JP2023553839A - Bidirectional optical flow in video coding - Google Patents

Abstract

A method for decoding video data includes determining that bidirectional optical flow (BDOF) is enabled for a block of video data and a block based on the determination that BDOF is enabled for a block of video data. dividing into a plurality of sub-blocks, determining a respective strain value for each sub-block of the one or more sub-blocks of the plurality of sub-blocks; determining, for each sub-block of the one or more sub-blocks, one of performing pixel-wise BDOF or bypassing BDOF based on the respective distortion values; determining predicted samples for each sub-block of the one or more sub-blocks based on a determination that BDOF is performed or BDOF is bypassed; and reconstructing the block based on the predicted samples. including.

Description

This application claims priority to U.S. Application No. 17/645,233, filed on December 20, 2021, and U.S. Provisional Application No. 63/129,190, filed on December 22, 2020, each of which , the entire contents of which are incorporated herein by reference. U.S. Application No. 17/645,233 claims the benefit of U.S. Provisional Application No. 63/129,190, filed on December 22, 2020.

TECHNICAL FIELD This disclosure relates to video encoding and video decoding.

Digital video capabilities include digital television, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video It can be incorporated into a wide variety of devices, including gaming devices, video game consoles, cellular or satellite radio telephones, so-called "smartphones," video teleconferencing devices, video streaming devices, and the like. Digital video devices include MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265/High Efficiency Implements video coding techniques, such as those described in the standards defined by HEVC, and extensions of such standards. By implementing such video coding techniques, video devices may more efficiently transmit, receive, encode, decode, and/or store digital video information.

Video coding techniques include spatial (intra-picture) prediction and/or temporal (inter-picture) prediction to reduce or eliminate redundancy inherent in video sequences. For block-based video coding, a video slice (e.g., a video picture, or a portion of a video picture) may be partitioned into video blocks, which are divided into coding tree units (CTUs), coding units (CUs), and /or sometimes called a coding node. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction on reference samples in adjacent blocks in the same picture. Video blocks in an inter-coded (P or B) slice of a picture use spatial prediction with respect to reference samples in adjacent blocks in the same picture, or temporal prediction with respect to reference samples in other reference pictures. obtain. A picture is sometimes called a frame, and a reference picture is sometimes called a reference frame.

“Series H: Audiovisual and Multimedia Systems, Infrastructure of Audiovisual Services-Coding of Moving Video, High efficiency Video Coding”, International Telecommunication Union, December 2016, 664 pages. VVC Test Model 10 (VTM10.0), https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM Bross et al., “Versatile Video Coding (Draft 10)”, Joint Video Expert Team (JVET) of ITU-T SG16 WP 3 and ISO/IEC JTC 1/SC29/WG11, 18th meeting via remote conference, June 2020. July 22nd to July 1st, JVET-S2001-vA Bross et al., “Versatile Video Coding Editorial Refinements on Draft 10”, Joint Video Expert Team (JVET) of ITU-T SG16 WP 3 and ISO/IEC JTC 1/SC29/WG11, 20th meeting via remote conference, 2020. October 7th to 16th, JVET-T2001-v2 J.Chen, Y.Ye, and S.Kim, "Algorithm description for Versatile Video Coding and Test Model 11 (VTM 11)", JVET-T2002, December 2020

In general, the present disclosure describes decoder-side motion vector derivation (e.g., template matching, bidirectional matching, decoder-side motion vector (MV) refinement), and/or bi-directional optical flow (BDOF) flow)). The techniques of this disclosure may be applied to any of the existing video codecs, such as HEVC (High Efficiency Video Coding), VVC (Versatile Video Coding), Essential Video Coding (EVC), or may be an efficient coding tool in any future video coding standard.

In one or more examples, for BDOF, video encoders and video decoders (e.g., video coders) may determine whether pixel-by-pixel BDOF is performed on subblocks of the block or whether BDOF is bypassed. may be configured to selectively determine whether or not. That is, the video coder may select one of the pixel-by-pixel BDOF, or the pixel-by-pixel BDOF (or BDOF in general) being bypassed. In this way, the example techniques, when combined together (e.g., one of the pixel-by-pixel BDOF is performed for the sub-block or the BDOF is bypassed for the sub-block), (as determined by the video coder) may facilitate selection between coding modes that may result in better coding performance.

Moreover, in some examples, deciding whether to perform pixel-wise BDOF or bypass BDOF on a subblock is based on determining a strain value and comparing the strain value to a threshold. It's fine. In some examples, the video coder determines the distortion values in such a way that the calculations used to determine the distortion values can be reused by the video coder when performing pixel-wise BDOF. can be configured. For example, if a video coder is to perform pixel-wise BDOF, the video coder may reuse the results from the calculations performed to determine the distortion values to perform pixel-wise BDOF.

In one example, this disclosure describes a method of decoding video data, the method comprising: determining that bidirectional optical flow (BDOF) is enabled for a block of video data; dividing the block into a plurality of subblocks based on a determination that the subblock is enabled; and determining a respective strain value for each subblock of the one or more subblocks of the plurality of subblocks. and for each subblock of one or more of the subblocks, one of the following: pixel-wise BDOF is performed or BDOF is bypassed. determining prediction samples for each subblock of the one or more subblocks based on a determination that pixel-wise BDOF is performed or BDOF is bypassed; and reconstructing the block based on the samples.

In one example, this disclosure describes a device for decoding video data, the device comprising a memory configured to store video data and processing circuitry coupled to the memory, the processing circuitry comprising: , determining that bidirectional optical flow (BDOF) is enabled for the block of video data, and dividing the block into a plurality of subblocks based on the determination that BDOF is enabled for the block; determining a respective strain value for each subblock of the one or more subblocks of the plurality of subblocks; and determining a respective strain value for each subblock of the one or more subblocks of the plurality of subblocks; , determine one of whether per-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values, and based on the determination that per-pixel BDOF is performed or BDOF is bypassed. is configured to determine predictive samples for each sub-block of the one or more sub-blocks and to reconstruct the block based on the predictive samples.

In one example, this disclosure describes a computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to perform a bidirectional optical flow (BDOF) operation on a block of video data. ) determines that BDOF is enabled, causes the block to be split into multiple subblocks based on the determination that BDOF is enabled for the block, and one or more of the multiple subblocks A respective distortion value is determined for each sub-block of the plurality of sub-blocks, and for each sub-block of one or more sub-blocks of the plurality of sub-blocks, a pixel-wise BDOF is performed or the BDOF is bypassed. for each sub-block of one or more sub-blocks based on the decision that pixel-wise BDOF is performed or BDOF is bypassed. the predicted samples are determined, and the block is reconstructed based on the predicted samples.

In one example, this disclosure describes a device for decoding video data, the device comprising: means for determining that bidirectional optical flow (BDOF) is enabled for a block of video data; means for dividing the block into a plurality of subblocks based on a determination that BDOF is enabled for the plurality of subblocks, and respectively for each subblock of the one or more subblocks of the plurality of subblocks; and whether pixel-wise BDOF is performed or BDOF is bypassed for each sub-block of one or more of the plurality of sub-blocks. for each sub-block of the one or more sub-blocks based on the determination that pixel-wise BDOF is performed or BDOF is bypassed. and means for reconstructing the block based on the predicted samples.

The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

FIG. 1 is a block diagram illustrating an example video encoding and decoding system that may implement techniques of this disclosure. 1 is a conceptual diagram illustrating an example quadtree binary tree (QTBT) structure and corresponding coding tree unit (CTU); FIG. 1 is a conceptual diagram illustrating an example quadtree binary tree (QTBT) structure and a corresponding coding tree unit (CTU). FIG. FIG. 1 is a block diagram illustrating an example video encoder that may implement the techniques of this disclosure. 1 is a block diagram illustrating an example video decoder that may implement techniques of this disclosure. FIG. FIG. 3 is a conceptual diagram illustrating an example of spatially adjacent motion vector candidates for merge mode. FIG. 2 is a conceptual diagram illustrating an example of spatially adjacent motion vector candidates for advanced motion vector predictor (AMVP) mode. FIG. 2 is a conceptual diagram showing an example of a temporal motion vector predictor (TMVP) candidate. FIG. 3 is a conceptual diagram showing an example of motion vector scaling. FIG. 2 is a conceptual diagram showing template matching performed on a search area around an initial motion vector (MV). FIG. 3 is a conceptual diagram showing an example of a motion vector difference proportional to time distance. FIG. 3 is a conceptual diagram showing an example of a motion vector difference that is reflected like a mirror regardless of time distance. FIG. 3 is a conceptual diagram showing an example of a 3×3 square search pattern within the search range [-8,8]. FIG. 3 is a conceptual diagram showing an example of decoding side motion vector improvement. FIG. 2 is a conceptual diagram illustrating an extended coding unit (CU) used in bidirectional optical flow (BDOF). 2 is a flowchart illustrating an example process for pixel-wise BDOF with sub-block bypass. FIG. 3 is a conceptual diagram showing an example of a pixel-by-pixel BDOF of 8×8 sub-blocks. 3 is a flowchart illustrating an example method for decoding a current block in accordance with techniques of this disclosure. 2 is a flowchart illustrating an example method for encoding a current block in accordance with techniques of this disclosure.

A video encoder may be configured to generate a predictive block from one or more reference blocks in one or more reference pictures using one or more motion vectors for the block. A video encoder determines a residual between a predictive block and that block and signals information indicative of the residual and information used to determine a motion vector. A video decoder receives information indicative of residuals and information used to determine motion vectors. A video decoder determines a motion vector, determines a reference block from the motion vector, and generates a predictive block. A video decoder adds the predictive block to the residual to reconstruct the block.

In some cases, the reference block and prediction block are the same block. However, the reference block and prediction block may not be required to be the same in all instances. In some examples, such as in bi-prediction, video encoders and video decoders determine a first reference block based on a first motion vector and a second reference block based on a second motion vector. It's fine. A video encoder and a video decoder may blend the first reference block and the second reference block to generate a predictive block.

Moreover, in some examples, the video encoder and video decoder may generate predictive blocks based on adjustments to sample values of the first reference block and the second reference block. One example method for adjusting sample values to generate samples for a predictive block is called bidirectional optical flow (BDOF). For example, assume that I ⁽⁰⁾ (x,y) points to a first reference block and I ⁽¹⁾ (x,y) points to a second reference block. In BDOF, the predictive block may be considered as I ⁽⁰⁾ (x,y) plus I ⁽¹⁾ (x,y). As explained below, video encoders and video decoders may determine an adjustment factor (i.e., b(x,y)) as part of the process of determining prediction samples, and add the adjustment factor to the prediction block. (i.e., I ⁽⁰⁾ (x,y) + I ⁽¹⁾ (x,y) + b(x,y)). There may be additional scaling and offsetting of the result of I ⁽⁰⁾ (x,y) + I ⁽¹⁾ (x,y) + b(x,y) to determine the predicted samples. .

In BDOF, video encoders and video decoders utilize motion vectors to determine adjustment factors (eg, factors that are multiplied or added to) to adjust sample values of predictive blocks to generate predictive samples. As an example, video encoders and video decoders generate predicted samples by adding corresponding samples of a first reference block, corresponding samples of a second reference block, and corresponding values generated from motion improvement. You may do so.

There may be various types of BDOF techniques. One example of BDOF is sub-block BDOF, and another example of a BDOF technique is pixel-by-pixel BDOF. In a sub-block BDOF, the video encoder and video decoder determine motion improvements (also referred to as improved motion) for the sub-block. For sub-block BDOF, the video encoder and video decoder use the same motion improvement to adjust the samples from the prediction block, where the prediction block uses the first reference block and the second reference block. (eg, the sum of the first reference block and the second reference block, or the weighted average of the first reference block and the second reference block). In pixel-by-pixel BDOF, video encoders and video decoders may determine motion improvement factors that may be different for two or more samples in the current block. For per-pixel BDOF, video encoders and video decoders may adjust the samples from the predictive block using the motion improvement (also referred to as improved motion) determined for the pixel-by-pixel samples, and the predictive block It may be generated using a first reference block and a second reference block.

BDOF or other improvement techniques may be selectively enabled at the block level, but whether BDOF is applied at the sub-block level may be estimated based on distortion values. For example, a video encoder may enable BDOF for a block and may signal information indicating that BDOF is enabled for the block.

In response, the video decoder may divide the block into multiple subblocks based on the determination that BDOF is enabled for the block. Although BDOF is enabled for a block, the video decoder may decide for each sub-block whether BDOF is actually to be performed or bypassed. For example, a video decoder determines a respective distortion value for each subblock of one or more of the plurality of subblocks.

According to one or more examples described in this disclosure, a video decoder performs pixel-by-pixel BDOF for each sub-block of one or more of the plurality of sub-blocks. One of the BDOFs to be bypassed may be determined based on the respective distortion values. For example, the video decoder may determine a first distortion value for a first sub-block, and may determine based on the first distortion value that pixel-wise BDOF is performed on the first sub-block. It's fine. The video decoder may determine a second distortion value for the second sub-block, and may determine based on the second distortion value that the BDOF is bypassed for the second sub-block, and the following: The same is true.

In one or more examples, if the video decoder determines that BDOF is performed, the video decoder may perform pixel-by-pixel BDOF and other BDOF techniques may not be available to the video decoder. . That is, the video decoder may determine for each sub-block one of: pixel-by-pixel BDOF is performed or BDOF is bypassed for each sub-block. When BDOF is implemented, the BDOF technique available to the video decoder may be pixel-by-pixel BDOF, and no other BDOF techniques may be available.

In one or more examples, as described above, the video decoder may provide distortion values for each subblock to determine whether pixel-by-pixel BDOF is performed or whether BDOF is bypassed. You may decide. In some examples, as described in more detail below, the video decoder reuses the calculations used to determine the distortion values to determine the per-pixel motion improvement relative to the per-pixel BDOF. good. For example, for a first sub-block, a video decoder may determine a first distortion value. Assume that the video decoder has decided that for the first sub-block, per-pixel BDOF is enabled. In some instances, rather than recalculating all the values needed to determine per-pixel motion improvement, the video decoder uses Results from the performed calculations may be configured to be reused to determine pixel-by-pixel motion improvement.

The video decoder may be configured to determine prediction samples for each subblock of the one or more subblocks based on a determination that pixel-by-pixel BDOF is performed or BDOF is bypassed. For example, assume that pixel-by-pixel BDOF is performed on a sub-block. In this example, the video decoder improves the predictive samples for the sub-block by improving the samples of the predictive block (e.g., the block generated from combining two reference blocks) based on pixel-by-pixel motion improvement. May be generated. As another example, assume that BDOF is bypassed for a sub-block. In this example, the video decoder does not have to perform enhancement of the samples of the predictive block to generate predictive samples. Rather, the samples of the prediction block may be the same as the prediction samples (or possibly with some non-BDOF-based adjustments). For example, when BDOF is bypassed, the video encoder and video decoder may generate predicted samples by determining a weighted average of corresponding samples in the first reference block and the second reference block. .

A video decoder may reconstruct blocks based on the predicted samples. For example, a video decoder may receive residual values indicating differences between predicted samples and samples of a block, and may add the residual values to the predicted samples to reconstruct the block. The above example is described from the perspective of a video decoder. Video encoders may be configured to perform similar techniques. For example, the predicted samples produced by a video decoder should be the same as the predicted samples produced by a video encoder. Accordingly, a video encoder may perform techniques similar to those described above to determine prediction samples in the same way as a video decoder.

FIG. 1 is a block diagram illustrating an example video encoding and decoding system 100 that may implement the techniques of this disclosure. The techniques of this disclosure are generally directed to coding (encoding and/or decoding) video data. Generally, video data includes any data for processing video. Accordingly, video data may include raw unencoded video, encoded video, decoded (eg, reconstructed) video, and video metadata such as signaling data.

As shown in FIG. 1, system 100 includes, in this example, a source device 102 that provides encoded video data to be decoded and displayed by a destination device 116. In particular, source device 102 provides video data to destination device 116 via computer-readable medium 110. Source device 102 and destination device 116 may include desktop computers, notebook (i.e., laptop) computers, mobile devices, tablet computers, set-top boxes, telephone handsets such as smartphones, televisions, cameras, display devices, digital media players, It may comprise any of a wide variety of devices, including video gaming consoles, video streaming devices, broadcast receiver devices, and the like. In some cases, source device 102 and destination device 116 may be equipped for wireless communication and may therefore be referred to as wireless communication devices.

In the example of FIG. 1, source device 102 includes video source 104, memory 106, video encoder 200, and output interface 108. Destination device 116 includes input interface 122, video decoder 300, memory 120, and display device 118. According to this disclosure, the video encoder 200 of the source device 102 and the video decoder 300 of the destination device 116 perform decoder side motion vector (MV) improvement, such as template matching, bidirectional matching, decoder side motion vector (MV) improvement, and bidirectional optical flow. The method may be configured to apply techniques for vector derivation techniques. Thus, source device 102 represents an example of a video encoding device, and destination device 116 represents an example of a video decoding device. In other examples, the source device and destination device may include other components or configurations. For example, source device 102 may receive video data from an external video source, such as an external camera. Similarly, destination device 116 may interface with an external display device rather than include an integrated display device.

A system 100 as shown in FIG. 1 is only one example. In general, any digital video encoding and/or decoding device supports decoder-side motion vector derivation techniques, such as template matching, bidirectional matching, decoder-side motion vector (MV) improvement, and bidirectional optical flow (BDOF). You may perform this technique. Source device 102 and destination device 116 are only examples of coding devices, such as source device 102 generating coded video data for transmission to destination device 116. This disclosure refers to "coding" devices as devices that perform coding (encoding and/or decoding) of data. Accordingly, video encoder 200 and video decoder 300 represent examples of coding devices, specifically video encoders and video decoders, respectively. In some examples, source device 102 and destination device 116 may operate substantially symmetrically, such that each of source device 102 and destination device 116 includes video encoding and decoding components. Thus, system 100 may support one-way or two-way video transmission between source device 102 and destination device 116, for example, for video streaming, video playback, video broadcasting, or video telephony.

Generally, video source 104 represents a source of video data (i.e., raw, unencoded video data) that provides a continuous series of pictures (also referred to as “frames”) of video data to video encoder 200. , video encoder 200 encodes data for pictures. Video source 104 of source device 102 includes a video capture device, such as a video camera, a video archive containing previously captured raw video, and/or a video feed interface for receiving video from a video content provider. good. As a further alternative, video source 104 may produce computer graphics-based data as the source video, or a combination of live video, archived video, and computer-generated video. In each case, video encoder 200 encodes captured video data, pre-captured video data, or computer-generated video data. Video encoder 200 may reorder the pictures from the order in which they were received (sometimes referred to as the "display order") to a coding order for coding. Video encoder 200 may generate a bitstream that includes encoded video data. Source device 102 may then output encoded video data via output interface 108 onto computer-readable medium 110 for reception and/or retrieval by input interface 122 of destination device 116, for example.

Memory 106 of source device 102 and memory 120 of destination device 116 represent general purpose memory. In some examples, memories 106, 120 may store raw video data, eg, raw video from video source 104 and raw decoded video data from video decoder 300. Additionally or alternatively, memories 106, 120 may store software instructions executable by video encoder 200 and video decoder 300, respectively, for example. Although memory 106 and memory 120 are shown separately from video encoder 200 and video decoder 300 in this example, video encoder 200 and video decoder 300 may also use internal memory for functionally similar or equivalent purposes. It should be understood that this may include Additionally, memories 106, 120 may store encoded video data, eg, output from video encoder 200 and input to video decoder 300. In some examples, portions of memory 106, 120 may be allocated as one or more video buffers, eg, for storing raw decoded and/or encoded video data.

Computer-readable medium 110 may represent any type of medium or device that can transport encoded video data from source device 102 to destination device 116. In one example, computer-readable medium 110 represents a communication medium that enables source device 102 to transmit encoded video data in real-time directly to destination device 116, such as over a radio frequency network or a computer-based network. . Output interface 108 may modulate the transmitted signal containing encoded video data and input interface 122 may demodulate the received transmitted signal in accordance with a communication standard, such as a wireless communication protocol. The communication medium may comprise any wireless or wired communication medium, such as the radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. Communication media may include routers, switches, base stations, or any other equipment that may be useful to facilitate communication from source device 102 to destination device 116.

In some examples, source device 102 may output encoded data from output interface 108 to storage device 112. Similarly, destination device 116 may access encoded data from storage device 112 via input interface 122. Storage device 112 can be various, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. The data storage media may include either distributed data storage media or locally accessed data storage media.

In some examples, source device 102 may output encoded video data to file server 114 or another intermediate storage device that may store encoded video data generated by source device 102. Destination device 116 may access stored video data from file server 114 via streaming or downloading.

File server 114 may be any type of server device capable of storing encoded video data and transmitting the encoded video data to destination device 116. File server 114 is a web server (e.g., for a website), and provides file transfer protocol services (such as File Transfer Protocol (FTP) or File Delivery over Unidirectional Transport (FLUTE) protocol). a server, content delivery network (CDN) device, hypertext transfer protocol (HTTP) server, multimedia broadcast multicast service (MBMS) or enhanced MBMS (eMBMS) server, and/or network attached storage (NAS) configured to May represent a device. File server 114 may additionally or alternatively support Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Real Time Streaming Protocol (RTSP), HTTP One or more HTTP streaming protocols may be implemented, such as dynamic streaming.

Destination device 116 may access encoded video data from file server 114 through any standard data connection, including an Internet connection. This can be used to access encoded video data stored on a wireless channel (e.g., Wi-Fi connection), a wired connection (e.g., digital subscriber line (DSL), cable modem, etc.), or on a file server 114. May include suitable combinations of both. Input interface 122 supports any one or more of the various protocols described above for retrieving or receiving media data from file server 114 or other such protocols for retrieving media data. may be configured to operate according to.

Output interface 108 and input interface 122 may be a wireless transmitter/receiver, modem, wired networking component (e.g., an Ethernet card), wireless communication component operating according to any of various IEEE 802.11 standards, or other may represent a physical component of In examples where the output interface 108 and the input interface 122 comprise wireless components, the output interface 108 and the input interface 122 provide encoded video according to cellular communication standards such as 4G, 4G-LTE (Long Term Evolution), LTE Advanced, 5G, etc. The device may be configured to transfer data such as data. In some examples where output interface 108 comprises a wireless transmitter, output interface 108 and input interface 122 may be compatible with other standards such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee(TM)), the Bluetooth(TM) standard, etc. may be configured to transfer data, such as encoded video data, in accordance with a wireless standard. In some examples, source device 102 and/or destination device 116 may include respective system-on-chip (SoC) devices. For example, source device 102 may include an SoC device for performing functionality due to video encoder 200 and/or output interface 108, and destination device 116 may include a video decoder 300 and/or input interface 122. May include an SoC device to perform the functionality.

The techniques of this disclosure can be applied to over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet streaming video transmissions such as Dynamic Adaptive Streaming over HTTP (DASH), and digital video encoded on a data storage medium. It may be applied to video coding to support any of a variety of multimedia applications, such as video, decoding of digital video stored on a data storage medium, or other applications.

Input interface 122 of destination device 116 receives an encoded video bitstream from computer-readable medium 110 (eg, a communication medium, storage device 112, file server 114, etc.). The encoded video bitstream is encoded by video decoder 300, such as syntax elements having values that describe the characteristics and/or processing of video blocks or other coded units (e.g., slices, pictures, picture groups, sequences, etc.). may also include signaling information used and defined by video encoder 200. Display device 118 displays decoded pictures of the decoded video data to the user. Display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

Although not shown in FIG. 1, in some examples, video encoder 200 and video decoder 300 may each be integrated with an audio encoder and/or audio decoder to include both audio and video in a common data stream. It may include a suitable MUX-DEMUX unit or other hardware and/or software to process the multiplexed streams. If applicable, the MUX-DEMUX unit may comply with the ITU H.223 multiplexer protocol or other protocols such as User Datagram Protocol (UDP).

Video encoder 200 and video decoder 300 each include one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, May be implemented as any of a variety of suitable encoder and/or decoder circuitry, such as firmware, or any combination thereof. When the techniques are partially implemented in software, the device may store instructions for the software on a suitable non-transitory computer-readable medium and implement one or more processors to execute the techniques of this disclosure. can be used to execute instructions in hardware. That is, there may be a computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to perform the example techniques described in this disclosure. Each of video encoder 200 and video decoder 300 may be included within one or more encoders or decoders, any of which may be integrated within their respective devices as part of a combined encoder/decoder (CODEC). It's okay to be. Devices including video encoder 200 and/or video decoder 300 may include integrated circuits, microprocessors, and/or wireless communication devices such as cellular telephones.

The following describes video coding standards. Video coding standards include ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-T H.262 or ISO/IEC, including its Scalable Video Coding (SVC) and Multiview Video Coding (MVC) extensions. Includes MPEG-2 Visual, ITU-T H.263, ISO/IEC MPEG-4 Visual, and ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC). In addition, High Efficiency Video Coding (HEVC) or ITU-T H.265, including its range extension, multi-view extension (MV-HEVC), and scalable extension (SHVC), has been approved by the Joint Video Coding Working Group (JCT-VC). ) and the Joint Working Group for 3D Video Coding Extensions (JCT-3V) between the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Motion Picture Experts Group (MPEG). HEVC specification available from ITU-T H.265, “Series H: Audiovisual and Multimedia Systems, Infrastructure of Audiovisual Services-Coding of Moving Video, High efficiency Video Coding”, International Telecommunication Union, December 2016, p. 664 It is possible.

ITU-T VCEG (Q6/16) and ISO/IEC MPEG (JTC 1/SC29/WG11) define the current HEVC standard, including its current and upcoming extensions for screen content coding and high dynamic range coding. The standardization of future video coding technologies with compression capabilities significantly exceeding those of The group worked together on this exploration effort in a joint research effort called the Joint Video Exploration Group (JVET) to evaluate compression technology designs proposed by those experts in this area. It's inside. The latest version of the reference software, namely VVC Test Model 10 (VTM10.0), can be downloaded from https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM.

Video encoder 200 and video decoder 300 operate according to a video coding standard, such as ITU-T H.265, also referred to as High Efficiency Video Coding (HEVC), or extensions thereto such as multi-view and/or scalable video coding extensions. obtain. Alternatively, video encoder 200 and video decoder 300 may operate according to other proprietary or industry standards, such as ITU-T H.266, also referred to as Versatile Video Coding (VVC). The draft of the VVC standard was published by Bross et al., "Versatile Video Coding (Draft 10)", Joint Video Expert Team (JVET) of ITU-T SG16 WP 3 and ISO/IEC JTC 1/SC29/WG11, No. 18 by teleconference. Meeting, June 22, 2020 - July 1, 2020, JVET-S2001-vA (hereinafter referred to as "VVC Draft 10"). Editorial refinements of VVC Draft 10 are presented in Bross et al., “Versatile Video Coding Editorial Refinements on Draft 10,” Joint Video Expert Team (JVET) between ITU-T SG16 WP 3 and ISO/IEC JTC 1/SC29/WG11, remote conference. 20th Meeting, October 7-16, 2020, JVET-T2001-v2. Algorithm description for Versatile Video Coding and Test Model 11 (VTM 11), J. Chen, Y. Ye, and S. Kim, “Algorithm description for Versatile Video Coding and Test Model 11 (VTM 11),” JVET- T2002, December 2020 (hereinafter referred to as JVET-T2002). However, the techniques of this disclosure are not limited to any particular coding standard.

Generally, video encoder 200 and video decoder 300 may perform block-based coding of pictures. The term "block" generally refers to a structure containing data to be processed (eg, encoded, decoded, or otherwise used in an encoding and/or decoding process). For example, a block may include a two-dimensional matrix of samples of luminance and/or chrominance data. Generally, video encoder 200 and video decoder 300 may code video data represented in YUV (eg, Y, Cb, Cr) format. That is, rather than coding red, green, and blue (RGB) data for samples of a picture, video encoder 200 and video decoder 300 may code luminance and chrominance components, where the chrominance component may include chrominance components in both the red and blue hues. In some examples, video encoder 200 converts the received RGB formatted data to a YUV representation and video decoder 300 converts the YUV representation to RGB format before encoding. Alternatively, a pre-processing unit and a post-processing unit (not shown) may perform these transformations.

This disclosure may generally refer to coding (eg, encoding and decoding) pictures to include the process of encoding or decoding data for a picture. Similarly, this disclosure may refer to coding blocks of pictures, such as predictive and/or residual coding, to include the process of encoding or decoding data for the blocks. A coded video bitstream typically includes a series of values for syntax elements representing coding decisions (eg, coding modes) and partitioning of pictures into blocks. Accordingly, references to coding a picture or block should generally be understood as coding values for the syntax elements forming the picture or block.

HEVC defines various blocks, including coding units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video coder (such as video encoder 200) partitions coding tree units (CTUs) into CUs according to a quadtree structure. That is, the video coder partitions the CTU and CU into four equal non-overlapping squares, where each node in the quadtree has either 0 or 4 child nodes. A node without child nodes may be referred to as a "leaf node," and the CU of such a leaf node may include one or more PUs and/or one or more TUs. A video coder may further partition PUs and TUs. For example, in HEVC, a residual quadrant tree (RQT) represents a partition of a TU. In HEVC, PU represents inter-predicted data and TU represents residual data. The intra-predicted CU includes intra-prediction information such as intra-mode indication.

As another example, video encoder 200 and video decoder 300 may be configured to operate according to VVC. According to VVC, a video coder (such as video encoder 200) partitions a picture into multiple coding tree units (CTUs). Video encoder 200 may partition CTUs according to a tree structure, such as a quadrilateral tree (QTBT) structure or a multitype tree (MTT) structure. The QTBT structure eliminates the concept of multiple partition types, such as the separation between CU, PU, and TU in HEVC. The QTBT structure includes two levels: a first level partitioned according to quadtree partitioning, and a second level partitioned according to binary partitioning. The root node of the QTBT structure corresponds to the CTU. Leaf nodes of the binary tree correspond to coding units (CUs).

In an MTT partition structure, blocks are divided into quadtree (QT) partitions, binary tree (BT) partitions, and one or more types of triple tree (TT) partitions. ) can be partitioned using partitions. A ternary tree partition or ternary tree partition is a partition in which a block is divided into three subblocks. In some examples, ternary tree partitioning or ternary tree partitioning partitions a block into three subblocks without splitting the original block through the center. The partition types in MTT (eg, QT, BT, and TT) can be symmetric or asymmetric.

In some examples, video encoder 200 and video decoder 300 may use a single QTBT or MTT structure to represent each of the luminance and chrominance components, while in other examples, video encoder 200 and video The decoder 300 includes two QTBT/MTT structures, such as one QTBT/MTT structure for the luminance component and another QTBT/MTT structure for both chrominance components (or two QTBT/MTT structures for each chrominance component). More than one QTBT or MTT structure may be used.

Video encoder 200 and video decoder 300 may be configured to use quadtree partitioning with HEVC, QTBT partitioning, MTT partitioning, or other partitioning structures. For purposes of explanation, descriptions of the techniques of this disclosure are presented in terms of the QTBT partition. However, it should be understood that the techniques of this disclosure may also be applied to video coders configured to use quadtree partitioning or other types of partitioning as well.

In some examples, a CTU is used to code a coding tree block (CTB) of luma samples, two corresponding CTBs of chroma samples of a picture with three sample arrays, or a monochrome picture, or samples. Contains sample CTBs of pictures coded using three separate color planes and syntax structures. A CTB may be an N×N block of samples for some value of N, such that the division of components into CTBs is piecewise. The components can be arrays or singles from one of the three arrays (luma and two chromas) that make up a picture in 4:2:0, 4:2:2, or 4:4:4 color format. An array of samples or a single sample of an array that creates a picture in monochrome format. In some examples, the coding block is an M×N block of samples for several values of M and N, such that the division of the CTB into coding blocks is partitioned.

Blocks (eg, CTUs or CUs) may be grouped in various ways within a picture. As an example, a brick may refer to a rectangular area of a CTU row within a particular tile within a picture. A tile may be a rectangular region of a CTU within a particular tile column and a particular tile row within a picture. A tile column refers to a rectangular region of a CTU with a height equal to the height of a picture and a width specified by a syntax element (eg, in a picture parameter set). A tile row refers to a rectangular region of the CTU with a height specified by a syntax element (eg, in a picture parameter set) and a width equal to the width of the picture.

In some examples, a tile may be partitioned into multiple bricks, and each brick may include one or more CTU rows within the tile. Tiles that are not divided into multiple bricks may also be called bricks. However, bricks that are a true subset of tiles may not be called tiles.

Bricks within a picture may also be arranged into slices. A slice may be an integer number of bricks of a picture that may be contained exclusively within a single network abstraction layer (NAL) unit. In some examples, a slice includes either a number of complete tiles or only a contiguous sequence of complete bricks of one tile.

This disclosure uses "N×N" and "N by N" interchangeably to refer to the sample dimension of a block (such as a CU or other video block) in terms of vertical and horizontal dimensions. ”, for example, 16×16 samples or 16 by 16 samples may be used. Generally, a 16x16 CU has 16 samples vertically (y=16) and 16 samples horizontally (x=16). Similarly, an N×N CU generally has N samples vertically and N samples horizontally, where N represents a non-negative integer value. Samples within a CU may be arranged in rows and columns. Moreover, a CU does not necessarily need to have the same number of samples in the horizontal direction as in the vertical direction. For example, a CU may comprise N×M samples, where M is not necessarily equal to N.

Video encoder 200 encodes video data for a CU representing prediction and/or residual information, as well as other information. Prediction information indicates how a CU is to be predicted to form a prediction block for the CU. The residual information generally represents the sample-by-sample difference between the samples of the CU and the samples of the predictive block before encoding.

To predict a CU, video encoder 200 may generally form a prediction block for the CU through inter-prediction or intra-prediction. Inter prediction generally refers to predicting a CU from data of a previously coded picture, and intra prediction generally refers to predicting a CU from previously coded data of the same picture. To perform inter prediction, video encoder 200 may use one or more motion vectors to generate a predictive block. Video encoder 200 may generally perform motion searching, for example, to identify reference blocks that closely match the CU with respect to the difference between the CU and the reference block. Video encoder 200 uses sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference ( MSD), or other such difference calculations may be used to calculate the difference metric. In some examples, video encoder 200 may predict the current CU using unidirectional prediction or bidirectional prediction.

Some examples of VVC also provide an affine motion compensation mode, which can be considered an inter-prediction mode. In affine motion compensation mode, video encoder 200 may determine two or more motion vectors representing non-translational motion, such as zooming in or out, rotation, perspective movement, or other irregular motion types.

To perform intra prediction, video encoder 200 may select an intra prediction mode to generate predictive blocks. Some examples of VVC provide 67 intra-prediction modes, including various directional modes, as well as planar and DC modes. Generally, video encoder 200 selects an intra prediction mode that describes adjacent samples to a current block (eg, a block of a CU) from which samples of the current block are to be predicted. Such samples are generally above, above, and above the current block in the same picture as the current block, assuming that video encoder 200 codes CTUs and CUs in raster scan order (from left to right, top to bottom). and may be on the left or on the left.

Video encoder 200 encodes data representing the prediction mode for the current block. For example, for inter-prediction modes, video encoder 200 may encode data representative of which of the various available inter-prediction modes will be used, as well as motion information for the corresponding mode. For unidirectional or bidirectional inter-prediction, for example, video encoder 200 may encode motion vectors using advanced motion vector prediction (AMVP) mode or merge mode. Video encoder 200 may use a similar mode to encode motion vectors for affine motion compensation mode.

Following prediction, such as intra-prediction or inter-prediction, of a block, video encoder 200 may calculate residual data for the block. Residual data, such as a residual block, represents the sample-by-sample difference between a block and a predicted block for that block formed using a corresponding prediction mode. Video encoder 200 may apply one or more transforms to the residual block to generate transform data in the transform domain rather than the sample domain. For example, video encoder 200 may apply a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to the residual video data. Additionally, video encoder 200 may perform a secondary transform, such as a mode-dependent non-separable secondary transform (MDNSST), a signal-dependent transform, or a Karhunen-Loeve transform (KLT), as the first transform. May be applied following. Video encoder 200 generates transform coefficients following application of one or more transforms.

As described above, following any transformation to generate transform coefficients, video encoder 200 may perform quantization of the transform coefficients. Quantization generally refers to a process in which transform coefficients are quantized to reduce as much as possible the amount of data used to represent the transform coefficients, providing further compression. By performing a quantization process, video encoder 200 may reduce the bit depth associated with some or all of the transform coefficients. For example, video encoder 200 may truncate an n-bit value to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, video encoder 200 may perform a bitwise right shift of the value to be quantized.

Following quantization, video encoder 200 may scan the transform coefficients to generate a one-dimensional vector from the two-dimensional matrix containing the quantized transform coefficients. The scan may be designed to place higher energy (and therefore lower frequency) transform coefficients at the front of the vector and lower energy (and therefore higher frequency) transform coefficients at the back of the vector. In some examples, video encoder 200 may utilize a default scanning order to scan the quantized transform coefficients to generate a serialized vector, and then entropy encode the quantized transform coefficients of the vector. good. In other examples, video encoder 200 may perform adaptive scanning. After scanning the quantized transform coefficients to form a one-dimensional vector, video encoder 200 may entropy encode the one-dimensional vector, for example, according to context-adaptive binary arithmetic coding (CABAC). Video encoder 200 may also entropy encode values for syntax elements that describe metadata associated with encoded video data for use by video decoder 300 in decoding the video data.

To perform CABAC, video encoder 200 may assign contexts in a context model to symbols to be transmitted. The context may relate, for example, to whether adjacent values of the symbol are zeroed. The probability determination may be based on the context assigned to the symbol.

Video encoder 200 may encode video data in a picture header, block header, slice header, or other syntax data such as a sequence parameter set (SPS), picture parameter set (PPS), or video parameter set (VPS), for example. Syntax data such as block-based syntax data, picture-based syntax data, and sequence-based syntax data to decoder 300 may also be generated. Video decoder 300 may similarly decode such syntax data to determine how to decode the corresponding video data.

In this manner, video encoder 200 includes syntax elements that describe encoded video data, e.g., partitioning of pictures into blocks (e.g., CUs) and prediction information and/or residual information for the blocks. A bitstream can be generated. Ultimately, video decoder 300 may receive the bitstream and decode the encoded video data.

Generally, video decoder 300 performs a reciprocal process to the process performed by video encoder 200 to decode encoded video data of a bitstream. For example, video decoder 300 may decode values for syntax elements of a bitstream using CABAC in a manner that is contrary to, but substantially similar to, the CABAC encoding process of video encoder 200. The syntax element may define partitioning information for partitioning pictures into CTUs and partitioning each CTU by a corresponding partitioning structure, such as a QTBT structure, to define CUs of CTUs. The syntax elements may further define prediction and residual information for a block of video data (eg, a CU).

The residual information may be represented by, for example, quantized transform coefficients. Video decoder 300 may dequantize and inverse transform the quantized transform coefficients of the block to reconstruct a residual block for the block. Video decoder 300 forms a prediction block for the block using the signaled prediction mode (intra-prediction or inter-prediction) and associated prediction information (eg, motion information for inter-prediction). Video decoder 300 may then combine the predictive block and the residual block (sample by sample) to reconstruct the original block. Video decoder 300 may perform additional processing, such as performing a deblocking process to reduce visual artifacts along block boundaries.

According to the techniques of this disclosure, video encoder 200 and video decoder 300 may be configured to perform bidirectional optical flow (BDOF). For example, video encoder 200 may be configured to perform BDOF as part of encoding the current block, and video decoder 300 may be configured to perform BDOF as part of decoding the current block. may be configured.

As described in more detail, in some examples, a video coder (e.g., video encoder 200 and/or video decoder 300) partitions an input block into multiple subblocks, the size of the input block being is smaller than or equal to the size of the coding unit, and bidirectional optical flow (BDOF) is to be applied to a subblock of the plurality of subblocks. determining an improved motion vector for one or more of the sub-sub-blocks, the method comprising: determining an improved motion vector for one or more of the sub-sub-blocks; The improved motion vector for a sub-subblock of a block is the same for multiple samples in the sub-subblock, and the improved motion vector for one or more sub-subblocks is and performing BDOF.

As another example, a video coder may partition an input block into multiple subblocks, the size of the input block being smaller than or equal to the size of the coding unit, and the size of the input block being smaller than or equal to the size of the coding unit. determining that bidirectional optical flow (BDOF) is to be applied to a sub-block based on the satisfaction of a condition; dividing the sub-block into multiple sub-sub-blocks; determining an improved motion vector for each of the one or more samples in the sub-block; and determining an improved motion vector for each of the one or more samples in the sub-block. and performing BDOF.

For example, as explained above, video encoder 200 or video decoder 300 may determine an improved motion vector for each of the one or more samples in the sub-block, and BDOF may be performed based on the improved motion vector for each of the one or more samples of . In this disclosure, performing BDOF based on the improved motion vector for each of one or more samples in a sub-block is referred to as "pixel-by-pixel BDOF." For example, in pixel-wise BDOF, instead of having one refined motion vector that is the same for all samples in a sub-block, the refined motion vector for each sample in a sub-block is determined separately. Ru.

An improved motion vector may not necessarily mean that the motion vector for the sub-block is changed. Rather, the improved motion vector for the sample may be used to determine the amount by which the samples in the prediction block are adjusted to generate the prediction sample. For example, for the first sample of the first sub-block, the first improved motion vector is how much the first sample in the prediction block should be adjusted to generate the first prediction sample. For the second sample of the first sub-block, the second improved motion vector indicates how much the second sample in the prediction block needs to be used to generate the second prediction sample. It may indicate whether the adjustment should be made, and the same applies hereafter.

According to one or more examples described in this disclosure, video encoder 200 and video decoder 300 determine a pixel-by-pixel BDOF for each sub-block of one or more sub-blocks of a block (e.g., an input block). One of execution or BDOF bypass may be determined based on the respective distortion values. For example, as explained above, video encoder 200 and video decoder 300 may perform pixel-by-pixel BDOF based on a condition being met. The condition being met may be whether the distortion value for the sub-block is greater than a threshold.

Thus, in some examples, the options for video encoder 200 and video decoder 300 are to determine whether the distortion value for a sub-block is greater than a threshold value or less than or equal to a threshold value. Can be configured to either perform pixel-by-pixel BDOF or bypass BDOF. For example, in some techniques it may be possible for video encoder 200 and video decoder 300 to perform pixel-by-pixel BDOF but not determine on a sub-block basis whether BDOF is bypassed. In some techniques where BDOF may be bypassed on a per-subblock basis, per-pixel BDOF may not have been available. Using the example techniques described in this disclosure, video encoder 200 and video decoder 300 may be configured to selectively perform pixel-by-pixel BDOF or bypass BDOF, as appropriate. Balances decoding overhead and may result in better video compression.

In one or more examples, to encode or decode video data, respectively, video encoder 200 and video decoder 300 determine that BDOF is enabled for a block of video data and that the block is Based on a determination that BDOF is enabled for a block, or more generally when BDOF is enabled for a block, the block may be configured to be divided into a plurality of sub-blocks. Video encoder 200 and video decoder 300 may determine respective distortion values for each subblock of one or more of the plurality of subblocks. Exemplary methods for determining respective strain values are described in more detail below. Video encoder 200 and video decoder 300 perform one of: pixel-by-pixel BDOF is performed or BDOF is bypassed for each sub-block of one or more of the plurality of sub-blocks. may be determined based on the respective distortion values, and the predicted samples for each sub-block of the one or more sub-blocks may be determined based on the determination that pixel-wise BDOF is performed or BDOF is bypassed. You may do so.

Video encoder 200 may determine a residual value that indicates the difference between the predicted samples and the samples of the block, and may signal the residual value. Video decoder 300 may receive residual values indicating the differences between the predicted samples and the samples of the block and may add the residual values to the predicted samples to reconstruct the block. In some examples, to receive the residual value, video decoder 300 may be configured to receive information indicative of the residual value, and video decoder 300 determines the residual value from such information. .

This disclosure may generally refer to "signaling" some information, such as syntax elements. The term "signaling" may generally refer to the communication of values for syntax elements and/or other data used to decode encoded video data. That is, video encoder 200 may signal values for syntax elements within the bitstream. Generally speaking, signaling refers to the generation of values within a bitstream. As mentioned above, the source device 102 may store the bitstream in substantially real-time or not in real-time, as may be done when storing the syntax elements on the storage device 112 for later retrieval by the destination device 116. and may be transported to destination device 116.

2A and 2B are conceptual diagrams illustrating an example quadtree binary tree (QTBT) structure 130 and a corresponding coding tree unit (CTU) 132. A solid line represents a quadtree partition, and a dotted line represents a binary tree partition. At each split (i.e., non-leaf) node of the binary tree, one flag is signaled to indicate which split type (i.e., horizontal or vertical) is used, where, in this example, 0 indicates horizontal division, 1 indicates vertical division. For quadtree partitioning, there is no need to indicate the partition type because the quadtree node partitions the block horizontally and vertically into four equal-sized subblocks. Therefore, syntax elements (such as partitioning information) for the region tree level (i.e., solid lines) of QTBT structure 130, and syntax elements (such as partitioning information) for the prediction tree level (i.e., dashed lines) of QTBT structure 130. Tax elements may be encoded by video encoder 200 and decoded by video decoder 300. Video encoder 200 may encode and video decoder 300 may decode video data, such as prediction data and transform data, for CUs represented by terminal leaf nodes of QTBT structure 130.

In general, CTU 132 of FIG. 2B may be associated with parameters that define the size of blocks corresponding to nodes of QTBT structure 130 at first and second levels. These parameters are CTU size (representing the size of CTU132 in the sample), minimum quadtree size (MinQTSize, representing the smallest allowed quadtree leaf node size), maximum binary size (MaxBTSize, ), the maximum binary tree depth (MaxBTDepth, which represents the maximum allowed binary tree root node size), and the minimum binary tree size (MinBTSize, which represents the minimum allowed binary tree depth). (representing the binary tree leaf node size).

The root node of the QTBT structure corresponding to a CTU may have four child nodes at the first level of the QTBT structure, and each of the child nodes may be partitioned according to the quadtree partition. That is, a first level node is either a leaf node (with no child nodes) or has four child nodes. An example QTBT structure 130 represents a node that includes a parent node and a child node with solid lines for branching. If the first level nodes are not larger than the maximum allowed binary tree root node size (MaxBTSize), the nodes may be further partitioned by their respective binary trees. Binary tree splitting of one node is iterated until the resulting node reaches the minimum allowed binary tree leaf node size (MinBTSize) or the maximum allowed binary tree depth (MaxBTDepth). obtain. The example QTBT structure 130 represents such nodes with dashed lines for branches. Binary tree leaf nodes are called coding units (CUs), and coding units (CUs) are used for prediction (e.g., intra-picture prediction or inter-picture prediction) and transformations without further partitioning. . As explained above, a CU is sometimes referred to as a "video block" or "block."

In an example QTBT partitioned structure, CTU size is set as 128x128 (luma sample and two corresponding 64x64 chroma samples), MinQTSize is set as 16x16, MaxBTSize is set as 64x64, and ( MinBTSize (for both width and height) is set as 4 and MaxBTDepth is set as 4. To generate quadtree leaf nodes, quadtree partitioning is first applied to the CTU. Quadtree leaf nodes may have a size from 16×16 (ie, MinQTSize) to 128×128 (ie, CTU size). If the quadtree leaf node is 128x128, the quadtree leaf node is not further divided by the binary tree because the size exceeds MaxBTSize (ie, 64x64 in this example). Otherwise, the quadtree leaf nodes are further partitioned by a binary tree. Therefore, the quadtree leaf node is also the root node for the binary tree and has the binary tree depth as 0. Once the binary tree depth reaches MaxBTDepth (4 in this example), no further splits are allowed. A binary tree node with a width equal to MinBTSize (4 in this example) implies that no further vertical splits (ie, width splits) are allowed for that binary tree node. Similarly, a binary tree node with a height equal to MinBTSize implies that no further horizontal splits (ie, height splits) are allowed for that binary tree node. As mentioned above, the leaf nodes of the binary tree are called CUs and are further processed according to predictions and transformations without further partitioning.

FIG. 3 is a block diagram illustrating an example video encoder 200 that may implement the techniques of this disclosure. FIG. 3 is provided for illustrative purposes and should not be considered a limitation of the techniques as broadly illustrated and described in this disclosure. For purposes of illustration, this disclosure describes a video encoder 200 in accordance with VVC (ITU-T H.266 under development) and HEVC (ITU-T H.265) techniques. However, the techniques of this disclosure may be performed by video encoding devices configured for other video coding standards.

In the example of FIG. 3, video encoder 200 includes video data memory 230, mode selection unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, It includes a configuration unit 214, a filter unit 216, a decoded picture buffer (DPB) 218, and an entropy encoding unit 220. Video data memory 230, mode selection unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filter unit 216, DPB 218, and Any or all of entropy encoding units 220 may be implemented in one or more processors or in processing circuitry. For example, a unit of video encoder 200 may be implemented as one or more circuits or logic elements as part of a hardware circuitry, or as part of a processor, ASIC, or FPGA. Moreover, video encoder 200 may include additional or alternative processors or processing circuitry to perform these and other functions.

Video data memory 230 may store video data to be encoded by components of video encoder 200. Video encoder 200 may receive video data stored in video data memory 230, for example, from video source 104 (FIG. 1). DPB 218 may act as a reference picture memory that stores reference video data for use in predicting subsequent video data by video encoder 200. Video data memory 230 and DPB 218 can be used in a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. It can be formed by any of the following. Video data memory 230 and DPB 218 may be provided by the same memory device or separate memory devices. In various examples, video data memory 230 may be on-chip with other components of video encoder 200, as shown, or off-chip with respect to those components.

In this disclosure, references to video data memory 230 are limited to memory internal to video encoder 200, unless specifically noted as such, or to memory external to video encoder 200, unless specifically noted as such. It should not be interpreted as such. Rather, references to video data memory 230 are understood as a reference memory that stores video data that video encoder 200 receives for encoding (e.g., video data for the current block to be encoded). Should. Memory 106 of FIG. 1 may also provide temporary storage of outputs from various units of video encoder 200.

The various units in FIG. 3 are shown to aid in understanding the operations performed by video encoder 200. A unit may be implemented as a fixed function circuit, a programmable circuit, or a combination thereof. Fixed function circuitry refers to circuitry that provides specific functionality and is preconfigured for operations that may be performed. Programmable circuit refers to a circuit that can be programmed to perform a variety of tasks, providing flexible functionality in the operations that can be performed. For example, a programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the software or firmware instructions. Although fixed function circuits may execute software instructions (eg, to receive parameters or output parameters), the types of operations that fixed function circuits perform generally remain unchanged. In some examples, one or more of the units may be different circuit blocks (fixed function or programmable), and in some examples, one or more of the units may be an integrated circuit. .

Video encoder 200 may include a programmable core formed from an arithmetic logic unit (ALU), an elementary function unit (EFU), digital circuitry, analog circuitry, and/or programmable circuitry. In examples where the operations of video encoder 200 are performed using software executed by programmable circuitry, memory 106 (FIG. 1) stores software instructions (e.g., object code) that video encoder 200 receives and executes. or another memory (not shown) within video encoder 200 may store such instructions.

Video data memory 230 is configured to store received video data. Video encoder 200 may retrieve pictures of video data from video data memory 230 and provide video data to residual generation unit 204 and mode selection unit 202. The video data in video data memory 230 may be raw video data that is to be encoded.

Mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an intra prediction unit 226. Mode selection unit 202 may include additional functional units for performing video prediction according to other prediction modes. By way of example, mode selection unit 202 may include a palette unit, an intra block copy unit (which may be part of motion estimation unit 222 and/or motion compensation unit 224), an affine unit, a linear model (LM) unit, etc. .

Mode selection unit 202 generally coordinates multiple encoding passes to test combinations of encoding parameters and resulting rate-distortion values for such combinations. The encoding parameters may include partitioning of the CTU into CUs, a prediction mode for the CU, a transform type for the residual data of the CU, a quantization parameter for the residual data of the CU, and the like. Mode selection unit 202 may ultimately select a combination of encoding parameters that has a better rate-distortion value than other combinations tested.

Video encoder 200 may partition a picture retrieved from video data memory 230 into a series of CTUs and may encapsulate one or more CTUs within a slice. Mode selection unit 202 may partition the CTU of a picture according to a tree structure, such as the HEVC QTBT structure or quadtree structure described above. As explained above, video encoder 200 may form one or more CUs from partitioning the CTUs according to a tree structure. Such a CU is also commonly referred to as a "video block" or "block."

In general, mode selection unit 202 also selects its components (e.g., motion estimation unit 222 , a motion compensation unit 224, and an intra prediction unit 226). For inter prediction of the current block, motion estimation unit 222 closely matches among one or more reference pictures (e.g., one or more previously coded pictures stored in DPB 218). Motion search may be performed to identify one or more reference blocks. In particular, the motion estimation unit 222 determines whether the possible reference blocks are currently A value representing how similar the blocks are can be calculated. Motion estimation unit 222 may generally perform these calculations using sample-by-sample differences between the current block and the reference block under consideration. Motion estimation unit 222 may identify the reference block with the smallest value resulting from these calculations, indicating the reference block that most closely matches the current block.

Motion estimation unit 222 may form one or more motion vectors (MVs) that define the position of a reference block in a reference picture relative to the position of a current block in a current picture. Motion estimation unit 222 may then provide the motion vector to motion compensation unit 224. For example, for unidirectional inter-prediction, motion estimation unit 222 may provide a single motion vector, whereas for bidirectional inter-prediction, motion estimation unit 222 may provide two motion vectors. Motion compensation unit 224 may then generate a predictive block using the motion vector. For example, motion compensation unit 224 may use motion vectors to retrieve data for reference blocks. As another example, if the motion vector has fractional sample precision, motion compensation unit 224 may interpolate the values for the predictive block according to one or more interpolation filters. Moreover, for bidirectional inter-prediction, motion compensation unit 224 may retrieve data for the two reference blocks identified by their respective motion vectors, e.g., through sample-by-sample averaging or weighted averaging. , the retrieved data may be combined.

As another example, for intra prediction or intra predictive coding, intra prediction unit 226 may generate a predictive block from samples adjacent to the current block. For example, for directional mode, intra prediction unit 226 may generally mathematically combine the values of adjacent samples and distribute these calculated values in a defined direction across the current block to form the predicted block. May be generated. As another example, for DC mode, intra prediction unit 226 may calculate the average of adjacent samples for the current block and generate a prediction block that should include this resulting average for each sample of the prediction block. obtain.

Mode selection unit 202 provides predictive blocks to residual generation unit 204. Residual generation unit 204 receives the raw, uncoded version of the current block from video data memory 230 and the predictive block from mode selection unit 202. Residual generation unit 204 calculates the sample-by-sample difference between the current block and the predicted block. The resulting sample-by-sample difference defines the residual block with respect to the current block. In some examples, residual generation unit 204 also generates sample values in the residual block to generate the residual block using residual differential pulse code modulation (RDPCM). The difference between can be determined. In some examples, residual generation unit 204 may be formed using one or more subtractor circuits that perform binary subtraction.

In examples where mode selection unit 202 partitions CUs into PUs, each PU may be associated with a luma prediction unit and a corresponding chroma prediction unit. Video encoder 200 and video decoder 300 may support PUs with various sizes. As indicated above, the size of a CU may refer to the size of the luma coding block of the CU, and the size of the PU may refer to the size of the luma prediction unit of the PU. Assuming that the size of a particular CU is 2N×2N, video encoder 200 uses a PU size of 2N×2N or N×N for intra prediction and 2N×2N, 2N×N for inter prediction. , N×2N, N×N, or similar symmetric PU sizes. Video encoder 200 and video decoder 300 may also support asymmetric partitioning for PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter prediction.

In examples where mode selection unit 202 no longer partitions CUs into PUs, each CU may be associated with a luma coding block and a corresponding chroma coding block. As mentioned above, the size of a CU may refer to the size of the luma coding block of the CU. Video encoder 200 and video decoder 300 may support CU sizes of 2N×2N, 2N×N, or N×2N.

For other video coding techniques, such as intra block copy mode coding, affine mode coding, and linear model (LM) mode coding, as some examples, mode selection unit 202 selects the respective units associated with the coding technique. Generate a predictive block for the current block being encoded via . In some examples, such as palette mode coding, mode selection unit 202 may not generate a predictive block, but instead generates a syntax element that indicates a scheme for reconstructing the block based on the selected palette. May be generated. In such a mode, mode selection unit 202 may provide these syntax elements to be encoded to entropy encoding unit 220.

As explained above, residual generation unit 204 receives video data for the current block and the corresponding predictive block. Residual generation unit 204 then generates a residual block for the current block. To generate a residual block, residual generation unit 204 calculates the sample-by-sample difference between the predicted block and the current block.

Transform processing unit 206 applies one or more transforms to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 may apply various transforms to the residual block to form a transform coefficient block. For example, transform processing unit 206 may apply a discrete cosine transform (DCT), a directional transform, a Karhunen-Loeve transform (KLT), or a conceptually similar transform to the residual block. In some examples, transform processing unit 206 may perform multiple transforms, eg, linear and quadratic transforms, such as rotation transforms, on the residual block. In some examples, transform processing unit 206 does not apply a transform to the residual block.

Quantization unit 208 may quantize the transform coefficients within the transform coefficient block to generate a quantized transform coefficient block. Quantization unit 208 may quantize the transform coefficients of the transform coefficient block according to a quantization parameter (QP) value associated with the current block. Video encoder 200 (eg, via mode selection unit 202) may adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization may result in a loss of information, and therefore the quantized transform coefficients may have lower accuracy than the original transform coefficients produced by transform processing unit 206.

Inverse quantization unit 210 and inverse transform processing unit 212 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficient block to reconstruct a residual block from the transform coefficient block. The reconstruction unit 214 generates a reconstructed block corresponding to the current block (potentially with some degree of distortion) based on the reconstructed residual block and the prediction block generated by the mode selection unit 202. can be generated. For example, reconstruction unit 214 may add samples of the reconstructed residual block to corresponding samples from the predictive block generated by mode selection unit 202 to generate a reconstructed block.

Filter unit 216 may perform one or more filter operations on the reconstructed block. For example, filter unit 216 may perform a deblocking operation to reduce blockiness artifacts along the edges of the CU. In some examples, operation of filter unit 216 may be skipped.

Video encoder 200 stores the reconstructed blocks in DPB 218. For example, in instances where the operations of filter unit 216 are not performed, reconstruction unit 214 may store reconstruction blocks in DPB 218. In examples where the operations of filter unit 216 are performed, filter unit 216 may store filtered reconstructed blocks in DPB 218. Motion estimation unit 222 and motion compensation unit 224 use reference pictures formed from the reconstructed (and potentially filtered) blocks to inter-predict blocks of pictures that are subsequently coded. Can be extracted from DPB218. Additionally, intra prediction unit 226 may use reconstructed blocks in DPB 218 of the current picture to intra predict other blocks in the current picture.

Generally, entropy encoding unit 220 may entropy encode syntax elements received from other functional components of video encoder 200. For example, entropy encoding unit 220 may entropy encode the quantized transform coefficient block from quantization unit 208. As another example, entropy encoding unit 220 may entropy encode prediction syntax elements (eg, motion information for inter prediction or intra mode information for intra prediction) from mode selection unit 202. Entropy encoding unit 220 may perform one or more entropy encoding operations on syntax elements that are another example of video data to generate entropy encoded data. For example, entropy encoding unit 220 may perform a context adaptive variable length coding (CAVLC) operation, a CABAC operation, a variable-to-variable (V2V) length coding operation, a syntax-based context adaptive binary arithmetic coding (SBAC) operation, a probability interval A piecewise entropy (PIPE) coding operation, an exponential Golomb coding operation, or another type of entropy coding operation may be performed on the data. In some examples, entropy encoding unit 220 may operate in a bypass mode in which syntax elements are not entropy encoded.

Video encoder 200 may output a bitstream that includes entropy-encoded syntax elements needed to reconstruct slices or blocks of pictures. In particular, entropy encoding unit 220 may output a bitstream.

The operations described above are described in terms of blocks. Such descriptions should be understood as being operations for luma coding blocks and/or chroma coding blocks. As explained above, in some examples, the luma and chroma coding blocks are the luma and chroma components of the CU. In some examples, the luma coding block and chroma coding block are the luma and chroma components of the PU.

In some examples, operations performed on luma coding blocks need not be repeated for chroma coding blocks. As an example, the operations for identifying motion vectors (MVs) and reference pictures for luma coding blocks do not need to be repeated to identify MVs and reference pictures for chroma blocks. Rather, the MV for the luma coding block may be scaled to determine the MV for the chroma block, and the reference picture may be the same. As another example, the intra prediction process may be the same for luma coding blocks and chroma coding blocks.

Video encoder 200 is a device configured to encode video data that includes a memory configured to store video data and one or more processing units implemented in circuitry. Representing one example, the one or more processing units may divide an input block into multiple sub-blocks, wherein the size of the input block is less than or equal to the size of the coding unit; determining that bidirectional optical flow (BDOF) is to be applied to a sub-block of the block based on a condition being met; and dividing the sub-block into a plurality of sub-sub-blocks; determining an improved motion vector for one or more of the sub-subblocks, wherein the improved motion vector for the sub-subblock of the one or more sub-subblocks is determined for one or more of the sub-subblocks; and performing BDOF on the sub-blocks based on the improved motion vectors for the one or more sub-sub-blocks.

As another example, one or more processing units implemented in a circuit configuration may partition an input block into multiple subblocks, the size of the input block being smaller than the size of the coding unit. or equivalent thereto, and determining that Bidirectional Optical Flow (BDOF) is to be applied to a sub-block of the plurality of sub-blocks based on the satisfaction of a condition; for each of the one or more samples in the sub-block, determining an improved motion vector for each of the one or more samples in the sub-block; and performing BDOF on the sub-blocks based on the improved motion vectors for the sub-blocks.

As yet another example, the processing circuitry of video encoder 200 determines that bidirectional optical flow (BDOF) is enabled for a block of video data, and that BDOF is enabled for the block. divide the block into multiple sub-blocks based on the determination, determine a respective strain value for each sub-block of one or more of the multiple sub-blocks, Determine whether per-pixel BDOF is performed or BDOF is bypassed for each sub-block of the one or more sub-blocks based on the respective distortion values, and per-pixel BDOF is performed. determine a predicted sample for each subblock of the one or more subblocks based on a determination that the BDOF is bypassed or that the BDOF is bypassed; and determine a residual value indicating the difference between the predicted sample and the block. and may be configured to signal information indicative of the residual value.

FIG. 4 is a block diagram illustrating an example video decoder 300 that may implement the techniques of this disclosure. FIG. 4 is provided for purposes of illustration and not limitation of the techniques as broadly illustrated and described in this disclosure. For purposes of illustration, this disclosure describes a video decoder 300 in accordance with VVC (ITU-T H.266 under development) and HEVC (ITU-T H.265) techniques. However, the techniques of this disclosure may be performed by video coding devices configured for other video coding standards.

In the example of FIG. 4, the video decoder 300 includes a coded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, and a filter unit. 312, and a decoded picture buffer (DPB) 314. Any or all of CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, filter unit 312, and DPB 314 may include one or more may be implemented in a processor or in processing circuitry. For example, a unit of video decoder 300 may be implemented as one or more circuits or logic elements as part of a hardware circuitry, or as part of a processor, ASIC, or FPGA. Moreover, video decoder 300 may include additional or alternative processors or processing circuitry to perform these and other functions.

Prediction processing unit 304 includes a motion compensation unit 316 and an intra prediction unit 318. Prediction processing unit 304 may include additional units for performing predictions according to other prediction modes. By way of example, prediction processing unit 304 may include a palette unit, an intra block copy unit (which may form part of motion compensation unit 316), an affine unit, a linear model (LM) unit, and the like. In other examples, video decoder 300 may include more, fewer, or different functional components.

CPB memory 320 may store video data, such as an encoded video bitstream to be decoded by components of video decoder 300. Video data stored in CPB memory 320 may be obtained from computer-readable medium 110 (FIG. 1), for example. CPB memory 320 may include a CPB that stores encoded video data (eg, syntax elements) from an encoded video bitstream. CPB memory 320 may also store video data other than syntax elements of coded pictures, such as temporary data representing outputs from various units of video decoder 300. DPB 314 generally stores decoded pictures that video decoder 300 may output and/or use as reference video data when decoding subsequent data or pictures of the encoded video bitstream. CPB memory 320 and DPB 314 may be formed by any of a variety of memory devices, such as DRAM, including SDRAM, MRAM, RRAM, or other types of memory devices. CPB memory 320 and DPB 314 may be provided by the same memory device or separate memory devices. In various examples, CPB memory 320 may be on-chip with other components of video decoder 300 or off-chip with respect to those components.

Additionally or alternatively, in some examples, video decoder 300 may retrieve coded video data from memory 120 (FIG. 1). That is, memory 120 may store data as described above using CPB memory 320. Similarly, memory 120 stores instructions to be executed by video decoder 300 when some or all of the functionality of video decoder 300 is implemented in software to be executed by processing circuitry of video decoder 300. It's fine.

The various units shown in FIG. 4 are shown to aid in understanding the operations performed by video decoder 300. A unit may be implemented as a fixed function circuit, a programmable circuit, or a combination thereof. Similar to FIG. 3, fixed function circuitry refers to circuitry that provides specific functionality and is preconfigured for operations that may be performed. Programmable circuit refers to a circuit that can be programmed to perform a variety of tasks, providing flexible functionality in the operations that can be performed. For example, a programmable circuit may execute software or firmware that causes the programmable circuit to operate in a manner defined by the software or firmware instructions. Although fixed function circuits may execute software instructions (eg, to receive parameters or output parameters), the types of operations that fixed function circuits perform generally remain unchanged. In some examples, one or more of the units may be different circuit blocks (fixed function or programmable), and in some examples, one or more of the units may be an integrated circuit. .

Video decoder 300 may include a programmable core formed from ALUs, EFUs, digital circuits, analog circuits, and/or programmable circuits. In examples where the operations of video decoder 300 are performed by software executing on programmable circuitry, on-chip or off-chip memory stores software instructions (e.g., object code) that video decoder 300 receives and executes. It's fine.

Entropy decoding unit 302 may receive encoded video data from the CPB and entropy decode the video data to reproduce syntax elements. Prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, and filter unit 312 may generate decoded video data based on syntax elements extracted from the bitstream.

Generally, video decoder 300 reconstructs pictures block by block. Video decoder 300 may perform a reconstruction operation on each block individually (where the block currently being reconstructed, ie, being decoded, may be referred to as the "current block").

Entropy decoding unit 302 may entropy decode syntax elements that define quantized transform coefficients of the quantized transform coefficient block and transform information such as quantization parameters (QPs) and/or transform mode indications. Inverse quantization unit 306 may use the QP associated with the quantized transform coefficient block to determine the degree of quantization, and likewise the degree of inverse quantization that inverse quantization unit 306 should apply. Dequantization unit 306 may, for example, perform a bitwise left shift operation to dequantize the quantized transform coefficients. Inverse quantization unit 306 may thereby form a transform coefficient block that includes transform coefficients.

After inverse quantization unit 306 forms the transform coefficient block, inverse transform processing unit 308 applies one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. good. For example, inverse transform processing unit 308 may apply an inverse DCT, an inverse integer transform, an inverse Karhunen-Loeve transform (KLT), an inverse rotation transform, an inverse transform, or another inverse transform to the block of transform coefficients.

Further, the prediction processing unit 304 generates a prediction block according to the prediction information syntax elements entropy decoded by the entropy decoding unit 302. For example, motion compensation unit 316 may generate a predictive block if the prediction information syntax element indicates that the current block is inter-predicted. In this case, the prediction information syntax element specifies the reference picture in the DPB 314 from which the reference block is to be retrieved, as well as a motion vector that identifies the location of the reference block in the reference picture relative to the location of the current block in the current picture. can be shown. Motion compensation unit 316 may generally perform an inter-prediction process in a manner substantially similar to that described with respect to motion compensation unit 224 (FIG. 3).

As another example, if the prediction information syntax element indicates that the current block is intra-predicted, intra prediction unit 318 may generate the prediction block according to the intra prediction mode indicated by the prediction information syntax element. Again, intra prediction unit 318 may generally perform the intra prediction process in a manner substantially similar to that described with respect to intra prediction unit 226 (FIG. 3). Intra prediction unit 318 may retrieve data for adjacent samples to the current block from DPB 314.

Reconstruction unit 310 may reconstruct the current block using the prediction block and the residual block. For example, reconstruction unit 310 may add samples of the residual block to corresponding samples of the predictive block to reconstruct the current block.

Filter unit 312 may perform one or more filter operations on the reconstructed block. For example, filter unit 312 may perform a deblocking operation to reduce blockiness artifacts along the edges of the reconstructed block. The operations of filter unit 312 are not necessarily performed in all instances.

Video decoder 300 may store reconstruction blocks in DPB 314. For example, in examples where the operations of filter unit 312 are not performed, reconstruction unit 310 may store reconstruction blocks in DPB 314. In examples where the operations of filter unit 312 are performed, filter unit 312 may store filtered reconstructed blocks in DPB 314. As explained above, DPB 314 may provide reference information to prediction processing unit 304, such as samples of the current picture for intra prediction and previously decoded pictures for subsequent motion compensation. Additionally, video decoder 300 may output decoded pictures (eg, decoded video) from DPB 314 for later presentation on a display device, such as display device 118 of FIG. 1.

Video decoder 300 thus represents an example of a video decoding device that includes a memory configured to store video data and one or more processing units implemented in circuitry; The one or more processing units are configured to divide the input block into a plurality of sub-blocks, the size of the input block being smaller than or equal to the size of the coding unit, and the size of the input block being smaller than or equal to the size of the coding unit; determining that bidirectional optical flow (BDOF) is to be applied to a sub-block based on a condition being met; dividing the sub-block into a plurality of sub-sub-blocks; determining an improved motion vector for one or more of the sub-sub-blocks, the improved motion vector for the sub-sub-block of the one or more sub-sub-blocks comprising: determining an improved motion vector for one or more of the sub-sub-blocks; and performing BDOF on the sub-blocks based on the improved motion vectors for the one or more sub-sub-blocks.

As another example, one or more processing units implemented in a circuit configuration may partition an input block into multiple subblocks, the size of the input block being smaller than the size of the coding unit. or equivalent thereto, and determining that Bidirectional Optical Flow (BDOF) is to be applied to a sub-block of the plurality of sub-blocks based on the satisfaction of a condition; for each of the one or more samples in the sub-block, determining an improved motion vector for each of the one or more samples in the sub-block; and performing BDOF on the sub-blocks based on the improved motion vectors for the sub-blocks.

As another example, processing circuitry (e.g., motion compensation unit 316) of video decoder 300 determines that bidirectional optical flow (BDOF) is enabled for a block of video data and that dividing the block into a plurality of subblocks based on the determination that BDOF is enabled; determining a respective strain value for each subblock of the one or more subblocks of the plurality of subblocks; Determining whether pixel-wise BDOF is performed or BDOF is bypassed for each subblock of one or more of the plurality of subblocks based on the respective distortion values. determine predicted samples for each subblock of the one or more subblocks based on the decision that pixel-wise BDOF is performed or BDOF is bypassed, and reconstruct the block based on the predicted samples. may be configured to do so. For example, processing circuitry may receive a residual value indicating a difference between the predicted samples and the samples of the block and may add the residual value to the predicted samples to reconstruct the block.

The following describes the CU structure and motion vector prediction in HEVC. The following may provide additional context to the above description of CU and motion vector prediction, and may include some repetitions of the above description to aid in understanding.

In HEVC, the largest coding unit in a slice is called a coding tree block (CTB) or coding tree unit (CTU). CTB contains a quadtree, and the nodes of the quadtree are coding units. The size of the CTB can range from 16x16 to 64x64 in the HEVC main profile (though technically an 8x8 CTB size could be supported). The coding unit (CU) may be as small as 8×8 from the same size of the CTB. Each coding unit is coded using one mode: inter mode or intra mode. When inter-coded, a CU may be further partitioned into two or four prediction units (PUs), or may become only one PU when no further partitioning is applied. When there are two PUs in one CU, they can be half rectangles in size, or two rectangles with the size of 1/4 or 3/4 of the CU. When CUs are inter-coded, each PU has one set of motion information derived using a unique inter-prediction mode.

The following describes motion vector prediction. The HEVC standard provides two inter-prediction modes for a prediction unit (PU), named merge mode (skip mode is considered a special case of merge mode) and advanced motion vector prediction (AMVP) mode, respectively. There is.

In either AMVP mode or merge mode, a motion vector (MV) candidate list is maintained for multiple motion vector predictors. The motion vector of the current PU as well as the reference index in merge mode are generated by taking one candidate from the MV candidate list.

The MV candidate list contains up to 5 candidates for merge mode and only 2 candidates for AMVP mode. A merge candidate may include a set of motion information, eg, motion vectors that correspond to both reference picture lists (List 0 and List 1) and reference indices. If a merge candidate is identified by the merge index, the reference picture used for prediction of the current block as well as the associated motion vector are determined. On the other hand, under AMVP mode, for each possible prediction direction from either list 0 or list 1, the AMVP candidates only contain motion vectors, so the reference index is the MV predictor to the MV candidate list. (MVP) will be explicitly signaled along with the index. In AMVP mode, the predicted motion vectors may be further improved. Candidates for both modes are similarly derived from the same spatial and temporal neighbors.

The following describes spatial neighbor candidates. For example, FIGS. 5A and 5B are conceptual diagrams illustrating examples of spatially adjacent motion vector candidates for merge mode and advanced motion vector predictor (AMVP) mode, respectively.

Spatial MV candidates are derived from neighboring blocks shown in FIGS. 5A and 5B for a particular PU (PU ₀ ) 500, but the method of generating candidates from blocks is different for merge mode and AMVP mode. In merge mode, up to 4 spatial MV candidates can be derived using numbers with the order shown in Figure 5A as follows: (0,A1), top (1,B1), top right (2,B0), bottom left (3,A0), and top left (4,B2).

In AVMP mode, the adjacent blocks are divided into two groups, the left group consisting of blocks 0 and 1, and the upper group consisting of blocks 2, 3, and 4, as shown in PU0 502 in Figure 5B. be divided. For each group, the possible candidates among the neighboring blocks that refer to the same reference picture as indicated by the signaled reference index have the highest priority to be chosen to form the final candidates of the group. . It is possible that all neighboring blocks do not contain motion vectors pointing to the same reference picture. Therefore, if no such candidate is found, the first available candidate may be scaled to form the final candidate, thus compensating for the time-distance difference.

The following describes temporal motion vector prediction in HEVC. Temporal motion vector predictor (TMVP) candidates, if enabled and available, are added into the MV candidate list after the spatial motion vector candidates. The process of motion vector derivation for TMVP candidates is the same for both merge mode and AMVP mode, but the target reference index for TMVP candidates in merge mode is always set to 0.

The primary block location for TMVP candidate derivation is as block “T”, denoted as block 602, to compensate for the bias towards the top and left blocks used to generate spatial neighbor candidates. As shown in Figure 6A is the outer lower right block of the colocated PU. However, if the block is currently located outside the CTB row or no motion information is available, then the block is replaced with the central block of the PU, shown as block 604.

Motion vectors for TMVP candidates are derived from colocated PUs of colocated pictures, indicated within the slice level. The motion vector for collocated PU is called collocated MV. Similar to the temporal direct mode in AVC, to derive the TMVP candidate motion vector, the collocated MVs will be scaled to compensate for the temporal distance difference, as shown in FIG. 6B.

The following describes additional aspects of motion prediction in HEVC. Some aspects of merge mode and AMVP mode are worth mentioning as follows. Motion Vector Scaling: It is assumed that the value of the motion vector is proportional to the distance of the picture at the presentation time. A motion vector associates two pictures: a reference picture and a picture containing the motion vector (ie, a stored picture). When that motion vector is used to predict other motion vectors, the distance between the stored picture and the reference picture is calculated based on the picture order count (POC) value.

For a motion vector to be predicted, both its associated stored picture and reference picture may be different. Therefore, a new distance (based on POC) is calculated. The motion vector is then scaled based on these two POC distances. For spatially adjacent candidates, the stored pictures for the two motion vectors are the same, but the reference pictures are different. In HEVC, motion vector scaling is applied to both TMVP and AMVP for spatial and temporal neighbor candidates.

Artificial motion vector candidate generation: If the motion vector candidate list is not complete, artificial motion vector candidates are generated and inserted at the end of the list until the list has all candidates.

In merge mode, there are two types of artificial MV candidates: composite candidates, which are derived only for B slices, and zeros, which are used only for AMVP if the first type does not provide enough artificial candidates. There are candidates. For each pair of candidates that are already in the candidate list and have the required motion information, the motion vector of the first candidate refers to a picture in list 0 and the second candidate refers to a picture in list 1. A bidirectional composite motion vector candidate is derived by combination with the candidate motion vector.

Pruning process for candidate insertion: Candidates from different blocks may happen to be the same, which reduces the efficiency of the merge/AMVP candidate list. A pruning process is applied to solve this problem. The pruning process compares one candidate against another in the current candidate list in order to avoid inserting identical candidates to some extent. To reduce complexity, rather than comparing each possible candidate with all other existing candidates, a pruning process is applied only a limited number of times.

The following describes template matching prediction. Template matching (TM) prediction is a special merging mode based on frame rate upconversion (FRUC) techniques. Using this mode, motion information for the block is not signaled, but is derived at the decoder side (eg, by video decoder 300). TM prediction applies to both AMVP mode and normal merge mode. In AMVP mode, MVP candidate selection is determined based on template matching to pick up the candidate that reaches the minimum difference between the current block template and the reference block template. In normal merge mode, a TM mode flag is signaled to indicate the use of a TM, and then the TM is applied to the merge candidates indicated by the merge index for MV improvement.

As shown in FIG. 7, template matching is used to match a template in current frame 700 (the adjacent block above and/or to the left of the current CU) with a block in reference frame 702 (same size as the template). Derive the motion information of the current CU by finding the closest match between. With the AMVP candidates selected based on the initial matching error, the MVP of the AMVP candidates is improved by template matching. With the merge candidates indicated by the signaled merge index, the merged MVs of the merge candidates corresponding to L0 and L1 are independently improved by template matching, and then the less accurate MVs are compared to the better MVs. is further improved using as a priority.

For the cost function, motion compensated interpolation may be utilized when the motion vector points to a fractional sample position. To reduce complexity, bilinear interpolation is used instead of the usual 8-tap DCT-IF interpolation for both template matching to generate templates on the reference picture. The matching cost C of template matching is calculated as follows.

In the above equation, w is a weighting factor that is empirically set to 4, and MV and MV ^s are the currently testing MV, and the initial MV (i.e., MVP candidate in AMVP mode, or merged (merged movement in mode). SAD (sum of absolute differences) is used as a matching cost for template matching.

When TM is used, motion is improved by using only luma samples. The derived motion can be used for both luma and chroma for MC (motion compensation) inter prediction. After the MV is determined, a final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma.

For search methods, MV improvement is a pattern-based MV search using the criterion of template matching cost. Two search patterns are supported for MV improvement: diamond search and cross search. The MV is searched directly with a 1/4 luma sample MVD accuracy using a diamond pattern, followed by a 1/4 luma sample MVD accuracy using a cross pattern, then 1/8 luma sample MVD accuracy using a cross pattern. This will be followed by sample MVD improvements. The search range for MV improvement is set equal to (-8,+8) luma samples around the initial MV.

The following describes bidirectional matching prediction. Bidirectional matching (also called bidirectional merging) (BM) prediction is another merging mode based on frame rate up conversion (FRUC) techniques. When it is determined that a block should apply BM mode, the two initial motion vectors MV0 and MV1 are determined by using the signaled merging candidate index to select the merging candidate in the configured merging list. is derived. A bidirectional matching search may be around MV0 and MV1. Based on the minimum two-way matching cost, the final MV0' and MV1' are derived.

The motion vector differences MVD0 800 (indicated by MV0' - MV0) and MVD1 802 (indicated by MV1' - MV1) pointing to the two reference blocks are the temporal distance (TD) between the current picture and the two reference pictures. , for example, may be proportional to TD0 and TD1. Figure 8 shows an example of MVD0 and MVD1, where TD1 is 4 times TD0.

However, there are optional designs in which MVD0 and MVD1 reflect like mirrors regardless of the time distances TD0 and TD1. Figure 9 shows an example of a mirror-reflecting MVD0 900 and MVD1 902, where TD1 is 4 times TD0.

Bidirectional matching performs a local search around the initial MV0 and MV1 to derive the final MV0' and MV1'. The local search applies a 3x3 square search pattern in a loop through the search range [-8,8]. At each search iteration, the two-way matching costs of the eight surrounding MVs in the search pattern are calculated and compared with the two-way matching cost of the central MV. The MV with the minimum two-way matching cost becomes the new central MV in the next search iteration. The local search ends when the current center MV has the minimum cost within the 3x3 square search pattern or the local search reaches a predetermined maximum search iterations. FIG. 10 shows an example of a 3×3 square search pattern 1000 within the search range [-8,8].

The following describes the decoder side motion vector improvement. To increase the accuracy of merge mode MV, decoder-side motion vector refinement (DMVR) is applied in VVC. In bi-prediction operations, the MV to be improved is searched around the initial MV in reference picture list L0 and reference picture list L1. The DMVR method calculates the distortion between two candidate blocks in reference picture list L0 and list L1. As shown in FIG. 11, the SAD between blocks 1102 and 1100 is calculated based on each MV candidate around the initial MV. The MV candidate with the smallest SAD becomes the improved MV and is used to generate the bi-predicted signal.

The improved MV derived by the DMVR process is used to generate inter-prediction samples and is also used in temporal motion vector prediction for future picture coding. The original MV is used in the deblocking process, but also in spatial motion vector prediction for future CU coding.

DMVR is a subblock-based merging mode with a default maximum processing unit of 16x16 luma samples. When the width and/or height of the CU is greater than 16 luma samples, the CU may be further divided into sub-blocks having a width and/or height equal to 16 luma samples.

The following describes the search method. In DVMR, the search points surrounding the initial MV and the MV offset may comply with MV differential mirroring rules. For example, any point checked by DMVR, indicated by a candidate MV pair (MV0, MV1), may conform to the following two equations:
MV0' = MV0 + MV_offset
MV1' = MV1 - MV_offset

In the above equation, MV_offset represents the improvement offset between the initial MV and the improved MV in one of the reference pictures. The improvement search range is two integer luma samples from the initial MV. The search includes an integer sample offset search stage and a fractional sample improvement stage.

A 25-point full search is applied for the integer sample offset search. The SAD of the initial MV pair is first calculated. If the SAD of the initial MV pair is less than the threshold, the integer sample stage of DMVR ends. If not, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search stage. To reduce the penalty of uncertainty in DMVR improvement, the original MV during the DMVR process may be preferred. The SAD between the reference blocks referenced by the initial MV candidate will be smaller by 1/4 of the SAD value.

Integer sample search is followed by fractional sample improvement. To save computational complexity, a fractional sample improvement is derived by using a parametric error surface equation rather than an additional search with SAD comparison. Fractional sample refinement is conditionally invoked based on the output of the integer sample search stage. Once the integer sample search stage ends with the center having the minimum SAD in either the first iterative search or the second iterative search, fractional sample refinement is further applied.

In parametric error surface-based subpixel offset estimation, the center location cost and the costs at four adjacent locations from the center are used to fit a 2D parabolic error surface equation of the form:
E(x,y) = A(x - x _min ) ² + B(y - y _min ) ² + C

In the above equation, (x _min ,y _min ) corresponds to the fractional position with minimum cost, and C corresponds to the minimum cost value. By solving the above equation by using the cost values of the five search points, (x _min ,y _min ) becomes
x _min = (E(-1,0) - E(1,0))/(2(E(-1,0) + E(1,0) - 2E(0,0)))
y _min = (E(0,-1) - E(0,1))/(2((E(0,-1) + E(0,1) - 2E(0,0)))
It is calculated as

Since all cost values are positive and the minimum value is E(0,0), the values of x _min and y _min are automatically constrained to be between -8 and 8. This corresponds to a 1/2 pel offset with 1/16 pel MV accuracy in VVC. The calculated fraction (x _min ,y _min ) is added to the integer distance improvement MV to obtain the sub-pixel accuracy improvement delta MV.

The following describes bilinear interpolation and sample padding. In VVC, the resolution of MV is 1/16 luma sample. Samples at fractional positions are interpolated using an 8-tap interpolation filter. In DMVR, search points surround an initial fractional pel MV with an integer sample offset, so samples at those fractional positions may be interpolated for the DMVR search process. To reduce computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in DMVR. In some examples, by using a bilinear filter with a search range of 2 samples, DVMR does not access more reference samples compared to regular motion compensation processes. After the improved MV is achieved using the DMVR search process, a regular 8-tap interpolation filter is applied to generate the final prediction. In order not to access more reference samples than the normal MC process, the samples that are not needed for the original MV-based interpolation process but are needed for the improved MV-based interpolation process are Padded from available samples.

The following describes exemplary enabling conditions for DMVR. DMVR is enabled if all of the following conditions are met:
a. CU level merge mode using bi-predictive MV.
b. One reference picture is in the past and another in the future for the current picture.
c. The distances (eg, POC difference) from both reference pictures to the current picture are the same.
d.CU has more than 64 luma samples.
e. Both CU height and CU width are greater than or equal to 8 luma samples.
f.BCW (bi-prediction using CU-level weights) weight index indicates equal weight.
g.WP (weighted prediction) is not enabled for the current block.
h. CIIP (Combined Inter and Intra Prediction) mode is not used for the current block.

The following describes bidirectional optical flow. Bidirectional optical flow (BDOF) is used to improve the bi-predictive signal of the luma samples in the CU at the 4×4 sub-block level. As its name suggests, BDOF mode is based on the optical flow concept, which assumes smooth object motion. For each 4×4 sub-block, the motion improvement (v _x ,v _y ) is computed by minimizing the difference between the L0 and L1 predicted samples. Motion enhancement is then used to adjust the bi-predicted sample values within the 4x4 sub-blocks.

For example, in the case of BDOF, video encoder 200 and video decoder determine that BDOF is enabled for a block, and when BDOF is enabled for a block, they divide the block into multiple subblocks. It's fine. In some examples, video encoder 200 and video decoder 300 may determine a first reference block from a first motion vector for the block and a second reference block from a second motion vector for the block. Video encoder 200 and video decoder 300 may blend (eg, weighted average) the samples in the first reference block and the samples in the second reference block to generate a predictive block. Video encoder 200 and video decoder 300 may determine motion improvements and adjust samples in a predictive block to generate predictive samples that are used to encode or decode samples of a sub-block. . In some examples, video encoder 200 and video decoder 300 may determine a motion improvement that is the same for each sample in a subblock (ie, subblock-level motion improvement, referred to as subblock BDOF). In some examples, video encoder 200 and video decoder 300 may determine motion improvement in sub-blocks or each sample (ie, sample-level motion improvement, referred to as pixel-by-pixel BDOF).

In the BDOF process that may be applicable to sub-block BDOF, the following steps are applied. The steps for pixel-wise BDOF are explained in further detail below.

First, the horizontal and vertical slopes of the two predicted signals

and

but directly computes the difference between two adjacent samples, i.e.

Calculated by

In the above example, I ^(k) (i,j) is the sample value at coordinates (i,j) of the predicted signal in list k (k=0,1), and shift1 is The luma bit depth is calculated based on bitDepth to be set equal. That is, I ⁽⁰⁾ refers to the sample of the first reference block, I ⁽¹⁾ refers to the sample of the second reference block, where the first reference block and the second reference block refer to the sample of the first reference block. was used to generate prediction blocks that are being adjusted according to the BDOF technique.

Then the autocorrelations and crosscorrelations of the slopes S ₁ , S ₂ , S ₃ , S ₅ , and S ₆ are
S ₁ = Σ _(i,j)∈Ω |ψ _x (i,j)|, S ₃ = Σ _(i,j)∈Ω θ(i,j)・(-sign(ψ _x (i,j) )
S ₂ = Σ _(i,j)∈Ω ψ _x (i,j)・sign(ψ _y (i,j))
S ₅ = Σ _(i,j)∈Ω |ψ _y (i,j)|, S ₆ = Σ _(i,j)∈Ω θ(i,j)・(-sign(ψ _y (i,j) ) (1-6-2)
is calculated as, where:

, where Ω is a 6×6 window around the 4×4 subblock, the value of shift2 is set equal to 4, and the value of shift3 is set equal to 1 .

The motion improvement (v _x ,v _y ) is then derived using the cross-correlation and autocorrelation terms using: In this example, the motion improvement is for sub-blocks. Pixel-by-pixel motion improvement calculations are explained in more detail below.

However, th' _BIO = 1 << 4.

is the floor function.

Based on the motion improvement and gradient, the following adjustments are calculated for each sample in a 4x4 sub-block.

Finally, the BDOF samples of the CU are calculated by adjusting the bi-predictive samples as follows.
pred _BDOF (x,y) = (I ⁽⁰⁾ (x,y) + I ⁽¹⁾ (x,y) + b(x,y) + ο _offset ) >> shift5 (1-6-6)
However, shift5 is set equal to Max(3,15 - BitDepth), and the variable ο _offset is set equal to (1 << (shift5 - 1)).

In the above example, I ⁽⁰⁾ refers to the first reference block, I ⁽¹⁾ refers to the second reference block, and b(x,y) is the motion improvement (v _x ,v _y ) is the adjustment value determined based on In some examples, I ⁽⁰⁾ (x,y) + I ⁽¹⁾ (x,y) may be considered a predictive block, and therefore b(x,y) is considered to adjust the predictive block. It's good to be considered. As shown in equation (1-6-6), there may be an addition of o _offset and a right shift operation of shift5 in order to generate the predicted sample (pred _BDOF (x,y)).

The above provides an example for a sub-block BDOF where video encoder 200 and video decoder 300 determine that the motion improvements (v _x ,v _y ) are the same for all samples in the sub-block. explain. The adjustment value b(x,y) may be different for each sample in the subblock due to the gradient, but the motion improvement may be the same.

As described in even more detail below, for pixel-wise BDOF, video encoder 200 and video decoder 300 may determine pixel-wise motion improvements (v _x ′,v _y ′). That is, rather than having one motion improvement for a sub-block as in sub-block BDOF, in pixel-wise BDOF there may be a different motion improvement for each sample (eg, pixel). Video encoder 200 and video decoder 300 base the adjustment value b'(x ,y) may be determined.

In some examples, the values from Equation 1-6-6 are selected such that the multiplier in the BDOF process does not exceed 15 bits and the maximum bit width of intermediate parameters in the BDOF process is kept within 32 bits.

To derive the gradient values, some predicted samples I ^(k) (i,j) in the list k (k=0,1) outside the current CU boundary are sent to the video encoder 200 and the video decoder Generated by 300. As shown in Figure 12, BDOF uses one extended row/column around the border of CU 1200. To control the amount of computation that generates predicted samples outside the boundary, video encoder 200 and video decoder 300 calculate directly (using the floor() operation on the coordinates) nearby integer locations without using interpolation. By taking reference samples at , we may generate predicted samples within the extended area (white positions), and a regular 8-tap motion compensated interpolation filter may generate predicted samples within the CU (gray positions). used to. These expanded sample values may only be used in gradient calculations. For the remaining steps in the BDOF process, if any samples and gradient values outside the CU boundary are needed, the samples and gradient values are padded (ie, repeated) from their nearest neighbors.

BDOF is used to improve the CU's bi-predicted signal (eg, the sum of the first reference block and the second reference block) at the 4×4 subblock level. BDOF applies to a CU if all of the following conditions are met:
a. The CU is coded using a "true" bi-prediction mode, i.e. one of the two reference pictures is before the current picture in the display order and the other is after the current picture in the display order.
b.CU is not coded using affine mode or ATMVP merge mode.
c.CU has more than 64 luma samples.
d. Both CU height and CU width are greater than or equal to 8 luma samples.
e.BCW weight index indicates equal weight.
f.WP is not currently enabled for the CU.
g. CIIP mode is not currently used for the CU.

There can be several problems with BDOF. As explained above, in the current version of VVC, the BDOF method is used to improve the bi-predictive signal of the luma samples in the coding block at the 4×4 sub-block level. The motion improvement (v _x ,v _y ) is derived by minimizing the difference between the L0 and L1 predicted samples within the 6×6 luma sample region. L0 predicted samples refer to samples of the first reference block, and L1 predicted samples refer to samples of the second reference block. Motion enhancement (v _x ,v _y ) is then used to adjust each predicted sample of the 4x4 sub-block.

However, luma samples within a 4x4 sub-block may have different motion improvement characteristics compared to other luma samples within a 4x4 sub-block. Computing the motion improvement (v' _x ,v' _y ) at the pixel level can improve the accuracy of the motion improvement for each pixel and thus the sub-block or block prediction quality.

However, BDOF is a decoder-side process, and the complexity of BDOF is also an important aspect to be considered when designing a video coding method. When the motion improvement (v' _x ,v' _y ) is computed at the pixel level, the complexity of the BDOF may be 16 times higher than the current BDOF at the 4x4 sub-block level. In other words, the current 4x4 sub-block BDOF does not achieve the best prediction quality. Although pixel-wise BDOF has better prediction quality, complexity is an issue for video coding.

In VVC Draft 10, when BDOF precedes decoder-side motion vector refinement (DMVR), the BDOF process may be bypassed based on the minimum SAD in the DMVR search process. The DMVR process is at the 16x16 subblock level. This BDOF bypass method can reduce complexity.

However, the prediction signal of a subarea within a 16x16 subblock may need to be improved by BDOF. In the BDOF bypass of the VVC Draft 10 method, BDOF cannot be applied in subareas within a 16×16 subblock, and BDOF is bypassed in other subareas in the meantime. In VVC Draft 10, there is no bypass BDOF scheme when BDOF is applied to bi-predicted coding blocks (not DMVR predicted).

The following describes example techniques that may address the above problems. However, the present techniques should not be considered limited to or required to address the above problems. The following techniques may be used separately or in virtually any combination. Although, for simplicity, the techniques below are described as various aspects, such aspects should not be considered as required to be separate; in fact, the various aspects may be combined. obtain. Unless otherwise specified, example aspects may be performed by video encoder 200 and/or video decoder 300.

The first aspect relates to bypassing the sub-block BDOF. In this first aspect, when it is determined that a W×H coding block should apply bidirectional optical flow (BDOF), video encoder 200 and/or video decoder 300 May bypass the BDOF process. The BDOF process for the first embodiment may be as follows.
a. The BDOF process starts with an input block (designated S1), S1 has dimensions W_1×H_1, and the dimensions of S1 are equal to or smaller than the dimensions of the coding block. When the previous process is block-based, the size of S1 is equal to the coding block. When the previously performed process is sub-block based (sub-block partitioning due to hardware constraints or from previous processing stages), the size of S1 is smaller than the coding block.
b. The input block S1 is divided into N sub-blocks (designated S2), S2 has dimensions W_2×H_2, and the dimensions of S2 are equal to or smaller than the dimensions of S1. For each S2 determined by condition T, it is determined whether S2 should apply BDOF. In some examples, condition T is to check whether the SAD between two predicted signals in reference picture 0 and reference picture 1 is less than a threshold. The sub-blocks in this step define the basic unit for the decision whether to apply BDOF to all samples in the unit.
c. Once it is decided that BDOF should be applied to S2, S2 is divided into M sub-blocks (designated as S3), S3 has dimensions W_3×H_3, and the dimensions of S3 are the dimensions of S2. Equal to or less than. For each S3, a BDOF process is applied to derive the improved motion vector (v' _x ,v' _y ) and (either through motion compensation or adding an offset to the initial predicted signal) The predicted signal of S3 is derived using the motion vector obtained. The sub-blocks in this step define a unit for refined motion vector granularity, and all samples within the unit share the same improved motion.

In the BDOF process of aspect 1, blocks S1, S2, and S3 are defined. The dimensions of S3 may be equal to or less than S2, and the dimensions of S2 may be equal to or less than S1. In other words, W_3 is less than or equal to W_2, H_3 is less than or equal to H_2, W_2 is less than or equal to W_1, and H_2 is less than or equal to H_1. . The size may be fixed, adapted to the picture resolution, or signaled within the bitstream.

One case is that W_3 equals 1 and H_3 equals 1, where S3 is pixel-based. This case may be a pixel-by-pixel BDOF process.

In some examples, S1 is a coding block regardless of whether a previously performed subblock-based process is applied to the coding block.

The second aspect relates to pixel-by-pixel BDOF using a sub-block BDOF bypass scheme. As in the first aspect, when it is determined that a W×H coding block (S1) should apply bidirectional optical flow (BDOF), the coding block is divided into N sub-blocks (S2). . For each subblock, check whether the SAD between the two predicted signals in reference picture 0 and reference picture 1 is smaller than a threshold whether to apply BDOF to the subblock. This is further determined by: If it is decided that BDOF should be applied to a sub-block, an improved motion vector (v' _x , v' _y ) is calculated for each pixel (S3) in the sub-block (S2). The improved motion vector (v' _x ,v' _y ) is used to adjust the prediction signal for that pixel (S3) within the sub-block (S2). An example of pixel-wise BDOF with sub-block bypass process is shown in FIG. 13.

For example, in FIG. 13, video encoder 200 and video decoder 300 may determine that BDOF is enabled for a block of video data, and video encoder 200 and video decoder 300 may determine that BDOF is enabled for the block. A block may be divided into multiple sub-blocks based on the determination to be enabled. As shown in FIG. 13, deriving the number of subblocks and N subblock indices <i = 0> (1300) involves video encoder 200 and video decoder 300 dividing the block into N subblocks. , where each sub-block is identified by a respective index, with the first index being 0. Therefore, the index ranges from 0 to N-1.

Video encoder 200 and video decoder 300 may determine whether prediction samples for all subblocks in the block have been determined (1302), as represented by i < N. If predictive samples for all sub-blocks have been determined (NO at 1302), video encoder 200 and video decoder 300 may finish the process of determining predictive samples for the sub-blocks. However, if prediction samples for all sub-blocks have not been determined (YES at 1302), video encoder 200 and video decoder 300 determine whether the current sub-block of the multiple sub-blocks into which the block has been partitioned The process of determining predictive samples of may continue.

For the current subblock, video encoder 200 and video decoder 300 may determine a distortion value (1304). Since decisions on distortion values may be made for each sub-block, video encoder 200 and video decoder 300 determine respective distortion values for each sub-block of one or more of the plurality of sub-blocks. (eg, a first strain value for a first sub-block, a second strain value for a second sub-block, and so on).

One exemplary method for determining the strain value for the current subblock is to determine the sum of absolute differences (SAD) between a first reference block (ref0) and a second reference block (ref1). This is due to a number of reasons. However, there may be other methods for determining strain values. For example, as described in further detail below, in some examples, video encoder 200 and video decoder 300 perform a Strain values may be determined in such a way that the values can be reused later.

As shown in FIG. 13, video encoder 200 and video decoder 300 may compare the distortion value to a threshold (1306). Based on the comparison, video encoder 200 and video decoder 300 may have two options. The first option may be to perform pixel-by-pixel BDOF, and the second option may be to bypass BDOF. Video encoder 200 and video decoder 300 may not have other options such as sub-block BDOF. Accordingly, video encoder 200 and video decoder 300 determine whether pixel-by-pixel BDOF is performed or BDOF is bypassed for each sub-block of one or more of the plurality of sub-blocks. One may be considered to be determined based on the respective strain value (eg, based on a comparison of the respective strain value with a fixed threshold or a respective threshold).

For example, if the distortion value for the current sub-block is greater than the threshold (NO at 1306), video encoder 200 and video decoder 300 may perform pixel-by-pixel BDOF (1308). If the distortion value for the current sub-block is less than the threshold (YES at 1306), video encoder 200 and video decoder 300 determine whether the distortion value in the sub-block (e.g., by bypassing the BDOF for the sub-block) A predicted signal may be derived (1310).

In one or more examples, video encoder 200 and video decoder 300 may perform for each sub-block of the one or more sub-blocks based on the determination that pixel-wise BDOF is performed or BDOF is bypassed. A predictive sample may be determined. For example, if video encoder 200 and video decoder 300 are to perform BDOF on the current sub-block, video encoder 200 and video decoder 300 may use pixel-wise BDOF techniques to determine prediction samples; , video encoder 200 and video decoder 300 may determine prediction samples without using BDOF techniques if video encoder 200 and video decoder 300 are to bypass BDOF for the current sub-block.

The upper example of FIG. 13 explained how the decision is made whether pixel-by-pixel BDOF is performed or BDOF is bypassed for the current sub-block. Video encoder 200 and video decoder 300 may perform the example techniques described above on a subblock-by-subblock basis.

For example, for a first subblock of one or more subblocks to determine a respective strain value for each subblock of one or more of the subblocks. video encoder 200 and video decoder 300 may determine a first distortion value of the respective distortion values, and for a second sub-block of the one or more sub-blocks, video encoder 200 and Video decoder 300 may determine a second distortion value for each distortion value.

For each sub-block of one or more of the plurality of sub-blocks, one of: per-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values. To determine, for a first subblock of the plurality of subblocks, video encoder 200 and video decoder 300 use a first distortion to determine that BDOF is enabled for the first subblock. The determination may be based on a value (eg, based on the first strain value being greater than a threshold). In this example, based on the determination that BDOF is enabled for the first sub-block, video encoder 200 and video decoder 300 improve the first set of prediction samples for the first sub-block. A pixel-by-pixel motion improvement may be determined (eg, pixel-by-pixel BDOF may be performed). For example, video encoder 200 and video decoder 300 may derive a first motion improvement for a first sample of a first sub-block to improve a first predicted sample; For a second sample of the block, a second motion improvement may be derived to improve the second predicted sample, and so on.

However, for a second subblock of the plurality of subblocks, video encoder 200 and video decoder 300 determine that the BDOF is bypassed based on the second distortion value (e.g., the second distortion value is smaller than a threshold). In this example, based on the decision that the BDOF is bypassed for the second block, video encoder 200 and video decoder 300 improve the second set of prediction samples for the second sub-block. (e.g., BDOF may be bypassed). For example, video encoder 200 and video decoder 300 may bypass deriving a first motion improvement to improve a first predicted sample for a first sample of a first sub-block; For the second sample of the sub-block of , the derivation of the second motion improvement to improve the second predicted sample may be bypassed, and so on.

To determine prediction samples for each subblock of the one or more subblocks based on the determination that pixel-by-pixel BDOF is performed or BDOF is bypassed, video encoder 200 and video decoder 300 include: For the first sub-block, an improved first set of prediction samples for the first sub-block may be determined based on pixel-by-pixel motion improvement for the first sub-block. For the second sub-block, video encoder 200 and video decoder 300 perform a second set of predicted samples without improving the second set of predicted samples based on pixel-by-pixel motion improvement to improve the second set of predicted samples. A second set of predictive samples may be determined.

Within the second aspect, the following describes bypassing the sub-block BDOF. Given a W×H coding block to which bidirectional optical flow (BDOF) is determined to be applied, the number N of subblocks is determined as follows.
a. numSbX = (W > thW) ? (W / thW) : 1
b. numSbY = (H > thH) ? (H / thH) : 1
c. N = numSbX * numSbY

In the above, thW represents the maximum subblock width and thH represents the maximum subblock height. The values of thW and thH are predetermined integer values (eg, thW = thH = 8).

For each sub-block, video encoder 200 and/or video decoder 300 may derive prediction signal predSig0 and prediction signal predSig1 from reference picture 0 and reference picture 1, respectively. The width (sbWidth) and height (sbHeight) of predSig0 and predSig1 are determined as follows.
a. sbWidth = (W > thW) ? thW : W
b. sbHeight = (H > thH) ? thH : H

Whether or not to bypass BDOF in a sub-block is determined by checking the SAD between predSig0 and predSig1. SAD is derived as follows.

In the above equation, Ω'' is a subblock of sbWidth×sbHeight, and I ^(k) (i,j) is the coordinate (i,j ) is the sample value at

If sbSAD is less than the threshold sbDistTh, video encoder 200 and/or video decoder 300 may decide that BDOF should be bypassed in the sub-block, otherwise (sbSAD is equal to or greater than sbDistTh) (large), video encoder 200 and/or video decoder 300 may decide to apply BDOF to the sub-block. The threshold value sbDistTh is derived as follows.
sbDistTh = (sbWidth・sbHeight・s) << n (3-1-1-2)

In the above formula, n and s are predetermined values. For example, n may be derived as n = InternalBitDepth - bitDepth + 1. In the above formula, s represents a scale factor, for example, s = 1. In the current version of VVC, InternalBitDepth is equal to 14 at bitDepth10, so n is equal to 5. The scale s may be 1, 2, 3, other predefined values, or may be signaled in the bitstream.

It should be appreciated that the above describes one example method of determining a threshold and one example method of determining a strain value. However, the example techniques are not so limited. As described in more detail below, in some examples, video encoder 200 and video decoder 300 use The distortion values may be determined in such a way that the calculated calculations can be reused to perform pixel-by-pixel BDOF.

Within the second aspect, the following describes a pixel-wise BDOF. If video encoder 200 and/or video decoder 300 determines that BDOF should be applied to a sub-block of sbWidth x sbHeight, the sub-block is expanded to an area of (sbWidth + 4) x (sbHeight + 4). For each pixel within a sub-block, video encoder 200 and/or video decoder 300 calculates a motion improvement (v' _x ,v' _y ) may be derived. FIG. 14 shows an example of a pixel-by-pixel BDOF of an 8×8 sub-block. Therefore, for pixel-by-pixel BDOF, video encoder 200 and video decoder 300 may determine pixel-by-pixel motion improvement. In sub-block BDOF, motion improvement is for sub-blocks and is not determined on a sample-by-sample (eg, pixel-by-pixel) basis.

In the above, given a sub-block of sbWidth×sbHeight, the following steps are applied in the pixel-wise BDOF process.
- horizontal and vertical slopes of the two predicted signals

and

is calculated by directly calculating the difference between two adjacent samples as in the bidirectional optical flow explained above, and (i,j) is the predicted signal in reference picture 0 and reference picture 1. It is a position with coordinates in the area of (sbWidth + 4) x (sbHeight + 4).
- For each pixel within a sub-block, the following steps are applied.
The auto- and cross-correlations of the gradients S ₁ , S ₂ , S ₃ , S ₅ , and S ₆ are calculated as in the bidirectional optical flow described above, and Ω' is the 5×5 window.
- The motion improvement (v' _x ,v' _y ) is then derived using the cross-correlation and autocorrelation terms.
- Based on the motion improvement and gradient, the following adjustments are calculated to derive the predicted signal of the pixel.

In the above example, I ⁽⁰⁾ refers to the first reference block and I ⁽¹⁾ refers to the second reference block. The adjustment value b'(x,y) is an adjustment value determined based on the pixel-by-pixel motion improvement (v' _x , v' _y ) for each sample in the sub-block. In some examples, I ⁽⁰⁾ (x,y) + I ⁽¹⁾ (x,y) may be considered a predictive block, and therefore b'(x,y) is the one that adjusts the predictive block. may be considered as As shown in equation (3-1-2-1), in order to generate the predicted sample (pred _BDOF (x,y)), there may be an addition of o _offset and a right shift operation of shift5.

A third aspect relates to alternative sub-block SAD derivation. This exemplary technique for deriving SAD may be such that the values determined for SAD derivation may be reused to perform pixel-by-pixel BDOF. That is, video encoder 200 and video decoder may first determine distortion values (eg, SAD values) for a sub-block to determine whether to perform pixel-by-pixel BDOF. If video encoder 200 and video decoder 300 determine that pixel-wise BDOF is to be performed, the calculations performed by video encoder 200 and video decoder 300 to determine whether to perform pixel-wise BDOF are , may be reused to perform pixel-wise BDOF.

For example, one method for determining distortion values for subblocks is to use a first reference block (e.g., identified by a first motion vector) and a second reference block (e.g., identified by a second motion vector). ) determining a second reference block and determining a difference value between samples of the first reference block and samples of the second reference block to determine a distortion value; As an example, as explained above, one method for determining strain values is

It is to decide.

In the above equation, I ⁽¹⁾ (i,j) refers to the samples of the first reference block, and I ⁽⁰⁾ (i,j) refers to the samples of the second reference block. As further explained above, to determine motion improvement, including pixel-by-pixel motion improvement (e.g., _{v '} S ₁ , S ₂ , S ₃ , S ₅ , and S ₆ may be determined. As explained in Equation 1-6-3, part of determining the autocorrelation and cross-correlation of the slopes is determining the intermediate value for θ, where θ = (I ⁽⁰⁾ (i ,j) >> shift2) - (I ⁽¹⁾ (i,j) >> shift2).

Therefore, if pixel-wise BDOF is to be performed on a sub-block, video encoder 200 and video decoder 300 will perform (I ⁽⁰⁾ (i,j) >> shift2) - (I ⁽¹⁾ (i ,j) >> shift2) may need to be determined. In one or more examples, as part of determining distortion values for sub-blocks, video encoder 200 and video decoder 300 determine (I ⁽¹⁾ (i,j)) - (I ⁽⁰⁾ (i,j)) instead of (or in addition to) determining the strain value based on (I ⁽⁰⁾ (i,j) >> shift2) - (I ⁽¹⁾ (i,j) >> shift2) may be used to determine the strain value for the sub-block. That is, to determine a distortion value for a sub-block, such as to determine whether pixel-wise BDOF is to be performed, video encoder 200 and video decoder 300 use (I ^{( 0)} (i,j) >> shift2) - (I ⁽¹⁾ (i,j) >> shift2) may be determined. In this way, if pixel-wise BDOF is to be performed, video encoder ²⁰⁰ and video decoder ^{300 will} ) >> shift2) and those values are the values of θ and are used to determine the motion improvement.

Accordingly, in one or more examples, to determine a respective distortion value for each sub-block of one or more of the plurality of sub-blocks, video encoder 200 and video decoder 300 may For each subblock of one or more of the plurality of subblocks, the first reference block and the second reference block may be configured to be determined. For example, I ⁽⁰⁾ (i,j) may be the first reference block and I ⁽¹⁾ (i,j) may be the second reference block.

Video encoder 200 and video decoder 300 may scale the samples of the first reference block and the samples of the second reference block. For example, video encoder 200 and video decoder 300 may perform the operation I ⁽⁰⁾ (i,j) >> shift2. In this example, the value of shift2 may define how much the value of I ⁽⁰⁾ (i,j) should be scaled to generate the scaled samples of the first reference block. Similarly, video encoder 200 and video decoder 300 may perform the operation I ⁽¹⁾ (i,j) >> shift2. In this example, the value of shift2 may define how much the value of I ⁽¹⁾ (i,j) should be scaled to generate the scaled samples of the second reference block.

Video encoder 200 and video decoder 300 determine a difference value between the scaled samples of the first reference block and the scaled samples of the second reference block to determine respective distortion values. good. For example, video encoder 200 and video decoder 300 may determine (I ⁽⁰⁾ (i,j) >> shift2) - (I ⁽¹⁾ (i,j) >> shift2). Video encoder 200 and video decoder 300 apply distortion values to sub-blocks based on the results of (I ⁽⁰⁾ (i,j) >> shift2) - (I ⁽¹⁾ (i,j) >> shift2). (eg, sbSAD).

As explained above, in some examples there may be a computational gain for video encoder 200 and video decoder 300 such that (I ⁽⁰⁾ (i,j) >> shift2) - (I ⁽¹⁾ The values of (i,j) >> shift2) can be reused for the pixel-by-pixel BDOF. For example, a video may indicate that pixel-wise BDOF is performed on the first subblock of one or more subblocks into which the block being encoded or decoded is divided. Assume that encoder 200 and video decoder 300 have made a decision.

In this example, video encoder 200 and video decoder 300 may determine respective motion improvements for each sample in the first sub-block. That is, video encoder 200 and video decoder 300 do not determine one motion improvement (v _x ,v _y ) that is the same for all samples in the first sub-block, or in addition to The motion improvement (v' _x ,v' _y ) may be determined for each sample of one sub-block.

Video encoder 200 and video decoder 300 perform respective motion improvement operations for each sample in the first sub-block from the samples in the predictive block for the first sub-block based on the respective motion improvement. The sample value may be configured to determine the sample value. For example, as explained above, the formula for determining prediction samples for a pixel-wise BDOF is pred _BDOF (x,y) = (I ⁽⁰⁾ (x,y) + I ⁽¹⁾ (x ,y) + b'(x,y) + ο _offset ) >> shift5.

To determine the pred _BDOF , video encoder ₂₀₀ and video decoder ₃₀₀ use b'(x ,y) may be determined. In some examples, the predicted block is considered the sum of the first reference block and the second reference block (i.e., I ⁽⁰⁾ (i,j) + I ⁽¹⁾ (i,j)). It's fine. As shown in the formula for determining pred _BDOF , video encoder 200 and video decoder 300 convert I ⁽⁰⁾ (i,j) + I ⁽¹⁾ (i,j) to b'(x,y). May be added. Therefore, as part of determining the pred _BDOF , video encoder 200 and video decoder 300 are used to determine their respective motion improvements (e.g., b'(x,y) (v' _x ,v ' _y )) in the predicted block (e.g., if the predicted block is equal to I ⁽⁰⁾ (i,j) + I ⁽¹⁾ (i,j)) From the samples, improved sample values (eg, pred _BDOF ) may be determined.

In other words, video encoder 200 and video decoder 300 may determine a first set of sample values in a first reference block for a first sub-block of the one or more sub-blocks. (For example, you may determine I ⁽⁰⁾ (i,j)). Video encoder 200 and video decoder 300 may scale the first set of sample values using a scale factor to generate a first set of scaled sample values. That is, to perform I ⁽⁰⁾ (i,j) >> shift2, video encoder 200 and video decoder 300 change the number of samples by the scale factor defined by the values of ">>" and "shift2". may be considered as scaling the set of 1.

Video encoder 200 and video decoder 300 may determine a second set of sample values in a second reference block for a first of the one or more subblocks (e.g., I ⁽¹⁾ (i,j) may be determined). Video encoder 200 and video decoder 300 may scale the second set of sample values using a scale factor to generate a second set of scaled sample values. That is, to perform I ⁽¹⁾ (i,j) >> shift2, video encoder 200 and video decoder 300 change the number of samples by the scale factor defined by the values of ">>" and "shift2". may be considered as scaling the set of 2.

Video encoder 200 and video decoder 300 are configured based on the first set of scaled sample values and the second set of scaled sample values (e.g., I ⁽⁰⁾ (i,j) >> shift2 and I ⁽¹⁾ (based on (i,j) >> shift2), a strain value may be determined for the first sub-block. For example, video encoder ²⁰⁰ and video decoder 300 ^may The strain value may be determined by

In one or more examples, assume that pixel-by-pixel BDOF is performed on the first sub-block, as described above. In this example, video encoder 200 and video decoder 300 replay a first set of scaled sample values and a second set of scaled sample values to determine pixel-by-pixel motion improvement for the pixel-by-pixel BDOF. May be used. For example, video encoder 200 and video decoder 300 may use (I ⁽⁰⁾ (i,j) >> shift2) - (I ⁽¹⁾ (i,j) >> shift2) may be reused to determine the per-pixel motion improvement (eg, (v' _x ,v' _y )). As explained above, video encoder 200 and video decoder 300 perform _b '( Pixel-by-pixel motion improvement may be used to determine the adjustment values x,y).

The foregoing indicates that video encoder 200 and video decoder 300 use a first set of scaled sample values and a second set of scaled sample values to determine pixel-by-pixel motion improvement relative to pixel-by-pixel BDOF. An example that can be reused will be explained. However, the technique is not so limited. In some examples, video encoder 200 and video decoder 300 reuse the first set of scaled sample values and the second set of scaled sample values to determine motion improvement for the BDOF. It's fine. That is, the example techniques are not limited to reusing the first set of scaled sample values and the second set of scaled sample values for pixel-by-pixel motion improvement for pixel-by-pixel BDOF. and may be used more generally for motion improvement for BDOF (eg, not limited to per-pixel motion improvement for pixel-by-pixel BDOF). There can be complexity reduction not only for pixel-by-pixel BDOF, but also for sub-block-based BDOF, such as in examples where the BDOF includes motion improvement for the entire sub-block rather than pixel-by-pixel.

Therefore, as in the second aspect, the following is used to derive the sub-block SAD that is used to decide whether a sub-block should be bypassed (i.e. whether the BDOF is bypassed or not). Describe an alternative method. As explained above, the exemplary method uses Equations 1-6 to calculate θ(i,j) as in the bidirectional optical flow described above using two reference signals. Calculate the difference diff(i,j) between .

If it is determined that a sub-block should be subjected to BDOF, diff(i,j) is Can be reused.

Formula (3-1-1-1) in the second embodiment is modified as follows.

In the above equation, I ^(k) (i,j) is the coordinate ( i, j). shift2 is a predetermined value, for example shift2 equals 4. Ω'' is a sub-block area of sbWidth×sbHeight.

To determine the strain value for a subblock based on θ(i,j) = (I ⁽⁰⁾ (i,j) >> shift2) - (I ⁽¹⁾ (i,j) >> shift2) Note that alternative techniques (eg, for determining sbSAD) should not be considered limited to examples where pixel-by-pixel BDOF is performed. Alternative techniques for determining distortion values for sub-blocks may be applicable even in instances where sub-block BDOF or some other BDOF technique is applied. For example, even for sub-block BDOF, video encoder 200 and video decoder 300 may utilize alternative techniques for determining distortion values to determine whether BDOF is performed on the sub-block. It's fine. If BDOF is to be performed, video encoder 200 and video decoder 300 may use alternative techniques to reuse the calculations for determining distortion values to determine motion improvements as part of the sub-block BDOF. (eg, there may be reuse of calculations for alternative techniques to determine strain values).

As explained above, the threshold against which the distortion value is compared to determine whether pixel-wise BDOF is performed or BDOF is bypassed is given by Equation 3-1-1-2 above. sbDistTh is calculated as (sbWidth*sbHeight*s) << n as shown. However, in an alternative technique for determining distortion values, video encoder 200 and video decoder 300 may scale I ⁽⁰⁾ (i,j) by >> shift2 and >> You may scale I ⁽¹⁾ by shift2. Accordingly, in some examples, the manner in which video encoder 200 and video decoder 300 determine sbDistTh may be modified to take >> shift2 scaling into account.

The formula (3-1-1-2) in the second embodiment for calculating sbDistTh is modified as follows.
sbDistTh = (sbWidth・sbHeight・s) << (n - shift2) (3-2-2)

In the above formula, n and s are predetermined values. For example, n may be derived as n = InternalBitDepth - bitDepth + 1. In the above formula, s represents a scale factor, for example, s = 1. In the current version of VVC, InternalBitDepth is equal to 14 at bitDepth10, so n is equal to 5. The scale s may be 1, 2, 3, other predefined values, or may be signaled in the bitstream.

Therefore, to determine the threshold, video encoder 200 and video decoder 300 determine the width of the first of the one or more subblocks (i.e., sbWidth in Equation 3-2-2) ), the height of the first subblock of the one or more subblocks (i.e., sbHeight in Equation 3-2-2), and the first scale factor (i.e., sbHeight in Equation 3-2-2 's' in ) to generate an intermediate value. Video encoder 200 and video decoder 300 may be configured to perform a left shift operation on the intermediate value based on the second scale factor to generate the threshold. For example, the second scale factor may be (n - shift2) in Equation 3-2-2, and the left shift operation is indicated as "<<" in Equation 3-2-2.

In one or more examples, video encoder 200 and video decoder 300 threshold a distortion value (e.g., a distortion value calculated using an alternative technique for determining the distortion value) for the first sub-block. (eg, sbDistTh as determined in equation 3-2-2). Video encoder 200 and video decoder 300 may determine one of whether pixel-wise BDOF is performed or BDOF is bypassed for the first sub-block based on the comparison. For example, if the distortion value is less than a threshold (eg, 1306 YES in FIG. 13), video encoder 200 and video decoder 300 may bypass the BDOF. If the distortion value is greater than the threshold (eg, NO at 1306 in FIG. 13), video encoder 200 and video decoder 300 may perform pixel-by-pixel BDOF.

A fourth aspect relates to determining the values of thW and thH. As in the above aspects, example techniques may be applied to bi-predicted coding blocks. The total number of subblocks is derived from the width and height of the current block and the maximum subblock width (thW) and height (thH) of the subblock.

When the current coding block applies a subblock-based method, e.g., DMVR, the values of thW and thH are equal to or less than the maximum subblock width and height of the previously performed method (e.g., DMVR). It should be smaller than .

The values of thW and thH may be fixed, predetermined values, for example, thW equals 8 and thH equals 8. The values of thW and thH may be adaptive, with the values determined by decoded information from the bitstream. The following describes how the values of thW and thH are made adaptive.
a. Determined by the previous coding method. If the current coding block applies a subblock-based method, thW and thH may be set to the same subblock size as the previously done method. For example, when DMVR is applied to the current coding block, thW is set equal to the DMVR maximum subblock width, e.g., 16, and thH is set equal to the DMVR maximum subblock height, e.g., 16. It is set as follows. Otherwise (if the current coding block does not apply any subblock-based method), thW and thH may be set to a predetermined value, eg, 8.
b. Determined by the current coding block dimensions. In this example, coding blocks that have a total number of luma samples greater than a threshold T (eg, T=128) are set to the larger value of thW and thH. Given a W×H coding block, if W*H is greater than T, set the values of thW and thH equal to 16. Otherwise (W*H is less than or equal to T), set the values of thW and thH equal to 8.

A fifth aspect relates to an example decoder process of applying pixel-wise BDOF with sub-block bypass. The above aspects may be applied within an encoder (eg, video encoder 200) and/or a decoder (eg, video decoder 300). A decoder (eg, video decoder 300) may perform the methods described herein by all or a subset of the following steps for decoding inter-predicted blocks in pictures from a bitstream.
1. Derive the position component (cbX, cbY) as the upper left luma position of the current block by decoding the syntax elements in the bitstream.
2. Derive the size of the current block as width value W and height value H by decoding the syntax elements in the bitstream.
3. Determine that the current block is an inter-predicted block from decoding the elements in the bitstream.
4. Derive the motion vector components (mvL0 and mvL1) and reference indices (refPicL0 and refPicL1) of the current block from decoding the elements in the bitstream.
5. Infer a flag from decoding elements in the bitstream, the flag indicating whether decoder-side motion vector derivation (e.g., DMVR, bidirectional merging, template matching) is applied to the current block. . The flag estimation scheme may be, but is not limited to, the same as the example described above regarding the enabling conditions for when DMVR is enabled. In another example, this flag may be explicitly signaled in the bitstream to avoid complex condition checks at the decoder.
6. Derive an improved motion vector if it is determined that DMVR should be applied to the current block.
7. If it is determined that two (W + 6) × (H + 6) luma prediction sample arrays predSampleL0 and predSampleL1 should be derived from the decoded refPicL0, refPicL1, and motion vectors, and DMVR should be applied, The motion vector is the improved motion vector, otherwise the motion vector is mvL0, mvL1.
8. Infer a flag from decoding elements in the bitstream, the flag indicating whether bidirectional optical flow is currently applied to the block. The flag estimation method may be, but is not limited to, the same as for bidirectional optical flow. In another example, this flag may be explicitly signaled in the bitstream to avoid complex condition checks at the decoder.
9. If the decision is to apply BDOF to the current block, the number of sub-blocks in the horizontal direction numSbX and the number of sub-blocks in the vertical direction numSbY, the sub-block width sbWidth and the sub-block height, according to the flag values mentioned above. Derive sbHeight as follows.
・numSbX = (W > thW) ? (W / thW) : 1
・numSbY = (H > thH) ? (H / thH) : 1
・sbWidth = (W > thW)? thW : W
・sbHeight = (H > thH) ? thH : H
where thW and thH are predetermined integer values (eg, thW = thH = 8).
10. Derive the variable sbDistTh as follows.
sbDistTh = sbWidth * sbHeight * s << (n - shift2)
however,
shift2 is a predetermined value, for example shift2 equals 4.
n is a predetermined value, for example, n = InternalBitDepth - bitDepth + 1 = 5.
s is a scale factor, for example s=1.
11. Set positional components (sbX,sbY) = (0,0) as the upper left luma position of the first subblock of the current block.
12. For each subblock in (sbX,sbY), when sbX is less than W and sbY is less than H, the following steps are applied.
12.1. For x = sbX - 2...sbX + sbWidth + 1, y = sbY - 2...sbY + sbHeight + 1, the variable diff[x][y] is derived as follows.
・diff[x][y] = (predSamplesL0[x][y] >> shift2) - (predSamplesL1[x][y] >> shift2)
However, shift2 is a predetermined value, for example shift2 is equal to 4.
12.2. Derive the variable sbDist as follows.
・sbDist = Σ _i Σ _j Abs(diff[sbX + i][sbY + j])
However, i = 0...sbWidth - 1, j = 0...sbHeight - 1
12.3. If sbDist (which bypasses the subblock BDOF) is smaller than sbDistTh, derive the subblock prediction signal as follows.
12.3.1. For x = sbX...sbX + sbWidth - 1, y = sbY...sbY + sbHeight - 1,
・predSamples[x + cbX][y + cbY] = Clip3(0,(2 ^BitDepth ) - 1,(predSamplesL0[x][y] + predSamplesL1[x][y] + offset5) >> shift5)
however,
shift5 is set equal to Max(3,15 - BitDepth) and
offset5 is set equal to (1 << (shift5 - 1)).
12.4. Otherwise (sbDist is greater than or equal to sbDistTh), the following steps apply.
12.4.1. Variables gradientHL0[x][y], gradientVL0[x][y for x = sbX - 2...sbX + sbWidth + 1, y = sbY - 2...sbY + sbHeight + 1 ], gradientHL1[x][y], and gradientVL1[x][y] are derived as follows.
・gradientHL0[x][y] = (predSamplesL0[x + 1][y] >> shift1) - (predSamplesL0[x - 1][y] >> shift1)
・gradientVL0[x][y] = (predSamplesL0[x][y + 1] >> shift1) - (predSamplesL0[x][y - 1] >> shift1)
・gradientHL1[x][y] = (predSamplesL1[x + 1][y] >> shift1) - (predSamplesL1[x - 1][y] >> shift1)
・gradientVL1[x][y] = (predSamplesL1[x][y + 1] >> shift1) - (predSamplesL1[x][y - 1] >> shift1)
However, shift1 is a predetermined value, for example shift1 is set equal to 6.
12.4.2. Variables tempH[x][y] and tempV[x][y for x = sbX - 2...sbX + sbWidth + 1, y = sbY - 2...sbY + sbHeight + 1 ] is derived as follows.
・tempH[x][y] = (gradientHL0[x][y] + gradientHL1[x][y]) >> shift3
・tempV[x][y] = (gradientVL0[x][y] + gradientVL1[x][y]) >> shift3
However, shift3 is a predetermined value, for example shift3 is set equal to 1.
12.4.3. For each pixel in (piX,piY), piX = sbX...sbX + sbWidth - 1, piY = sbY...sbY + sbHeight - 1, and the following steps are applied: .
12.4.3.1. The variables sGx2, sGy2, sGxGy, sGxdI, and sGydI are derived as follows.
・sGx2 = Σ _i Σ _j Abs(tempH[piX + i][piY + j])
・sGy2 = Σ _i Σ _j Abs(tempV[piX + i][piY + j])
・sGxGy = Σ _i Σ _j (Sign(tempV[piX + i][piY + j]) * tempH[piX + i][piY + j])
・sGxdI = Σ _i Σ _j (-Sign(tempH[piX + i][piY + j]) * diff[piX + i][piY + j])
・sGydI = Σ _i Σ _j (-Sign(tempV[piX + i][piY + j]) * diff[piX + i][piY + j])
However, i = -2...2, j = -2...2.
12.4.3.2. The horizontal and vertical motion offsets of the current pixel are derived as follows.
・v _x = sGx2 > 0 ? Clip3(-mvRefineThres + 1, mvRefineThres - 1, (sGxdI << 2) >> Floor(Log2(sGx2))) : 0
・v _y = sGy2 > 0 ? Clip3(-mvRefineThres + 1, mvRefineThres - 1, ((sGydI << 2) - ((v _x * sGxGy) >> 1)) >> Floor(Log2(sGy2))): 0
However, mvRefineThres is a predetermined value, for example mvRefineThres is set equal to (1 << 4).
12.4.3.3. The predicted signal for the current pixel is derived as follows.
・bdofOffset = v _x * (gradientHL0[piX][piY] - gradientHL1[piX][piY]) + v _y * (gradientVL0[piX][piY] - gradientVL1[piX][piY])
・predSamples[piX + cbW][piY + cbY] = Clip3(0,(2 ^BitDepth ) - 1,(predSamplesL0[xPix][yPix] + predSamplesL1[xPix][yPix] + bdofOffset + offset5) >> shift5)
however,
shift5 is set equal to Max(3,15 - BitDepth) and
offset5 is set equal to (1 << (shift5 - 1)).
12.5. Update the subblock as the upper left luma position as follows.
・sbX = (sbX + sbWidth) < W ? sbX + sbWidth : 0
・sbY = (sbX + sbWidth) < W ? sbY : sbY + sbHeight
13. Using the derived prediction signal of each sub-block to derive a predicted block and using the derived and predicted block for video decoding.

FIG. 15 is a flowchart illustrating an example method for decoding video data in accordance with the techniques of this disclosure. A current block may currently comprise a CU. Although described with respect to video decoder 300 (FIGS. 1 and 4), it should be understood that other devices may be configured to perform a method similar to that of FIG. 15. For example, prediction processing unit 304 and/or motion compensation unit 316 may be configured to perform the example technique of FIG. 15. Prediction processing unit 304 and/or motion compensation unit 316 may be coupled to memory such as DPB 314 or other memory of video decoder 300. In some examples, video decoder 300 may be coupled to memory 120 that stores information used by video decoder 300 to perform the example technique of FIG. 15.

Video decoder 300 may determine that bidirectional optical flow (BDOF) is enabled for a block of video data (1500). For example, video decoder 300 may receive signaling indicating that BDOF is enabled for a block. In some examples, video decoder 300 may infer that BDOF is enabled for a block, such as based on some criteria being met (e.g., without receiving any signaling) (you may decide).

Video decoder 300 may partition the block into multiple subblocks based on a determination that BDOF is enabled for the block (1502). For example, video decoder 300 may divide a block into N subblocks. In some cases, two or more of the sub-blocks may be of different sizes, but it is possible for the sub-blocks to have the same size. Video decoder 300 may determine how to partition blocks based on signaled information or by estimation.

Video decoder 300 may determine a respective distortion value for each subblock of one or more of the plurality of subblocks (1504). There may be various ways in which video decoder 300 may determine the respective distortion values. As an example, video decoder 300 may determine a first reference block (e.g., I ⁽⁰⁾ (i,j)) and a second reference block (e.g., I ⁽¹⁾ (i,j)). may be determined. Video decoder 300 may calculate the sum of absolute differences (SAD) between I ⁽⁰⁾ (i,j) and I ⁽¹⁾ (i,j).

However, the example techniques are not so limited. In some examples, video decoder 300 may perform alternative techniques for determining distortion values, as described above. For example, video decoder 300 may determine a first set of sample values in a first reference block for a first sub-block of the one or more sub-blocks (e.g., I ^{( 0)} (i,j) may be determined). Video decoder 300 may scale the first set of sample values using a scale factor to generate a first set of scaled sample values (e.g., the first set of scaled sample values to generate the set I ⁽⁰⁾ (i,j) << shift2). Video decoder 300 may determine a second set of sample values in a second reference block for a first of the one or more subblocks (e.g., I ⁽¹⁾ (i,j) may be determined). Video decoder 300 may scale the second set of sample values using a scale factor to generate a second set of scaled sample values (e.g., a second set of scaled sample values). to generate the set I ⁽¹⁾ (i,j) << shift2). In one or more examples, to determine a respective distortion value for each subblock of one or more of the plurality of subblocks, video decoder 300 may for each strain value based on the first set of scaled sample values and the second set of scaled sample values (e.g., the first set of scaled sample values and determining the SAD based on the second set of scaled sample values).

Video decoder 300 determines one of: pixel-by-pixel BDOF is performed or BDOF is bypassed for one of the plurality of sub-blocks or each sub-block of the plurality of sub-blocks. The determination may be made based on the strain value (1506). For example, as discussed with respect to FIG. 13, there may be two options for video decoder 300: either perform pixel-by-pixel BDOF on the sub-blocks or bypass BDOF. In some examples, there may be no other options for video decoder 300 when evaluating a sub-block.

In some examples, video encoder 200 and video decoder 300 may determine a threshold to determine whether to perform pixel-by-pixel BDOF or bypass BDOF. One exemplary method for determining the threshold is sbDistTh = (sbWidth·sbHeight·s) << n. However, in examples where alternative techniques for determining distortion values are utilized, video decoder 300 may determine the threshold as sbDistTh = (sbWidth·sbHeight·s) << (n − shift2).

That is, video decoder 300 determines the width of the first of the one or more sub-blocks (e.g., sbWidth), the width of the first of the one or more sub-blocks, to generate the intermediate value. may be multiplied by the sub-block height (eg, sbHeight), and a first scale factor (eg, "s"). Video decoder 300 may perform a left shift operation on the intermediate value based on the second scale factor (e.g., may perform << (n - shift2)) to generate the threshold. , where (n - shift2) is the second scale factor).

Video decoder 300 may compare the respective distortion values for the first sub-block to a threshold value. As shown in decision block 1306 of FIG. 13, for each subblock of one or more of the plurality of subblocks, pixel-wise BDOF is performed or BDOF is bypassed. based on the respective distortion values, video decoder 300 determines whether pixel-wise BDOF is performed or BDOF is bypassed for the first sub-block. May be determined based on comparison.

Video decoder 300 may be configured to determine prediction samples for each subblock of the one or more subblocks based on a determination that pixel-by-pixel BDOF is performed or BDOF is bypassed (1508 ). As an example, to determine prediction samples, video decoder 300 may determine that pixel-by-pixel BDOF is performed on a first sub-block of the one or more sub-blocks. In this example, video decoder 300 may determine a respective motion improvement for each sample in the first sub-block, and for each sample in the first sub-block, video decoder 300 may determine a respective motion improvement for each sample in the first sub-block. Respective improved sample values from samples in the predictive block for the block may be determined based on the respective motion improvement.

For example, video decoder 300 pred _BDOF (x,y) = (I ⁽⁰⁾ (x,y) + I ⁽¹⁾ (x,y) + b'(x,y) + ο _offset ) >> shift5 You can perform the following operation. pred _BDOF may represent the improved sample value. In this example, I ⁽⁰⁾ (x,y) + I ⁽¹⁾ (x,y) may be considered the prediction block. The value for b'(x,y) may be determined by the respective motion refinement (v' _x ,v' _y ) for each sample in the sub-block. Therefore, each improved sample value (eg, pred _BDOF ) is based on the predictive block and the respective motion improvement.

There may be various ways to determine motion improvement (v' _x ,v' _y ). As part of determining motion improvement, video decoder 300 determines θ(i,j) = (I ⁽⁰⁾ (i,j) >> shift2) - (I ⁽¹⁾ (i,j) >> autocorrelation and cross-correlation, including shift2) may be determined. In one or more examples, such as when an alternative technique for determining the distortion value is used, video decoder 300 uses (I ⁽⁰⁾ (i ,j) >> shift2) - (I ⁽¹⁾ (i,j) >> shift2) may have already been determined. In such an example, video decoder 300 uses a first set of scaled sample values (e.g., I ⁽⁰⁾ (i,j) >> shift2 ) and a second set of scaled sample values (e.g., I ⁽¹⁾ (i,j) >> shift2) may be reused (e.g., the value for θ(i,j) is I ^{(0 )} (i,j) >> shift2 and I ⁽¹⁾ (i,j) >> shift2 can be determined without recalculating).

Video decoder 300 may reconstruct the block based on the predicted samples (1510). For example, reconstructing a block based on predicted samples involves video decoder 300 receiving a residual value indicating the difference between the predicted sample and the sample of the block, and adding the residual value to the predicted sample. and reconfiguring the block.

The above provides examples for each sub-block of a block. The following is an example where there are two sub-blocks and pixel-wise BDOF is performed on one sub-block and BDOF is bypassed on the other sub-block.

For example, for a first sub-block of the one or more sub-blocks, video decoder 300 may determine a first of the respective distortion values; For a second sub-block of the blocks, video decoder 300 may determine a second of the respective distortion values.

For a first subblock of the plurality of subblocks, video decoder 300 determines based on a first distortion value (e.g., a threshold) that BDOF is enabled for the first subblock. (based on a comparison of the first strain value with the first strain value). Based on the determination that BDOF is enabled for the first sub-block, video decoder 300 performs pixel-by-pixel motion improvement to improve the first set of prediction samples for the first sub-block. You may decide. For example, video decoder 300 may derive a first motion improvement for a first sample of a first sub-block to improve a first predicted sample, and a first motion improvement for a first sample of a first sub-block to improve a first predicted sample. A second motion improvement may be derived for the sample to improve the second predicted sample, and so on.

For the second subblock of the plurality of subblocks, video decoder 300 determines that the BDOF is bypassed based on the second distortion value (e.g., the second distortion value with a threshold value). (based on comparison). Based on the determination that the BDOF is bypassed for the second block, video decoder 300 determines a pixel-by-pixel motion improvement to improve the second set of prediction samples for the second sub-block. You can bypass that. For example, video decoder 300 may bypass deriving a first motion improvement to improve the first predicted sample for the first sample of the first sub-block, and For the second sample, the derivation of the second motion improvement to improve the second predicted sample may be bypassed, and so on.

For the first sub-block, video decoder 300 may determine an improved first set of predictive samples for the first sub-block based on pixel-wise motion improvement for the first sub-block. (For example, the pred _BDOF may be determined using the example techniques described in this disclosure). For the second sub-block, video decoder 300 performs a second set of predicted samples without improving the second set of predicted samples based on pixel-by-pixel motion improvement to improve the second set of predicted samples. You may decide on a set of 2. That is, for the second sub-block, the BDOF is bypassed. Video decoder 300 may determine predictive samples for the second sub-block based on various techniques, such as determining a predictive block based on a weighted average of reference blocks.

FIG. 16 is a flowchart illustrating an example method of encoding video data in accordance with the techniques of this disclosure. A current block may currently comprise a CU. Although described with respect to video encoder 200 (FIGS. 1 and 3), it should be understood that other devices may be configured to perform a method similar to that of FIG. 16. For example, motion selection unit 202 and/or motion compensation unit 224 may be configured to perform the example technique of FIG. 16. Motion selection unit 202 and/or motion compensation unit 224 may be coupled to memory such as DPB 218 or other memory of video encoder 200. In some examples, video encoder 200 may be coupled to memory 106 that stores information used by video encoder 200 to perform the example technique of FIG. 16. In general, video encoder 200 may perform the same operations as video decoder 300 to generate predictive samples.

Video encoder 200 may determine that bidirectional optical flow (BDOF) is enabled for a block of video data (1600). For example, video encoder 200 may determine rate-distortion costs associated with different coding modes and may determine that BDOF is enabled for a block based on the rate-distortion costs.

Video encoder 200 may divide the block into multiple subblocks when BDOF is enabled for the block (1602). Video encoder 200 may determine a respective distortion value for each subblock of one or more of the plurality of subblocks (1604). Video encoder 200 may perform the same techniques described for video decoder 300 to determine respective distortion values.

Video encoder 200 determines one of: pixel-by-pixel BDOF is performed or BDOF is bypassed for one of the plurality of sub-blocks or each sub-block of the plurality of sub-blocks. The determination may be made based on the strain value (1606). For example, video encoder 200 may not signal information indicating whether per-pixel BDOF is performed or BDOF is bypassed, so video encoder 200 may not perform per-pixel BDOF for each sub-block. The same operations as video decoder 300 may be performed to determine whether BDOF is used or BDOF is bypassed.

Video encoder 200 may determine prediction samples for each subblock of the one or more subblocks based on the determination that pixel-by-pixel BDOF is performed or BDOF is bypassed (1608). Video encoder 200 may signal (1610) residual values between the predicted samples and samples of a block (eg, a respective subblock).

The following describes some example techniques that may be applied together or separately.

Clause 1. A method for decoding video data, the method comprising: determining that bidirectional optical flow (BDOF) is enabled for a block of video data; and determining that BDOF is enabled for the block. dividing the block into a plurality of sub-blocks based on a determination that the plurality of sub-blocks is determined to be a plurality of sub-blocks; and determining a respective strain value for each sub-block of the one or more sub-blocks of the plurality of sub-blocks; For each sub-block of one or more of the plurality of sub-blocks, one of: per-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values. determining a predicted sample for each subblock of the one or more subblocks based on the determination that pixel-wise BDOF is performed or the BDOF is bypassed; and reconstructing the block.

Clause 2. The method of Clause 1, wherein determining a respective strain value for each sub-block of one or more of the sub-blocks comprises: determining the first of the respective strain values for the first sub-block of the sub-block; and determining the first of the respective strain values for the first sub-block of the sub-block; determining a second strain value of the strain values; BDOF is valid for the first sub-block of the multiple sub-blocks, determining which one is bypassed based on the respective distortion values based on the first distortion value and the determination that BDOF is enabled for the first sub-block. 1 and determining, based on the second distortion value, that the BDOF is bypassed for a second subblock of the plurality of subblocks. and determining a pixel-by-pixel motion improvement to improve a second set of predictive samples for the second sub-block based on the determination that the BDOF is bypassed for the second block. and determining prediction samples for each sub-block of the one or more sub-blocks based on the determination that pixel-wise BDOF is performed or BDOF is bypassed. determining, for one sub-block, an improved first set of predictive samples of the first sub-block based on pixel-by-pixel motion improvement for the first sub-block; and and determining a second set of predicted samples for, without improving the second set of predicted samples based on pixel-wise motion improvement to improve the second set of predicted samples. .

Clause 3. The method of clauses 1 and 2, wherein pixel-by-pixel BDOF is performed or BDOF is performed on each sub-block of one or more of the sub-blocks. Determining which one of the subblocks to be bypassed based on the respective distortion values means that pixel-wise BDOF is performed on the first subblock of the one or more subblocks. determining, the method further comprising determining a respective motion improvement for each sample in the first sub-block, a determination that pixel-wise BDOF is performed or BDOF is bypassed; Determining a predicted sample for each sub-block of one or more sub-blocks based on the predicted sample for the first sub-block is determined based on the predicted sample for the first sub-block. determining respective improved sample values from the samples in based on the respective motion improvement.

Clause 4. The method of any of clauses 1 to 3, in which the width of the first of the one or more sub-blocks, the width of the first of the one or more sub-blocks, Multiplying the height of the first sub-block of the blocks, and the first scale factor and left shift operation for the intermediate value based on the second scale factor to generate the threshold and comparing the strain value of each strain value for the first sub-block with a threshold, each sub-block of the one or more sub-blocks of the plurality of sub-blocks. For the first sub-block, determining whether pixel-wise BDOF is performed or BDOF is bypassed is determined based on the respective distortion values. or the BDOF is bypassed based on the comparison.

Clause 5. In the method of any of clauses 1 to 4, the first of the sample values in the first reference block for the first sub-block of the one or more sub-blocks. determining a set of the one or more sub-blocks; scaling the first set of sample values with a scale factor to generate a first set of scaled sample values; Determine a second set of sample values in the second reference block for the first sub-block of and use the scale factor to generate a second set of scaled sample values. and determining a respective distortion value for each sub-block of the one or more sub-blocks of the plurality of sub-blocks comprises scaling the second set of sample values by for a sub-block of , determining a distortion value for each distortion value based on the first set of scaled sample values and the second set of scaled sample values.

Clause 6. The method of Clause 5, wherein for each sub-block of one or more of the plurality of sub-blocks, pixel-wise BDOF is performed or BDOF is bypassed. determining one of the two based on the respective distortion values comprises determining that a pixel-wise BDOF is performed on the first sub-block, and the method includes pixel-wise motion improvement for the pixel-wise BDOF. The method further comprises reusing the first set of scaled sample values and the second set of scaled sample values to determine .

Clause 7. The method of Clause 5, wherein for each sub-block of one or more of the plurality of sub-blocks, pixel-wise BDOF is performed or BDOF is bypassed. determining one of the two based on the respective distortion values comprises determining that a pixel-wise BDOF is performed on the first sub-block, the method for determining a motion improvement for the BDOF. further comprising reusing the first set of scaled sample values and the second set of scaled sample values.

Clause 8. The method of any of Clauses 1 to 7, wherein reconstructing a block comprises receiving a residual value indicating a difference between a predicted sample and a sample of the block; adding the values to the predicted samples to reconstruct the block.

Clause 9. A device for decoding video data, the device comprising a memory configured to store video data and processing circuitry coupled to the memory, the processing circuitry configured to store video data. determines that bidirectional optical flow (BDOF) is enabled for a block, divides the block into multiple sub-blocks based on the decision that BDOF is enabled for the block, and splits the block into multiple sub-blocks based on the decision that BDOF is enabled for the block. Determine a respective strain value for each sub-block of one or more sub-blocks of the blocks, and for each sub-block of one or more sub-blocks of the plurality of sub-blocks, one of determining whether BDOF is performed or BDOF is bypassed based on the respective distortion values; one based on the determination that per-pixel BDOF is performed or BDOF is bypassed; or configured to determine prediction samples for each subblock of the plurality of subblocks and reconstruct the block based on the prediction samples.

Clause 10. In the device of Clause 9, for determining a respective strain value for each sub-block of one or more of the plurality of sub-blocks, the processing circuitry comprises one or determining a first of the respective strain values for a first subblock of the plurality of subblocks; and determining a first of the respective strain values for a first subblock of the one or more subblocks; , configured to determine a second strain value of the respective strain values, and a pixel-wise BDOF is performed for each sub-block of the one or more sub-blocks of the plurality of sub-blocks. or the BDOF is bypassed, based on the respective distortion values, the processing circuitry configures the first sub-block of the plurality of sub-blocks to determining that BDOF is enabled for the sub-block based on the first strain value; Determine pixel-by-pixel motion improvement to improve the first set of predictive samples for the second distortion value that, for the second sub-block of the plurality of sub-blocks, the BDOF is bypassed. and determining a pixel-by-pixel motion improvement to improve a second set of predicted samples for the second sub-block based on the determination that the BDOF is bypassed for the second block. and to determine prediction samples for each subblock of the one or more subblocks based on the determination that pixel-wise BDOF is performed or BDOF is bypassed. , the processing circuitry determines, for the first sub-block, an improved first set of prediction samples of the first sub-block based on pixel-by-pixel motion improvement for the first sub-block; For the second sub-block, determine a second set of predicted samples without improving the second set of predicted samples based on pixel-wise motion improvement to improve the second set of predicted samples. configured to do so.

Clause 11. The device of any of clauses 9 and 10, wherein pixel-by-pixel BDOF is performed or BDOF is performed on each sub-block of one or more of the plurality of sub-blocks. The processing circuitry determines which one of the subblocks is bypassed based on the respective distortion values. the processing circuitry is further configured to determine a respective motion improvement for each sample in the first sub-block to perform pixel-wise BDOF; To determine a predicted sample for each subblock of the one or more subblocks based on the determination that the BDOF is bypassed or the BDOF is bypassed, the processing circuitry includes a In contrast, each improved sample value from samples in the predictive block for the first sub-block is configured to be determined based on the respective motion improvement.

Clause 12. The device of any of clauses 9 to 11, wherein the processing circuitry is configured to reduce the width of the first sub-block of the one or more sub-blocks, 1 Multiply the height of the first subblock of the one or more subblocks, and the first scale factor, and for the intermediate value based on the second scale factor to generate the threshold each subblock of the one or more subblocks of the plurality of subblocks configured to perform a left shift operation and compare the strain value of the respective strain value for the first subblock with a threshold; For the first sub-block, the processing circuitry determines whether pixel-wise BDOF is performed or BDOF is bypassed based on the respective distortion values. The method is configured to determine one of whether pixel-by-pixel BDOF is performed or BDOF is bypassed based on the comparison.

Clause 13. A device according to any of Clauses 9 to 12, wherein the processing circuitry is a sample in a first reference block for a first sub-block of the one or more sub-blocks. Determine a first set of values and scale the first set of sample values using a scale factor to generate a first set of scaled sample values of one or more subblocks. Determine the second set of sample values in the second reference block for the first sub-block out of and use the scale factor to generate the second set of scaled sample values. the processing circuitry is configured to scale the second set of sample values and to determine a respective distortion value for each sub-block of the one or more sub-blocks of the plurality of sub-blocks; , configured to determine a respective strain value for the first sub-block based on the first set of scaled sample values and the second set of scaled sample values.

Clause 14. The device of Clause 13, wherein for each sub-block of one or more of the plurality of sub-blocks, pixel-by-pixel BDOF is performed or BDOF is bypassed. The processing circuitry is configured to determine that a pixel-by-pixel BDOF is performed on the first sub-block to determine one of the first sub-blocks based on the respective distortion values; The first set of scaled sample values and the second set of scaled sample values are configured to reuse the first set of scaled sample values to determine a pixel-by-pixel motion improvement for the pixel-by-pixel BDOF.

Clause 15. The device of Clause 13, wherein for each sub-block of one or more of the plurality of sub-blocks, pixel-by-pixel BDOF is performed or BDOF is bypassed. The processing circuitry is configured to determine that a pixel-by-pixel BDOF is performed on the first sub-block to determine one of the first sub-blocks based on the respective distortion values; The first set of scaled sample values and the second set of scaled sample values are configured to reuse the first set of scaled sample values to determine a motion improvement for the BDOF.

Clause 16. The device of any of Clauses 9 to 15, wherein the processing circuitry receives a residual value indicative of a difference between a predicted sample and a sample of the block, in order to reconstruct the block. , configured to add the residual values to the predicted samples to reconstruct the block.

Clause 17. The device of any of Clauses 9 to 16, further comprising a display configured to display decoded video data.

Clause 18. The device of Clauses 9 to 17, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

Clause 19. A computer-readable storage medium having instructions stored thereon, the instructions, when executed, causing one or more processors to enable bidirectional optical flow (BDOF) for a block of video data. cause the block to be split into multiple sub-blocks based on the decision that BDOF is enabled for the block, each sub-block of one or more of the multiple sub-blocks Let the blocks determine their respective distortion values, and for each sub-block of one or more of the sub-blocks, whether pixel-wise BDOF is performed or BDOF is bypassed. one of the predicted samples for each sub-block of one or more sub-blocks based on the decision that pixel-wise BDOF is performed or BDOF is bypassed. is determined and the block is reconstructed based on the predicted samples.

Clause 20. The computer-readable storage medium of Clause 19, wherein the one or more processors determine a respective strain value for each sub-block of one or more of the plurality of sub-blocks. , instructions cause the one or more processors to determine a first of the respective strain values for a first of the one or more sub-blocks; instructions for causing the one or more processors to determine a second of the respective strain values for a second sub-block of the plurality of sub-blocks; For each sub-block of one or more sub-blocks of , the instructions cause one of performing pixel-wise BDOF or BDOF to be bypassed to be determined based on the respective distortion values. causing the one or more processors to determine, for a first subblock of the plurality of subblocks, that BDOF is enabled for the first subblock based on the first distortion value; , based on the decision that BDOF is enabled for the first sub-block, determine a pixel-by-pixel motion improvement to improve the first set of predictive samples for the first sub-block; for a second sub-block of the sub-blocks of , having the BDOF determined to be bypassed based on the second distortion value; , comprising instructions for causing one or more processors to perform a pixel-by-pixel BDOF, bypassing determining a pixel-by-pixel motion improvement to improve a second set of predicted samples for a second sub-block; The instructions cause the one or more processors to determine prediction samples for each sub-block of the one or more sub-blocks based on a determination that the first sub-block , let an improved first set of predicted samples for the first sub-block be determined based on the pixel-wise motion improvement for the first sub-block, and for the second sub-block, let the improved first set of predicted samples for the first sub-block instructions for determining a second set of predictive samples without improving the second set of predictive samples based on pixel-by-pixel motion improvement to improve the second set of predictive samples.

Clause 21. A computer-readable storage medium according to any of clauses 19 and 20, wherein one or more processors, for each sub-block of one or more of the plurality of sub-blocks, , instructions cause one or more processors to determine one of whether pixel-by-pixel BDOF is performed or BDOF is bypassed based on their respective distortion values. instructions for causing the one or more processors to determine that a pixel-by-pixel BDOF is performed on a first sub-block of the further comprising instructions for causing the one or more processors to determine respective motion improvements for each of the one or more sub-blocks based on the determination that pixel-by-pixel BDOF is performed or BDOF is bypassed; Instructions that cause one or more processors to determine predicted samples for a subblock include, for each sample in a first subblock, a sample in the predicted block for the first subblock. instructions for causing each improved sample value from to be determined based on the respective motion improvement.

Clause 22. The computer-readable storage medium of clauses 19 to 21, in which the width of a first sub-block of the one or more sub-blocks is transmitted to one or more processors to generate an intermediate value; Multiply the height of the first subblock of one or more subblocks, and the first scale factor for the intermediate value based on the second scale factor to generate the threshold. further comprising instructions for causing the one or more processors to perform a left shift operation on one of the plurality of subblocks and to compare the strain value of each strain value for the first subblock with a threshold; instructions for determining, for each sub-block of the one or more sub-blocks, one of whether pixel-wise BDOF is performed or BDOF is bypassed based on the respective distortion values; or instructions for causing the plurality of processors to determine, based on the comparison, one of performing pixel-by-pixel BDOF or bypassing BDOF for the first sub-block.

Clause 23. The computer-readable storage medium of any of clauses 19 to 22, wherein the computer-readable storage medium of any of clauses 19 to 22 is configured to provide one or more processors with a first storage medium for a first sub-block of one or more sub-blocks. determine a first set of sample values in a reference block, and scale the first set of sample values with a scale factor to generate a first set of scaled sample values; or to determine a second set of sample values in a second reference block for a first sub-block of the plurality of sub-blocks and to generate a second set of scaled sample values. , further comprising instructions for causing the one or more processors to scale the second set of sample values using a scale factor, for each sub-block of the one or more sub-blocks of the plurality of sub-blocks. instructions that cause the one or more processors to determine, for a first sub-block, a first set of scaled sample values and a second set of scaled sample values. instructions for determining a strain value for each strain value based on the strain value.

Clause 24. A device for decoding video data, the device comprising: means for determining that bidirectional optical flow (BDOF) is enabled for a block of video data; means for dividing the block into multiple sub-blocks based on a determination that BDOF is enabled and a respective strain value for each sub-block of one or more of the multiple sub-blocks; and one of: pixel-wise BDOF is performed or BDOF is bypassed for each sub-block of the one or more sub-blocks of the plurality of sub-blocks; a means for determining the predicted samples for each sub-block of the one or more sub-blocks based on the respective distortion values and the determination that pixel-wise BDOF is performed or BDOF is bypassed. and means for reconstructing the block based on the predicted samples.

Clause 25. A method of coding video data, the method comprising dividing an input block into a plurality of sub-blocks, the size of the input block being smaller than or equal to the size of the coding unit; Determining that bidirectional optical flow (BDOF) is to be applied to a sub-block of the plurality of sub-blocks based on a condition being satisfied, and dividing the sub-block into a plurality of sub-sub-blocks. determining an improved motion vector for one or more of the sub-subblocks, the improved motion vector for the sub-subblock of the one or more sub-subblocks being the same for multiple samples in the block, and performing a BDOF on the sub-block based on the improved motion vector for the one or more sub-sub-blocks.

Clause 26. A method of coding video data, the method comprising dividing an input block into a plurality of sub-blocks, the size of the input block being smaller than or equal to the size of the coding unit; Determining that bidirectional optical flow (BDOF) is to be applied to a sub-block of the plurality of sub-blocks based on a condition being satisfied, and dividing the sub-block into a plurality of sub-sub-blocks. determining an improved motion vector for each of the one or more samples in the sub-block; and determining an improved motion vector for each of the one or more samples in the sub-block. and performing BDOF on the sub-blocks based on the vectors.

Clause 27. The method of any of Clauses 25 and 26, further comprising bypassing the BDOF for other subblocks of the plurality of subblocks.

Clause 28. In any method of clauses 25 to 27, the condition is satisfied if the sum of absolute differences (SAD) between the two predicted signals in reference picture 0 and reference picture 1 is Involves determining whether it is less than a threshold.

Clause 29. The method of any of Clauses 25 to 28, wherein the size of the input block is thW × thH, where thW and thH are fixed, predetermined values of the value decoded from the bitstream. or based on the size of the block used before the BDOF when encoding or decoding the coding unit.

Clause 30. A method of coding video data, the method comprising any one or a combination of clauses 25 to 29.

Clause 31. The method of any of clauses 25 to 30, wherein performing BDOF comprises performing BDOF as part of decoding the video data.

Clause 32. The method of any of clauses 25 to 31, wherein performing BDOF includes performing BDOF as part of encoding the video data, including within the encoding reconstruction loop. be prepared to do something.

Clause 33. A device for coding video data, the device comprising a memory for storing video data and processing circuitry coupled to the memory, the processing circuitry as described in Clauses 25 to 32. configured to perform any one or a combination thereof.

Clause 34. A device for coding video data, the device comprising one or more means for performing the method of any of clauses 25 to 32.

Clause 35. The device of any of Clauses 33 and 34, further comprising a display configured to display decoded video data.

Clause 36. The device of any of clauses 33 to 35, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

Clause 37. The device of any of clauses 33 to 36, wherein the processing circuitry or means for carrying out comprises a video decoder.

Clause 38. The device of any of Clauses 33 to 37, wherein the processing circuitry or means for carrying out comprises a video encoder.

Clause 39. A computer readable storage medium having instructions stored thereon which, when executed, cause one or more processors to carry out the method of any of Clauses 25 to 32.

Depending on the example, some acts or events of any of the techniques described herein may be performed in different sequences, may be added, integrated, or completely (e.g., not all acts or events described may be necessary for the practice of a technique). Moreover, in some examples, acts or events may be performed in parallel, eg, through multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially.

In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code and executed by a hardware-based processing unit. It's fine. Computer-readable media refers to a computer-readable storage medium that represents a tangible medium, such as a data storage medium or a communication protocol, including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. May include a medium. In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media that is non-transitory, or (2) a communication medium such as a signal or carrier wave. A data storage medium can be accessed by one or more computers or one or more processors to retrieve instructions, code, and/or data structures for implementation of the techniques described in this disclosure. Any possible medium may be used. A computer program product may include a computer readable medium.

By way of example and not limitation, such computer readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any form of instruction or data structure. Any other medium that can be used to store desired program code and that can be accessed by a computer can be included. Also, any connection is properly termed a computer-readable medium. For example, instructions may be transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave. If so, coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included within the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other transitory media, and instead refer to non-transitory tangible storage media. As used herein, "disk" and "disc" refer to compact disc (disc) (CD), laser disc (disc), optical disc (disc), digital versatile disc (disc) (DVD), and floppy disc. discs, and Blu-ray discs, where discs typically reproduce data magnetically and discs typically reproduce data optically using a laser. Combinations of the above should also be included within the scope of computer-readable media.

The instructions may be executed by one or more processors, such as one or more DSPs, general purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuitry. Accordingly, the terms "processor" and "processing circuitry" as used herein refer to any of the structures described above or any other structure suitable for implementing the techniques described herein. Sometimes. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or within a composite codec. may be incorporated into. Also, the techniques may be implemented entirely in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or sets of ICs (eg, chipsets). Although various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, they do not necessarily require implementation by different hardware units. It doesn't necessarily mean you need it. Rather, as explained above, the various units may be combined in a codec hardware unit, or in conjunction with suitable software and/or firmware, one or more of the units as explained above. It may be provided by a collection of interoperable hardware units, including a processor.

Various examples are explained. These and other examples are within the scope of the following claims.

100 video encoding and decoding systems
102 Source device
104 Video Source
106 Memory
108 output interface
110 Computer-readable media
112 Storage devices
114 File server
116 Destination Device
118 Display devices
120 memory
122 input interface
130 Quadrant binary tree (QTBT) structure
132 Coding Tree Unit (CTU)
200 video encoder
202 Mode selection unit
204 Residual generation unit
206 Conversion processing unit
208 Quantization unit
210 Inverse quantization unit
212 Inverse transformation processing unit
214 Reconfiguration unit
216 Filter unit
218 Decoded Picture Buffer (DPB)
220 entropy coding units
222 Motion estimation unit
224 Motion Compensation Unit
226 Intra prediction unit
230 video data memory
300 video decoder
302 Entropy decoding unit
304 Prediction processing unit
306 Inverse quantization unit
308 Inverse conversion processing unit
310 Reconfiguration unit
312 Filter unit
314 Decoded Picture Buffer (DPB)
316 Motion Compensation Unit
318 Intra prediction unit
320 coded picture buffer (CPB) memory
602, 604 block
700 current frames
702 Reference frame
1000 square search patterns
1100, 1102 block

Claims (24)

A method for decoding video data, the method comprising:
determining that bidirectional optical flow (BDOF) is enabled for the block of video data;
dividing the block into a plurality of sub-blocks based on the determination that BDOF is enabled for the block;
determining a respective strain value for each subblock of one or more of the plurality of subblocks;
For each sub-block of the one or more sub-blocks of the plurality of sub-blocks, pixel-by-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values. a step of determining one of the
determining prediction samples for each sub-block of the one or more sub-blocks based on the determination that pixel-wise BDOF is performed or BDOF is bypassed;
reconstructing the block based on the predicted samples. determining a respective strain value for each sub-block of the one or more sub-blocks of the plurality of sub-blocks;
determining a first of the respective strain values for a first sub-block of the one or more sub-blocks;
determining a second of the respective strain values for a second sub-block of the one or more sub-blocks;
For each sub-block of the one or more sub-blocks of the plurality of sub-blocks, pixel-by-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values. The step of determining one of the
determining, for the first sub-block of the plurality of sub-blocks, that BDOF is enabled for the first sub-block based on the first distortion value;
determining a pixel-by-pixel motion improvement to improve a first set of predictive samples for the first sub-block based on the determination that BDOF is enabled for the first sub-block; step and
determining that BDOF is bypassed for the second sub-block of the plurality of sub-blocks based on the second distortion value;
determining a pixel-by-pixel motion improvement to improve a second set of predictive samples for the second sub-block based on the determination that BDOF is bypassed for the second sub-block; and a step to bypass the
determining the predicted samples for each sub-block of the one or more sub-blocks based on the determination that pixel-wise BDOF is performed or BDOF is bypassed;
determining, for the first sub-block, the improved first set of predictive samples of the first sub-block based on the pixel-by-pixel motion improvement for the first sub-block; ,
For the second sub-block, the second set of predictive samples is modified without improving the second set of predictive samples based on the pixel-by-pixel motion improvement to improve the second set of predictive samples. 2. The method of claim 1, comprising: determining a set of 2. For each sub-block of the one or more sub-blocks of the plurality of sub-blocks, pixel-by-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values. determining that pixel-by-pixel BDOF is performed on a first sub-block of the one or more sub-blocks;
The method further comprises determining a respective motion improvement for each sample in the first sub-block;
determining the predicted samples for each sub-block of the one or more sub-blocks based on the determination that pixel-wise BDOF is performed or BDOF is bypassed; 4. Determining, for each sample in a block, a respective improved sample value from a sample in a predictive block for the first sub-block based on the respective motion improvement. The method described in Section 1. to generate an intermediate value, a width of a first sub-block of the one or more sub-blocks, a height of the first sub-block of the one or more sub-blocks, and a first sub-block of the one or more sub-blocks. multiplying by a scale factor of 1;
performing a left shift operation on the intermediate value based on a second scale factor to generate a threshold;
further comprising comparing a strain value of the respective strain values for the first sub-block with the threshold;
For each sub-block of the one or more sub-blocks of the plurality of sub-blocks, pixel-by-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values. determining one of whether pixel-by-pixel BDOF is performed or BDOF is bypassed for the first sub-block based on the comparison; The method according to claim 1. determining a first set of sample values in a first reference block for a first sub-block of the one or more sub-blocks;
scaling the first set of sample values using a scale factor to generate a first set of scaled sample values;
determining a second set of sample values in a second reference block for the first sub-block of the one or more sub-blocks;
scaling the second set of sample values using the scale factor to generate a second set of scaled sample values;
Determining the respective distortion values for each sub-block of one or more of the plurality of sub-blocks includes, for the first sub-block, determining the respective distortion values of the scaled sample values. 2. The method of claim 1, comprising determining a distortion value of the respective distortion values based on the first set and the second set of scaled sample values. For each sub-block of the one or more sub-blocks of the plurality of sub-blocks, pixel-by-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values. determining that a pixel-by-pixel BDOF is performed on the first sub-block;
The method further comprises reusing the first set of scaled sample values and the second set of scaled sample values to determine a pixel-by-pixel motion improvement for a pixel-by-pixel BDOF. 6. The method according to claim 5. For each sub-block of the one or more sub-blocks of the plurality of sub-blocks, pixel-by-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values. determining that a pixel-by-pixel BDOF is performed on the first sub-block;
6. The method further comprises reusing the first set of scaled sample values and the second set of scaled sample values to determine motion improvement for BDOF. Method described. The step of reconfiguring the block comprises:
receiving a residual value indicating a difference between the predicted samples and the samples of the block;
and adding the residual values to the predicted samples to reconstruct the block. A device for decoding video data, the device comprising:
a memory configured to store the video data;
processing circuitry coupled to the memory, the processing circuitry comprising:
determining that bidirectional optical flow (BDOF) is enabled for the block of video data;
dividing the block into a plurality of sub-blocks based on the determination that BDOF is enabled for the block;
determining a respective strain value for each subblock of the one or more subblocks of the plurality of subblocks;
For each sub-block of the one or more sub-blocks of the plurality of sub-blocks, pixel-by-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values. determine one of
determining prediction samples for each subblock of the one or more subblocks based on the determination that pixel-wise BDOF is performed or BDOF is bypassed;
A device configured to reconstruct the block based on the predicted samples. the processing circuitry for determining a respective strain value for each subblock of one or more of the plurality of subblocks;
determining a first of the respective strain values for a first sub-block of the one or more sub-blocks;
configured to determine a second of the respective strain values for a second sub-block of the one or more sub-blocks;
For each sub-block of the one or more sub-blocks of the plurality of sub-blocks, pixel-by-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values. In order to determine one of the processing circuit configurations,
determining, for the first sub-block of the plurality of sub-blocks, that BDOF is enabled for the first sub-block based on the first distortion value;
Based on the determination that BDOF is enabled for the first sub-block, determining a pixel-by-pixel motion improvement to improve a first set of prediction samples for the first sub-block. ,
determining that BDOF is bypassed for the second sub-block of the plurality of sub-blocks based on the second distortion value;
determining a pixel-by-pixel motion improvement to improve a second set of prediction samples for the second sub-block based on the determination that BDOF is bypassed for the second sub-block; configured to bypass
the processing circuitry for determining the prediction samples for each sub-block of the one or more sub-blocks based on the determination that pixel-wise BDOF is performed or BDOF is bypassed; ,
for the first sub-block, determining the improved first set of predictive samples of the first sub-block based on the pixel-by-pixel motion improvement for the first sub-block;
For the second sub-block, the second set of predictive samples is modified without improving the second set of predictive samples based on the pixel-by-pixel motion improvement to improve the second set of predictive samples. 10. The device of claim 9, configured to determine a set of 2. For each sub-block of the one or more sub-blocks of the plurality of sub-blocks, pixel-by-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values. the processing circuitry is configured to determine that a pixel-by-pixel BDOF is performed on a first sub-block of the one or more sub-blocks;
the processing circuitry is further configured to determine a respective motion improvement for each sample in the first sub-block;
the processing circuitry for determining the prediction samples for each sub-block of the one or more sub-blocks based on the determination that pixel-wise BDOF is performed or BDOF is bypassed; , for each sample in the first sub-block, determine a respective improved sample value from a sample in the predictive block for the first sub-block based on the respective motion improvement. 10. The device of claim 9, configured to. The processing circuit configuration is
to generate an intermediate value, a width of a first sub-block of the one or more sub-blocks, a height of the first sub-block of the one or more sub-blocks, and a first sub-block of the one or more sub-blocks. Multiply by a scale factor of 1,
performing a left shift operation on the intermediate value based on a second scale factor to generate a threshold;
configured to compare a strain value of the respective strain values for the first sub-block to the threshold;
For each sub-block of the one or more sub-blocks of the plurality of sub-blocks, pixel-by-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values. The processing circuitry determines one of whether pixel-wise BDOF is performed or BDOF is bypassed for the first sub-block to determine one of: 10. The device of claim 9, configured to determine. The processing circuit configuration is
determining a first set of sample values in a first reference block for a first sub-block of the one or more sub-blocks;
scaling the first set of sample values using a scale factor to generate a first set of scaled sample values;
determining a second set of sample values in a second reference block for the first sub-block of the one or more sub-blocks;
configured to scale the second set of sample values using the scale factor to generate a second set of scaled sample values;
In order to determine the respective distortion values for each sub-block of one or more sub-blocks of the plurality of sub-blocks, the processing circuitry is configured to perform scaling for the first sub-block. 10. The first set of scaled sample values and the second set of scaled sample values are configured to determine a distortion value of the respective distortion values. device. For each sub-block of the one or more sub-blocks of the plurality of sub-blocks, pixel-by-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values. the processing circuitry is configured to determine that a pixel-by-pixel BDOF is performed on the first sub-block;
The processing circuitry is configured to reuse the first set of scaled sample values and the second set of scaled sample values to determine a pixel-by-pixel motion improvement for a pixel-by-pixel BDOF. 14. The device of claim 13. For each sub-block of the one or more sub-blocks of the plurality of sub-blocks, pixel-by-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values. the processing circuitry is configured to determine that a pixel-by-pixel BDOF is performed on the first sub-block;
4. The processing circuitry is configured to reuse the first set of scaled sample values and the second set of scaled sample values to determine motion improvement for a BDOF. Devices described in Section 13. In order to reconfigure the block, the processing circuitry comprises:
receiving a residual value indicating a difference between the predicted sample and the sample of the block;
10. The device of claim 9, configured to add the residual values to the predicted samples to reconstruct the block. 10. The device of claim 9, further comprising a display configured to display decoded video data. 10. The device of claim 9, comprising one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box. A computer-readable storage medium having instructions stored thereon, the instructions, when executed, causing one or more processors to:
determining that bidirectional optical flow (BDOF) is enabled for a block of video data;
dividing the block into a plurality of sub-blocks based on the determination that BDOF is enabled for the block;
determining respective strain values for each subblock of one or more subblocks of the plurality of subblocks;
For each sub-block of the one or more sub-blocks of the plurality of sub-blocks, pixel-by-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values. let them decide on one of the
determining prediction samples for each sub-block of the one or more sub-blocks based on the determination that pixel-wise BDOF is performed or BDOF is bypassed;
A computer-readable storage medium that causes the block to be reconstructed based on the predicted samples. The instructions cause the one or more processors to determine a respective strain value for each sub-block of one or more of the plurality of sub-blocks. to the processor,
determining a first strain value of the respective strain values for a first subblock of the one or more subblocks;
causing a second sub-block of the one or more sub-blocks to determine a second strain value of the respective strain values;
equipped with commands,
causing the one or more processors to perform pixel-by-pixel BDOF on each sub-block of the one or more sub-blocks of the plurality of sub-blocks based on the respective distortion values; or the instruction causes the one or more processors to determine one of: or BDOF is bypassed;
causing the first sub-block of the plurality of sub-blocks to determine, based on the first distortion value, that BDOF is enabled for the first sub-block;
determining a pixel-by-pixel motion improvement to improve a first set of predictive samples for the first sub-block based on the determination that BDOF is enabled for the first sub-block; ,
determining that BDOF is bypassed for the second sub-block of the plurality of sub-blocks based on the second distortion value;
determining a pixel-by-pixel motion improvement to improve a second set of prediction samples for the second sub-block based on the determination that BDOF is bypassed for the second sub-block; bypass the
equipped with commands,
determining the prediction samples for each subblock of the one or more subblocks based on the determination that pixel-by-pixel BDOF is performed or BDOF is bypassed; causing the one or more processors to:
for the first sub-block, causing the improved first set of predictive samples of the first sub-block to be determined based on the pixel-by-pixel motion improvement for the first sub-block;
For the second sub-block, the second set of predictive samples is modified without improving the second set of predictive samples based on the pixel-by-pixel motion improvement to improve the second set of predictive samples. Let the set of 2 be determined,
20. The computer readable storage medium of claim 19, comprising instructions. causing the one or more processors to perform pixel-by-pixel BDOF on each sub-block of the one or more sub-blocks of the plurality of sub-blocks based on the respective distortion values; or a BDOF is bypassed, the instructions cause the one or more processors to determine, pixel by pixel, for a first subblock of the one or more subblocks. comprises an instruction that causes the BDOF to decide to be executed;
the instructions further comprising instructions causing the one or more processors to determine a respective motion improvement for each sample in the first sub-block;
determining the prediction samples for each subblock of the one or more subblocks based on the determination that pixel-by-pixel BDOF is performed or BDOF is bypassed; causing the one or more processors to calculate, for each sample in the first sub-block, a respective improvement from a sample in the predictive block for the first sub-block. 20. The computer-readable storage medium of claim 19, comprising instructions for causing a determined sample value to be determined based on the respective motion improvement. said one or more processors;
to generate an intermediate value, a width of a first sub-block of the one or more sub-blocks, a height of the first sub-block of the one or more sub-blocks, and a first sub-block of the one or more sub-blocks. Multiply by a scale factor of 1,
performing a left shift operation on the intermediate value based on a second scale factor to generate a threshold;
comparing a strain value of the respective strain values for the first sub-block with the threshold;
With more instructions,
causing the one or more processors to perform pixel-by-pixel BDOF on each sub-block of the one or more sub-blocks of the plurality of sub-blocks based on the respective distortion values; or BDOF is bypassed, said instructions causing said one or more processors to determine one of: pixel-by-pixel BDOF is performed for said first sub-block or BDOF is bypassed. 20. The computer-readable storage medium of claim 19, comprising instructions for causing one of the following to be determined based on the comparison. said one or more processors;
determining a first set of sample values in a first reference block for a first sub-block of the one or more sub-blocks;
scaling the first set of sample values using a scale factor to generate a first set of scaled sample values;
determining a second set of sample values in a second reference block for the first sub-block of the one or more sub-blocks;
scaling the second set of sample values using the scale factor to generate a second set of scaled sample values;
With more instructions,
The instructions cause the one or more processors to determine the respective strain value for each sub-block of one or more of the plurality of sub-blocks. a processor for determining a distortion value of the respective distortion values for the first sub-block based on the first set of scaled sample values and the second set of scaled sample values; 20. The computer readable storage medium of claim 19, comprising instructions for determining. A device for decoding video data, the device comprising:
means for determining that bidirectional optical flow (BDOF) is enabled for the block of video data;
means for dividing the block into a plurality of sub-blocks based on the determination that BDOF is enabled for the block;
means for determining a respective strain value for each subblock of one or more of the plurality of subblocks;
For each sub-block of the one or more sub-blocks of the plurality of sub-blocks, pixel-by-pixel BDOF is performed or BDOF is bypassed based on the respective distortion values. a means for determining one of the
means for determining prediction samples for each sub-block of the one or more sub-blocks based on the determination that pixel-wise BDOF is performed or BDOF is bypassed;
and means for reconstructing the block based on the predicted samples.

Images