RU2783333C2 - Transmission of a motion vector predictor in service signals based on prediction subunits - Google Patents

Abstract

FIELD: video data processing.

SUBSTANCE: group of inventions relates to technology for encoding/decoding video data. Proposed is a method for decoding video data. The method includes a stage of receiving encoded video data. The motion flag is then syntactically analysed based on prediction subunits from the encoded video data. In response to determining that the motion flag based on prediction subunits is active, a list of motion prediction candidates at the level of prediction subunits is extracted. In response to determining the motion flag based on prediction subunits is not active, a list of motion prediction candidates at the level of prediction units is extracted. The list of motion prediction candidates at the level of prediction subunits or the list of motion prediction candidates at the level of prediction units is therein at least partially ordered based on motion prediction occurrences in the adjacent blocks. The encoded video data are also decoded using the selected motion vector predictor.

EFFECT: increase in the efficiency of decoding video data.

26 cl, 13 dwg, 4 tbl

Description

[0001] This application claims priority of U.S. Non-Provisional Application No: 16/443113, filed June 17, 2019, which claims priority of U.S. Provisional Application No. 62/687,052, filed June 19, 2018, the contents of which are referenced in their entirety in this document.

The field of technology to which the invention belongs

[0002] This disclosure relates to video coding and video decoding.

State of the art

[0003] Digital video capabilities can be incorporated into a wide range of devices, including digital television receivers, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-readers books, digital cameras, digital recorders, digital media players, video game devices, video game consoles, cellular or satellite radio telephones, so-called smartphones, video conferencing devices, video streaming devices, and the like. Digital video devices implement video coding technologies such as the video coding technologies described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4 Part 10 Advanced Coding video (AVC), the High Efficiency Video Coding (HEVC) standard, and the ITU-T H.265/High Efficiency Video Coding (HEVC) standard and extensions of such standards. Video devices can transmit, receive, encode, decode and/or store digital video information more efficiently by implementing such video coding technologies.

[0004] Video coding techniques include spatial (intra-picture) prediction and/or temporal (inter-picture) prediction in order to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video slice (eg, a video image or part of a video image) may be divided into video blocks, which may also be referred to as coding tree units (CTUs), coding units (CUs), and/or coding nodes. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks in the same picture. Video blocks in an outer-coded (P or B) slice of a picture may use spatial prediction relative to reference samples in neighboring blocks in the same picture, or temporal prediction relative to reference samples in other reference pictures. Pictures may be referred to as frames, and reference pictures may be referred to as reference frames.

The essence of the invention

Первый список включает в себя кандидатов предсказания векторов движения на уровне PU, второй список включает в себя кандидатов предсказания векторов движения на уровне суб-PU.[0005] In general, this disclosure relates to motion vector prediction in video codecs. For example, a motion vector predictor is selected adaptively from two lists of motion vector prediction candidates that are extracted.

The first list includes motion vector prediction candidates at the PU level, the second list includes motion vector prediction candidates at the sub-PU level.

[0006] In one exemplary embodiment, a method for decoding video data is discussed. The method includes receiving encoded video data, parsing a motion flag based on prediction subunits from the encoded video data, in response to determining that the motion flag based on prediction subunits is active, extracting a list of motion prediction candidates at the level of prediction subunits, in response to determining, that the prediction sub-unit-level motion flag is not active, extract the prediction unit-level motion prediction candidate list, select the motion vector predictor from either the prediction sub-unit level motion prediction candidate list or the prediction unit-level motion prediction candidate list, and decode the encoded video data using the selected motion vector predictor. The encoded video data includes the current block, and wherein the motion prediction candidate list at the prediction subunit level and the motion prediction candidate list at the prediction unit level are derived from adjacent blocks of the current block. Neighbor blocks are spatial neighbors of said current block in the current picture, or temporal neighbors of the current block in a previously encoded picture. The motion prediction candidate list at the prediction subunit level or the motion prediction candidate list at the prediction unit level is at least partially ordered based on motion prediction occurrences in neighboring blocks. The pixels in the prediction unit share the first motion vector information and the pixels in the prediction subunit share the second motion vector information, the first motion vector information or the second vector information being determined from the selected motion vector predictor. The motion vector candidate list at the prediction unit level includes at least one of the following: spatial neighbor candidates and temporal neighbor candidates. The list of motion prediction candidates at the prediction subunit level includes at least one of the following: affine motion vector prediction, alternative temporal motion vector prediction (ATMVP), space-time motion vector prediction (STMVP), planar motion vector prediction, and motion vector extraction based on pattern matching (PMVD). The method includes deriving a merge candidate index in response to determining that a motion flag based on prediction subunits is active, wherein the merge candidate index indicates a motion vector predictor to be selected.

[0007] In another exemplary embodiment, equipment for decoding video data is explained. The equipment includes a storage device for storing received encoded video data and a processor. The processor is configured to parse the motion flag based on the prediction subunits from the encoded video data, responsive to determining that the motion flag based on the prediction subunits is active, extract the list of motion prediction candidates at the prediction subunit level, responsive to determining that the motion flag is on is not active, extract a list of motion prediction candidates at the prediction unit level, select a motion vector predictor from either the list of motion prediction candidates at the prediction subunit level or from the list of motion prediction candidates at the prediction unit level, and decode the encoded video data using the selected predictor. motion vector. The encoded video data includes the current block, wherein the motion prediction candidate list at the prediction subunit level and the motion prediction candidate list at the prediction unit level are derived from neighboring blocks of the current block. Neighbor blocks are spatial neighbors of the current block in the current picture, or temporal neighbors of the current block in a previously encoded picture. The motion prediction candidate list at the prediction subunit level or the motion prediction candidate list at the prediction unit level is at least partially ordered based on motion prediction occurrences in neighboring blocks. The pixels in the prediction unit share the first motion vector information and the pixels in the prediction subunit share the second motion vector information, the first motion vector information or the second vector information being determined from the selected motion vector predictor. The motion vector candidate list at the prediction unit level includes at least one of the following: spatial neighbor candidates and temporal neighbor candidates. The list of motion prediction candidates at the prediction subunit level includes at least one of the following: affine motion vector prediction, alternative temporal motion vector prediction (ATMVP), space-time motion vector prediction (STMVP), planar motion vector prediction, and motion vector extraction based on pattern matching (PMVD). The processor is further configured to extract a merge candidate index in response to determining that a motion flag based on prediction subunits is active, wherein the merge candidate index indicates a motion vector predictor to be selected.

[0008] In another exemplary embodiment, equipment for decoding video data is explained. The equipment includes a memory means for storing received encoded video data and a processor means. The processor means is configured to parse a motion flag based on the prediction subunits from the encoded video data, in response to determining that the motion flag based on the prediction subunits is active, extract a list of motion prediction candidates at the level of the prediction subunits, in response to determining that the motion flag is is not active, extract a list of motion prediction candidates at the prediction unit level, select a motion vector predictor from either the list of motion prediction candidates at the prediction subunit level or from the list of motion prediction candidates at the prediction unit level, and decode the encoded video data using the selected motion vector predictor. The encoded video data includes the current block, wherein the motion prediction candidate list at the prediction subunit level and the motion prediction candidate list at the prediction unit level are derived from neighboring blocks of the current block. Neighbor blocks are spatial neighbors of the current block in the current picture, or temporal neighbors of the current block in a previously encoded picture. The motion prediction candidate list at the prediction subunit level or the motion prediction candidate list at the prediction unit level is at least partially ordered based on motion prediction occurrences in neighboring blocks. The pixels in the prediction unit share the first motion vector information and the pixels in the prediction subunit share the second motion vector information, the first motion vector information or the second vector information being determined from the selected motion vector predictor. The motion vector candidate list at the prediction unit level includes at least one of the following: spatial neighbor candidates and temporal neighbor candidates. The list of motion prediction candidates at the prediction subunit level includes at least one of the following: affine motion vector prediction, alternative temporal motion vector prediction (ATMVP), space-time motion vector prediction (STMVP), planar motion vector prediction, and motion vector extraction based on pattern matching (PMVD). The processor means is further configured to extract the merge candidate index in response to determining that the motion flag based on the prediction subunits is active, wherein the merge candidate index indicates the motion vector predictor to be selected.

[0009] In another exemplary embodiment, a non-volatile computer-readable storage medium having stored instructions that, when executed, instructs one or more processors to perform a method. The method includes receiving encoded video data, parsing a motion flag based on prediction subunits from the encoded video data, in response to determining that the motion flag based on prediction subunits is active, extracting a list of motion prediction candidates at the level of prediction subunits, in response to determining, that the prediction sub-unit-level motion flag is not active, extract the prediction unit-level motion prediction candidate list, select the motion vector predictor from either the prediction sub-unit level motion prediction candidate list or the prediction unit-level motion prediction candidate list, and decode the encoded video data using the selected motion vector predictor. The encoded video data includes the current block, wherein the motion prediction candidate list at the prediction subunit level and the motion prediction candidate list at the prediction unit level are derived from neighboring blocks of the current block. Neighbor blocks are spatial neighbors of the current block in the current picture, or temporal neighbors of the current block in a previously encoded picture. The motion prediction candidate list at the prediction subunit level or the motion prediction candidate list at the prediction unit level is at least partially ordered based on motion prediction occurrences in neighboring blocks. The pixels in the prediction unit share the first motion vector information and the pixels in the prediction subunit share the second motion vector information, the first motion vector information or the second vector information being determined from the selected motion vector predictor. The motion vector candidate list at the prediction unit level includes at least one of the following: spatial neighbor candidates and temporal neighbor candidates. The list of motion prediction candidates at the prediction subunit level includes at least one of the following: affine motion vector prediction, alternative temporal motion vector prediction (ATMVP), space-time motion vector prediction (STMVP), planar motion vector prediction, and motion vector extraction based on pattern matching (PMVD). The method includes deriving a merge candidate index in response to determining that a motion flag based on prediction subunits is active, wherein the merge candidate index indicates a motion vector predictor to be selected.

[0010] Details of one or more examples are set forth in the accompanying drawings and in the description below. Other features, purposes and advantages should become apparent from the description, drawings and claims.

Brief description of the drawings

[0011] FIG. 1 is a block diagram illustrating an exemplary video encoding and decoding system that may perform the techniques of this disclosure.

[0012] FIG. 2A and 2B are conceptual diagrams illustrating an exemplary binary quadtree (QTBT) structure and corresponding coding tree unit (CTU).

[0013] FIG. 3 is a flowchart illustrating the extraction of motion vector predictors.

[0014] FIG. 4 is a conceptual diagram illustrating spatial neighbor motion vector candidates for the merge mode.

[0015] FIG. 5 is a conceptual diagram illustrating temporal motion vector candidates.

[0016] FIG. 6 illustrates motion vector candidate block selection techniques.

[0017] FIG. 7 is a conceptual diagram illustrating Space Time Motion Vector Prediction (STMVP).

[0018] FIG. 8 is a conceptual diagram illustrating bilateral matching technologies.

[0019] FIG. 9 is a conceptual diagram illustrating pattern matching technologies.

[0020] FIG. 10 is a conceptual diagram illustrating planar motion vector prediction.

[0021] FIG. 11 is a block diagram illustrating an exemplary video encoder that may perform the techniques of this disclosure.

[0022] FIG. 12 is a block diagram illustrating an exemplary video decoder that may perform the techniques of this disclosure.

Detailed description of the invention

[0023] This disclosure relates to decoder-side motion vector extraction (DMVD). The decoder side motion vector extraction techniques described in this disclosure may be used in conjunction with any of the existing video codecs such as HEVC (High Efficiency Video Coding Standard) or may be used as coding techniques for any future video coding standards such as H.266 Universal Video Coding (VVC) and Fundamental Video Coding (EVC).

[0024] FIG. 1 is a block diagram illustrating an exemplary video encoding and decoding system 100 that may perform the techniques of this disclosure. The technologies of this disclosure are generally directed to encoding (encoding and/or decoding) video data and, in particular, relate to the technologies explained herein. In general, video data includes any video processing data. Thus, the video data may include raw unencoded video, encoded video, decoded (eg, reconstructed) video, and video metadata such as ancillary data.

[0025] As shown in FIG. 1, system 100 includes a source device 102 that provides encoded video data to be decoded and displayed by target device 116, in this example. In particular, source device 102 provides video data to destination device 116 via computer-readable medium 110. Source device 102 and destination device 116 may comprise any of a wide range of devices including desktop computers, laptops (i.e., laptops), tablet computers. , set-top boxes, handsets, such as smartphones, televisions, cameras, display devices, digital media players, video game consoles, video streaming device, etc. In some cases, source device 102 and destination device 116 may be equipped with wireless communication capabilities and may therefore be referred to as "wireless devices".

[0026] In the example of FIG. 1, source device 102 includes a video source 104, a storage device 106, a video encoder 200, and an output interface 108. The target device 116 includes an input interface 122, a video decoder 300, a storage device 120, and a display device 118. In accordance with this disclosure, the video encoder 200 of the source device 102 and the video decoder 300 of the target device 116 may be configured to apply techniques to extract motion vectors on the decoder side. Thus, source device 102 represents an example of a video encoding device, while destination device 116 represents an example of a video decoding device. In other examples, the source device and the target device may include other components or arrangements. For example, source device 102 may receive video data from an external video source such as an external camera. Likewise, target device 116 may interact with an external display device instead of including an integrated display device.

[0027] System 100, as shown in FIG. 1 is just one example. In general, any digital video encoding and/or decoding device may perform techniques for extracting motion vectors on the decoder side. Source device 102 and destination device 116 are merely examples of such encoders, in which source device 102 generates encoded video data for transmission to destination device 116. This disclosure refers to an "encoder" as the device that performs the encoding (encoding and/or decoding) data. Thus, video encoder 200 and video decoder 300 represent examples of encoding devices, specifically a video encoder and a video decoder, respectively. In some examples, devices 102, 116 may operate substantially symmetrically such that each of devices 102, 116 includes video encoding and decoding components. Therefore, system 100 can support one-way and two-way video transmission between video devices 102, 116, such as for video streaming, video playback, video broadcast, or video telephony.

[0028] In general, video source 104 represents a source of video data (ie, raw, uncoded video data) and provides a sequential series of pictures (also referred to as frames) of video data to video encoder 200, which encodes the data for the pictures. Video source 104 of source device 102 may include a video capture device such as a video camera, a video archive containing previously captured raw video, and/or a live video interface to receive video from a video content provider. As a further alternative, video source 104 may generate computer graphics data as source video, or a combination of live video, archived video, and computer-generated video. In each case, video encoder 200 encodes captured, pre-captured, or machine-generated video data. Video encoder 200 may re-arrange pictures from received order (sometimes referred to as "display order") to encoding order for encoding. Video encoder 200 may generate a bit stream including encoded video data. Source device 102 may then output the encoded video data via output interface 108 to computer-readable media 110 for reception and/or retrieval, such as via input interface 122 of target device 116.

[0029] The storage device 106 of the source device 102 and the storage device 120 of the target device 116 are general purpose storage devices. In some example, memories 106, 120 may store raw video data, such as raw video from video source 104 and raw decoded video data from video decoder 300. Additionally or alternatively, memories 106, 120 may store program instructions executed, for example, by video encoder 200 and video decoder 300, respectively. Although shown separately from video encoder 200 and video decoder 300 in this example, it should be understood that video encoder 200 and video decoder 300 may also include internal storage devices for functionally similar or equivalent purposes. Additionally, memories 106, 120 may store encoded video data, such as output from video encoder 200 and input to video decoder 300. In some examples, portions of memories 106, 120 may be allocated as one or more video buffers, such as to store raw, decoded and/or encoded video data.

[0030] Computer-readable media 110 may represent any type of media or device capable of transporting encoded video data from source device 102 to destination device 116. In one example, computer-readable media 110 represents a communication medium to allow source device 102 to transmit encoded video data directly to a target device. device 116 in real time, such as via an RF network or a computer network. The output interface 108 may modulate a transmitted signal including encoded video data, and the input interface 122 may modulate a received transmission signal according to a communication standard such as a wireless communication protocol. The communication medium may comprise any wireless or wired communication medium such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-switched network such as a local area network, a wide area network, or a wide area network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful to facilitate transmission from source device 102 to destination device 116.

[0031] In some examples, source device 102 may output coded data from output interface 108 to storage device 112. Similarly, the target device 116 can access the encoded data from the storage device 112 via the input interface 122 . Storage device 112 may include any number of distributed or locally available storage media such as a hard drive, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile storage, or any other suitable digital media. data storage for storing encoded video data.

[0032] In some examples, source device 102 may output encoded video data to file server 114 or other intermediate storage device that may store encoded video generated by source device 102. Target device 116 may access stored video data from file server 114 via streaming or download. File server 114 may be any type of server device capable of storing encoded video data and transmitting that encoded video data to target device 116. File server 114 may be a web server (e.g. for a web site), file transfer protocol (FTP) server, , a networked content delivery device, or a Network Attached Storage System (NAS) protocol device. The target device 116 may access the encoded video data from the file server 114 via any standard data connection, including an Internet connection. It may include a wireless link (eg, Wi-Fi connection), a wired connection (eg, DSL, cable modem, etc.), or a combination thereof that is appropriate for accessing encoded video data stored on a file server. 114. File server 114 and input interface 122 may be configured to operate according to a streaming protocol, a download-based transfer protocol, or a combination of the above.

[0033] Output interface 108 and input interface 122 may represent wireless transmitter/receiver devices, modems, wired network components (eg, Ethernet cards), wireless communication components that operate according to any of a variety of IEEE 802.11 standards, or other physical Components. In examples in which the output interface 108 and the input interface 122 comprise wireless components, the output interface 108 and the input interface 122 may be configured to transmit data such as encoded video data according to a cellular communication standard such as 4G, 4G LTE (standard long-term development), advanced LTE, 5G, etc. In some examples, in which the output interface 108 comprises a wireless transmitter, the output interface 108 and the input interface 122 may be configured to transmit data, such as encoded video data, according to other wireless communication standards such as the IEEE 802.11 specification, IEEE 802.15 - specification (eg ZigBee™), Bluetooth™ standard, etc. In some examples, source device 102 and/or target device 116 may include respective on-chip (SoC) devices. For example, source device 102 may include an SoC device to perform functionality attributed to video encoder 200 and/or output interface 108, and target device 116 may include an SoC device to perform functionality attributed to video decoder 300 and/or interface 122 entries.

[0034] The techniques of this disclosure may be applied to video coding in support of any of a variety of multimedia applications such as over-the-air television broadcasts, cable television, satellite television, video streaming over the Internet such as dynamic adaptive streaming over HTTP (DASH), digital video that is encoded to a storage medium, decoding of digital video stored on a storage medium, or other applications.

[0035] The input interface 122 of the target device 116 receives an encoded video bitstream from a computer-readable medium 110 (eg, non-volatile storage device 112, file server 114, etc.). The machine-readable medium 110 of the encoded video bitstreams may include overhead information specified by video encoder 200 that is also used by video decoder 300, such as syntax elements having values that describe the characteristics and/or processing of video blocks or other coded units (e.g., runs macroblocks, pictures, groups of pictures, sequences, etc.). The display device 118 displays the decoded pictures of the decoded video data to the user. The display device 118 may represent any of a variety of display devices such as a cathode ray tube display (CRT), liquid crystal display (LCD), plasma display, organic light emitting diode (OLED) display, or other type of display device.

[0036] Although not shown in FIG. 1, in some examples, video encoder 200 and video decoder 300 may be integrated with an audio encoder and/or audio decoder and may include appropriate multiplexer-demultiplexer modules or other hardware and software in order to process media streams including both audio , and video in the general data stream. If applicable, the multiplexer-demultiplexer modules may conform to the ITU H.223 Multiplexer Protocol or other protocols such as User Datagram Protocol (UDP).

[0037] Video encoder 200 and video decoder 300 may be implemented as any of a variety of suitable encoder and/or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays ( FPGA) as discrete logic, software, hardware, firmware, or any combination of the above. When the technologies are implemented in part in software, a device may store instructions for the software on a suitable non-volatile computer-readable medium and execute the instructions in hardware using one or more processors to implement the technologies of this disclosure. Each of video encoder 200 and video decoder 300 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder/decoder (codec) in a respective device. The device including video encoder 200 and/or video decoder 300 may include an integrated circuit, a microprocessor, and/or a wireless communication device such as a cellular phone.

[0038] Video coding standards include ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-T H.262 or ISO/IEC MPEG-2 Visual, ITU-T H.263, ISO/IEC MPEG -4 Visual and ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC), which includes Scalable Video Coding (SVC) and Multi-View Video Coding (MVC) extensions.

[0039] A new video coding standard, namely the High Efficiency Video Coding (HEVC) or ITU-T H.265 standard, including its range extension, multi-view extension (MV-HEVC) and scalable extension (SHVC), is developed through the Joint Group Joint Working Group on Video Standards (JCT-VC), and the Joint Working Group on 3D Video Coding Extensions (JCT-3V) of the Video Coding Expert Group (VCEG) of ITU-T and the Motion Picture Expert Group (MPEG) of ISO/ IEC.

[0040] ITU-T VCEG (Q6/16) and ISO/IEC MPEG (JTC 1/SC 29/WG 11) are now exploring the potential need to standardize a future video coding technology with compression capabilities that exceed the compression capabilities of the current HEVC standard ( including its current extensions and near-term extensions for screen content coding and high dynamic range coding). The groups jointly conduct these studies in a collaborative collective effort known as the Joint Video Research Team (JVET) to evaluate compression technology design solutions proposed by their experts in the field. JVET held its first meeting on October 19-21, 2015. In addition, the latest version of the reference software, ie. The Joint Research Group Test Model 7 (JEM7) standard can be downloaded at:

https://jvet.hhi.fraunhofer.de/svn/svn_HMJEMSoftware/tags/HM-16.6-JEM-57.0/

The algorithm description of the Joint Research Group Test Model 7 (JEM7) standard may be referred to as "JVET-G1001".

[0041] Video encoder 200 and video decoder 300 may operate according to a video coding standard such as ITU-T H.265, also referred to as the "High Efficiency Video Coding (HEVC) standard", or extensions thereof, such as multi-view and/or scalable video coding extensions. video encoding. Alternatively, video encoder 200 and video decoder 300 may operate according to other proprietary or industry standards, such as the Joint Research Establishment Test Model (JEM) standard or ITU-T H.266, also referred to as "Versatile Video Coding (VVC)". A recent draft VVC standard is described in Bross et al. "Versatile Video Coding (Draft 3)", Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 13th Conference: Marrakesh, MA, January 9-18, 2019, JVET-M1001-v3 (hereinafter referred to as "VVC Project 4"). However, the techniques of this disclosure are not limited to any particular coding standard.

As explained, video decoder 300 may be configured to perform one or more of the techniques of this disclosure. For example, video decoder 300 may parse a motion flag based on prediction subunits from the received encoded video data, extract a list of motion prediction candidates at the prediction subunit level if the motion flag based on prediction subunits is active, extract a list of motion prediction candidates at the prediction subunit level if the flag motion based prediction subunit is not active, and decode the encoded video data using the selected motion vector predictor.

[0042] In general, video encoder 200 and video decoder 300 may perform block-based picture coding. The term "block" generally means a structure including data to be processed (eg, encoded, decoded, or otherwise used in an encoding and/or decoding process). For example, the block may include a two-dimensional matrix of luma and/or chrominance data samples. In general, video encoder 200 and video decoder 300 may encode video data represented in YUV- (eg, Y, Cb, Cr) format. Thus, instead of encoding the red, green, and blue (RGB) color data for the picture samples, video encoder 200 and video decoder 300 may encode luma and chroma components, where the chroma components may include red hue and blue hue chrominance components. . In some examples, video encoder 200 converts received RGB-formatted data to YUV prior to encoding, and video decoder 300 converts YUV to RGB. Alternatively, pre- and post-processing modules (not shown) can perform these transformations.

[0043] This disclosure may generally mean encoding (eg, encoding and decoding) of pictures, which includes the process of encoding or decoding picture data. Likewise, this disclosure may refer to picture block coding, which includes the process of encoding or decoding data for blocks, such as prediction and/or residual coding. An encoded video bitstream generally includes a sequence of values for syntax elements representing encoding decisions (eg, encoding modes) and segmentation of pictures into blocks. Thus, references to encoding a picture or block are generally to be understood as encoding values for the syntax elements that form the picture or block.

[0044] HEVC defines various blocks including coding units (CU), prediction units (PU), and transform units (TU). According to HEVC, a video encoder (eg, video encoder 200) segments a coding tree unit (CTU) into CUs according to a quadtree structure. Thus, the video encoder segments the CTU and CU into four equal non-overlapping squares, and each quadtree node has either zero or four child nodes. Nodes without child nodes may be referred to as "leaf nodes" and the CUs of such leaf nodes may include one or more PUs and/or one or more TUs. The video encoder may optionally segment the PU and TU. For example, in HEVC, a residual quadtree (RQT) represents the segmentation of a TU. In HEVC, PUs represent inter prediction data while TUs represent residual data. CUs that are intra predicted include intra prediction information such as an intra mode indicator.

[0045] As another example, video encoder 200 and video decoder 300 may be configured to operate according to JEM or VVC. According to JEM or VVC, a video encoder (eg, video encoder 200) segments a picture into multiple coding tree units (CTUs). Video encoder 200 may segment the CTU according to a tree structure, such as a quadtree binary tree (QTBT) structure. The QTBT framework removes the concepts of several types of segmentation, such as the division between CUs, PUs, and HEVC TUs. The JEM QTBT structure includes two levels: a first level segmented according to quadtree segmentation and a second level segmented according to binary tree segmentation. The root node of the QTBT structure corresponds to the CTU. The leaf nodes of binary trees correspond to coding units (CUs).

[0046] In an MTT segmentation structure, blocks may be segmented using a quadtree segment (QT), a binary tree segment (BT), and one or more types of ternary tree segments (TT). A ternary tree segment is a segment in which a block is split into three subblocks. In some examples, a ternary tree segment divides a block into three subblocks without splitting the original block through the center. Segmentation types in MTT (eg, QT, BT, and TT) can be symmetrical or asymmetric.

[0047] In some examples, video encoder 200 and video decoder 300 may use a single QTBT or MTT structure to represent each of the luminance and chrominance components, while in other examples, video encoder 200 and video decoder 300 may use two or more than a QTBT or MTT structure, eg one QTBT/MTT structure for the luminance component and another QTBT or MTT structure for both chrominance components (or two QTBT/MTT structures for the respective chrominance components).

[0048] Video encoder 200 and video decoder 300 may be configured to use quadtree segmentation according to HEVC, QTBT segmentation, or MTT segmentation, or other segmentation structures. For purposes of explanation, the description of the technologies of this disclosure is provided in relation to QTBT segmentation. However, it should be understood that the techniques of this disclosure may also be applied to video encoders configured to also use quadtree segmentation or other types of segmentation.

[0049] This disclosure may use "NxN" and "N by N" interchangeably to refer to sizes in samples of a block (e.g., CU or other video block) in terms of vertical and horizontal dimensions, e.g., 16x samples 16 or 16 by 16 samples. In general, a 16×16 CU should have 16 pixels in the vertical direction (y=16) and 16 pixels in the horizontal direction (x=16). Similarly, an NxN CU generally has N samples in the vertical direction and N samples in the horizontal direction, with N representing a non-negative integer value. Samples in a CU can be placed in rows and columns. In addition, CUs do not need to have the same number of samples in the horizontal direction and in the vertical direction. For example, CUs may contain NxM samples, with M not necessarily equal to N.

[0050] Video encoder 200 encodes video data for CUs representing prediction information and/or residual information and other information. The prediction information indicates how the CU should be predicted in order to generate a prediction block for the CU. The residual information is generally the successive sample differences between pre-coding CU samples and the prediction block.

[0051] To predict the CU, video encoder 200 may generally generate a prediction block for the CU via inter prediction or intra prediction. Mutual prediction generally means predicting a CU from previously encoded picture data, while intra prediction generally means predicting a CU from previously encoded identical picture data. To perform inter prediction, video encoder 200 may generate a prediction block using one or more motion vectors. Video encoder 200 may generally perform a motion search to identify a reference block that closely matches the CU, eg, in terms of differences between the CU and the reference block. Video encoder 200 may compute a difference metric using sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute differences (MAD), root mean square differences (MSD), or other such difference calculations to determine whether or not a reference block closely matches. with the current CU. In some examples, video encoder 200 may predict the current CU using unidirectional prediction or bidirectional prediction.

[0052] Some examples of JEM and VVC also provide an affine motion compensation mode, which can be considered a mutual prediction mode. In affine motion compensation mode, video encoder 200 may determine two or more motion vectors that represent non-translational motion in space, such as zoom in or out, rotation, perspective motion, or other types of irregular motion.

[0053] To perform intra prediction, video encoder 200 may select an intra prediction mode to generate a prediction block. Some examples of JEM and VVC provide sixty-seven intra prediction modes including various directional modes as well as planar mode and DC mode. In general, video encoder 200 selects an intra prediction mode that describes adjacent samples for the current block (eg, CU) from which the current block samples can be predicted. Such samples may generally be above, above, and to the left or left of the current block in the same picture as the current block, as long as video encoder 200 encodes CTUs and CUs in raster scan order (left to right, top to bottom).

[0054] Video encoder 200 encodes data representing a prediction mode for the current block. For example, for inter prediction modes, video encoder 200 may encode data representing which of the various available inter prediction modes is in use, as well as motion information for the corresponding mode. For unidirectional or bidirectional inter prediction, for example, video encoder 200 may encode the motion vectors using advanced motion vector prediction (AMVP) or merge mode. Video encoder 200 may use similar modes to encode motion vectors for the affine motion compensation mode.

[0055] After prediction, such as intra-prediction or inter-prediction of a block, video encoder 200 may calculate residual data for the block. Residual data, such as a residual block, represents a sample by way of approximate differences between a block and a prediction block for a block generated using the corresponding prediction mode. Video encoder 200 may apply one or more transforms to the residual block to generate transformed data in a transform region instead of a sample region. For example, video encoder 200 may apply a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to the video residual data. Additionally, video encoder 200 may apply a secondary transform after the first transform, such as mode-dependent non-separable secondary transform (MDNSST), signal-dependent transform, Karhunen-Loeve (KLT) transform, and the like. Video encoder 200 generates transform coefficients after applying one or more transforms.

[0056] As noted above, after transforms to generate transform coefficients, video encoder 200 may perform quantization of the transform coefficients. Quantization generally means a process in which transform coefficients are quantized to possibly reduce the amount of data used to represent the coefficients, providing additional compression. By performing the quantization process, video encoder 200 may reduce the bit depth associated with some or all of the coefficients. For example, video encoder 200 may round down an n-bit value to an m-bit value during quantization where n is greater than m. In some examples, to perform quantization, video encoder 200 may perform a bitwise right shift of the value to be quantized.

[0057] After quantization, video encoder 200 may scan transform coefficients forming a one-dimensional vector from a two-dimensional matrix including the quantized transform coefficients. The scan can be designed to place higher energy (and therefore lower frequency) coefficients at the beginning of the vector and place lower energy (and therefore higher frequency) coefficients at the end of the vectors. In some examples, video encoder 200 may use a predefined scan order to scan the quantized transform coefficients to generate a serialized vector and then entropy encode the quantized vector transform coefficients. In other examples, video encoder 200 may perform adaptive scanning. After scanning the quantized transform coefficients to generate a one-dimensional vector, video encoder 200 may entropy encode the one-dimensional vector, such as according to context adaptive binary arithmetic coding (CABAC). Video encoder 200 may also entropy encode other syntax elements describing metadata associated with encoded video data for use by video decoder 300 when decoding video data.

[0058] To perform CABAC, video encoder 200 may assign a context in a context model to a character to be transmitted. The context may be related, for example, to whether adjacent symbol values are null or not. The probability determination may be based on the context assigned to the symbol.

[0059] The video encoder 200 may further generate syntax data, such as block-based syntax data, picture-based syntax data, and sequence-based syntax data, to video decoder 300, for example, in a picture header, a block header, a slice header, or other syntax data, such as a sequence parameter set (SPS), a picture parameter set (PPS), or a video parameter set (VPS). Video decoder 300 can similarly decode such syntax data to determine how to decode the corresponding video data.

[0060] Thus, video encoder 200 may generate a bit stream including encoded video data, such as syntax elements describing segmentation of a picture into blocks (eg, CUs), and prediction information and/or residual information for blocks. Ultimately, video decoder 300 may receive the bit stream and decode the encoded video data.

[0061] In general, video decoder 300 performs a reciprocal process of the process performed by video encoder 200 to decode encoded video bitstream data. For example, video decoder 300 may decode values for bitstream syntax elements using CABAC in a manner substantially similar to, although reciprocal of, video encoder 200's CABAC coding process. such as a QTBT structure to define the CU CTU. The syntax elements may further define prediction information and residual information for blocks (eg, CUs) of video data.

[0062] The residual information may be represented, for example, by quantized transform coefficients. Video decoder 300 may inversely quantize and inversely transform the quantized transform coefficients of the block to reproduce the residual block for the block. Video decoder 300 uses the signaled prediction mode (intra or inter prediction) and associated prediction information (eg, motion information for inter prediction) to generate a prediction block for the block. Video decoder 300 may then combine the prediction block and the residual block (on a per-sample basis) to reproduce the original block. Video decoder 300 may perform additional processing, such as performing a deblocking process, to reduce visual artifacts along block boundaries.

[0063] This disclosure may generally refer to "signaling" certain information, such as syntax elements. The term "signaling" can generally mean the transmission of syntax element values and/or other data used to decode encoded video data. Thus, video encoder 200 may signal values for syntax elements in the bitstream. In general, signaling means generating a value in a bitstream. As noted above, source device 102 may transport the bitstream to destination device 116 in near real time or non-real time, such as when syntax elements are stored in storage device 112 for later retrieval by destination device 116.

[0064] FIG. 2A and 2B is a conceptual diagram illustrating an exemplary binary quadtree tree (QTBT) structure 130 and a corresponding coding tree unit (CTU) 132 . The solid lines represent the quadtree split, and the dotted lines indicate the binary tree split. At each split (i.e. non-leaf) node of the binary tree, one flag is signaled to indicate what type of split (i.e. horizontal or vertical) is used, where 0 indicates that horizontal split and 1 indicates vertical split in this example. For quadtree splitting, it is not necessary to specify the type of splitting, because the quadtree nodes split the block horizontally and vertically into 4 equally sized subblocks. Accordingly, video encoder 200 may encode, and video decoder 300 may decode, syntax elements (eg, split information) for the tree level of the area of the QTBT structure 130 (i.e., solid lines) and syntax elements (eg, split information) for the tree level. predictions of the QTBT structure 130 (ie, dotted lines). Video encoder 200 may encode and video decoder 300 may decode video data, such as prediction and transform data, for CUs represented by leaf terminal nodes of QTBT structure 130.

[0065] In general, the CTU 132 of FIG. 2B may be associated with parameters specifying block sizes corresponding to the nodes of the QTBT structure 130 at the first and second levels. These parameters may include CTU size (representing the size of the CTU 132 in samples), minimum quadtree size (MinQTSize, representing the minimum allowed size of quadtree leaf nodes), maximum binary tree size (MaxBTSize, representing the maximum allowed size of binary root nodes). tree), the maximum depth of the binary tree (MaxBTDepth, representing the maximum allowed depth of the binary tree), and the minimum size of the binary tree (MinBTSize, representing the minimum allowed size of the leaf nodes of the binary tree).

[0066] The root node of the QTBT structure corresponding to the CTU may have four child nodes at the first level of the QTBT structure, each of which may be segmented according to segmentation into a quadtree. Thus, first-level nodes are either leaf nodes (having child nodes) or have four child nodes. An example QTBT structure 130 represents such nodes as including a parent node and child nodes having solid lines for branches. If the nodes of the first level do not exceed the maximum allowed size of the root nodes of the binary tree (MaxBTSize), they can additionally be segmented by the corresponding binary trees. Splitting into a binary tree of a single node can be iterated until the resulting nodes of the split reach the minimum allowed leaf node size of the binary tree (MinBTSize) or the maximum allowed binary tree depth (MaxBTDepth). Example QTBT structure 130 represents such nodes as having dashed lines for branches. A leaf node of a binary tree is called a "coding unit (CU)" which is used for prediction (eg, intra-picture or inter-picture prediction) and transformation, without further segmentation. As explained above, CUs may also be referred to as "video blocks" or "blocks".

[0067] In one example of a QTBT segmentation structure, the CTU size is set to 128x128 (luminance samples and two corresponding 64x64 chrominance samples), MinQTSize is set to 16x16, MaxBTSize is set to 64x64, MinBTSize( for width and height) is set to 4, and MaxBTDepth is set to 4. Quadtree segmentation is applied to the CTU first to form the leaf nodes of the quadtree. The leaf nodes of the quadtree can be up to 16×16 (i.e. MinQTSize) up to 128×128 (i.e. CTU-size). If the leaf node of the quadtree is 128×128, then it should not be further split by the binary tree because the size exceeds MaxBTSize (ie 64×64, in this example). Otherwise, the leaf node of the quadtree is further segmented by a binary tree. Therefore, the leaf node of the quadtree is also the root node for the binary tree and has a binary tree depth of 0. When the binary tree depth reaches MaxBTDepth (4, in this example), no further splitting is allowed. When a binary tree node has a width equal to MinBTSize (4, in this example), this implies that no additional horizontal splitting is allowed. Similarly, a binary tree node having a height equal to MinBTSize implies that additional vertical splitting is not allowed for that binary tree node. As noted above, the leaf nodes of the binary tree are called "CU" and are further processed according to the prediction and transformation without additional segmentation.

[0068] In this section, video coding standards, in particular those related to motion vector prediction technologies of the previous standards, are explained.

[0069] Video coding standards include ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-T H.262 or ISO/IEC MPEG-2 Visual, ITU-T H.263, ISO/IEC MPEG -4 Visual and ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC), which includes Scalable Video Coding (SVC) and Multi-View Video Coding (MVC) extensions. The latest collaborative MVC project is described in "Advanced video coding for generic audiovisual services", ITU-T Recommendation H.264, March 2010.

[0070] In addition, a new developed video coding standard is provided, namely the High Efficiency Video Coding (HEVC) standard being developed through the Joint Video Standards Collaborative Group (JCT-VC) of the Video Coding Expert Group (VCEG) of ITU-T and Motion Picture Expert Group (MPEG) ISO/IEC. The latest HEVC draft is available at http://phenix.int-evry.fr/jct/doc_end_user/documents/12_Geneva/wg11/JCTVC-L1003-v34.zip.

[0071] Motion information

[0072] For each block, a set of motion information may be available. The motion information set contains motion information for forward and backward prediction directions. Here, the forward and reverse prediction directions are two prediction directions corresponding to reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1) for the current picture or slice. The terms "forward" and "inverse" do not necessarily have the meaning of geometry. Instead, they are used to distinguish which list of reference pictures the motion vector is based on. Forward prediction means a prediction generated based on reference list 0, while backward prediction means a prediction generated based on reference list 1. In case both reference list 0 and reference list 1 are used to generate a prediction for a given block, it is called "bidirectional prediction".

[0073] For a given picture or slice, if only one reference picture list is used, each block within the picture or slice undergoes forward prediction. If both reference picture lists are used for a given picture or slice, a block within a picture or slice may be subjected to forward prediction or inverse prediction, or bidirectional prediction.

[0074] For each prediction direction, the motion information contains a reference index and a motion vector. The reference index is used to identify the reference picture in the corresponding reference picture list (eg, RefPicList0 or RefPicList1). The motion vector has both a horizontal and a vertical component, each indicating an offset value along the horizontal and vertical directions, respectively. In some descriptions, for simplicity, the phrase "motion vector" may be used interchangeably with motion information to indicate both the motion vector and its associated reference index.

[0075] POC

[0076] A Picture Sequence Number (POC) is used in video coding standards to identify the display order of a picture. While there are cases where two pictures in the same coded video sequence may have the same POC value, this typically does not occur in the coded video sequence. When multiple encoded video sequences are present in a bitstream, pictures with the same POC value may be closer to each other in terms of decoding order.

[0077] Picture POC values are typically used for reference picture listing, reference picture set extraction, similar to HEVC, and motion vector scaling.

[0078] Prediction unit

[0079] A prediction unit (PU) means a base unit of samples that share identical prediction information. In the PU for mutual prediction, the prediction information may be the same set of motion information or the same method for extracting motion information for the PU. In one example, the PU may be identical in the HEVC block structure and may also be the basic unit in other block segmentation, quadtree plus binary tree segmentation, multitype tree segmentation, and others.

[0080] A sub-PU is a sub-unit within a PU in which samples share an identical set of motion information. In motion at the sub-PU level, the samples in the PU share the same method/model for retrieving motion information, but the sub-PU may have its own set of motion information and may be different from another sub-PU in the same PU.

[0081] FIG. 3 illustrates an exemplary flowchart of a method for extracting motion vector predictors as explained herein. Two lists of motion vector predictor candidates are derived from neighboring block information in the 3D domain. The decoder adaptively selects the list according to the decoded is_sub_pu_motion_flag syntax element from the bitstream.

[0082] In one example, the sub_pu list may only exist if the current block exceeds a predetermined value. In another example, the block may contain only sub-PUs, such as a 4x4 block, the flag in such a case serves to indicate that the motion information generation is based on the sub-PU method or not. The two lists may include a different number of motion prediction candidates, which may be predefined or signaled in the SPS, PPS, or slice header. Neighbor blocks may be spatial neighbors in the current picture or temporal neighbors in previous coded pictures.

[0083] As illustrated in FIG. 3, the process of extracting the two lists is explained below. At 300, the decoder determines whether the flag is active or not. If the decoder determines that the flag is not active, the decoder proceeds to 302. If the flag is active, the decoder proceeds to 306.

[0084] At 302, the first list of motion prediction candidates at the PU level is retrieved. At 306, a second sub-PU level motion prediction candidate list is retrieved. An option for motion prediction at the PU level means that all pixels in an identical PU share an identical set of motion information. An option for motion prediction at the sub-PU level means that all pixels in an identical sub-PU share the same set of motion information, but another sub-PU in a PU may have different sets of motion information. The set of motion information may include an inter prediction direction, an index or indices of reference pictures when using multiple links, a motion vector or motion vectors when using multiple links.

[0085] An example of motion vector candidates at the PU level, the list is a list of HEVC merge candidates. Examples of motion prediction at the sub-PU level include, but are not limited to, affine motion vector prediction (affine), alternative temporal motion vector prediction (ATMVP), space-time motion vector prediction (STMVP), planar motion vector prediction, and vector extraction. motion based pattern matching (PMVD), and the like. An exemplary syntax table is shown in Table 1 below. The sub_pu_motion_idx syntax element may be used to indicate the selected candidate in the sub-PU level motion prediction candidate list, and the pu_motion_idx syntax element is used to indicate the selected candidate in the prediction candidate list. movements at the PU level.

Tab. one

B

S

C

D

E". A, B, C, D, E обозначают кандидаты предсказания движения на уровне PU, и S представляет собой индикатор sub_pu_motion_flag. Если S выбирается, то sub_pu_motion_flag логически выводится как истинный. Один пример A может представлять собой левый нижний возможный вариант пространственного слияния, и один пример B может представлять собой правый верхний возможный вариант пространственного слияния. Следует отметить, что также может применяться другой порядок вставки.[0086] In another method, motion vector candidates at the PU level may be divided into two groups; is_sub_pu_motion_flag is signaled when the decoder receives a syntax indicating that the selected opportunity is not in the first PU layer motion information candidate group. Is_sub_pu_motion_flag may be implicitly signaled in the PU layer merge index as one of the indexes. Then, the sub-PU level merge index shall additionally be signaled if is_sub_pu_motion_flag is true. In one example, the insertion order in the motion prediction candidate lists at the PU level is "A

B

S

C

D

E". A, B, C, D, E denote motion prediction candidates at the PU level, and S is the sub_pu_motion_flag indicator. If S is selected, then sub_pu_motion_flag is inferred to be true. One example of A may represent the lower left spatial fusion candidate , and one example B may be the upper right spatial fusion candidate.It should be noted that a different insertion order may also apply.

[0087] It should be noted that motion prediction candidates at the sub-PU level are grouped in all of the methods described above.

[0088] Signaling for Motion Vector Prediction Candidate Selection

[0089] The syntax element is_sub_pu_motion_flag is first used to indicate whether or not a motion prediction candidate is a sub-PU layer. Is_sub_pu_motion_flag may be binarized using one bin (0/1) and encoded with a context binary arithmetic encoder. The context may depend on the PU-size/area or PU-depth in the block segmentation tree. A larger PU may tend to be more frequently used in sub-PU level motion vector prediction selection than a smaller PU. The context may also depend on the sub_pu_motion_flag of the spatial/temporal neighboring blocks. The probability that the current PU uses sub-PU based motion is higher if neighboring blocks have sub-PU based motion.

[0090] If is_sub_pu_motion_flag is true ("1"), the sub_pu_motion_idx syntax element is used to indicate a method to extract a motion prediction candidate based on the sub-PU. The total number of ways, i.e. the total number of motion prediction candidates at the sub-PU level, num_sub_pu_motion may be signaled in the high-level syntax. Sub_pu_motion_idx can be binarized using a truncated unary code depending on num_sub_pu_motion. However, another binarization method may also be used.

[0091] If is_sub_pu_motion_flag is false ("0"), the nor_pu_motion_idx syntax element is used to indicate a method to extract a motion prediction candidate at the PU level. The total number of motion vector prediction candidates at the PU level, num_nor_pu_motion, may be signaled in the high-level syntax. Nor_pu_motion_idx can be converted to binary using a truncated unary code depending on num_nor_pu_motion. However, another binarization method may also be used.

[0092] Extraction of motion prediction candidates at the PU level

[0093] A motion prediction candidate at the PU level may be derived from spatial or temporal neighboring coded blocks, similar to HEVC. In HEVC fusion mode,

[0094] In HEVC, the MV candidate list contains up to 5 candidates for the merge mode and only two candidates for the AMVP mode. An exemplary fusion may comprise a set of motion information, such as motion vectors, corresponding to both reference picture lists (list 0 and list 1) and reference indices. If a merge candidate is identified by the merge index, reference pictures are used to predict the current blocks and associated motion vectors are determined.

[0095] Based on the above, one or more motion vector predictors are selected based on the decoded candidate index.

[0096] In HEVC merge mode, up to four spatial MV candidates 402, 404, 406, and 408 of block 400 can be extracted with the orders shown in FIG. 4. The order is left (0, A1), top (1, B1), top right (2, B0), bottom left (3, A0), and top left (4, B2).

[0097] Temporary neighbor candidates for HEVC are explained below. A temporal motion vector predictor (TMVP) candidate, if enabled and available, is added to the list of MV candidates after the spatial motion vector candidates. The primary block location for extracting TMVP candidates is the bottom right block outside the co-located PU, as shown in FIG. 5 as block "T" 500 to compensate for the offset in the top and left blocks used to generate spatial neighbor candidates. However, if this block 502 is located outside the current CTB row, or motion information is not available, the block is replaced by the central PU block 504. The motion vector for the TMVP candidate is derived from the co-located PU of the co-located picture , indicated at the slice level. The motion vector for a co-located PU is referred to as a "co-located MV".

[0098] Extraction of motion prediction candidates at the sub-PU level

[0099] Candidates for motion prediction at the sub-PU level may include, but are not limited to, affine motion prediction, alternative temporal motion vector prediction (ATMVP), space-time motion vector prediction (STMVP), planar motion vector prediction, vector extraction motion based pattern matching (PMVD), and the like. Next, examples of this motion prediction at the sub-PU level are illustrated. However, some variation or other motion prediction at the sub-PU level may also be added.

[0100] Affine Motion Prediction

[0101] In the 4-parameter affine motion prediction method, the block motion vector field is described by equation (1):

, (1)

, (one)

является вектором движения левой верхней угловой управляющей точки, и

является вектором движения правой верхней угловой управляющей точки.[0102] where (w, h) is the block size and (x, y) is the coordinate;

is the motion vector of the top left corner control point, and

is the motion vector of the upper right corner control point.

[0103] In the 6-parameter affine motion prediction method, the block motion field is described by equation (2):

, (2)

, (2)

является вектором движения левой нижней угловой управляющей точки.[0104] where, in addition,

is the motion vector of the lower left corner control point.

[0105] In sub-PU-based affine motion prediction, the MV of the sub-PU may be extracted by calculating the MV at the center of the sub-PU. Alternatively, (w, h) and (x, y) can be downscaled according to the sub-PU size.

может извлекаться из соседнего блока, который ранее кодируется посредством аффинного движения, с учетом того, что текущий блок совместно использует идентичную модель аффинного движения с предыдущим кодированным соседним блоком.[0106] In one way,

may be derived from a neighboring block that was previously encoded with affine motion, given that the current block shares the same affine motion model with the previous encoded neighboring block.

может извлекаться посредством вектора движения в соседнем кодированном блоке. Например, как показано на фиг. 6,

может извлекаться из вектора движения в блоке A 600, B 602 или C 604,

может извлекаться из вектора движения в блоке C 606 или D 608,

может извлекаться из блока E 610 или F 612.[0107] In another way,

may be extracted by a motion vector in an adjacent coded block. For example, as shown in FIG. 6,

may be extracted from the motion vector in block A 600, B 602, or C 604,

may be extracted from the motion vector in block C 606 or D 608,

can be removed from block E 610 or F 612.

[0108] Alternative temporal motion vector prediction

[0109] In an alternative temporal motion vector prediction (ATMVP) method (or sometimes referred to as "advanced temporal motion vector prediction"), temporal motion vector prediction (TMVP) is modified by fetching multiple sets of motion information (including motion vectors and reference indices) from a sub-PU of the current PU.

[0110] Spatio-temporal prediction of motion vectors

[0111] In the spatiotemporal motion vector prediction method, sub-PU motion vectors are extracted recursively according to the raster scan order. Fig. 7 illustrates the concept of STMVP. Consider an 8x8 PU that contains four 4x4 sub-PUs A, B, C, and D. Neighboring 4×4 blocks in the current frame are labeled a, b, c, and d. Motion extraction for a sub-CU begins by identifying its two spatial neighbors. The first neighbor is an NxN block above the sub-CU (block c). If this block c is not available or intra-coded, other NxN blocks above sub-CU A are checked (from left to right, starting from block c). The second neighbor is the block to the left of the sub-CU (block b). If block b is not available or intra-coded, other blocks to the left of sub-CU A are checked (top to bottom, starting from block b). Motion information obtained from adjacent blocks for each list is scaled to the first reference frame for that list. Next, the temporal motion vector predictor (TMVP) of sub-block A is extracted by performing the identical TMVP extraction procedure as specified in HEVC. The motion information of the co-located block at location D is sampled and scaled, respectively. Finally, after extracting and scaling the motion information, all available motion vectors (up to 3) are averaged separately for each reference list. The average motion vector is assigned as the motion vector of the current sub-CU.

[0112] Extraction of motion vectors based on pattern matching

[0113] A template-based motion vector extraction (PMMVD) method is based on picture upscaling techniques. The motion extraction process has two steps. Motion search at the PU level is first performed, followed by motion detailing at the sub-PU level. At the PU level, the initial motion vector is derived for the entire PU based on bilateral matching or pattern matching. First, a list of MV candidates is generated, and the candidate that results in the least matching overhead is selected as the starting point for further granularity at the PU level. A local search is then performed based on bilateral matching or pattern matching around the starting point, and the MV resulting in the lowest matching overhead is considered as the MV for the entire CU. Thereafter, the motion information is further refined at the sub-PU level with the extracted CU motion vectors as starting points.

[0114] The concept of bilateral matching is illustrated in FIG. 8. Bilateral matching is used to extract motion information by finding the closest match between two blocks along the motion path of the current block in two different reference pictures. Assuming a trajectory of continuous motion, the motion vectors MV0 and MV1 pointing to the two reference blocks must be proportional to the time distances, i.e. TD0 and TD1, between the current picture and two reference pictures. As a special case, when the current picture is temporarily between two reference pictures, and the temporal distance from the current picture to the two reference pictures is identical, the bilateral matching becomes a mirror-based bi-directional MV.

[0115] Matching with a template, as illustrated in FIG. 9 is used to extract motion information by finding the closest match between a template (top and/or left adjacent blocks of the current block) in the current picture and a block (of the same size as the template) in the reference picture.

[0116] Planar Motion Vector Prediction

[0117] Planar motion vector prediction is achieved by averaging horizontal and vertical linear interpolation based on 4×4 blocks as follows.

[0118] W and H denote the width and height of the block; (x, y) are the coordinates of the current sub-block relative to the above left corner sub-block. All distances are denoted by pixel distances divided by 4. P(x, y) is the motion vector of the current subblock.

[0119] Horizontal prediction P _h (x, y) and vertical prediction P _v (x, y) for location (x, y) are calculated as follows:

[0120] where L(-1, y) and R(W, y) are motion vectors of 4×4 blocks to the left and right of the current block; A(x, -1) and B(x, H) are the motion vectors of the 4×4 blocks above and below the current block, as shown in FIG. 9.

[0121] The movement reference information of neighboring blocks from the left column and top row is extracted from the spatial neighboring blocks of the current block.

[0122] The movement reference information of adjacent blocks from the right column and bottom row is extracted as follows.

1) Extraction of motion information of the right lower temporal neighboring block 4×4.

2) Calculation of the motion vectors of the neighboring 4×4 blocks in the right column, using the extracted motion information of the right lower neighboring block 4×4 along with the motion information of the above right neighboring block 4×4, as R(W, y)=((H-y -1)xAR+(y+1)xBR)/H.

3) Computing the motion vectors of the 4×4 neighboring blocks in the bottom row, using the extracted motion information of the right lower neighboring 4×4 block along with the motion information of the left lower neighboring 4×4 block, as B(x, H)=((W-x -1)xBL+(x+1)xBR)/W.

[0123] - where AR is the motion vector of the right upper 4x4 spatial neighbor block, BR is the motion vector of the right lower 4x4 temporal neighbor block, and BL is the motion vector of the left lower 4x4 spatial neighbor block, as shown in FIG. ten.

[0124] Motion information obtained from adjacent blocks for each list is scaled to the first reference picture for that list.

[0125] Insertion in Candidate Lists

ATMVP

STMVP

PMMVD

планарный". Порядок по умолчанию может быть предварительно задан или передаваться в служебных сигналах на основе типа серии последовательных макроблоков, временного слоя, модели аффинного движения и/или доступности временного предиктора движения. Порядок по умолчанию также может отличаться на основе типа блока, формы блока или размера блока. Максимальное число кандидатов предсказания движения на уровне суб-PU определяется посредством предварительно заданного числа и/или общего числа доступных кандидатов предсказания движения на уровне суб-PU, как указано посредством высокоуровневого синтаксиса. В одном примере, если предварительно заданное число равно 3, и все из аффинного, ATMVP-, STMVP-, PMMVD- и планарного кандидатов доступны, то максимальное число равно 3. Но если только аффинный и ATMVP доступны, то максимальное число равно 2. Если максимальное число равно 0, то is_sub_pu_motion_flag логически выводится как ложный (0) и не передается в служебных сигналах в потоке битов.[0126] If present, sub-PU level motion prediction candidates are inserted into the list in the following default order: "affine

ATMVP

STMVP

PMMVD

planar". The default order may be predefined or signaled based on the slice type, time slice, affine motion model, and/or availability of the temporal motion predictor. The default order may also differ based on the block type, block shape, or The maximum number of sub-PU level motion prediction candidates is determined by a predefined number and/or the total number of available sub-PU level motion prediction candidates as specified by high-level syntax.In one example, if the predefined number is 3, and all of the affine, ATMVP-, STMVP-, PMMVD- and planar candidates are available, then the maximum number is 3. But if only affine and ATMVP are available, then the maximum number is 2. If the maximum number is 0, then is_sub_pu_motion_flag is logically inferred to be false (0) and is not signaled in the bitstream.

[0127] A different default order of candidates may also be used.

PMMVD

ATMVP

STMVP

планарный".In one example, "affine

PMMVD

ATMVP

STMVP

planar".

аффинный

ATMVP

STMVP

планарный".In another example, "PMMVD

affine

ATMVP

STMVP

planar".

In another example, two or more affine possibilities may be used:

аффинный 2

PMMVD

ATMVP

STMVP

планарный";"affine 1

affine 2

PMMVD

ATMVP

STMVP

planar";

Affine1

ATMVP

аффинный 2

планарный"or, "affine 1

Affine1

ATMVP

affine 2

planar"

[0128] Prioritized Candidate Reordering

[0129] The list of motion prediction candidates at the default sub-PU level may be reordered based on their occurrences in neighboring coded blocks. In one example, a candidate that has more occurrences in adjacent code blocks is placed at a lower index position in the list.

[0130] Partial prioritized candidate reordering

[0131] To reduce the complexity of candidate reordering, prioritized candidate reordering is applied to only one or more sublists. For example, the 1-2, 3-4 option in the default order is reordered separately based on its occurrence in adjacent code blocks.

[0132] Clipping

[0133] In order to reduce complexity, pruning or partial pruning cannot be applied in the sub-PU candidate list. In one example, pruning between ATMVP, STMVP, and planar may be applied, but pruning is not applied between affine and the rest of the sub-PU options.

[0134] In another example, the clipping is only applied to the number of sub-PUs less than or equal to a predetermined value.

[0135] Alternative methods for signaling a motion prediction candidate based on sub-PU

PMMVD

ATMVP

STMVP

планарный", то передача служебных сигналов является такой, как показано в таблице 2.[0136] The selection of a motion prediction candidate based on the sub-PU may be signaled by activating flags according to the insertion order of the candidates instead of the index. In one example, if the insertion order is "affine

PMMVD

ATMVP

STMVP

planar", then signaling is as shown in Table 2.

[0137] In another alternative method, sub-PU based motion prediction candidates may be grouped into multiple subgroups. For example, affine, planar are grouped into subgroup 1, ATMVP and STMVP are grouped into subgroup 2, and PMMVD is yet another subgroup 3. An exemplary signaling is shown in Table 3. The pmmvd_flag syntax is first signaled in signaling to indicate which represents is it a possible PMMVD variant or not. Then, if it is not a PMMVD candidate, another sub_group1_flag syntax element is signaled to indicate whether it is a subgroup 1 candidate or not. If sub_group1_flag is true, indicating that it is a subgroup 1 candidate, affine_flag is signaled to indicate whether or not it is an affine candidate. If it does not represent an affine candidate, then planar_mv_flag is set to true to indicate that it represents a planar candidate. If sub_group1_flag is false, then atmvp_flag is signaled to indicate whether it is an ATMVP candidate or not. If it does not represent an ATMVP candidate, stmvp_flag is set to true to indicate that it is an STMVP candidate.

Tab. 2

Tab. 3

[0138] It should be noted that in Table 2 and Table 3, the availability check is ignored for the sake of example. The flag is logically output as false (0) if the corresponding option is not available. It should also be noted that other grouping variations may also apply.

[0139] FIG. 11 is a block diagram illustrating an exemplary video encoder 200 that may perform the techniques of this disclosure. Fig. 11 is provided for purposes of explanation and should not be considered a limitation of the technologies as illustrated and outlined in this disclosure. For purposes of explanation, this disclosure describes video encoder 200 in the context of video coding standards such as the HEVC video coding standard and the upcoming H.266 video coding standard. However, the techniques of this disclosure are not limited to these video coding standards and are generally applicable to video encoding and decoding.

[0140] In the example of FIG. 11, the video encoder 200 includes a video data memory 230, a mode selection unit 202, a residual unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 202, an inverse transform processing unit 212, a reconstruction unit 214, a filter unit 216, buffer 218 decoded pictures (DPB) and module 220 entropy encoding.

[0141] Video data storage 230 may store video data to be encoded by components of video encoder 200. Video encoder 200 may receive video data stored in video data storage 230, such as from video source 104 (FIG. 1). DPB 218 may act as a reference picture store that stores reference video data for use in predicting subsequent video data by video encoder 200. Video store 230 and DPB 218 may be formed by any of a variety of memories, such as dynamic random access memory (DRAM) DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of storage devices. The video storage device 230 and the DPB 218 may be provided via an identical storage device or separate storage devices. In various examples, video data storage 230 may be on-chip with other components of video encoder 200, as illustrated, or off-chip with respect to those components.

[0142] In this disclosure, reference to video storage 230 is not to be interpreted as being limited to storage internal to video encoder 200, unless specifically described as such, or storage external to video encoder 200, unless specifically described as such. quality. Conversely, a reference to video data storage 230 is to be understood as reference storage that stores video data that video encoder 200 receives for encoding (eg, video data for the current block to be encoded). The storage device 106 of FIG. 1 may also provide temporary storage of outputs from various modules of video encoder 200.

[0143] The various modules of FIG. 11 are illustrated to assist in understanding the operations performed by video encoder 200. The modules may be implemented as fixed function circuits, programmable circuits, or a combination of the above. Fixed function circuits means circuits that provide specific functionality and are preset in terms of operations that can be performed. Programmable circuits means circuits that can be programmed to perform various tasks and provide flexible functionality in operations that can be performed. For example, the programmable circuits may execute software or firmware that causes the programmable circuits to operate in a manner specified by software or firmware instructions. Fixed-function circuits can execute program instructions (eg, to receive parameters or output parameters), but the types of operations that fixed-function circuits perform are generally immutable. In some examples, one or more modules may be various circuit blocks (fixed function or programmable), and in some examples, one or more modules may be integrated circuits.

[0144] Video encoder 200 may include arithmetic logic units (ALUs), elementary functional units (EFUs), digital circuits, analog circuits, and/or programmable cores formed from programmable circuits. In examples where the operations of video encoder 200 are performed using software executable by programmable circuitry, memory 106 (FIG. 1) may store the object code of the software that video encoder 200 receives and executes or other memory in video encoder 200 (not shown) can store such instructions.

[0145] The video storage device 230 is configured to store received video data. The video encoder 200 may retrieve a picture of the video data from the video data storage device 230 and provide the video data to the residual generation module 204 and the mode selection module 202 . The video data in the video data storage 230 may be raw video data to be encoded.

[0146] The mode selection module 202 includes a motion estimation module 222, a motion compensation module 224, and an intra prediction module 226. The mode selection module 202 may include additional functional modules to perform video prediction in accordance with other prediction modes. By way of example, mode selection module 202 may include a palette processing module, an intra-block copy module (which may be part of motion estimation module 222 and/or motion compensation module 224), affine processing module, linear model (LM ) etc.

[0147] Mode selection module 202 generally coordinates multiple encoding passes to test combinations of encoding parameters and resulting distortion values versus rate for such combinations. The coding parameters may include segmentation of the CTU per CU, prediction modes for the CU, transform types for the CU residual data, quantization parameters for the CU residual data, and so on. The mode selection module 202 may ultimately select a combination of coding parameters having rate-distortion values that are better than the other tested combinations.

[0148] Video encoder 200 may segment the picture retrieved from video storage 230 into a sequence of CTUs and encapsulate one or more CTUs in a slice. The mode selector 210 may segment the picture CTUs according to a tree structure such as the QTBT structure or the HEVC quadtree structure described above. As described above, video encoder 200 may generate one or more CUs from CTU segmentation according to a tree structure. Such a CU may also be referred to generically as a "video block" or "block".

[0149] In general, mode selection module 202 also controls its components (e.g., motion estimation module 222, motion compensation module 224, and intra prediction module 226) to generate a prediction block for the current block (e.g., the current CU or, in HEVC, overlapping part of PU and TU). For inter-prediction of the current block, motion estimation module 222 may perform a motion search to identify one or more closely matching reference blocks in one or more reference pictures (eg, one or more previously encoded pictures stored in DPB 218). Specifically, motion estimation module 222 may calculate a value representing how similar a potential reference block is to the current block, for example, according to Sum Absolute Difference (SAD), Sum Squared Difference (SSD), Mean Absolute Difference (MAD), RMS Differences (MSD), etc. Motion estimation module 222 can generally perform these calculations using the sample differences between the current block and the reference block under consideration. Motion estimation module 222 may identify the reference block having the smallest value resulting from these calculations, indicating the reference block that most closely matches the current block.

[0150] The motion estimation module 222 may generate one or more motion vectors (MVs) that define the positions of the reference blocks in the reference pictures x relative to the position of the current block in the current picture. Motion estimation module 222 may then provide motion vectors to motion compensation module 224 . For example, for unidirectional inter prediction, motion estimator 222 may provide one motion vector, while for bidirectional inter prediction, motion estimator 222 may provide two motion vectors. Motion compensation module 224 may then generate a prediction block using the motion vectors. For example, motion compensation module 224 may extract reference block data using a motion vector. As another example, if the motion vector has fractional precision, motion compensation module 224 may interpolate values for the prediction block according to one or more interpolation filters. In addition, for bidirectional inter-prediction, motion compensation module 224 may extract data for two reference blocks identified by respective motion vectors and combine the extracted data, for example, by successive sample averaging or weighted averaging.

[0151] As another example, for intra prediction or intra predictive coding, intra prediction module 226 may generate a prediction block from samples adjacent to the current block. For example, for directional modes, intra prediction module 226 can generally mathematically combine adjacent sample values and fill in those computed values in a given direction for the current block to form a prediction block. As another example, for DC mode, intra prediction module 226 may calculate an average of adjacent samples with respect to the current block and generate a prediction block that includes this resulting average for each prediction block sample.

[0152] The mode selection module 202 provides a prediction block to the residual generation module 204 . Residue generation module 204 receives a raw, uncoded version of the current block from video data storage 230 and a prediction block from mode selection module 202 . Residualizer 204 calculates successive sample differences between the current block and the prediction block. The resulting successive sample differences define the residual block for the current block. In some examples, residual block 204 may also determine differences between sample values in the residual block to generate a residual block using residual differential pulse code modulation (RDPCM). In some examples, residual module 204 may be generated using one or more subtractor circuits that perform binary subtraction.

[0153] In examples in which mode selector 202 segments CUs into PUs, each PU may be associated with a luminance prediction unit and corresponding chrominance prediction units. Video encoder 200 and video decoder 300 may support PUs having different sizes. As mentioned above, the size of the CU may mean the size of the luminance coding unit of the CU, and the size of the PU may mean the size of the luminance prediction unit of the PU. Assuming that the size of a particular CU is 2Nx2N, video encoder 200 may support PU sizes of 2Nx2N or NxN for intra prediction and symmetric PU sizes of 2Nx2N, 2NxN, Nx2N, NxN or similar for inter prediction. Video encoder 200 and video decoder 300 may also support asymmetric segmentation for PU sizes of 2NxnU, 2NxnD, nLx2N, and nRx2N for inter prediction.

[0154] In examples where mode selector 202 does not further segment CUs into PUs, each CU may be associated with a luminance coding block and corresponding chrominance coding blocks. As described above, the size of the CU may mean the size of the luminance coding block of the CU. Video encoder 200 and video decoder 300 may support CU sizes of 2Nx2N, 2NxN, or Nx2N.

[0155] For other video coding technologies such as intra-block copy mode coding, affine mode coding, and linear model (LM) mode coding, as a few examples, the mode selection module 202, through the respective modules associated with the technologies coding, generates a prediction block for the current block being encoded. In some examples, such as palette mode coding, mode selector 202 may not generate a prediction block and instead generate syntax elements that indicate the manner in which the block should be reconstructed based on the selected palette. In such modes, mode selector 202 may provide these syntax elements to entropy encoding module 220 for encoding.

[0156] As described above, the residual generation unit 204 receives video data for the current block and the corresponding prediction block. The residual block 204 then generates a residual block for the current block. To generate a residual block, the residual block 204 calculates successive sample differences between the prediction block and the current block. In this way:

[0157] The transform processing module 206 applies one or more transforms to the residual block to generate a transform coefficient block (referred to herein as a "transform coefficient block"). Transform processing module 206 may apply various transforms to the residual block to generate a block of transform coefficients. For example, transform processing module 206 may apply a discrete cosine transform (DCT), a directional transform, a Karhunen-Loeve (KLT) transform, or a conceptually similar transform to the residual block. In some examples, the transform processing module 206 may perform multiple transforms on the residual block, such as a primary transform and a secondary transform such as a rotational transform. In some examples, transform processing module 206 does not apply transforms to the residual block.

[0158] Quantization module 208 may quantize the transform coefficients in the transform coefficient block to generate a block of quantized transform coefficients. Quantization module 208 may quantize the transform coefficients of a block of transform coefficients according to a quantization parameter (QP) value associated with the current block. Video encoder 200 (eg, via mode selector 202) may adjust the amount of quantization applied to coefficient blocks associated with the current block by adjusting the QP value associated with the CU. Quantization may introduce loss of information, and because of this, the quantized transform coefficients may have less accuracy than the original transform coefficients generated by the transform processing unit 206 .

[0159] The inverse quantization unit 210 and the inverse transform processing unit 212 may apply inverse quantization and inverse transforms to the block of quantized transform coefficients, respectively, in order to recover the residual block from the block of transform coefficients. Recovery module 214 may generate a reconstructed block corresponding to the current block (albeit potentially with a certain degree of distortion) based on the reconstructed residual block and the prediction block generated by mode selection module 202 . For example, reconstruction module 214 may sum the samples of the reconstructed residual block with corresponding samples from the prediction block generated by mode selection module 202 to form a reconstructed block.

[0160] The filtering module 216 may perform one or more filtering operations on the recovered blocks. For example, filter module 216 may perform deblocking operations to reduce deblocking artifacts along the edges of a CU. The operations of filter module 216 may be omitted in some examples.

[0161] The video encoder 200 stores the reconstructed blocks in the DPB 218. For example, in examples in which the operations of the filter module 216 are not required, the recovery module 216 may store the reconstructed blocks in the DPB 218. In the examples in which the operations of the filter module 224 are required, the module 216 filtering may store the filtered reconstructed blocks in DPB 218. Motion estimation module 222 and motion compensation module 224 may extract a reference picture from DPB 218 formed from reconstructed (and potentially filtered) blocks to mutually predict blocks of subsequent encoded pictures. In addition, intra prediction module 226 may use the recovered blocks in the current picture DPB 218 to intra predict other blocks in the current picture.

[0162] In general, entropy encoding unit 220 may entropy encode syntax elements received from other functional components of video encoder 200. For example, entropy encoding unit 220 may entropy encode blocks of quantized transform coefficients from quantization unit 208. As another example, entropy encoding unit 220 may entropy encode prediction syntax elements (eg, motion information for inter prediction or intra mode information for intra prediction) from mode selection unit 202 . Entropy encoding module 220 may perform one or more entropy encoding operations on syntax elements that are another example of video data to generate entropy encoded data. For example, entropy encoding unit 220 may perform a context adaptive variable length coding (CAVLC) operation, a CABAC operation, a variable length variable (V2V) coding operation, a syntactic context adaptive binary arithmetic coding (SBAC) operation, an entropy coding operation based on segmentation into probability intervals (PIPE), an exponential Golomb encoding operation, or another type of entropy encoding operation for the data. In some examples, entropy encoding module 220 may operate in a bypass mode in which syntax elements are not entropy encoded.

[0163] Video encoder 200 may output a bitstream that includes the entropy-coded syntax elements required in order to reconstruct the slices or pictures. In particular, entropy encoding unit 220 may output a bitstream.

[0164] The operations described above are described with respect to a block. Such description is to be understood as operations for the luminance coding block and/or chrominance coding blocks. As described above, in some examples, the luminance coding block and the chrominance coding blocks are luminance signal and chrominance signal components of the CU. In some examples, the luminance coder and the chrominance coders are luma and chroma components of the PU.

[0165] In some examples, operations performed on a luminance coding block need not be repeated for chrominance coding blocks. As one example, operations to identify a motion vector (MV) and a reference picture for a luminance coding block need not be repeated to identify a MV and a reference picture for chrominance blocks. Conversely, the MV for the luminance coding block may be scaled to determine the MV for the chrominance blocks, and the reference picture may be the same. As another example, the intra prediction process may be the same for luminance coding blocks and chrominance coding blocks.

[0166] The video encoder 200 is an example of an apparatus configured to encode video data, including a memory configured to store video data, and one or more processing modules implemented in the circuit and configured to extract a picture upscaling conversion (FRUC) reference. ) using the prediction pixel value, and perform motion vector extraction technology on the decoder side using the extracted reference. In another example, video decoder 300 may be configured to determine respective motion vectors from one or more neighboring video blocks and extract a motion vector for the current video block using the corresponding motion vectors from one or more neighboring video blocks. In another example, video decoder 300 may be configured to determine appropriate motion vectors from one or more collocation video blocks and extract a motion vector for the current video block using the corresponding motion vectors from one or more collateral video blocks.

[0167] FIG. 12 is a block diagram illustrating an exemplary video decoder 300 that may perform the techniques of this disclosure. Fig. 12 is provided for purposes of explanation and is not a limitation of the technologies as illustrated and outlined in this disclosure. For purposes of explanation, this disclosure describes a video decoder 300 described according to JEM, H.266/VVC, and HEVC technologies. However, the techniques of this disclosure may be performed by video coding devices that are configured for other video coding standards.

[0168] In the example of FIG. 12, the video decoder 300 includes a coded picture buffer (CPB) 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, a filtering unit 312, and a decoded picture buffer 314. (DPB). The prediction processing module 304 includes a motion compensation module 316 and an intra prediction module 318 . The prediction processing unit 304 may include summing units to perform prediction in accordance with other prediction modes. As an example, prediction processing module 304 may include a palette processing module, an intra-block copy module (which may be part of motion compensation module 316), an affine processing module, a linear model (LM) processing module, and the like. In other examples, video decoder 300 may include more, fewer, or other functional components.

[0169] The CPB storage device 320 may store video data, such as an encoded video bitstream, to be decoded by the components of the video decoder 300. The video data stored in the CPB storage device 320 may be obtained, for example, from a computer-readable medium 110 (FIG. 1 ). CPB storage 320 may include a CPB that stores encoded video data (eg, syntax elements) from an encoded video bitstream. In addition, CPB memory 320 may store video data other than encoded picture syntax elements, such as temporal data representing outputs from various modules of video decoder 300. DPB 314 generally stores decoded pictures that video decoder 300 may output and/or be used as reference video data when decoding subsequent data or pictures of the encoded video bitstream. The CPB memory 320 and DPB 314 may be formed by any of a variety of memories, such as dynamic random access memory (DRAM) including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of storage devices. The CPB storage device 320 and DPB 314 may be provided via an identical storage device or separate storage devices. In various examples, CPB memory 320 may be on-chip with other components of video decoder 300 or off-chip with respect to these components.

[0170] Additionally or alternatively, in some examples, video decoder 300 may retrieve encoded video data from memory 120 (FIG. 1). Thus, memory 120 may store data as discussed above for CPB memory 320. Similarly, memory 120 may store instructions to be executed by video decoder 300 when some or all of the functionality of video decoder 300 is implemented in software for execution by video decoder processing circuitry 300.

[0171] The various modules shown in FIG. 12 are illustrated to assist in understanding the operations performed by video decoder 300. The modules may be implemented as fixed function circuits, programmable circuits, or a combination of the above. Similarly to FIG. 11, fixed function circuits means circuits that provide a specific functionality and are predetermined with respect to operations that can be performed. Programmable circuits means circuits that can be programmed to perform various tasks and provide flexible functionality in operations that can be performed. For example, the programmable circuits may execute software or firmware that causes the programmable circuits to operate in a manner specified by software or firmware instructions. Fixed-function circuits can execute program instructions (eg, to receive parameters or output parameters), but the types of operations that fixed-function circuits perform are generally immutable. In some examples, one or more modules may be various circuit blocks (fixed function or programmable), and in some examples, one or more modules may be integrated circuits.

[0172] Video decoder 300 may include ALUs, EFUs, digital circuits, analog circuits, and/or programmable cores formed from programmable circuits. In instances where operations of video decoder 300 are performed by software executing on programmable circuits, on-chip or off-chip memory may store instructions (e.g., object code) of the software that video decoder 300 receives and executes.

[0173] The entropy decoding module 302 may receive the encoded video data from the CPB and entropy decode the video data to reproduce the syntax elements. Prediction processing module 304, inverse quantization module 306, inverse transform processing module 308, reconstruction module 310, and filter module 312 may generate decoded video data based on syntax elements extracted from the bitstream.

[0174] In general, video decoder 300 reconstructs a picture on a block-by-block basis. Video decoder 300 may perform the recovery operation for each block separately (and the block currently being recovered, ie, being decoded, may be referred to as the "current block").

[0175] Entropy decoding module 302 can entropy decode syntax elements specifying quantized transform coefficients of a block of quantized transform coefficients, as well as transform information such as a quantization parameter (QP), and/or transform mode indicator(s). Inverse quantization module 306 may use the QP associated with the block of quantized transform coefficients to determine the degree of quantization, and likewise, the degree of inverse quantization for inverse quantization module 306 to be applied. Inverse quantizer 306, for example, may perform a bitwise left shift operation to inversely quantize the quantized transform coefficients. The inverse quantizer 306 can therefore generate a transform coefficient block including the transform coefficients.

[0176] After inverse quantization module 306 generates a transform coefficient block, inverse transform processing module 308 may apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, inverse transform processing module 308 may apply an inverse DCT, inverse integer transform, inverse Karhunen-Loeve (KLT), inverse rotational, inverse directional transform, or other inverse transform to the coefficient block.

[0177] In addition, the prediction processing unit 304 generates a prediction block according to syntax elements with prediction information that are entropy decoded by the entropy decoding unit 302 . For example, if the prediction information syntax elements indicate that the current block is mutually predicted, motion compensation module 316 may generate a prediction block. In this case, the syntax elements with prediction information may indicate a reference picture in DPB 314 from which the reference block can be retrieved, as well as a motion vector identifying the location of the reference block in the reference picture relative to the location of the current block in the current picture. The motion compensation module 316 can generally perform the inter prediction process in a manner that is substantially similar to that described with respect to the motion compensation module 224 (FIG. 11).

[0178] As another example, if the prediction information syntax elements indicate that the current block is intra-predicted, the intra prediction module 318 may generate a prediction block according to the intra prediction mode indicated by the prediction information syntax elements. On the other hand, the intra prediction module 318 can generally perform the intra prediction process in a manner that is substantially similar to that described with respect to the intra prediction module 226 (FIG. 11). Intra prediction module 318 may extract adjacent sample data with respect to the current block from DPB 314.

[0179] Recovery module 310 may recover the current block using the prediction block and the residual block. For example, recovery module 310 may sum the residual block samples with the corresponding prediction block samples to reconstruct the current block.

[0180] The filtering module 312 may perform one or more filtering operations on the recovered blocks. For example, filter module 312 may perform deblocking operations to reduce deblocking artifacts along the edges of the reconstructed blocks. The operations of filter module 312 are not necessarily performed in all examples.

[0181] Video decoder 300 may store the reconstructed blocks in DPB 314. As explained above, DPB 314 may provide reference information, such as current picture samples for intra prediction and previously decoded pictures for subsequent motion compensation, to prediction processing unit 304. In addition, video decoder 300 may output decoded pictures from the DPB for subsequent presentation on a display device, such as display device 118 of FIG. one.

[0182] Thus, video decoder 300 is an example of a video decoding apparatus including a memory configured to store video data and one or more processing modules implemented in circuitry and configured to implement the methods and processes explained herein.

[0183] For example, the encoded video data may include the following syntax:

Table 4. Syntax of merge data

[0184] Merge_subblock_flag[x0][y0] indicates whether or not the subblock-based inter prediction parameters for the current coding unit from neighboring blocks are inferred. The array indices x0, y0 indicate the location (x0, y0) of the top left luma sample of the coding block in question, relative to the top left luma sample of the picture . When merge_subblock_flag[x0][y0] is not present, it is logically inferred to be 0.

[0185] Merge_subblock_idx[x0][y0] indicates the index of the merge candidate from the subblock-based merge candidate list, where x0, y0 indicate the location (x0, y0) of the upper left luma sample of the coding block in question relative to the upper left picture luma sample.

[0186] When merge_subblock_idx[x0][y0] is not present, it is logically inferred to be 0.

[0187] If merge_subblock_flag[xCb][yCb] is equal to 1, the extraction process for motion vectors and reference indices in sub-block merge mode as specified in 8.4.4.2 is activated with the location (xCb, yCb) of the luminance coding block, width cbWidth of the luminance coding block and cbHeight of the luminance coding block as inputs, the number of luminance coding subblocks in the horizontal direction numSbX and in the vertical direction numSbY, reference indices refIdxL0, refIdxL1, arrays predFlagL0[xSbIdx][ySbIdx] and predFlagL1[xSbIdx] [ySbIdx] flags for using prediction lists, arrays mvL0[xSbIdx][ySbIdx] and mvL0[xSbIdx][ySbIdx] of motion vectors of luma signal subblocks and arrays mvCL0[xSbIdx][ySbIdx], and mvCL1[xSbIdx][ySbIdx] of motion vectors of subblocks chrominance signals, with xSbIdx=0...numSbX-1, ySbIdx=0...numSbY-1 and biprediction weight index gbiIdx as outputs.

[0188] It should be recognized that, depending on the example, certain steps or events of any of the technologies described herein may be performed in a different sequence, may be added, combined, or omitted altogether (for example, not all of the described steps or events are required for practical technology implementation). In addition, in certain examples, steps or events may be executed simultaneously, such as through multithreading, interrupt handling, or across multiple processors, rather than sequentially.

[0189] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination of the foregoing. When implemented in software, the functions may be stored or transmitted, as one or more instructions or code, on a computer-readable medium and executed by a hardware processing module. Computer-readable media may include computer-readable storage media that corresponds to a tangible medium, such as storage media, or communication media, including any medium that facilitates the transfer of a computer program from one place to another, such as according to a communication protocol. Thus, computer-readable media can generally correspond to (1) a tangible computer-readable storage medium that is non-volatile, or (2) a communication medium such as a signal or carrier. Storage media can be any available media that can be accessed by one or more computers or one or more processors in order to retrieve instructions, code, and/or data structures for implementing the technologies described in this disclosure. The computer program product may include a computer readable medium.

[0190] By way of example, and not limitation, these computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disc storage device, magnetic disk storage device or other magnetic storage devices, flash memory or any other medium that can be used to store the required program code in the form of instructions or data structures and that can be accessed by a computer. In addition, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a Web site, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave media, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave media are included in the definition of media. However, it should be understood that computer-readable storage media and storage media do not include connections, carriers, signals, or other volatile media, but are instead directed to nonvolatile tangible storage media. Disc (disk) and disc (disc) as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, while discs (disk ) usually play data magnetically, while disks (disc) usually play data optically using lasers. Combinations of the above should also be included in computer readable media.

[0191] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. . Accordingly, the term "processor" as used herein can mean any of the above structures, or another structure suitable for implementing the technologies described herein. In addition, in some aspects, the functionality described herein may be provided within specialized hardware and/or software modules capable of encoding or decoding or embedded in a combined codec. In addition, technologies may be fully implemented in one or more circuits or logic elements.

[0192] The technologies of this disclosure may be implemented in a wide variety of devices or equipment, including a wireless handset, an integrated circuit (IC), or an IC stack (eg, a chipset). Various components, modules, or blocks are described in this disclosure in order to emphasize the functional aspects of devices capable of implementing the disclosed technologies, but do not necessarily require implementation by different hardware modules. Conversely, as described above, various modules may be combined into a codec hardware module or provided through a set of cooperating hardware modules including one or more processors as described above in combination with appropriate software and/or firmware.

[0193] Various examples have been described above. These and other examples are within the scope of the appended claims.

Claims (63)

1. A method for decoding video data, the method comprising the steps of: - receive encoded video data; parsing the motion flag based on prediction subunits from the encoded video data; - in response to determining that the motion flag based on prediction subunits is active, retrieve the list of motion prediction candidates at the level of prediction subunits; - in response to determining that the motion flag based on prediction subunits is not active, retrieve the list of motion prediction candidates at the level of prediction units; selecting a motion vector predictor from either a list of motion prediction candidates at the prediction subunit level or from a list of motion prediction candidates at the prediction unit level, wherein the prediction subunit level motion prediction candidate list or the prediction unit level motion prediction candidate list is at least partially ordered based on motion prediction occurrences in neighboring blocks; and - decoding the encoded video data using the selected motion vector predictor. 2. The method of claim 1, wherein the encoded video data includes a current block, and wherein the prediction subunit level motion prediction candidate list and the prediction unit level motion prediction candidate list are derived from adjacent blocks of the current block. 3. The method of claim 2, wherein the neighboring blocks are spatial neighbors of the current block in the current picture, or temporal neighbors of the current block in a previously encoded picture. 4. The method of claim 1, wherein pixels in the prediction unit share first motion vector information and pixels in the prediction subunit share second motion vector information, wherein the first motion vector information or the second vector information is determined from a selected motion vector predictor. 5. The method of claim 1, wherein the motion vector candidate list at the prediction unit level includes at least one of spatial neighbor candidates and temporal neighbor candidates. 6. The method of claim 1, wherein the motion prediction candidate list at the prediction subunit level includes at least one of the following: affine motion vector prediction, alternative temporal motion vector prediction (ATMVP), space-time motion vector prediction (STMVP) ), planar motion vector prediction, and template-based motion vector extraction (PMVD). 7. The method according to claim 1, further comprising the step of: extracting the merge candidate index in response to determining that the motion flag based on the prediction subunits is active, wherein the merge candidate index indicates the motion vector predictor to be selected. 8. Equipment for decoding video data, and the equipment contains: - a storage device for storing the received encoded video data; and - a processor, wherein the processor is configured to: - parse the motion flag based on the prediction subunits from the encoded video data, - in response to determining that the movement flag based on prediction subunits is active, retrieve the list of motion prediction candidates at the level of prediction subunits, - in response to determining that the movement flag based on prediction subunits is not active, retrieve the list of motion prediction candidates at the level of prediction units, - select a motion vector predictor either from a list of motion prediction candidates at the prediction subunit level, or from a list of motion prediction candidates at the prediction unit level, wherein the motion prediction candidate list at the prediction subunit level or the motion prediction candidate list at the prediction unit level is at least partially ordered based on motion prediction occurrences in neighboring blocks, and - decode the encoded video data using the selected motion vector predictor. 9. The equipment of claim 8, wherein the encoded video data includes a current block, and wherein the prediction subunit level motion prediction candidate list and the prediction unit level motion prediction candidate list are derived from neighboring blocks of the current block. 10. The equipment of claim 9, wherein the neighboring blocks are the spatial neighbors of the current block in the current picture, or the temporal neighbors of the current block in the previously encoded picture. 11. The equipment of claim 8, wherein the pixels in the prediction unit share the first motion vector information and the pixels in the prediction subunit share the second motion vector information, the first motion vector information or the second vector information being determined from the selected motion vector predictor. 12. The equipment of claim 8, wherein the motion vector candidate list at the prediction unit level includes at least one of spatial neighbor candidates and temporal neighbor candidates. 13. The equipment of claim 8, wherein the motion prediction candidate list at the prediction subunit level includes at least one of the following: affine motion vector prediction, alternative temporal motion vector prediction (ATMVP), space-time motion vector prediction (STMVP) ), planar motion vector prediction, and template-based motion vector extraction (PMVD). 14. Equipment according to claim 8, in which the processor is additionally configured to: - retrieve the merge candidate index in response to determining that the motion flag based on prediction subunits is active, wherein the merge candidate index indicates the motion vector predictor to be selected. 15. Equipment for decoding video data, and the equipment contains: - storage means for storing received encoded video data; and - a processor means, wherein the processor means is configured to: - parse the motion flag based on the prediction subunits from the encoded video data, - in response to determining that the movement flag based on prediction subunits is active, retrieve the list of motion prediction candidates at the level of prediction subunits, - in response to determining that the movement flag based on prediction subunits is not active, retrieve the list of motion prediction candidates at the level of prediction units, - select a motion vector predictor either from a list of motion prediction candidates at the prediction subunit level, or from a list of motion prediction candidates at the prediction unit level, wherein the motion prediction candidate list at the prediction subunit level or the motion prediction candidate list at the prediction unit level is at least partially ordered based on motion prediction occurrences in neighboring blocks, and - decode the encoded video data using the selected motion vector predictor. 16. The equipment of claim 15, wherein the encoded video data includes a current block, and wherein the prediction subunit level motion prediction candidate list and the prediction unit level motion prediction candidate list are derived from neighboring blocks of the current block. 17. The equipment of claim 16, wherein the neighboring blocks are the spatial neighbors of the current block in the current picture, or the temporal neighbors of the current block in the previously encoded picture. 18. The equipment of claim 15, wherein pixels in the prediction unit share first motion vector information and pixels in the prediction subunit share second motion vector information, wherein the first motion vector information or the second vector information is determined from a selected motion vector predictor. 19. The equipment of claim 15, wherein the motion vector candidate list at the prediction unit level includes at least one of spatial neighbor candidates and temporal neighbor candidates. 20. The equipment of claim 15, wherein the motion prediction candidate list at the prediction subunit level includes at least one of the following: affine motion vector prediction, alternative temporal motion vector prediction (ATMVP), space-time motion vector prediction (STMVP) ), planar motion vector prediction, and template-based motion vector extraction (PMVD). 21. The equipment of claim 15, wherein the processor means is further configured to: - retrieve the merge candidate index in response to determining that the motion flag based on prediction subunits is active, wherein the merge candidate index indicates the motion vector predictor to be selected. 22. A non-volatile computer-readable storage medium having stored instructions that, when executed, instructs one or more processors to perform a method, comprising: - reception of encoded video data; - parsing the motion flag based on prediction subunits from encoded video data; - in response to determining that the motion flag based on prediction subunits is active, extracting a list of motion prediction candidates at the level of prediction subunits; - in response to determining that the motion flag based on prediction subunits is not active, extracting a list of motion prediction candidates at the level of prediction units; - selecting a motion vector predictor from either a list of motion prediction candidates at the prediction subunit level or from a list of motion prediction candidates at the prediction unit level, wherein the prediction subunit level motion prediction candidate list or the prediction unit level motion prediction candidate list is at least partially ordered based on motion prediction occurrences in neighboring blocks; and - decoding the encoded video data using the selected motion vector predictor. 23. The medium of claim 22, wherein the encoded video data includes a current block, wherein the prediction subunit level motion prediction candidate list and the prediction unit level motion prediction candidate list are derived from neighboring blocks of the current block. 24. The carrier of claim 23, wherein the neighboring blocks are the spatial neighbors of the current block in the current picture, or the temporal neighbors of the current block in the previously encoded picture. 25. The carrier according to claim 22, in which: the pixels in the prediction unit share the first motion vector information and the pixels in the prediction subunit share the second motion vector information, wherein the first motion vector information or the second vector information is determined from the selected motion vector predictor, wherein the motion vector candidate list at the prediction unit level includes at least one of the following: spatial neighbor candidates and temporal neighbor candidates, and wherein the list of motion prediction candidates at the prediction subunit level includes at least one of the following: affine motion vector prediction, alternative temporal motion vector prediction (ATMVP), spatiotemporal motion vector prediction (STMVP), planar motion vector prediction, and motion vector extraction based on pattern matching (PMVD). 26. The carrier of claim 22, wherein the method further comprises: - extracting the merge candidate index in response to determining that the motion flag based on prediction subunits is active, wherein the merge candidate index indicates the motion vector predictor to be selected.

Images