RU2820638C2 - Method for calculating position of reference sample of integer grid for calculating gradient of boundary sample of block level in calculating optical flow with dual prediction and correction with dual prediction - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to image processing and relates to a method for predicting a bidirectional optical flow based on a sample value from a sampling position belonging to an extended region of a predicted block used in predicting a bidirectional optical stream of the current block. Method includes obtaining a motion vector of a current block, obtaining a subpixel sampling position in an extended region of a predicted block in a reference picture based on the obtained motion vector, processing the obtained subpixel sampling position to obtain an integer sampling position of pixels in a reference picture and performing bidirectional optical flow prediction by accessing the reference sample value in the obtained integer pixel sample position in the reference picture.

EFFECT: high degree of image compression with minimal loss of quality.

35 cl, 10 dwg

Description

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to Indian Patent Application IN201931025013 filed on June 24, 2019 with the Indian Patent Office, the disclosure of which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

Embodiments of the present application generally relate to the field of image processing and, in particular, to optical flow refinement.

BACKGROUND OF THE ART

Video encoding (video encoding and decoding) is used in a wide range of digital video applications, such as digital broadcast television, video transmission over the Internet and mobile networks, real-time conversational applications such as video chat, video conferencing, DVD and Blu-ray discs, systems for collecting and editing video content and video cameras for security applications.

The amount of video data required to present even a relatively short video can be significant, which can lead to difficulties when the data must be streamed or otherwise communicated over a communications network with limited bandwidth. Thus, video data is typically compressed before being transmitted over modern telecommunications networks. Video size can also be an issue when the video is stored on a storage device, as memory resources may be limited. Video compression devices often use software and/or hardware at the source to encode video data before transmission or storage, thereby reducing the amount of data required to represent digital video images. The compressed data is then received by the recipient's video decompressor, which decodes the video data. With limited network resources and ever-increasing demands for higher video quality, improved compression and decompression techniques are desired that improve the compression ratio with little or no loss in picture quality.

ESSENCE OF DISCLOSURE

Embodiments of the present application provide apparatus and methods for encoding and decoding according to independent claims.

The above and other goals are achieved by the invention according to independent claims. Additional forms of implementation are obvious from the dependent claims, description and figures.

A first embodiment of the present disclosure provides a method, implemented by an encoding device or a decoding device, for predicting a bidirectional optical flow based on a sample value from a sample position belonging to an extended area of a predicted block used in predicting a bidirectional optical flow of a current block, the method comprising obtaining a vector motion (MV) of the current block, obtaining a subpixel sampling position in the extended region of the predicted block in the reference picture based on the MV, processing the obtained subpixel sampling position to obtain an integer pixel sampling position in the reference picture, and performing bidirectional optical flow prediction by referring to the reference sampling value in the resulting integer position of a sample of pixels in the reference picture.

Processing the resulting subpixel sample position to obtain an integer pixel sample position in the reference picture can reduce the memory footprint for bidirectional optical flow (BDOF) prediction and allow the establishment of a subblock level pipeline in hardware designs as long as extended sample values (that is, sample positions that extend beyond the positions M x N samples corresponding to the sub-block) are not obtained through motion-compensated interpolation of the corresponding reference samples.

The reference sample value at the obtained integer pixel sampling position in the reference picture may be called the sampling value from the sub-pixel sampling position in bidirectional optical flow prediction.

Processing the received subpixel sample position may include rounding the obtained subpixel sample position to obtain an integer pixel sample position in the reference picture.

By rounding the subpixel position in the horizontal and vertical directions, boundary gradients can be more precise. This can improve the calculated optical flow based on these sampling gradients, as well as the corrections applied using the calculated optical flow. This may provide a consistent encoding gain over the earlier method of ignoring fractional offsets in the horizontal and vertical directions when choosing an integer sample value for extended sample positions. At the same time, the complexity cannot be increased much, since adding horizontal and vertical rounding offsets is the only increase in complexity.

Processing the obtained subpixel sampling position may include rounding the obtained subpixel sampling position separately horizontally and vertically to the corresponding nearest integer pixel sampling position in the corresponding direction, and obtaining an integer pixel sampling position in the reference picture based on the nearest integer pixel sampling positions in the horizontal and vertical directions. .

The method may further comprise, before performing bidirectional optical flow prediction, selecting (extracting) a reference sample value at the obtained integer sample position of pixels in the reference picture as a prediction sample value for use in bidirectional optical flow prediction, wherein performing bidirectional optical flow prediction with reference the reference sample value comprises performing a bidirectional optical flow prediction using the prediction sample value.

According to the implementation, rounding the obtained subpixel sample position separately in the horizontal direction and vertical direction may include comparing a fractional part of the obtained subpixel sample position in the corresponding direction with a threshold value, wherein if the fractional part is equal to or greater than the threshold value, the nearest upper integer the pixel sampling position in the corresponding direction is obtained as the nearest integer pixel sampling position in the corresponding direction, and wherein in case the fractional part is less than the threshold value, the nearest lower integer pixel sampling position in the corresponding direction is obtained as the nearest integer pixel sampling position in the corresponding direction.

The nearest upper integer pixel sampling position in the corresponding direction can be obtained by adding an offset of 1 to the truncated integer pixel sampling position in the corresponding direction obtained by discarding the fractional part of the resulting sub-pixel sampling position, and the nearest lower integer pixel sampling position in the corresponding direction can be obtained by adding an offset of 0 to the truncated integer position of the truncated pixel sample in the appropriate direction, obtained by discarding the fractional part of the resulting subpixel sample position.

The threshold can be set to P/2 for sub-pixel precision of 1/P pel (pixel). The threshold value can be 8 for sub-pixel precision of 1/16 pel.

According to the implementation, rounding the obtained subpixel sample position separately in the horizontal direction and the vertical direction may include applying a shift operation and a reverse shift operation to the components of the obtained subpixel sample position in the respective directions.

The integer position (x_pos, y_pos) of a sample of pixels in the reference picture can be obtained according to the following equations:

x_pos = (x_spel_pos + (1 << (shift_val – 1))) >> shift_val

y_pos = (y_spel_pos + (1 << (shift_val – 1))) >> shift_val

where (x_spel_pos, y_spel_pos) is the resulting subpixel sampling position and shift_val is chosen according to the subpixel sampling precision used. For subpixel sampling precision equal to half pixel, quarter pixel, 1/8 pixel or 1/16 pixel, you can use shift_val equal to 1, 2, 3 or 4 respectively.

According to the implementation, the MV of the current block may be a refined MV obtained by motion vector refinement (MVR) based on the initial MV.

The MVR may comprise an integer distance refinement step with respect to the initial MV, wherein, in the event that the integer distance MVR refinement step results in a non-zero offset of the refined MV relative to the initial MV, the integer pixel sample position in the reference picture may be obtained by rounding the resulting subpixel sample position.

In case the integer MVR distance refinement step results in a zero offset of the refined MV with respect to the initial MV, the integer pixel sample position in the reference picture can be obtained by discarding the fractional part of the resulting subpixel sample position.

According to the implementation, in the case that the MVR integer distance refinement step results in a zero offset of the refined MV relative to the initial MV, the integer position (x_pos, y_pos) of the sample pixels in the reference picture can be obtained according to the following equations:

x_pos = x_spel_pos >> shift_val

y_pos = y_spel_pos >> shift_val

where (x_spel_pos, y_spel_pos) is the resulting subpixel sampling position and shift_val is chosen according to the subpixel sampling precision used. For subpixel sampling precision equal to half pixel, quarter pixel, 1/8 pixel or 1/16 pixel, you can use shift_val equal to 1, 2, 3 or 4 respectively.

The current block may be a current encoding block or a current sub-encoding block or a current prediction block.

The reference sample value at the resulting integer sample position of pixels in the reference picture can be used in calculating a boundary sample gradient corresponding to the boundary of a predicted block in the reference picture.

According to an aspect of the first embodiment, an encoder is provided comprising processing circuitry for performing any of the methods according to the first embodiment.

According to a further aspect of the first embodiment, a decoder is provided including processing circuitry for performing any of the methods according to the first embodiment.

According to a further aspect of the first embodiment, a computer program product is provided containing instructions that, when the program is executed by a computer, cause the computer to perform any of the methods according to the first embodiment.

According to a further aspect of the first embodiment, an encoder is provided comprising one or more processors and a non-transitory computer-readable storage medium coupled to the one or more processors and storing instructions for execution by the one or more processors, wherein the instructions, when executed by the one or more processors, configure the encoder to performing any of the methods according to the first embodiment.

According to a further aspect of the first embodiment, a decoder is provided comprising one or more processors and a non-volatile computer-readable storage medium coupled to the one or more processors and storing instructions for execution by the one or more processors, wherein the instructions, when executed by the one or more processors, configure the decoder to performing any of the methods according to the first embodiment.

According to a further aspect of the first embodiment, an encoder is provided, comprising a prediction unit configured to perform bidirectional optical flow prediction of a current block, a first acquiring unit configured to obtain a motion vector (MV) of the current block, a second acquiring unit configured to obtain a subpixel sampling position in an extended area of the predicted block in the reference picture based on the MV, and a processing unit configured to process the obtained sub-pixel sampling position to obtain an integer pixel sampling position in the reference picture, wherein the prediction unit is configured to perform bidirectional optical flow prediction by referring to the reference sampling value at the resulting integer sample position of pixels in the reference picture.

According to the implementation, processing the received sub-pixel sample position may include rounding the received sub-pixel sample position to obtain an integer pixel sample position in the reference picture.

According to the implementation, processing the obtained subpixel sampling position may include rounding the obtained subpixel sampling position separately in the horizontal direction and vertical direction to the corresponding nearest integer pixel sampling position in the corresponding direction, and obtaining the integer pixel sampling position in the reference picture based on the nearest integer pixel sampling positions in horizontal direction and vertical direction.

Rounding the resulting subpixel sample position separately in the horizontal direction and the vertical direction may involve comparing a fractional portion of the resulting subpixel sample position in the corresponding direction with a threshold value,

wherein, in the case where the fractional part is equal to or greater than the threshold value, the nearest integer pixel sampling position in the corresponding direction can be obtained as the nearest integer pixel sampling position in the corresponding direction, and in this case, in the case where the fractional part is less than the threshold value, the nearest integer the pixel sampling position in the corresponding direction can be obtained as the nearest integer pixel sampling position in the corresponding direction.

The nearest upper integer pixel sampling position in the corresponding direction can be obtained by adding an offset of 1 to the truncated integer pixel sampling position in the corresponding direction obtained by discarding the fractional part of the resulting sub-pixel sampling position, and the nearest lower integer pixel sampling position in the corresponding direction can be obtained by adding an offset of 0 to the truncated integer pixel sample position in the appropriate direction, obtained by discarding the fractional part of the resulting subpixel sample position.

According to a further aspect of the first embodiment, a decoder is provided, comprising a prediction unit configured to perform bidirectional optical flow prediction of the current block, a first acquiring unit configured to obtain a motion vector (MV) of the current block from the encoded video bitstream, a second acquiring unit configured to obtaining a subpixel sampling position in an extended region of a predicted block in a reference picture based on the MV, and a processing unit configured to process the obtained subpixel sampling position to obtain an integer pixel sampling position in the reference picture, wherein the prediction unit is configured to perform bidirectional optical flow prediction by reversing to the reference sample value at the resulting integer sample position of pixels in the reference picture.

According to the implementation, processing the received sub-pixel sample position may include rounding the received sub-pixel sample position to obtain an integer pixel sample position in the reference picture.

According to the implementation, processing the obtained subpixel sampling position may include rounding the obtained subpixel sampling position separately in the horizontal direction and vertical direction to the corresponding nearest integer pixel sampling position in the corresponding direction, and obtaining the integer pixel sampling position in the reference picture based on the nearest integer pixel sampling positions in horizontal direction and vertical direction.

Rounding the resulting subpixel sample position separately in the horizontal direction and the vertical direction may involve comparing a fractional portion of the resulting subpixel sample position in the corresponding direction with a threshold value,

wherein, in the case where the fractional part is equal to or greater than the threshold value, the nearest integer pixel sample position in the corresponding direction can be obtained as the nearest integer pixel sample position in the corresponding direction, and in this case, in the case where the fractional part is less than the threshold value, the nearest lower the integer pixel sampling position in the corresponding direction can be obtained as the nearest integer pixel sampling position in the corresponding direction.

The nearest upper integer pixel sampling position in the corresponding direction can be obtained by adding an offset of 1 to the truncated integer pixel sampling position in the corresponding direction obtained by discarding the fractional part of the resulting sub-pixel sampling position, and the nearest lower integer pixel sampling position in the corresponding direction can be obtained by adding an offset of 0 to the truncated integer pixel sample position in the appropriate direction, obtained by discarding the fractional part of the resulting subpixel sample position.

According to a further aspect of the first embodiment, a storage medium storing a bit stream that is obtained using any of the methods of the first embodiment is provided.

A second embodiment of the present disclosure provides a method used for obtaining a predicted sample value for a sample position belonging to an extended area of a predicted block used in bidirectional optical flow prediction of a current encoding block, comprising obtaining a first MV corresponding to a predicted sample in an extended area of the encoding block, obtaining a second MV by discarding the fractional part of the first MV, obtaining the position of the first integer pixel of the extended region in the reference picture based on the second MV, determining the fractional offset vector based on the first MV, and obtaining the position of the second integer pixel in the reference picture based on the position of the first integer pixel and the fractional offset vector , wherein the position of the second integer pixel is set equal to the position of the first integer pixel plus a position offset, where the position offset is determined by comparing each component of the fractional offset vector and a threshold value, and further padding the predicted sample using the pixel value at the position of the second integer pixel in the reference picture.

Whether the position offset component is 1 or 0 may depend on the result of a comparison between the corresponding fractional offset vector component and the threshold value, where the component includes an X component and a Y component.

The threshold value may be K, and the position offset component may be set to 1 when the corresponding fractional offset component is equal to or greater than K, otherwise set to 0. K may be 7.

According to an aspect of the second embodiment, an encoder or decoder is provided including processing circuitry for performing any of the methods according to the second embodiment.

According to a further aspect of the second embodiment, a computer program product is provided comprising program code for performing any of the methods according to the second embodiment.

According to a further aspect of the second embodiment, a decoder or encoder is provided comprising one or more processors and a non-volatile computer readable storage medium coupled to the processors and storing programs for execution by the processors, wherein the program, when executed by the processors, configures the decoder or encoder to perform any of the methods according to the second embodiment.

Details of one or more embodiments are set forth in the accompanying drawings and in the following description. Other features, objects and advantages will be apparent from the description, drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the disclosure are now described in more detail with reference to the accompanying figures and drawings, in which:

Fig. 1A is a block diagram showing an example of a video encoding system configured to implement embodiments of the present disclosure;

Fig. 1B is a block diagram showing another example of a video encoding system configured to implement embodiments of the present disclosure;

Fig. 2 is a block diagram showing an example of a video encoder configured to implement embodiments of the present disclosure;

Fig. 3 is a block diagram showing an example structure of a video decoder configured to implement embodiments of the present disclosure;

Fig. 4 is a block diagram illustrating an example of encoding equipment or decoding equipment;

Fig. 5 is a block diagram illustrating another example of encoding equipment or decoding equipment;

Fig. 6 is a diagram illustrating the process of rounding a subpixel position in one of the directions in a reference picture used in the integer sampling process for calculating the BDOF gradient;

Fig. 7 is a diagram illustrating integer positions using a process of rounding a subpixel position in a reference picture in an integer sample selection process for calculating a BDOF gradient;

Fig. 8 shows a flowchart of a video encoding/decoding method according to an embodiment of the disclosure;

Fig. 9 shows a block diagram illustrating an example of encoding/decoding equipment according to an embodiment of the disclosure.

In the following, like reference numerals refer to the same or at least functionally equivalent features, unless explicitly stated otherwise.

DETAILED DESCRIPTION OF IMPLEMENTATION OPTIONS

In the following description, reference is made to the accompanying drawings, which form a part of the disclosure and show by way of illustration specific aspects of embodiments of the present disclosure or specific aspects in which embodiments of the present disclosure may be used. It is understood that embodiments of this disclosure may be used in other aspects and contain structural or logical changes not shown in the figures. Therefore, the following detailed description is not to be taken in a limiting sense, and the scope of the present disclosure is determined by the appended claims.

For example, it will be understood that the disclosure associated with the described method may also apply to a corresponding device or system configured to perform the method, and vice versa. For example, if one or a plurality of specific method steps are described, the corresponding apparatus may include one or a plurality of blocks, such as functional blocks, to perform the one or multiple method steps described (e.g., one block performing one or a plurality of steps, or a plurality of blocks, each of which performs one or more of a plurality of steps), even if such one or more blocks are not explicitly described or illustrated in the figures. On the other hand, for example, if a particular apparatus is described in terms of one or a plurality of blocks, such as functional blocks, the corresponding method may include one step for performing the functionality of the one or multiple blocks (for example, one step performing the functionality of the one or multiple blocks , or a plurality of steps, each of which performs the functionality of one or more of a plurality of blocks), even if such one or multiple steps are not explicitly described or illustrated in the figures. It is further understood that features of various exemplary embodiments and/or aspects described herein may be combined with each other unless specifically stated otherwise.

Video encoding generally refers to the processing of the sequence of pictures that form a video or video sequence. Instead of the term "picture", the term "frame" or "image" can be used interchangeably in the field of video encoding. Video encoding (or encoding in general) consists of two parts: video encoding and video decoding. Video encoding is performed at the source side and typically involves processing (eg, through compression) of the source video images to reduce the amount of data required to represent the video images (for more efficient storage and/or transmission). Video decoding is performed on the receiver (destination) side and usually contains reverse processing compared to the encoder for reconstructing video images. Embodiments relating to "encoding" of video pictures (or pictures in general) are to be understood as referring to "encoding" or "decoding" video pictures or corresponding video sequences. The combination of an encoding part and a decoding part is also called CODEC (encoding and decoding).

In the case of lossless video encoding, the original video images can be restored, i.e., the restored video images have the same quality as the original video images (provided that no transmission loss or other data loss occurs during storage or transmission). In the case of lossy video encoding, additional compression is performed, for example through quantization, to reduce the amount of data representing video images that cannot be fully reconstructed at the decoder, i.e. the quality of the restored video images is lower or worse compared to the quality of the original video images.

Several video coding standards belong to the group of "hybrid lossy video codecs" (i.e., they combine spatial and temporal prediction in the sample domain and 2D transform coding to apply quantization in the transform domain). Each picture of a video sequence is usually divided into a set of non-overlapping blocks, and encoding is usually done at the block level (block level). In other words, in a video encoder, video is typically processed, that is, encoded, at the block (video block) level, for example using spatial (in-picture) prediction and/or temporal (out-of-picture) prediction to generate a prediction block, the prediction block being subtracted from the current block ( block that is currently being processed/to be processed) to obtain a residual block, the residual block is transformed and this residual block is quantized in the transform domain to reduce the amount of data (compression) to be transmitted, whereas in the decoder, reverse processing compared to the encoder is applied to an encoded or compressed block to reconstruct the current block for presentation. In addition, the encoder duplicates the decoder's processing loop so that they will both generate identical predictions (eg, intra and outer predictions) and/or reconstructions for processing, ie encoding, of subsequent blocks.

In the following embodiments, video encoding systems 10, video encoder 20, and video decoder 30 are described based on FIGS. from 1 to 3.

Fig. 1A is a schematic block diagram illustrating an exemplary encoding system 10, such as video encoding system 10 (or encoding system 10 for short), which may use the techniques of the present application. Video encoder 20 (or encoder 20 for short) and video decoder 30 (or decoder 30 for short) of video encoding system 10 are examples of devices that may be configured to perform methods in accordance with various examples described herein.

As shown in FIG. 1A, encoding system 10 includes a source device 12 configured to provide encoded picture data 21, for example, to a destination device 14 for decoding the encoded picture data 13.

The source device 12 contains an encoder 20 and can additionally, i.e. optionally comprising a picture source 16, a preprocessor (or preprocessing unit) 18, such as a picture preprocessor 18, and a communications interface or communications unit 22.

The picture source 16 may contain or be any type of picture capturing device, such as a camera for capturing a real world picture, and/or any type of picture generating device, such as a computer graphics processor for generating a computer animated picture, or any type of other device for obtaining and /or providing a real-world image, a computer-generated image (e.g., screen content, virtual reality (VR) image), and/or any combination thereof (e.g., augmented reality (AR) image). The source of the picture can be any type of memory or storage that stores any of the above pictures.

In contrast to the preprocessor 18 and the processing performed by the preprocessing unit 18, the picture or picture data 17 may also be referred to as a raw picture or raw picture data 17.

The preprocessor 18 may be configured to receive (raw) picture data 17 and perform preprocessing on the picture data 17 to produce a preprocessed picture 19 or preprocessed picture data 19. Preprocessing performed by preprocessor 18 may, for example, comprise cropping, color format conversion (eg, RGB to YCbCr), color correction, or noise reduction. It can be understood that the preprocessing unit 18 may be an optional component.

Video encoder 20 may be configured to receive preprocessed picture data 19 and provide encoded picture data 21 (further details will be described below, eg based on FIG. 2).

The communication interface 22 of the source device 12 may be configured to receive encoded picture data 21 and transmit this encoded picture data 21 (or any further processed version thereof) over the communication channel 13 to another device, for example, the destination device 14 or any other device. , for saving or immediate restoration.

The recipient device 14 includes a decoder 30 (eg, video decoder 30) and may additionally, i.e. optionally comprising a communication interface or communication unit 28, a post processor 32 (or post processing unit 32) and a display device 34.

The communications interface 28 of the destination device 14 may be configured to receive the encoded picture data 21 (or any further processed version thereof), for example, directly from the source device 12 or from any other source, for example, a storage device such as an encoded data storage device pictures, and provide the encoded picture data 21 to the decoder 30.

The communication interface 22 and the communication interface 28 may be configured to transmit or receive encoded picture data 21 or encoded data 13 through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or through a network of any type, for example, a wired or wireless network or any combination thereof, or any private and public network, or any combination thereof.

The communications interface 22 may be configured to package the encoded picture data 21 into an appropriate format, such as packets, and/or process the encoded picture data using any form of transmission encoding or processing for transmission over a communications link or network.

Communication interface 28, constituting a counterpart to communication interface 22, may be configured to receive transmitted data and process the transmission data using any form of appropriate decoding or transmission processing and/or decompression to obtain encoded picture data 21.

Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces, as shown by the arrow for communication channel 13 in FIG. 1A pointing from a source device 12 to a destination device 14, or as bidirectional communication interfaces, and may be configured to send and receive messages, for example, to establish a connection, to confirm and exchange any other information related to the communication link and/ or data transmission, such as transmission of encoded picture data.

Decoder 30 may be configured to receive encoded picture data 21 and provide decoded picture data 31 or decoded picture 31 (further details will be described below, for example based on FIG. 3 or FIG. 5).

The post-processor 32 of the recipient device 14 may be configured to post-process the decoded picture data 31 (also referred to as reconstructed picture data), such as the decoded picture 31, to obtain post-processed picture data 33, such as the post-processed picture 33. Post-processing performed by the post-processing unit 32 , may include any one or more color format conversions (eg, from YCbCr to RGB), color correction, cropping or resampling, or any other processing, for example, to prepare decoded picture data 31 for display, for example, by display device 34.

The display device 34 of the recipient device 14 may be configured to receive post-processed picture data 33 for displaying the picture, for example, to a user or viewer. The display device 34 may be or include any kind of display for presenting the reconstructed picture, such as a built-in or external display or monitor. The display may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other display.

Although fig. 1A depicts source device 12 and destination device 14 as separate devices, embodiments of the devices may also comprise both devices or both functionality, that is, source device 12 or related functionality and destination device 14 or related functionality. In such embodiments, source device 12 or related functionality and destination device 14 or related functionality may be implemented using the same hardware and/or software, or using separate hardware and/or software, or any combination thereof. .

As will be apparent to one skilled in the art from the description, the presence and (precise) separation of functionality of various blocks or functionality in source device 12 and/or destination device 14, as shown in FIG. 1A, may vary depending on the actual device and applications.

An encoder 20 (eg, video encoder 20) or a decoder 30 (eg, video decoder 30), or both, encoder 20 and decoder 30 may be implemented using the processing circuit shown in FIG. 1B, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video encoding, or any combination thereof. Encoder 20 may be implemented through processing circuit 46 to implement various modules, as described with respect to encoder 20 in FIG. 2 and/or any other encoder system or subsystem described in this document. Decoder 30 may be implemented through processing circuit 46 to implement various modules, as discussed with respect to decoder 30 of FIG. 3 and/or any other decoder system or subsystem described herein. The processing circuit may be configured to perform various operations, which will be described below. As shown in FIG. 5, if the techniques are partially implemented in software, the apparatus may store instructions for the software on a suitable non-volatile computer readable storage medium and may execute the instructions in hardware using one or more processors to execute the techniques of this disclosure. Video encoder 20 and video decoder 30 may be integrated as part of a combined encoder/decoder (CODEC) in one device, for example, as shown in FIG. 1B.

The video encoding system 40 shown in FIG. 1B includes processing circuitry implementing both a video encoder 20 and a video decoder 30. In addition, one or more imaging devices 41, such as a real world camera, an antenna 42, one or more storage devices 44, one or more processors 43 and/or display device 45, such as display device 34 described above, may be provided as part of video encoding system 40.

Source device 12 and destination device 14 may include any of a wide range of devices, including any type of portable or desktop devices, such as notebooks or laptops, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, game consoles, video streaming devices (such as content service servers or content delivery servers), broadcast receiver devices, broadcast transmitter devices, and the like. and can use any or any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

In some cases, the video encoding system 10 illustrated in FIG. 1A is merely an example, and the techniques of the present application may be applied to video encoding systems (eg, video encoding or video decoding) that do not necessarily involve any data transfer between encoding and decoding devices. In other examples, data is retrieved from local memory, streamed over a network, or the like. A video encoding apparatus may encode and store data in memory, and/or a video decoding apparatus may retrieve and decode data from memory. In some examples, encoding and decoding are performed by devices that do not communicate with each other, but simply encode data into memory and/or retrieve and decode data from memory.

For convenience of description, embodiments of this disclosure are described herein, for example, with reference to High Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC) reference software, a next-generation video coding standard being developed by the Joint Collaboration on Video Coding (JCT-VC) Expert Group. ITU-T Video Coding (VCEG) and ISO/IEC Moving Picture Expert Group (MPEG). One of ordinary skill in the art will appreciate that embodiments of this disclosure are not limited to HEVC or VVC.

Encoder and encoding method

Fig. 2 shows a schematic block diagram of an exemplary video encoder 20 that is configured to implement the methods of the present application. In the example of FIG. 2, video encoder 20 includes an input 201 (or input interface 201), a remainder calculation block 204, a transform processing block 206, a quantization block 208, an inverse quantization block 210 and an inverse transform processing block 212, a restoration block 214, a contour block 220 filter, decoded picture buffer (DPB) 230, mode selection unit 260, entropy encoding unit 270, and output 272 (or output interface 272). The mode selection block 260 may include an inter prediction block 244, an intra prediction block 254, and a separating block 262. The inter prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 shown in FIG. 2 may also be called a hybrid video encoder or a video encoder according to a hybrid video codec.

The remainder calculation block 204, transform processing block 206, quantization block 208, and mode selection block 260 may be referred to as forming the forward signal path of the encoder 20, while the inverse quantization block 210, inverse transform processing block 212, reconstruction block 214, loop filter 220, buffer decoded picture (DPB) 230, inter prediction block 244 and intra prediction block 254 may be said to form a return signal path of video encoder 20, wherein the return signal path of video encoder 20 corresponds to the signal path of the decoder (see video decoder 30 in FIG. 3). The inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded picture buffer (DPB) 230, the inter prediction unit 244, and the intra prediction unit 254 are also relevant to forming the “on-chip decoder” of the video encoder 20.

Pictures and picture division (pictures and blocks)

Encoder 20 may be configured to receive, for example, through input 201, a picture 17 (or picture data 17), for example, a picture of a sequence of pictures forming a video or video sequence. The received picture or picture data may also be a preprocessed picture 19 (or preprocessed picture data 19). For simplicity, the following description refers to picture 17. Picture 17 may also be referred to as the current picture or the picture that is to be encoded (particularly in video encoding to distinguish the current picture from other pictures, such as previously encoded and/or decoded pictures of the same video sequences, i.e. video sequences that also contain the current picture).

A (digital) picture is, or can be viewed as, a two-dimensional array or matrix of samples with intensity values. A selection in an array may also be referred to as a pixel (short form of picture element) or pel (picture element). The number of samples in the horizontal and vertical direction (or axis) of the array or picture determines the size and/or resolution of the picture. To represent color, three color components are usually used, that is, the picture can be represented as three arrays of samples or can include them. In RBG format, or color space, a picture contains a corresponding array of red, green, and blue color samples. However, in video encoding, each pixel is typically represented in a luminance and chrominance format or color space, such as YCbCr, which contains a luminance component indicated by Y (sometimes L is also used instead) and two chrominance components (chrominance components) indicated by Cb and Cr. The Y component of luminance (or luma for short) represents the brightness or intensity of the gray level (such as in a halftone picture), while the two components Cb and Cr are chrominance (or chrominance for short). chroma) represent components of chromaticity or color information. Accordingly, a picture in YCbCr format contains an array of luminance samples with luminance sample values (Y) and two arrays of chrominance samples with chrominance values (Cb and Cr). Pictures in RGB format can be converted or converted to YCbCr format and vice versa. This process is also known as color conversion or conversion. If the picture is monochrome, it can only contain an array of brightness samples. Accordingly, a picture can be, for example, an array of luminance samples in a monochrome format or an array of luminance samples and two corresponding arrays of chrominance samples in 4:2:0, 4:2:2 and 4:4:4 color formats.

Embodiments of video encoder 20 may include a picture splitter (not shown in FIG. 2) configured to divide picture 17 into multiple (usually non-overlapping) picture blocks 203. These blocks may also be called root blocks, macroblocks (H.264/AVC) or coding tree blocks (CTB) or coding tree units (CTU) (as per H.265/HEVC and VVC). The picture division unit may be configured to use the same block size for all pictures in a video sequence and a corresponding grid defining the block size, or to vary the block size between pictures or subsets or groups of pictures and divide each picture into appropriate blocks.

In additional embodiments, the video encoder may be configured to receive directly a block 203 of a picture 17, such as one, more, or all of the blocks forming a picture 17. The picture block 203 may also be referred to as a current picture block or a picture block to be encoded.

Like picture 17, picture block 203 is or can be thought of as a two-dimensional array or matrix of samples with intensity values (sample values), albeit smaller in size than picture 17. In other words, block 203 may contain, for example, a single array of samples (e.g. a luminance array in the case of a monochrome picture 17 or a luminance or chrominance array in the case of a color picture) or three arrays of samples (for example, luminance and two chrominance arrays in the case of a color picture 17) or any other number and/or type of arrays depending on the color format used . The number of samples in the horizontal and vertical direction (or axis) of block 203 determines the size of block 203. Accordingly, the block may, for example, contain an M×N array of samples (M-column by N-row) or an M×N array of transform coefficients.

Embodiments of video encoder 20 shown in FIG. 2 may be configured to encode the picture 17 block by block, for example, encoding and prediction is performed for each block 203.

Embodiments of video encoder 20, as shown in FIG. 2 may further be configured to divide and/or encode a picture using slices (also referred to as video slices), wherein the picture may be divided into or encoded using one or more slices (usually non-overlapping), and each slice may contain one or more units (for example, CTU). A slice may contain a collection of rectangular tiles (tiled images) or may be a collection of raster order CTU strings within a tile.

Embodiments of video encoder 20 shown in FIG. 2 may be further configured to divide and/or encode a picture using groups of tiles (also called video tile groups) and/or tiles (also called video tiles), wherein the picture may be divided or encoded using one or more groups of tiles (usually not overlapping), and each group of tiles may contain one or more blocks (such as a CTU) or one or more tiles, where each tile may be rectangular in shape and may contain one or more blocks ( e.g. CTU), such as full or fractional blocks.

Calculating the remainder

Residual calculation block 204 may be configured to calculate residual block 205 (also referred to as residual 205) based on picture block 203 and prediction block 265 (additional details about prediction block 265 are provided below), for example, by subtracting the sample values of prediction block 265 from the values samples the picture block 203, sample by sample (pixel by pixel), to obtain a residual block 205 in the sample area.

Conversion

Transform processing unit 206 may be configured to apply a transform, such as a discrete cosine transform (DCT) or a discrete sine transform (DST), to the sample values of the residual block 205 to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 may also be referred to as residual transform coefficients and represent the residual block 205 in the transform domain.

Transform processing block 206 may be configured to apply DCT/DST integer approximations, such as the transforms defined for H.265/HEVC. Compared to the orthogonal DCT transform, such integer approximations are usually scaled by some factor. To preserve the norm of the residual block that is processed by the forward and inverse transforms, additional scaling factors are applied as part of the transform process. Scaling factors are typically chosen based on some constraints, e.g., scaling factors are powers of two for shift operations, bit depth of transform factors, tradeoff between accuracy and implementation cost, etc. Specific scaling factors, for example, are specified for inverse transform, e.g. by the inverse transform processing unit 212 (and the corresponding inverse transform, such as the inverse transform processing unit 312 in the video decoder 30), and the corresponding scaling factors for the forward transform, for example, by the transform processing unit 206, can be set appropriately in the encoder 20.

Embodiments of video encoder 20 (respectively, transform processing unit 206) may be configured to output transform parameters, such as the type of transform or transforms, for example, directly or encoded or compressed via entropy encoding unit 270, so that, for example, video decoder 30 may receive and use conversion parameters for decoding.

Quantization

The quantization unit 208 may be configured to quantize the transform coefficients 207 to obtain quantized coefficients 209, for example, by applying scalar quantization or vector quantization. Quantized coefficients 209 may also be referred to as quantized transform coefficients 209 or quantized residual coefficients 209.

The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207. For example, an n-bit transform factor may be rounded to an m-bit transform factor during quantization, where n is greater than m. The degree of quantization can be modified by adjusting the quantization parameter (QP). For example, for scalar quantization, different scalings can be applied to achieve finer or coarser quantization. Smaller quantization step sizes correspond to finer quantization, while larger quantization step sizes correspond to coarser quantization. The applicable quantization step size may be specified by the quantization parameter (QP). The quantization parameter may, for example, be an index to a predetermined set of applicable quantization step sizes. For example, small quantization parameters may correspond to fine quantization (small quantization step sizes), and large quantization parameters may correspond to coarse quantization (large quantization step sizes), or vice versa. Quantization may include dividing by a quantization step size, and corresponding and/or inverse dequantization, such as by inverse quantization unit 210, may include multiplying by a quantization step size. Embodiments in accordance with some standards, such as HEVC, may be configured to use a quantization parameter to determine the quantization step size. Typically, the quantization step size can be calculated based on the quantization parameter using a fixed point approximation of the equation involving division. Additional scaling factors can be introduced for quantization and dequantization to recover the residual block norm that may have been changed due to the scaling used in the fixed point approximation of the above equation for the quantization step size and the quantization parameter. In one exemplary implementation, inverse transform scaling and dequantization may be combined. Alternatively, customized quantization tables can be used and signaled from encoder to decoder, for example in a bitstream. Quantization is a lossy operation in which losses increase as the size of the quantization steps increases.

Embodiments of video encoder 20 (respectively, quantization block 208) may be configured to output quantization parameters (QPs), for example, directly or encoded through entropy encoding block 270, so that, for example, video decoder 30 can receive and apply quantization parameters for decoding.

Inverse quantization

The inverse quantization block 210 is configured to apply the inverse quantization of the quantization block 208 to the quantized coefficients to obtain the dequantized coefficients 211, for example, by applying an inverse quantization scheme to that applied by the quantization block 208, based on or using the same quantization step size as the block 208 quantization. The dequantized coefficients 211 may also be referred to as the dequantized residual coefficients 211 and correspond, although they are typically not identical to the transform coefficients due to quantization loss, to the transform coefficients 207 .

Reverse conversion

The inverse transform processing unit 212 is configured to apply an inverse transform relative to the transform applied by the transform processing unit 206, such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST) or other inverse transforms to obtain the reconstructed residual block 213 (or corresponding dequantized coefficients 213) in the sample area. The reconstructed residual block 213 may also be referred to as a transform block 213.

Recovery

A reconstruction block 214 (e.g., an add block or an adder 214) is configured to add a transform block 213 (i.e., a reconstructed residual block 213) with a prediction block 265 to obtain a reconstructed block 215 in the sample domain, for example, by means of an addition, sample per sample, the sample values of the reconstructed residual block 213, and the sample values of the prediction block 265.

Filtration

Loop filter block 220 (or “loop filter” 220 for short) is configured to filter the reconstructed block 215 to obtain a filtered block 221, or more generally to filter the reconstructed samples to obtain filtered samples. The edge filter block can be configured to smooth pixel transitions or otherwise improve video quality. The edge filter block 220 may include one or more edge filters, such as a deblocking filter, an adaptive offset filter (SAO), or one or more other filters, such as a two-way filter, an adaptive edge filter (ALF), a sharpening filter, an anti-aliasing filter, or a collaborative filter. filter, or any combination of them. Although loop filter block 220 is shown in FIG. 2 as a loop filter, in other configurations the loop filter block 220 may be implemented as a post-loop filter. Filtered block 221 may also be referred to as filtered reconstructed block 221.

Embodiments of video encoder 20 (respectively, loop filter block 220) may be configured to output loop filter parameters (such as adaptive sample offset information), for example, directly or encoded through entropy encoding block 270, so that, for example, decoder 30 can receive and apply the same loop filter parameters or corresponding loop filters for decoding.

Buffer of decoded pictures

Decoded picture buffer (DPB) 230 may be a memory that stores reference pictures, or generally reference picture data, for encoding video data by video encoder 20. DPB 230 may be formed by any of a variety of storage devices, such as dynamic random access memory (DRAM). DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer (DPB) 230 may be configured to store one or more filtered blocks 221. The decoded picture buffer 230 may be further configured to store other previously filtered blocks, such as previously reconstructed and filtered blocks 221, the same current picture, or different pictures, for example, previously restored pictures, and can provide complete previously restored ones, i.e. decoded, pictures (and corresponding reference blocks and samples) and/or partially reconstructed current picture (and corresponding reference blocks and samples), for example, for inter-prediction. The decoded picture buffer (DPB) 230 may also be configured to store one or more unfiltered reconstructed blocks 215, or generally unfiltered reconstructed samples, for example, if the reconstructed block 215 is not filtered by the loop filter block 220, or any other further processed version of the reconstructed blocks. or samples.

Mode selection (separation and prediction)

The mode selection block 260 includes a division block 262, an inter prediction block 244, and an intra prediction block 254, and is configured to receive or obtain original picture data such as the original block 203 (current block 203 of the current picture 17), and reconstructed picture data such as filtered and/or unfiltered reconstructed samples or blocks of the same (current) picture and/or from one or more previously decoded pictures, for example, from decoded picture buffer 230 or other buffers (eg, a line buffer, not shown). The reconstructed picture data is used as reference picture data for prediction, for example, inter-prediction or intra-prediction to obtain a prediction block 265 or a predictor 265.

The mode selection block 260 may be configured to determine or select a split for the current block prediction mode (including no split) and prediction mode (eg, intra or inter prediction mode) and generate a corresponding prediction block 265 that is used to calculate the residual block 205 and to recovery of the restored block 215.

Embodiments of the mode selector 260 may be configured to select a split and prediction mode (e.g., from those supported by or available to the mode selector 260) that provides the best match, or in other words, the minimum remainder (the minimum remainder means the best compression for transmission or storage), or minimal signaling overhead (minimum signaling overhead means better compression for transmission or storage), or which takes into account or balances both factors. The mode selector 260 may be configured to determine a division and prediction mode based on rate distortion optimization (RDO), that is, selecting a prediction mode that provides minimal rate distortion. Terms such as "best", "minimum", "optimal", etc. in this context do not necessarily refer to the all-encompassing "best", "minimum", "optimal", etc., but may also refer to meeting a selection or termination criterion, such as when a value exceeds or falls below a threshold, or other constraints, potentially leading to "suboptimal selection" but reducing complexity and processing time.

In other words, partitioning block 262 may be configured to partition block 203 into smaller sections or sub-blocks (which form blocks again), for example, iteratively using quad-tree (QT) partitioning, binary-tree (BT) partitioning, or partitioning by triple tree (TT) or any combination thereof, and to perform prediction for each of the partitions or sub-blocks of the block, wherein the mode selection comprises selecting a tree structure of the partitioned block 203 and the prediction modes are applied to each of the block partitions or sub-blocks.

Next, the division (e.g., by division block 262) and prediction processing (by inter prediction block 244 and intra prediction block 254) performed by the exemplary video encoder 20 will be explained in more detail.

Separation

Divider 262 may divide (or split) the current block 203 into smaller divisions, such as smaller blocks of square or rectangular size. These smaller blocks (which may also be called subblocks) can be further divided into even smaller sections. This is also called tree partitioning or hierarchical tree partitioning, in which the root block, for example at root tree level 0 (hierarchy level 0, depth 0), can be recursively partitioned, for example divided into two or more blocks of the next lower tree level. for example, nodes at tree level 1 (hierarchy level 1, depth 1), where these blocks can again be divided into two or more blocks of the next lower level, for example tree level 2 (hierarchy level 2, depth 2), etc. until the split is completed, for example because a completion criterion is met, such as the maximum tree depth or minimum block size being reached. Blocks that are not further divided are also called leaf blocks or leaf nodes of a tree. A tree that uses a two-partition partition is called a binary tree (BT), a tree that uses a three-partition partition is called a triple-tree (TT), and a tree that uses a four-partition partition is called a quad-tree (QT).

As mentioned earlier, the term "block" as used herein can be a part, particularly a square or rectangular part of a picture. With reference to, for example, HEVC and VVC, a block can either correspond to a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU) or a transform unit (TU) and/or to corresponding blocks, e.g. a tree block coding block (CTB), coding block (CB), transform block (TB) or prediction block (PB).

For example, a coding tree unit (CTU) may be either a CTB of luma samples and two corresponding CTBs of chroma samples of a picture that has three arrays of samples, or a CTB of samples of a monochrome picture or a picture that is encoded using three separate color planes and syntax structures used for encoding samples. Accordingly, a coding tree block (CTB) may be an N×N block of samples for some value of N, such that dividing a component by the CTB is a division. A coding unit (CU) may be or comprise a luma sample encoding unit and two corresponding chrominance sample encoding units of a picture that has three sample arrays, or a monochrome sample encoding unit or a picture that is encoded using three separate color planes and syntactic structures used for encoding samples. Accordingly, a coding block (CB) may be an M×N block of samples for some values of M and N, so that dividing the CTB into coding blocks is a division.

In some embodiments, for example, according to HEVC, a coding tree unit (CTU) may be divided into CUs using a quad-tree structure, referred to as a coding tree. The decision of whether to encode a picture region using inter-picture (temporal) or intra-picture (spatial) prediction is made at the CU level. Each CU can be further partitioned into one, two or four PUs according to the PU partition type. Within a single PU, the same prediction process is applied, and the relevant information is passed to the PU-based decoder. After obtaining the residual block by applying a prediction process based on the partitioning type of the PU, the CU can be divided into transformation units (TUs) according to another quad-tree structure similar to the encoding tree for the CU.

In embodiments, for example, in accordance with a recent video coding standard currently under development called Versatile Video Coding (VVC), a combined quad-tree and binary tree (QTBT) partition, for example, is used to partition a coding block. In a block structure, a QTBT CU can be either square or rectangular in shape. For example, a coding tree unit (CTU) is first divided by a quad-tree structure. The leaf nodes of a quad-tree are further separated by a binary tree or ternary (or triple) tree structure. The leaf nodes of a partitioning tree are called coding units (CUs), and this partitioning is used for prediction and transformation processing without any additional partitioning. This means that CU, PU and TU have the same block size in the QTBT coding block structure. Multiple partitions can be used in parallel with the QTBT block structure, such as a ternary tree partition.

In one example, the mode selector 260 of the video encoder 20 may be configured to perform any combination of the division methods described herein.

As described above, video encoder 20 is configured to determine or select the best or optimal prediction mode from a set of (eg, predefined) prediction modes. The set of prediction modes may comprise intra prediction modes and/or inter prediction modes.

Internal prediction

The set of intra prediction modes may contain 35 different intra prediction modes, such as non-directional modes, such as DC (or average) mode and planar mode, or directional modes, for example, as defined in HEVC, or may contain 67 different intra prediction modes, such as as non-directional modes such as DC (or average) mode and planar mode, or directional modes such as those defined for VVC.

The intra prediction block 254 is configured to use reconstructed samples of neighboring blocks of the same current picture to generate an (intra) prediction block 265 in accordance with an intra prediction mode from a set of intra prediction modes.

Intra prediction block 254 (or more generally, mode selector 260) may be further configured to output intra prediction parameters (or more generally, information indicating the selected intra prediction mode for a block) to entropy encoding block 270 in the form of syntax elements 266 to include pictures in the encoded data 21 so that, for example, video decoder 30 can receive and use prediction parameters for decoding.

External prediction

The set of inter-prediction modes (or possible ones) depends on the available reference pictures (i.e., previously at least partially decoded pictures, e.g., stored in DBP 230) and other inter-prediction parameters, e.g., whether the entire reference picture is or only part of it, for example, the search window area around the current block area, the reference picture is used to find the most suitable reference block, and/or, for example, whether pixel interpolation such as half/half-pixel and/or quarter-pixel interpolation is applied or not .

In addition to the above prediction modes, a skip mode and/or a direct mode may be used.

The inter prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both not shown in FIG. 2). The motion estimator may be configured to receive or obtain a picture block 203 (the current picture block 203 of the current picture 17) and a decoded picture 231, or at least one or a plurality of previously reconstructed blocks, such as reconstructed blocks of one or a plurality of previously decoded pictures. 231 for motion assessment. As an example, a video sequence may comprise a current picture and previously decoded pictures 231, or in other words, the current picture and previously decoded pictures 231 may be part of or form a sequence of pictures forming a video sequence.

Encoder 20 may be configured to select a reference block from a plurality of reference blocks of the same or different pictures from a plurality of previously decoded pictures and provide a reference picture (or reference picture index) and/or an offset (spatial offset) between the position (x, y coordinates) of the reference block and the position of the current block as inter-frame prediction parameters for the motion estimation block. This displacement is also called motion vector (MV). In some implementations, the MV may be further refined on both the encoding and decoding sides to improve prediction accuracy and ultimately improve compression performance. Moreover, the prediction obtained by MV or refined MV can be further modified or corrected using bidirectional prediction optical flow (BPOF) refinement, whereby the prediction can be compensated in the case where fine motion exists at the pixel level. Decoding-side motion vector refinement (DMVR) and (BPOF) technology will be described in detail later.

The motion compensation block may be configured to receive, for example, receive an inter prediction parameter and perform inter prediction based on or using the inter prediction parameter to obtain an (inter) prediction block 265. The motion compensation performed by the motion compensation block may include obtaining or generating a prediction block based on a motion vector/block determined by motion estimation, possibly performing interpolations with sub-pixel accuracy. Interpolation filtering can generate additional pixel samples from known pixel samples, thereby potentially increasing the number of candidate prediction blocks that can be used to encode a picture block. After receiving the motion vector for the PU of the current picture block, the motion compensation unit can determine the position of the prediction block pointed to by the motion vector in one of the reference picture lists.

The motion compensation block may also generate syntax elements associated with video blocks and slices for use by video decoder 30 when decoding the picture blocks of the video slice. In addition to or as an alternative to slices and corresponding syntax elements, groups of tiles and/or tiles and corresponding syntax elements may be generated or used.

Entropy coding

Entropy encoding block 270 is configured to apply, for example, an entropy encoding algorithm or scheme (e.g., variable length coding (VLC) scheme, context adaptive VLC (CAVLC), arithmetic coding scheme, binarization, context adaptive binary arithmetic coding (CABAC) ), based on the syntax of context-adaptive binary arithmetic coding (SBAC), probability interval entropy encoding (PIPE), or other entropy encoding method or technique) or bypass (uncompressed) quantized coefficients 209, inter prediction parameters, intra prediction parameters, parameters loop filter and/or other syntax elements to obtain encoded picture data 21, which may be output through output 272, for example, in the form of an encoded bit stream 21, so that, for example, video decoder 30 can receive and use parameters for decoding. The encoded bitstream 21 may be transmitted to video decoder 30 or stored in memory for later transmission or retrieval by video decoder 30.

Other changes to the structure of video encoder 20 may be used to encode the video stream. For example, non-transform encoder 20 may quantize the residual signal directly without transform processing unit 206. In another implementation, encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

Decoder and decoding method

Fig. 3 shows an example of a video decoder 30 that is configured to implement the methods of the present application. Video decoder 30 is configured to receive encoded picture data 21 (e.g., encoded bit stream 21), such as encoded by encoder 20, to obtain decoded picture 331. The encoded picture data or bit stream contains information for decoding the encoded picture data, e.g., data that represent picture blocks of an encoded video slice (and/or group of tiles or tile) and associated syntax elements.

In the example in FIG. 3, decoder 30 includes an entropy decoding block 304, an inverse quantization block 310, an inverse transform processing block 312, a recovery block 314 (e.g., adder 314), a loop filter 320, a decoded picture buffer (DBP) 330, a mode application block 360, an external block 344 prediction and intra prediction block 354. The inter prediction unit 344 may be or include a motion compensation unit. Video decoder 30 may, in some examples, perform a decoding step that is generally the opposite of the encoding step described with respect to video encoder 20 in FIG. 2.

As explained with respect to the encoder 20, the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded picture buffer (DPB) 230, the inter prediction unit 244, and the intra prediction unit 254 is also referred to as “embedded decoder” generation. video encoder 20. Accordingly, the inverse quantization unit 310 may be identical in function to the inverse quantization unit 210, the inverse transform processing unit 312 may be identical in function to the inverse transform processing unit 212, the restoration unit 314 may be identical in function to the restoration unit 214, and the loop filter 320. may be identical in function to the loop filter 220, and the decoded picture buffer 330 may be identical in function to the decoded picture buffer 230. Therefore, the explanations provided for the corresponding blocks and functions of the video encoder 20 apply correspondingly to the corresponding blocks and functions of the video decoder 30.

Entropy Decoding

Entropy decoding block 304 is configured to parse bitstream 21 (or generally encoded picture data 21) and perform, for example, entropy decoding on encoded picture data 21 to obtain, for example, quantized coefficients 309 and/or decoded encoding parameters 366 , for example, any or all of the inter-prediction parameters (eg, reference picture index and motion vector), intra-prediction parameters (eg, intra-prediction mode or index), transform parameters, quantization parameters, loop filter parameters, and/or other syntax elements. Entropy decoding block 304 may be configured to employ decoding algorithms or schemes corresponding to encoding schemes as described with respect to entropy encoding block 270 of encoder 20. Entropy decoding block 304 may be further configured to provide inter prediction parameters, intra prediction parameters, and/or other syntax elements to mode application block 360 and other parameters to other blocks of decoder 30. Video decoder 30 may receive syntax elements at the video slice level and/or video block level. In addition to or as an alternative to slices and corresponding syntax elements, groups of tiles and/or tiles and corresponding syntax elements may be received and/or used.

Inverse quantization

An inverse quantization block 310 may be configured to receive quantization parameters (QPs) (or, more generally, information related to inverse quantization) and quantized coefficients from the encoded picture data 21 (e.g., by parsing and/or decoding, e.g., by a block 304 entropy decoding) and apply, based on the quantization parameters, inverse quantization to the decoded quantized coefficients 309 to obtain dequantized coefficients 311, which may also be referred to as transform coefficients 311. The inverse quantization process may include using a quantization parameter determined by video encoder 20 for each video block in a video slice (or tile or group of tiles) to determine the degree of quantization and, likewise, the degree of inverse quantization to be applied.

Reverse conversion

The inverse transform processing unit 312 may be configured to receive the dequantized coefficients 311, also referred to as transform coefficients 311, and apply a transform to the dequantized coefficients 311 to obtain reconstructed residual blocks 313 in the sample region. The reconstructed residual blocks 313 may also be referred to as transform blocks 313. The conversion may be an inverse transform, such as an inverse DCT, an inverse DST, an integer integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 may be further configured to receive transform parameters or corresponding information from the encoded picture data 21 (e.g., by parsing and/or decoding, e.g., by entropy decoding unit 304) to determine a transform to be applied to dequantized coefficients 311.

Recovery

A reconstruction block 314 (e.g., an add block or an adder 314) may be configured to add the reconstructed residual block 313 with a prediction block 365 to obtain a reconstructed block 315 in the sample domain, for example by adding the sample values of the reconstructed residual block 313 and the sample values of the block 365 predictions.

Filtration

A loop filter block 320 (either in the encoding loop or after the encoding loop) is configured to filter the reconstructed block 315 to produce a filtered block 321, for example, to smooth pixel transitions or otherwise improve video quality. The edge filter block 320 may include one or more edge filters, such as a deblocking filter, an adaptive offset filter (SAO), or one or more other filters, such as a two-way filter, an adaptive edge filter (ALF), a sharpener, an anti-aliasing filter, or a collaborative filter. filter, or any combination of them. Although loop filter block 320 is shown in FIG. 3 as a loop filter, in other configurations the loop filter block 320 may be implemented as a post-loop filter.

Buffer of decoded pictures

The decoded video picture blocks 321 are then stored in a decoded picture buffer 330, which stores the decoded pictures 331 as reference pictures for subsequent motion compensation for other pictures and/or for output or display, respectively.

The decoder 30 is configured to output the decoded picture 311, for example, through an output 312 for presentation or viewing to a user.

Prediction

The inter prediction block 344 may be identical to the inter prediction block 244 (specifically, the motion compensation module), and the intra prediction block 354 may be identical to the intra prediction block 254 in function and performs division or division decisions and prediction based on division parameters and/or prediction or corresponding information received from the encoded picture data 21 (eg, by parsing and/or decoding, eg, by entropy decoding unit 304). Mode application block 360 may be configured to perform prediction (intra or inter prediction) for each block based on the reconstructed pictures, blocks, or corresponding samples (filtered or unfiltered) to obtain prediction block 365.

When a video or picture slice is encoded as an intra-encoded (I) slice, the intra prediction block 354 of the mode application block 360 is configured to generate a prediction block 365 for the picture block of the current video slice based on the signaled intra prediction mode and data from previously decoded current picture blocks. When a video slice or picture is encoded as an outer-encoded slice (i.e., B or P), an inter-prediction block 344 (e.g., a motion compensation block) of the mode application block 360 is configured to create a prediction block 365 for the video block of the current video slice based on the motion vectors and other syntax elements received from entropy decoding block 304. For inter-prediction, prediction blocks may be created from one of the reference pictures in one of the reference picture lists. Video decoder 30 may create reference picture lists, List 0 and List 1, using default construction methods based on reference pictures stored in DPB 330. The same or similar approach may be applied to or with embodiments using groups of tiles (eg , video tile groups) and/or slices (e.g., video tiles) in addition to or as an alternative to slices (e.g., video slices), for example, a video may be encoded using I, P, or B tile groups and/or tiles.

Mode application block 360 is configured to determine prediction information for a video/picture block of the current video slice by analyzing motion vectors or related information and other syntax elements and using the prediction information to create prediction blocks for the current video block that is being decoded. For example, mode application block 360 uses some of the received syntax elements to determine the prediction mode (e.g., intra or inter prediction) used to encode video blocks of a video slice, the type of inter-prediction slice (e.g., B-slice, P-slice, or GPB slice ), design information for one or more reference picture lists for a slice, motion vectors for each externally encoded video block of the slice, inter-prediction status for each externally encoded video block of the slice, and other information for decoding video blocks in the current slice. The same or similar approach can be applied to or with embodiments using groups of tiles (e.g., a group of video tiles) and/or slices (e.g., video tiles) in addition to or as an alternative to slices (e.g., video slices), e.g. video can be encoded using groups of I, P or B tiles and/or tals.

Embodiments of video decoder 30, as shown in FIG. 3 may be configured to split and/or decode a picture using slices (also referred to as video slices), wherein the picture may be divided into or encoded using one or more slices (usually non-overlapping), and each slice may contain one or more blocks (for example, CTU).

Embodiments of video decoder 30, as shown in FIG. 3, may be configured to divide and/or decode a picture using groups of tiles (also called tile groups) and/or tiles (also called video tiles), wherein the picture can be divided into or decoded using one or more groups of tiles (usually non-overlapping), and each group of tiles may contain one or more blocks (such as a CTU) or one or more tiles, where each tile may be rectangular in shape and may contain one or more blocks (e.g. CTU), such as full or fractional blocks.

Other embodiments of video decoder 30 may be used to decode the encoded picture data 21. For example, decoder 30 may produce an output video stream without loop filtering block 320. For example, the non-transform decoder 30 may perform inverse quantization of the residual signal directly without detransforming processing unit 312 for some blocks or frames. In another implementation, video decoder 30 may have an inverse quantization block 310 and an inverse transform processing block 312 combined into one block.

It should be understood that in the encoder 20 and decoder 30, the result of processing of some current stage may be further processed and then output to the next stage. For example, after interpolation filtering, motion vector acquisition, or contour filtering, an additional operation such as Clip or offset may be performed on the result of the interpolation filtering, motion vector acquisition, or contour filtering processing.

It should be noted that additional operations may be applied to the derived motion vectors of the current block (including, but not limited to, affine mode checkpoint motion vectors, affine mode subblock motion vectors, planar mode motion vectors, ATMVP mode motion vectors, temporary motion vectors, and so on). For example, the value of a motion vector is limited to a predefined range according to the number of bits representing it. If the number of bits of the motion vector representation is bitDepth, then the range is -2^(bitDepth-1) ~ 2^(bitDepth-1)-1, where "^" means exponentiation. For example, if bitDepth is 16, the range is -32768~32767; if bitDepth is 18, the range is -131072~131071. For example, the value of the derived motion vector (e.g., the MVs of four 4x4 subblocks in one 8x8 block) is limited such that the maximum difference between the integer parts of the four MVs of a 4x4 subblock is no more than N pixels, e.g., no more than 1 pixel . The following description provides two ways to constrain the motion vector according to bitDepth.

Method 1: Remove the most significant overflow bit (most significant bit) using the following operations:

ux= ( mvx+2 ^bitDepth ) % 2 ^bitDepth (1)

mvx = ( ux >= 2 ^bitDepth-1 ) ? (ux − 2 ^bitDepth ): ux (2)

uy= ( mvy+2 ^bitDepth ) % 2 ^bitDepth (3)

mvy = ( uy >= 2 ^bitDepth-1 ) ? (uy − 2 ^bitDepth ): uy (4)

where mvx is the horizontal component of the motion vector of an image block or subblock, mvy is the vertical component of the motion vector of an image block or subblock, and ux and uy indicate the corresponding intermediate values.

For example, if the value of mvx is -32769, after applying formulas (1) and (2), the resulting value is 32767. In a computer system, decimal numbers are stored as two's complement. Two's complement -32769 is 1.0111,1111,1111,1111 (17 bits). The MSB is then discarded, so the resulting two's complement is 0111,1111,1111,1111 (decimal 32767), which is the same as the result of applying formulas (1) and (2).

ux= ( mvpx + mvdx +2 ^bitDepth ) % 2 ^bitDepth (5)

mvx = ( ux >= 2 ^bitDepth-1 ) ? (ux − 2 ^bitDepth ): ux (6)

uy= ( mvpy + mvdy +2 ^bitDepth ) % 2 ^bitDepth (7)

mvy = ( uy >= 2 ^bitDepth-1 ) ? (uy − 2 ^bitDepth ): uy (8)

Operations may be applied during the summation of the motion vector predictor mvp and the motion vector difference mvd, as shown in formulas (5) to (8).

Method 2: Removing MSB overflow by truncating the value:

vx = Clip3(-2 ^bitDepth-1 , 2 ^bitDepth-1 -1, vx)

vy = Clip3(-2 ^bitDepth-1 , 2 ^bitDepth-1 -1, vy)

where vx is the horizontal component of the motion vector of a block or subblock of the image, vy is the vertical component of the motion vector of the block or subblock of the image; x, y and z respectively correspond to the three input values of the MV clipping process, and the definition of the Clip3 function is as follows:

Clip3( x, y, z )=

Fig. 4 is a schematic block diagram of a video encoding apparatus 400 according to an embodiment of the present disclosure. Video encoding apparatus 400 is suitable for implementing the disclosed embodiments, as described below. In an embodiment, video encoding device 400 may be a decoder, such as video decoder 30 of FIG. 1A, or an encoder such as video encoder 20 of FIG. 1A.

The video encoding device 400 may include input ports 410 (or input ports 410) and one or more receive units (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; one or more transmit units (Tx) 440 and output ports 450 (or output ports 450) for transmitting data; and a memory 460 for storing data. Video encoding device 400 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components connected to input ports 410, receiver blocks 420, transmitter blocks 440, and output ports 450 to provide optical or optical input or output. electrical signals.

Processor 430 may be implemented in hardware or software. Processor 430 may be implemented as one or more CPU chips, cores (eg, a multi-core processor), FPGA, ASIC, and DSP. Processor 430 may communicate with input ports 410, receiver blocks 420, transmitter blocks 440, output ports 450, and memory 460. Processor 430 may include an encoding block 470. Encoding block 470 implements the disclosed embodiments described above and below. For example, encoding block 470 may implement, process, prepare, or provide various encoding operations. Therefore, the inclusion of the encoding block 470 provides a significant improvement in the functionality of the video encoding device 400 and allows the video encoding device 400 to be transformed into a different state. Alternatively, encoding block 470 may be implemented as instructions stored in memory 460 and executed by processor 430.

Memory 460 may include one or more disks, tape drives, and solid-state drives and may be used as an overflow storage device for storing programs when such programs are selected for execution and for storing instructions and data that are read during execution of the programs. Memory 460 may be, for example, volatile and/or non-volatile and may be read only memory (ROM), random access memory (RAM), ternary memory associative memory (TCAM), and/or static random access memory (SRAM).

Fig. 5 is a simplified block diagram of an apparatus 500 that may be used as one or both of the source device 12 and destination device 14 of FIG. 1A according to an exemplary embodiment .

The processor 502 in the hardware 500 may be a central processing unit. Alternatively, processor 502 may be any other type of device or plurality of devices capable of manipulating or processing information that currently exists or will be developed in the future. Although the disclosed implementations may be practiced with a single processor, as shown, for example, with processor 502, speed and efficiency benefits may be achieved by using more than one processor.

Memory 504 in hardware 500 may be read-only memory (ROM) or random access memory (RAM) in an implementation. Memory 504 may be any other suitable type of memory device. Memory 504 may include code and data 506 accessed by processor 502 using bus 512. Memory 504 may further include an operating system 508 and application programs 510, application programs 510 including at least one program that allows processor 502 to perform the methods described herein. For example, application programs 510 may include applications 1 through N, which further include a video encoding application that performs the methods described herein.

The apparatus 500 may also include one or more output devices, such as a display 518. The display 518 may be, in one example, a touch display that combines a display with a touch element capable of sensing touch inputs. Display 518 may be coupled to processor 502 via bus 512.

Although a single bus is depicted here, bus 512 of device 500 may be composed of multiple buses. In addition, secondary storage (not shown) may be directly coupled to other components of hardware 500 or may be accessible over a network and may comprise a single integrated unit, such as a memory card, or multiple units, such as multiple memory cards. Thus, the apparatus 500 can be implemented in a wide variety of configurations.

Motion Vector Refinement (MVR)

Motion vectors are typically at least partially determined on the encoder side and transmitted to the decoder in a coded bit stream. However, motion vectors may also be refined at the decoder (as well as at the encoder), starting from the initial motion vectors specified in the bitstream. In such a case, for example, the similarity between fragments or blocks of already decoded pixels indicated by the initial motion vectors can be used to improve the accuracy of the initial motion vectors. This motion vector refinement provides the benefit of reducing signaling overhead: the accuracy of the initial motion vector is improved equally in both the encoder and decoder, and thus no additional signaling is required for refinement.

It should be noted that the original motion vectors before refinement may not be the best motion vectors that lead to the best prediction. Since the initial motion vectors are transmitted in a bit stream, it may not be possible to represent the initial motion vector with very high accuracy (which would increase the data rate). Therefore, the motion vector refinement process is used to improve the initial motion vector. The initial motion vectors may, for example, be motion vectors that are used in predicting a neighbor block of the current block. In this case, it is sufficient to signal an indication in the bitstream indicating the motion vectors of which neighboring block is being used by the current block. This prediction mechanism is very effective in reducing the number of bits to represent the initial motion vectors. However, the accuracy of the initial motion vectors may be low because in general the motion vectors of two adjacent blocks are not expected to be identical.

To further improve the accuracy of the motion vectors without further increasing signaling overhead, it may be useful to further refine the motion vectors derived at the encoder side and provided (signaled) in the bit stream. Motion vector refinement can be performed in the decoder without the help of an encoder. The encoder in its decoder loop can use the same refinement to obtain corresponding refined motion vectors that would be available at the decoder. Refinement for the current block that is reconstructed in the current image is performed by determining the pattern of the reconstructed samples, defining a search space around the original motion information for the current block, and finding the part of the reference image with the best match to the pattern in the search space. The best-fit part determines refined motion vectors for the current block, which are then used to obtain inter-predicted samples for the current block, that is, the current block that is being reconstructed.

Motion vector refinement is part of inter prediction block 244 in FIG. 2 and 344 in FIG. 3.

Refinement of the motion vector can be performed in accordance with the following steps:

Typically, the initial motion vector can be determined based on the indication in the bit stream. For example, an index may be reported in the bitstream that indicates a position in a list of possible motion vectors. In another example, the motion vector predictor index and the motion vector difference value may be signaled in the bitstream. Motion vectors that are determined based on the indication in the bit stream are defined as initial motion vectors. In the case of dual prediction, where the outer prediction for the current block is obtained as a weighted combination of predicted sample blocks that are determined according to two motion vectors, let the initial motion vector in the first reference picture in the first reference picture list L0 be denoted as MV0, and the initial motion vector in the second the reference picture in the second reference picture list L1 is designated as MV1.

Using the initial motion vectors, candidate motion vector (MV) pairs for refinement are determined. At least two candidate pairs for refinement can be identified. Typically, candidate pairs of motion vectors for refinement are determined based on a pair of initial motion vectors (MV0, MV1). Additionally, candidate MV pairs can be determined by adding small motion vector differences to MV0 and MV1. For example, candidate MV pairs could include the following:

(MV0, MV1)

(MV0 + (0,1), MV1 + (0, -1))

(MV0 + (1.0), MV1 + (-1.0))

(MV0 + (0, -1), MV1 + (0,1))

(MV0 + (-1.0), MV1 + (1.0))

...

Where (1, -1) denotes a vector that has an offset of 1 integer pixel in the horizontal (or x) direction and an offset of -1 integer pixel in the vertical (or y) direction.

It should be noted that the above list of candidate MV pairs are merely examples for illustration purposes, and the disclosure is not limited to a specific list of candidates.

Refinement of candidate motion vector (MV) pairs form the search space of the motion vector refinement process.

In dual prediction of the current block, two prediction blocks obtained using the corresponding first list motion vector L0 and second list motion vector L1 are combined into one signal or prediction block, which can provide better adaptation to the original signal than with single prediction, resulting in less residual information and possibly more efficient compression.

In motion vector refinement, two prediction blocks obtained using the corresponding first motion vector and the second motion vector of a candidate MV pair are compared with each other based on a similarity metric for each of the candidate MV pairs for refinement. The candidate MV pair leading to the highest similarity is usually selected as refined motion vectors, denoted as MV0' and MV1', that is, the refined motion vector in the first reference picture in the list L0 and the refined motion vector in the second reference picture in the list L1, respectively. In other words, predictions are obtained according to the motion vector of the list L0 and the motion vector of the list L1 of the candidate pair of motion vectors, which are then compared with each other based on a similarity metric. The candidate motion vector pair that has the highest similarity is selected as the refined MV pair.

Typically, the output of the refinement process is refined MVs. The refined MVs may be the same as the initial MVs, or they may be different from the initial MVs. Depending on which candidate MV pair achieves the highest similarity, the candidate MV pair formed by the initial MVs is also among the candidate MV pairs. In other words, if the candidate pair of MVs that achieves the highest similarity is formed by the initial MVs, the refined MVs and the initial MVs are equal to each other.

Instead of choosing a position that maximizes the similarity metric, another way is to choose a position that minimizes the dissimilarity metric. The dissimilarity comparison measure can be SAD (Sum of Absolute Differences), MRSAD (Mean Remote Sum of Absolute Differences, SSE (Sum of Squared Errors), etc. The SAD between two prediction blocks can be obtained using candidate MV pair (CMV0, CMV1), in this case, SAD can be calculated as follows:

where nCbH and nCbW are the height and width of the prediction blocks, the function abs(a) determines the absolute value of the argument a, predSamplesL0 and predSamplesL1 are the prediction block samples obtained in accordance with the candidate MV pair, which is denoted (CMV0, CMV1).

Alternatively, the dissimilarity comparison measure can be obtained by evaluating only a subset of samples in the prediction block to reduce the amount of computation. Below is an example where sample rows are alternatively included in the SAD calculation (every other row is evaluated).

One illustrative example of motion vector refinement is explained in JVET-M1001-v3, “Versatile Video Coding (Draft 4)” of JVET (of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11) which is publicly available under http://phenix.it-sudparis.eu/jvet/. Section “8.4.3 Decoder side motion vector refinement process” in the document illustrates motion vector refinement.

To reduce internal memory requirements for refinement, in some embodiments, the motion vector refinement process may be performed independently on blocks of luminance samples obtained by dividing a sample encoding block that exceeds a certain predetermined width and/or predetermined height in the luminance samples into subblocks of samples. , which are less than or equal to a predefined width and a predefined height in luminance. The refined MV pair for each subblock in a divided encoding block may be different. External prediction is then performed for luma and chrominance for each sub-block using the refined MV pair of that sub-block.

Each MV of an initial MV pair may have fractional pixel precision. In other words, the MV indicates an offset between the current block of samples and the resampled reference region, and this offset may indicate a fractional position in the horizontal and vertical directions from an integer grid of reconstructed reference samples. Typically, a 2D integer interpolation of the reconstructed reference sample is performed to obtain sample values at the fractional position of the sample offset. The process of obtaining predicted samples from reconstructed reference pictures using a candidate MV pair can be carried out in one of the following ways:

Round the fractional part of the pair of initial MVs to the nearest integer position and obtain the integer grid values of the reconstructed reference pictures.

Perform 2-tap (e.g., bilinear) separable bilinear interpolation to obtain predicted sample values to the fractional pixel precision specified by the pair of initial MVs.

Perform divisive interpolation with a higher number of stages (e.g., 8 or 6 taps) to obtain predicted sample values to the fractional pixel precision specified by the initial MV pair.

Although candidate MV pairs can have an arbitrary subpixel offset relative to the seed MV pair, in some embodiments, for ease of finding a candidate MV pair, MVs are selected with an integer inter-pixel distance or an integer offset relative to the seed MV pair. Such candidate MV pairs with an integer pixel distance relative to the seed MV pair can also be used in the first stage of the motion vector refinement (MVR) process, the so-called integer MVR distance refinement stage. In such cases, predicted samples over all candidate MV pairs can be obtained by performing prediction on a block of samples around the seed MV pair to cover all refinement positions around the seed MV pair.

In some embodiments, after the dissimilarity cost values in all candidate MV pairs at an integer distance (integer offset) from the pair of initial MVs have been estimated, additional candidate MV pairs at a distance of subpixel offsets from the position of the best cost value can be added. This second step can be performed as a fractional distance refinement step of the MVR process. Predicted samples can be obtained for each of these positions using one of the methods described above, and the dissimilarity costs are estimated and compared with each other to obtain the position with the lowest dissimilarity cost. In some other embodiments, to avoid this computationally expensive process of predicting for each subpixel distance position around the least-overhead integer distance position, the estimated integer distance overhead values are stored and the parametric error surface is set near the best integer distance. The minimum of this error surface is then analytically calculated and used as the position with minimum dissimilarity. In such cases, the value of the dissimilarity cost is said to be derived from the calculated values of the integer distance cost.

The application of motion vector refinement for a given sample encoding block may be due to certain coding properties of the sample encoding block. Some examples of such encoding properties may include:

The distances in number of pictures (when sampling at the same frame rate) or differences in picture order counter (POC) from the current picture to the two reference pictures used for the double prediction of the sample encoding block are equal and fall on opposite sides of the current picture.

The initial dissimilarity between two predicted blocks obtained using a pair of initial MVs is less than a predefined threshold for each sample.

Bidirectional Predictive Optical Flow (BPOF) refinement

Bidirectional optical flow (BPOF) or bidirectional optical flow (BDOF) refinement is the process of improving the dual prediction accuracy of a block without explicit additional signaling in the bitstream other than the signaling for dual prediction. This is part of inter prediction block 244 in FIG. 2 and 344 in FIG. 3.

In dual prediction, 2 outer predictions are obtained according to two motion vectors, then the predictions are combined by applying weighted averaging. Combined prediction can result in reduced residual energy as quantization noise in two reference tiles or blocks is removed, thereby providing higher coding efficiency than single prediction. Weighted combination in dual prediction can be performed according to the following equation:

Double prediction=prediction1*W1+prediction2*W2+K,

where W1 and W2 are weights that may be signaled in the bit stream or may be predetermined at the encoder side and/or at the decoder side. K is an additive coefficient, which may also be signaled in the bitstream or predetermined on the encoder side and/or on the decoder side. For example, double prediction can be obtained using

Double prediction=(prediction1+prediction2)/2

where W1 and W2 are equal to 1/2, and K is equal to 0.

The purpose of optical flow refinement is to improve the accuracy of dual prediction. Optical flow is the pattern of apparent movement of objects in an image between two successive frames. Optical flow is caused by the movement of the object and/or camera. The optical flow refinement process improves the accuracy of dual prediction by applying the optical flow equation (solving the optical flow equation).

In the example, pixel I (x, y, t) is located in the first frame (x and y correspond to the spatial coordinates, t corresponds to the time dimension). The object represented by the pixel moves a distance (dx, dy) in the next frame taken after time dt. Assuming that these pixels are the same and the intensity does not change, the optical flow equation is defined as follows:

I (x, y, t)== I (x+ dx , y+ dy , t+dt

where I(x, y, t) indicates the intensity (sampling value) of the pixel in coordinates (x, y, t).

In another example, for small displacements, when the higher order terms in the Taylor series expansion are ignored, the optical flow equation can also be written as:

= 0

Where And - horizontal and vertical gradients of spatial sampling at position (x, y), the derivative with respect to time at position (x, y), and v _x =dx/dt and v _y =dy/dt represents the speed of movement.

Optical flow refinement uses the above principle to improve the quality of dual prediction.

Implementation of optical flow refinement usually includes the following steps:

1. Calculation of sampling gradients;

2. Calculate the differences between the first prediction and the second prediction;

3. Calculate the displacement (dx, dy) of pixels or a group of pixels that minimizes the error between two supporting fragments or blocks using the optical flow equation.

where I⁽⁰⁾ corresponds to the sample value in the first prediction, I⁽¹⁾ is the sample value in the second prediction, v_x and v_y - speed displacements calculated in the x and y directions, and ∂I⁽⁰⁾/ ∂x, ∂I⁽⁰⁾/ ∂y, ∂I⁽¹⁾/ ∂x and ∂I⁽¹⁾/ ∂y are the gradients in the x and y directions in the first and second predictions, respectively. And denote the time distances from the current picture in display order to the corresponding reference pictures where the first prediction and the second prediction are obtained. Some approaches minimize the sum of squared errors, while some approaches minimize the sum of absolute errors.

4. Using an implementation of the optical flow equation such as the one below:

where pred_BIO indicates a modified prediction that is the result of an optical flow refinement process.

Sample gradients can be obtained using the following formulas:

∂I(x, y, t) / ∂x = I(x + 1, y, t) - I(x - 1, y, t)

∂I(x, y, t) / ∂y = I(x, y + 1, t) - I(x, y - 1, t)

In some embodiments, to simplify the complexity of estimating motion for each pixel, motion is estimated for a group of pixels. In some examples, to calculate improved dual prediction for a 4x4 luma sample block, the offsets are estimated using the sample values of the 8x8 luma sample block with the 4x4 luma sample block at its center.

The input to the optical flow refinement process is the prediction samples from two reference pictures, and the output of the optical flow refinement is a combined prediction (predBIO), which is calculated according to the optical flow equation.

One example of optical flow refinement is explained in section 8.5.6.4 "Bidirectional optical flow prediction process" of JVET-N1001, Versatile Video Coding (Draft 5), which is publicly available under http://phenix.it-sudparis.eu/jvet/ , which is linked below for your reference.

The inputs for this process are as follows:

two variables nCbW and nCbH, defining the width and height of the current encoding block,

two (nCbW+2) x (nCbH+2) arrays of brightness prediction samples predSamplesL0 and predSamplesL1,

prediction list usage flags predFlagL0 and predFlagL1,

reference indexes refIdxL0 and refIdxL1,

flags bdofUtilizationFlag[ xIdx ][ yIdx ] for using bidirectional optical flow with xIdx = 0..( nCbW >> 2 ) − 1, yIdx = 0..( nCbH >> 2 ) − 1.

The output of this process is pbSamples of an array of (nCbW) x (nCbH) brightness prediction sample values.

The bitDepth, shift1, shift2, shift3, shift4, offset4 and mvRefineThres variables are output as follows:

The bitDepth variable is set equal to BitDepthY, then is the bit depth of the luma component.

The shift1 variable is set to Max(6, bitDepth - 6).

The shift2 variable is set to Max(4, bitDepth - 8).

The shift3 variable is set to Max(1, bitDepth - 11).

The shift4 variable is set to Max(3, 15 - bitDepth) and the offset4 variable is set to 1  << (shift4 − 1).

– The mvRefineThres variable is set to 1 << Max(5, bitDepth − 7).

For xIdx = 0..( nCbW >> 2 ) − 1 and yIdx = 0..( nCbH >> 2 ) − 1 the following applies:

The variable xSb is set to (( xIdx << 2) + 1 and ySb is set to ( yIdx << 2) + 1.

If bdofUtilizationFlag [xIdx] [yIdx] is FALSE, for x=xSb - 1..xSb+2, y=ySb - 1 .. ySb+2, the prediction sample values for the current subblock are output as follows:

pbSamples[ x ][ y ] = Clip3( 0, ( 2 ^bitDepth ) − 1, ( predSamplesL0[ x + 1 ][ y + 1 ] + offset4 +
predSamplesL1[ x + 1 ][ y + 1 ] ) >> shift4 )

Otherwise (bdofUtilizationFlag [xIdx] [yIdx] is TRUE), the prediction sample values of the current subblock are output as follows:

For x=xSb - 1..xSb+4, y=ySb - 1..ySb+4 the following ordered steps apply:

1. Positions (h_x, v_y ) for each from the corresponding sample positions (x, y) within the sample arrays, predictions are derived as follows:

hx=Clip3 (1, nCbW, x)

vy=Clip3 (1, nCbH, y)

2. The variables gradientHL0 [x] [y], gradientVL0 [x] [y], gradientHL1 [x] [y] and gradientVL1 [x] [y] are output as follows:

gradientHL0[ x ][ y ] = (predSamplesL0[ h _x + 1 ][v _y ] − predSampleL0[ h _x − 1 ][ v _y ] ) >> shift1

gradientVL0[ x ][ y ] = (predSampleL0[ h _x ][ v _y + 1 ] − predSampleL0[ h _x ][v _y − 1 ] ) >> shift1

gradientHL1[ x ][ y ] = (predSamplesL1[ h _x + 1 ][v _y ] − predSampleL1[ h _x − 1 ][ v _y ] ) >> shift1

gradientVL1[ x ][ y ] = (predSampleL1[ h _x ][ v _y + 1 ] − predSampleL1[ h _x ][v _y − 1 ] ) >> shift1

3. The variables diff [x] [y], tempH [x] [y] and tempV [x] [y] are output as follows:

diff[ x ][ y ] = (predSamplesL0[ h _x ][ v _y ] >> shift2 ) − ( predSamplesL1[ h _x ][ v _y ] >> shift2 )

tempH[ x ][ y ] = (gradientHL0[ x ][ y ] + gradientHL1[ x ][ y ]) >> shift3

tempV[ x ][ y ] = (gradientVL0[ x ][ y ] + gradientVL1[ x ][ y ]) >> shift3

The variables sGx2, sGy2, sGxGy, sGxdI and sGydI are output as follows:

sGx2 = Σ _i Σ _j ( tempH[ xSb + i ][ ySb + j ] * tempH[ xSb + i ][ ySb + j ] ) with i, j = −1..4

sGy2 = Σ _i Σ _j ( tempV[ xSb + i ][ ySb + j ] * tempV[ xSb + i ][ ySb + j ] ) with i, j = −1..4

sGxGy = Σ _i Σ _j ( tempH[ xSb + i ][ ySb + j ] * tempV[ xSb + i ][ ySb + j ] ) with i, j −1..4

sGxGy _m = sGxGy >> 12

sGxGy _s = sGxGy & ( ( 1 << 12 ) − 1 )

sGxdI = Σ _i Σ _j ( −tempH[ xSb + i ][ ySb + j ] * diff[ xSb + i ][ ySb + j ] ) with i, j = −1..4

sGydI = Σ _i Σ _j ( −tempV[ xSb + i ][ ySb + j ] * diff[ xSb + i ][ ySb + j ] ) with i, j = −1..4

The horizontal and vertical offsets of the current subblock are output as:

v _x = sGx2 > 0 ? Clip3( −mvRefineThres, mvRefineThres, −( sGxdI << 3 ) >> Floor( Log2( sGx2 ) ) : 0

v _y = sGy2 > 0 ? Clip3( −mvRefineThres, mvRefineThres, ( ( sGydI << 3 ) –

( ( v _x * sGxGy _m ) << 12 + v _x * sGxGy _s ) >> 1 ) >> Floor( Log2( sGx2 ) ) : 0

For x=xSb - 1..xSb+2, y=ySb - 1..ySb+2, the prediction sample values of the current subblock are output as follows:

bdofOffset = Round( ( v _x * ( gradientHL1[ x + 1 ][ y + 1 ] − gradientHL0[ x + 1 ][ y + 1 ] ) ) >> 1 )

+ Round( ( v _y * (gradientVL1[ x + 1 ][ y + 1 ] − gradientVL0[ x + 1 ][ y + 1 ] ) ) >> 1 )

[The Round() operation is defined for floating point input. The Round() operation may be omitted here.]

pbSamples[ x ][ y ] = Clip3( 0, ( 2 ^bitDepth ) − 1, ( predSamplesL0[ x + 1 ][ y + 1 ] + offset4 + predSamplesL1[ x + 1 ][ y + 1 ] + bdofOffset ) >> shift4)

The terms “optical flow refinement,” “dual prediction optical flow refinement,” and “bidirectional optical flow refinement” are used interchangeably in the disclosure because the terms are equivalent in nature.

Technical problems addressed in the disclosure

In the version of the currently accepted BPOF, the encoding block is divided into subblocks to reduce the memory footprint of BPOF and allow the installation of a subblock level pipeline in hardware designs. Any coding unit with a width greater than 16 is split horizontally with a subblock width of 16. Any coding unit with a height greater than 16 is split vertically with a subblock height of 16.

Sample gradients are required for all sample positions in a 4x4 block (as described in the prior art section regarding BPOF) to calculate optical flow and to apply correction based on the calculated optical flow. To obtain 4x4 sampling gradients, extended sampling values are required for both, with reference points extending beyond the 4x4 sampling values. When using the [-1 0 1] filter, computing gradients at each reference point requires one oversampled value across the entire 4x4 block. In other words, 4x4 sample gradients require a 6x6 sample area (except for the four corner samples of the 6x6 region).

At the subblock level, a subblock of size M x N requires (M+2) x (N+2) sample values to compute sampling gradients for M x N sample positions. To keep complexity low, the version of currently accepted BPOF has extended sampling values (t e. sample positions that extend beyond the M x N sample positions corresponding to the subblock) are not obtained by motion-compensated interpolation of the corresponding reference samples. Instead, the extended sample position is obtained with subpixel precision in each reference picture, and fractional parts in the horizontal and vertical directions are ignored to obtain the corresponding integer grid value of the reconstructed reference sample position, at each reference point.

These extended sample values are used to calculate the boundary sample gradients at each reference point. Since the optical flow calculation for 4x4 uses the sample gradient values and the sample position sample values of 6x6 centered on 4x4, the position gradient value of the extended sample is set equal to the sample gradient value of the nearest sub-block boundary sample. After the extended sample values obtained from the reconstructed integer grid reference sample position values have been used to calculate the boundary gradient, the sample value of some extended sample is set equal to the sample value of the nearest subblock boundary sample.

This aspect of removing the fractional part in the horizontal and vertical directions of the sample position to obtain the value of the reference sample reconstructed from the integer grid for the extended sample introduces a small compression loss compared to performing 2D separable K-tap interpolation with the filled samples taking the closest value boundary sampling as shown in JVET-M0487.

Suggested method

This disclosure provides a method that retains the low complexity aspect of obtaining an extended sample value while substantially eliminating coding losses caused by removing fractional horizontal and vertical portions.

Specifically, rather than ignoring the fractional horizontal and vertical offsets of the extended sample position in each reconstructed reference sample grid, the fractional offsets are used to perform a rounding operation in both the horizontal and vertical directions to identify the nearest integer sample position in the corresponding integer grid of the reconstructed reference frame. . The sample value at such identified position is then assigned to the extended sample.

In general, the proposed bidirectional optical flow prediction method based on a sample value from a sampling position belonging to the extended area of the predicted block used in bidirectional optical flow prediction of the current block can be implemented in the following steps, as shown in Fig. 8:

Step 1310, obtaining a motion vector (MV) of the current block;

Step 1320, obtaining the position of the subpixel sample in the expanded region of the predicted block in the reference picture based on the MV;

Here, the expanded area of the predicted block in the reference picture is based on the MV. In particular, using the MV, a prediction block at an offset specified by the MV as a reference to a co-located block corresponding to the current block can be found. An extended region is an area that includes at least one row/column of pixels adjacent to and surrounding/encompassing the prediction block. The extended region can be thought of as a padding region that surrounds a prediction block in a reference picture and is used, for example, to apply K-tap interpolation filters and/or calculate 3-tap sampling gradients at a block boundary.

Step 1330, processing the obtained sub-pixel sampling position to obtain an integer pixel sampling position in the reference picture; And

Step 1340, performing a bidirectional optical flow prediction by referring to a reference sample value at the obtained integer sample position of pixels in the reference picture.

The following embodiments illustrate how this method can be implemented.

Option 1:

Let the top left sample of the current subblock of size M x N samples for which dual prediction with BPOF/BDOF is performed be located at (sb_start_x, sb_start_y). Let (mvx0, mvy0) be the MV with accurate to 1/16 pixel for the current subblock relative to the first reference frame L0, and (mvx1, mvxy1) will be MV with accuracy up to 1/16 pixel for the current subblock relative to the second reference frame L1.

Position with precision to 1/16 pixel in L0, corresponding to the top left position of the current subblock in L0, is given by ((sb_start_x << 4) + mvx0, (sb_start_y << 4) + mvy0). Position with precision to 1/16 pixel in L1, corresponding to the top left position of the current subblock in L0, is given as ((sb_start_x << 4) + mvx1, (sb_start_y << 4) + mvy1). It will be appreciated that various subpixel accuracies may be used, such as half pixel, quarter pixel or 1/8 pixel, with the top left position of the current subblocks determined by applying an appropriate left shift of 1, 2, or 3 respectively.

For M x N positions in reference frames L0 and L1 corresponding to a subblock, 2D separable motion-compensated K-tap interpolation is performed on the corresponding reference frame based on the subpixel precision, e.g. 1/16, corresponding sample position and used as predicted sample values that are used in the BDOF process to calculate sample gradients.

To obtain the predicted sample values for the remaining extended corresponding sample positions that extend beyond the central M x N sample positions in a block of (M+2) x (N+2) sample positions for the purpose of calculating the boundary sample gradient of the M x N subblock, the following is performed procedure (for overall subpixel sampling accuracy):

Let x_spel_pos be the horizontal fine subpixel sampling position and y_spel_pos be the vertical fine subpixel sampling position corresponding to the extended sampling position. The sample value for this extended sample position is obtained as the reconstructed value of the reference sample at position (x_pos, y_pos), calculated as follows:

x_pos = (x_spel_pos + (1 << (shift_val – 1))) >> shift_val

y_pos = (y_spel_pos + (1 << (shift_val – 1))) >> shift_val

where shift_val is chosen according to the subpixel sampling precision used. For subpixel sampling precision of half pixel, quarter pixel, 1/8 pixel and 1/16 pixels, shift_val are 1, 2, 3 and 4 respectively.

Through this process of rounding the subpixel position in the horizontal and vertical directions, the boundary gradients become more accurate, as can be seen in Figures 6 and 7.

FIG. 6 shows from which position in the reference picture the pixel value is selected. In the diagram (A) showing the prior art, as an MV indicating a fractional position in reference pictures, the fractional part is discarded and only the integer part is used to select a pixel value. For example, for MV(4, 0), MV(9, 0), which are in Q4 format, the position (0, 0) in the reference picture is used to select the pixel value (reference sample). In contrast, in diagram (B), according to the present disclosure, the fractional part is considered for rounding to the nearest integer, which is ultimately used to select the pixel value. For example, for MV (4, 0), MV (9, 0), which are in Q4 format, position (0, 0) and position (16, 0), respectively, in the reference picture are used to select the corresponding value of the pixels that can significantly improve the accuracy of reference sample selection.

In fig. 7B illustrates a described method for determining the position of an integer grid reference sample based on the position of a subpixel precision (subpixel accurate) sample position (i.e., 1/16 pixel). It can be seen that in the horizontal and vertical directions, the nearest integer sample position is selected based on (half) rounding of the horizontal and vertical components of the position. Thus, subpixel precision offsets from (i - 8/16) to (i+7/16) in the horizontal direction and from (j - 8/16) to (j+7/16) in the vertical direction use ( i, j ) as the nearest integer sample position. Compared to this Fig. 7A illustrates a problem with selecting an integer sample position via clipping, where the exact subpixel offset from i to (i+15/16) in the horizontal direction and j to (j+15/16) in the vertical direction uses (i, j) as the integer sample position .

At the left boundary sampling position of an M x N block, the sampling gradient of the horizontal boundary is calculated by taking the difference between the sampling value of the position on the right (which is calculated using 2-dimensional shared K-tap interpolation) and the sampling value of the extended sampling position to the left of it (which is assigned the rounded nearest the reconstructed value of the reference sample as described above). Similarly, at the position of the right sample boundary of an M x N block, the horizontal sampling gradient is calculated by taking the difference between the sample value of the position to the right (which is assigned the rounded nearest reconstructed value of the reference sample, as described above) and the approximate value of the position to the left of it (which is calculated with using 2D separable K-tap interpolation).

At the top boundary sampling position of an M x N block, the vertical boundary sampling gradient is calculated by taking the difference between the sampling value of the position below it (which is calculated using 2D separable K-tap interpolation) and the sampling value of the extended sampling position above (which is assigned the rounded nearest reconstructed reference sample value as described above). Similarly, at the bottom boundary sampling position of an M x N block, the vertical boundary sampling gradient is calculated by taking the difference between the sample value of the position below (which is assigned the rounded nearest reconstructed value of the reference sample as described above) and the approximate value of the position above (which is calculated using 2 -dimensional separable interpolation of K-taps).

Option 2 implementation:

In this embodiment, the extended sample position values are output using the procedure described in Embodiment 1 whenever BPOF/BDOF is applied and when motion vector refinement (MVR) is not applied for a given coding block to which the current subblock belongs, or when MVR and BDOF are both applied for a given encoding block to which the current subblock belongs, but MVR does not result in an integer offset in the integer distance refinement step of the MVR relative to the initial MV pair. In other words, when both MVR and BDOF are applied, when the MVR integer distance refinement step results in an initial MV pair, that is, with zero offset, the extended sample position values are output using the procedure described in Embodiment 1.

When MVR and BDOF are both applied for a given encoding block to which the current subblock belongs, but MVR results in an integer offset in the integer distance refinement step of the MVR relative to the initial MV pair, i.e., the refined MV pair after the integer distance refinement step is different from the initial MV pair , the sample value for the extended sample position (x_spel_pos, y_spel_pos) is obtained as the reconstructed value of the reference sample at position (x_pos, y_pos), calculated as follows:

x_pos = x_spel_pos >> shift_val

y_pos = y_spel_pos >> shift_val

where shift_val is chosen according to the subpixel sampling precision used. For subpixel sampling precision of half pixel, quarter pixel, 1/8 pixel and 1/16 pixels used shift_val 1, 2, 3 and 4 respectively. In other words, the fractional part of the position in both the horizontal and vertical directions is ignored when obtaining the sample position for this case.

Advantages of the proposed method:

By using the described methods of rounding the subpixel position in the horizontal and vertical directions, the boundary gradients become more accurate. This improves the calculated optical flow based on these sampling gradients, as well as the corrections applied using the calculated optical flow. This provides a consistent coding gain over the earlier method of ignoring fractional offsets in the horizontal and vertical directions when selecting an integer sample value for extended sample positions. At the same time, the difficulty doesn't change much, since adding horizontal and vertical rounding offsets is the only increase in difficulty.

Based on the solution, a possible combination of a standard VVC 5 project can be shown as follows.

Integer luminance sampling process (Luma)

The inputs for this process are as follows:

– location of brightness in full sample blocks ( xInt _L , yInt _L ),

brightness location in fractional sampling blocks (( xFrac _L , yFrac _L ),

array of reference brightness samples refPicLX _L ,

The output of this process is the predicted luminance sample value predSampleLX _L .

The variable shift is set to Max (2, 14 - BitDepth _Y ).

The picW variable is set to pic_width_in_luma_samples and the picH variable is set to pic_height_in_luma_samples.

Luminance positions in full sample units (xInt, yInt) are output as follows:

xOffset = xFrac _L > 7 ? 10

yOffset = yFrac _L > 7 ? 10

xInt = Clip3( 0, picW − 1, sps_ref_wraparound_enabled_flag ?
ClipH( ( sps_ref_wraparound_offset_minus1 + 1 ) * MinCbSizeY, picW, xInt_L − 1 + xOffset) : xInt_L − 1 + xOffset)

yInt = Clip3( 0, picH − 1, yInt _L − 1 + yOffset)

The predicted luminance sample value predSampleLX _L is output as follows:

predSampleLX _L = refPicLX _L [ xInt ][ yInt ] << shift3

It should be noted that these example changes in the VVC design are made only for the luma component, but they can be used for the chrominance component in the same way or with some modifications to adopt the characteristic of the chrominance component.

Fig. 8 shows a flowchart of a video encoding/decoding method according to an embodiment of the disclosure. The illustrated method performs dual optical flow prediction based on a sample value from a sample position belonging to an extended region of a predicted block used in bidirectional optical flow prediction of the current block as follows: At step 1310, a motion vector (MV) of the current block is obtained. At step 1320, the position of the subpixel sample in the expanded region of the predicted block in the reference picture is obtained based on the MV. At step 1330, the resulting subpixel sample position is processed, specifically rounded, to obtain an integer pixel sample position in the reference picture. At step 1340, a dual optical flow prediction is finally performed by referring to the reference sample value at the obtained integer sample position of the pixels in the reference picture.

Fig. 9 is a block diagram illustrating an example of encoding/decoding equipment according to an embodiment of this disclosure. The encoding/decoding apparatus 20/30 includes a prediction unit 1440 configured to perform bidirectional optical flow prediction of the current block, a first acquiring unit 1410 configured to obtain the motion vector (MV) of the current block, a second acquiring unit 1420 configured to obtain the subpixel sampling position in extended area of the predicted block in the MV-based reference picture, and

a processing unit 1430 configured to process, in a particular round, the obtained sub-pixel sampling position to obtain an integer pixel sampling position in the reference picture, wherein the prediction unit 1440 is configured to perform bidirectional optical flow prediction by referring to the reference sample value at the obtained integer pixel sampling position in the reference picture.

Acquisition units 1410 and 1420, processing unit 1430, and prediction unit 1440 may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on a computer-readable medium or transmitted through a communication medium in the form of one or more instructions or code and executed by a hardware processing unit. The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term “processor” as used herein can refer to any of the above structure or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within specialized hardware and/or software modules configured for encoding and decoding or included in a bundled codec. In addition, the techniques may be fully implemented in one or more circuits or logic elements.

Some specific embodiments are set forth in the following aspects of the disclosure.

According to a first aspect, there is provided a method used for obtaining a predicted sample value for a sample position belonging to an extended area of a predicted block used in bidirectional optical flow prediction of a current encoding block, which comprises obtaining an MV of the current encoding block, obtaining a subpixel position in a reference picture based on the MV, and an extended sample position in the current picture that surrounds the sample positions of the current encoding block, obtaining an integer pixel position in the reference picture by rounding/moving the resulting sub-pixel sample position to the nearest integer pixel position, and setting a predicted sample value for the sample position belonging to the extended region of the predicted block , as the reference sample value at the resulting integer pixel position.

Rounding/moving the resulting subpixel sample position toward the nearest integer pixel position may include rounding the resulting subpixel sample position separately in the horizontal and vertical directions to the nearest integer sample position in that direction, with the half sample position between the two integer sample positions rounded up to a higher integer sample position in that direction.

Rounding/moving the received subpixel sample position towards the nearest integer pixel position may involve comparing the obtained subpixel sample position separately in the horizontal and vertical directions with a threshold value, in the case where the subpixel sample position is equal to or greater than the threshold value, the nearest higher integer position sample position can be obtained as an integer pixel position, and in the case where the subpixel sample position is less than a threshold value, the nearest lower integer sample position can be obtained as an integer pixel position.

According to a second aspect, there is provided a method used for obtaining a predicted sample value for a sample position belonging to an extended area of a prediction block used in bidirectional optical flow prediction of a current encoding block, which comprises obtaining a first MV corresponding to a predicted sample in an extended area of the encoding block, obtaining a second MV by discarding the fractional part of the first MV, obtaining the position of the first integer pixel of the extended region in the reference picture based on the second MV, determining the fractional offset vector based on the first MV, obtaining the position of the second integer pixel in the reference picture based on the position of the first integer pixel and the fractional offset vector, wherein the position of the second integer pixel is set equal to the position of the first integer pixel plus a position offset, wherein the position offset is determined by comparing each component of the fractional offset vector and the threshold value, and further comprises padding the predicted sample using the pixel value at the position of the second integer pixel in the reference picture.

Whether the position offset component is 1 or 0 may depend on the result of a comparison between the corresponding fractional offset vector component and the threshold value, where the component includes an X component and a Y component.

The threshold value may be K, with the position offset component set to 1 when the corresponding fractional offset component is equal to or greater than K, otherwise set to 0. K may be 8.

According to a further aspect, an encoder or decoder is provided including processing circuitry for performing any of the methods according to the first and second aspects described above.

According to a further aspect, a computer program product is provided comprising program code for performing any of the methods according to the first and second aspects described above.

According to a further aspect, a decoder or encoder is provided, comprising one or more processors and a non-volatile computer-readable storage medium coupled to the processors and storing a program for execution by the processors, wherein the program, when executed by the processors, configures the decoder to perform any of the methods according to the first and second aspects described above.

Mathematical operators

The mathematical operators used in this application are similar to those used in the C programming language. However, the results of integer division and arithmetic shift operations are defined more precisely, and additional operations such as exponentiation and real-valued division are also defined. Numbering and counting rules usually start from 0, i.e. "first" is equivalent to 0, "second" is equivalent to 1, etc.

Arithmetic operators

The following arithmetic operators are defined as follows:

+ Addition − Subtraction (as a two-argument operator) or negation (as a unary prefix operator) * Multiplication, including matrix multiplication x ^y Exponentiation. Defines x to the power of y. In other contexts, this notation is used as a superscript that should not be interpreted as exponentiation. / Integer division with the result truncated towards zero. For example, 7/4
and -7/-4 are truncated to 1, and -7/4 and 7/-4 are truncated to -1. ÷ Used to indicate division in mathematical equations where truncation or rounding is not intended. Used to indicate division in mathematical equations where truncation or rounding is not intended. Summation f( i), where i takes all integer values from x to y inclusive. x % y Modulo operation. The remainder of x divided by y is defined only for integers x and y
with x >= 0 and y > 0.

Logical operators

The following logical operators are defined as follows:

x && y

x || y Boolean logical "or" for x and y

! Boolean logical "not"

x?y:z If x is TRUE or not 0, the value of y is calculated; otherwise, the z value is calculated.

Relational operators

The following relational operators are defined as follows:

> More

>= Greater than or equal to

< Less

<= Less than or equal to

== Equal

!= Not equal

When a comparison operator is applied to a syntax element or variable that is assigned the value "na" (not applicable), the value "na" is treated as a separate value for the syntax element or variable. The value "na" is not considered equal to any other value.

Bit operators

The following bitwise operators are defined as follows:

& bit "and". When operating on integer arguments, operates on the two's complement representation of an integer value. When working on a binary argument that contains fewer bits than another argument, the shorter argument is expanded by adding more significant bits equal to 0.

| bit or. When operating on integer arguments, operates on the two's complement representation of an integer value. When working on a binary argument that contains fewer bits than another argument, the shorter argument is expanded by adding more significant bits equal to 0.

^ exclusive or. When operating on integer arguments, operates on the two's complement representation of an integer value. When working on a binary argument that contains fewer bits than another argument, the shorter argument is expanded by adding more significant bits equal to 0.

x >> y Arithmetic right shift of the two's complement integer representation of x by y binary digits. This function is defined only for non-negative integer values of y. The bits shifted to the most significant bits (MSB) by a right shift have a value equal to the MSB of x before the shift operation.

x << y Arithmetic left shift of the two's complement integer representation of x by y binary digits. This function is defined only for non-negative integer values of y. Bits shifted to the least significant bits (LSB) by left shifting have a value of 0.

Assignment Operators

The following arithmetic operators are defined as follows:

= assignment operator

+ + Increment, i.e. x+ + is equivalent to x = x+ 1; when used in an array index, evaluates to the value of the variable before the increment operation.

− − Decrease, i.e. x− − is equivalent to x = x - 1; when used in an array index, evaluates the value of the variable before the decrement operation.

+=Increment by the specified amount, i.e. x += 3 is equivalent to x=x+3, and x += (−3) is equivalent to x=x + (−3).

−=Decrease by the specified amount, i.e. x −= 3 is equivalent to x=x − 3, and x −= (−3) is equivalent to x=x − (−3).

Range designation

The following notation is used to indicate a range of values:

x=y..z x takes integer values from y to z, inclusive, where x, y and z are integers and z is greater than y.

Mathematical functions

The following mathematical functions are defined:

Abs(x)=

Asin(x) is a trigonometric inverse sine function operating on an argument x that ranges from −1.0 to 1.0 inclusive, with an output value ranging from −π ÷ 2 to π ÷ 2 inclusive, in units of radians.

Atan( x ) is a trigonometric arctangent function operating on an argument x, with an output value ranging from −π÷2 to π÷2 inclusive, in units of radians.

Atan2( y, x )=

Ceil( x ) smallest integer greater than or equal to x

Clip1Y( x )=Clip3( 0, ( 1 << BitDepthY ) − 1, x )

Clip1C( x )=Clip3( 0, ( 1 << BitDepthC ) − 1, x )

Clip3( x, y, z )=

Cos( x ) is a trigonometric cosine function that operates on x in units of radians.

Floor( x ) is the largest integer less than or equal to x.

GetCurrMsb( a, b, c, d )=

Ln( x ) natural logarithm of x (logarithm to base e, where e is the base constant of the natural logarithm 2.718 281 828...).

Log2( x ) logarithm of x to base 2.

Log10( x ) is the logarithm of x to base 10.

Min( x, y )=

Max( x, y )=

Round( x )=Sign( x ) * Floor( Abs( x ) + 0.5)

Sign( x )=

Sin( x ) is a trigonometric sine function that operates on x in units of radians.

Sqrt( x )=

Swap( x, y )=( y, x )

Tan( x ) is a trigonometric tangent function that operates on x in units of radians.

Order of priority of operations

When the order of precedence in an expression is not explicitly specified using parentheses, the following rules apply:

Operations with higher priority are evaluated before any operation with lower priority.

Operations with the same priority are evaluated sequentially from left to right.

The table below lists the priority of operations from highest to lowest; a higher position in the table indicates higher priority.

For those operators that are also used in the C programming language, the order of precedence used in this description is the same as in the C programming language.

Table: Operation priority from highest (at the top of the table) to lowest (at the bottom of the table)

operations (with operands x, y and z) "x++", "x− −" "!x", "−x" (as unary prefix operator) x ^y "x * y", "x/y", "x ÷ y", " ", "x % y" "x+y", "x − y" (as an operator with two arguments), " " "x << y", "x >> y" "x < y", "x <= y", "x > y", "x >= y" "x==y", "x != y" "x&y" "x | y" "x && y" "x | | y" "x ? y : z" "x..y" "x=y", "x += y", "x −= y"

Text description of logical operations

The text contains a definition of logical operations, which would be described mathematically in the following form:

if ( condition 0 ) definition 0

else if( condition 1)

definition 1...

else /* informative comment on the remaining condition */ definition n

can be described as follows:

... as below / ... the following applies:

- if condition 0, definition 0

- Otherwise, if condition 1, definition 1

- ...

- Otherwise (informative comment on the remaining condition), definition n.

Each "if...else, if...else,..." definition in the text is introduced with the words "...as below" or "...the following applies" followed immediately by "if..." . The final condition "if...else, if...else..." can always be "otherwise..." Alternating definitions of "if...else if...else,..." can be identified by comparison "...as stated below" or "...the following applies" followed by a final "otherwise,...".

The text contains a definition of logical operations, which would be described mathematically in the following form:

if( condition 0a && condition 0b ) definition 0

else if ( condition 1a | | condition 1b ) definition 1 ... else definition n

can be described as follows:

... as below / ... the following applies:

- If all of the following conditions are true, definition 0:

- condition 0a

- condition 0b

- Else, if one or more of the following conditions are true, Definition 1:

- condition 1a

- condition 1b

- ...

- Otherwise, the definition of n

The text contains a definition of logical operations, which would be described mathematically in the following form:

if (condition 0) definition 0 if (condition 1) definition 1

can be described as follows:

When condition is 0, definition is 0

When condition 1, definition 1

Although embodiments of the disclosure have been primarily described in terms of video encoding, it should be noted that embodiments of encoding system 10, encoder 20, and decoder 30 (and thus system 10) and other embodiments described herein may also be configured for processing or encoding a picture, that is, processing or encoding a single picture independently of any previous or subsequent picture, as in video encoding. In general, only the inter prediction (encoder) and 344 (decoder) blocks 244 may not be available if the encoding for picture processing is limited to a single picture 17. All other functions (also referred to as tools or techniques) of video encoder 20 and video decoder 30 may equally used for processing static images, for example, residual calculation 204/304, transform 206, quantization 208, inverse quantization 210/310, (inverse) transform 212/312, division 262, intra prediction 254/354 and/or loop filtering 220 ,320, entropy encoding 270 and entropy decoding 304.

Embodiments of, for example, encoder 20 and decoder 30, as well as the functions described herein, for example, with respect to encoder 20 and decoder 30 may be implemented by hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on a computer-readable medium or transmitted through a communication medium in the form of one or more instructions or code and executed by a hardware processing unit. Computer-readable media may include computer-readable storage media that correspond to tangible media, such as data storage media, or communication media, including any media that facilitates the transfer of a computer program from one place to another, for example, according to a communications protocol. Thus, computer-readable media may generally correspond to (1) tangible computer-readable storage media that are non-volatile, or (2) communication media such as a signal or carrier wave. Storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementing the techniques described in this disclosure. The computer program product may include computer-readable media.

By way of example, and not limitation, such computer-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other media that can used to store desired program code in the form of instructions or data structures and can be accessed by a computer. Additionally, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave communications, then coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave communications are included in the definition of media. However, it should be understood that computer readable and storage media do not include connections, carrier waves, signals or other transient media, but are instead directed to non-transitory, tangible storage media. Disc and disc used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where discs typically reproduce data magnetically and discs reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer readable media.

The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term “processor” as used herein can refer to any of the above structure or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within specialized hardware and/or software modules configured for encoding and decoding or included in a bundled codec. In addition, the techniques may be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide variety of devices or hardware components, including a wireless telephone, an integrated circuit (IC), or a set of ICs (eg, a chipset). This disclosure describes various components, modules, or blocks to highlight functional aspects of devices configured to implement the disclosed technologies, but not necessarily required to be implemented by different hardware units. Rather, as described above, the various blocks may be combined into a codec hardware block or provided by a collection of interacting hardware blocks, including one or more processors as described above, together with suitable software and/or firmware.

Claims (79)

1. A method for predicting bidirectional optical flow based on a sample value from a sample position belonging to an extended area of a predicted block used in bidirectional optical flow prediction of a current block, comprising: obtaining the motion vector (MV) of the current block; obtaining a sample position of subpixels in the expanded region of the predicted block in the reference picture based on said MV; processing the obtained sub-pixel sampling position to obtain an integer pixel sampling position in the reference picture; And performing a bidirectional optical flow prediction by referring to a reference sample value at the resulting integer sample position of pixels in the reference picture. 2. The method of claim 1, wherein the reference sample value at the obtained integer pixel sampling position in the reference picture is referred to as the sampling value from the sub-pixel sampling position in bidirectional optical flow prediction. 3. The method of claim 1 or 2, wherein processing the obtained subpixel sampling position includes rounding the obtained subpixel sampling position to obtain an integer pixel sampling position in the reference picture. 4. The method as claimed in any one of the preceding claims, wherein processing the obtained subpixel sample position includes: rounding the obtained sub-pixel sampling position separately in the horizontal direction and vertical direction to the corresponding nearest integer pixel sampling position in the corresponding direction, and obtaining an integer pixel sampling position in the reference picture based on the nearest integer pixel sampling positions in the horizontal direction and vertical direction. 5. The method according to any of the preceding paragraphs, characterized in that the method additionally contains: before performing bidirectional optical flow prediction, selecting a reference sample value at the obtained integer sample position of pixels in the reference picture as a prediction sample value for use in bidirectional optical flow prediction, wherein performing a bidirectional optical flow prediction with reference to the reference sample value comprises: performing bidirectional optical flow prediction using the prediction sample value. 6. The method according to claim 4 or 5, wherein rounding the obtained sub-pixel sampling position separately in the horizontal direction and the vertical direction includes: comparison of the fractional part of the resulting subpixel sample position in the corresponding direction with a threshold value, wherein, in the case where the fractional part is equal to or greater than the threshold value, the nearest upper integer pixel sampling position in the corresponding direction is obtained as the nearest integer pixel sampling position in the corresponding direction, and wherein, in the case where the fractional part is less than the threshold value, the nearest lower integer pixel sampling position in the corresponding direction is obtained as the nearest integer pixel sampling position in the corresponding direction. 7. The method of claim 6, wherein the nearest upper integer pixel sample position in the corresponding direction is obtained by adding an offset of 1 to the integer position of the truncated pixel sample in the corresponding direction obtained by discarding the fractional part of the resulting subpixel sample position; And wherein the nearest lower integer pixel sample position in the corresponding direction is obtained by adding an offset of 0 to the integer position of the truncated pixel sample in the corresponding direction obtained by discarding the fractional part of the resulting subpixel sample position. 8. The method of claim 6 or 7, wherein the threshold value is set to P/2 for sub-pixel precision 1/P pixel. 9. The method of claim 8, wherein the threshold value is 8 for a subpixel precision of 1/16 pixel. 10. The method of claim 4, wherein rounding the obtained subpixel sampling position separately in the horizontal direction and the vertical direction includes applying a shift operation to the components of the resulting subpixel sample position in appropriate directions. 11. The method of claim 10, wherein the integer position (x_pos, y_pos) of the sample pixels in the reference picture is obtained in accordance with the following equations: x_pos = (x_spel_pos + (1 << (shift_val – 1))) >> shift_val, y_pos = (y_spel_pos + (1 << (shift_val – 1))) >> shift_val, where (x_spel_pos, y_spel_pos) is the resulting subpixel sampling position and shift_val is chosen according to the subpixel sampling precision used. 12. The method of claim 11, wherein for subpixel sampling precision equal to half a pixel, quarter pixel, 1/8 pixel, or 1/16 pixel, a shift_val of 1, 2, 3, or 4 is used, respectively. 13. The method as claimed in one of the preceding claims, wherein the MV of the current block is a refined MV obtained by motion vector refinement (MVR) based on the initial MV. 14. The method according to claim 13, in which the MVR comprises the step of refining the integer distance with respect to the initial MV; And Moreover, in the event that the integer MVR distance refinement step results in a non-zero offset of the refined MV relative to the initial MV, the integer position of the pixel sample in the reference picture is obtained by rounding the resulting subpixel sample position. 15. The method of claim 14, wherein if the integer MVR distance refinement step results in a zero offset of the refined MV relative to the initial MV, the integer pixel sample position in the reference picture is obtained by discarding the fractional part of the resulting subpixel sample position. 16. The method of claim 14, wherein if the integer MVR distance refinement step results in a zero offset of the refined MV relative to the initial MV, the integer position (x_pos, y_pos) of the sample pixels in the reference picture is obtained according to the following equations: x_pos = x_spel_pos >> shift_val, y_pos = y_spel_pos >> shift_val, where (x_spel_pos, y_spel_pos) is the resulting subpixel sampling position and shift_val is chosen according to the subpixel sampling precision used. 17. The method of claim 16, wherein for subpixel sampling precision equal to half a pixel, quarter pixel, 1/8 pixel, or 1/16 pixel, a shift_val of 1, 2, 3, or 4 is used, respectively. 18. The method as claimed in any one of the preceding claims, wherein the current block is a current encoding block, or a current sub-encoding block, or a current prediction block. 19. The method as claimed in any one of the preceding claims, wherein the reference sample value at the obtained integer sample position of pixels in the reference picture is used in calculating a boundary sample gradient corresponding to the boundary of a predicted block in the reference picture. 20. An encoder (20) containing a processing circuit for performing the method according to any one of claims. 1-19. 21. A decoder (30) comprising a processing circuit for performing the method according to any one of claims. 1-19. 22. Encoder (20), containing: one or more processors; And a non-volatile computer-readable storage medium coupled to one or more processors and storing instructions for execution by the one or more processors, wherein the instructions, when executed by the one or more processors, configure an encoder to perform the method of any one of claims. 1-19. 23. Decoder (30), containing: one or more processors; And a non-volatile computer-readable storage medium coupled to one or more processors and storing instructions for execution by the one or more processors, wherein the instructions, when executed by the one or more processors, configure a decoder to perform the method of any one of claims. 1-19. 24. Encoder (20), containing: a prediction unit configured to perform bidirectional optical flow prediction of the current block; a first receiving unit configured to obtain a motion vector (MV) of the current block; a second acquiring unit configured to obtain a sample position of subpixels in an expanded region of a predicted block in a reference picture based on said MV; And a processing unit configured to process the obtained sub-pixel sampling position to obtain an integer pixel sampling position in the reference picture; wherein the prediction unit is configured to perform bidirectional optical flow prediction by referring to a reference sample value at the obtained integer sample position of pixels in the reference picture. 25. The encoder (20) of claim 24, wherein processing the obtained sub-pixel sample position includes rounding the obtained sub-pixel sample position to obtain an integer pixel sample position in the reference picture. 26. The encoder (20) according to claim 24 or 25, wherein processing the received subpixel sample position includes: rounding the obtained sub-pixel sampling position separately in the horizontal direction and vertical direction to the corresponding nearest integer pixel sampling position in the corresponding direction, and obtaining an integer pixel sampling position in the reference picture based on the nearest integer pixel sampling positions in the horizontal direction and vertical direction. 27. The encoder (20) according to claim 26, wherein rounding the obtained sub-pixel sample position separately in the horizontal direction and the vertical direction includes: comparison of the fractional part of the resulting subpixel sample position in the corresponding direction with a threshold value, wherein, in the case where the fractional part is equal to or greater than the threshold value, the nearest upper integer pixel sampling position in the corresponding direction is obtained as the nearest integer pixel sampling position in the corresponding direction, and wherein, in the case where the fractional part is less than the threshold value, the nearest lower integer pixel sampling position in the corresponding direction is obtained as the nearest integer pixel sampling position in the corresponding direction. 28. The encoder (20) of claim 27, wherein the nearest upper integer pixel sample position in the corresponding direction is obtained by adding an offset of 1 to the integer position of the truncated pixel sample in the corresponding direction obtained by discarding the fractional part of the resulting sub-pixel sample position; And wherein the nearest lower integer pixel sample position in the corresponding direction is obtained by adding an offset of 0 to the integer position of the truncated pixel sample in the corresponding direction obtained by discarding the fractional part of the resulting sub-pixel sample position. 29. Decoder (30), containing: a prediction unit configured to perform bidirectional optical flow prediction of the current block; a first receiving unit configured to obtain a motion vector (MV) of the current block from the encoded video bit stream; a second acquiring unit configured to obtain a sample position of subpixels in an expanded region of a predicted block in a reference picture based on said MV; And a processing unit configured to process the obtained sub-pixel sampling position to obtain an integer pixel sampling position in the reference picture; wherein the prediction unit (1440) is configured to perform bidirectional optical flow prediction by referring to a reference sample value at the obtained integer sample position of pixels in the reference picture. 30. The decoder (30) of claim 29, wherein processing the obtained sub-pixel sample position includes rounding the obtained sub-pixel sample position to obtain an integer pixel sample position in the reference picture. 31. The decoder (30) according to claim 29 or 30, wherein processing the received subpixel sample position includes: rounding the obtained sub-pixel sampling position separately in the horizontal direction and vertical direction to the corresponding nearest integer pixel sampling position in the corresponding direction, and obtaining an integer pixel sampling position in the reference picture based on the nearest integer pixel sampling positions in the horizontal direction and vertical direction. 32. The decoder (30) according to claim 31, wherein rounding the obtained sub-pixel sample position separately in the horizontal direction and the vertical direction includes: comparison of the fractional part of the resulting subpixel sample position in the corresponding direction with a threshold value, wherein, in the case where the fractional part is equal to or greater than the threshold value, the nearest upper integer pixel sampling position in the corresponding direction is obtained as the nearest integer pixel sampling position in the corresponding direction, and wherein, in the case where the fractional part is less than the threshold value, the nearest lower integer pixel sampling position in the corresponding direction is obtained as the nearest integer pixel sampling position in the corresponding direction. 33. The decoder (30) of claim 32, wherein the nearest upper integer pixel sample position in the corresponding direction is obtained by adding an offset of 1 to the integer position of the truncated pixel sample in the corresponding direction obtained by discarding the fractional part of the resulting sub-pixel sample position; And wherein the nearest lower integer pixel sample position in the corresponding direction is obtained by adding an offset of 0 to the integer position of the truncated pixel sample in the corresponding direction obtained by discarding the fractional part of the resulting subpixel sample position. 34. A storage medium storing a bitstream obtained using a decoding/encoding method that uses the method according to any one of claims. 1-19. 35. A computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, upon execution of which the processor executes the method according to any one of claims. 1-19.

Images