CN119032566A - Video encoding and decoding method, device, equipment, system and storage medium - Google Patents

Abstract

The embodiment of the application provides a video coding and decoding method, device, equipment, system and storage medium, which aim to improve the accuracy of a reconstructed image, and the quantized first characteristic information is fused with the characteristic information of a previous reconstructed image of a current image, namely the quantized first characteristic information is fused with the characteristic information of a plurality of reconstructed images before the current image, so that when certain information in the previous reconstructed image of the current image is blocked, the blocked information can be obtained from a plurality of reconstructed images before the current image, and further the generated mixed space-time characterization comprises more accurate, rich and detailed characteristic information. When the previous reconstructed image is subjected to motion compensation based on the mixed space-time characterization, the reconstructed image of the current image can be accurately obtained based on the high-precision P predicted images when the high-precision P predicted images can be generated, and then the video compression effect is improved.

Description

Video encoding and decoding method, device, equipment, system and storage medium Technical Field

The present application relates to the field of video coding and decoding technology, and in particular to a video coding and decoding method, device, equipment, system and storage medium.

Background Art

Digital video technology can be incorporated into a variety of video devices, such as digital televisions, smart phones, computers, e-readers or video players, etc. With the development of video technology, the amount of data included in video data is large. In order to facilitate the transmission of video data, video devices implement video compression technology to make video data more efficiently transmitted or stored.

With the rapid development of neural network technology, neural network technology has been widely used in video compression technology, for example, in loop filtering, coding block division and coding block prediction, etc. However, the current video compression technology based on neural network has poor compression effect.

Summary of the invention

The embodiments of the present application provide a video encoding and decoding method, apparatus, device, system and storage medium to improve the video compression effect.

In a first aspect, the present application provides a video decoding method, comprising:

Decoding a first bitstream to determine quantized first feature information, where the first feature information is obtained by performing feature fusion on a current image and a reconstructed image before the current image;

Performing multi-level time-domain fusion on the quantized first feature information to obtain a mixed time-space representation;

Performing motion compensation on the previous reconstructed image according to the mixed spatiotemporal representation to obtain P predicted images of the current image, where P is a positive integer;

A reconstructed image of the current image is determined according to the P predicted images.

In a second aspect, an embodiment of the present application provides a video encoding method, including:

Performing feature fusion on a current image and a previous reconstructed image of the current image to obtain first feature information;

quantizing the first feature information to obtain quantized first feature information;

The quantized first feature information is encoded to obtain the first code stream.

In a third aspect, the present application provides a video encoder for executing the method in the first aspect or its respective implementations. Specifically, the encoder includes a functional unit for executing the method in the first aspect or its respective implementations.

In a fourth aspect, the present application provides a video decoder for executing the method in the second aspect or its respective implementations. Specifically, the decoder includes a functional unit for executing the method in the second aspect or its respective implementations.

In a fifth aspect, a video encoder is provided, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method in the first aspect or its implementations.

In a sixth aspect, a video decoder is provided, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method in the second aspect or its implementations.

In a seventh aspect, a video coding and decoding system is provided, including a video encoder and a video decoder. The video encoder is used to execute the method in the first aspect or its respective implementations, and the video decoder is used to execute the method in the second aspect or its respective implementations.

In an eighth aspect, a chip is provided for implementing the method in any one of the first to second aspects or their respective implementations. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip executes the method in any one of the first to second aspects or their respective implementations.

In a ninth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the method of any one of the first to second aspects or any of their implementations.

In a tenth aspect, a computer program product is provided, comprising computer program instructions, which enable a computer to execute the method in any one of the first to second aspects or their respective implementations.

In an eleventh aspect, a computer program is provided, which, when executed on a computer, enables the computer to execute the method in any one of the first to second aspects or in each of their implementations.

In the twelfth aspect, a code stream is provided, including the code stream generated by any aspect of the second aspect.

Based on the above technical solution, in order to improve the accuracy of the reconstructed image, the present application performs multi-level time-domain fusion on the quantized first feature information, that is, the quantized first feature information is not only fused with the feature information of the previous reconstructed image of the current image, but also the quantized first feature information is feature fused with multiple reconstructed images before the current image. This can avoid that when certain information in the previous reconstructed image of the current image is blocked, the blocked information can be obtained from several reconstructed images before the current image, thereby making the generated mixed spatiotemporal representation include more accurate, rich and detailed feature information. In this way, when motion compensation is performed on the previous reconstructed image based on the mixed spatiotemporal representation, P high-precision predicted images can be generated, and the reconstructed image of the current image can be accurately obtained based on the high-precision P predicted images, thereby improving the video compression effect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic block diagram of a video encoding and decoding system according to an embodiment of the present application;

FIG2 is a schematic diagram of a flow chart of a video decoding method provided in an embodiment of the present application;

FIG3 is a schematic diagram of a network structure of an inverse transformation module involved in an embodiment of the present application;

FIG4 is a schematic diagram of a network structure of a recursive aggregation module according to an embodiment of the present application;

FIG5 is a schematic diagram of a network structure of a first decoder involved in an embodiment of the present application;

FIG6 is a schematic diagram of a network structure of a second decoder involved in an embodiment of the present application;

FIG7 is a schematic diagram of a network structure of a third decoder involved in an embodiment of the present application;

FIG. 8 is a schematic diagram of the network structure of the fourth decoder involved in the embodiment of the present application

FIG9 is a schematic diagram of a network structure of a decoder based on a neural network according to an embodiment of the present application;

FIG10 is a schematic diagram of a video decoding process provided by an embodiment of the present application;

FIG11 is a schematic diagram of a flow chart of a video encoding method provided in an embodiment of the present application;

FIG12 is a schematic diagram of a network structure of an encoder based on a neural network according to an embodiment of the present application;

FIG13 is a schematic diagram of a video encoding process provided by an embodiment of the present application;

FIG14 is a schematic block diagram of a video decoding device provided in an embodiment of the present application;

FIG15 is a schematic block diagram of a video encoding device provided in an embodiment of the present application;

FIG16 is a schematic block diagram of an electronic device provided in an embodiment of the present application;

FIG. 17 is a schematic block diagram of a video encoding system provided in an embodiment of the present application.

DETAILED DESCRIPTION

The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

The present application can be applied to the field of image coding and decoding, the field of video coding and decoding, the field of hardware video coding and decoding, the field of dedicated circuit video coding and decoding, the field of real-time video coding and decoding, etc. Alternatively, the scheme of the present application can be combined with other proprietary or industry standards and operated, and the standards include ITU-TH.261, ISO/IECMPEG-1Visual, ITU-TH.262 or ISO/IECMPEG-2Visual, ITU-TH.263, ISO/IECMPEG-4Visual, ITU-TH.264 (also known as ISO/IECMPEG-4AVC), including scalable video coding (SVC) and multi-view video coding (MVC) extensions. It should be understood that the technology of the present application is not limited to any specific coding standard or technology.

For ease of understanding, the video encoding and decoding system involved in the embodiment of the present application is first introduced in conjunction with Figure 1.

FIG1 is a schematic block diagram of a video encoding and decoding system involved in an embodiment of the present application. It should be noted that FIG1 is only an example, and the video encoding and decoding system of the embodiment of the present application includes but is not limited to that shown in FIG1. As shown in FIG1, the video encoding and decoding system 100 includes an encoding device 110 and a decoding device 120. The encoding device is used to encode (which can be understood as compression) the video data to generate a code stream, and transmit the code stream to the decoding device. The decoding device decodes the code stream generated by the encoding device to obtain decoded video data.

The encoding device 110 of the embodiment of the present application can be understood as a device with a video encoding function, and the decoding device 120 can be understood as a device with a video decoding function, that is, the embodiment of the present application includes a wider range of devices for the encoding device 110 and the decoding device 120, such as smartphones, desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, vehicle-mounted computers, etc.

In some embodiments, the encoding device 110 may transmit the encoded video data (eg, a code stream) to the decoding device 120 via the channel 130. The channel 130 may include one or more media and/or devices capable of transmitting the encoded video data from the encoding device 110 to the decoding device 120.

In one example, the channel 130 includes one or more communication media that enable the encoding device 110 to transmit the encoded video data directly to the decoding device 120 in real time. In this example, the encoding device 110 can modulate the encoded video data according to the communication standard and transmit the modulated video data to the decoding device 120. The communication medium includes a wireless communication medium, such as a radio frequency spectrum, and optionally, the communication medium may also include a wired communication medium, such as one or more physical transmission lines.

In another example, the channel 130 includes a storage medium, which can store the video data encoded by the encoding device 110. The storage medium includes a variety of locally accessible data storage media, such as optical disks, DVDs, flash memories, etc. In this example, the decoding device 120 can obtain the encoded video data from the storage medium.

In another example, the channel 130 may include a storage server that can store the video data encoded by the encoding device 110. In this example, the decoding device 120 can download the stored encoded video data from the storage server. Alternatively, the storage server can store the encoded video data and transmit the encoded video data to the decoding device 120, such as a web server (e.g., for a website), a file transfer protocol (FTP) server, etc.

In some embodiments, the encoding device 110 includes a video encoder 112 and an output interface 113. The output interface 113 may include a modulator/demodulator (modem) and/or a transmitter.

In some embodiments, the encoding device 110 may further include a video source 111 in addition to the video encoder 112 and the input interface 113 .

The video source 111 may include at least one of a video acquisition device (eg, a video camera), a video archive, a video input interface, and a computer graphics system, wherein the video input interface is used to receive video data from a video content provider, and the computer graphics system is used to generate video data.

The video encoder 112 encodes the video data from the video source 111 to generate a code stream. The video data may include one or more pictures or a sequence of pictures. The code stream contains the coding information of the picture or the sequence of pictures in the form of a bit stream. The coding information may include the coded picture data and associated data. The associated data may include a sequence parameter set (SPS for short), a picture parameter set (PPS for short) and other syntax structures. The SPS may contain parameters applied to one or more sequences. The PPS may contain parameters applied to one or more pictures. The syntax structure refers to a set of zero or more syntax elements arranged in a specified order in the code stream.

The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data may also be stored in a storage medium or a storage server for subsequent reading by the decoding device 120.

In some embodiments, the decoding device 120 includes an input interface 121 and a video decoder 122 .

In some embodiments, the decoding device 120 may include a display device 123 in addition to the input interface 121 and the video decoder 122 .

The input interface 121 includes a receiver and/or a modem. The input interface 121 can receive the encoded video data through the channel 130 .

The video decoder 122 is used to decode the encoded video data to obtain decoded video data, and transmit the decoded video data to the display device 123 .

The display device 123 displays the decoded video data. The display device 123 may be integrated with the decoding device 120 or external to the decoding device 120. The display device 123 may include a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.

In addition, FIG1 is only an example, and the technical solution of the embodiment of the present application is not limited to FIG1 . For example, the technology of the present application can also be applied to unilateral video encoding or unilateral video decoding.

In some embodiments, the video encoder 112 may be applied to image data in a luminance-chrominance (YCbCr, YUV) format. For example, the YUV ratio may be 4:2:0, 4:2:2, or 4:4:4, where Y represents brightness (Luma), Cb (U) represents blue chrominance, Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) for describing color and saturation. For example, in color format, 4:2:0 means that every 4 pixels have 4 luminance components and 2 chrominance components (YYYYCbCr), 4:2:2 means that every 4 pixels have 4 luminance components and 4 chrominance components (YYYYCbCrCbCr), and 4:4:4 represents full pixel display (YYYYCbCrCbCrCbCrCbCr).

Since there is a strong correlation between adjacent pixels in a video frame, the intra-frame prediction method is used in video coding and decoding technology to eliminate the spatial redundancy between adjacent pixels. Since there is a strong similarity between adjacent frames in a video, the inter-frame prediction method is used in video coding and decoding technology to eliminate the temporal redundancy between adjacent frames, thereby improving coding efficiency.

The embodiments of the present application may be used for inter-frame coding to improve the efficiency of inter-frame coding.

Video coding technology mainly encodes serialized video data and is mainly used for data storage, transmission and presentation in the Internet era. Video currently accounts for more than 85% of traffic space and entry. With the increasing demand of users for video data resolution, frame rate and dimension in the future, the role and value of video coding technology will also increase significantly. The technical improvement and demand for video coding are huge opportunities and challenges. Traditional video coding technology has undergone decades of development and change, and has greatly satisfied and served the world's video services in every era. Traditional video coding technology has been iteratively updated and used to this day under the hybrid coding framework based on multi-scale block level. With the rapid development of hardware technology, video coding has brought great coding performance improvement through the improvement of sub-technology at the expense of certain complexity. However, the way of obtaining performance by replacing complexity has gradually become more obviously limited due to the bottleneck of hardware development, which puts higher requirements on hardware design and update, so that the traditional codecs currently in commercial use usually need to be simplified to a certain extent.

At the same time, deep learning technology, especially deep neural network technology, is becoming more and more mature, and has been widely studied and used in multiple tasks of video images, including video enhancement, video detection, and video segmentation. The deep learning technology applied to the field of video coding initially focused on the research and replacement of sub-techniques in traditional video coding. By studying the relevant modules in traditional video coding, the original video coding framework is used as a data generation tool to obtain paired training data to train the corresponding neural network, and then the corresponding modules are replaced after the final neural network converges. The replaceable modules include in-loop filtering, out-of-loop filtering, coding block division, coding block prediction, etc. However, the current video compression technology based on neural networks has poor compression effect.

In order to further improve the video compression effect, this application proposes a purely data-driven neural network coding framework, that is, the entire coding and decoding system is designed, trained and ultimately used for video coding based on a deep neural network, and a new hybrid lossy motion expression method is adopted to realize the neural network-based inter-frame coding and decoding technology.

The technical solution provided in the embodiments of the present application is described in detail below in conjunction with specific embodiments.

First, in conjunction with FIG. 2 , the decoding end is taken as an example for introduction.

FIG2 is a flow chart of a video decoding method provided by an embodiment of the present application, and the embodiment of the present application is applied to the video decoder shown in FIG1. As shown in FIG2, the method of the embodiment of the present application includes:

S201. Decode a first bit stream to determine quantized first feature information.

The first feature information is obtained by fusing the features of the current image and a previous reconstructed image of the current image.

An embodiment of the present application proposes a neural network-based decoder, which is obtained by end-to-end training of a neural network-based encoder.

In the embodiment of the present application, the previous reconstructed image of the current image can be understood as the previous frame image before the current image in the video sequence, and the previous frame image has been decoded and reconstructed.

Since there is a strong similarity between the two adjacent frames of the current image and the previous reconstructed image of the current image, the encoding end performs feature fusion on the current image and the previous reconstructed image of the current image during encoding to obtain the first feature information. For example, the encoding end cascades the current image and the previous reconstructed image of the current image, and extracts features from the cascaded image to obtain the first feature information. Exemplarily, the encoding end extracts features from the cascaded image through a feature extraction module to obtain the first feature information. The present application does not limit the specific network structure of the feature extraction module. The first feature information obtained above is a floating point type, for example, a 32-bit floating point number representation. Further, in order to reduce the encoding cost, the encoding end quantizes the first feature information obtained above to obtain the quantized first feature information. Then, the quantized first feature information is encoded to obtain a first code stream. For example, the encoding end performs arithmetic encoding on the first feature information to obtain the first code stream. In this way, after obtaining the first code stream, the decoding end decodes the first code stream to obtain the quantized first feature information, and obtains the reconstructed image of the current image based on the quantized first feature information.

In the embodiment of the present application, the decoding end decodes the first bit stream in the above S201, and determines the quantized first feature information in the following ways, but not limited to:

In method 1, if the encoding end directly uses the probability distribution of the quantized first feature information to encode the quantized first feature information to obtain the first bitstream, then the decoding end directly decodes the first bitstream to obtain the quantized first feature information.

The quantized first feature information includes a large amount of redundant information. When the quantized first feature information is directly encoded, many codewords are required for encoding and the encoding cost is large. In order to reduce the encoding cost, in some embodiments, the encoding end performs feature transformation according to the first feature information to obtain the second feature information, and quantizes the second feature information and then encodes it to obtain the second code stream; decodes the second code stream to obtain the quantized second feature information, and determines the probability distribution of the quantized first feature information according to the quantized second feature information; and then encodes the quantized first feature information according to the probability distribution of the quantized first feature information to obtain the first code stream. In other words, in order to reduce the encoding cost, the encoding end determines the super-prior feature information corresponding to the first feature information, that is, the second feature information, and determines the probability distribution of the quantized first feature information based on the second feature information. Since the second feature information is the super-prior feature information of the first feature information, it includes less redundancy. In this way, the probability distribution of the quantized first feature information is determined based on the second feature information with less redundancy, and the first feature information is encoded using the probability distribution, which can reduce the encoding cost of the first feature information.

Based on the above description, the decoding end can determine the quantized first feature information through the steps of the following method 2.

Mode 2, the above S201 includes the following steps S201-A to S201-C:

S201-A. Decode the second code stream to obtain quantized second feature information.

The second feature information is obtained by performing feature transformation on the first feature information.

As can be seen from the above, in order to reduce the coding cost, the encoder performs feature transformation on the first feature information to obtain the super-prior feature information of the first feature information, that is, the second feature information, and uses the second feature information to determine the probability distribution of the quantized first feature information, and uses the probability distribution to encode the quantized first feature information to obtain the first code stream. At the same time, in order to enable the decoder to decode the first code stream using the same probability distribution as the encoding, the second feature information is encoded to obtain the second code stream. That is to say, in this second method, the encoder generates two code streams, namely the first code stream and the second code stream.

In this way, after the decoding end obtains the first code stream and the second code stream, it first decodes the second code stream to determine the probability distribution of the quantized first feature information. Specifically, the second code stream is decoded to obtain the quantized second feature information, and the probability distribution of the quantized first feature information is determined based on the quantized second feature information. Then, the decoding end decodes the first code stream using the determined probability distribution to obtain the quantized first feature information, thereby achieving accurate decoding of the first feature information.

In the present application, since the second feature information is the super-prior feature information of the first feature information and includes less redundant information, the encoder can directly use the probability distribution of the quantized second feature information to encode the quantized second feature information to obtain the second bitstream when encoding. Correspondingly, the decoder can directly decode the second bitstream to obtain the quantized second feature information when decoding.

S201-B. Determine the probability distribution of the quantized first feature information according to the quantized second feature information.

After the decoding end determines the quantized second feature information according to the above steps, it determines the probability distribution of the quantized first feature information according to the quantized second feature information.

In the embodiment of the present application, there is no limitation on the specific method of determining the probability distribution of the quantized first feature information according to the quantized second feature information in the above S201-B.

In some embodiments, since the second feature information is obtained by performing feature transformation on the first feature information, based on this, S201-B includes the following steps S201-B1 to S201-B3:

S201-B1. Perform an inverse transformation on the quantized second feature information to obtain reconstructed feature information.

In this implementation, the decoding end performs an inverse transformation on the quantized second feature information to obtain the reconstructed feature information, wherein the inverse transformation method adopted by the decoding end can be understood as the inverse operation of the transformation method adopted by the encoding end. For example, the encoding end performs N times of feature extraction on the first feature information to obtain the second feature information, and correspondingly, the decoding end performs N times of reverse feature extraction on the quantized second feature information to obtain the inversely transformed feature information, which is recorded as the reconstructed feature information.

The embodiment of the present application does not limit the inverse transformation method adopted by the decoding end.

In some embodiments, the inverse transformation method adopted by the decoding end includes N times of feature extraction. That is, the decoding end performs N times of feature extraction on the obtained quantized second feature information to obtain reconstructed feature information.

In some embodiments, the inverse transformation method adopted by the decoding end includes N times of feature extraction and N times of upsampling. That is, the decoding end performs N times of feature extraction and N times of upsampling on the obtained quantized second feature information to obtain reconstructed feature information.

The embodiment of the present application does not limit the specific execution order of the above-mentioned N feature extractions and N upsamplings.

In an example, the decoding end may first perform N consecutive feature extractions on the quantized second feature information, and then perform N consecutive upsamplings.

In another example, the above-mentioned N feature extractions and N upsamplings are performed alternately, that is, upsampling is performed once after performing one feature extraction. For example, assuming that N=2, the decoding end performs an inverse transformation on the quantized second feature information, and the specific process of obtaining the reconstructed feature information is: input the quantized second feature information into the first feature extraction module for the first feature extraction to obtain feature information 1, upsample feature information 1 to obtain feature information 2, input feature information 2 into the second feature extraction module for the second feature extraction to obtain feature information 3, upsample feature information 3 to obtain feature information 4, and record feature information 4 as the reconstructed feature information.

It should be noted that the embodiment of the present application does not limit the N-time feature extraction method adopted by the decoding end, for example, at least one of the feature extraction methods including multi-layer convolution, residual connection, dense connection, etc.

In some embodiments, the decoding end performs feature extraction by non-local attention. In this case, the above S201-B1 includes the following steps S201-B11:

S201-B11, perform N non-local attention transformations and N upsamplings on the quantized second feature information to obtain reconstructed feature information, where N is a positive integer.

Since the non-local attention method can achieve more efficient feature extraction, the extracted features can retain more information, and the calculation efficiency is high, therefore, in the embodiment of the present application, the decoding end uses the non-local attention method to extract features from the quantized second feature information to achieve fast and accurate feature extraction of the quantized second feature information. In addition, when the encoding end generates the second feature information based on the first feature information, it performs N downsampling, so the decoding end correspondingly performs N upsampling to make the reconstructed feature information consistent with the size of the first feature information.

In some embodiments, as shown in FIG3 , the decoding end obtains the reconstructed feature information through the inverse transformation module, and the inverse transformation module includes N non-local attention modules and N upsampling modules. Among them, the non-local attention module is used to realize the non-local attention transformation, and the upsampling module is used to realize the upsampling. Exemplarily, as shown in FIG3 , an upsampling module is connected after a non-local attention module. In actual application, the decoding end inputs the decoded quantized second feature information into the inverse transformation module, and the first non-local attention module in the inverse transformation module performs non-local attention feature transformation extraction on the quantized second feature information to obtain feature information 1, and then inputs feature information 1 into the first upsampling module for upsampling to obtain feature information 2. Next, feature information 2 is input into the second non-local attention module for non-local attention feature transformation extraction to obtain feature information 3, and then feature information 3 is input into the second upsampling module for upsampling to obtain feature information 4. By analogy, the feature information output by the Nth upsampling module is obtained, and the feature information is determined as the reconstructed feature information.

S201-B2. Determine the probability distribution of the reconstructed feature information.

From the above, it can be seen that the second quantized feature information is obtained by transforming the first feature information. The decoding end dequantizes the quantized second feature information through the above steps to obtain the reconstructed feature information. Therefore, the reconstructed feature information can be understood as the reconstructed information of the first feature information, that is, the probability distribution of the reconstructed feature information is similar to or related to the probability distribution of the quantized first feature information. In this way, the decoding end can first determine the probability distribution of the reconstructed feature information, and then predict the probability distribution of the first feature information after quantization based on the probability distribution of the reconstructed feature information.

In some embodiments, the probability distribution of the reconstructed feature information is a normal distribution or a Gaussian distribution. In this case, the process of determining the probability distribution of the reconstructed feature information is to determine the mean and variance matrix of the reconstructed feature information based on each eigenvalue in the reconstructed feature information, and generate a Gaussian distribution of the reconstructed feature information based on the mean and variance matrix.

S201-B3. According to the probability distribution of the reconstructed feature information, predict the probability distribution of the quantized first feature information.

Since the reconstructed feature information is the reconstructed information of the first feature information, the probability distribution of the reconstructed feature information is similar or related to the probability distribution of the quantized first feature information. Therefore, the embodiment of the present application can achieve accurate prediction of the probability distribution of the quantized first feature information through the probability distribution of the reconstructed feature information.

The embodiment of the present application does not limit the specific implementation method of the above S201-B3.

In a possible implementation manner, the probability distribution of the reconstructed feature information is determined as the probability distribution of the quantized first feature information.

In another possible implementation, the probability of the encoded pixels in the quantized first feature information is predicted based on the probability distribution of the reconstructed feature information; and the probability distribution of the quantized first feature information is obtained based on the probability of the encoded pixels in the quantized first feature information.

S201-C. Decode the first bitstream according to the probability distribution of the quantized first feature information to obtain the quantized first feature information.

According to the above steps, after the probability distribution of the quantized first feature information is determined, the first code stream is decoded using the probability distribution, thereby achieving accurate decoding of the quantized first feature information.

In the embodiment of the present application, the decoding end decodes the first bit stream according to the above-mentioned method 1 or method 2, determines the quantized first feature information, and then performs the following step S202.

S202, performing multi-level time-domain fusion on the quantized first feature information to obtain a mixed time-space representation.

In an embodiment of the present application, in order to improve the accuracy of the reconstructed image, the quantized first feature information is subjected to multi-level time domain fusion, that is, the quantized first feature information is not only fused with the feature information of the previous reconstructed image of the current image, but also the quantized first feature information is fused with the multiple reconstructed images before the current image, for example, the reconstructed images at multiple times such as time t-1, time t-2..., time t-k are fused with the quantized first feature information. In this way, when certain information in the previous reconstructed image of the current image is blocked, the blocked information can be obtained from several reconstructed images before the current image, thereby making the generated mixed spatiotemporal representation include more accurate, rich and detailed feature information. In this way, when motion compensation is performed on the previous reconstructed image based on the mixed spatiotemporal representation to generate P predicted images of the current image, the accuracy of the generated predicted image can be improved, and then the reconstructed image of the current image can be accurately obtained based on the accurate predicted image, thereby improving the video compression effect.

The embodiment of the present application does not limit the specific method in which the decoding end performs multi-level time domain fusion on the quantized first feature information to obtain a mixed time-space representation.

In some embodiments, the decoding end mixes the spatiotemporal representations through a recursive aggregation module, that is, the above S202 includes the following step S202-A:

S202-A, the decoding end fuses the quantized first feature information with the implicit feature information of the recursive aggregation module at the previous moment through the recursive aggregation module to obtain a mixed spatiotemporal representation.

The recursive aggregation module of the embodiment of the present application will learn and retain the deep-level feature information learned from the feature information each time it generates a mixed space-time representation, and use the learned deep-level features as implicit feature information for the next generation of the mixed space-time representation, thereby improving the accuracy of the generated mixed space-time representation. That is to say, in the embodiment of the present application, the implicit feature information of the recursive aggregation module at the previous moment includes the feature information of multiple reconstructed images before the current image learned by the recursive aggregation module. In this way, the decoding end fuses the quantized first feature information with the implicit feature information of the recursive aggregation module at the previous moment through the recursive aggregation module, so as to generate a more accurate, rich and detailed mixed space-time representation.

The embodiment of the present application does not limit the specific network structure of the recursive aggregation module, for example, it can be any network structure that can implement the above functions.

In some embodiments, the recursive aggregation module is formed by stacking at least one spatiotemporal recursive network ST-LSTM. In this case, the expression formula of the above hybrid spatiotemporal representation Gt is as shown in formula (1):

in, is the first feature information after quantization, and h is the implicit feature information included in ST-LSTM.

In one example, assuming that the recursive aggregation module includes two ST-LSTM components, as shown in FIG4 , the decoder reconstructs the quantized first feature information The input is sent to the recursive aggregation module, and the two ST-LSTMs in the recursive aggregation module sequentially quantize the first feature information Processing is performed to generate feature information. Specifically, as shown in Figure 4, the implicit feature information h1 generated by the first ST-LSTM is used as the input of the next ST-LSTM, and the two ST-LSTMs generate transmission band update values c1 and c2 respectively during this operation to update their respective transmission band values, where m is the memory information, which is transmitted between the two ST-LSTMs, and finally the feature information h2 output by the second ST-LSTM is obtained. Furthermore, in order to improve the accuracy of the generated hybrid spatiotemporal representation, the feature information h2 generated by the second ST-LSTM is compared with the quantized first feature information. Perform residual connection, that is, the feature information h generated by the second ST-LSTM is connected to the quantized first feature information Add them together to generate a mixed spatiotemporal representation Gt.

After the decoder obtains the mixed spatiotemporal representation according to the above method, it executes the following S203.

S203 . Perform motion compensation on the previous reconstructed image according to the hybrid spatiotemporal representation to obtain P predicted images of the current image.

Wherein, P is a positive integer.

From the above, it can be seen that the hybrid spatiotemporal representation of the embodiment of the present application fuses the feature information of the current image and multiple reconstructed images before the current image, so that motion compensation is performed on the previous reconstructed image according to the hybrid spatiotemporal representation, and P accurate predicted images of the current image can be obtained.

The embodiment of the present application does not limit the specific number of the generated P predicted images. That is, in the embodiment of the present application, the decoding end can use different methods to perform motion compensation on the previous reconstructed image according to the mixed spatiotemporal representation to obtain P predicted images of the current image.

The embodiment of the present application does not limit the specific manner in which the decoding end performs motion compensation on the previous reconstructed image according to the mixed spatiotemporal representation.

In some embodiments, the P predicted images include a first predicted image, and the first predicted image is obtained by the decoding end using an optical flow motion compensation method. In this case, the S203 includes the following steps S203-A1 and S203-A2:

S203-A1, determining optical flow motion information according to the mixed spatiotemporal representation;

S203-A2: Perform motion compensation on the previous reconstructed image according to the optical flow motion information to obtain a first predicted image.

The embodiment of the present application does not limit the specific manner in which the decoding end determines the optical flow motion information based on the mixed spatiotemporal representation.

In some embodiments, the decoding end obtains the optical flow motion information through a pre-trained neural network model, that is, the neural network model can predict the optical flow motion information based on the mixed spatiotemporal representation. In some embodiments, the neural network model can be called a first decoder, or an optical flow signal decoder Df. The decoding end inputs the mixed spatiotemporal representation Gt into the optical flow signal decoder Df to predict the optical flow motion information, and obtains the optical flow motion information _fx,y output by the optical flow signal decoder Df. Optionally, fx _,y is the optical flow motion information with a channel of 2.

Exemplarily, the formula for generating f _x,y is shown in formula (2):

f _x,y = D _f (G _t ) (2)

The embodiment of the present application does not limit the specific network structure of the optical flow signal decoder Df.

In some embodiments, the optical flow signal decoder Df is composed of multiple NLAMs and multiple upsampling modules. Exemplarily, as shown in FIG5, the optical flow signal decoder Df includes 1 NLAM, 3 LAMs and 4 downsampling modules, wherein one NLAM is connected to a downsampling module, and one LAM is connected to a downsampling module. Optionally, the NLAM includes multiple convolutional layers, for example, including 3 convolutional layers, each of which has a convolution kernel size of 3*3 and a channel number of 192. Optionally, the three LAMs include multiple convolutional layers, for example, including 3 convolutional layers, each of which has a convolution kernel size of 3*3, and the number of channels of the convolutional layers included in the three LAMs is 128, 96 and 64, respectively. Optionally, the four downsampling modules each include a convolutional layer Conv, the convolution kernel size of the convolutional layer is 5*5, and the number of channels of the convolutional layers included in the four downsampling modules is 128, 96, 64 and 2, respectively. In this way, the decoding end inputs the mixed spatiotemporal representation Gt into the optical flow signal decoder Df, and the NLAM extracts the features of the spatiotemporal representation Gt to obtain feature information a with a channel number of 192, and inputs the feature information a into the first downsampling module for downsampling to obtain feature information b with a channel number of 128. Then, the feature information b is input into the first LAM for feature re-extraction to obtain feature information c with a channel number of 128, and the feature information c is input into the second downsampling module for downsampling to obtain feature information d with a channel number of 96. Then, the feature information d is input into the second LAM for feature re-extraction to obtain feature information e with a channel number of 96, and the feature information e is input into the third downsampling module for downsampling to obtain feature information f with a channel number of 64. Then, the feature information f is input into the third LAM for feature re-extraction to obtain feature information g with a channel number of 64, and the feature information g is input into the fourth downsampling module for downsampling to obtain feature information j with a channel number of 2, and the feature information j is the optical flow motion information.

It should be noted that the above FIG. 5 is only an example, and the settings of the parameters in FIG. 5 are also only examples. The network structure of the optical flow signal decoder Df in the embodiment of the present application includes but is not limited to that shown in FIG. 5 .

After the decoder generates the optical flow motion information f _x,y , it uses the optical flow motion information f _x,y to reconstruct the previous image Motion compensation is performed to obtain a first predicted image X ₁ .

In the embodiment of the present application, the decoding end performs motion compensation on the previous reconstructed image according to the optical flow motion information to obtain the first predicted image without limiting the specific method. For example, the decoding end uses the optical flow motion information f _x,y to compensate the previous reconstructed image. Linear interpolation is performed, and the image generated by interpolation is recorded as the first predicted image X ₁ .

In a possible implementation, the decoding end obtains the first predicted image X ₁ by the following formula (3):

In this implementation, as shown in FIG5 , the decoder uses the optical flow motion information fxy to reconstruct the previous image by performing a Warping operation. Motion compensation is performed to obtain a first predicted image X ₁ .

In some embodiments, the P predicted images include a second predicted image, and the second predicted image is obtained by the decoding end using an offset motion compensation method. In this case, the above S203 includes the following steps S203-B1 to S203-B3:

S203-B1, obtaining an offset corresponding to the current image according to the mixed spatiotemporal representation;

S203-B2, extracting spatial features from the previous reconstructed image to obtain reference feature information;

S203-B3. Use the offset to perform motion compensation on the reference feature information to obtain a second predicted image.

The embodiment of the present application does not limit the specific manner in which the decoding end obtains the offset corresponding to the current image based on the mixed spatiotemporal representation.

In some embodiments, the decoding end obtains the offset corresponding to the current image through a pre-trained neural network model, that is, the neural network model can predict the offset based on the mixed spatiotemporal representation, and the offset is lossy offset information. In some embodiments, the neural network model can be called a second decoder, or a variable convolution decoder Dm. The decoding end inputs the mixed spatiotemporal representation Gt into the variable convolution decoder Dm to predict the offset information.

At the same time, the decoding end extracts spatial features from the previous reconstructed image to obtain reference feature information. For example, the decoding end extracts spatial features from the previous reconstructed image through a spatial feature extraction module SFE to obtain reference feature information.

Next, the decoding end uses the offset to perform motion compensation on the extracted reference feature information to obtain a second predicted image of the current image.

In the embodiment of the present application, the decoding end uses the offset to perform motion compensation on the extracted reference feature information, and the specific method of obtaining the second predicted image of the current image is not limited.

In a possible implementation, the decoding end uses the offset to perform deformable convolution-based motion compensation on the reference feature information to obtain a second predicted image.

In some embodiments, since the transformable convolution can generate an offset corresponding to the current image based on the mixed space-time representation, in an embodiment of the present application, the decoding end inputs the mixed space-time representation Gt and the reference feature information into the transformable convolution. The transformable convolution generates an offset corresponding to the current image based on the mixed space-time representation Gt, and applies the offset to the reference feature information for motion compensation, thereby obtaining a second predicted image.

Based on this, exemplary, as shown in FIG6 , the variable convolution decoder Dm of the embodiment of the present application includes a transformable convolution DCN, and the decoding end reconstructs the previous image The mixed spatiotemporal representation Gt and the reference feature information are then input into the transformable convolution DCN to extract the offset and perform motion compensation to obtain the second predicted image X ₂ .

Exemplarily, the decoding end generates the second predicted image X ₂ by formula (4):

The embodiment of the present application does not limit the specific network structure of the optical flow signal decoder Df.

In some embodiments, as shown in FIG6 , in order to further improve the accuracy of the second predicted image, the variable convolution decoder Dm includes, in addition to the transformable convolution DCN, 1 NLAM, 3 LAMs and 4 downsampling modules, wherein one NLAM is connected to a downsampling module, and one LAM is connected to a downsampling module. Optionally, the network structure of the 1 NLAM, 3 LAMs and the first 3 downsampling modules included in the variable convolution decoder Dm is the same as the network structure of the 1 NLAM, 3 LAMs and the first 3 downsampling modules included in the above-mentioned optical flow signal decoder Df, and will not be repeated here. Optionally, the number of channels included in the last downsampling module included in the variable convolution decoder Dm is 5.

It should be noted that the above-mentioned Figure 6 is only an example, and the settings of the parameters in Figure 6 are also only examples. The network structure of the variable convolution decoder Dm of the embodiment of the present application includes but is not limited to that shown in Figure 6.

In the embodiment of the present application, as shown in FIG6 , the decoding end first converts the previous reconstructed image The mixed spatiotemporal representation Gt and the reference feature information are then input into the transformable convolution DCN in the variable convolution decoder Dm for offset extraction and motion compensation to obtain feature information, which is then input into the NLAM. After feature extraction by the NLAM, 3 LAMs and 4 downsampling modules, the second predicted image X ₂ is finally restored.

According to the above method, the decoding end can determine P predicted images, for example, determine the first predicted image and the second predicted image, and then perform the following step S204.

S204: Determine a reconstructed image of the current image according to the P predicted images.

In some embodiments, if the P predicted images include one predicted image, a reconstructed image of the current image is determined according to the predicted image.

For example, the predicted image is compared with one or several previous reconstructed images of the current image to calculate the loss. If the loss is small, it means that the prediction accuracy of the predicted image is high, and the predicted image can be determined as the reconstructed image of the current image.

For another example, if the above-mentioned loss is large, it means that the prediction accuracy of the predicted image is low. At this time, the reconstructed image of the current image can be determined based on the previous one or several reconstructed images of the current image and the predicted image. For example, the predicted image and the previous one or several reconstructed images of the current image are input into a neural network to obtain the reconstructed image of the current image.

In some embodiments, the above S204 includes the following steps S204-A and S204-B:

S204-A. Determine a target predicted image for the current image based on the P predicted images.

In this implementation, the decoding end first determines the target prediction image of the current image based on P prediction images, and then realizes the reconstructed image of the current image based on the target prediction image of the current image, thereby improving the determination accuracy of the reconstructed image.

The embodiment of the present application does not limit the specific method of determining the target predicted image of the current image based on P predicted images.

In some embodiments, if P=1, the one predicted image is determined as the target predicted image of the current image.

In some embodiments, if P is greater than 1, S204-A includes S204-A11 and S204-A12:

S204-A11, determining a weighted image according to the P predicted images;

In this implementation, if multiple predicted images of the current image are generated according to the above method, for example, when a first predicted image and a second predicted image are generated, these P predicted images are weighted to generate a weighted image, and then the target predicted image is obtained based on the weighted image.

The embodiment of the present application does not limit the specific method of determining the weighted image based on P predicted images.

For example, weights corresponding to P predicted images are determined; and according to the weights corresponding to the P predicted images, the P predicted images are weighted to obtain a weighted image.

Exemplarily, if the P predicted images include a first predicted image and a second predicted image, the decoding end determines a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image, and weights the first predicted image and the second predicted image according to the first weight and the second weight to obtain a weighted image.

The methods for determining the weights corresponding to the P prediction images include but are not limited to the following:

Method 1: The weights corresponding to the above P predicted images are preset weights. Assume that P=2, that is, the first weight corresponding to the first predicted image and the second weight corresponding to the second predicted image can be that the first weight is equal to the second weight, or the ratio of the first weight to the second weight is 1/2, 1/4, 1/2, 1/3, 2/1, 3/1, 4/1, etc.

In the second method, the decoder performs adaptive masking based on the mixed spatiotemporal representation to obtain weights corresponding to P predicted images.

Exemplarily, the decoding end generates weights corresponding to P predicted images through a neural network model, and the neural network model is pre-trained and can be used to generate weights corresponding to P predicted images. In some embodiments, the neural network model is also called a third decoder or an adaptive mask compensation decoder _Dw . Specifically, the decoding end inputs the mixed spatiotemporal representation into the adaptive mask compensation decoder _Dw for adaptive masking to obtain weights corresponding to the P predicted images. For example, the decoding end inputs the mixed spatiotemporal representation Gt into the adaptive mask compensation decoder _Dw for adaptive masking, and the adaptive mask compensation decoder _Dw outputs a first weight w1 of the first predicted image and a second weight w2 of the second predicted image, and performs the first weight w1 and the second weight w2 on the first predicted image _X1 and the second predicted image _X2 obtained above, and can adaptively select the corresponding information representing different regions in the predicted frame, thereby generating a weighted image.

Exemplarily, the weighted image X ₃ is generated according to the following formula (5):

X ₃ =w ₁ *X ₁ +w ₂ *X ₂ (5)

In some embodiments, the weights corresponding to the above-mentioned P predicted images are a matrix, including the weights corresponding to each pixel in the predicted image. When generating a weighted image, for each pixel in the current image, the predicted values and their weights corresponding to the pixel in the P predicted images are weighted to obtain the weighted predicted value of the pixel. In this way, the weighted prediction values corresponding to each pixel in the current image constitute the weighted image of the current image.

The embodiment of the present application does not limit the specific network structure of the adaptive mask compensation decoder _Dw .

In some embodiments, as shown in FIG7 , the adaptive mask compensation decoder D _w includes 1 NLAM, 3 LAMs, 4 downsampling modules and a sigmoid function, wherein one NLAM is connected to a downsampling module, and one LAM is connected to a downsampling module. Optionally, the network structure of the 1 NLAM, 3 LAMs, and 4 downsampling modules included in the adaptive mask compensation decoder D _w is consistent with the network structure of the 1 NLAM, 3 LAMs, and 4 downsampling modules included in the above-mentioned variable convolution decoder D m, which is not repeated here.

It should be noted that FIG. 7 is only an example, and the settings of the parameters in FIG. 7 are also only examples. The network structure of the adaptive mask compensation decoder _Dw in the embodiment of the present application includes but is not limited to that shown in FIG. 7.

In this implementation, the decoding end weights the P predicted images according to the above method, and after obtaining the weighted images, executes the following S204-A12.

S204-A12: Obtain a target prediction image based on the weighted image.

For example, the weighted image is determined as the target prediction image.

In some embodiments, the decoding end may also obtain a residual image of the current image based on the mixed spatiotemporal representation.

Exemplarily, the decoding end obtains the residual image of the current image through a neural network model, and the neural network model is pre-trained and can be used to generate the residual image of the current image. In some embodiments, the neural network model is also called a fourth decoder or a spatial texture enhancement decoder Dt. Specifically, the decoding end inputs the mixed spatiotemporal representation into the spatial texture enhancement decoder Dt for spatial texture enhancement, and obtains the residual image _Xr = D_t ( _Gt ) of the current image, and the residual image _Xr can perform texture enhancement on the predicted image.

In the embodiment of the present application, there is no limitation on the specific network structure of the above-mentioned spatial texture enhancement decoder Dt.

In some embodiments, as shown in FIG8 , the spatial texture enhancement decoder Dt includes 1 NLAM, 3 LAMs, and 4 downsampling modules, wherein one NLAM is connected to a downsampling module, and one LAM is connected to a downsampling module. Optionally, the 1 NLAM, 3 LAMs, and the first 3 downsampling modules included in the spatial texture enhancement decoder Dt are consistent with the network structure of the 1 NLAM, 3 LAMs, and the first 3 downsampling modules included in the above-mentioned optical flow signal decoder Df, which are not repeated here. The number of channels included in the last downsampling module of the spatial texture enhancement decoder Dt is 3.

It should be noted that the above FIG. 8 is only an example, and the settings of the parameters in FIG. 8 are also only examples. The network structure of the spatial texture enhancement decoder Dt in the embodiment of the present application includes but is not limited to that shown in FIG. 8 .

Since the residual image _Xr can perform texture enhancement on the predicted image, in some embodiments, determining the target predicted image of the current image according to the P predicted images in S204-A includes the following step S204-A21:

S204-A21. Obtain a target prediction image based on P prediction images and residual images.

For example, if P=1, a target predicted image is obtained based on the predicted image and the residual image. For example, the predicted image and the residual image are added to generate the target predicted image.

For another example, if P is greater than 1, a weighted image is first determined based on the P predicted images; and then a target predicted image is determined based on the weighted image and the residual image.

The specific process of the decoding end determining the weighted image based on the P predicted images can refer to the specific description of S204-A11 above, which will not be repeated here.

For example, taking P=2 as an example, according to the above method, a first weight w1 corresponding to the first predicted image and a second weight w2 corresponding to the second predicted image are determined. Optionally, the first predicted image and the second predicted image are weighted according to the above formula (5) to obtain a weighted image _X3 . Then, the weighted image _X3 is enhanced using the residual image _Xr to obtain a target predicted image.

Exemplarily, the target predicted image X ₄ is generated according to the following formula (6):

X ₄ =X ₃ +X _r (6)

According to the above method, after the decoding end determines the target predicted image of the current image, the following step S204-B is performed.

S204-B, determining a reconstructed image of the current image according to the target predicted image.

In some embodiments, the target predicted image is compared with one or several previous reconstructed images of the current image to calculate the loss. If the loss is small, it means that the prediction accuracy of the target predicted image is high, and the target predicted image can be determined as the reconstructed image of the current image. If the above loss is large, it means that the prediction accuracy of the target predicted image is low. At this time, the reconstructed image of the current image can be determined based on one or several previous reconstructed images of the current image and the target predicted image. For example, the target predicted image and one or several previous reconstructed images of the current image are input into a neural network to obtain the reconstructed image of the current image.

In some embodiments, in order to further improve the accuracy of determining the reconstructed image, the embodiment of the present application further includes residual decoding. In this case, the above S204-B includes the following steps S204-B1 and S204-B2:

S204-B1, decoding the residual code stream to obtain a residual value of the current image;

S204-B2, obtaining a reconstructed image based on the target predicted image and the residual value.

In the embodiment of the present application, in order to improve the effect of reconstructing the image, the encoding end also generates a residual code stream by residual coding. Specifically, the encoding end determines the residual value of the current image, encodes the residual value to generate a residual code stream. Correspondingly, the decoding end decodes the residual code stream to obtain the residual value of the current image, and obtains the reconstructed image according to the target predicted image and the residual value.

The embodiment of the present application does not limit the specific representation form of the residual value of the current image.

In a possible implementation, the residual value of the current image is a matrix, and each element in the matrix is the residual value corresponding to each pixel in the current image. In this way, the decoding end can add the residual value and the predicted value corresponding to each pixel in the target predicted image pixel by pixel to obtain the reconstruction value of each pixel, and then obtain the reconstructed image of the current image. Taking the i-th pixel in the current image as an example, in the target predicted image, the predicted value corresponding to the i-th pixel is obtained, and the residual value corresponding to the i-th pixel is obtained from the residual value of the current image. Then, the predicted value and the residual value corresponding to the i-th pixel are added to obtain the reconstruction value corresponding to the i-th pixel. For each pixel in the current image, referring to the above i-th pixel, the reconstruction value corresponding to each pixel in the current image can be obtained, and the reconstruction value corresponding to each pixel in the current image constitutes the reconstructed image of the current image.

The embodiment of the present application does not limit the specific method in which the decoding end obtains the residual value of the current image, that is, the embodiment of the present application does not limit the residual encoding and decoding method adopted by both ends of the encoding and decoding.

In one example, the encoding end determines the target prediction image of the current image in the same manner as the decoding end, and then obtains the residual value of the current image according to the current image and the target prediction image, for example, the difference between the current image and the target prediction image is determined as the residual value of the current image. Then, the residual value of the current image is encoded to generate residual coding. Optionally, the residual value of the current image can be transformed to obtain a transformation coefficient, the transformation coefficient is quantized to obtain a quantization coefficient, and the quantization coefficient is encoded to obtain a residual code stream. Correspondingly, the decoding end decodes the residual code stream to obtain the residual value of the current image, for example, decodes the residual code stream to obtain a quantization coefficient, dequantizes and de-transforms the quantization coefficient to obtain the residual value of the current image. Then, according to the above method, the residual values corresponding to the target prediction image and the current image are added to obtain a reconstructed image of the current image.

In some embodiments, the encoding end may use a neural network method to process the current image and the target predicted image of the current image, generate a residual value of the current image, and encode the residual value of the current image to generate a residual code stream. Correspondingly, the decoding end decodes the residual code stream to obtain the residual value of the current image, and then, according to the above method, adds the residual values corresponding to the target predicted image and the current image to obtain a reconstructed image of the current image.

In the embodiment of the present application, the decoding end can obtain a reconstructed image of the current image according to the above method.

Optionally, the reconstructed image may be directly displayed.

Optionally, the reconstructed image may also be stored in a cache for use in decoding subsequent images.

The video decoding method provided by the embodiment of the present application is that the decoding end determines the quantized first feature information by decoding the first bit stream, and the first feature information is obtained by feature fusion of the current image and the previous reconstructed image of the current image; the quantized first feature information is subjected to multi-level time domain fusion to obtain a mixed space-time representation; motion compensation is performed on the previous reconstructed image according to the mixed space-time representation to obtain P predicted images of the current image, where P is a positive integer; and the reconstructed image of the current image is determined according to the P predicted images. In the present application, in order to improve the accuracy of the reconstructed image, the quantized first feature information is subjected to multi-level time domain fusion, that is, the quantized first feature information is not only fused with the feature information of the previous reconstructed image of the current image, but also the quantized first feature information is feature fused with multiple reconstructed images before the current image, so that when certain information in the previous reconstructed image of the current image is blocked, the blocked information can be obtained from several reconstructed images before the current image, thereby making the generated mixed space-time representation include more accurate, rich and detailed feature information. In this way, when motion compensation is performed on the previous reconstructed image based on the mixed spatiotemporal representation, P high-precision predicted images can be generated. Based on the high-precision P predicted images, the reconstructed image of the current image can be accurately obtained, thereby improving the video compression effect.

In an embodiment of the present application, an end-to-end neural network-based encoding and decoding framework is proposed, and the neural network-based encoding and decoding framework includes a neural network-based encoder and a neural network-based decoder. The decoding process of the embodiment of the present application is introduced below in combination with a possible neural network-based decoder of the present application.

FIG9 is a schematic diagram of the network structure of a neural network-based decoder according to an embodiment of the present application, including: an inverse transformation module, a recursive aggregation module and a hybrid motion compensation module.

The inverse transformation module is used to perform an inverse transformation on the quantized second feature information to obtain reconstructed feature information of the first feature information. Exemplarily, its network structure is shown in FIG3 .

The recursive aggregation module is used to perform multi-level time-domain fusion on the quantized first feature information to obtain a mixed spatiotemporal representation. Exemplarily, its network structure is shown in FIG4 .

The hybrid motion compensation module is used to perform hybrid motion compensation on the hybrid spatiotemporal representation to obtain a target predicted image of the current image. Exemplarily, the hybrid motion compensation module may include the first decoder shown in FIG5 and/or the second decoder shown in FIG6. Optionally, if the hybrid motion compensation module includes the first decoder and the second decoder, the hybrid motion compensation module may also include the third decoder shown in FIG7. In some embodiments, the hybrid motion compensation module may also include a fourth decoder as shown in FIG8.

Exemplarily, the embodiment of the present application is described by taking the example that the motion compensation module includes a first decoder, a second decoder, a third decoder and a fourth decoder.

Based on the neural network-based decoder shown in FIG. 9 above, a possible video decoding method of an embodiment of the present application is introduced in combination with FIG. 10 .

FIG10 is a schematic diagram of a video decoding process provided by an embodiment of the present application, as shown in FIG10 , including:

S301. Decode the second bit stream to obtain quantized second feature information.

The specific implementation process of the above S301 refers to the description of the above S201-A, which will not be repeated here.

S302: Perform an inverse transformation on the quantized second feature information through an inverse transformation module to obtain reconstructed feature information.

Exemplarily, the specific network structure of the inverse transformation module is shown in FIG3 , including two non-local self-attention modules and two upsampling modules.

For example, the decoding end inputs the quantized second feature information into the inverse transformation module for inverse transformation, and the inverse transformation module outputs the reconstructed feature information. The specific implementation process of the above S302 refers to the description of the above S201-B1, which will not be repeated here.

S303: Determine the probability distribution of the reconstructed feature information.

S304: predicting the probability distribution of the quantized first feature information according to the probability distribution of the reconstructed feature information.

S305 . Decode the first bitstream according to the probability distribution of the quantized first feature information to obtain the quantized first feature information.

For the specific implementation process of S303 to S305, please refer to the specific description of S201-B2, S201-B3 and S201-C, which will not be repeated here.

S306 , performing multi-level time-domain fusion on the quantized first feature information through a recursive aggregation module to obtain a mixed time-space representation.

Optionally, the recursive aggregation module is formed by stacking at least one spatiotemporal recursive network.

Exemplarily, the network structure of the recursive aggregation module is shown in FIG4 .

For example, the decoding end inputs the quantized first feature information into the recursive aggregation module, so that the recursive aggregation module fuses the quantized first feature information with the implicit feature information of the recursive aggregation module at the previous moment, and then outputs a mixed spatiotemporal representation. The specific implementation process of the above S306 refers to the description of the above S202-A, which will not be repeated here.

S307 . Process the mixed spatiotemporal representation through a first decoder to obtain a first predicted image.

After the mixed spatiotemporal representation is obtained according to the above S306, the mixed spatiotemporal representation and the previous reconstructed image are input into the mixed motion compensation module for motion mixed compensation to obtain the target predicted image of the current image.

Specifically, the mixed spatiotemporal representation is processed by a first decoder to determine optical flow motion information, and motion compensation is performed on a previous reconstructed image according to the optical flow motion information to obtain a first predicted image.

Optionally, the network structure of the first decoder is shown in FIG5 .

For the specific implementation process of S307, please refer to the specific description of S203-A1 and S203-A2, which will not be repeated here.

S308 . Process the mixed spatiotemporal representation through a second decoder to obtain a second predicted image.

Specifically, spatial features of a previous reconstructed image are extracted through SFE to obtain reference feature information; the reference feature information and the mixed spatiotemporal representation are input into a second decoder so that the offset performs motion compensation on the reference feature information to obtain a second predicted image.

Optionally, the network structure of the second decoder is shown in FIG6 .

For the specific implementation process of the above S308, please refer to the specific description of S203-B1 to S203-B3, which will not be repeated here.

S309 , processing the mixed spatiotemporal representation through a third decoder to obtain a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image.

Specifically, the mixed spatiotemporal representation is input into a third decoder for adaptive masking to obtain a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image.

Optionally, the network structure of the third decoder is shown in FIG7 .

For the specific implementation process of the above S309, refer to the specific description of the second method in the above S204-A11, which will not be repeated here.

S310 . Weight the first predicted image and the second predicted image according to the first weight and the second weight to obtain a weighted image.

For example, the product of the first weight and the first predicted image is added to the product of the second weight and the second predicted image to obtain a weighted image.

S311 . Process the mixed spatiotemporal representation through a fourth decoder to obtain a residual image of the current image.

Specifically, the mixed spatiotemporal representation is input into the fourth decoder for processing to obtain a residual image of the current image.

Optionally, the network structure of the fourth decoder is shown in FIG8 .

The specific implementation process of the above S311 refers to the specific description of the above S204-A12, which will not be repeated here.

S312: Determine a target prediction image according to the weighted image and the residual image.

For example, the weighted image and the residual image are added together to determine the target prediction image.

S313: Decode the residual code stream to obtain a residual value of the current image.

S314: Obtain a reconstructed image according to the target predicted image and the residual value.

For the specific implementation process of the above S313 and S314, refer to the specific description of the above S204-B1 and S204-B2, which will not be repeated here.

In the embodiment of the present application, when decoding is performed by the neural network-based decoder shown in FIG9 , the quantized first feature information is subjected to multi-level time-domain fusion, that is, the quantized first feature information is subjected to feature fusion with multiple reconstructed images before the current image, so that the generated mixed spatiotemporal representation includes more accurate, rich and detailed feature information. In this way, motion compensation is performed on the previous reconstructed image based on the mixed spatiotemporal representation to generate multiple decoding information. For example, the multiple decoding information includes the first predicted image, the second predicted image, the weights corresponding to the first predicted image and the second predicted image, and the residual image. In this way, when the target predicted image of the current image is determined based on the multiple decoding information, the accuracy of the target predicted image can be effectively improved, and then the reconstructed image of the current image can be accurately obtained based on the accurate predicted image, thereby improving the video compression effect.

The above describes the video decoding method involved in the embodiment of the present application. On this basis, the following describes the video encoding method involved in the present application for the encoding end.

Fig. 11 is a schematic diagram of a flow chart of a video encoding method provided by an embodiment of the present application. The execution subject of the embodiment of the present application may be the encoder shown in Fig. 1 above.

As shown in FIG11 , the method of the embodiment of the present application includes:

S401 , performing feature fusion on a current image and a previous reconstructed image of the current image to obtain first feature information.

An embodiment of the present application proposes a neural network-based encoder, which is obtained by end-to-end training of the neural network-based encoder and the neural network-based decoder.

In the embodiment of the present application, the previous reconstructed image of the current image can be understood as the previous frame image before the current image in the video sequence, and the previous frame image has been decoded and reconstructed.

Since the current image _Xt and the previous reconstructed image of the current image There is a strong similarity between these two adjacent frames. Therefore, when encoding, the encoder replaces the current image _Xt with the previous reconstructed image of the current image. Perform feature fusion to obtain the first feature information. For example, the encoder combines the current image _Xt and the previous reconstructed image of the current image Cascade connection between channels Get the cascaded input data X _cat , X _{t and X t} are 3-channel video frame inputs in the SRGB domain, and X _cat synthesizes the two video frames by stacking channels one by one to obtain an input signal with 6 channels. Next, feature extraction is performed on the cascaded image X _cat to obtain first feature information.

The embodiment of the present application does not limit the specific manner in which the encoder extracts features from _Xcat , for example, including at least one of multi-layer convolution, residual connection, dense connection and other feature extraction methods.

In some embodiments, the encoder performs Q non-local attention transformations and Q downsampling on the cascaded image to obtain first feature information, where Q is a positive integer.

For example, the encoder inputs the cascaded 6-channel high-dimensional input signal X _cat into a spatiotemporal feature extraction module (STFE) for multi-layer feature transformation and extraction.

Optionally, the spatiotemporal feature extraction module includes Q non-local attention modules and Q downsampling modules. Among them, the non-local attention module is used to implement non-local attention transformation, and the downsampling module is used to implement downsampling. Exemplarily, as shown in FIG12, a downsampling module is connected after a non-local attention module. In actual application, the encoder inputs the cascaded 6-channel high-dimensional input signal X _cat into the STFE, and the first non-local attention module in the STFE performs non-local attention feature transformation extraction on X _cat to obtain feature information 11, and then inputs the feature information 11 into the first downsampling module for downsampling to obtain feature information 12. Then, the feature information 12 is input into the second non-local attention module for non-local attention feature transformation extraction to obtain feature information 13, and then the feature information 13 is input into the second downsampling module for downsampling to obtain feature information 14. By analogy, the feature information output by the Qth downsampling module is obtained, and the feature information is determined as the first feature information X _F.

The embodiment of the present application does not limit the specific value of Q.

Optionally, Q=4.

S402: quantize the first feature information to obtain quantized first feature information.

The first feature information obtained above is of floating point type, for example, represented by a 32-bit floating point number. Further, in order to reduce the encoding cost, the encoding end quantizes the first feature information obtained above to obtain the quantized first feature information.

Exemplarily, the encoding end uses a rounding function Round(.) to quantize the first feature information.

In some embodiments, during the model training process, during forward propagation, the first feature information is quantized using the method shown in the following formula (7):

Among them, U(-0.5,0.5) is a uniform noise distribution of plus or minus 0.5, which is used to approximate the actual rounding quantization function Round(.).

During the training process, the derivative of formula (7) is used to obtain the corresponding back-propagation gradient of 1, which is used as the back-propagation gradient to update the model.

S403: Encode the quantized first feature information to obtain a first code stream.

Method 1: The encoder directly uses the probability distribution of the quantized first feature information to encode the quantized first feature information to obtain the first bitstream

The quantized first feature information includes a large amount of redundant information. When the quantized first feature information is directly encoded, many codewords are required for encoding and the encoding cost is large. In order to reduce the encoding cost, in some embodiments, the encoding end performs feature transformation according to the first feature information to obtain the second feature information, and quantizes the second feature information and then encodes it to obtain the second code stream; decodes the second code stream to obtain the quantized second feature information, and determines the probability distribution of the quantized first feature information according to the quantized second feature information; and then encodes the quantized first feature information according to the probability distribution of the quantized first feature information to obtain the first code stream. In other words, in order to reduce the encoding cost, the encoding end determines the super-prior feature information corresponding to the first feature information, that is, the second feature information, and determines the probability distribution of the quantized first feature information based on the second feature information. Since the second feature information is the super-prior feature information of the first feature information, it includes less redundancy. In this way, the probability distribution of the quantized first feature information is determined based on the second feature information with less redundancy, and the first feature information is encoded using the probability distribution, which can reduce the encoding cost of the first feature information.

Based on the above description, the encoding end may encode the quantized first feature information through the following steps of method 2 to obtain a first code stream.

Mode 2, the above S403 includes the following steps S403-A1 to S403-A4:

S403-A1. Perform feature transformation according to the first feature information to obtain second feature information.

In the second method, in order to reduce the coding cost, the encoder performs feature transformation on the first feature information to obtain the super-prior feature information of the first feature information, that is, the second feature information, and uses the second feature information to determine the probability distribution of the quantized first feature information, and encodes the quantized first feature information using the probability distribution to obtain the first code stream. At the same time, in order to enable the decoder to decode the first code stream using the same probability distribution as the encoding, the second feature information is encoded to obtain the second code stream. That is to say, in the second method, the encoder generates two code streams, namely the first code stream and the second code stream.

In the embodiment of the present application, the encoding end performs feature transformation according to the first feature information to obtain the second feature information in the following ways, but not limited to:

Method 1, performing N non-local attention transformations and N downsampling on the first feature information to obtain the second feature information.

Method 2, performing N non-local attention transformations and N downsampling on the quantized first feature information to obtain the second feature information.

That is to say, the encoding end can perform N non-local attention transformations and N downsamplings on the first feature information or the quantized first feature information to obtain the second feature information.

S403-A2, quantize and then encode the second characteristic information to obtain a second code stream.

For example, the second feature information is quantized to obtain quantized second feature information; the probability distribution of the quantized second feature information is determined; and according to the probability distribution of the quantized second feature information, the quantized second feature information is encoded to obtain a second code stream.

In the present application, since the second feature information is the super-prior feature information of the first feature information and includes less redundant information, the encoding end directly uses the probability distribution of the quantized second feature information during encoding to encode the quantized second feature information to obtain a second code stream.

S403-A3, decode the second code stream to obtain quantized second feature information, and determine the probability distribution of the quantized first feature information according to the quantized second feature information.

In the embodiment of the present application, the encoding end performs arithmetic decoding on the second super-prior code stream to restore the quantized super-prior spatiotemporal features. That is, the quantized second feature information. Then, according to the quantized second feature information, the probability distribution of the quantized first feature information is determined, and then the quantized first feature information is encoded according to the probability distribution of the quantized first feature information to obtain the first code stream.

The following is an introduction to the process of determining the probability distribution of the quantized first feature information according to the quantized second feature information in the above S403-A3.

In some embodiments, determining the probability distribution of the quantized first feature information according to the quantized second feature information in S403-A3 comprises the following steps:

S403-A31. Perform an inverse transformation on the quantized second feature information to obtain reconstructed feature information.

In this implementation, the encoding end performs an inverse transformation on the quantized second feature information to obtain the reconstructed feature information, wherein the inverse transformation method adopted by the encoding end can be understood as the inverse operation of the transformation method adopted by the encoding end. For example, the encoding end performs N times of feature extraction on the first feature information to obtain the second feature information, and correspondingly, at this time, the encoding end performs N times of reverse feature extraction on the quantized second feature information to obtain the inversely transformed feature information, which is recorded as the reconstructed feature information.

The embodiment of the present application does not limit the inverse transformation method adopted by the encoding end.

In some embodiments, the inverse transformation method adopted by the encoding end includes N times of feature extraction. That is, the encoding end performs N times of feature extraction on the obtained quantized second feature information to obtain reconstructed feature information.

In some embodiments, the inverse transformation method adopted by the encoding end includes N times of feature extraction and N times of upsampling. That is, the encoding end performs N times of feature extraction and N times of upsampling on the obtained quantized second feature information to obtain reconstructed feature information.

The embodiment of the present application does not limit the specific execution order of the above-mentioned N feature extractions and N upsamplings.

In an example, the encoding end may first perform N consecutive feature extractions on the quantized second feature information, and then perform N consecutive upsamplings.

In another example, the N times of feature extraction and N times of upsampling are performed alternately, that is, one upsampling is performed after one feature extraction.

It should be noted that the embodiment of the present application does not limit the N-time feature extraction method adopted by the encoding end, for example, at least one of the feature extraction methods including multi-layer convolution, residual connection, dense connection, etc.

In some embodiments, the encoding end performs N non-local attention transformations and N upsamplings on the quantized second feature information to obtain reconstructed feature information, where N is a positive integer.

Since the non-local attention method can achieve more efficient feature extraction, the extracted features can retain more information, and the calculation efficiency is high, therefore, in the embodiment of the present application, the encoding end uses the non-local attention method to extract features from the quantized second feature information to achieve fast and accurate feature extraction of the quantized second feature information. In addition, when the encoding end generates the second feature information based on the first feature information, it performs N downsampling. Therefore, at this time, the encoding end performs N upsampling correspondingly during the inverse transformation, so that the reconstructed feature information obtained by reconstruction is consistent with the size of the first feature information.

In some embodiments, as shown in FIG3 , the encoding end obtains the reconstructed feature information through an inverse transformation module, and the inverse transformation module includes N non-local attention modules and N upsampling modules.

S403-A32, determine the probability distribution of the reconstructed feature information.

From the above, it can be seen that the second quantized feature information is obtained by transforming the first feature information. Through the above steps, the encoding end dequantizes the quantized second feature information to obtain the reconstructed feature information. Therefore, the reconstructed feature information can be understood as the reconstructed information of the first feature information, that is, the probability distribution of the reconstructed feature information is similar to or related to the probability distribution of the quantized first feature information. In this way, the encoding end can first determine the probability distribution of the reconstructed feature information, and then predict the probability distribution of the first feature information after quantization based on the probability distribution of the reconstructed feature information.

In some embodiments, the probability distribution of the reconstructed feature information is a normal distribution or a Gaussian distribution. In this case, the process of determining the probability distribution of the reconstructed feature information is to determine the mean and variance matrix of the reconstructed feature information based on each eigenvalue in the reconstructed feature information, and generate a Gaussian distribution of the reconstructed feature information based on the mean and variance matrix.

S403-A33. Determine the probability distribution of the quantized first feature information according to the probability distribution of the reconstructed feature information.

For example, according to the probability distribution of the reconstructed feature information, the probability of the encoded pixel in the quantized first feature information is predicted; according to the probability of the encoded pixel in the quantized first feature information, the probability distribution of the quantized first feature information is obtained.

S403-A4. Encode the quantized first feature information according to the probability distribution of the quantized first feature information to obtain a first code stream.

According to the above steps, after the probability distribution of the quantized first feature information is determined, the quantized first feature information is encoded using the probability distribution to obtain a first code stream.

In some embodiments, the embodiment of the present application further includes a step of determining a reconstructed image of the current image, that is, the embodiment of the present application further includes the following S404:

S404: Determine a reconstructed image of the current image.

In some embodiments, the above S404 includes the following steps:

S404-A, performing multi-level time domain fusion on the quantized first feature information to obtain a mixed time-space representation.

In some embodiments, the quantized first feature information is feature information obtained by quantizing the first feature information at the encoding end.

In some embodiments, the quantized first feature information is reconstructed by the encoding end. For example, the encoding end decodes the second code stream to obtain the quantized second feature information, and determines the probability distribution of the quantized first feature information based on the quantized second feature information. Exemplarily, the encoding end obtains the probability distribution of the quantized first feature information according to the above methods S403-A31 to S403-A33, and then uses the probability distribution of the quantized first feature information to decode the first code stream to obtain the quantized first feature information.

Next, the encoding end performs multi-level time domain fusion on the quantized first feature information obtained above to obtain a mixed time-space representation.

In an embodiment of the present application, in order to improve the accuracy of the reconstructed image, the quantized first feature information is subjected to multi-level time domain fusion, that is, the quantized first feature information is not only fused with the feature information of the previous reconstructed image of the current image, but also the quantized first feature information is fused with the multiple reconstructed images before the current image, for example, the reconstructed images at multiple times such as time t-1, time t-2..., time t-k are fused with the quantized first feature information. In this way, when certain information in the previous reconstructed image of the current image is blocked, the blocked information can be obtained from several reconstructed images before the current image, thereby making the generated mixed spatiotemporal representation include more accurate, rich and detailed feature information. In this way, when motion compensation is performed on the previous reconstructed image based on the mixed spatiotemporal representation to generate P predicted images of the current image, the accuracy of the generated predicted image can be improved, and then the reconstructed image of the current image can be accurately obtained based on the accurate predicted image, thereby improving the video compression effect.

The embodiment of the present application does not limit the specific method of performing multi-level time-domain fusion on the quantized first feature information at the encoding end to obtain a mixed time-space representation.

In some embodiments, the encoding end mixes the spatiotemporal representations through a recursive aggregation module, that is, the above S404-A includes the following step S404-A1:

S404-A1. The encoding end fuses the quantized first feature information with the implicit feature information of the recursive aggregation module at the previous moment through the recursive aggregation module to obtain a mixed spatiotemporal representation.

The recursive aggregation module of the embodiment of the present application will learn and retain the deep-level feature information learned from the feature information each time it generates a mixed spatiotemporal representation, and use the learned deep-level features as implicit feature information for the next generation of the mixed spatiotemporal representation, thereby improving the accuracy of the generated mixed spatiotemporal representation. That is to say, in the embodiment of the present application, the implicit feature information of the recursive aggregation module at the previous moment includes the feature information of multiple reconstructed images before the current image learned by the recursive aggregation module. In this way, the encoding end fuses the quantized first feature information with the implicit feature information of the recursive aggregation module at the previous moment through the recursive aggregation module, so as to generate a more accurate, rich and detailed mixed spatiotemporal representation.

The embodiment of the present application does not limit the specific network structure of the recursive aggregation module, for example, it can be any network structure that can implement the above functions.

In some embodiments, the recursive aggregation module is formed by stacking at least one spatiotemporal recursive network ST-LSTM. In this case, the expression formula of the above-mentioned mixed spatiotemporal representation Gt is as shown in the above formula (1).

S404-B, performing motion compensation on the previous reconstructed image according to the hybrid spatiotemporal representation to obtain P predicted images of the current image, where P is a positive integer.

From the above, it can be seen that the hybrid spatiotemporal representation of the embodiment of the present application fuses the feature information of the current image and multiple reconstructed images before the current image, so that motion compensation is performed on the previous reconstructed image according to the hybrid spatiotemporal representation, and P accurate predicted images of the current image can be obtained.

The embodiment of the present application does not limit the specific number of the generated P predicted images. That is, in the embodiment of the present application, the encoding end can use different methods to perform motion compensation on the previous reconstructed image according to the mixed spatiotemporal representation to obtain P predicted images of the current image.

The embodiment of the present application does not limit the specific manner in which the encoding end performs motion compensation on the previous reconstructed image according to the mixed spatiotemporal representation.

In some embodiments, the P predicted images include a first predicted image, and the first predicted image is obtained by the encoder using optical flow motion compensation. In this case, S404-B includes the following steps S404-B1 and S404-B2:

S404-B1, determining optical flow motion information according to the mixed spatiotemporal representation;

S404-B2. Perform motion compensation on the previous reconstructed image according to the optical flow motion information to obtain a first predicted image.

The embodiment of the present application does not limit the specific manner in which the encoding end determines the optical flow motion information based on the mixed spatiotemporal representation.

In some embodiments, the encoding end obtains the optical flow motion information through a pre-trained neural network model, that is, the neural network model can predict the optical flow motion information based on the mixed spatiotemporal representation. In some embodiments, the neural network model can be called a first decoder, or an optical flow signal decoder Df. The encoding end inputs the mixed spatiotemporal representation Gt into the optical flow signal decoder Df to predict the optical flow motion information, and obtains the optical flow motion information _fx,y output by the optical flow signal decoder Df. Optionally, fx _,y is the optical flow motion information with a channel of 2.

Exemplarily, the generation formula of f _x,y is shown in the above formula (2).

The embodiment of the present application does not limit the specific network structure of the optical flow signal decoder Df.

In some embodiments, the optical flow signal decoder Df is composed of multiple NLAMs and multiple upsampling modules. Exemplarily, as shown in Figure 5, the optical flow signal decoder Df includes 1 NLAM, 3 LAMs and 4 downsampling modules, wherein one NLAM is connected to a downsampling module and one LAM is connected to a downsampling module.

It should be noted that the above FIG. 5 is only an example, and the settings of the parameters in FIG. 5 are also only examples. The network structure of the optical flow signal decoder Df in the embodiment of the present application includes but is not limited to that shown in FIG. 5 .

After the encoder generates the optical flow motion information f _x,y , it uses the optical flow motion information f _x,y to reconstruct the previous image Motion compensation is performed to obtain a first predicted image X ₁ .

In the embodiment of the present application, the encoder performs motion compensation on the previous reconstructed image according to the optical flow motion information to obtain the first predicted image without limiting the specific method. For example, the encoder uses the optical flow motion information f _x,y to compensate the previous reconstructed image. Linear interpolation is performed, and the image generated by interpolation is recorded as the first predicted image X ₁ .

In a possible implementation, the encoder obtains the first predicted image X ₁ through the following formula (3).

In this implementation, as shown in FIG5 , the encoder uses the optical flow motion information f _x,y to reconstruct the previous image by performing a warping operation. Motion compensation is performed to obtain a first predicted image X ₁ .

In some embodiments, the P predicted images include a second predicted image, and the second predicted image is obtained by the decoding end using an offset motion compensation method. In this case, the S404-B includes the following steps S404-B-1 to S404-B-3:

S404-B-1. Obtain an offset corresponding to the current image according to the mixed spatiotemporal representation;

S404-B-2, extracting spatial features from the previous reconstructed image to obtain reference feature information;

S404-B-3. Use the offset to perform motion compensation on the reference feature information to obtain a second predicted image.

The embodiment of the present application does not limit the specific manner in which the encoding end obtains the offset corresponding to the current image based on the mixed spatiotemporal representation.

In some embodiments, the encoder obtains the offset corresponding to the current image through a pre-trained neural network model, that is, the neural network model can predict the offset based on the mixed spatiotemporal representation, and the offset is lossy offset information. In some embodiments, the neural network model can be called a second decoder, or a variable convolution decoder Dm. The encoder inputs the mixed spatiotemporal representation Gt into the variable convolution decoder Dm to predict the offset information.

At the same time, the encoding end extracts spatial features from the previous reconstructed image to obtain reference feature information. For example, the encoding end extracts spatial features from the previous reconstructed image through a spatial feature extraction module SFE to obtain reference feature information.

Next, the encoder uses the offset to perform motion compensation on the extracted reference feature information to obtain a second predicted image of the current image.

In the embodiment of the present application, the encoding end uses the offset to perform motion compensation on the extracted reference feature information, and the specific method of obtaining the second predicted image of the current image is not limited.

In a possible implementation, the encoder uses the offset to perform deformable convolution-based motion compensation on the reference feature information to obtain a second predicted image.

In some embodiments, since the transformable convolution can generate an offset corresponding to the current image based on the mixed space-time representation, in an embodiment of the present application, the encoding end inputs the mixed space-time representation Gt and the reference feature information into the transformable convolution. The transformable convolution generates an offset corresponding to the current image based on the mixed space-time representation Gt, and applies the offset to the reference feature information for motion compensation, thereby obtaining a second predicted image.

Based on this, exemplary, as shown in FIG6 , the variable convolution decoder Dm of the embodiment of the present application includes a transformable convolution DCN, and the encoding end reconstructs the previous image The mixed spatiotemporal representation Gt and the reference feature information are then input into the transformable convolution DCN to extract the offset and perform motion compensation to obtain the second predicted image X ₂ .

Exemplarily, the encoder generates the second predicted image X ₂ by using the above formula (4).

The embodiment of the present application does not limit the specific network structure of the optical flow signal decoder Df.

In some embodiments, as shown in Figure 6, in order to further improve the accuracy of the second predicted image, the variable convolution decoder Dm includes, in addition to the transformable convolution DCN, 1 NLAM, 3 LAMs and 4 downsampling modules, wherein one NLAM is connected to a downsampling module and one LAM is connected to a downsampling module.

It should be noted that the above-mentioned Figure 6 is only an example, and the settings of the parameters in Figure 6 are also only examples. The network structure of the variable convolution decoder Dm of the embodiment of the present application includes but is not limited to that shown in Figure 6.

In the embodiment of the present application, as shown in FIG6 , the encoding end first reconstructs the previous image The mixed spatiotemporal representation Gt and the reference feature information are then input into the transformable convolution DCN in the variable convolution decoder Dm for offset extraction and motion compensation to obtain feature information, which is then input into the NLAM. After feature extraction by the NLAM, 3 LAMs and 4 downsampling modules, the second predicted image X ₂ is finally restored.

According to the above method, the encoding end may determine P predicted images, for example, determine a first predicted image and a second predicted image, and then perform the following step S204.

S404-C. Determine a reconstructed image of the current image according to the P predicted images.

In some embodiments, if the P predicted images include one predicted image, a reconstructed image of the current image is determined according to the predicted image.

For example, the predicted image is compared with one or several previous reconstructed images of the current image to calculate the loss. If the loss is small, it means that the prediction accuracy of the predicted image is high, and the predicted image can be determined as the reconstructed image of the current image.

For another example, if the above-mentioned loss is large, it means that the prediction accuracy of the predicted image is low. At this time, the reconstructed image of the current image can be determined based on the previous one or several reconstructed images of the current image and the predicted image. For example, the predicted image and the previous one or several reconstructed images of the current image are input into a neural network to obtain the reconstructed image of the current image.

In some embodiments, the above S404-C includes the following steps S404-C-A and S404-C-B:

S404-C-A. Determine a target predicted image for the current image based on P predicted images.

In this implementation, the encoding end first determines the target prediction image of the current image based on P prediction images, and then realizes the reconstructed image of the current image based on the target prediction image of the current image, thereby improving the determination accuracy of the reconstructed image.

The embodiment of the present application does not limit the specific method of determining the target predicted image of the current image based on P predicted images.

In some embodiments, if P=1, the one predicted image is determined as the target predicted image of the current image.

In some embodiments, if P is greater than 1, S404-C-A includes S404-C-A11 and S404-C-A12:

S404-C-A11, determining a weighted image according to the P predicted images;

In this implementation, if multiple predicted images of the current image are generated according to the above method, for example, when a first predicted image and a second predicted image are generated, these P predicted images are weighted to generate a weighted image, and then the target predicted image is obtained based on the weighted image.

The embodiment of the present application does not limit the specific method of determining the weighted image based on P predicted images.

For example, weights corresponding to P predicted images are determined; and according to the weights corresponding to the P predicted images, the P predicted images are weighted to obtain a weighted image.

Exemplarily, if the P predicted images include a first predicted image and a second predicted image, the encoding end determines a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image, and weights the first predicted image and the second predicted image according to the first weight and the second weight to obtain a weighted image.

The methods for determining the weights corresponding to the P predicted images include but are not limited to the following:

Method 1: The weights corresponding to the above P predicted images are preset weights. Assume that P=2, that is, the first weight corresponding to the first predicted image and the second weight corresponding to the second predicted image can be that the first weight is equal to the second weight, or the ratio of the first weight to the second weight is 1/2, 1/4, 1/2, 1/3, 2/1, 3/1, 4/1, etc.

In the second method, the encoder performs adaptive masking based on the mixed spatiotemporal representation to obtain weights corresponding to P predicted images.

Exemplarily, the encoder generates weights corresponding to P predicted images through a neural network model, and the neural network model is pre-trained and can be used to generate weights corresponding to P predicted images. In some embodiments, the neural network model is also called a third decoder or an adaptive mask compensation decoder _Dw . Specifically, the encoder inputs the mixed spatiotemporal representation into the adaptive mask compensation decoder _Dw for adaptive masking to obtain weights corresponding to the P predicted images. For example, the encoder inputs the mixed spatiotemporal representation Gt into the adaptive mask compensation decoder _Dw for adaptive masking, and the adaptive mask compensation decoder _Dw outputs a first weight w1 of the first predicted image and a second weight w2 of the second predicted image, and performs the first weight w1 and the second weight w2 on the first predicted image _X1 and the second predicted image _X2 obtained above, and can adaptively select the corresponding information representing different regions in the predicted frame, thereby generating a weighted image.

Exemplarily, the weighted image X ₃ is generated according to the above formula (5).

In some embodiments, the weights corresponding to the above-mentioned P predicted images are a matrix, including the weights corresponding to each pixel in the predicted image. When generating a weighted image, for each pixel in the current image, the predicted values and their weights corresponding to the pixel in the P predicted images are weighted to obtain the weighted predicted value of the pixel. In this way, the weighted prediction values corresponding to each pixel in the current image constitute the weighted image of the current image.

The embodiment of the present application does not limit the specific network structure of the adaptive mask compensation decoder _Dw .

In some embodiments, as shown in FIG. 7 , the adaptive mask compensation decoder D _w includes 1 NLAM, 3 LAMs, 4 downsampling modules and a sigmoid function, wherein a downsampling module is connected after one NLAM and a downsampling module is connected after one LAM.

It should be noted that FIG. 7 is only an example, and the settings of the parameters in FIG. 7 are also only examples. The network structure of the adaptive mask compensation decoder _Dw in the embodiment of the present application includes but is not limited to that shown in FIG. 7.

In this implementation, the encoding end weights the P predicted images according to the above method, and after obtaining the weighted images, executes the following S404-C-A12.

S404-C-A12. Obtain a target prediction image based on the weighted image.

For example, the weighted image is determined as the target prediction image.

In some embodiments, the encoding end may also obtain a residual image of the current image according to the mixed spatiotemporal representation.

Exemplarily, the encoder obtains a residual image of the current image through a neural network model, and the neural network model is pre-trained and can be used to generate a residual image of the current image. In some embodiments, the neural network model is also called a fourth decoder or a spatial texture enhancement decoder Dt. Specifically, the encoder inputs the mixed spatiotemporal representation into the spatial texture enhancement decoder Dt for spatial texture enhancement, and obtains a residual image _Xr = D_t ( _Gt ) of the current image, and the residual image _Xr can perform texture enhancement on the predicted image.

In the embodiment of the present application, there is no limitation on the specific network structure of the above-mentioned spatial texture enhancement decoder Dt.

In some embodiments, as shown in FIG8 , the spatial texture enhancement decoder Dt includes 1 NLAM, 3 LAMs, and 4 downsampling modules, wherein one NLAM is connected to a downsampling module, and one LAM is connected to a downsampling module.

It should be noted that the above FIG. 8 is only an example, and the settings of the parameters in FIG. 8 are also only examples. The network structure of the spatial texture enhancement decoder Dt in the embodiment of the present application includes but is not limited to that shown in FIG. 8 .

Since the residual image _Xr can perform texture enhancement on the predicted image, in some embodiments, determining the target predicted image of the current image according to the P predicted images in S404-CA includes the following step S404-C-A21:

S404-C-A21. Obtain a target prediction image based on P prediction images and residual images.

For example, if P=1, a target predicted image is obtained based on the predicted image and the residual image. For example, the predicted image and the residual image are added to generate the target predicted image.

For another example, if P is greater than 1, a weighted image is first determined based on the P predicted images; and then a target predicted image is determined based on the weighted image and the residual image.

The specific process of the encoder determining the weighted image based on the P predicted images can refer to the specific description of S204-A11 above, which will not be repeated here.

For example, taking P=2 as an example, according to the above method, a first weight w1 corresponding to the first predicted image and a second weight w2 corresponding to the second predicted image are determined. Optionally, the first predicted image and the second predicted image are weighted according to the above formula (5) to obtain a weighted image _X3 . Then, the weighted image _X3 is enhanced using the residual image _Xr to obtain a target predicted image.

Exemplarily, the target predicted image X ₄ is generated according to the above formula (6).

According to the above method, after the encoding end determines the target predicted image of the current image, the following steps S404-C-B are executed.

S404-C-B. Determine a reconstructed image of the current image based on the target predicted image.

In some embodiments, the target predicted image is compared with one or several previous reconstructed images of the current image to calculate the loss. If the loss is small, it means that the prediction accuracy of the target predicted image is high, and the target predicted image can be determined as the reconstructed image of the current image. If the above loss is large, it means that the prediction accuracy of the target predicted image is low. At this time, the reconstructed image of the current image can be determined based on one or several previous reconstructed images of the current image and the target predicted image. For example, the target predicted image and one or several previous reconstructed images of the current image are input into a neural network to obtain the reconstructed image of the current image.

In some embodiments, in order to further improve the accuracy of determining the reconstructed image, the encoder determines the residual value of the current image based on the current image and the target predicted image; encodes the residual value to obtain a residual code stream. At this time, the embodiment of the present application also includes residual decoding, and the above S404-C-B includes the following steps S404-C-B1 and S404-C-B2:

S404-C-B1, decoding the residual code stream to obtain a residual value of the current image;

S404-C-B2. Obtain a reconstructed image based on the target predicted image and the residual value.

In the embodiment of the present application, in order to improve the effect of reconstructing the image, the encoding end also generates a residual code stream by residual coding. Specifically, the encoding end determines the residual value of the current image, encodes the residual value to generate a residual code stream. Correspondingly, the encoding end decodes the residual code stream to obtain the residual value of the current image, and obtains the reconstructed image according to the target predicted image and the residual value.

The embodiment of the present application does not limit the specific representation form of the residual value of the current image.

In a possible implementation, the residual value of the current image is a matrix, and each element in the matrix is the residual value corresponding to each pixel in the current image. In this way, the encoding end can add the residual value and the predicted value corresponding to each pixel in the target predicted image pixel by pixel to obtain the reconstruction value of each pixel, and then obtain the reconstructed image of the current image. Taking the i-th pixel in the current image as an example, in the target predicted image, the predicted value corresponding to the i-th pixel is obtained, and the residual value corresponding to the i-th pixel is obtained from the residual value of the current image. Then, the predicted value and the residual value corresponding to the i-th pixel are added to obtain the reconstruction value corresponding to the i-th pixel. For each pixel in the current image, referring to the above i-th pixel, the reconstruction value corresponding to each pixel in the current image can be obtained, and the reconstruction value corresponding to each pixel in the current image constitutes the reconstructed image of the current image.

The embodiment of the present application does not limit the specific method in which the encoding end obtains the residual value of the current image, that is, the embodiment of the present application does not limit the residual encoding and decoding method adopted by both ends of the encoding and decoding.

In one example, the encoding end determines the target prediction image of the current image, and then obtains the residual value of the current image based on the current image and the target prediction image, for example, the difference between the current image and the target prediction image is determined as the residual value of the current image. Then, the residual value of the current image is encoded to generate residual coding. Optionally, the residual value of the current image can be transformed to obtain a transformation coefficient, the transformation coefficient is quantized to obtain a quantization coefficient, and the quantization coefficient is encoded to obtain a residual code stream. Correspondingly, the encoding end decodes the residual code stream to obtain the residual value of the current image, for example, decodes the residual code stream to obtain a quantization coefficient, dequantizes and de-transforms the quantization coefficient to obtain the residual value of the current image. Then, according to the above method, the residual values corresponding to the target prediction image and the current image are added to obtain a reconstructed image of the current image.

In some embodiments, the encoding end may use a neural network method to process the current image and the target predicted image of the current image to generate a residual value of the current image, and encode the residual value of the current image to generate a residual code stream.

In the embodiment of the present application, the encoding end can obtain a reconstructed image of the current image according to the above method.

Optionally, the reconstructed image may be displayed directly.

Optionally, the reconstructed image may also be stored in a cache for use in encoding subsequent images.

The video encoding method provided in the embodiment of the present application is that the encoding end obtains the first feature information by performing feature fusion on the current image and the previous reconstructed image of the current image; quantizes the first feature information to obtain the quantized first feature information; encodes the quantized first feature information to obtain the first code stream, so that the decoding end decodes the first code stream, determines the quantized first feature information, performs multi-level time domain fusion on the quantized first feature information to obtain a mixed time-space representation; performs motion compensation on the previous reconstructed image according to the mixed time-space representation to obtain P predicted images of the current image; and then determines the reconstructed image of the current image according to the P predicted images. That is, in order to improve the accuracy of the reconstructed image, the present application performs multi-level time domain fusion on the quantized first feature information, for example, the quantized first feature information is feature fused with multiple reconstructed images before the current image, so that when certain information in the previous reconstructed image of the current image is blocked, the blocked information can be obtained from several reconstructed images before the current image, so that the generated mixed time-space representation includes more accurate, rich and detailed feature information. In this way, when motion compensation is performed on the previous reconstructed image based on the mixed spatiotemporal representation, P high-precision predicted images can be generated. Based on the high-precision P predicted images, the reconstructed image of the current image can be accurately obtained, thereby improving the video compression effect.

In an embodiment of the present application, an end-to-end neural network-based encoding and decoding framework is proposed, and the neural network-based encoding and decoding framework includes a neural network-based encoder and a neural network-based decoder. The encoding process of the embodiment of the present application is introduced below in combination with a possible neural network-based encoder of the present application.

12 is a schematic diagram of the network structure of a neural network-based encoder according to an embodiment of the present application, including: a spatiotemporal feature extraction module, an inverse transformation module, a recursive aggregation module and a hybrid motion compensation module.

The spatiotemporal feature extraction module is used to perform feature extraction and down-sampling on the cascaded current image and the previous reconstructed image to obtain first feature information.

The inverse transformation module is used to perform an inverse transformation on the quantized second feature information to obtain reconstructed feature information of the first feature information. Exemplarily, its network structure is shown in FIG3 .

The recursive aggregation module is used to perform multi-level time-domain fusion on the quantized first feature information to obtain a mixed spatiotemporal representation. Exemplarily, its network structure is shown in FIG4 .

The hybrid motion compensation module is used to perform hybrid motion compensation on the hybrid spatiotemporal representation to obtain a target predicted image of the current image. Exemplarily, the hybrid motion compensation module may include the first decoder shown in FIG5 and/or the second decoder shown in FIG6. Optionally, if the hybrid motion compensation module includes the first decoder and the second decoder, the hybrid motion compensation module may also include the third decoder shown in FIG7. In some embodiments, the hybrid motion compensation module may also include a fourth decoder as shown in FIG8.

Exemplarily, the embodiment of the present application is described by taking the example that the motion compensation module includes a first decoder, a second decoder, a third decoder and a fourth decoder.

Based on the neural network-based encoder shown in FIG. 12 above, a possible video encoding method of an embodiment of the present application is introduced in combination with FIG. 13 .

FIG. 13 is a schematic diagram of a video encoding process provided by an embodiment of the present application, as shown in FIG. 13 , including:

S501: Perform feature fusion on a current image and a previous reconstructed image of the current image to obtain first feature information.

For example, the encoder combines the current image _Xt and the previous reconstructed image of the current image The channels are cascaded to obtain X _cat , and then, feature extraction is performed on the cascaded image X _cat to obtain first feature information.

The specific implementation process of the above S501 refers to the description of the above S401 and will not be repeated here.

S502: quantize the first feature information to obtain quantized first feature information.

The specific implementation process of the above S502 refers to the description of the above S402 and will not be repeated here.

S503: Perform feature transformation according to the first feature information to obtain second feature information.

The specific implementation process of the above S503 refers to the description of the above S403-A1, which will not be repeated here.

S504: quantize and then encode the second feature information to obtain a second bit stream.

The specific implementation process of the above S504 refers to the description of the above S403-A2, which will not be repeated here.

S505: Decode the second code stream to obtain quantized second feature information.

The specific implementation process of the above S505 refers to the description of the above S403-A3, which will not be repeated here.

S506 . Perform an inverse transformation on the quantized second feature information through an inverse transformation module to obtain reconstructed feature information.

Exemplarily, the specific network structure of the inverse transformation module is shown in FIG3 , including two non-local self-attention modules and two upsampling modules.

For example, the decoding end inputs the quantized second feature information into the inverse transformation module for inverse transformation, and the inverse transformation module outputs the reconstructed feature information.

The specific implementation process of the above S506 refers to the description of the above S403-A31, which will not be repeated here.

S507: Determine the probability distribution of the reconstructed feature information.

S508: Predict the probability distribution of the quantized first feature information according to the probability distribution of the reconstructed feature information.

S509: Encode the quantized first feature information according to the probability distribution of the quantized first feature information to obtain a first code stream.

The specific implementation process of the above S507 to S509 refers to the description of the above S403-A32, S403-A33 and S403-A4, which will not be repeated here.

The embodiment of the present application also includes a process of determining a reconstructed image.

S510: Decode the first bitstream according to the probability distribution of the quantized first feature information to obtain the quantized first feature information.

S511. Perform multi-level time-domain fusion on the quantized first feature information through a recursive aggregation module to obtain a mixed time-space representation.

Optionally, the recursive aggregation module is formed by stacking at least one spatiotemporal recursive network.

Exemplarily, the network structure of the recursive aggregation module is shown in FIG4 .

For example, the decoding end inputs the quantized first feature information into the recursive aggregation module, so that the recursive aggregation module fuses the quantized first feature information with the implicit feature information of the recursive aggregation module at the previous moment, and then outputs a mixed spatiotemporal representation. The specific implementation process of the above S511 refers to the description of the above S404-A, which will not be repeated here.

S512: Process the mixed spatiotemporal representation through a first decoder to obtain a first predicted image.

After the mixed spatiotemporal representation is obtained according to the above S512, the mixed spatiotemporal representation and the previous reconstructed image are input into the mixed motion compensation module for motion mixed compensation to obtain the target predicted image of the current image.

Specifically, the mixed spatiotemporal representation is processed by a first decoder to determine optical flow motion information, and motion compensation is performed on a previous reconstructed image according to the optical flow motion information to obtain a first predicted image.

Optionally, the network structure of the first decoder is shown in FIG5 .

For the specific implementation process of the above S512, please refer to the specific description of the above S404-B1 and S404-B2, which will not be repeated here.

S513: Process the mixed spatiotemporal representation through a second decoder to obtain a second predicted image.

Specifically, spatial features of a previous reconstructed image are extracted through SFE to obtain reference feature information; the reference feature information and the mixed spatiotemporal representation are input into a second decoder so that the offset performs motion compensation on the reference feature information to obtain a second predicted image.

Optionally, the network structure of the second decoder is shown in FIG6 .

For the specific implementation process of the above S513, please refer to the specific description of S404-B-1 to S404-B-3, which will not be repeated here.

S514 . Process the mixed spatiotemporal representation through a third decoder to obtain a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image.

Specifically, the mixed spatiotemporal representation is input into the third decoder for adaptive masking to obtain a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image.

Optionally, the network structure of the third decoder is shown in FIG7 .

For the specific implementation process of the above S514, please refer to the specific description of method 2 in the above S404-C-A11, which will not be repeated here.

S515 . Weight the first predicted image and the second predicted image according to the first weight and the second weight to obtain a weighted image.

For example, the product of the first weight and the first predicted image is added to the product of the second weight and the second predicted image to obtain a weighted image.

S516: Process the mixed spatiotemporal representation through a fourth decoder to obtain a residual image of the current image.

Specifically, the mixed spatiotemporal representation is input into the fourth decoder for processing to obtain a residual image of the current image.

Optionally, the network structure of the fourth decoder is shown in FIG8 .

For the specific implementation process of the above S516, please refer to the specific description of the above S404-C-A12, which will not be repeated here.

S517: Determine a target prediction image according to the weighted image and the residual image.

For example, the weighted image and the residual image are added together to determine the target prediction image.

S518: Decode the residual code stream to obtain a residual value of the current image.

S519: Obtain a reconstructed image according to the target predicted image and the residual value.

For the specific implementation process of the above S518 and S519, please refer to the specific description of the above S404-C-B1 and S404-C-B2, which will not be repeated here.

In the embodiment of the present application, when encoding is performed by the neural network-based encoder shown in FIG12, the quantized first feature information is subjected to multi-level time-domain fusion, that is, the quantized first feature information is subjected to feature fusion with multiple reconstructed images before the current image, so that the generated mixed spatiotemporal representation includes more accurate, rich and detailed feature information. In this way, motion compensation is performed on the previous reconstructed image based on the mixed spatiotemporal representation to generate multiple decoding information. For example, the multiple decoding information includes the first predicted image, the second predicted image, the weights corresponding to the first predicted image and the second predicted image, and the residual image. In this way, when the target predicted image of the current image is determined based on the multiple decoding information, the accuracy of the target predicted image can be effectively improved, and then the reconstructed image of the current image can be accurately obtained based on the accurate predicted image, thereby improving the video compression effect.

It should be understood that Figures 2 to 13 are merely examples of the present application and should not be construed as limitations to the present application.

The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

It should also be understood that in the various method embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in the embodiments of the present application, the term "and/or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. Specifically, A and/or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character "/" in this article generally indicates that the objects associated before and after are in an "or" relationship.

The above text describes in detail a method embodiment of the present application in conjunction with Figures 2 to 13 , and the following text describes in detail a device embodiment of the present application in conjunction with Figures 14 to 17 .

FIG. 14 is a schematic block diagram of a video decoding device provided in an embodiment of the present application.

As shown in FIG. 14 , the video decoding device 10 includes:

A decoding unit 11 is used to decode the first bit stream and determine quantized first feature information, where the first feature information is obtained by performing feature fusion of a current image and a previous reconstructed image of the current image;

A fusion unit 12, configured to perform multi-level time-domain fusion on the quantized first feature information to obtain a mixed time-space representation;

A compensation unit 13, configured to perform motion compensation on the previous reconstructed image according to the mixed spatiotemporal representation to obtain P predicted images of the current image, where P is a positive integer;

The reconstruction unit 14 is used to determine a reconstructed image of the current image according to the P predicted images.

In some embodiments, the fusion unit 12 is specifically configured to fuse the quantized first feature information with the implicit feature information of the recursive aggregation module at a previous moment through a recursive aggregation module to obtain the mixed spatiotemporal representation.

Optionally, the recursive aggregation module is formed by stacking at least one spatiotemporal recursive network.

In some embodiments, the P predicted images include a first predicted image, and the compensation unit 13 is specifically used to determine the optical flow motion information according to the mixed spatiotemporal representation; perform motion compensation on the previous reconstructed image according to the optical flow motion information to obtain the first predicted image.

In some embodiments, the P predicted images include a second predicted image, and the compensation unit 13 is specifically used to obtain an offset corresponding to the current image based on the mixed spatiotemporal representation; perform spatial feature extraction on the previous reconstructed image to obtain reference feature information; and use the offset to perform motion compensation on the reference feature information to obtain the second predicted image.

In some embodiments, the compensation unit 13 is specifically configured to use the offset to perform deformable convolution-based motion compensation on the reference feature information to obtain the second predicted image.

In some embodiments, the reconstruction unit 14 is used to determine a target prediction image of the current image based on the P prediction images; and determine a reconstructed image of the current image based on the target prediction image.

In some embodiments, the reconstruction unit 14 is used to determine a weighted image according to the P predicted images; and obtain the target predicted image according to the weighted image.

In some embodiments, the reconstruction unit 14 is further configured to obtain a residual image of the current image according to the mixed spatiotemporal representation; and obtain the target predicted image according to the P predicted images and the residual image.

In some embodiments, the reconstruction unit 14 is specifically configured to determine a weighted image according to the P predicted images; and determine the target predicted image according to the weighted image and the residual image.

In some embodiments, the reconstruction unit 14 is specifically used to determine the weights corresponding to the P predicted images; and weight the P predicted images according to the weights corresponding to the P predicted images to obtain the weighted image.

In some embodiments, the reconstruction unit 14 is specifically configured to perform adaptive masking according to the mixed spatiotemporal representation to obtain weights corresponding to the P predicted images.

In some embodiments, if the P predicted images include a first predicted image and a second predicted image, the reconstruction unit 14 is specifically used to determine a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image; and weight the first predicted image and the second predicted image according to the first weight and the second weight to obtain the weighted image.

In some embodiments, the reconstruction unit 14 is specifically configured to decode the residual code stream to obtain the residual value of the current image; and obtain the reconstructed image according to the target predicted image and the residual value.

In some embodiments, the decoding unit 11 is specifically used to decode the second code stream to obtain quantized second feature information, where the second feature information is obtained by performing feature transformation on the first feature information; determine the probability distribution of the quantized first feature information based on the quantized second feature information; decode the first code stream based on the probability distribution of the quantized first feature information to obtain the quantized first feature information.

In some embodiments, the decoding unit 11 is specifically used to perform an inverse transformation on the quantized second feature information to obtain reconstructed feature information; determine the probability distribution of the reconstructed feature information; and predict the probability distribution of the quantized first feature information based on the probability distribution of the reconstructed feature information.

In some embodiments, the decoding unit 11 is specifically used to perform N non-local attention transformations and N upsamplings on the quantized second feature information to obtain the reconstructed feature information, where N is a positive integer.

In some embodiments, the decoding unit 11 is specifically used to predict the probability of the encoded pixels in the first feature information after quantization according to the probability distribution of the reconstructed feature information; and obtain the probability distribution of the first feature information after quantization according to the probability of the encoded pixels in the first feature information after quantization.

It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the video decoding device 10 shown in FIG. 14 may correspond to the corresponding subject in the method for executing the embodiment of the present application, and the aforementioned and other operations and/or functions of each unit in the video decoding device 10 are respectively for implementing the corresponding processes in each method such as the method, and for the sake of brevity, no further description is given here.

FIG15 is a schematic block diagram of a video encoding device provided in an embodiment of the present application.

As shown in FIG. 15 , the video encoding apparatus 20 includes:

A fusion unit 21 is used to perform feature fusion on a current image and a previous reconstructed image of the current image to obtain first feature information;

A quantization unit 22, configured to quantize the first feature information to obtain quantized first feature information;

The encoding unit 23 is used to encode the quantized first feature information to obtain the first code stream.

In some embodiments, the fusion unit 21 is specifically used to perform channel cascading on the current image and the reconstructed image to obtain a cascaded image; and perform feature extraction on the cascaded image to obtain the first feature information.

In some embodiments, the fusion unit 21 is specifically used to perform Q non-local attention transformations and Q downsampling on the cascaded image to obtain the first feature information, where Q is a positive integer.

In some embodiments, the encoding unit 23 is further used to perform feature transformation according to the first feature information to obtain second feature information; quantize and then encode the second feature information to obtain a second code stream; decode the second code stream to obtain the quantized second feature information, and determine the probability distribution of the quantized first feature information according to the quantized second feature information; encode the quantized first feature information according to the probability distribution of the quantized first feature information to obtain a first code stream.

In some embodiments, the encoding unit 23 is specifically used to perform N non-local attention transformations and N downsampling on the first feature information to obtain the second feature information, where N is a positive integer.

In some embodiments, the encoding unit 23 is specifically used to perform N non-local attention transformations and N downsamplings on the quantized first feature information to obtain the second feature information.

In some embodiments, the encoding unit 23 is further used to quantize the second feature information to obtain the quantized second feature information; determine the probability distribution of the quantized second feature information; and encode the quantized second feature information according to the probability distribution of the quantized second feature information to obtain the second code stream.

In some embodiments, the encoding unit 23 is specifically used to perform an inverse transformation on the quantized second feature information to obtain reconstructed feature information; determine the probability distribution of the reconstructed feature information; and determine the probability distribution of the quantized first feature information based on the probability distribution of the reconstructed feature information.

In some embodiments, the encoding unit 23 is specifically used to perform N non-local attention transformations and N upsamplings on the quantized second feature information to obtain the reconstructed feature information, where N is a positive integer.

In some embodiments, the encoding unit 23 is specifically used to determine the probability of the encoded pixels in the quantized first feature information according to the probability distribution of the reconstructed feature information; and obtain the probability distribution of the quantized first feature information according to the probability of the encoded pixels in the quantized first feature information.

In some embodiments, the encoding unit 23 is further configured to determine a reconstructed image of the current image.

In some embodiments, the encoding unit 23 is specifically used to perform multi-level time-domain fusion on the quantized first feature information to obtain a mixed space-time representation; perform motion compensation on the previous reconstructed image according to the mixed space-time representation to obtain P predicted images of the current image, where P is a positive integer; and determine the reconstructed image of the current image based on the P predicted images.

In some embodiments, the encoding unit 23 is specifically configured to fuse the quantized first feature information with the implicit feature information of the recursive aggregation module at a previous moment through a recursive aggregation module to obtain the mixed spatiotemporal representation.

Optionally, the recursive aggregation module is formed by stacking at least one spatiotemporal recursive network.

In some embodiments, the P predicted images include a first predicted image, and the encoding unit 23 is specifically used to determine optical flow motion information according to the mixed spatiotemporal representation; perform motion compensation on the previous reconstructed image according to the optical flow motion information to obtain the first predicted image.

In some embodiments, the P predicted images include a second predicted image, and the encoding unit 23 is specifically used to obtain an offset corresponding to the current image based on the mixed spatiotemporal representation; perform spatial feature extraction on the previous reconstructed image to obtain reference feature information; and use the offset to perform motion compensation on the reference feature information to obtain the second predicted image.

In some embodiments, the encoding unit 23 is specifically configured to use the offset to perform deformable convolution-based motion compensation on the reference feature information to obtain the second predicted image.

In some embodiments, the encoding unit 23 is specifically configured to determine a target prediction image of the current image according to the P prediction images; and determine a reconstructed image of the current image according to the target prediction image.

In some embodiments, the encoding unit 23 is specifically configured to determine a weighted image according to the P predicted images; and obtain the target predicted image according to the weighted image.

In some embodiments, the encoding unit 23 is further configured to obtain a residual image of the current image according to the mixed spatiotemporal representation; and obtain the target predicted image according to the P predicted images and the residual image.

In some embodiments, if the P is greater than 1, the encoding unit 23 is specifically configured to determine a weighted image according to the P predicted images; and determine the target predicted image according to the weighted image and the residual image.

In some embodiments, the encoding unit 23 is specifically configured to determine weights corresponding to the P predicted images; and weight the P predicted images according to the weights corresponding to the P predicted images to obtain the weighted image.

In some embodiments, the encoding unit 23 is specifically configured to perform adaptive masking according to the mixed spatiotemporal representation to obtain weights corresponding to the P predicted images.

In some embodiments, if the P predicted images include a first predicted image and a second predicted image, the encoding unit 23 is specifically used to determine the P predicted images, determine a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image; and weight the first predicted image and the second predicted image according to the first weight and the second weight to obtain the weighted image.

In some embodiments, the encoding unit 23 is further configured to determine a residual value of the current image according to the current image and the target predicted image; and encode the residual value to obtain a residual code stream.

In some embodiments, the encoding unit 23 is specifically configured to decode the residual code stream to obtain a residual value of the current image; and obtain the reconstructed image according to the target predicted image and the residual value.

It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the video encoding device 20 shown in FIG. 15 may correspond to the corresponding subject in the method for executing the embodiment of the present application, and the aforementioned and other operations and/or functions of each unit in the video encoding device 20 are respectively for implementing the corresponding processes in each method such as the method, and for the sake of brevity, no further description is given here.

The above describes the device and system of the embodiment of the present application from the perspective of the functional unit in conjunction with the accompanying drawings. It should be understood that the functional unit can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software units. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and/or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software units in the decoding processor to perform. Optionally, the software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

FIG. 16 is a schematic block diagram of an electronic device provided in an embodiment of the present application.

As shown in FIG. 16 , the electronic device 30 may be a video encoder or a video decoder as described in the embodiment of the present application, and the electronic device 30 may include:

The memory 33 and the processor 32, the memory 33 is used to store the computer program 34 and transmit the program code 34 to the processor 32. In other words, the processor 32 can call and run the computer program 34 from the memory 33 to implement the method in the embodiment of the present application.

For example, the processor 32 may be configured to execute the steps in the above method according to the instructions in the computer program 34 .

In some embodiments of the present application, the processor 32 may include but is not limited to:

General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

In some embodiments of the present application, the memory 33 includes but is not limited to:

Volatile memory and/or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

In some embodiments of the present application, the computer program 34 may be divided into one or more units, which are stored in the memory 33 and executed by the processor 32 to complete the method provided by the present application. The one or more units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 34 in the electronic device 30.

As shown in FIG. 16 , the electronic device 30 may further include:

The transceiver 33 may be connected to the processor 32 or the memory 33 .

The processor 32 may control the transceiver 33 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 33 may include a transmitter and a receiver. The transceiver 33 may further include an antenna, and the number of antennas may be one or more.

It should be understood that the various components in the electronic device 30 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

FIG. 17 is a schematic block diagram of a video encoding and decoding system 40 provided in an embodiment of the present application.

As shown in FIG. 17 , the video encoding and decoding system 40 may include: a video encoder 41 and a video decoder 42 , wherein the video encoder 41 is used to execute the video encoding method involved in the embodiment of the present application, and the video decoder 42 is used to execute the video decoding method involved in the embodiment of the present application.

In some embodiments, the present application also provides a code stream, which is obtained by the above encoding method.

The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the unit is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (50)

1. A video decoding method, comprising:

   Decoding a first code stream, and determining quantized first characteristic information, wherein the first characteristic information is obtained by carrying out characteristic fusion on a current image and a reconstructed image before the current image;

   performing multi-stage time domain fusion on the quantized first characteristic information to obtain a hybrid space-time representation;

   Performing motion compensation on the previous reconstructed image according to the mixed space-time representation to obtain P predicted images of the current image, wherein P is a positive integer;

   and determining a reconstructed image of the current image according to the P predicted images.
2. The method of claim 1, wherein the performing multi-level time-domain fusion on the quantized first feature information to obtain a hybrid space-time representation comprises:

   And fusing the quantized first characteristic information with implicit characteristic information of the recursion aggregation module at the previous moment through the recursion aggregation module to obtain the mixed space-time characterization.
3. The method of claim 2, wherein the recursive aggregation module is stacked by at least one spatio-temporal recursive network.
4. The method of claim 1, wherein the P predicted pictures comprise a first predicted picture, wherein the motion compensating the previous reconstructed picture based on the hybrid spatio-temporal characterization results in P predicted pictures for the current picture, comprising:

   determining optical flow motion information according to the mixed space-time representation;

   And performing motion compensation on the previous reconstructed image according to the optical flow motion information to obtain the first predicted image.
5. The method of claim 1, wherein the P predicted pictures comprise a second predicted picture, wherein the motion compensating the previous reconstructed picture based on the hybrid spatio-temporal characterization results in P predicted pictures for the current picture, comprising:

   Obtaining the offset corresponding to the current image according to the mixed space-time representation;

   extracting spatial features of the previous reconstructed image to obtain reference feature information;

   And performing motion compensation on the reference characteristic information by using the offset to obtain the second predicted image.
6. The method of claim 5, wherein the motion compensating the reference feature information using the offset to obtain the second predicted image comprises:

   And performing deformable convolution-based motion compensation on the reference characteristic information by using the offset to obtain the second predicted image.
7. The method according to any one of claims 1-6, wherein said determining a reconstructed image of the current image from the P predicted images comprises:

   Determining a target predicted image of the current image according to the P predicted images;

   and determining a reconstructed image of the current image according to the target predicted image.
8. The method of claim 7, wherein if P is greater than 1, the determining the target predicted picture for the current picture from the P predicted pictures comprises:

   determining a weighted image according to the P predicted images;

   and obtaining the target prediction image according to the weighted image.
9. The method of claim 7, wherein the method further comprises:

   Obtaining a residual image of the current image according to the mixed space-time representation;

   The determining the target predicted image of the current image according to the P predicted images comprises:

   And obtaining the target predicted image according to the P predicted images and the residual image.
10. The method according to claim 9, wherein if the P is greater than 1, the obtaining the target prediction image from the P prediction images and the residual image comprises:

    determining a weighted image according to the P predicted images;

    and determining the target prediction image according to the weighted image and the residual image.
11. The method according to claim 8 or 10, wherein said determining weighted pictures from said P predicted pictures comprises:

    determining weights corresponding to the P predicted images;

    and weighting the P predicted images according to the weights corresponding to the P predicted images to obtain the weighted images.
12. The method of claim 11, wherein determining weights for the P predicted pictures comprises:

    And carrying out self-adaptive masking according to the mixed space-time characterization to obtain weights corresponding to the P predicted images.
13. The method of claim 11, wherein if the P predicted pictures include a first predicted picture and a second predicted picture, the determining weights corresponding to the P predicted pictures comprises:

    determining a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image;

    the step of weighting the P predicted images according to the weights corresponding to the P predicted images to obtain the weighted images includes:

    And weighting the first predicted image and the second predicted image according to the first weight and the second weight to obtain the weighted image.
14. The method of claim 7, wherein the determining a reconstructed image of the current image from the target predicted image comprises:

    decoding the residual code stream to obtain a residual value of the current image;

    And obtaining the reconstructed image according to the target predicted image and the residual value.
15. The method of any of claims 1-6, wherein decoding the first code stream to determine quantized first characteristic information comprises:

    Decoding a second code stream to obtain quantized second characteristic information, wherein the second characteristic information is obtained by carrying out characteristic transformation on the first characteristic information;

    Determining probability distribution of the quantized first characteristic information according to the quantized second characteristic information;

    and decoding the first code stream according to the quantized probability distribution of the first characteristic information to obtain the quantized first characteristic information.
16. The method of claim 15, wherein determining probability distribution information of the quantized first feature information based on the quantized second feature information comprises:

    Inversely transforming the quantized second characteristic information to obtain reconstructed characteristic information;

    determining probability distribution of the reconstruction feature information;

    and predicting the probability distribution of the quantized first characteristic information according to the probability distribution of the reconstructed characteristic information.
17. The method of claim 16, wherein the inverse transforming the quantized second feature information to obtain reconstructed feature information comprises:

    And carrying out N times of non-local attention transformation and N times of up-sampling on the quantized second characteristic information to obtain the reconstructed characteristic information, wherein N is a positive integer.
18. The method of claim 16, wherein predicting the quantized probability distribution of the first feature information based on the probability distribution of the reconstructed feature information comprises:

    Predicting the probability of the encoded pixels in the quantized first characteristic information according to the probability distribution of the reconstructed characteristic information;

    And obtaining the probability distribution of the quantized first characteristic information according to the probability of the encoded pixels in the quantized first characteristic information.
19. A video encoding method, comprising:

    performing feature fusion on a current image and a reconstructed image before the current image to obtain first feature information;

    Quantizing the first characteristic information to obtain quantized first characteristic information;

    And encoding the quantized first characteristic information to obtain a first code stream.
20. The method of claim 19, wherein the feature fusion of the current image and the reconstructed image before the current image to obtain the first feature information includes:

    carrying out channel cascade on the current image and the reconstructed image to obtain a cascade image;

    and extracting the characteristics of the cascaded images to obtain the first characteristic information.
21. The method of claim 20, wherein the feature extracting the concatenated image to obtain the first feature information includes:

    And carrying out Q times of non-local attention conversion and Q times of downsampling on the cascaded images to obtain the first characteristic information, wherein Q is a positive integer.
22. The method of claim 19, wherein encoding the quantized first characteristic information to obtain the first code stream comprises:

    Performing feature transformation according to the first feature information to obtain second feature information;

    quantizing and then encoding the second characteristic information to obtain a second code stream;

    decoding the second code stream to obtain quantized second characteristic information, and determining probability distribution of the quantized first characteristic information according to the quantized second characteristic information;

    And encoding the quantized first characteristic information according to the probability distribution of the quantized first characteristic information to obtain a first code stream.
23. The method of claim 22, wherein performing the feature transformation according to the first feature information to obtain second feature information includes:

    And carrying out N times of non-local attention conversion and N times of downsampling on the first characteristic information to obtain the second characteristic information, wherein N is a positive integer.
24. The method of claim 22, wherein performing the feature transformation according to the first feature information to obtain second feature information includes:

    And carrying out N times of non-local attention transformation and N times of downsampling on the quantized first characteristic information to obtain the second characteristic information.
25. The method of claim 22, wherein the quantizing the second characteristic information and then encoding to obtain a second code stream comprises:

    quantizing the second characteristic information to obtain quantized second characteristic information;

    determining a probability distribution of the quantized second feature information;

    And encoding the quantized second characteristic information according to the probability distribution of the quantized second characteristic information to obtain the second code stream.
26. The method of claim 22, wherein determining probability distribution information of the quantized first feature information based on the quantized second feature information comprises:

    Inversely transforming the quantized second characteristic information to obtain reconstructed characteristic information;

    determining probability distribution of the reconstruction feature information;

    and determining the probability distribution of the quantized first characteristic information according to the probability distribution of the reconstructed characteristic information.
27. The method of claim 26, wherein said inversely transforming the quantized second feature information to obtain reconstructed feature information comprises:

    And carrying out N times of non-local attention transformation and N times of up-sampling on the quantized second characteristic information to obtain the reconstructed characteristic information, wherein N is a positive integer.
28. The method of claim 26, wherein determining the quantized probability distribution of the first feature information based on the probability distribution of the reconstructed feature information comprises:

    Determining the probability of the encoded pixels in the quantized first characteristic information according to the probability distribution of the reconstructed characteristic information;

    And obtaining the probability distribution of the quantized first characteristic information according to the probability of the encoded pixels in the quantized first characteristic information.
29. The method according to any one of claims 19-28, further comprising:

    A reconstructed image of the current image is determined.
30. The method of claim 29, wherein said determining a reconstructed image of said current image comprises:

    performing multi-stage time domain fusion on the quantized first characteristic information to obtain a hybrid space-time representation;

    Performing motion compensation on the previous reconstructed image according to the mixed space-time representation to obtain P predicted images of the current image, wherein P is a positive integer;

    and determining a reconstructed image of the current image according to the P predicted images.
31. The method of claim 30, wherein the performing multi-level time-domain fusion on the quantized first feature information to obtain a hybrid space-time representation comprises:

    And fusing the quantized first characteristic information with implicit characteristic information of the recursion aggregation module at the previous moment through the recursion aggregation module to obtain the mixed space-time characterization.
32. The method of claim 31, wherein the recursive aggregation module is stacked by at least one spatio-temporal recursive network.
33. The method of claim 30, wherein the P predicted pictures comprise a first predicted picture, wherein the motion compensating the previous reconstructed picture based on the hybrid spatio-temporal characterization results in P predicted pictures for the current picture, comprising:

    determining optical flow motion information according to the mixed space-time representation;

    And performing motion compensation on the previous reconstructed image according to the optical flow motion information to obtain the first predicted image.
34. The method of claim 30, wherein the P predicted pictures comprise a second predicted picture, wherein the motion compensating the previous reconstructed picture based on the hybrid spatio-temporal characterization results in P predicted pictures for the current picture, comprising:

    Obtaining the offset corresponding to the current image according to the mixed space-time representation;

    extracting spatial features of the previous reconstructed image to obtain reference feature information;

    And performing motion compensation on the reference characteristic information by using the offset to obtain the second predicted image.
35. The method of claim 34, wherein the motion compensating the reference feature information using the offset to obtain the second predicted image comprises:

    And performing deformable convolution-based motion compensation on the reference characteristic information by using the offset to obtain the second predicted image.
36. The method according to any one of claims 30-35, wherein said determining a reconstructed image of said current image from said P predicted images comprises:

    Determining a target predicted image of the current image according to the P predicted images;

    and determining a reconstructed image of the current image according to the target predicted image.
37. The method of claim 36, wherein if P is greater than 1, the determining the target predicted picture for the current picture from the P predicted pictures comprises:

    determining a weighted image according to the P predicted images;

    and obtaining the target prediction image according to the weighted image.
38. The method of claim 36, wherein the method further comprises:

    Obtaining a residual image of the current image according to the mixed space-time representation;

    The determining the target predicted image of the current image according to the P predicted images comprises:

    And obtaining the target predicted image according to the P predicted images and the residual image.
39. The method of claim 38, wherein if P is greater than 1, the obtaining the target predicted image from the P predicted images and the residual image comprises:

    determining a weighted image according to the P predicted images;

    and determining the target prediction image according to the weighted image and the residual image.
40. The method of claim 37 or 39, wherein said determining a weighted image from said P predicted images comprises:

    determining weights corresponding to the P predicted images;

    and weighting the P predicted images according to the weights corresponding to the P predicted images to obtain the weighted images.
41. The method of claim 40, wherein determining weights for the P predicted pictures comprises:

    And carrying out self-adaptive masking according to the mixed space-time characterization to obtain weights corresponding to the P predicted images.
42. The method of claim 41, wherein if the P predicted pictures include a first predicted picture and a second predicted picture, the determining weights for the P predicted pictures comprises:

    Determining the P predicted images, and determining a first weight corresponding to the first predicted image and a second weight corresponding to the second predicted image;

    the step of weighting the P predicted images according to the weights corresponding to the P predicted images to obtain the weighted images includes:

    And weighting the first predicted image and the second predicted image according to the first weight and the second weight to obtain the weighted image.
43. The method of claim 36, wherein the method further comprises:

    Determining a residual value of the current image according to the current image and the target predicted image;

    and encoding the residual error value to obtain a residual error code stream.
44. A method according to claim 43, wherein said determining a reconstructed image of the current image from the target predicted image comprises:

    decoding the residual code stream to obtain a residual value of the current image;

    And obtaining the reconstructed image according to the target predicted image and the residual value.
45. A video decoding apparatus, comprising:

    The decoding unit is used for decoding the first code stream, and determining quantized first characteristic information, wherein the first characteristic information is obtained by carrying out characteristic fusion on a current image and a reconstructed image before the current image;

    the fusion unit is used for carrying out multi-stage time domain fusion on the quantized first characteristic information to obtain a hybrid space-time characterization;

    The compensation unit is used for performing motion compensation on the previous reconstructed image according to the mixed space-time representation to obtain P predicted images of the current image, wherein P is a positive integer;

    And the reconstruction unit is used for determining a reconstructed image of the current image according to the P predicted images.
46. A video encoding apparatus, comprising:

    The fusion unit is used for carrying out feature fusion on the current image and a reconstructed image before the current image to obtain first feature information;

    the quantization unit is used for quantizing the first characteristic information to obtain quantized first characteristic information;

    and the encoding unit is used for encoding the quantized first characteristic information to obtain a first code stream.
47. A video codec system comprising a video encoder and a video decoder;

    The video decoder for performing the video decoding method of any of claims 1-19;

    The video encoder is configured to perform the video encoding method of any of claims 20-44.
48. An electronic device, comprising: a memory, a processor;

    the memory is used for storing a computer program;

    The processor being adapted to execute the computer program to implement the method of any of the preceding claims 1 to 19 or 20 to 44.
49. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are for implementing the method of any of claims 1 to 19 or 20 to 44.
50. A code stream comprising the code stream obtained by the method of any one of claims 20 to 44.