CN118590650A - A decoding and encoding method, device and equipment thereof - Google Patents

Abstract

The application provides a decoding and encoding method, a device and equipment thereof, wherein the decoding method comprises the following steps: acquiring a standard deviation corresponding to the current image block through a probability super-parameter decoding network based on a first code stream corresponding to the current image block; determining a probability distribution model based on the standard deviation, and decoding a second code stream corresponding to the current image block based on the probability distribution model to obtain decoded image features; determining residual characteristics corresponding to the current image block based on the decoded image characteristics; determining initial image features corresponding to the current image block based on the residual features; acquiring an entropy gradient offset step length corresponding to a current image block; performing feature enhancement on the initial image features based on the entropy gradient offset step length to obtain target image features; based on the target image characteristics, obtaining a reconstructed image block corresponding to the current image block through a synthesis transformation network. By the technical scheme of the application, the coding performance and decoding performance can be improved.

Description

A decoding and encoding method, device and equipment thereof

Technical Field

The present application relates to the field of coding and decoding technology, and in particular to a decoding and coding method, device and equipment thereof.

Background Art

In order to save space, video images are transmitted after being encoded. Complete video encoding can include prediction, transformation, quantization, entropy coding, filtering and other processes. For the prediction process, the prediction process can include intra-frame prediction and inter-frame prediction. Inter-frame prediction refers to the use of the correlation in the video time domain to use the pixels of the adjacent encoded images to predict the current pixel, so as to effectively remove the redundancy in the video time domain. Intra-frame prediction refers to the use of the correlation in the video space domain to use the pixels of the encoded blocks of the current frame image to predict the current pixel, so as to remove the redundancy in the video space domain.

With the rapid development of deep learning, deep learning has achieved success in many high-level computer vision problems, such as image classification and target detection. Deep learning has also gradually begun to be applied in the field of encoding and decoding, that is, neural networks can be used to encode and decode images. Although the encoding and decoding methods based on neural networks have shown great performance potential, they still have problems such as poor encoding performance, poor decoding performance and high complexity.

Summary of the invention

In view of this, the present application provides a decoding and encoding method, device and equipment thereof to improve encoding performance and decoding performance.

The present application provides a decoding method, which is applied to a decoding end, and the method includes:

Based on the first bitstream corresponding to the current image block, a standard deviation corresponding to the current image block is obtained through a probabilistic hyperparameter decoding network; a probability distribution model is determined based on the standard deviation, and a second bitstream corresponding to the current image block is decoded based on the probability distribution model to obtain decoded image features; based on the decoded image features, a residual feature corresponding to the current image block is determined; based on the residual feature, an initial image feature corresponding to the current image block is determined;

Obtaining the entropy gradient offset step corresponding to the current image block;

Performing feature enhancement on the initial image features based on the entropy gradient offset step to obtain target image features;

Based on the target image feature, a reconstructed image block corresponding to the current image block is obtained through a synthetic transformation network.

The present application provides a coding method, which is applied to a coding end, and the method includes:

Based on the first bitstream corresponding to the current image block, a standard deviation corresponding to the current image block is obtained through a probabilistic hyperparameter decoding network; a probability distribution model is determined based on the standard deviation, and a second bitstream corresponding to the current image block is decoded based on the probability distribution model to obtain decoded image features; based on the decoded image features, a residual feature corresponding to the current image block is determined; based on the residual feature, an initial image feature corresponding to the current image block is determined;

For each entropy gradient offset step, the initial image feature is enhanced based on the entropy gradient offset step to obtain a target image feature; based on the target image feature, a reconstructed image block corresponding to the entropy gradient offset step is obtained through a synthetic transformation network;

Based on the current image block and the reconstructed image block corresponding to each entropy gradient offset step, the entropy gradient offset step corresponding to the current image block is selected from all entropy gradient offset steps, and the entropy gradient offset step corresponding to the current image block is encoded in the header information code stream corresponding to the current image block.

The present application provides a decoding device, applied to a decoding end, the device comprising:

A determination module is used to obtain a standard deviation corresponding to the current image block through a probabilistic hyperparameter decoding network based on a first bitstream corresponding to the current image block; determine a probability distribution model based on the standard deviation, and decode the second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features; determine a residual feature corresponding to the current image block based on the decoded image features; and determine an initial image feature corresponding to the current image block based on the residual feature;

An acquisition module, used to acquire the entropy gradient offset step corresponding to the current image block;

A processing module, used for performing feature enhancement on the initial image features based on the entropy gradient offset step to obtain target image features;

The acquisition module is further used to acquire the reconstructed image block corresponding to the current image block through a synthetic transformation network based on the target image feature.

The present application provides an encoding device, applied to an encoding end, the device comprising:

A determination module is used to obtain a standard deviation corresponding to the current image block through a probabilistic hyperparameter decoding network based on a first bitstream corresponding to the current image block; determine a probability distribution model based on the standard deviation, and decode the second bitstream corresponding to the current image block based on the probability distribution model to obtain decoded image features; determine a residual feature corresponding to the current image block based on the decoded image features; and determine an initial image feature corresponding to the current image block based on the residual feature;

A processing module, configured to perform feature enhancement on the initial image feature based on each entropy gradient offset step to obtain a target image feature; and obtain a reconstructed image block corresponding to the entropy gradient offset step through a synthetic transformation network based on the target image feature;

A selection module, configured to select, based on the current image block and a reconstructed image block corresponding to each entropy gradient offset step, an entropy gradient offset step corresponding to the current image block from all entropy gradient offset step lengths;

The encoding module is used to encode the entropy gradient offset step corresponding to the current image block in the header information code stream corresponding to the current image block.

The present application provides a decoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

The processor is used to execute machine executable instructions to implement the above decoding method.

The present application provides an encoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

The processor is used to execute machine executable instructions to implement the above encoding method.

The present application provides an electronic device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-mentioned decoding method; or, the processor is used to execute the machine-executable instructions to implement the above-mentioned encoding method.

The present application provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the above-mentioned decoding method is implemented; or, the above-mentioned encoding method is implemented.

It can be seen from the above technical solutions that in the embodiment of the present application, the initial image features corresponding to the current image block can be obtained, and the entropy gradient offset step corresponding to the current image block can be obtained. The target image features are determined based on the initial image features and the entropy gradient offset step, and the reconstructed image block corresponding to the current image block is obtained based on the target image features, thereby proposing an end-to-end video image compression method, which can realize the encoding and decoding of video images based on neural networks, and achieve the purpose of improving the encoding efficiency and decoding efficiency by combining the entropy gradient offset step. Combined with the network structure design and the header information code stream, the neural network can effectively guarantee the quality of the reconstructed image block while maintaining low complexity, so as to achieve the purpose of improving the encoding performance and decoding performance, and reduce the complexity. The entropy gradient is used to enhance the quality of the feature. The encoding end does not directly change the feature information, but encodes the entropy gradient offset step into the header information code stream. The decoding end enhances the feature through the entropy gradient offset step, improves the encoding performance, and improves the reconstructed image quality. Adding a regularization term improves the structural stability of feature enhancement.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a three-dimensional feature matrix in one embodiment of the present application;

FIG2 is a flowchart of a decoding method in one embodiment of the present application;

FIG3 is a flow chart of an encoding method in one embodiment of the present application;

FIG4 is a schematic diagram of a processing process of an encoding end in one embodiment of the present application;

FIG5 is a schematic diagram of a processing process of a decoding end in one embodiment of the present application;

FIG6A is a schematic diagram of a decoding method in one embodiment of the present application;

FIG6B is a schematic diagram of an end-to-end encoding and decoding framework in one implementation of the present application;

FIG6C is a schematic diagram showing the relationship between the self-information of a feature point and the quantized residual of the main information in one embodiment of the present application;

6D and 6E are schematic flow diagrams of an encoding method in one embodiment of the present application;

FIG6F is a processing flow of adding a LEGS module to the decoding end of JPEG-AI in one embodiment of the present application;

FIG6G is a flowchart of reconstruction of a JPEG-AI component in one embodiment of the present application;

FIG7A is a flowchart of a decoding method in one embodiment of the present application;

FIG7B is a flowchart of an encoding method in one embodiment of the present application;

FIG8A is a hardware structure diagram of a decoding end device in one embodiment of the present application;

FIG8B is a hardware structure diagram of an encoding end device in one implementation of the present application.

DETAILED DESCRIPTION

The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than for limiting the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and/or" used herein refers to any or all possible combinations of one or more associated listed items. It should be understood that, although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, depending on the context. In addition, the word "if" used can be interpreted as "at ... ", or "when ... ", or "in response to determination".

In the embodiments of the present application, a decoding method and an encoding method are proposed, which may involve the following concepts:

JPEG (Joint Photographic Experts Group): JPEG is a standard for continuous tone static image compression. The file extension can be .jpg or .jpeg. JPEG is a commonly used image file format. JPEG uses a joint coding method of predictive coding (DPCM), discrete cosine transform (DCT) and entropy coding to remove redundant image and color data. It is a lossy compression format that can compress images in a very small storage space, which will cause damage to image data to a certain extent. In particular, when using too high a compression ratio, the quality of the image restored after the final decompression will be reduced. If you pursue high-quality images, JPEG should not use too high a compression ratio.

JPEG-AI (Joint Photographic Experts Group Artificial Intelligence): The scope of JPEG-AI is to create a learning-based image coding standard that provides a single-stream, compact compressed domain representation that significantly improves compression efficiency over commonly used image coding standards at the same subjective quality, effectively improving performance in image processing and computer vision tasks. JPEG-AI targets a wide range of applications such as cloud storage, vision management, self-driving cars and devices, image acquisition storage and management, real-time management of visual data, and media distribution. The goal of JPEG-AI is to design a codec solution that needs to significantly improve compression efficiency at the same subjective quality and provide efficient compressed domain processing for machine learning-based image processing and computer vision tasks. JPEG-AI requires hardware- and software-friendly encoding and decoding, support for 8-bit and 10-bit depths, and efficient encoding and progressive decoding of images with text and graphics.

Entropy coding: Entropy coding is the coding process that does not lose any information according to the entropy principle. Information entropy is the average amount of information of the source (a measure of uncertainty). Entropy coding methods may include but are not limited to: Shannon coding, Huffman coding, and arithmetic coding.

Neural Network (NN): Neural network refers to artificial neural network. Neural network is a computing model composed of a large number of nodes (or neurons) connected to each other. In a neural network, neuron processing units can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. The types of processing units in a neural network can be divided into three categories: input units, output units, and hidden units. Input units receive signals and data from the outside world; output units realize the output of processing results; hidden units are units between input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units, and the representation and processing of information are reflected in the connection relationship of processing units. Neural network is a non-programmed, brain-like information processing method. The essence of neural network is to obtain a parallel distributed information processing function through the transformation and dynamic behavior of neural network, and imitate the information processing function of human brain nervous system to different degrees and levels. In the field of video processing, commonly used neural networks may include but are not limited to: convolutional neural network (CNN), recurrent neural network (RNN), fully connected network, etc.

Convolutional Neural Network (CNN): Convolutional neural network is a feedforward neural network and one of the most representative network structures in deep learning technology. The artificial neurons of convolutional neural network can respond to surrounding units within a certain coverage range and have excellent performance in large image processing. The basic structure of convolutional neural network includes two layers. One is the feature extraction layer (also called convolution layer). The input of each neuron is connected to the local receptive field of the previous layer and extracts the local features. Once the local feature is extracted, its positional relationship with other features is also determined. The second is the feature mapping layer (also called activation layer). Each computing layer of the neural network consists of multiple feature maps. Each feature map is a plane, and the weights of all neurons on the plane are equal. The feature mapping structure can use Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, GDN function, etc. as the activation function of the convolutional network. In addition, since neurons on a mapping surface share weights, the number of free parameters of the network can be reduced.

For example, one of the advantages of convolutional neural networks over image processing algorithms is that they avoid the complex pre-processing of images (extracting artificial features, etc.), and can directly input the original image for end-to-end learning. One of the advantages of convolutional neural networks over ordinary neural networks is that ordinary neural networks use a fully connected approach, that is, all neurons from the input layer to the hidden layer are connected, which will result in a huge number of parameters, making network training time-consuming or even difficult to train, while convolutional neural networks avoid this difficulty through local connections, weight sharing and other methods.

Deconvolution layer: Deconvolution layer is also called transposed convolution layer. The working process of deconvolution layer and convolution layer is very similar. The main difference is that deconvolution layer will make the output larger than the input through padding (of course, it can also remain the same). If stride is 1, it means that the output size is equal to the input size; if stride is N, it means that the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

Generalization Ability: Generalization ability refers to the ability of a machine learning algorithm to adapt to fresh samples. The purpose of learning is to learn the rules hidden behind the data pairs. For data outside the learning set with the same rules, the trained network can also give appropriate outputs. This ability can be called generalization ability.

Feature: The feature involved in this application is a three-dimensional feature matrix of C*W*H. See Figure 1, which is a schematic diagram of a three-dimensional feature matrix. In the three-dimensional feature matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be the input of the neural network or the output of the neural network.

Rate-Distortion Optimized: There are two major indicators for evaluating coding efficiency: bit rate and PSNR (Peak Signal to Noise Ratio). The smaller the bit stream, the greater the compression rate, and the greater the PSNR, the better the quality of the reconstructed image. When selecting a mode, the discriminant formula is essentially a comprehensive evaluation of the two. For example, the cost corresponding to the mode: J(mode) = D + λ*R, where D represents Distortion, which can usually be measured using the SSE indicator. SSE refers to the mean square sum of the difference between the reconstructed image block and the source image. In order to achieve cost considerations, the SAD indicator can also be used. SAD refers to the sum of the absolute values of the difference between the reconstructed image block and the source image; λ is the Lagrange multiplier, and R is the actual number of bits required for encoding the image block in this mode, including the sum of bits required for encoding mode information, motion information, residuals, etc. When selecting a mode, if the rate-distortion principle is used to make a comparison decision on the encoding mode, the best encoding performance can usually be guaranteed.

A large number of coding tools have been proposed for each module of the encoding end, and each tool often has multiple modes. For different video sequences, the coding tools that can achieve the best coding performance are often different. Therefore, in the encoding process, RDO (Rate-Distortion Optimize) is usually used to compare the coding performance of different tools or modes to select the best mode. After determining the optimal tool or mode, the decision information of the tool or mode is transmitted by encoding the marker information in the bitstream. Although this method brings higher coding complexity, it can adaptively select the optimal mode combination for different content to obtain the best coding performance. The decoding end can obtain relevant mode information by directly parsing the flag information, and the complexity impact is small.

In the general framework of end-to-end image coding, it mainly includes the feature main information part and the super prior side information part. The feature main information part includes analysis network, quantization, normal entropy coding, normal entropy decoding and synthesis network, and the super prior side information includes super prior analysis network, quantization, factored entropy coding, factored entropy decoding and super prior synthesis network. The image component x is compressed and encoded and reconstructed and restored by the analysis network and synthesis network of the feature main information part respectively; the super prior side information part is mainly used to model the probability of the feature main information and guide the entropy coding and decoding of the feature main information. In the general framework of end-to-end image coding, there is a problem that the main information feature entropy gradient is not fully utilized, and the image reconstruction quality is not fully improved.

In view of the above findings, in this embodiment, the end-to-end image coding framework features are utilized, and the decoding end utilizes the entropy gradient information of the quantized residual of the main information to enhance the quality of the feature, thereby improving the quality of the reconstructed image, and by adding a regularization term, the structural stability of the feature enhancement is improved. The encoding end does not directly change the main information, but the encoding step length is entered into the header information bit stream (that is, the entropy gradient offset step length is encoded into the header information bit stream), and the decoding end enhances the main information feature through the entropy gradient offset step length.

The decoding method and encoding method in the embodiments of the present application are described in detail below in combination with several specific embodiments.

Embodiment 1: A decoding method is proposed in the embodiment of the present application. Referring to FIG. 2 , it is a flow chart of the decoding method. The method can be applied to a decoding end (also called a video decoder). The method may include:

Step 201: obtaining initial image features corresponding to the current image block through a first neural network, and obtaining an entropy gradient offset step corresponding to the current image block. The first neural network may be a probabilistic hyperparameter decoding network including at least one convolutional layer.

In a possible implementation, obtaining the initial image features corresponding to the current image block through the first neural network may include but is not limited to: based on the first bitstream corresponding to the current image block, obtaining the standard deviation corresponding to the current image block through the probabilistic hyperparameter decoding network; determining the probability distribution model based on the standard deviation, decoding the second bitstream corresponding to the current image block based on the probability distribution model to obtain the decoded image features; determining the residual features corresponding to the current image block based on the decoded image features; and determining the initial image features corresponding to the current image block based on the residual features. Exemplarily, based on the first bitstream corresponding to the current image block, obtaining the standard deviation corresponding to the previous image block through the probabilistic hyperparameter decoding network may include but is not limited to: decoding the first bitstream corresponding to the current image block based on the factorized probability model to obtain the decoded image features, and determining the coefficient hyperparameter features based on the decoded image features; inputting the coefficient hyperparameter features into the probabilistic hyperparameter decoding network, and inversely transforming the coefficient hyperparameter features through the probabilistic hyperparameter decoding network to obtain the standard deviation corresponding to the current image block.

In a possible implementation, obtaining the entropy gradient offset step corresponding to the current image block may include, but is not limited to: decoding the header information code stream corresponding to the current image block to obtain the entropy gradient offset step. Alternatively, decoding the header information code stream corresponding to the current image block to obtain indication information corresponding to the entropy gradient offset step, and determining the entropy gradient offset step based on the indication information. Exemplarily, determining the entropy gradient offset step based on the indication information may include, but is not limited to: if the configured entropy gradient offset step table includes at least one entropy gradient offset step, and the indication information is used to indicate the index value of the entropy gradient offset step, then determining the entropy gradient offset step corresponding to the current image block from the entropy gradient offset step table based on the index value.

Step 202: Enhance the initial image features based on the entropy gradient offset step to obtain target image features.

Exemplarily, performing feature enhancement on initial image features based on the entropy gradient offset step to obtain target image features may include: performing feature enhancement on initial image features based on the entropy gradient offset step and entropy gradient feature information corresponding to the current image block to obtain target image features; wherein the entropy gradient offset step represents the offset step of the entropy gradient, and the entropy gradient offset step is a correction step used to enhance the initial image features.

In a possible implementation, the entropy gradient feature information includes an entropy gradient offset value, and the method for determining the entropy gradient offset value corresponding to the current image block may include, but is not limited to: determining the entropy gradient offset value corresponding to the current image block based on the residual feature. Exemplarily, determining the entropy gradient offset value corresponding to the current image block based on the residual feature may include, but is not limited to: determining the residual incremental feature corresponding to the residual feature; calculating the entropy of the residual feature, and calculating the entropy of the residual incremental feature; determining the entropy gradient of the residual feature based on the entropy of the residual feature and the entropy of the residual incremental feature, and determining the entropy gradient offset value based on the entropy gradient of the residual feature and the regularization term.

In another possible implementation, the entropy gradient feature information may include a target entropy gradient and a regularization term. The method for determining the target entropy gradient corresponding to the current image block may include but is not limited to: querying the target quantization scale corresponding to the standard deviation from the entropy rate fitting table, the target quantization scale is the quantization scale closest to the standard deviation, and querying the target quadratic term corresponding to the target quantization scale from the entropy rate fitting table; wherein the entropy rate fitting table includes the correspondence between the quantization scale and the quadratic term; and determining the target entropy gradient corresponding to the current image block based on the target quadratic term.

Exemplarily, the method of obtaining the entropy rate fitting table may include but is not limited to: for multiple standard deviations between the minimum standard deviation and the maximum standard deviation, the multiple standard deviations are uniformly quantized on a logarithmic scale to obtain multiple quantization scales corresponding to the multiple standard deviations; for each quantization scale, the entropy rate can be fitted to the quantization scale to obtain the quadratic term corresponding to the quantization scale, and the correspondence between the quantization scale and the quadratic term is recorded in the entropy rate fitting table.

Exemplarily, for the above two implementations, the method of determining the regularization term may include, but is not limited to: determining the regularization term based on the initial image features corresponding to the current image block, the standard deviation, and the residual features corresponding to the current image block.

Step 203: Based on the target image features, a reconstructed image block corresponding to the current image block is obtained through a synthetic transformation network.

Exemplarily, the target image features may be input into a synthetic transformation network, and the synthetic transformation may be performed on the target image features through the synthetic transformation network to obtain a reconstructed image block.

In the above embodiment, the initial image feature corresponding to the current image block is the initial image feature corresponding to the brightness component; and/or, the initial image feature corresponding to the current image block is the initial image feature corresponding to the chrominance component. If the initial image feature corresponding to the current image block is the initial image feature corresponding to the brightness component, the target image feature is the target image feature corresponding to the brightness component, and the reconstructed image block is the reconstructed image block corresponding to the brightness component. And/or, if the initial image feature corresponding to the current image block is the initial image feature corresponding to the chrominance component, the target image feature is the target image feature corresponding to the chrominance component, and the reconstructed image block is the reconstructed image block corresponding to the chrominance component.

Exemplarily, the above execution order is only for the convenience of describing the examples given. In practical applications, the execution order between the steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

It can be seen from the above technical scheme that in the embodiment of the present application, by obtaining the initial image features corresponding to the current image block and obtaining the entropy gradient offset step corresponding to the current image block, the target image features are determined based on the initial image features and the entropy gradient offset step, and based on the target image features, the reconstructed image block corresponding to the current image block is obtained, thereby proposing an end-to-end video image compression method, which can realize the encoding and decoding of video images based on neural networks, and achieve the purpose of improving the encoding efficiency and decoding efficiency by combining the entropy gradient offset step. Combined with the network structure design and the header information code stream, the neural network can effectively guarantee the quality of the reconstructed image block while maintaining low complexity, so as to achieve the purpose of improving the encoding performance and decoding performance, and reduce the complexity. The entropy gradient is used to enhance the quality of the feature. The encoding end does not directly change the feature information, but encodes the entropy gradient offset step into the header information code stream. The decoding end enhances the feature through the entropy gradient offset step, improves the encoding performance, and improves the reconstructed image quality. Adding a regularization term improves the structural stability of feature enhancement.

Embodiment 2: In the embodiment of the present application, a coding method is proposed. Referring to FIG. 3 , it is a schematic diagram of the flow of the coding method. The method can be applied to a coding end (also called a video encoder). The method may include:

Step 301: Acquire initial image features corresponding to the current image block through a first neural network. The first neural network may be a probabilistic hyperparameter decoding network, and the first neural network may include at least one convolutional layer.

In a possible implementation, obtaining the initial image features corresponding to the current image block through the first neural network may include but is not limited to: based on the first bitstream corresponding to the current image block, obtaining the standard deviation corresponding to the current image block through the probabilistic hyperparameter decoding network; determining the probability distribution model based on the standard deviation, decoding the second bitstream corresponding to the current image block based on the probability distribution model to obtain the decoded image features; determining the residual features corresponding to the current image block based on the decoded image features; and determining the initial image features corresponding to the current image block based on the residual features. Exemplarily, based on the first bitstream corresponding to the current image block, obtaining the standard deviation corresponding to the previous image block through the probabilistic hyperparameter decoding network may include but is not limited to: decoding the first bitstream corresponding to the current image block based on the factorized probability model to obtain the decoded image features, and determining the coefficient hyperparameter features based on the decoded image features; inputting the coefficient hyperparameter features into the probabilistic hyperparameter decoding network, and inversely transforming the coefficient hyperparameter features through the probabilistic hyperparameter decoding network to obtain the standard deviation corresponding to the current image block.

Step 302: for each entropy gradient offset step, the initial image features are enhanced based on the entropy gradient offset step to obtain the target image features; based on the target image features, a reconstructed image block corresponding to the entropy gradient offset step is obtained through a synthetic transformation network.

Step 303: Based on the current image block and the reconstructed image block corresponding to each entropy gradient offset step, select the entropy gradient offset step corresponding to the current image block from all entropy gradient offset step sizes (i.e., the entropy gradient offset step size finally adopted), and encode the entropy gradient offset step size corresponding to the current image block in the header information code stream corresponding to the current image block.

Exemplarily, performing feature enhancement on the initial image features based on the entropy gradient offset step to obtain the target image features may include: performing feature enhancement on the initial image features based on the entropy gradient offset step and the entropy gradient feature information corresponding to the current image block to obtain the target image features; wherein the entropy gradient offset step represents the offset step of the entropy gradient, and the entropy gradient offset step is the correction step used to enhance the initial image features.

In a possible implementation, the entropy gradient feature information includes an entropy gradient offset value, and the method for determining the entropy gradient offset value corresponding to the current image block may include, but is not limited to: determining the entropy gradient offset value corresponding to the current image block based on the residual feature. Exemplarily, determining the entropy gradient offset value corresponding to the current image block based on the residual feature may include, but is not limited to: determining the residual incremental feature corresponding to the residual feature; calculating the entropy of the residual feature, and calculating the entropy of the residual incremental feature; determining the entropy gradient of the residual feature based on the entropy of the residual feature and the entropy of the residual incremental feature, and determining the entropy gradient offset value based on the entropy gradient of the residual feature and the regularization term.

In another possible implementation, the entropy gradient feature information may include a target entropy gradient and a regularization term. The method for determining the target entropy gradient corresponding to the current image block may include but is not limited to: querying the target quantization scale corresponding to the standard deviation from the entropy rate fitting table, the target quantization scale is the quantization scale closest to the standard deviation, and querying the target quadratic term corresponding to the target quantization scale from the entropy rate fitting table; wherein the entropy rate fitting table includes the correspondence between the quantization scale and the quadratic term; and determining the target entropy gradient corresponding to the current image block based on the target quadratic term.

Exemplarily, the method of obtaining the entropy rate fitting table may include, but is not limited to: for multiple standard deviations between the minimum standard deviation and the maximum standard deviation, uniformly quantizing the multiple standard deviations on a logarithmic scale to obtain multiple quantization scales corresponding to the multiple standard deviations; for each quantization scale, entropy rate fitting can be performed on the quantization scale to obtain a quadratic term corresponding to the quantization scale, and the correspondence between the quantization scale and the quadratic term is recorded in the entropy rate fitting table.

Exemplarily, for the above two implementations, the method of determining the regularization term may include, but is not limited to: determining the regularization term based on the initial image features corresponding to the current image block, the standard deviation, and the residual features corresponding to the current image block.

Exemplarily, based on the target image features, a reconstructed image block corresponding to the entropy gradient offset step is obtained through a synthetic transformation network, which may include but is not limited to: the target image features may be input into the synthetic transformation network, and the target image features may be synthetically transformed through the synthetic transformation network to obtain a reconstructed image block corresponding to the entropy gradient offset step.

In a possible implementation, based on the current image block and the reconstructed image block corresponding to each entropy gradient offset step, the entropy gradient offset step corresponding to the current image block is selected from all entropy gradient offset steps, which may include but is not limited to: for each entropy gradient offset step, based on the current image block and the reconstructed image block corresponding to the entropy gradient offset step, determining the cost value corresponding to the entropy gradient offset step; based on the cost value corresponding to each entropy gradient offset step, the entropy gradient offset step corresponding to the minimum cost value can be selected as the entropy gradient offset step corresponding to the current image block.

In another possible implementation, based on the current image block and the reconstructed image block corresponding to each entropy gradient offset step, the entropy gradient offset step corresponding to the current image block is selected from all entropy gradient offset steps, which may include but is not limited to: for each entropy gradient offset step, based on the current image block and the reconstructed image block corresponding to the entropy gradient offset step, determining the target cost value corresponding to the entropy gradient offset step, and determining the fidelity corresponding to the entropy gradient offset step based on the target cost value and the reference cost value; based on the fidelity corresponding to each entropy gradient offset step, selecting the entropy gradient offset step corresponding to the maximum fidelity as the entropy gradient offset step corresponding to the current image block; wherein the process of obtaining the reference cost value may include: after obtaining the initial image features corresponding to the current image block, based on the initial image features, obtaining the reconstructed image block through a synthetic transformation network, and determining the reference cost value based on the current image block and the reconstructed image block.

Exemplarily, after obtaining the entropy gradient offset step corresponding to the current image block, the entropy gradient offset step corresponding to the current image block can be encoded in the header information code stream corresponding to the current image block. For example, the value of the entropy gradient offset step is encoded in the header information code stream corresponding to the current image block; or, the indication information corresponding to the entropy gradient offset step is encoded in the header information code stream corresponding to the current image block, for example, if the configured entropy gradient offset step table includes at least one entropy gradient offset step, the indication information corresponding to the entropy gradient offset step is used to indicate the index value of the entropy gradient offset step.

In the above embodiment, the initial image feature corresponding to the current image block is the initial image feature corresponding to the brightness component; and/or, the initial image feature corresponding to the current image block is the initial image feature corresponding to the chrominance component. If the initial image feature corresponding to the current image block is the initial image feature corresponding to the brightness component, the target image feature is the target image feature corresponding to the brightness component, and the reconstructed image block is the reconstructed image block corresponding to the brightness component. And/or, if the initial image feature corresponding to the current image block is the initial image feature corresponding to the chrominance component, the target image feature is the target image feature corresponding to the chrominance component, and the reconstructed image block is the reconstructed image block corresponding to the chrominance component.

Exemplarily, the above execution order is only for the convenience of describing the examples given. In practical applications, the execution order between the steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

It can be seen from the above technical scheme that in the embodiment of the present application, by obtaining the initial image features corresponding to the current image block, and obtaining the entropy gradient offset step corresponding to the current image block, the target image features are determined based on the initial image features and the entropy gradient offset step, and the reconstructed image block corresponding to the current image block is obtained based on the target image features, thereby proposing an end-to-end video image compression method, which can realize the encoding and decoding of video images based on neural networks, and achieve the purpose of improving the encoding efficiency and decoding efficiency by combining the entropy gradient offset step. Combined with the network structure design and the header information code stream, the neural network can effectively guarantee the quality of the reconstructed image block while maintaining low complexity, so as to achieve the purpose of improving the encoding performance and decoding performance, and reduce the complexity. The entropy gradient is used to enhance the quality of the feature. The encoding end does not directly change the feature information, but encodes the entropy gradient offset step into the header information code stream. The decoding end enhances the feature through the entropy gradient offset step, improves the encoding performance, and improves the quality of the reconstructed image. Adding a regularization term improves the structural stability of feature enhancement.

Embodiment 3: For Embodiment 1 and Embodiment 2, the processing process of the encoding end can be referred to as shown in FIG4. Of course, FIG4 is only an example of the processing process of the encoding end, and does not limit the processing process of the encoding end.

After obtaining the current image block x (the current image block x may be the original image block x, i.e., the input image block), the encoder may analyze and transform the current image block x through an analysis transformation network (i.e., a neural network) to obtain an image feature y corresponding to the current image block x. Among them, performing feature transformation on the current image block x through the analysis transformation network means: transforming the current image block x to an image feature y in the latent domain, thereby facilitating all subsequent processes to operate on the latent domain.

The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the encoding and decoding process for the image block can also be directly used for the image.

After obtaining the image feature y, the encoding end performs coefficient hyperparameter feature transformation on the image feature y to obtain coefficient hyperparameter feature z. For example, the image feature y can be input into a hyperparameter encoding network (i.e., a neural network), and the hyperparameter encoding network performs coefficient hyperparameter feature transformation on the image feature y to obtain coefficient hyperparameter feature z. The hyperparameter encoding network can be a trained neural network, and there is no restriction on the training process of the hyperparameter encoding network, as long as the coefficient hyperparameter feature transformation can be performed on the image feature y. The image feature y in the latent domain obtains the super prior latent information z after passing through the hyperparameter encoding network.

After obtaining the coefficient hyperparameter feature z, the encoding end can quantize the coefficient hyperparameter feature z to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, that is, the Q operation in Figure 4 is the quantization process. After obtaining the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, the hyperparameter quantization feature is encoded to obtain Bitstream#1 (i.e., the first bitstream) corresponding to the current image block, that is, the AE operation in Figure 4 represents the encoding process, such as the entropy encoding process. Alternatively, the encoding end can also directly encode the coefficient hyperparameter feature z to obtain Bitstream#1 corresponding to the current image block. Among them, the hyperparameter quantization feature or coefficient hyperparameter feature z carried in Bitstream#1 is mainly used to obtain the parameters of the mean and probability distribution model.

After obtaining the Bitstream#1 corresponding to the current image block, the encoder may send the Bitstream#1 corresponding to the current image block to the decoder. For the processing process of the decoder for the Bitstream#1 corresponding to the current image block, refer to the subsequent embodiments.

After obtaining the Bitstream#1 corresponding to the current image block, the encoder can also decode Bitstream#1 to obtain the hyperparameter quantization feature, that is, AD in Figure 4 represents the decoding process, and then the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat and the coefficient hyperparameter feature z can be the same or different. The IQ operation in Figure 4 is the dequantization process. Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the encoder can also decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the dequantization process of the coefficient hyperparameter feature z_hat.

For the encoding process of Bitstream#1, the encoding method of the fixed probability density model can be adopted, and for the decoding process of Bitstream#1, the decoding method of the fixed probability density model can be adopted. There is no restriction on this encoding and decoding process.

After obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block (the determination process of the residual feature y_hat is described in the subsequent embodiments) to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat are input to the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu. The prediction value mu is used to obtain the residual by subtracting from the original feature and to obtain the reconstructed y by adding the decoded residual.

It should be noted that the mean prediction network is an optional neural network, that is, there is no mean prediction network, that is, there is no need to determine the predicted value mu through the mean prediction network. The dotted box in Figure 4 indicates that the mean prediction network is optional.

After obtaining the image feature y, the encoder can determine the residual feature r based on the image feature y and the prediction value mu, such as taking the difference between the image feature y and the prediction value mu as the residual feature r. Then, the residual feature r is feature processed to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, it is necessary to deploy a mean prediction network, and the mean prediction network provides the prediction value mu. Alternatively, after obtaining the image feature y, the encoder can perform feature processing on the image feature y to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, there is no need to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

After obtaining the image feature s, the encoding end can quantize the image feature s to obtain the image quantization feature corresponding to the image feature s, that is, the Q operation in Figure 4 is the quantization process. After obtaining the image quantization feature corresponding to the image feature s, the encoding end can encode the image quantization feature to obtain Bitstream#2 (i.e., the second bitstream) corresponding to the current image block, that is, the AE operation in Figure 4 represents the encoding process, such as the entropy encoding process. Alternatively, the encoding end can also directly encode the image feature s to obtain Bitstream#2 corresponding to the current image block without involving the quantization process of the image feature s.

After obtaining Bitstream#2 corresponding to the current image block, the encoder may send Bitstream#2 corresponding to the current image block to the decoder. For the processing process of the decoder for Bitstream#2 corresponding to the current image block, refer to the subsequent embodiments.

After obtaining the Bitstream#2 corresponding to the current image block, the encoder can also decode Bitstream#2 to obtain the image quantization feature, that is, AD in FIG4 represents the decoding process, and then the encoder can dequantize the image quantization feature to obtain the image feature s', which can be the same as or different from the image feature s. The IQ operation in FIG4 is the dequantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the encoder can also decode Bitstream#2 to obtain the image feature s' without involving the dequantization process of the image quantization feature.

After obtaining the image feature s', the encoder can perform feature recovery (i.e., the inverse process of feature processing) on the image feature s'. There is no restriction on this feature recovery process, and it can be any feature recovery method to obtain the residual feature r_hat. The residual feature r_hat and the residual feature r can be the same or different. After obtaining the residual feature r_hat, the encoder determines the image feature y_hat based on the residual feature r_hat and the prediction value mu. The image feature y_hat and the image feature y can be the same or different. For example, the sum of the residual feature r_hat and the prediction value mu is used as the image feature y_hat. In this case, it is necessary to deploy a mean prediction network, and the mean prediction network provides the prediction value mu. Alternatively, after obtaining the image feature s', the encoder can perform feature recovery (i.e., the inverse process of feature processing) on the image feature s' to obtain the image feature y_hat. The image feature y_hat and the image feature y can be the same or different. In this case, it is not necessary to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

After obtaining the image feature y_hat, the encoding end can perform a synthetic transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

In a possible implementation, when the encoding end encodes the image quantization feature or the image feature s to obtain the Bitstream#2 corresponding to the current image block, the encoding end needs to first determine the probability distribution model, and then encode the image quantization feature or the image feature s based on the probability distribution model. In addition, when the encoding end decodes Bitstream#2, it also needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model.

In order to obtain the probability distribution model, as shown in Figure 4, after obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input to the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probability hyperparameter decoding network can be a trained neural network, and there is no restriction on the training process of this probability hyperparameter decoding network, and it is sufficient to be able to perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat.

In a possible implementation, the processing process at the encoding end may be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process, without any limitation on the encoding process.

Embodiment 4: For Embodiment 1 and Embodiment 2, the processing process of the decoding end can be referred to as shown in FIG5. Of course, FIG5 is only an example of the processing process of the decoding end, and does not limit the processing process of the decoding end.

After obtaining the Bitstream#1 corresponding to the current image block, the decoding end can also decode Bitstream#1 to obtain the hyperparameter quantization feature, that is, AD in Figure 5 represents the decoding process, and then the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat and the coefficient hyperparameter feature z can be the same or different. The IQ operation in Figure 5 is the dequantization process. Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the decoding end can also decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the dequantization process of the coefficient hyperparameter feature z_hat.

For the decoding process of Bitstream#1, a decoding method of a fixed probability density model may be used, and there is no restriction on this.

The image can be divided into one image block or into multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the decoding process for the image block can also be directly used for the image.

After obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block (the determination process of the residual feature y_hat is shown in the subsequent embodiments) to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat are input to the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu.

It should be noted that the mean prediction network is an optional neural network, that is, there is no mean prediction network, that is, there is no need to determine the predicted value mu through the mean prediction network. The dotted box in Figure 5 indicates that the mean prediction network is optional.

After obtaining the Bitstream#2 corresponding to the current image block, the decoding end can also decode Bitstream#2 to obtain the image quantization feature, that is, AD in Figure 5 represents the decoding process, and then the decoding end can dequantize the image quantization feature to obtain the image feature s', which can be the same as or different from the image feature s. The IQ operation in Figure 5 is the dequantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the decoding end can also decode Bitstream#2 to obtain the image feature s' without involving the dequantization process of the image quantization feature.

After obtaining the image feature s', the decoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing) to obtain the residual feature r_hat, which is the same as or different from the residual feature r. After obtaining the residual feature r_hat, the decoder determines the image feature y_hat based on the residual feature r_hat and the prediction value mu, and the image feature y_hat is the same as or different from the image feature y, such as taking the sum of the residual feature r_hat and the prediction value mu as the image feature y_hat. In this case, it is necessary to deploy a mean prediction network, and the prediction value mu is provided by the mean prediction network. Alternatively, after obtaining the image feature s', the decoder can perform feature recovery on the image feature s' to obtain the image feature y_hat, which is the same as or different from the image feature y. In this case, it is not necessary to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

After obtaining the image feature y_hat, the decoding end can perform a synthetic transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

In a possible implementation, when decoding Bitstream#2, the decoding end needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model. In order to obtain the probability distribution model, referring to FIG5 , after obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input to the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probability hyperparameter decoding network can be a trained neural network, and there is no restriction on the training process of the probability hyperparameter decoding network. It is sufficient to perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p.

In a possible implementation, the processing process at the decoding end may be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process, without limiting the decoding process.

Embodiment 5: A decoding method is proposed in the embodiment of the present application. Referring to FIG. 6A , it is a flowchart of the decoding method. The method can be applied to a decoding end (also called a video decoder). The method may include:

Step S11: The decoding end receives the first bitstream (Bitstream#1) corresponding to the current image block. The first bitstream can also be called the side information bitstream. The decoding end decodes the first bitstream to obtain the hyperparametric quantization feature. For example, the first bitstream is losslessly entropy decoded according to the factorized probability model p _f (.) to obtain the hyperparametric quantization feature. The decoding end dequantizes the hyperparametric quantization feature to obtain the coefficient hyperparametric feature z_hat. The coefficient hyperparametric feature z_hat can also be called the quantization side information. Alternatively, the first bitstream may be losslessly entropy decoded according to the factorized probability model p _f (.) to obtain the coefficient hyperparameter feature z_hat.

In summary, the decoding end can decode the first bitstream corresponding to the current image block based on the factorized probability model to obtain decoded image features, and determine the coefficient hyperparameter features based on the decoded image features. For example, the decoded image features can be hyperparameter quantization features, and the decoding end can dequantize the decoded image features to obtain the coefficient hyperparameter features z_hat. Alternatively, the decoded image features can be directly the coefficient hyperparameter features z_hat.

Step S12: The decoding end receives the header information code stream corresponding to the current image block, and decodes the entropy gradient offset step corresponding to the current image block from the header information code stream. The entropy gradient offset step may also be referred to as the entropy gradient offset step ρ of the optimal main information.

In a possible implementation, the header information bitstream may carry the value of the entropy gradient offset step, so the decoding end may decode the header information bitstream corresponding to the current image block to directly obtain the value of the entropy gradient offset step.

In another possible implementation, the header information bitstream may carry indication information corresponding to the entropy gradient offset step, so the decoding end may decode the header information bitstream corresponding to the current image block to obtain indication information corresponding to the entropy gradient offset step, the indication information is used to indicate the index value of the entropy gradient offset step, the index value is used to indicate the entropy gradient offset step is the number of entropy gradient offset step in the entropy gradient offset step table. For example, the configured entropy gradient offset step table may include at least one entropy gradient offset step, and the decoding end selects the entropy gradient offset step corresponding to the index value from the entropy gradient offset step table based on the index value corresponding to the indication information as the entropy gradient offset step corresponding to the current image block.

Step S13: The decoding end obtains the coefficient hyperparameter feature z_hat (quantized edge information ) After that, the coefficient hyperparameter feature z_hat is inversely transformed to obtain the standard deviation σ. For example, the coefficient hyperparameter feature z_hat is input to the probabilistic hyperparameter decoding network, and the probabilistic hyperparameter decoding network inversely transforms the coefficient hyperparameter feature z_hat to obtain the standard deviation σ. Among them, the probabilistic hyperparameter decoding network is also called the super-prior scale decoding network, and the standard deviation σ is also called the probability density parameter of the main information.

Exemplarily, after obtaining the standard deviation σ, the probability distribution parameter p can also be determined based on the standard deviation σ. For example, if the probability distribution parameter p only includes the standard deviation σ, the standard deviation σ can be used as the probability distribution parameter p, that is, the probability distribution parameter p with zero mean. Alternatively, if the probability distribution parameter p includes the standard deviation σ and the mean, the mean can also be obtained. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat are input to the mean prediction network, and the mean prediction network determines the mean mu (that is, the predicted value mu) based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. There is no restriction on this prediction process. In this way, the probability distribution parameter p can be determined based on the standard deviation σ and the mean mu. After obtaining the probability distribution parameter p, the decoding end can also generate a probability distribution model N(mu, σ) based on the probability distribution parameter p. There is no restriction on this process in this embodiment.

In summary, it can be seen that after the decoding end obtains the coefficient hyperparameter feature z_hat, the coefficient hyperparameter feature z_hat can be input into the probabilistic hyperparameter decoding network, and the coefficient hyperparameter feature z_hat can be inversely transformed through the probabilistic hyperparameter decoding network to obtain the standard deviation.

Step S14: The decoding end receives the second bitstream (Bitstream#2) corresponding to the current image block. The second bitstream can also be called the main information bitstream. The decoding end decodes the second bitstream based on the probability distribution model to obtain the decoded image features, and determines the residual features corresponding to the current image block based on the decoded image features. The residual features are the quantized residuals of the main information.

For example, the decoding end decodes the second bitstream based on the probability distribution model to obtain the image quantization feature (i.e., the decoded image feature), and dequantizes the image quantization feature to obtain the residual feature r_hat, i.e., the quantized residual Alternatively, the decoding end decodes the second bitstream based on the probability distribution model to obtain the image quantization feature (i.e., the decoded image feature), and dequantizes the image quantization feature to obtain the image feature s'. Alternatively, the decoding end decodes the second bitstream based on the probability distribution model to obtain the image feature s' (i.e., the decoded image feature). After obtaining the image feature s', the decoding end can perform feature recovery on the image feature s' to obtain the residual feature r_hat, i.e., the quantized residual

Step S15: The decoding end determines the initial image features corresponding to the current image block based on the residual features.

Exemplarily, after obtaining the residual feature r_hat, the decoder determines the image feature y_hat based on the residual feature r_hat and the prediction value mu (i.e., the mean value mu), and the image feature y_hat is the initial image feature corresponding to the current image block. For example, the sum of the residual feature r_hat and the prediction value mu is used as the initial image feature, which can also be called the main information quantization value

In order to obtain the predicted value mu, then: after obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block to obtain the predicted value mu corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat can be input into the mean prediction network, and the mean prediction network determines the predicted value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input can include the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat, and the two are combined to obtain a more accurate prediction value mu.

Step S16: The decoding end determines the target image feature based on the initial image feature, the entropy gradient offset step length, and the entropy gradient offset value, wherein the entropy gradient offset step length represents the offset step length of the entropy gradient and is used to indicate the direction in which the feature code rate increases.

Exemplarily, based on the entropy gradient offset step size ρ, the initial image features The entropy gradient offset value shift can be used to determine the target image feature y using the following expression (1). That is, the initial image feature y can be determined based on the entropy gradient offset step size ρ and the entropy gradient offset value shift. Perform an offset, such as shifting the initial image features based on the entropy gradient offset step size ρ and the entropy gradient offset value shift The feature value of each feature point in is offset. Of course, the following expression (1) is only an example and is not limited to this.

In expression (1), represents the initial image feature, which is obtained based on the residual feature r_hat and the predicted value mu, ρ represents the entropy gradient shift step, which is decoded from the header information code stream corresponding to the current image block, and shift represents the entropy gradient shift value. The entropy gradient shift value can be determined in a manner including but not limited to: based on the residual feature corresponding to the current image block (such as the quantized residual ) determine the entropy gradient offset value shift corresponding to the current image block, for example, determine the residual incremental feature corresponding to the residual feature, calculate the entropy of the residual feature, and calculate the entropy of the residual incremental feature; determine the entropy gradient of the residual feature based on the entropy of the residual feature and the entropy of the residual incremental feature, and determine the entropy gradient offset value based on the entropy gradient of the residual feature and the regularization term.

In a possible implementation, the regularization term may be determined in a manner including, but not limited to: based on the initial image features Standard deviation σ and residual features (such as quantized residual ) Determine the regularization term in, It can be a regularization term used to improve the structural stability of the feature domain representation and enhance the performance of the entropy of different distortion evaluation indicators.

Step S17: The decoding end obtains a reconstructed image block corresponding to the current image block based on the target image feature. For example, the target image feature is input to the synthesis transformation network, and the target image feature is synthesized and transformed by the synthesis transformation network to obtain a reconstructed image block. The target image feature is the enhanced main information, and the reconstructed image block can be recorded as the reconstructed image block.

Example 6: In Example 1, Example 2 and Example 5, the target image features are determined based on the initial image features, the entropy gradient offset step, and the entropy gradient offset value, as shown in expression (1). The process is described below.

As shown in FIG6B , it is a schematic diagram of an end-to-end encoding and decoding framework, which is mainly divided into two parts, the feature main information part and the super priori side information part. The feature main information part includes an analysis network, quantization, normal entropy coding, normal entropy decoding and a synthesis network. The super priori side information part includes a super priori analysis network, quantization, factored entropy coding, factored entropy decoding and a super priori synthesis network. The current image block x is compressed and encoded and reconstructed and restored by the analysis network and synthesis network of the feature main information part, respectively. The super priori side information part is used to model the probability of the feature main information and guide the entropy coding and entropy decoding of the feature main information.

In the end-to-end encoding and decoding framework, the initial image features (i.e., main information characteristic value ) obeys the normal distribution of N(μ, σ), N(μ, σ) is a probability distribution model, and the network needs to minimize the following loss function during training:

In expression (2), yes The entropy rate loss or bit rate loss, for The entropy rate loss or bit rate loss, is the initial image feature of the above embodiment, is the coefficient hyperparameter feature z_hat (i.e., quantized edge information) of the above embodiment. ), is the distortion loss, which is the reconstructed image block corresponding to the current image block x and the current image block x. The distortion loss between λ and λ is the Lagrangian coefficient. This problem is a multi-objective Pareto optimization problem. When the neural network model training converges, the expression (3) can be inferred from the KKT condition (Karush-Kuhn-Tucker Condition):

From expression (3), it can be seen that the main information entropy gradient and the distortion gradient can be negatively correlated, that is, the initial image features are adjusted along the direction of the main information entropy gradient. Can cause distortion loss Reduce, thereby improving encoding performance.

Exemplary, initial image features The amount of self-information As shown in expression (4), for example, for the initial image features Each feature point Feature Points The amount of self-information See expression (4):

In expression (4), σ is the initial image feature The scale parameter of the normal distribution is The residual is the main information quantization, and erf can be the error function. can be the residual feature, can be the initial image feature, and μ can be the mean.

As shown in FIG6C , the relationship between the self-information of the feature point and the quantized residual of the main information under different σ can be selected. Upsampling is marked with circles in Figure 6C, and quadratic polynomial curve fitting is used, marked with dashed lines in Figure 6C. As shown in Figure 6C, fixed The larger σ is, the smaller the slope of the curve is, that is, the smaller the main information entropy gradient is.

For example, the calculation method of the code rate can be shown in expression (5), and the gradient of the code rate with respect to a certain feature point and the derivation process of the gradient of the code rate with respect to the feature point can be shown in expression (6):

In expression (5), is the code rate of the main information, and _Ii is the self-information of the ith feature point. In expression (6), _ai , _bi , and _ci are the quadratic coefficient, the linear coefficient, and the constant term of the quadratic curve fitting, respectively, and ∝ is a proportional relationship. From expression (6), it can be seen that the residual feature can be used The scale parameter σ of the normal distribution approximately describes the entropy gradient of the main information.

In a possible implementation, the main information feature points may be offset using the following expression:

1. Calculate residual features Entropy For example, the residual feature is calculated using the following expression (7): Entropy

In expression (7), erf is an error function. Of course, expression (7) is only an example and is not limiting.

2. Determine the residual incremental feature corresponding to the residual feature, such as the residual incremental feature is s can be greater than 0, such as 0.01, 0.02, or less than 0, such as -0.01, -0.02. Take the residual incremental feature as an example. Entropy For example, the residual incremental feature is calculated using the following expression (8): Entropy

In expression (8), erf is an error function. Of course, expression (8) is only an example and is not limiting.

3. Based on residual features Entropy and residual incremental features Entropy Determine residual characteristics The entropy gradient For example, the residual feature is calculated using the following expression (9): The entropy gradient

4. Based on initial image features Standard deviation σ and residual characteristics Determine the regularization term f, as shown in expression (10):

5. Based on residual features The entropy gradient of And the regularization term f determines the entropy gradient offset value shift. For example, the entropy gradient offset value shift can be determined by using expression (11), that is, the offset value shift of the main information quantization value of the regularization term and the main information quantization residual entropy gradient is calculated. Of course, expression (11) is only an example and there is no limitation on this determination method.

6. Entropy gradient-based offset step size ρ and initial image features The entropy gradient offset value shift uses the following expression (12) to determine the target image feature y, that is, the entropy gradient offset value and the entropy gradient offset step size are used to offset the main information quantization value:

At this point, the definite expression of the target image feature y is obtained, and the initial image feature can be shifted based on the entropy gradient shift step size ρ and the entropy gradient shift value shift Perform an offset, such as shifting the initial image features based on the entropy gradient offset step size ρ and the entropy gradient offset value shift The eigenvalues of each feature point in are offset. In expression (12), Represents the initial image features, which are based on residual features and the predicted value mu, ρ represents the entropy gradient offset step, which is decoded from the header information bitstream corresponding to the current image block, and shift represents the entropy gradient offset value. The method for determining the entropy gradient offset value can refer to the above expression.

Embodiment 7: A decoding method is proposed in the embodiment of the present application. The method can be applied to a decoding end. The method includes:

Step S21: The decoding end receives the first bitstream (Bitstream#1) corresponding to the current image block. The first bitstream can also be called the side information bitstream. The decoding end decodes the first bitstream to obtain the hyperparametric quantization feature. For example, the first bitstream is losslessly entropy decoded according to the factorized probability model p _f (.) to obtain the hyperparametric quantization feature. The decoding end dequantizes the hyperparametric quantization feature to obtain the coefficient hyperparametric feature z_hat. The coefficient hyperparametric feature z_hat can also be called the quantization side information. Alternatively, the first bitstream may be losslessly entropy decoded according to the factorized probability model p _f (.) to obtain the coefficient hyperparameter feature z_hat.

In summary, the decoding end can decode the first bitstream corresponding to the current image block based on the factorized probability model to obtain decoded image features, and determine the coefficient hyperparameter features based on the decoded image features. For example, the decoded image features can be hyperparameter quantization features, and the decoding end can dequantize the decoded image features to obtain the coefficient hyperparameter features z_hat. Alternatively, the decoded image features can be directly the coefficient hyperparameter features z_hat.

Step S22: The decoding end receives the header information code stream corresponding to the current image block, and decodes the entropy gradient offset step corresponding to the current image block from the header information code stream. The entropy gradient offset step may also be referred to as the entropy gradient offset step ρ of the optimal main information.

In a possible implementation, the header information bitstream may carry the value of the entropy gradient offset step, so the decoding end may decode the header information bitstream corresponding to the current image block to directly obtain the value of the entropy gradient offset step.

In another possible implementation, the header information bitstream may carry indication information corresponding to the entropy gradient offset step, so the decoding end may decode the header information bitstream corresponding to the current image block to obtain indication information corresponding to the entropy gradient offset step, the indication information is used to indicate the index value of the entropy gradient offset step, the index value is used to indicate the entropy gradient offset step is the number of entropy gradient offset step in the entropy gradient offset step table. For example, the configured entropy gradient offset step table may include at least one entropy gradient offset step, and the decoding end selects the entropy gradient offset step corresponding to the index value from the entropy gradient offset step table based on the index value corresponding to the indication information as the entropy gradient offset step corresponding to the current image block.

Step S23, the decoding end obtains the coefficient hyperparameter feature z_hat (quantized edge information ) After that, the coefficient hyperparameter feature z_hat is inversely transformed to obtain the standard deviation σ. For example, the coefficient hyperparameter feature z_hat is input to the probabilistic hyperparameter decoding network, and the probabilistic hyperparameter decoding network inversely transforms the coefficient hyperparameter feature z_hat to obtain the standard deviation σ. Among them, the probabilistic hyperparameter decoding network is also called the super-prior scale decoding network, and the standard deviation σ is also called the probability density parameter of the main information.

Exemplarily, after obtaining the standard deviation σ, the probability distribution parameter p can also be determined based on the standard deviation σ. For example, if the probability distribution parameter p only includes the standard deviation σ, the standard deviation σ can be used as the probability distribution parameter p, that is, the probability distribution parameter p with zero mean. Alternatively, if the probability distribution parameter p includes the standard deviation σ and the mean, the mean can also be obtained. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat are input to the mean prediction network, and the mean prediction network determines the mean mu (that is, the predicted value mu) based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. There is no restriction on this prediction process. In this way, the probability distribution parameter p can be determined based on the standard deviation σ and the mean mu. After obtaining the probability distribution parameter p, the decoding end can also generate a probability distribution model N(mu, σ) based on the probability distribution parameter p. There is no restriction on this process in this embodiment.

In summary, it can be seen that after the decoding end obtains the coefficient hyperparameter feature z_hat, the coefficient hyperparameter feature z_hat can be input into the probabilistic hyperparameter decoding network, and the coefficient hyperparameter feature z_hat can be inversely transformed through the probabilistic hyperparameter decoding network to obtain the standard deviation.

Step S24: The decoding end receives the second bitstream (Bitstream#2) corresponding to the current image block. The second bitstream can also be called the main information bitstream. The decoding end decodes the second bitstream based on the probability distribution model to obtain the decoded image features, and determines the residual features corresponding to the current image block based on the decoded image features. The residual features are the quantized residuals of the main information.

For example, the decoding end decodes the second bitstream based on the probability distribution model to obtain the image quantization feature (i.e., the decoded image feature), and dequantizes the image quantization feature to obtain the residual feature r_hat, i.e., the quantized residual Alternatively, the decoding end decodes the second bitstream based on the probability distribution model to obtain the image quantization feature (i.e., the decoded image feature), and dequantizes the image quantization feature to obtain the image feature s'. Alternatively, the decoding end decodes the second bitstream based on the probability distribution model to obtain the image feature s' (i.e., the decoded image feature). After obtaining the image feature s', the decoding end can perform feature recovery on the image feature s' to obtain the residual feature r_hat, i.e., the quantized residual

Step S25: The decoding end determines the initial image feature corresponding to the current image block based on the residual feature.

Exemplarily, after obtaining the residual feature r_hat, the decoder determines the image feature y_hat based on the residual feature r_hat and the prediction value mu (i.e., the mean value mu), and the image feature y_hat is the initial image feature corresponding to the current image block. For example, the sum of the residual feature r_hat and the prediction value mu is used as the initial image feature, which can also be called the main information quantization value

In order to obtain the predicted value mu, then: after obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block to obtain the predicted value mu corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat can be input into the mean prediction network, and the mean prediction network determines the predicted value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input can include the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat, and the two are combined to obtain a more accurate prediction value mu.

Step S26: The decoding end queries the target quantization scale corresponding to the standard deviation σ from the entropy rate fitting table Target Quantification Scale is the quantization scale closest to the standard deviation σ, and the target quantization scale is queried from the entropy rate fitting table (used to record the correspondence between the quantization scale and the quadratic term) The corresponding target quadratic term is obtained, and the target entropy gradient is determined based on the target quadratic term.

Exemplarily, an entropy rate fitting table can be maintained in advance, and the method for obtaining the entropy rate fitting table includes but is not limited to: for multiple standard deviations between the minimum standard deviation and the maximum standard deviation, the multiple standard deviations are uniformly quantized on a logarithmic scale to obtain multiple quantization scales corresponding to the multiple standard deviations; for each quantization scale, an entropy rate fitting can be performed on the quantization scale to obtain a quadratic term corresponding to the quantization scale, and the correspondence between the quantization scale and the quadratic term is recorded in the entropy rate fitting table.

For example, multiple standard deviations are divided between the minimum standard deviation σ _min and the maximum standard deviation σ _max . There is no limit on the number of standard deviations. For example, 64 standard deviations are divided. Assuming that the minimum standard deviation σ _min is 0.11 and the maximum standard deviation σ _max is 256, 64 values can be divided between 0.11 and 256 according to the geometric progression, that is, 64 standard deviations are obtained. Then, multiple standard deviations are uniformly quantized on a logarithmic scale to obtain multiple quantization scales corresponding to the multiple standard deviations. In other words, 64 quantization scales corresponding to 64 standard deviations can be obtained, which can be obtained by Indicates the quantitative scale corresponding to the standard deviation.

For each quantitative scale Quantitative scale Corresponding to an entropy about The curve can be fitted using a quadratic curve That is, we can quantify the scale Perform entropy rate fitting to obtain the quantitative scale The corresponding quadratic term a (i.e. the quadratic term in the quadratic curve). By taking the derivative of the above formula, we can know that Since each main information feature point has its corresponding main information feature residual Each Corresponding to the standard deviation σ _i of a feature point, for each standard deviation σ _i , the quantization scale closest to the standard deviation σ _i can be found from the 64 quantization scales, denoted as Each corresponds to an a _i , therefore, The calculation method can be as follows:

In summary, when enhancing the main information, the entropy gradient can be modeled using the fitted quadratic polynomial coefficients. Can be used to quantify the scale Perform entropy rate fitting to obtain the quantitative scale The corresponding quadratic term a, so that the quantitative scale can be recorded in the entropy rate fitting table Assuming that there are 64 quantization scales, the corresponding relationship between the 64 quantization scales and the quadratic term a can be recorded in the entropy rate fitting table.

For example, the decoder obtains the coefficient hyperparameter feature z_hat (quantized edge information ) After that, the coefficient hyperparameter feature z_hat is inversely transformed to obtain the standard deviation σ. On this basis, the decoder can query the target quantization scale corresponding to the standard deviation σ from the entropy rate fitting table Target Quantification Scale is the quantization scale closest to the standard deviation σ, and then the target quantization scale can be queried from the entropy rate fitting table The corresponding target quadratic term is obtained, and the target entropy gradient is determined based on the target quadratic term.

Step S27: The decoding end determines the target image feature based on the initial image feature, the entropy gradient offset step, the target entropy gradient, and the regularization term. The target entropy gradient and regularization term can be used to determine the target image feature y using the following expression (13). That is, the initial image feature y can be determined based on the entropy gradient offset step size ρ, the target entropy gradient and the regularization term. Perform an offset, such as offsetting the initial image features based on the entropy gradient offset step size ρ, the target entropy gradient and the regularization term The feature value of each feature point in is offset. Of course, expression (13) is just an example.

In expression (13), f represents a regularization term. The determination method of the regularization term f may include but is not limited to: based on the initial image features Standard deviation σ and residual features (such as quantized residual ) Determine the regularization term in, It can be a regularization term used to improve the structural stability of the feature domain representation and enhance the performance of different distortion evaluation index entropy. In expression (13), represents the target entropy gradient, a represents the target quadratic term, and N represents the total number of quantization scales, such as 64. In expression (13), It is used to represent the initial image features, which is obtained based on the residual feature r_hat and the predicted value mu. ρ is used to represent the entropy gradient offset step, which is decoded from the header information code stream corresponding to the current image block.

Step S28, the decoding end obtains the reconstructed image block corresponding to the current image block based on the target image feature. For example, the target image feature is input to the synthesis transformation network, and the target image feature is synthesized and transformed by the synthesis transformation network to obtain the reconstructed image block. Among them, the target image feature is the enhanced main information, and the reconstructed image block can be recorded as the reconstructed image block

Embodiment 8: In the embodiment of the present application, a coding method is proposed. Referring to FIG. 6D and FIG. 6E , a flow chart of the coding method is shown. The method can be applied to a coding end (also called a video encoder). The method may include:

Step S31: After obtaining the current image block x, the encoder performs analysis and transformation on the current image block x through an analysis and transformation network to obtain an image feature y corresponding to the current image block x. The image feature y may also be referred to as feature main information y.

Step S32: After obtaining the image feature y, the encoding end may also perform coefficient hyperparameter feature transformation on the image feature y to obtain coefficient hyperparameter feature z. For example, the encoding end may input the image feature y into a hyperparameter encoding network, and the hyperparameter encoding network may perform coefficient hyperparameter feature transformation on the image feature y to obtain coefficient hyperparameter feature z. The hyperparameter encoding network may also be referred to as a super priori encoding network, and the coefficient hyperparameter feature z may also be referred to as feature side information z.

Step S33: After obtaining the coefficient super-parameter feature z, the encoder can quantize the coefficient super-parameter feature z to obtain the super-parameter quantization feature corresponding to the coefficient super-parameter feature z. The super-parameter quantization feature can also be called the quantization feature side information.

Step S34: After obtaining the hyperparametric quantization feature corresponding to the coefficient hyperparametric feature z, the encoder encodes the hyperparametric quantization feature to obtain Bitstream#1 (i.e., the first bitstream). For example, the encoder uses a fixed factorized probability density p _f (·) for the hyperparametric quantization feature and uses a lossless entropy encoder to encode it into the first bitstream. The first bitstream can also be called a side information bitstream.

After obtaining the Bitstream#1 corresponding to the current image block, the encoder may send the Bitstream#1 corresponding to the current image block to the decoder. For the processing process of the decoder for the Bitstream#1 corresponding to the current image block, refer to Example 5-7.

Step S35, after obtaining Bitstream#1, the encoding end can also decode Bitstream#1 to obtain a hyperparametric quantization feature. For example, a fixed factorized probability density p _f (.) can be used to decode Bitstream#1 using a lossless entropy decoder to obtain a hyperparametric quantization feature, and then the hyperparametric quantization feature is dequantized to obtain a coefficient hyperparametric feature z_hat.

Step S36, after obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain a standard deviation σ. For example, the coefficient hyperparameter feature z_hat is input to a probabilistic hyperparameter decoding network, and the probabilistic hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain a standard deviation σ. After obtaining the standard deviation σ, a probability distribution model can be generated based on the standard deviation σ. Among them, the probabilistic hyperparameter decoding network can also be called a super-prior scale decoding network, and the standard deviation σ can also be called the probability distribution scale σ of the main information quantization feature.

Step S37: After obtaining the image feature y, the encoder can also determine the residual feature r based on the image feature y and the prediction value mu. For example, the difference between the image feature y and the prediction value mu can be used as the residual feature r. After obtaining the residual feature r, the encoder can also quantize the residual feature r to obtain a quantized image feature.

Exemplarily, after obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat are input into the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat.

Step S38, after obtaining the image quantization features, the encoding end can encode the image quantization features to obtain Bitstream#2 (i.e., the second code stream) corresponding to the current image block. For example, the encoding end can generate a probability distribution model based on the standard deviation σ, and use the probability distribution model to encode the image quantization features to obtain Bitstream#2.

After obtaining Bitstream#2 corresponding to the current image block, the encoder may send Bitstream#2 corresponding to the current image block to the decoder. For the processing of Bitstream#2 corresponding to the current image block by the decoder, see Example 5-7.

Step S39: After obtaining Bitstream#2, the encoder can also decode Bitstream#2 to obtain decoded image features. The decoded image features can be image quantization features. For example, the encoder decodes Bitstream#2 based on a probability distribution model to obtain image quantization features. After obtaining the image quantization features, the encoder can also dequantize the image quantization features to obtain residual features r_hat. The residual features r_hat are the quantization residual features.

Step S40, the encoder determines the initial image feature corresponding to the current image block based on the residual feature. For example, after obtaining the residual feature r_hat, the encoder determines the image feature y_hat based on the residual feature r_hat and the prediction value mu (i.e., the mean value mu), and the image feature y_hat is the initial image feature corresponding to the current image block. For example, the encoder uses the sum of the residual feature r_hat and the prediction value mu as the initial image feature, which can also be called the primary information quantization value

Step S41: For each entropy gradient offset step, the encoding end determines the target image feature based on the initial image feature and the entropy gradient offset step, and obtains the reconstructed image block corresponding to the entropy gradient offset step based on the target image feature.

In a possible implementation, multiple entropy gradient offset step sizes may be preconfigured, such as 10 entropy gradient offset step sizes. For each entropy gradient offset step size, the target image feature may be determined based on the initial image feature, the entropy gradient offset step size, and the entropy gradient offset value. The entropy gradient offset value is determined by: determining the residual incremental feature corresponding to the residual feature; calculating the entropy of the residual feature, calculating the entropy of the residual incremental feature; determining the entropy gradient of the residual feature based on the entropy of the residual feature and the entropy of the residual incremental feature, and determining the entropy gradient offset value based on the entropy gradient of the residual feature and the regularization term. Regarding how to determine the target image feature, please refer to step S16 of Example 5 and Example 6, which will not be repeated here.

In another possible implementation, multiple entropy gradient offset step sizes can be preconfigured. For each entropy gradient offset step size, the target quantization scale corresponding to the standard deviation can be queried from the entropy rate fitting table. The target quantization scale is the quantization scale closest to the standard deviation. The target quadratic term corresponding to the target quantization scale is queried from the entropy rate fitting table, and the target entropy gradient corresponding to the current image block is determined based on the target quadratic term. Then, the target image feature can be determined based on the initial image feature, the entropy gradient offset step size, the target entropy gradient, and the regularization term. Among them, the acquisition method of the entropy rate fitting table may include but is not limited to: for multiple standard deviations between the minimum standard deviation and the maximum standard deviation, the multiple standard deviations are uniformly quantized on a logarithmic scale to obtain multiple quantization scales corresponding to the multiple standard deviations; for each quantization scale, the entropy rate can be fitted to the quantization scale to obtain the quadratic term corresponding to the quantization scale, and the corresponding relationship between the quantization scale and the quadratic term is recorded in the entropy rate fitting table. Regarding how to determine the target image feature, please refer to step S26 and step S27 of embodiment 5, which will not be repeated here.

For each entropy gradient offset step, after obtaining the target image feature corresponding to the entropy gradient offset step, the reconstructed image block corresponding to the entropy gradient offset step can be obtained based on the target image feature. For example, the target image feature is input to the synthesis transformation network, and the target image feature is synthetically transformed through the synthesis transformation network to obtain the reconstructed image block.

Step S42: The encoder selects the entropy gradient offset step size corresponding to the current image block (i.e., the entropy gradient offset step size finally adopted) from all entropy gradient offset step sizes based on the current image block and the reconstructed image block corresponding to each entropy gradient offset step size.

Exemplarily, for each entropy gradient offset step, the encoding end may determine the cost value corresponding to the entropy gradient offset step based on the current image block and the reconstructed image block corresponding to the entropy gradient offset step (i.e., the cost value is determined based on the difference between the current image block and the reconstructed image block, and there is no limitation on this determination method). Based on the cost value corresponding to each entropy gradient offset step, the entropy gradient offset step corresponding to the minimum cost value may be selected as the entropy gradient offset step corresponding to the current image block.

Step S43, the encoding end encodes the entropy gradient offset step corresponding to the current image block in the header information code stream corresponding to the current image block. For example, the value of the entropy gradient offset step is encoded in the header information code stream corresponding to the current image block. Alternatively, if the configured entropy gradient offset step table includes at least one entropy gradient offset step, the indication information corresponding to the entropy gradient offset step is encoded in the header information code stream corresponding to the current image block, and the indication information corresponding to the entropy gradient offset step is used to indicate the index value of the entropy gradient offset step, that is, it indicates the number of the entropy gradient offset step in the entropy gradient offset step table.

In the above embodiment, the process of enhancing the initial image features by entropy gradient shift step to obtain the target image features can be called LEGS (Latent Entrophy Gradient Shift, feature entropy gradient shift) enhancement method. When the LEGS module is used in the end-to-end image coding framework, it can be seen as shown in Figures 6D and 6E. In Figures 6D and 6E, the dotted lines are used to distinguish new modules and new functions, and do not represent optional. For example, header information encoding and header information decoding are new functions of this embodiment, and the LEGS module is a new module of this embodiment. Exemplarily, the LEGS module can be a hardware module, the LEGS module can also be executed by a neural network, and the LEGS module can also be implemented by software code, which is not limited as long as the function of the LEGS module can be realized. For example, for the encoding end, the LEGS module receives the main information quantization features and the probability model parameters, evaluates the optimal shift step (i.e., the entropy gradient shift step corresponding to the current image block), and encodes the entropy gradient shift step corresponding to the current image block into the header information code stream. For the decoding end, the LEGS module uses the entropy gradient offset step corresponding to the current image block to offset and enhance the main information quantization feature (i.e., the initial image feature) to obtain the target image feature. Exemplarily, the LEGS module can select ρ with the smallest distortion loss as the optimal offset step, and encode the optimal offset step into the header information bitstream in the form of a fixed-length code, such as explicitly encoding it into the header information bitstream.

Exemplarily, when the encoder uses the LEGS module to determine the optimal offset step size, the change in the code stream size caused by encoding the optimal offset step size is negligible. Therefore, the following distortion loss can be used as a criterion for determining the optimal offset step size:

Embodiment 9: In embodiment 8, it is necessary to configure multiple entropy gradient offset steps, determine the reconstructed image block corresponding to each entropy gradient offset step, and select the entropy gradient offset step corresponding to the current image block from all entropy gradient offset steps based on the reconstructed image block corresponding to each entropy gradient offset step. In order to select the entropy gradient offset step corresponding to the current image block, the following method can be used: for each entropy gradient offset step, based on the current image block and the reconstructed image block corresponding to the entropy gradient offset step, determine the target cost value corresponding to the entropy gradient offset step, and determine the fidelity corresponding to the entropy gradient offset step based on the target cost value and the reference cost value; based on the fidelity corresponding to each entropy gradient offset step, select the entropy gradient offset step corresponding to the maximum fidelity as the entropy gradient offset step corresponding to the current image block; wherein the process of obtaining the reference cost value may include: after obtaining the initial image features corresponding to the current image block through the first neural network, based on the initial image features, obtain the reconstructed image block through the second neural network, and determine the reference cost value based on the current image block and the reconstructed image block.

For example, in the end-to-end image encoding and decoding framework of JPEG-AI, VMAF (Visual Multimethod Assessment Fusion) and FSIM (Feature Similarity Index Mersure) evaluation indicators can be used as criteria for evaluating the distortion of the LGBS module. The VMAF evaluation index will increase after the LGBS module is adjusted, while the FSIM index will decrease after the LGBS module is adjusted.

After obtaining the initial image features corresponding to the current image block, the initial image features are not adjusted, and are directly input into the synthesis transformation network to obtain the reconstructed image block. Based on the current image block and the reconstructed image block, the reconstruction value indicators fsim _org and VMAF _org can be obtained, and the reconstruction value indicators fsim _org and VMAF _org are used as reference cost values.

The initial image feature can be adjusted by using an entropy gradient offset step of 1 to obtain a target image feature 1, and the target image feature 1 is input into the synthetic transformation network to obtain a reconstructed image block corresponding to the entropy gradient offset step of 1. Based on the current image block and the reconstructed image block corresponding to the entropy gradient offset step of 1, the reconstruction value indicators fsim _now and VMAF _now can be obtained, and the reconstruction value indicators fsim _now and VMAF _now can be used as the target cost value corresponding to the entropy gradient offset step of 1.

The entropy gradient offset step 2 can be used to adjust the initial image features to obtain the target image features 2, and the target image features 2 are input to the synthetic transformation network to obtain the reconstructed image block corresponding to the entropy gradient offset step 2. Based on the current image block and the reconstructed image block corresponding to the entropy gradient offset step 2, the reconstruction value indicators fsim _now and VMAF _now can be obtained, and the reconstruction value indicators fsim _now and VMAF _now can be used as the target cost values corresponding to the entropy gradient offset step 2.

By analogy, the target cost value corresponding to each entropy gradient offset step can be obtained.

Exemplarily, for each entropy gradient offset step, based on the target cost value and the reference cost value corresponding to the entropy gradient offset step, the fidelity corresponding to the entropy gradient offset step can be determined using expression (15):

In expression (15), fsim _now and VMAF _now represent the target cost values corresponding to the entropy gradient offset step, and fsim _org and VMAF _org represent the reference cost values. 1 and 100 are numerical values configured based on experience, which are used to normalize the VMAF evaluation index and the FSIM evaluation index to the same order of magnitude, and can also be adjusted to other values without limitation.

Of course, expression (15) is just an example, and this embodiment does not limit the expression for determining the degree of fidelity.

Exemplarily, after obtaining the fidelity level corresponding to each entropy gradient offset step, the entropy gradient offset step corresponding to the maximum fidelity level may be selected as the entropy gradient offset step corresponding to the current image block.

For example, when configuring multiple entropy gradient offset step sizes (such as 10, 12, or 16 entropy gradient offset step sizes), one entropy gradient offset step size can be 0. In this way, if the entropy gradient offset step size corresponding to the maximum fidelity level is 0, it means that the entropy gradient offset step size does not need to be encoded, that is, the entropy gradient offset step size does not need to be used for feature enhancement processing.

Embodiment 10: As shown in FIG6F, the processing flow of adding a LEGS module to the decoding end of JPEG-AI, the position of the LEGS module in the JPEG-AI end-to-end image encoding and decoding framework can be seen in FIG6F. The side information code stream (i.e., the first code stream) obtains the side information feature through a fixed factorization model, and the side information feature obtains the scale of the normal distribution probability model through a super-prior scale decoding network. The normal probability density model and the main information code stream (i.e., the second code stream) are subjected to lossless entropy decoding to obtain the main information feature residual. Main information feature quantization residual Input the super prior decoding network to obtain The main information quantitative features are obtained through the autoregressive process of the context network and the prediction fusion network. After the LEGS module obtains the entropy gradient offset step from the header information stream, it uses Entropy gradient adjustment The reconstructed image block is obtained after the synthetic variation network.

Embodiment 11: Taking the end-to-end image coding and decoding framework JPEG-AI as an example, for an image sample x, such as a current image block x or a current image x, the current image block x or the current image x can be a luminance component or a chrominance component. The size of the image sample x is C _p ×H×W, which can be divided into a luminance component x _Y and a chrominance component x _UV . The size of the luminance component x _Y is 1×H×W, and the size of the chrominance component x _UV is 2×H×W. The reconstruction process of the JPEG-AI component encoding and decoding and the location of the LEGS module can be seen in FIG6G , which is a reconstruction flow chart of the JPEG-AI component.

Taking the brightness component x _Y as an example, x _Y is subjected to the Analysis Transform Net ( _ga ) to obtain the feature tensor y _Y , whose size is C _p ×h _Y ×w _Y . _{y Y} is subjected to the Hyper Encoder Net to obtain the hyper priori feature tensor z _Y , whose size is C _p ×h _hpY ×w _hpY . Each element (i.e., each feature point) in _{z Y} has a fixed cumulative probability distribution function (Cumulative Distribution Function, CDF), and the same channel of z _Y has the same CDF. The CDF of all elements is obtained by the Factorize Entrophy Model. z _Y is rounded and quantized to obtain the quantized value Input a lossless entropy encoder together with CDF to obtain bitstream 1 (BitStream#1); bitstream 1 is passed through a lossless entropy decoder to obtain Obtained through super-prior decoding Its size is 2C _p ×h _hpY ×w _hpY , is the quantized value of y _Y , whose size is 2C _p ×h _pY ×w _pY . ζ is obtained through the context model (Context Model Net), with a size of 2C _p ×h _hpY ×w _hpY ; Connect with ζ in the channel dimension and input the prediction fusion network (Prediction Fusion Net) to obtain the mean μ _y , whose size is C _p ×h _hpY ×w _hpY . y _Y -μ _y obtains the residual _ry , and _ry is rounded and quantized to obtain the quantized residual Adding μ _y gives μ _y ， _ry ， The generation of is an autoregressive process, i.e., pixel-by-pixel along the raster order in two spatial dimensions. Through a super prior scale decoding network, σ _Y is obtained, whose size is C _p ×h _Y ×w _Y , and its parameters are Generate a CDF for each element in The corresponding CDF is input into the lossless entropy encoder to obtain bitstream 2 (BitStream#2). Bitstream 2 (BitStream#2) is decoded by lossless entropy decoding to obtain Adding μ _y gives After entering the synthesis transformation network, the brightness reconstruction component is obtained

For example, the processing of the chrominance component _xUV is similar to the processing of the luminance component _xY , and the chrominance reconstruction component can be obtained. Luminance reconstruction component Chroma reconstruction component After connection, the reconstructed image is obtained after the chroma upsampling module

Adding LEGS module to the end-to-end image encoding and decoding network enhances and The representation capability before entering the synthetic variation network reduces the distortion of the reconstructed image and improves the encoding performance. In the encoding process, before the bitstream 2 is generated, the impact of the offset on the distortion is evaluated, and the optimal entropy gradient offset step size is selected. The optimal entropy gradient offset step size is included in the header information bitstream, but not in the bitstream. During the decoding process, the entropy gradient offset step can be obtained by decoding the header information stream. Before synthesizing the change network, the LEGS module Regarding the position of the entropy gradient offset step in the JPEG-AI framework, please refer to FIG. 6G , in which the dotted box shows the position of the entropy gradient offset step.

Embodiment 12: A decoding method is proposed in the embodiment of the present application. Referring to FIG. 7A , it is a flowchart of the decoding method. The method can be applied to a decoding end (also called a video decoder). The method may include:

Step 711: Obtain residual features corresponding to the current image block through the first neural network.

Exemplarily, obtaining the residual features corresponding to the current image block through the first neural network may include but is not limited to: obtaining the standard deviation through the first neural network based on the first bitstream corresponding to the current image block; determining the probability distribution model based on the standard deviation, and decoding the second bitstream corresponding to the current image block based on the probability distribution model to obtain the decoded image features; the residual features corresponding to the current image block may be determined based on the decoded image features. Exemplarily, the first neural network may include but is not limited to a probabilistic hyperparameter decoding network, and obtaining the standard deviation through the first neural network based on the first bitstream corresponding to the current image block may include but is not limited to: decoding the first bitstream corresponding to the current image block based on a factorized probability model to obtain the decoded image features, and determining the coefficient hyperparameter features based on the decoded image features; inputting the coefficient hyperparameter features into the probabilistic hyperparameter decoding network, and inversely transforming the coefficient hyperparameter features through the probabilistic hyperparameter decoding network to obtain the standard deviation.

Step 712: Generate initial image features based on the residual features and the mean features corresponding to the current image block.

Step 713: Perform feature domain enhancement on the target channel of the initial image feature to obtain the target image feature.

Exemplarily, the initial image features may include multiple channels. Based on the consumption bit rate corresponding to each channel, the multiple channels may be sorted in descending order of the consumption bit rate, and the top K channels in the sorting may be selected as target channels, where K is a positive integer; alternatively, a channel whose consumption bit rate is greater than a preset bit rate threshold may be selected as the target channel.

Exemplarily, the feature domain enhancement of the target channel of the initial image feature may include, but is not limited to: if the target channel includes multiple feature points, then the target feature point is selected from the multiple feature points; wherein all feature points of the target channel are target feature points, or, for each feature point, if the standard deviation corresponding to the feature point is greater than the standard deviation threshold, then the feature point is the target feature point, and if the standard deviation corresponding to the feature point is not greater than the standard deviation threshold, then the feature point is not the target feature point. After obtaining the target feature point, for each target feature point, the feature compensation value corresponding to the target feature point can be determined, and the feature value of the target feature point can be compensated based on the feature compensation value.

Exemplarily, determining the feature compensation value corresponding to the target feature point may include, but is not limited to: decoding the header information code stream corresponding to the current image block, obtaining the adjustment sign bit corresponding to the target feature point, and determining the sign of the feature compensation value based on the adjustment sign bit. In addition, determining the feature compensation value corresponding to the target feature point may also include, but is not limited to: decoding the header information code stream corresponding to the current image block, obtaining the amplitude indication information corresponding to the target feature point, and determining the amplitude of the feature compensation value based on the amplitude indication information; or obtaining a configured fixed amplitude, and determining the amplitude of the feature compensation value based on the fixed amplitude.

Exemplarily, determining the amplitude of the characteristic compensation value based on the amplitude indication information may include, but is not limited to: if the amplitude indication information includes a position index in the amplitude list (used to indicate the number of the amplitude in the amplitude list), then selecting the amplitude corresponding to the position index from the amplitude list, and determining the selected amplitude as the amplitude of the characteristic compensation value.

Step 714: Based on the target image features, a reconstructed image block is obtained through a second neural network.

Exemplarily, the first neural network includes at least one convolutional layer, and the second neural network includes at least one convolutional layer.

Exemplarily, the second neural network may include a synthetic transformation network. Based on the target image features, obtaining the reconstructed image block through the second neural network may include but is not limited to: inputting the target image features into the synthetic transformation network, performing a synthetic transformation on the target image features through the synthetic transformation network, and obtaining a reconstructed image block corresponding to the current image block.

Exemplarily, the above execution order is only for the convenience of describing the examples given. In practical applications, the execution order between the steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

It can be seen from the above technical scheme that in the embodiment of the present application, the residual features corresponding to the current image block are obtained through the first neural network, and the initial image features are generated based on the residual features. The target image features are obtained by performing feature domain enhancement on the target channel of the initial image features. Based on the target image features, the reconstructed image block corresponding to the current image block is obtained through the second neural network, thereby proposing an end-to-end video image compression method, which can realize encoding and decoding of video images based on neural networks, and achieve the purpose of improving encoding efficiency and decoding efficiency by performing feature domain enhancement on the target channel, so that the neural network can effectively guarantee the quality of the reconstructed image block while maintaining low complexity, thereby achieving the purpose of improving encoding performance and decoding performance.

Embodiment 13: In the embodiment of the present application, a coding method is proposed. Referring to FIG. 7B , it is a schematic diagram of the flow of the coding method. The method can be applied to a coding end (also called a video encoder). The method may include:

Step 721: Obtain the residual features corresponding to the current image block.

Step 722: quantize the residual features to obtain the initial image features corresponding to the current image block.

Step 723: Determine a feature compensation value corresponding to a target feature point in a target channel based on a target channel and a residual feature of the initial image feature corresponding to the current image block (ie, a residual feature corresponding to the current image block).

Exemplarily, the initial image features may include multiple channels. Based on the consumption bit rate corresponding to each channel, the multiple channels may be sorted in descending order of the consumption bit rate, and the top K channels in the sorting may be selected as target channels, where K is a positive integer; alternatively, a channel whose consumption bit rate is greater than a preset bit rate threshold may be selected as the target channel.

Step 724: Encode the indication information of the feature compensation value in the header information code stream corresponding to the current image block.

Exemplarily, the indication information of the feature compensation value may include an adjustment sign bit corresponding to the target feature point; if the difference between the eigenvalue corresponding to the target feature point in the residual feature and the eigenvalue corresponding to the target feature point in the target channel is greater than 0, the adjustment sign bit corresponding to the target feature point is positive; if the difference between the eigenvalue corresponding to the target feature point in the residual feature and the eigenvalue corresponding to the target feature point in the target channel is less than 0, the adjustment sign bit corresponding to the target feature point is negative.

Exemplarily, the indication information of the feature compensation value includes amplitude indication information corresponding to the target feature point; if the amplitude indication information includes a position index in the amplitude list, the position index indicates the position corresponding to the amplitude of the feature compensation value in the amplitude list.

Exemplarily, the above execution order is only for the convenience of describing the examples given. In practical applications, the execution order between the steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

It can be seen from the above technical scheme that in the embodiment of the present application, an end-to-end video image compression method is proposed by performing feature domain enhancement on the target channel of the initial image feature, which can realize encoding and decoding of video images based on a neural network, and achieve the purpose of improving the encoding efficiency and decoding efficiency by performing feature domain enhancement on the target channel, so that the neural network can effectively guarantee the quality of the reconstructed image block while maintaining low complexity, thereby achieving the purpose of improving the encoding performance and decoding performance.

Example 14: In Example 12 and Example 13, the encoding end and the decoding end can obtain residual features and initial image features. The acquisition process can refer to Example 1-Example 11 and will not be repeated here. After obtaining the target image features, the reconstructed image blocks can be obtained based on the target image features. The acquisition process can refer to Example 1-Example 11 and will not be repeated here. The following describes the process of obtaining the target image features based on the initial image features.

Exemplarily, the key channel of the initial image feature (the key channel is referred to as the target channel) can be compensated to perform feature domain enhancement and enhance the quality of the reconstructed image. For example, the following steps can be used for compensation:

Step S51: Select a target channel from all channels of the initial image feature. For example, the initial image feature includes multiple channels. The target channel can be selected from all channels of the initial image feature based on the consumed bit rate corresponding to each channel.

In a possible implementation, the initial image feature may include multiple channels. Based on the consumed bit rate corresponding to each channel, the multiple channels may be sorted in descending order of the consumed bit rate, and the top K channels are selected as target channels, where K is a positive integer. Alternatively, based on the consumed bit rate corresponding to each channel, the multiple channels may be sorted in ascending order of the consumed bit rate, and the bottom K channels are selected as target channels.

In another possible implementation, the initial image feature may include multiple channels, and based on the consumed bit rate corresponding to each channel, a channel whose consumed bit rate is greater than a preset bit rate threshold may be selected as a target channel.

Of course, the above are just two examples, and there is no limitation on the method of selecting the target channel.

For example, the brightness component feature of the initial image feature has 128 channels. If the number of target channels is 1, the initial image feature can be used to calculate the channel with the largest bit rate consumption, and the channel with the largest bit rate consumption can be used as the target channel.

Step S52: Perform feature domain enhancement on the target channel of the initial image feature to obtain the target image feature.

For example, the target channel of the initial image feature can be recorded as The height of the target channel is h, the width of the target channel is w, and the target channel is a feature channel map of the initial image feature. Therefore, the target channel includes h*w feature points, and all feature points of the target channel are selected as target feature points (that is, there are a total of h*w target feature points).

For each target feature point, a feature compensation value corresponding to the target feature point can be determined, and the feature value of the target feature point can be compensated based on the feature compensation value to obtain a compensated feature value. After obtaining the compensated feature value corresponding to each target feature point, the compensated feature values corresponding to all target feature points can be combined into a target image feature.

In order to determine the characteristic compensation value corresponding to the target feature point, the adjustment sign bit corresponding to the target feature point can be determined first. If the adjustment sign bit is positive or negative, the amplitude corresponding to the target feature point can be determined, such as the amplitude A. If the adjustment sign bit is positive, the characteristic compensation value corresponding to the target feature point is -A. When compensating the characteristic value of the target feature point based on the characteristic compensation value, A is subtracted from the characteristic value of the target feature point to obtain the compensated characteristic value, that is, when the adjustment sign bit is positive, the characteristic value of the target feature point is compensated by -A. If the adjustment sign bit is negative, the characteristic compensation value corresponding to the target feature point is +A. When compensating the characteristic value of the target feature point based on the characteristic compensation value, A is added to the characteristic value of the target feature point to obtain the compensated characteristic value, that is, when the adjustment sign bit is negative, the characteristic value of the target feature point is compensated by +A.

In order to determine the adjustment sign bit corresponding to the target feature point, the encoder can calculate the adjustment sign bit corresponding to the target feature point. If the difference between the eigenvalue corresponding to the target feature point in the residual feature and the eigenvalue corresponding to the target feature point in the target channel is greater than 0, the adjustment sign bit corresponding to the target feature point is positive; if the difference between the eigenvalue corresponding to the target feature point in the residual feature and the eigenvalue corresponding to the target feature point in the target channel is less than 0, the adjustment sign bit corresponding to the target feature point is negative. For example, the encoder can use the following expression to calculate the difference between the main information value and the main information quantization value of the target channel: y _impotant represents the eigenvalue corresponding to the target feature point in the target channel of the residual feature. It represents the eigenvalue corresponding to the target feature point in the target channel of the initial image feature. If diff is greater than 0, the adjustment sign bit corresponding to the target feature point is positive. If diff is less than 0, the adjustment sign bit corresponding to the target feature point is negative.

Assuming that the target channel includes h*w target feature points, the adjustment sign bit corresponding to each target feature point can be obtained, and the adjustment sign bits corresponding to the h*w target feature points can be encoded into the header information code stream with a 1-bit fixed-length code, that is, the header information code stream can transmit h*w bits, each bit represents the adjustment sign bit (positive or negative) corresponding to a target feature point.

The decoding end can decode the header information code stream corresponding to the current image block, obtain the adjustment sign bit corresponding to each target feature point, and obtain the sign corresponding to each target feature point based on the adjustment sign bit, such as positive or negative.

In order to determine the amplitude corresponding to the target feature point, the decoding end can use the fixed amplitude as the amplitude corresponding to the target feature point. The fixed amplitude can be configured based on experience. For example, the fixed amplitude can be 0.25, 0.75, 1.25, 1.75, etc. Assuming that the fixed amplitude is 0.25, then the decoding end can use 0.25 as the amplitude corresponding to the target feature point.

In summary, if the adjustment sign bit corresponding to the target feature point is positive, the feature compensation value corresponding to the target feature point is -0.25, and if the adjustment sign bit corresponding to the target feature point is negative, the feature compensation value corresponding to the target feature point is +0.25.

It should be noted that if the number of target channels is 1, that is, the channel with the largest bit rate consumption is used as the target channel, then the header information bitstream does not need to transmit the identifier of the target channel, and the decoding end can implicitly derive the target channel, that is, the channel with the largest bit rate consumption is used as the target channel. If the number of target channels is multiple, such as X, the header information bitstream does not need to transmit the identifiers of the X target channels, and the decoding end can implicitly derive the X target channels. The derivation process can refer to step S51.

Example 15: In Example 12 and Example 13, the encoding end and the decoding end can obtain residual features and initial image features. The acquisition process can refer to Example 1-Example 11 and will not be repeated here. After obtaining the target image features, the reconstructed image blocks can be obtained based on the target image features. The acquisition process can refer to Example 1-Example 11 and will not be repeated here. The following describes the process of obtaining the target image features based on the initial image features.

For example, the quantization module has a residual skipping technique, which directly sets the main information residual to zero according to the set σ threshold, resulting in that some main information residuals have large values but are directly set to zero, resulting in the point-based There is a large gap between y and y. Therefore, in addition to the channel with the largest bit rate consumption as the target channel, some channels in other channels are also target channels.

Exemplarily, the key channel of the initial image feature (the key channel is referred to as the target channel) can be compensated to perform feature domain enhancement and enhance the quality of the reconstructed image. For example, the following steps can be used for compensation:

Step S61: Select a target channel from all channels of the initial image feature. For example, the initial image feature includes multiple channels. The target channel can be selected from all channels of the initial image feature based on the consumed bit rate corresponding to each channel.

In a possible implementation, the initial image feature may include multiple channels. Based on the consumed bit rate corresponding to each channel, the multiple channels may be sorted in descending order of the consumed bit rate, and the top K channels are selected as target channels, where K is a positive integer. Alternatively, based on the consumed bit rate corresponding to each channel, the multiple channels may be sorted in ascending order of the consumed bit rate, and the bottom K channels are selected as target channels.

In another possible implementation, the initial image feature may include multiple channels, and based on the consumed bit rate corresponding to each channel, a channel whose consumed bit rate is greater than a preset bit rate threshold may be selected as a target channel.

Of course, the above are just two examples, and there is no limitation on the method of selecting the target channel.

Step S62: The target channel (can be multiple) of the initial image feature can be recorded as The height of the target channel is h, the width of the target channel is w, and the target channel is the feature channel map of the initial image feature. Therefore, the target channel includes h*w feature points. For each feature point, if the standard deviation corresponding to the feature point is greater than the standard deviation threshold, the feature point is selected as the target feature point. If the standard deviation corresponding to the feature point is not greater than the standard deviation threshold, the feature point is not selected as the target feature point. At this point, the target feature point can be selected from all the feature points of the target channel.

Exemplarily, the target feature points in the target channel are determined according to the standard deviation threshold. The target feature points can be distributed in each feature channel (it can be allowed that there is no target feature point in the feature channel). The standard deviation thresholds for determining the target feature points in each feature channel can be different or the same, and the number of target feature points depends on the setting of the standard deviation threshold.

Step S63: Perform feature domain enhancement on the target channel of the initial image feature to obtain the target image feature.

For each target feature point, a feature compensation value corresponding to the target feature point can be determined, and the feature value of the target feature point can be compensated based on the feature compensation value to obtain a compensated feature value. After obtaining the compensated feature value corresponding to each target feature point, the compensated feature values corresponding to all target feature points can be combined into a target image feature.

In order to determine the characteristic compensation value corresponding to the target feature point, the adjustment sign bit corresponding to the target feature point can be determined first. For example, the adjustment sign bit can be positive or negative, and the amplitude corresponding to the target feature point can be determined, such as the amplitude A. If the adjustment sign bit is positive, the characteristic compensation value corresponding to the target feature point is -A. When compensating the characteristic value of the target feature point based on the characteristic compensation value, A is subtracted from the characteristic value of the target feature point to obtain the compensated characteristic value, that is, when the adjustment sign bit is positive, the characteristic value of the target feature point is compensated by -A. If the adjustment sign bit is negative, the characteristic compensation value corresponding to the target feature point is +A. When compensating the characteristic value of the target feature point based on the characteristic compensation value, A is added to the characteristic value of the target feature point to obtain the compensated characteristic value, that is, when the adjustment sign bit is negative, the characteristic value of the target feature point is compensated by +A.

In order to determine the adjustment sign bit corresponding to the target feature point, the encoder can calculate the adjustment sign bit corresponding to the target feature point. If the difference between the eigenvalue corresponding to the target feature point in the residual feature and the eigenvalue corresponding to the target feature point in the target channel is greater than 0, the adjustment sign bit corresponding to the target feature point is positive; if the difference between the eigenvalue corresponding to the target feature point in the residual feature and the eigenvalue corresponding to the target feature point in the target channel is less than 0, the adjustment sign bit corresponding to the target feature point is negative. For example, the encoder can use the following expression to calculate the difference between the main information value and the main information quantization value of the target channel: y _impotant represents the eigenvalue corresponding to the target feature point in the target channel of the residual feature. It represents the eigenvalue corresponding to the target feature point in the target channel of the initial image feature. If diff is greater than 0, the adjustment sign bit corresponding to the target feature point is positive. If diff is less than 0, the adjustment sign bit corresponding to the target feature point is negative.

For each target feature point in the target channel, a 1-bit fixed-length code can be used to encode the adjustment sign bit corresponding to the target feature point into the header information code stream, and each bit represents the adjustment sign bit (positive or negative) corresponding to a target feature point.

The decoding end can decode the header information code stream corresponding to the current image block, obtain the adjustment sign bit corresponding to each target feature point, and obtain the sign corresponding to each target feature point based on the adjustment sign bit, such as positive or negative.

In order to determine the amplitude corresponding to the target feature point, both the encoding end and the decoding end need to maintain an amplitude list, which may include multiple amplitudes, such as 0.25, 0.75, 1.25, 1.75, etc. Of course, here are just a few examples of amplitudes, and there is no restriction on this. When the amplitude list includes amplitudes such as 0.25, 0.75, 1.25, 1.75, the order of these amplitudes can be arbitrarily configured without restriction, such as the order of these amplitudes can be 0.25, 0.75, 1.25, 1.75.

Exemplarily, the encoding end can determine the amplitude corresponding to the target feature point, such as determining the amplitude for each target feature point separately (that is, when the target channel includes multiple target feature points, the amplitude is determined separately for each target feature point), or determining the same amplitude for all target feature points of the same target channel (that is, when the target channel includes multiple target feature points, the same amplitude is determined for all target feature points). Taking the same amplitude as an example, the encoding end can determine the amplitude of the target channel according to the average size of the diff of the target channel (that is, the average value of the diff of all target feature points of the target channel), and the amplitude of the target channel is used as the amplitude of all target feature points of the target channel.

Then, the encoder encodes the amplitude indication information corresponding to the target feature point in the header information code stream corresponding to the current image block, and the amplitude indication information includes the position index in the amplitude list. For example, assuming that there are X target channels in total, X amplitude indication information is encoded in the header information code stream (X amplitude indication information corresponds to X target channels), and each amplitude indication information can be a 2-bit code word (of course, the number of bits of the code word can be more or less, and there is no restriction on this). It should be noted that for the X target channels, the identifiers of the X target channels do not need to be transmitted in the header information code stream, and the decoder can implicitly derive the X target channels. The derivation process can refer to step S61. Obviously, according to the threshold value of σ, the encoder and decoder can obtain a unified target channel, that is, X does not need to be transmitted through the header information code stream.

For X amplitude indication information (the amplitude indication information can be a 2-bit codeword), the first codeword represents the amplitude of the first target channel. For example, "0" represents the first amplitude in the amplitude list, such as 0.25. The decoding end determines that the amplitudes of all target feature points in the first target channel are 0.25 based on the amplitude indication information; if "1" represents the second amplitude in the amplitude list, such as 0.75, the decoding end determines that the amplitudes of all target feature points in the first target channel are 0.75 based on the amplitude indication information; if "2" represents the third amplitude in the amplitude list, such as 1.25, the decoding end determines that the amplitudes of all target feature points in the first target channel are 1.25 based on the amplitude indication information; if "3" represents the fourth amplitude in the amplitude list, such as 1.75, the decoding end determines that the amplitudes of all target feature points in the first target channel are 1.75 based on the amplitude indication information.

The second codeword represents the amplitude of the second target channel, such as "0" represents the first amplitude in the amplitude list, such as 0.25, "1" represents the second amplitude in the amplitude list, such as 0.75, "2" represents the third amplitude in the amplitude list, such as 1.25, and "3" represents the fourth amplitude in the amplitude list, such as 1.75. The third codeword represents the amplitude of the second target channel, and so on. The Xth codeword represents the amplitude of the Xth target channel, so that the amplitude of each target channel can be obtained, that is, for each target channel, the decoding end can determine the amplitudes of all target feature points in the target channel.

To sum up, if the adjustment sign bit corresponding to the target feature point is positive and the amplitude corresponding to the target feature point is 0.25, then the feature compensation value corresponding to the target feature point is -0.25; if the adjustment sign bit corresponding to the target feature point is negative and the amplitude corresponding to the target feature point is 0.25, then the feature compensation value corresponding to the target feature point is +0.25.

Illustratively, the above-mentioned embodiments can be implemented individually or in combination. For example, each embodiment in Embodiment 1 to Embodiment 15 can be implemented individually, and at least two embodiments in Embodiment 1 to Embodiment 15 can be implemented in combination.

Illustratively, in the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.

Based on the same application concept as the above method, a decoding device is also proposed in the embodiment of the present application. The device is applied to the decoding end, and the device includes: a memory, which is configured to store video data; a decoder, which is configured to implement the decoding method in the above embodiments 1 to 15, that is, the processing flow of the decoding end.

For example, in one possible implementation, the decoder is configured to implement:

Based on the first bitstream corresponding to the current image block, a standard deviation corresponding to the current image block is obtained through a probabilistic hyperparameter decoding network; a probability distribution model is determined based on the standard deviation, and a second bitstream corresponding to the current image block is decoded based on the probability distribution model to obtain decoded image features; based on the decoded image features, a residual feature corresponding to the current image block is determined; based on the residual feature, an initial image feature corresponding to the current image block is determined;

Obtaining the entropy gradient offset step corresponding to the current image block;

Performing feature enhancement on the initial image features based on the entropy gradient offset step to obtain target image features;

Based on the target image feature, a reconstructed image block corresponding to the current image block is obtained through a synthetic transformation network.

Exemplarily, the decoder is configured to further implement: decoding the first code stream corresponding to the current image block based on a factorized probability model to obtain decoded image features, and determining coefficient hyperparameter features based on the decoded image features; inputting the coefficient hyperparameter features to the probabilistic hyperparameter decoding network, and inversely transforming the coefficient hyperparameter features through the probabilistic hyperparameter decoding network to obtain the standard deviation corresponding to the current image block.

Exemplarily, the decoder is configured to further implement: decoding the header information code stream corresponding to the current image block to obtain the entropy gradient offset step; or, decoding the header information code stream corresponding to the current image block to obtain indication information corresponding to the entropy gradient offset step, and determining the entropy gradient offset step based on the indication information.

Exemplarily, a decoder is configured to further implement: if the configured entropy gradient offset step table includes at least one entropy gradient offset step, and the indication information is used to indicate an index value of the entropy gradient offset step, then the entropy gradient offset step corresponding to the current image block is determined from the entropy gradient offset step table based on the index value.

Exemplarily, a decoder is configured to further implement: based on the entropy gradient offset step and the entropy gradient feature information corresponding to the current image block, the feature of the initial image is enhanced to obtain the target image feature; wherein the entropy gradient offset step represents the offset step of the entropy gradient, and the entropy gradient offset step is a correction step used to enhance the initial image feature.

Exemplarily, the entropy gradient feature information includes an entropy gradient offset value, and the decoder is configured to further implement: determining the entropy gradient offset value corresponding to the current image block based on the residual feature.

Exemplarily, the decoder is configured to further implement: determining the residual incremental feature corresponding to the residual feature; calculating the entropy of the residual feature, and calculating the entropy of the residual incremental feature; determining the entropy gradient of the residual feature based on the entropy of the residual feature and the entropy of the residual incremental feature, and determining the entropy gradient offset value based on the entropy gradient of the residual feature and a regularization term.

Exemplarily, the entropy gradient feature information includes a target entropy gradient, and the decoder is configured to further implement: querying a target quantization scale corresponding to the standard deviation from an entropy rate fitting table, wherein the target quantization scale is a quantization scale closest to the standard deviation, and querying a target quadratic term corresponding to the target quantization scale from an entropy rate fitting table; wherein the entropy rate fitting table includes a correspondence between a quantization scale and a quadratic term; and determining a target entropy gradient corresponding to the current image block based on the target quadratic term.

Exemplarily, the decoder is configured to further implement: for multiple standard deviations between the minimum standard deviation and the maximum standard deviation, uniformly quantizing the multiple standard deviations on a logarithmic scale to obtain multiple quantization scales corresponding to the multiple standard deviations; for each quantization scale, performing entropy rate fitting on the quantization scale to obtain a quadratic term corresponding to the quantization scale, and recording the correspondence between the quantization scale and the quadratic term in an entropy rate fitting table.

Exemplarily, the decoder is configured to further implement: determining the regularization term based on the initial image feature, the standard deviation and the residual feature.

Based on the same application concept as the above method, an encoding device is also proposed in the embodiment of the present application. The device is applied to the encoding end, and the device includes: a memory, which is configured to store video data; an encoder, which is configured to implement the encoding method in the above embodiments 1 to 15, that is, the processing flow of the encoding end.

For example, in one possible implementation, the encoder is configured to implement:

Based on the first bitstream corresponding to the current image block, a standard deviation corresponding to the current image block is obtained through a probabilistic hyperparameter decoding network; a probability distribution model is determined based on the standard deviation, and a second bitstream corresponding to the current image block is decoded based on the probability distribution model to obtain decoded image features; based on the decoded image features, a residual feature corresponding to the current image block is determined; based on the residual feature, an initial image feature corresponding to the current image block is determined;

For each entropy gradient offset step, the initial image feature is enhanced based on the entropy gradient offset step to obtain a target image feature; based on the target image feature, a reconstructed image block corresponding to the entropy gradient offset step is obtained through a synthetic transformation network;

Based on the current image block and the reconstructed image block corresponding to each entropy gradient offset step, the entropy gradient offset step corresponding to the current image block is selected from all entropy gradient offset steps, and the entropy gradient offset step corresponding to the current image block is encoded in the header information code stream corresponding to the current image block.

Exemplarily, the encoder is configured to further implement: for each entropy gradient offset step, based on the current image block and the reconstructed image block corresponding to the entropy gradient offset step, determine the target cost value corresponding to the entropy gradient offset step, and determine the fidelity corresponding to the entropy gradient offset step based on the target cost value and the reference cost value; based on the fidelity corresponding to each entropy gradient offset step, select the entropy gradient offset step corresponding to the maximum fidelity as the entropy gradient offset step corresponding to the current image block; wherein the process of obtaining the reference cost value includes: after obtaining the initial image features corresponding to the current image block, based on the initial image features, obtaining the reconstructed image block through a synthetic transformation network, and determining the reference cost value based on the current image block and the reconstructed image block.

Based on the same application concept as the above method, the decoding end device (also referred to as a video decoder) provided in the embodiment of the present application, from the hardware level, its hardware architecture diagram can be specifically shown in Figure 8A. It includes: a processor 811 and a machine-readable storage medium 812, the machine-readable storage medium 812 stores machine-executable instructions that can be executed by the processor 811; the processor 811 is used to execute the machine-executable instructions to implement the decoding method of the above embodiments 1-15 of the present application.

For example, in one possible implementation, the processor 811 executes the machine executable instructions to implement:

Based on the first bitstream corresponding to the current image block, a standard deviation corresponding to the current image block is obtained through a probabilistic hyperparameter decoding network; a probability distribution model is determined based on the standard deviation, and a second bitstream corresponding to the current image block is decoded based on the probability distribution model to obtain decoded image features; based on the decoded image features, a residual feature corresponding to the current image block is determined; based on the residual feature, an initial image feature corresponding to the current image block is determined;

Obtaining the entropy gradient offset step corresponding to the current image block;

Performing feature enhancement on the initial image features based on the entropy gradient offset step to obtain target image features;

Based on the target image feature, a reconstructed image block corresponding to the current image block is obtained through a synthetic transformation network.

Based on the same application concept as the above method, the encoding end device (also referred to as a video encoder) provided in the embodiment of the present application, from the hardware level, its hardware architecture diagram can be specifically shown in Figure 8B. It includes: a processor 821 and a machine-readable storage medium 822, the machine-readable storage medium 822 stores machine-executable instructions that can be executed by the processor 821; the processor 821 is used to execute the machine-executable instructions to implement the encoding method of the above embodiments 1-15 of the present application.

For example, in one possible implementation, the processor 821 executes the machine executable instructions to implement:

Based on the first bitstream corresponding to the current image block, a standard deviation corresponding to the current image block is obtained through a probabilistic hyperparameter decoding network; a probability distribution model is determined based on the standard deviation, and a second bitstream corresponding to the current image block is decoded based on the probability distribution model to obtain decoded image features; based on the decoded image features, a residual feature corresponding to the current image block is determined; based on the residual feature, an initial image feature corresponding to the current image block is determined;

For each entropy gradient offset step, the initial image feature is enhanced based on the entropy gradient offset step to obtain a target image feature; based on the target image feature, a reconstructed image block corresponding to the entropy gradient offset step is obtained through a synthetic transformation network;

Based on the current image block and the reconstructed image block corresponding to each entropy gradient offset step, the entropy gradient offset step corresponding to the current image block is selected from all entropy gradient offset steps, and the entropy gradient offset step corresponding to the current image block is encoded in the header information code stream corresponding to the current image block.

Based on the same application concept as the above method, an embodiment of the present application provides an electronic device, including: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the decoding method or encoding method of the above embodiments 1-15 of the present application.

Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented, such as the decoding method or encoding method in the above embodiments.

Based on the same application concept as the above method, an embodiment of the present application also provides a computer application, which, when executed by a processor, can implement the decoding method or encoding method disclosed in the above example of the present application.

Based on the same application concept as the above method, a decoding device is also proposed in an embodiment of the present application. The decoding device can be applied to a decoding end (also referred to as a video decoder). The decoding device may include:

A determination module is used to obtain the standard deviation corresponding to the current image block through a probabilistic hyperparameter decoding network based on the first code stream corresponding to the current image block; determine a probability distribution model based on the standard deviation, and decode the second code stream corresponding to the current image block based on the probability distribution model to obtain decoded image features; determine the residual features corresponding to the current image block based on the decoded image features; determine the initial image features corresponding to the current image block based on the residual features; an acquisition module is used to obtain the entropy gradient offset step corresponding to the current image block; a processing module is used to perform feature enhancement on the initial image features based on the entropy gradient offset step to obtain target image features; the acquisition module is also used to obtain a reconstructed image block corresponding to the current image block through a synthetic transformation network based on the target image features.

Exemplarily, the determination module is specifically used to obtain the standard deviation corresponding to the current image block through a probabilistic hyperparameter decoding network based on the first code stream corresponding to the current image block, and is specifically used to: decode the first code stream corresponding to the current image block based on a factorized probability model to obtain decoded image features, and determine coefficient hyperparameter features based on the decoded image features; input the coefficient hyperparameter features to the probabilistic hyperparameter decoding network, and inversely transform the coefficient hyperparameter features through the probabilistic hyperparameter decoding network to obtain the standard deviation corresponding to the current image block.

Exemplarily, when the acquisition module acquires the entropy gradient offset step corresponding to the current image block, it is specifically used to: decode the header information code stream corresponding to the current image block to obtain the entropy gradient offset step; or, decode the header information code stream corresponding to the current image block to obtain indication information corresponding to the entropy gradient offset step, and determine the entropy gradient offset step based on the indication information.

Exemplarily, when the acquisition module determines the entropy gradient offset step based on the indication information, it is specifically used to: if the configured entropy gradient offset step table includes at least one entropy gradient offset step, and the indication information is used to indicate the index value of the entropy gradient offset step, then determine the entropy gradient offset step corresponding to the current image block from the entropy gradient offset step table based on the index value.

Exemplarily, the processing module performs feature enhancement on the initial image features based on the entropy gradient offset step to obtain the target image features, and is specifically used to: perform feature enhancement on the initial image features based on the entropy gradient offset step and the entropy gradient feature information corresponding to the current image block to obtain the target image features; wherein the entropy gradient offset step represents the offset step of the entropy gradient, and the entropy gradient offset step is a correction step used to enhance the initial image features.

Exemplarily, the entropy gradient feature information includes an entropy gradient offset value, and the processing module determines the entropy gradient offset value corresponding to the current image block by specifically determining the entropy gradient offset value corresponding to the current image block based on the residual feature.

Exemplarily, when the processing module determines the entropy gradient offset value corresponding to the current image block based on the residual feature, it is specifically used to: determine the residual incremental feature corresponding to the residual feature; calculate the entropy of the residual feature, and calculate the entropy of the residual incremental feature; determine the entropy gradient of the residual feature based on the entropy of the residual feature and the entropy of the residual incremental feature, and determine the entropy gradient offset value based on the entropy gradient of the residual feature and the regularization term.

Exemplarily, the entropy gradient feature information includes a target entropy gradient, and the processing module determines the target entropy gradient corresponding to the current image block by: querying the target quantization scale corresponding to the standard deviation from an entropy rate fitting table, the target quantization scale is the quantization scale closest to the standard deviation, and querying the target quadratic term corresponding to the target quantization scale from the entropy rate fitting table; wherein the entropy rate fitting table includes a correspondence between the quantization scale and the quadratic term; and determining the target entropy gradient corresponding to the current image block based on the target quadratic term.

Among them, the method for obtaining the entropy rate fitting table includes: for multiple standard deviations between the minimum standard deviation and the maximum standard deviation, uniformly quantizing the multiple standard deviations on a logarithmic scale to obtain multiple quantization scales corresponding to the multiple standard deviations; for each quantization scale, performing entropy rate fitting on the quantization scale to obtain a quadratic term corresponding to the quantization scale, and recording the corresponding relationship between the quantization scale and the quadratic term in the entropy rate fitting table.

Exemplarily, when determining the regularization term, the processing module is specifically used to: determine the regularization term based on the initial image feature, the standard deviation and the residual feature.

Exemplarily, the initial image feature corresponding to the current image block is an initial image feature corresponding to a brightness component; or, the initial image feature corresponding to the current image block is an initial image feature corresponding to a chrominance component.

Based on the same application concept as the above method, an encoding device is also proposed in an embodiment of the present application. The device is applied to an encoding end (also referred to as a video encoder), and the device may include: a determination module, which is used to obtain a standard deviation corresponding to the current image block through a probabilistic hyperparameter decoding network based on a first code stream corresponding to the current image block; determine a probability distribution model based on the standard deviation, and decode the second code stream corresponding to the current image block based on the probability distribution model to obtain decoded image features; determine the residual features corresponding to the current image block based on the decoded image features; determine the initial image features corresponding to the current image block based on the residual features; a processing module, which is used to enhance the initial image features based on the entropy gradient offset step for each entropy gradient offset step to obtain a target image feature; based on the target image feature, obtain a reconstructed image block corresponding to the entropy gradient offset step through a synthetic transformation network; a selection module, which is used to select the entropy gradient offset step corresponding to the current image block from all entropy gradient offset steps based on the current image block and the reconstructed image block corresponding to each entropy gradient offset step; and an encoding module, which is used to encode the entropy gradient offset step corresponding to the current image block in the header information code stream corresponding to the current image block.

Exemplarily, the selection module selects the entropy gradient offset step corresponding to the current image block from all entropy gradient offset steps based on the current image block and the reconstructed image block corresponding to each entropy gradient offset step, and is specifically used to: for each entropy gradient offset step, determine the target cost value corresponding to the entropy gradient offset step based on the current image block and the reconstructed image block corresponding to the entropy gradient offset step, and determine the fidelity corresponding to the entropy gradient offset step based on the target cost value and the reference cost value; based on the fidelity corresponding to each entropy gradient offset step, select the entropy gradient offset step corresponding to the maximum fidelity as the entropy gradient offset step corresponding to the current image block; wherein the process of obtaining the reference cost value includes: after obtaining the initial image features corresponding to the current image block, based on the initial image features, obtaining the reconstructed image block through a synthetic transformation network, and determining the reference cost value based on the current image block and the reconstructed image block.

Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. The present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. The embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The above is only an embodiment of the present application and is not intended to limit the present application.

For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.

Claims (19)

1. A decoding method, the method comprising:

Acquiring a standard deviation corresponding to the current image block through a probability super-parameter decoding network based on a first code stream corresponding to the current image block; determining a probability distribution model based on the standard deviation, and decoding a second code stream corresponding to the current image block based on the probability distribution model to obtain decoded image features; determining residual characteristics corresponding to the current image block based on the decoded image characteristics; determining initial image features corresponding to the current image block based on the residual features;

Acquiring an entropy gradient offset step length corresponding to the current image block;

Performing feature enhancement on the initial image features based on the entropy gradient offset step length to obtain target image features;

and acquiring a reconstructed image block corresponding to the current image block through a synthesis transformation network based on the target image characteristics.

2. The method according to claim 1, wherein the obtaining, based on the first code stream corresponding to the current image block, the standard deviation corresponding to the current image block through the probabilistic super-parameter decoding network includes:

Decoding a first code stream corresponding to the current image block based on a factorized probability model to obtain decoded image features, and determining coefficient super-parameter features based on the decoded image features;

And inputting the coefficient super-parameter characteristics into the probability super-parameter decoding network, and performing inverse transformation on the coefficient super-parameter characteristics through the probability super-parameter decoding network to obtain standard deviation corresponding to the current image block.

3. The method of claim 1, wherein the step of determining the position of the substrate comprises,

The step of obtaining the entropy gradient offset step corresponding to the current image block includes:

decoding the head information code stream corresponding to the current image block to obtain the entropy gradient offset step length;

Or decoding the header information code stream corresponding to the current image block to obtain indication information corresponding to the entropy gradient offset step length, and determining the entropy gradient offset step length based on the indication information.

4. The method of claim 3, wherein the step of,

The determining the entropy gradient offset step based on the indication information includes: if the configured entropy gradient offset step list comprises at least one entropy gradient offset step, and the indication information is used for indicating an index value of the entropy gradient offset step, determining the entropy gradient offset step corresponding to the current image block from the entropy gradient offset step list based on the index value.

5. The method of claim 1, wherein the feature enhancing the initial image feature based on the entropy gradient offset step size to obtain a target image feature comprises:

Performing feature enhancement on the initial image feature based on the entropy gradient offset step length and entropy gradient feature information corresponding to the current image block to obtain a target image feature; the entropy gradient offset step represents an offset step of an entropy gradient, and the entropy gradient offset step is a correction step used for enhancing the initial image feature.

6. The method according to claim 5, wherein the entropy gradient feature information includes an entropy gradient offset value, and the determining manner of the entropy gradient offset value corresponding to the current image block includes:

and determining an entropy gradient offset value corresponding to the current image block based on the residual characteristics.

7. The method of claim 6, wherein the step of providing the first layer comprises,

The determining the entropy gradient offset value corresponding to the current image block based on the residual feature includes:

determining residual increment characteristics corresponding to the residual characteristics;

calculating entropy of the residual error feature, and calculating entropy of the residual error increment feature;

Determining an entropy gradient of the residual feature based on the entropy of the residual feature and the entropy of the residual delta feature, and determining the entropy gradient offset value based on the entropy gradient of the residual feature and a regularization term.

8. The method according to claim 5, wherein the entropy gradient feature information includes a target entropy gradient, and the determining manner of the target entropy gradient corresponding to the current image block includes:

Inquiring a target quantization scale corresponding to the standard deviation from an entropy rate fitting table, wherein the target quantization scale is the quantization scale closest to the standard deviation, and inquiring a target quadratic term corresponding to the target quantization scale from the entropy rate fitting table; the entropy rate fitting table comprises a corresponding relation between a quantization scale and a quadratic term;

and determining a target entropy gradient corresponding to the current image block based on the target quadratic term.

9. The method of claim 8, wherein the obtaining the entropy rate fit table comprises:

Uniformly quantizing a plurality of standard deviations between the minimum standard deviation and the maximum standard deviation on a logarithmic scale to obtain a plurality of quantization scales corresponding to the plurality of standard deviations;

And performing entropy rate fitting on each quantization scale to obtain a quadratic term corresponding to the quantization scale, and recording the corresponding relation between the quantization scale and the quadratic term in an entropy rate fitting table.

10. The method of claim 7, wherein the determining of the regularization term comprises:

The regularization term is determined based on the initial image features, the standard deviation, and the residual features.

11. The method according to any one of claims 1 to 10, wherein,

The initial image features corresponding to the current image block are initial image features corresponding to the brightness component; or, the initial image feature corresponding to the current image block is the initial image feature corresponding to the chroma component.

12. A method of encoding, the method comprising:

Acquiring a standard deviation corresponding to the current image block through a probability super-parameter decoding network based on a first code stream corresponding to the current image block; determining a probability distribution model based on the standard deviation, and decoding a second code stream corresponding to the current image block based on the probability distribution model to obtain decoded image features; determining residual characteristics corresponding to the current image block based on the decoded image characteristics; determining initial image features corresponding to the current image block based on the residual features;

Aiming at each entropy gradient offset step length, carrying out feature enhancement on the initial image features based on the entropy gradient offset step length to obtain target image features; based on the target image characteristics, obtaining a reconstructed image block corresponding to the entropy gradient offset step length through a synthetic transformation network;

and selecting the entropy gradient offset step length corresponding to the current image block from all entropy gradient offset step lengths based on the current image block and the reconstructed image block corresponding to each entropy gradient offset step length, and encoding the entropy gradient offset step length corresponding to the current image block in the header information code stream corresponding to the current image block.

13. The method of claim 12, wherein the step of determining the position of the probe is performed,

The selecting the entropy gradient offset step length corresponding to the current image block from all entropy gradient offset step lengths based on the current image block and the reconstructed image block corresponding to each entropy gradient offset step length comprises the following steps:

For each entropy gradient offset step length, determining a target cost value corresponding to the entropy gradient offset step length based on the current image block and a reconstructed image block corresponding to the entropy gradient offset step length, and determining the fidelity degree corresponding to the entropy gradient offset step length based on the target cost value and a reference cost value; based on the fidelity degree corresponding to each entropy gradient offset step, selecting the entropy gradient offset step corresponding to the maximum fidelity degree as the entropy gradient offset step corresponding to the current image block;

The reference cost value obtaining process comprises the following steps: after obtaining initial image features corresponding to a current image block, obtaining a reconstructed image block through a synthesis transformation network based on the initial image features, and determining the reference cost value based on the current image block and the reconstructed image block.

14. A decoding device, the device comprising:

The determining module is used for acquiring the standard deviation corresponding to the current image block through the probability super-parameter decoding network based on the first code stream corresponding to the current image block; determining a probability distribution model based on the standard deviation, and decoding a second code stream corresponding to the current image block based on the probability distribution model to obtain decoded image features; determining residual characteristics corresponding to the current image block based on the decoded image characteristics; determining initial image features corresponding to the current image block based on the residual features;

The acquisition module is used for acquiring the entropy gradient offset step length corresponding to the current image block;

the processing module is used for carrying out feature enhancement on the initial image features based on the entropy gradient offset step length to obtain target image features;

The acquisition module is further used for acquiring a reconstructed image block corresponding to the current image block through a synthesis transformation network based on the target image characteristics.

15. An encoding apparatus, the apparatus comprising:

The determining module is used for acquiring the standard deviation corresponding to the current image block through the probability super-parameter decoding network based on the first code stream corresponding to the current image block; determining a probability distribution model based on the standard deviation, and decoding a second code stream corresponding to the current image block based on the probability distribution model to obtain decoded image features; determining residual characteristics corresponding to the current image block based on the decoded image characteristics; determining initial image features corresponding to the current image block based on the residual features;

The processing module is used for carrying out feature enhancement on the initial image features based on the entropy gradient offset step length aiming at each entropy gradient offset step length to obtain target image features; based on the target image characteristics, obtaining a reconstructed image block corresponding to the entropy gradient offset step length through a synthetic transformation network;

the selecting module is used for selecting the entropy gradient offset step length corresponding to the current image block from all entropy gradient offset step lengths based on the current image block and the reconstructed image block corresponding to each entropy gradient offset step length;

And the encoding module is used for encoding the entropy gradient offset step length corresponding to the current image block in the header information code stream corresponding to the current image block.

16. A decoding end apparatus, comprising: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor;

the processor is configured to execute machine executable instructions to implement the method of any of claims 1-11.

17. An encoding end device, comprising: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor;

the processor is configured to execute machine executable instructions to implement the method of any of claims 12-13.

18. An electronic device, comprising: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; the processor is configured to execute machine executable instructions to implement the method of any one of claims 1-11 or to implement the method of any one of claims 12-13.

19. A machine-readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the method of any of claims 1 to 11 or which, when executed by a processor, implement the method of any of claims 12 to 13.