CN110300301A - Image coding/decoding method and device - Google Patents

Abstract

This application provides a kind of image coding/decoding method and devices.The image encoding method includes: to determine target up-sampling filter from pre-set up-sampling filter set according to the Coding cost of image block to be encoded, wherein, the up-sampling filter set includes at least finite impulse response FIR up-sampling filter and convolutional neural networks CNN up-sampling filter；It generates up-sampling filter corresponding with the target up-sampling filter and indicates information；Down-sampling is carried out to the image block to be encoded using pre-set FIR downsampling filter, obtains the first image block；The first image block is encoded, code stream is obtained；The code stream is written into up-sampling filter instruction information.The application can be improved encoding efficiency.

Description

Image encoding and decoding method and device

technical field

The present application relates to the technical field of image encoding and decoding, and more specifically, relates to an image encoding and decoding method and device.

Background technique

In the process of image storage and transmission, it is often necessary to compress and encode the image to reduce the storage capacity and transmission bandwidth occupied by the image code stream. In addition, considering the cost of hardware and software implementation in the image encoding process, when encoding an image, the image to be encoded is usually divided into multiple image blocks first, and then the encoding operation is performed on each image block. Typical image encoding The process is shown in Figure 1.

The encoding process shown in Figure 1 mainly includes the following steps:

101. Acquire an input image, and divide the input image into image blocks;

102. Predict the image block to obtain a prediction signal;

103. Obtain an original residual signal according to the image block and the prediction signal;

104. Change and quantize the original residual signal to obtain a quantization coefficient;

105. Perform inverse quantization and inverse transformation operations on the quantized coefficients to obtain a reconstructed residual signal;

106. Obtain a reconstruction signal of the image block according to the reconstruction residual signal and the prediction signal;

107. Perform entropy coding on the quantization coefficients and other indication information in the coding process to obtain a compressed code stream.

Corresponding to the encoding process shown in FIG. 1 , FIG. 2 shows the decoding process of an image. The decoding process shown in Figure 2 mainly includes the following steps:

201. Perform entropy decoding on the compressed code stream to obtain quantized coefficients;

202. Perform inverse quantization and inverse transformation on the quantized coefficients to obtain a reconstruction residual signal of the current image block;

203. Acquire a prediction signal of an image block to be decoded;

204. Obtain a reconstruction signal of the current image block according to the reconstruction residual signal and the prediction signal;

205. Output the decoded image.

In some application scenarios, for example, in video conferencing, online live broadcast and other scenarios, in order to improve the encoding quality of the image and reduce the bandwidth occupied by the compressed code stream, the image block can be encoded in the variable resolution mode.

Specifically, the encoder first performs downsampling operations on the image blocks according to the downsampling filter to obtain low-resolution image blocks, then predicts, transforms, and quantizes the low-resolution image blocks to obtain quantization coefficients, and then quantizes The coefficients are entropy coded to obtain a compressed code stream. In addition, the encoder also needs to perform inverse quantization and inverse transformation on the quantized coefficients to obtain the reconstructed residual signal, and obtain the initial reconstructed image block according to the reconstructed residual signal and the prediction signal of the current image block to be decoded, and finally the initial reconstructed image Blocks are up-sampled to obtain a final reconstructed image, which can be used as a reference image when encoding subsequent images.

Correspondingly, the decoder obtains a low-resolution reconstructed image after decoding the compressed code stream, and then upsamples the low-resolution reconstructed image according to the upsampling filter to obtain the same resolution as the original image block. reconstructed image.

However, the up-sampling filter and down-sampling filter used in the traditional solution for variable-resolution coding of image blocks both use a preset fixed finite impulse response (Finite Impulse Response, FIR) filter, and the coding effect is not good. ideal.

Contents of the invention

The present application provides an image encoding and decoding method to improve the encoding and decoding effect.

In a first aspect, an image coding method is provided, the method comprising: determining a target upsampling filter from a preset set of upsampling filters; generating upsampling filter indication information corresponding to the target upsampling filter; Downsampling the image block to be coded to obtain a first image block; encoding the first image block to obtain a code stream; writing the upsampling filter instruction information into the code stream.

Wherein, the above-mentioned upsampling filter set includes at least FIR upsampling filter and convolutional neural network (Convolutional Neural Network, CNN) upsampling filter, FIR downsampling filter can be bicubic upsampling filter, CNN upsampling filter The filter can be an upsampling filter composed of a convolutional neural network.

The above upsampling filter indication information may be used to indicate that a certain upsampling filter in the upsampling filter set is a target upsampling filter. For example, the upsampling filter indication information may indicate the FIR upsampling filter in the upsampling filter set as the target upsampling filter, or the upsampling filter indication information may also indicate the CNN upsampling filter in the upsampling filter set filter as the target upsampling filter.

Optionally, the upsampling filter indication information is specifically an upsampling filter selection flag Flag2, and a value of Flag2 is used to indicate a target upsampling filter.

For example, when the FIR filter is determined to be the target upsampling filter, the value of Flag2 is 0; and when the CNN filter is determined to be the target upsampling filter, the value of Flag2 is 1. Alternatively, when the FIR filter is determined to be the target upsampling filter, the value of Flag2 is 1; and when the CNN filter is determined to be the target upsampling filter, the value of Flag2 is 0.

It should be understood that determining the target up-sampling filter from the preset up-sampling filter set and determining the target down-sampling filter from the down-sampling filter set are two independent processes. Determining the target up-sampling filter and determining the target down-sampling filter have no priority in terms of time. The target up-sampling filter can be determined first, or the target down-sampling filter can be determined first, or at the same time A target up-sampling filter and a target down-sampling filter are determined.

Optionally, encoding the first image block to obtain a code stream includes: performing prediction, transformation, quantization, and entropy encoding on the first image block to obtain a code stream.

Specifically, the prediction number can be obtained by predicting the first image block. After obtaining the prediction signal, the original signal can be subtracted from the original signal of the first image block to obtain the original residual signal, and then the original residual signal can be obtained. Transform and quantize the difference signal to obtain quantized coefficients, and finally perform entropy coding on the quantized coefficients to obtain a code stream.

In this application, a target upsampling filter can be selected from a variety of pre-set upsampling filters for subsequent upsampling operations, compared with directly using an upsampling filter with a fixed parameter value for subsequent upsampling operations , the selection range of the up-sampling filter in this application is wider, and an appropriate up-sampling filter can be selected as the target up-sampling filter according to the situation.

In a possible implementation manner, downsampling the image block to be encoded to obtain the first image block specifically includes: downsampling the image block to be encoded by using a preset FIR downsampling filter to obtain the first image block.

By using the preset FIR down-sampling filter to directly perform the down-sampling operation, it is no longer necessary to select a target down-sampling filter from various down-sampling filters, which can reduce the complexity of encoding operations and improve encoding efficiency.

In a possible implementation, the above method further includes: performing inverse transformation and inverse quantization on the quantized coefficients to obtain an initial reconstructed image block; using a target upsampling filter to upsample the initial reconstructed image block to obtain a target reconstructed image block ; Perform entropy coding on the quantized coefficients to obtain a code stream; write the upsampling filter indication information into the code stream.

Since the downsampling operation is performed on the image block, the resolution of the first image block obtained by downsampling is smaller than the original resolution of the image block to be encoded. When the image block is reconstructed, the resolution of the initial image block is also smaller than the original resolution of the image block to be encoded, and the resolution of the target reconstructed image block obtained after upsampling the initial image block is the same as the original The resolution is the same.

In a possible implementation manner, determining the target upsampling filter from the upsampling filter set specifically includes: determining the target upsampling filter from the preset upsampling filter set according to the encoding cost of the image block to be encoded filter.

Optionally, determining the target upsampling filter from the preset upsampling filter set according to the encoding cost of the image block to be encoded includes: upsampling the corresponding encoding cost in the upsampling filter set to meet the preset requirements The filter is determined to be the target upsampling filter.

In a possible implementation manner, determining an upsampling filter whose corresponding coding cost in the upsampling filter set satisfies a preset requirement as a target upsampling filter includes: determining each type of upsampling filter in the upsampling filter set The encoding cost of the image block to be encoded when the sampling filter is used as the target up-sampling filter; the first up-sampling filter in the up-sampling filter set is determined as the target up-sampling filter, wherein, in the up-sampling filter set, When the first upsampling filter is used as the target upsampling filter, the encoding cost of the image block to be encoded is the smallest.

By comparing the encoding costs of the corresponding image blocks to be encoded when each upsampling filter in the upsampling filter set performs an upsampling operation, the upsampling filter corresponding to the smallest encoding cost can be selected as the target upsampling filter. It reduces the encoding cost generated in the encoding process and achieves better encoding effect.

It should be understood that the above encoding cost may specifically be a rate-distortion cost of an image block in an encoding process, a distortion of an image block, and the like.

In a possible implementation manner, determining the target upsampling filter from a preset set of upsampling filters specifically includes: determining the target upsampling filter from the upsampling filter according to the texture feature of the image block to be encoded .

For example, when the texture of the image block to be encoded is relatively sparse, the FIR upsampling filter can be selected as the target upsampling filter; when the texture of the image block to be encoded is relatively dense, in order to ensure the encoding effect, the CNN upsampling filter can be selected as the target upsampling filter. Target upsampling filter.

According to the texture characteristics of the image block, an upsampling filter that matches the image block can be flexibly selected as the target upsampling filter, and the coding effect of image blocks with different texture characteristics can be improved.

In a possible implementation manner, determining the target upsampling filter from a preset set of upsampling filters specifically includes: determining the sampling filter from the upsampling filter according to the spectral characteristics of the image block to be encoded.

For example, when there are many high-frequency components of the image block to be encoded, the CNN upsampling filter can be selected as the target upsampling filter, and when the high-frequency components of the image block to be encoded are small, the FIR filter can be selected as the upsampling filter filter.

According to the spectral characteristics of the image block, an upsampling filter matching the image block can be selected as a target upsampling filter, which can improve the coding effect of image blocks with different spectral characteristics.

In a possible implementation manner, the parameter value of the CNN upsampling filter is preset, and the parameter value of the CNN upsampling filter is obtained by performing offline training on a preset image training set.

Since the parameter values of the CNN upsampling filter are obtained through offline training, if the CNN upsampling filter is used for upsampling during the encoding process, the information loss of the image during the upsampling process can be reduced and the image encoding quality can be improved.

In a possible implementation, before determining the target upsampling filter from the pre-set upsampling filter set, the above method further includes: performing online training on the CNN upsampling filter according to the image block to be coded to obtain the CNN upsampling filter The update parameter value for the sampling filter.

Wherein, the above updated parameter value of the CNN upsampling filter is used to replace the preset parameter value of the CNN upsampling filter.

Optionally, the above method further includes: writing the updated parameter value of the CNN upsampling filter into the code stream.

By training the CNN upsampling filter online, an upsampling filter with more matching filter parameters can be used for the upsampling operation according to the image texture characteristics. Compared with the preset CNN upsampling filter, the upsampling output can be further improved. The image quality of the image block.

In a possible implementation manner, before determining the target upsampling filter from the preset upsampling filter set according to the encoding cost of the image block to be encoded, the above method further includes: according to the encoding cost of the image block to be encoded Determine the target encoding mode from the original resolution encoding mode and the variable resolution encoding mode; generate encoding mode indication information corresponding to the target encoding mode; write the encoding mode indication information into the code stream.

Optionally, the target upsampling filter is determined from the preset upsampling filter set according to the encoding cost of the image block to be encoded, which specifically includes: determining the encoding cost of the image block to be encoded in the original resolution encoding mode; determining the variable The encoding cost of the image block to be encoded in the resolution encoding mode; the encoding mode with the smallest encoding cost is selected from the original resolution encoding mode and the variable resolution encoding mode as the target encoding mode.

The above coding mode indication information is used to indicate which coding mode among the candidate coding modes is the target coding mode. For example, the above encoding mode indication information may indicate that the original resolution encoding mode is the target encoding mode, or the above encoding mode indication information may also indicate the variable resolution encoding mode as the target encoding mode.

According to the encoding cost of the image block to be encoded, an encoding mode with a lower encoding cost can be selected as the target encoding mode, which can reduce the loss of image encoding in the encoding process.

It should be understood that the target upsampling filter is determined from the preset upsampling filter set according to the coding cost of the image block when it is determined to use the variable resolution coding mode to encode the image block to be coded. When it is determined to use the variable resolution coding mode to encode the image block to be coded, the original resolution coding mode is directly used to encode the image block to be coded, and there is no need to determine the target upsampling filter, because in the original resolution coding mode, there is no Upsampling is required.

In a second aspect, an image coding method is provided, which includes: determining a target upsampling filter from a preset set of upsampling filters according to the coding cost of an image block to be coded, wherein the set of upsampling filters is at least Including FIR upsampling filter and CNN upsampling filter; generating upsampling filter indication information corresponding to the target upsampling filter; downsampling the same type of downsampling filter as the target upsampling filter in the pre-set downsampling filter set The filter is determined as a target downsampling filter, wherein the downsampling filter set includes at least an FIR downsampling filter and a CNN downsampling filter; the target downsampling filter is used to downsample the image block to be coded to obtain the first image block ; Encoding the first image block to obtain a code stream; writing the upsampling filter indication information into the code stream.

For example, the upsampling filter indication information may indicate the FIR upsampling filter in the upsampling filter set as the target upsampling filter, or the upsampling filter indication information may also indicate the CNN upper sampling filter in the upsampling filter set The sampling filter acts as the target upsampling filter.

In this application, the target upsampling filter can be determined from the set of upsampling filters according to the encoding cost. Compared with directly using the upsampling filter with fixed parameter values for the upsampling operation, this application scheme selects the target upsampling filter The encoding cost of the image block to be encoded is fully considered when the encoder is used, and better encoding effect can be achieved.

In a possible implementation, determining the target upsampling filter from a preset upsampling filter set according to the encoding cost of the image block to be encoded includes: determining each type of upsampling filter in the upsampling filter set The encoding cost of the image block to be encoded when the filter is used as the target upsampling filter; the first upsampling filter in the upsampling filter set is determined as the target upsampling filter, wherein, in the upsampling filter set, the upsampling When the first upsampling filter in the filter set is used as the target upsampling filter, the encoding cost of the image block to be encoded is the smallest.

In this application, by selecting the upsampling filter with the smallest coding cost as the target upsampling filter, the coding cost generated in the coding process can be reduced as much as possible, and better coding effect can be achieved.

In a possible implementation manner, the parameter value of the CNN upsampling filter is preset, and the parameter value of the CNN upsampling filter is obtained by performing offline training on a preset image training set.

Since the parameter values of the CNN upsampling filter are obtained through offline training, if the CNN upsampling filter is used for upsampling during the encoding process, the information loss of the image during the upsampling process can be reduced and the image encoding quality can be improved.

In a possible implementation manner, the parameter value of the CNN downsampling filter is preset, and the parameter value of the CNN downsampling filter is obtained by performing offline training on a preset image training set.

Since the parameter values of the CNN downsampling filter are obtained through offline training, if the CNN downsampling filter is used for downsampling during the encoding process, the information loss of the image during the downsampling process can be reduced and the image encoding quality can be improved.

In a possible implementation, the parameter values of the CNN upsampling filter and the CNN downsampling filter are preset, and the parameter values of the CNN upsampling filter and the CNN downsampling filter are It is obtained by joint training on the preset image training set offline.

Since the CNN upsampling and downsampling filters are obtained through joint training offline, the information loss caused by the upsampling and downsampling of the image texture can be reduced during the encoding process and the quality of the encoded image can be improved.

In a possible implementation, before using the target downsampling filter to downsample the to-be-coded image block, the above method further includes: performing online training on the CNN downsampling filter according to the to-be-coded image block to obtain the CNN downsampling filter The update parameter value of the CNN downsampling filter, wherein the update parameter value of the CNN downsampling filter is used to replace the preset parameter value of the CNN downsampling filter.

Through online training of CNN downsampling filter parameters, the information loss caused by the downsampling operation can be reduced as much as possible, and the quality of the reconstructed image can be improved. In addition, since the CNN downsampling filter is only used in the encoder, there is no need to pass the CNN downsampling filter parameters to the decoder, which will not increase the encoding overhead.

In a possible implementation, before determining the target upsampling filter from the preset upsampling filter set according to the encoding cost of the image block to be encoded, the above method further includes: The sampling filter is trained online to obtain an updated parameter value of the CNN upsampling filter, wherein the updated parameter value of the CNN upsampling filter is used to replace the preset parameter value of the CNN upsampling filter.

By training the CNN upsampling filter online, an upsampling filter with more matching filter parameters can be used for the upsampling operation according to the image texture characteristics. Compared with the preset CNN upsampling filter, the upsampling output can be further improved. The image quality of the image block.

In a possible implementation, before using the target downsampling filter to downsample the image block to be encoded, the method further includes: performing joint online training on the CNN downsampling filter and the CNN upsampling filter according to the image block to be encoded , get the update parameter value of the CNN downsampling filter and the update parameter value of the CNN upsampling filter; wherein, the update parameter value of the CNN downsampling filter is used to replace the preset parameter value of the CNN downsampling filter, and the CNN upsampling filter The updated parameter values of the filter are used to replace the preset parameter values of the CNN upsampling filter.

Optionally, the CNN downsampling filter and the CNN upsampling filter are jointly trained online according to the image block to be encoded, including: using the CNN downsampling filter to downsample the image block to be encoded to obtain a low-resolution image to be encoded block; use the encoder emulator to encode the low-resolution image block to be encoded to obtain a low-resolution reconstructed image block; use the CNN upsampling filter to up-sample the low-resolution reconstructed image block to obtain the target reconstruction Image block; The upsampling filter parameter and the downsampling filter parameter when the difference between the image block to be encoded and the target reconstruction image block are the smallest are determined as the update parameter value of the CNN upsampling filter and the update of the CNN downsampling filter parameter value.

Through online training of CNN upsampling filter and CNN downsampling filter, it is possible to use upsampling filter and downsampling filter with more matching filter parameters to perform upsampling and downsampling operations according to image texture characteristics, which is different from using preset Compared with the CNN downsampling filter, the CNN upsampling filter can further improve the image quality of the upsampling output image block.

In a possible implementation manner, the above method further includes: writing the updated parameter value of the CNN upsampling filter into the code stream.

In a possible implementation manner, before determining an upsampling filter whose corresponding encoding cost meets a preset requirement in the upsampling filter set as a target upsampling filter, the above method further includes:

According to the image block to be encoded, the CNN upsampling filter and the CNN downsampling filter are jointly trained online to obtain the updated parameter value of the CNN upsampling filter and the updated parameter value of the CNN downsampling filter.

In a possible implementation manner, before determining the target upsampling filter from the preset upsampling filter set according to the coding cost of the image block to be coded, the above method further includes: The encoding mode whose cost meets the preset requirements is determined as the target encoding mode, wherein the candidate encoding mode includes the original resolution encoding mode and the variable resolution encoding mode; encoding mode indication information is generated, and the encoding mode indication information is used to indicate the target encoding mode; The encoding mode instruction information is written into the code stream.

In a possible implementation manner, before determining the target upsampling filter from the preset upsampling filter set according to the encoding cost of the image block to be encoded, the above method further includes: according to the encoding cost of the image block to be encoded Determine the target encoding mode from the original resolution encoding mode and the variable resolution encoding mode; generate encoding mode indication information corresponding to the target encoding mode; write the encoding mode indication information into the code stream.

Optionally, the target upsampling filter is determined from the preset upsampling filter set according to the encoding cost of the image block to be encoded, which specifically includes: determining the encoding cost of the image block to be encoded in the original resolution encoding mode; determining the variable The encoding cost of the image block to be encoded in the resolution encoding mode; the encoding mode with the smallest encoding cost is selected from the original resolution encoding mode and the variable resolution encoding mode as the target encoding mode.

The above coding mode indication information is used to indicate which coding mode among the candidate coding modes is the target coding mode. For example, the above encoding mode indication information may indicate that the original resolution encoding mode is the target encoding mode, or the above encoding mode indication information may also indicate the variable resolution encoding mode as the target encoding mode.

According to the encoding cost of the image block to be encoded, an encoding mode with a lower encoding cost can be selected as the target encoding mode, which can reduce the loss of image encoding in the encoding process.

It should be understood that the target upsampling filter is determined from the preset upsampling filter set according to the coding cost of the image block when it is determined to use the variable resolution coding mode to encode the image block to be coded. When it is determined to use the variable resolution coding mode to encode the image block to be coded, the original resolution coding mode is directly used to encode the image block to be coded, and there is no need to determine the target upsampling filter, because in the original resolution coding mode, there is no Upsampling is required.

In a third aspect, an image coding method is provided, which includes: determining a target downsampling filter from a preset downsampling filter set according to the coding cost of the image block to be coded, wherein the downsampling filter set includes at least The finite impulse response FIR downsampling filter and the convolutional neural network CNN downsampling filter; the target downsampling filter is used to downsample the image block to be encoded to obtain the first image block; the first image block is encoded to obtain the code stream ;Determine the upsampling filter with the same type as the target downsampling filter in the preset upsampling filter set as the target upsampling filter, wherein the upsampling filter set includes at least an FIR upsampling filter and a CNN upsampling filter filter; generate upsampling filter indication information corresponding to the target upsampling filter; write the upsampling filter indication information into the code stream.

For example, the upsampling filter indication information may indicate the FIR upsampling filter in the upsampling filter set as the target upsampling filter, or the upsampling filter indication information may also indicate the CNN upper sampling filter in the upsampling filter set The sampling filter acts as the target upsampling filter.

In this application, the target downsampling filter can be determined from the downsampling filter set according to the encoding cost, and the upsampling filter can be determined according to the filter type of the downsampling filter, and the upsampling filter directly using fixed parameter values Compared with the upsampling operation, the scheme of the present application fully considers the coding cost of the image block to be coded when selecting the target upsampling filter, and can achieve better coding effect.

In a possible implementation manner, determining the target downsampling filter from a preset set of downsampling filters according to the encoding cost of the image block to be encoded includes: determining each downsampling filter in the set of downsampling filters The encoding cost of the image block to be encoded when the filter is used as the target downsampling filter; the first downsampling filter in the downsampling filter set is determined as the target downsampling filter, wherein, in the downsampling filter set, the downsampling filter When the first downsampling filter in the filter set is used as the target downsampling filter, the encoding cost of the image block to be encoded is the smallest.

In this application, by selecting the down-sampling filter with the smallest coding cost as the target down-sampling filter, the coding cost generated in the coding process can be reduced as much as possible, and a better coding effect can be achieved.

In a fourth aspect, an image decoding method is provided, the method comprising: obtaining a code stream; performing entropy decoding, inverse quantization, and inverse transformation on the code stream to obtain a reconstructed residual signal of an image block to be decoded; obtaining a code stream of the image block to be decoded Prediction signal; add the reconstructed residual signal and the prediction signal to obtain the initial reconstructed image block of the image block to be decoded; analyze the code stream to obtain the coding mode indication information; according to the coding mode indication information, decode the mode from the original resolution and the variable resolution Determine the target decoding mode in the decoding mode; in the case that the target decoding mode is the variable resolution decoding mode, analyze the code stream of the image block to be encoded to obtain the upsampling filter indication information; according to the upsampling filter indication information from the preset The target upsampling filter is determined in the upsampling filter set, wherein the upsampling filter set includes at least a finite impulse response FIR upsampling filter and a convolutional neural network CNN upsampling filter; the target upsampling filter is used for the initial The reconstructed image block is up-sampled to obtain the target reconstructed image block.

Wherein, the above encoding mode may include an original resolution encoding mode and a variable resolution encoding mode, and the encoding mode indication information is used to indicate whether the encoding end adopts the original resolution encoding mode or the variable resolution encoding mode during encoding. That is to say, the above coding mode indication information will indicate in which coding mode the above code stream is obtained. For example, the above coding mode indication information may indicate that the code stream is obtained by coding in the original resolution coding mode or the variable resolution coding mode.

Optionally, determining the target decoding mode from the original resolution decoding mode and the variable resolution decoding mode according to the coding mode indication information includes: when the coding mode is the variable resolution coding mode, determining that the target decoding mode is the variable resolution decoding mode ; When the encoding mode is the original resolution encoding mode, determine that the target decoding mode is the original resolution decoding mode.

The decoding end selects the decoding mode corresponding to the encoding mode of the encoding end as the target decoding mode, which can ensure the decoding effect of the decoding end.

In this application, when decoding an image block in the variable resolution decoding mode, the decoding end can select the corresponding upsampling filter from the preset upsampling filter set as the target upsampling filter according to the upsampling filter indication information Filter, and then perform upsampling operation, compared with the way of directly using fixed parameter upsampling filter for upsampling operation, the matching upsampling filter can be selected according to the situation of the image block for upsampling operation, thereby improving the decoding effect .

Optionally, the above coding mode indication information is specifically a value of a coding mode flag Flag1, and different values of Flag1 represent different coding modes.

For example, when the value of Flag1 is 0, it indicates the original resolution encoding mode, and when the value of Flag1 is 1, it indicates the variable resolution encoding mode.

In a possible implementation manner, the parameter value of the CNN upsampling filter is preset, and the parameter value of the CNN upsampling filter is obtained by performing offline training on a preset image training set.

Since the parameter values of the CNN upsampling filter are obtained through offline training, using the CNN upsampling filter for upsampling during the decoding process can reduce the information loss of the image during the upsampling process and improve the image decoding quality.

In a possible implementation, the above method further includes: parsing the code stream to obtain an updated parameter value of the CNN upsampling filter, where the updated parameter value of the CNN upsampling filter is used to replace the preset value of the CNN upsampling filter parameter value.

By updating the parameter values of the CNN upsampling filter, filter parameters that better match the image content can be obtained, thereby further reducing image information loss and improving image decoding quality when using the CNN upsampling filter for upsampling operations .

In a possible implementation, before parsing the code stream and obtaining the update parameter value of the CNN upsampling filter, the method further includes: parsing the code stream, obtaining filter parameter update indication information, and the filter parameter update indication information is used for Indicates whether to update the parameter value of the target upsampling filter; parse the code stream to obtain the updated parameter value of the CNN upsampling filter, including: indicating that the parameters of the target upsampling filter are updated in the filter parameter update instruction information In this case, parse the code stream to obtain the updated parameter value of the CNN upsampling filter.

The update value of the CNN upsampling filter is obtained from the code stream only when the filter parameter update indication information indicates that the target upsampling filter needs to be updated, which can reduce the decoding burden and improve decoding efficiency.

Optionally, the above filter parameter update instruction information is carried in a sequence parameter set (Sequence ParamaterSet, SPS) or a picture parameter set (Picture Paramater Set, PPS).

Optionally, the update parameter value of the CNN upsampling filter is obtained by online training of the CNN upsampling network according to the image block to be encoded, wherein the image block to be decoded is obtained by encoding the image block to be encoded.

By training the CNN upsampling filter online, an upsampling filter with more matching filter parameters can be used for the upsampling operation according to the image texture characteristics. Compared with the preset CNN upsampling filter, the upsampling output can be further improved. The image quality of the image block.

In a fifth aspect, an image encoding device is provided, and the image encoding device includes a module for performing the method described in the first aspect, the second aspect, or the third aspect.

According to a sixth aspect, an image decoding device is provided, and the image decoding device includes a module for performing the method described in the fourth aspect above.

In a seventh aspect, there is provided an image coding device, the above image coding device includes: a memory for storing programs; a processor for executing the programs stored in the memory, and when the above programs are executed, the above processor is used for executing the above first The method described in one aspect, the second aspect or the third aspect.

In an eighth aspect, there is provided an image decoding device, the above-mentioned image decoding device includes: a memory for storing a program; a processor for executing the program stored in the memory, and when the above-mentioned program is executed, the above-mentioned processor is used for executing the above-mentioned first The method described in the four aspects.

In a ninth aspect, an image encoding device is provided, including a non-volatile storage medium, and a central processing unit, the non-volatile storage medium stores an executable program, and the central processing unit is connected to the non-volatile storage medium , and execute the above-mentioned executable program to implement the method described in the above-mentioned first aspect, second aspect or third aspect.

In a tenth aspect, an image decoding device is provided, including a non-volatile storage medium, and a central processing unit, the non-volatile storage medium stores an executable program, and the central processing unit is connected to the non-volatile storage medium , and execute the above-mentioned executable program to implement the method described in the above-mentioned fourth aspect.

In an eleventh aspect, a computer-readable medium is provided. The above-mentioned computer-readable medium stores program code for device execution, and the above-mentioned program code includes a program code for executing the above-mentioned first aspect, second aspect, third aspect or fourth aspect. the method described.

In a twelfth aspect, there is provided a computer program product including instructions, which, when run on a computer, cause the computer to execute the method described in the first aspect, the second aspect, the third aspect or the fourth aspect.

In a thirteenth aspect, there is provided an electronic device, including the image encoding device in the fifth aspect to the tenth aspect above, and/or the image decoding device in the fifth aspect to the tenth aspect above.

Description of drawings

Fig. 1 is a schematic diagram of an image encoding process;

Fig. 2 is a schematic diagram of an image decoding process;

Fig. 3 is the schematic diagram of digital image;

FIG. 4 is a schematic flowchart of an image encoding method according to an embodiment of the present application;

Fig. 5 is the schematic diagram of the online training process of CNN filter;

FIG. 6 is a schematic flowchart of an image encoding method according to an embodiment of the present application;

FIG. 7 is a schematic flowchart of an image encoding method according to an embodiment of the present application;

FIG. 8 is a schematic flowchart of an image decoding method according to an embodiment of the present application;

Figure 9 is a schematic diagram of the architecture of the encoder;

FIG. 10 is a flowchart of an image encoding method according to an embodiment of the present application;

Figure 11 is a schematic structural diagram of a CNN downsampling network;

FIG. 12 is a flowchart of an image decoding method according to an embodiment of the present application;

Fig. 13 is a schematic structural diagram of a CNN upsampling network;

Fig. 14 is a schematic diagram of the offline training process of the CNN filter;

Figure 15 is a schematic diagram of the architecture of the encoder;

Fig. 16 is a schematic diagram of the architecture of an encoder;

FIG. 17 is a flowchart of an image decoding method according to an embodiment of the present application;

FIG. 18 is a flowchart of an image decoding method according to an embodiment of the present application;

FIG. 19 is a schematic block diagram of an image encoding device according to an embodiment of the present application;

FIG. 20 is a schematic block diagram of an image encoding device according to an embodiment of the present application;

FIG. 21 is a schematic block diagram of an image decoding device according to an embodiment of the present application;

Fig. 22 is a schematic block diagram of an encoder according to an embodiment of the present application;

Fig. 23 is a schematic block diagram of a decoder according to an embodiment of the present application;

Fig. 24 is a schematic block diagram of a codec device according to an embodiment of the present application;

Fig. 25 is a schematic block diagram of a video encoding and decoding system according to an embodiment of the present application.

Detailed ways

The technical solution in this application will be described below with reference to the accompanying drawings.

The image coding and decoding method in the embodiment of the present application can code and decode digital images. Specifically, a digital image is image information recorded as a digital signal. Figure 3 shows a digital image, which can be regarded as a two-dimensional array of M×N, the digital image contains MxN samples in total, and the position of each sample is called a sampling position, and each sample The value of is called the sampling value. In general, MxN can be called the resolution of the image, that is, the number of samples contained in the image. For example, the image resolution of 2K video is 1920*1080, and the image resolution of 4K video is 3840*2160. Customarily, a sample can also be referred to as a pixel, and a pixel usually contains two pieces of information: the pixel position and the pixel value.

An image (specifically, it may be a digital image) usually includes one or more color components. For example, a black and white image contains only brightness components, and a color image contains RGB three color components. A color image is composed of samples of three color components, which can be intuitively understood as three color planes (planes), and each color plane is obtained by sampling the corresponding color components, as shown in Figure 3 The sample array shown. The three color components of a color image can be expressed in different ways, commonly used ways include RGB, YCbCr, ICtCp, etc.

The three color planes of a color image can have different resolutions. Taking the color image in YCbCr format as an example, since the luminance component Y contains detailed texture information, and the two chrominance components Cb and Cr contain less high-frequency signals, therefore, the two chrominance component planes of Cb and Cr are usually in the Perform a 2:1 downsampling operation in the horizontal and vertical directions to reduce the resolution of the two chroma components to 1/4 of the brightness resolution. At this time, an MxN color image is composed of an MxN brightness component plane and two (M/2)x(N/2) chrominance component planes.

It should be understood that the image encoding and decoding method in the embodiment of the present application can encode and decode video images.

Fig. 4 is a schematic flowchart of an image encoding method according to an embodiment of the present application. The method shown in FIG. 4 can be executed by an encoding end device, and the encoding end device can specifically be a smart terminal, a personal digital assistant (Personal Digital Assistant, PAD), a video player, an Internet of Things device, or other devices with a video image encoding function. The method shown in FIG. 4 includes steps 301 to 305, and the steps 301 to 305 will be described in detail below with reference to specific examples.

301. Determine a target upsampling filter from a preset set of upsampling filters according to an encoding cost of an image block to be encoded.

Wherein, the upsampling filter set includes at least an FIR upsampling filter and a CNN upsampling filter. The FIR upsampling filter may specifically be a bicubic upsampling filter, and the CNN upsampling filter is a filter composed of a convolutional neural network.

In this application, in addition to determining the target upsampling filter from the upsampling filter set according to the encoding cost, it can also be determined according to the texture characteristics or spectral characteristics of the image block to be coded. Target upsampling filter.

Specifically, the upsampling filter may be selected according to the dense texture of the image block to be encoded (the denser the texture, the richer the image features contained). When the texture of the image block is relatively dense, the CNN filter is selected as the upsampling filter; when the texture of the image block to be encoded is relatively sparse, the FIR filter is selected as the upsampling filter.

In addition, frequency spectrum analysis may also be performed on the image block to be coded to obtain the frequency spectrum features to be coded. When there are many high-frequency components of the image block to be encoded, the CNN filter can be selected as the upsampling filter; and when the high-frequency component of the image block to be encoded is small, the FIR filter can be selected as the upsampling filter.

302. Generate upsampling filter indication information corresponding to the target upsampling filter.

The above upsampling filter indication information may be used to indicate that a certain upsampling filter in the upsampling filter set is a target upsampling filter.

For example, the upsampling filter indication information may indicate the FIR upsampling filter in the upsampling filter set as the target upsampling filter, or the upsampling filter indication information may also indicate the CNN upsampling filter in the upsampling filter set filter as the target upsampling filter.

Optionally, the upsampling filter indication information is specifically an upsampling filter selection flag Flag2, and a value of Flag2 is used to indicate a target upsampling filter.

For example, when the FIR filter is determined to be the target upsampling filter, the value of Flag2 is 0; and when the CNN filter is determined to be the target upsampling filter, the value of Flag2 is 1. Alternatively, when the FIR filter is determined to be the target upsampling filter, the value of Flag2 is 1; and when the CNN filter is determined to be the target upsampling filter, the value of Flag2 is 0.

In short, the present application does not limit the specific value of Flag2 to indicate which filter, as long as different values of Flag2 can indicate different upsampling filters.

After obtaining the upsampling filter indication information, the decoder can directly determine the target upsampling filter according to the corresponding relationship between the value of Flag2 and the upsampling filter.

303. Use a preset FIR downsampling filter to downsample the image block to be coded to obtain a first image block.

It should be understood that step 301 and step 303 are two mutually independent processes, and step 301 may occur before or after step 303, or step 301 and step 303 may also occur simultaneously.

304. Encode the first image block to obtain a code stream.

Encoding the first image block may include: performing prediction, transformation, quantization, and entropy encoding on the first image block to obtain a code stream.

Specifically, the prediction number can be obtained by predicting the first image block. After obtaining the prediction signal, the original signal can be subtracted from the original signal of the first image block to obtain the original residual signal, and then the original residual signal can be obtained. Transform and quantize the difference signal to obtain quantized coefficients, and finally perform entropy coding on the quantized coefficients to obtain a code stream.

When predicting the first image block, the first image block may be predicted by using the encoded reconstruction signal in the same image as the first image block, and this type of prediction operation is generally called intra-frame prediction. Typical intra prediction includes DC component prediction, planar prediction, horizontal prediction, vertical prediction, etc.

The entropy encoding module performs entropy encoding on the quantized coefficients to obtain a code stream (also called a compressed code stream). In addition, in the process of generating the code stream, the entropy encoding module also performs entropy encoding on the prediction modes generated during the encoding process. Entropy coding operation is performed on the instruction information to obtain the code stream.

In addition to encoding the image block to be encoded to generate a code stream, the encoding end usually needs to obtain the reconstructed signal of the image block to be encoded as a reference for the intra prediction operation on the subsequent encoded image block. Therefore, the encoding end also needs to perform inverse quantization and inverse transformation operations on the quantized coefficients obtained during the encoding process of the image block to be encoded to obtain the reconstruction residual signal, and then add the reconstruction residual signal and the prediction signal to obtain the reconstruction signal of the image block.

However, the original residual signal will have information loss after transformation and quantization, and the information loss is irreversible. Therefore, the reconstructed residual signal after dequantization and inverse transformation is inconsistent with the original residual signal, which further causes the reconstructed signal of the image block to be different from the original The signal is inconsistent, so this compression coding method is called lossy compression.

In the image coding method based on block division, signal discontinuity is usually generated at the image block boundary position of the acquired reconstructed image, which is manifested as block effect. In order to eliminate the block effect, smoothing and filtering operations are usually performed on the border pixels of the image block, which is called deblocking filtering to improve the quality of the reconstructed image. In addition to filtering the reconstructed image to remove block effects, various filtering operations such as Wiener filtering and bilateral filtering can also be used to improve the quality of the reconstructed image.

305. Write the upsampling filter indication information into the code stream.

By writing the upsampling filter into the code stream, the decoder can obtain the information of the upsampling filter through the code stream and select the corresponding filter as the upsampling filter from the preset filters, so that the decoder can use the same The upsampling filter performs upsampling.

In this application, according to the coding cost of the image block, one filter can be selected from various candidate filters as the up filter, which can achieve better coding effect compared with the way of using a fixed filter as the up sampling filter .

Optionally, the FIR upsampling filter and the CNN upsampling filter in the upsampling filter set are preset in the encoder and the decoder, and the filter parameters of the FIR upsampling filter and the CNN upsampling filter can be It is pre-set or calculated.

Optionally, as an embodiment, determining the target upsampling filter from the preset upsampling filter set according to the encoding cost of the image block to be encoded includes: determining each type of upsampling filter in the upsampling filter set The encoding cost of the image block to be encoded when the filter is used as the target upsampling filter; the first upsampling filter in the upsampling filter set is determined as the target upsampling filter, wherein, in the upsampling filter set, the first When the upsampling filter is used as the target upsampling filter, the encoding cost of the image block to be encoded is the smallest.

By comparing the encoding costs of the corresponding image blocks to be encoded when each upsampling filter in the upsampling filter set performs an upsampling operation, the upsampling filter corresponding to the smallest encoding cost can be selected from the upsampling filter set as the target upsampling The filter can reduce the encoding cost generated in the encoding process as much as possible and achieve better encoding effect.

It should be understood that the above encoding cost may specifically be a rate-distortion cost of an image block in an encoding process, a distortion of an image block, and the like.

When determining the target upsampling filter from the preset upsampling filter set according to the encoding cost of the image block to be encoded, it is first necessary to determine when each filter in the upsampling filter set is used as the target upsampling filter The encoding cost of the image block, and then select the corresponding minimum encoding cost filter as the up-sampling filter.

For example, the encoding cost of an image block when the FIR upsampling filter is used as the target upsampling filter is less than the encoding cost of the image block when the CNN upsampling filter is used as the target upsampling filter, then the FIR upsampling filter is determined to be an upsampling filter.

The above encoding cost may specifically be a rate-distortion cost of an image block, or may be a distortion of an image block, and the like.

Since the upsampling filter set includes the CNN upsampling filter and the FIR upsampling filter, when determining the target upsampling filter, the CNN upsampling filter and the FIR upsampling filter can be determined as the target upsampling filter respectively. When the filter performs an upsampling operation, the encoding cost of the image block to be encoded is selected, and then the filter corresponding to the smaller encoding cost is selected as the target upsampling filter.

Wherein, determining the encoding cost of the image block to be encoded when the FIR upsampling filter is used as the target upsampling filter specifically includes the following process:

(1) adopt FIR down-sampling filter to carry out down-sampling operation to the image block to be coded, obtain the image block of low resolution;

(2) Encoding the low-resolution image block and outputting the compressed code stream;

(3) Obtain the reconstructed image block of the image block to be encoded;

(4) Determine the mean square error D _FIR of the reconstructed image block and the image block to be encoded;

(5) Determine the size R _FIR of the compressed code stream of the image block to be encoded;

(6) Determine the encoding cost of the image block to be encoded when the FIR upsampling filter is used as the target upsampling filter according to D _FIR and R _FIR , and record the encoding cost as cost _FIR .

Determining the encoding cost of the image block to be encoded when the CNN upsampling filter is used as the target upsampling filter specifically includes the following process:

(7) adopt CNN downsampling filter to carry out downsampling operation to the image block to be coded, obtain the image block of low resolution;

(8) Encoding the low-resolution image block and outputting the compressed code stream;

(9) Obtain the reconstructed image block of the image block to be encoded;

(10) Determine the mean square error _DCNN between the reconstructed image block and the image block to be encoded;

(11) Determine the size R _CNN of the compressed code stream of the image block to be encoded;

(12) Determine the encoding cost of the image block to be encoded when the CNN upsampling filter is used as the target upsampling filter according to _DCNN and R _CNN , and record the encoding cost as cost _CNN .

After the above process (1) to (6), it is determined that the encoding cost of the image block to be encoded is cost _FIR when the FIR upsampling filter is used as the target upsampling filter, and the CNN upsampling is determined through the above process (7) to (12). When the filter is used as the target upsampling filter, the encoding cost of the image block to be encoded is cost _CNN . Next, the upsampling filter can be selected from the FIR upsampling filter and the CNN upsampling filter according to the size relationship between the cost _FIR and the cost _CNN .

When cost _FIR <cost _CNN , determine the FIR upsampling filter as the target upsampling filter;

When cost _FIR > cost _CNN , determine the CNN upsampling filter as the target upsampling filter;

When cost _CNN = cost _FIR , select any one filter from the FIR upsampling filter and the CNN upsampling filter as the target upsampling filter.

Optionally, in addition to determining the target upsampling filter from the preset upsampling filter set according to the coding cost, it can also be determined from the preset upsampling filter set according to the texture feature or spectral characteristic of the image block to be encoded Determine the target upsampling filter in .

For example, when the texture of the image block to be encoded is relatively sparse, the FIR upsampling filter can be selected as the target upsampling filter; when the texture of the image block to be encoded is relatively dense, in order to ensure the encoding effect, the CNN upsampling filter can be selected as the target upsampling filter. Target upsampling filter.

According to the texture characteristics of the image block, an upsampling filter that matches the image block can be flexibly selected as the target upsampling filter, and the coding effect of image blocks with different texture characteristics can be improved.

For example, spectral analysis is performed before the image block to be encoded is formally encoded to obtain the spectral characteristics of the image block to be encoded. When the image block to be encoded has many high-frequency components, the CNN upsampling filter can be selected as the target upsampling filter; And when the high-frequency components of the image block to be coded are few, the FIR filter can be selected as the up-sampling filter.

According to the spectral characteristics of the image block, an upsampling filter matching the image block can be selected as a target upsampling filter, which can improve the coding effect of image blocks with different spectral characteristics.

Optionally, in the present application, the target up-sampling filter may also be determined from the up-sampling filter set according to at least two features among the coding cost, the texture characteristic and the spectrum feature.

Specifically, the target up-sampling filter may be determined from the up-sampling filter set according to the coding cost and the texture feature of the image block to be coded.

For example, when the texture of the image block to be encoded is relatively sparse, the FIR upsampling filter can be selected as the target upsampling filter; when the texture of the image block to be encoded is relatively dense, then the FIR upsampling filter and the CNN upsampling filter can be determined The encoding cost of the image block to be encoded when upsampling is carried out separately. If the encoding cost of the image block to be encoded is small when the CNN upsampling filter is used for upsampling, then the CNN upsampling filter is selected as the target upsampling filter, otherwise , still choose to select the FIR upsampling filter as the target upsampling filter.

Specifically, the target up-sampling filter may be determined from the set of up-sampling filters according to the coding cost and frequency spectrum characteristics of the image block to be coded.

For example, when there are few high-frequency components of the image block to be encoded, the FIR filter can be selected as the upsampling filter; when there are many high-frequency components of the image block to be encoded, then the FIR upsampling filter and the CNN upsampling filter can be determined. The encoding cost of the image block to be encoded when the filter is upsampled respectively, if the encoding cost of the image block to be encoded is small when the CNN upsampling filter is used for upsampling, then the CNN upsampling filter is selected as the target upsampling filter, Otherwise, select the FIR upsampling filter as the target upsampling filter still.

Optionally, as an embodiment, the parameter value of the CNN upsampling filter is preset, and the parameter value (also referred to as the initial parameter value) of the CNN upsampling filter is to perform offline training on the preset image training set owned.

Specifically, when training the CNN upsampling filter offline, the preset FIR downsampling filter can be used to downsample the original image x _i in the image training set, and then the downsampled image y _i can be used as the CNN upsampling The input of the filter, and the original image _xi is used as the training target.

Next, the CNN upsampling filter (also called CNN upsampling network) can be trained according to the general training method of the CNN network to obtain the optimal parameters of the CNN upsampling filter and the optimal parameter as the parameter value of the CNN upsampling filter.

Specifically, the optimal parameters of the CNN upsampling filter can be obtained according to formula (1)

In formula (1), _xi represents the original image in the image training set, y _i represents the low-resolution image obtained after downsampling the original image, n represents the number of training samples, and θ _g represents the CNN upsampling filter All parameters, g(y _i ; θ _g ) indicate that the upsampling network performs an upsampling operation on the input y _i .

In addition, in the above training process, in addition to using the downsampled image _yi as the network input for training. It is also possible to obtain the reconstructed image y′ _i of y _i first, and then use y′ _i as input to train the CNN upsampling filter. The upsampling network trained in this way can not only restore the details of the original resolution image, but also eliminate the image defects introduced by the encoding operation to a certain extent, and further improve the reconstruction quality of the original resolution image block output by the upsampling operation.

Optionally, as an embodiment, before determining the target upsampling filter from the preset upsampling filter set according to the encoding cost of the image block to be encoded, the above method further includes: The sampling filter is trained online to obtain the updated parameter values of the CNN upsampling filter.

Wherein, the updated parameter value of the CNN upsampling filter obtained by online training is used to replace the preset parameter value of the CNN upsampling filter. The preset parameter values of the CNN upsampling filter may be obtained by training the CNN upsampling filter on a preset image training set.

By training the CNN upsampling filter online, an upsampling filter with more matching filter parameters can be used for the upsampling operation according to the image texture characteristics. Compared with the preset CNN upsampling filter, the upsampling output can be further improved. The image quality of the image block.

Further, after obtaining the updated parameter value of the CNN upsampling filter, the updated parameter value of the CNN upsampling filter can also be written into the code stream, so that the decoder can also obtain the CNN upsampling filter by parsing the code stream. The parameters are updated, and the preset parameter values of the CNN upsampling filter are updated according to the update parameters of the CNN upsampling filter.

Specifically, the online training process shown in FIG. 5 can be used to perform online training on the CNN upsampling filter (or called the CNN upsampling network). As shown in Figure 5, I _O is the current image to be encoded, I _L represents the low-resolution image to be encoded after downsampling the input image I _O using the CNN downsampling filter, and the encoder emulator simulates the image encoding process A CNN network of , which can simulate the image encoding process and output low-resolution reconstructed image I′ _L and encoding bit overhead. Among them, the CNN downsampling network can be obtained through offline training, and it is preset in the encoder for downsampling the original image block to be encoded.

When performing online training on the CNN upsampling filter, the CNN upsampling network can be trained according to the formula (2) to obtain the optimal parameters of the CNN upsampling filter

In formula (2), g represents the upsampling operation mapping function, θ _g represents the upsampling filter parameter, and g represents the mapping function of the upsampling filter.

The optimal parameters of the CNN upsampling filter are obtained through the formula (2) Afterwards, the Determines the updated parameter value for the upsampling filter of this CNN.

Through online training of CNN upsampling filter parameters, matching parameters can be used for upsampling operation according to image texture features. Compared with CNN upsampling filter with preset parameters for upsampling operation, the image block output by upsampling can be further improved. quality. In addition, the encoder can comprehensively consider the image quality improvement and parameter transfer cost brought about by updating the upsampling filter parameters, for example, improve the overall rate-distortion performance of the encoding scheme of this embodiment by means of (Rate Distortion Optimization, RDO).

Optionally, as an embodiment, before determining the target upsampling filter from the preset upsampling filter set according to the encoding cost of the image block to be encoded, the above method further includes: according to the encoding cost of the image block to be encoded Determine the target encoding mode from the original resolution encoding mode and the variable resolution encoding mode; generate encoding mode indication information corresponding to the target encoding mode; write the encoding mode indication information into the code stream.

The above coding mode indication information may be used to indicate a target coding mode. Specifically, the coding mode indication information may be represented by the value of the variable resolution coding mode flag Flag1, and different values of Flag1 are used to indicate different coding modes. For example, when Flag1 is 0, it indicates that the target encoding mode is the original resolution encoding mode, and when Flag1 is 1, it indicates that the target encoding mode is the variable resolution encoding mode.

In addition, the above encoding mode information may also be represented by the value of the original resolution encoding mode flag Flag1, and different values of Flag1 are used to indicate different encoding modes. For example, when Flag1 is 0, it indicates that the target encoding mode is a variable resolution encoding mode, and when Flag1 is 1, it indicates that the target encoding mode is an original resolution encoding mode.

Optionally, the target upsampling filter is determined from the preset upsampling filter set according to the encoding cost of the image block to be encoded, which specifically includes: determining the encoding cost of the image block to be encoded in the original resolution encoding mode; determining the variable The encoding cost of the image block to be encoded in the resolution encoding mode; the encoding mode with the smallest encoding cost is selected from the original resolution encoding mode and the variable resolution encoding mode as the target encoding mode.

According to the encoding cost of the image block to be encoded, an encoding mode with a lower encoding cost can be selected as the target encoding mode, which can reduce the loss of image encoding in the encoding process.

Wherein, determining the encoding cost of the image block to be encoded in the original resolution encoding mode specifically includes the following process:

Firstly, the image block to be encoded is encoded in the original resolution mode according to the preset codec scheme, and the compressed code stream of the image block and the reconstructed image block of the image block to be encoded are obtained;

Secondly, it is determined that the mean square error between the reconstructed image block and the image block to be encoded is Dorg;

Again, determine the size of the compressed code stream of the image block to be encoded as Rorg;

Finally, the rate-distortion cost costorg for encoding the image block to be encoded in the original resolution encoding mode is calculated according to Dorg and Rorg, and the rate-distortion cost costorg is determined as the encoding cost of the image block to be encoded in the original resolution encoding mode.

The specific process of determining the encoding cost of the image block to be encoded in the variable resolution encoding mode is the same as the specific process of determining the encoding cost of the image block to be encoded in the original resolution encoding mode.

In the encoding cost of the image block to be encoded in the variable-resolution encoding mode, since the up-sampling filter and the down-sampling filter can have different choices, the up-sampling filter and the down-sampling filter can be both FIR filters The coding cost of the image block at the time of the image block is used as the coding cost of the image block in the variable resolution coding mode, and the coding cost of the image block when both the upsampling filter and the downsampling filter are CNN filters can be used as the coding cost of the image block in the variable resolution coding mode. Encoding cost in rate encoding mode. Alternatively, a filter can be arbitrarily selected from the CNN upsampling filter and the FIR upsampling filter as the upsampling filter in the encoding end, and a filter can be arbitrarily selected from the CNN downsampling filter and the FIR downsampling filter As a downsampling filter, and calculate the encoding cost of the image block to be encoded.

It should be understood that the target upsampling filter is determined from the preset upsampling filter set according to the coding cost of the image block when it is determined to use the variable resolution coding mode to encode the image block to be coded. When it is determined to use the variable resolution coding mode to encode the image block to be coded, the original resolution coding mode is directly used to encode the image block to be coded, and there is no need to determine the target upsampling filter, because in the original resolution coding mode, there is no Upsampling is required.

Fig. 5 is a schematic flowchart of an image coding method according to an embodiment of the present application. The method shown in FIG. 5 may be executed by an encoding end device, which specifically may be a smart terminal, a PAD, a video player, an Internet of Things device, or other devices with a video image encoding function. The method shown in FIG. 5 includes steps 401 to 406, and the steps 401 to 406 will be described in detail below with reference to specific examples.

401. Determine a target upsampling filter from a preset set of upsampling filters according to an encoding cost of an image block to be encoded.

Wherein, the upsampling filter set includes at least an FIR upsampling filter and a CNN upsampling filter. For example, the FIR upsampling filter is specifically a bicubic upsampling filter.

402. Generate upsampling filter indication information corresponding to the target upsampling filter.

The upsampling filter indication information may be an upsampling filter selection flag Flag2, and the value of Flag2 is used to indicate a target upsampling filter.

For example, when the FIR filter is determined to be the target upsampling filter, the value of Flag2 may be set to 0; and when the CNN filter is determined to be the target upsampling filter, the value of Flag2 may be set to 1.

403. Determine a downsampling filter of the same type as the target upsampling filter in the preset downsampling filter set as the target downsampling filter.

Wherein, the downsampling filter set includes at least an FIR downsampling filter and a CNN downsampling filter.

Specifically, when the FIR upsampling filter is the target upsampling filter, the FIR downsampling filter is determined as the target downsampling filter; when the CNN upsampling filter is the target upsampling filter, the CNN downsampling filter The filter is determined as the target downsampling filter.

404. Use a target downsampling filter to downsample the image block to be coded to obtain a first image block.

405. Encode the first image block to obtain a code stream.

406. Write the upsampling filter indication information into the code stream.

In this application, the target upsampling filter can be determined from the set of upsampling filters according to the encoding cost. Compared with directly using the upsampling filter with fixed parameter values for the upsampling operation, this application scheme selects the target upsampling filter The encoding cost of the image block to be encoded is fully considered when the encoder is used, and better encoding effect can be achieved.

It should be understood that steps 401 and 402 in the method shown in FIG. 6 are essentially the same as steps 301 and 302 in the method shown in FIG. 402. In addition, steps 405 and 406 in the method shown in FIG. 6 are essentially the same as steps 304 and 305 in the method shown in FIG. 305.

Optionally, as an embodiment, determining the target upsampling filter from the preset upsampling filter set according to the encoding cost of the image block to be encoded includes: determining each type of upsampling filter in the upsampling filter set The encoding cost of the image block to be encoded when the filter is used as the target upsampling filter; the first upsampling filter in the upsampling filter set is determined as the target upsampling filter, wherein, in the upsampling filter set, the first When the upsampling filter is used as the target upsampling filter, the encoding cost of the image block to be encoded is the smallest.

In this application, by selecting the upsampling filter with the smallest coding cost as the target upsampling filter, the coding cost generated in the coding process can be reduced as much as possible, and better coding effect can be achieved.

The above-mentioned encoding cost may specifically be the rate-distortion cost of the image block in the encoding process, the distortion of the image block, and the like.

Optionally, in addition to determining the target upsampling filter from the preset upsampling filter set according to the coding cost, it can also be determined from the preset upsampling filter set according to the texture feature or spectral characteristic of the image block to be encoded Determine the target upsampling filter in .

Optionally, as an embodiment, the parameter value of the CNN upsampling filter is preset, and the parameter value of the CNN upsampling filter is obtained by performing offline training on a preset image training set.

Since the parameter values of the CNN upsampling filter are obtained through offline training, if the CNN upsampling filter is used for upsampling during the encoding process, the information loss of the image during the upsampling process can be reduced and the image encoding quality can be improved.

It should be understood that the specific process of obtaining the parameter values of the CNN upsampling filter through offline training can refer to the relevant description of offline training of the CNN upsampling filter above.

Optionally, as an embodiment, the parameter value of the CNN downsampling filter is preset, and the parameter value of the CNN downsampling filter is obtained by performing offline training on a preset image training set.

Since the parameter values of the CNN downsampling filter are obtained through offline training, if the CNN downsampling filter is used for downsampling during the encoding process, the information loss of the image during the downsampling process can be reduced and the image encoding quality can be improved.

It should be understood that when performing offline training on the CNN downsampling filter, upsampling filtering needs to be used for upsampling processing. When performing upsampling processing, either the FIR upsampling filter or the CNN upsampling filter can be used. The offline training process of the CNN downsampling filter when the FIR upsampling filter is used is described in detail (the training process when the CNN upsampling filter is used for upsampling processing is similar).

When training the CNN downsampling filter, the original image in the training set can be used as the input of the CNN downsampling filter, and the output of the CNN downsampling filter can be used as the input of the FIR upsampling filter to calculate the FIR upsampling filter The mean square error between the output image and the original image is used as the loss function, and the filter parameter with the smallest loss function is used as the parameter of the CNN downsampling filter.

Specifically, the parameters of the CNN downsampling filter can be obtained according to formula (3).

In formula (3), Indicates all parameters in the CNN downsampling network, f( _xi ; θ _f ) indicates that the downsampling network performs a downsampling operation on the input _xi , g _FIR indicates that the FIR filter is used for upsampling operation, Indicates the initial parameters of the CNN downsampling network.

Optionally, as an embodiment, the parameter value of the CNN upsampling filter and the parameter value of the CNN downsampling filter are all preset, and the parameter value of the CNN upsampling filter and the parameter value of the CNN downsampling filter are It is obtained by joint training on the preset image training set offline.

Since the CNN upsampling and downsampling filters are obtained through joint training offline, the information loss caused by the upsampling and downsampling of the image texture can be reduced during the encoding process and the quality of the encoded image can be improved.

In the case of offline, the joint training of the preset image training set to obtain the parameter values of the CNN upsampling filter and the CNN downsampling filter specifically includes the following process:

First, the CNN upsampling filter is trained to obtain the initial parameters of the CNN upsampling filter.

The pictures in the training set are down-sampled, and the obtained down-sampled images are used as the input of the CNN upsampling network, and the original pictures in the training set are used as the training target. During the training process, the reconstructed image should be as close as possible to the training target. To get the parameters of the CNN upsampling filter. Specifically, the CNN upsampling filter can be trained by formula (4), and then the initial parameter value of the CNN upsampling filter can be obtained.

In the above formula (4), _xi represents the original image of the training set, y _i represents the low-resolution image obtained after downsampling the original image, n represents the number of training samples, θ _g represents the CNN upsampling network All parameters, g(y _i ; θ _g ) means that the upsampling network performs an upsampling operation on the input y _i , Indicates the initial parameters of the CNN upsampling network obtained after the training in step 1.

Secondly, the CNN downsampling filter is trained based on the CNN upsampling filter obtained through training, and the initial parameter value of the CNN downsampling filter is obtained.

The original image in the training set is used as the input of the CNN downsampling filter, and the output of the CNN downsampling filter is used as the input of the CNN upsampling filter to obtain the image output by the CNN upsampling filter, and the output of the CNN upsampling network is calculated. The mean square error between the image and the original image is used as the loss function to minimize the loss function, thereby obtaining the initial parameter value of the CNN downsampling filter.

Specifically, the CNN down-sampling filter may be trained through formula (5), and then the initial parameter value of the CNN down-sampling filter may be obtained.

In formula (5), _xi represents the original image of the training set, θ _f represents all the parameters of the CNN downsampling filter, Indicates the initial parameters of the CNN upsampling network obtained after the training in step 1, f( _xi ; θ _f ) indicates that the downsampling filter performs a downsampling operation on the input x _i , Indicates the initial parameters of the resulting CNN downsampling filter.

Again, jointly train the upsampling filter and the downsampling filter to obtain updated network parameters.

The parameters obtained through the above training process are the initial parameter values of the upsampling filter and the downsampling filter, respectively. Next, use the original resolution image in the image training set as the input and training target, and train the parameters of the upsampling and downsampling filters at the same time.

Specifically, CNN upsampling network and downsampling network can be trained simultaneously according to formula (6) to obtain parameter values of CNN upsampling filter and CNN downsampling filter.

In formula (6), is the optimal parameter of the CNN downsampling filter, θ′ _g is the update parameter value of the CNN upsampling filter during the training process (the initial parameter value of the CNN upsampling filter is ), and the meanings of other relevant parameters in formula (6) can be found in the explanations below formula (4) and formula (5).

The loss function represented by formula (6) consists of two items, the first item The calculation is the error between the image output by the CNN downsampling filter and the image output by a FIR filter downsampling, the second item What is calculated is the error between the image output by the CNN upsampling filter and the original resolution image. According to the formula (6), the up-sampling filter and down-sampling filter are trained, and the optimal parameters of the CNN down-sampling filter can be obtained

Optionally, as an embodiment, before using the target downsampling filter to downsample the image block to be coded, the method shown in FIG. 6 further includes: according to the image block to be coded, the CNN downsampling filter and/or The sampling filter is trained online to obtain the updated parameter value of the CNN downsampling filter and/or the updated parameter value of the CNN downsampling filter, wherein the updated parameter value of the CNN downsampling filter is used to replace the CNN downsampling filter in advance The set parameter value, the updated parameter value of the CNN upsampling filter is used to replace the preset parameter value of the CNN upsampling filter.

Through online training of CNN downsampling filter parameters, the information loss caused by the downsampling operation can be reduced as much as possible, and the quality of the reconstructed image can be improved. In addition, since the CNN downsampling filter is only used in the encoder, there is no need to pass the CNN downsampling filter parameters to the decoder, which will not increase the encoding overhead.

By training the CNN upsampling filter online, an upsampling filter with more matching filter parameters can be used for the upsampling operation according to the image texture characteristics. Compared with the preset CNN upsampling filter, the upsampling output can be further improved. The image quality of the image block.

By simultaneously training the CNN upsampling filter and the CNN downsampling filter online, it is possible to use the upsampling filter and downsampling filter with better matching filter parameters to perform upsampling and downsampling operations according to the image texture characteristics, which is different from using preset Compared with the CNN downsampling filter, the CNN upsampling filter can further improve the image quality of the upsampling output image block.

Optionally, the method shown in FIG. 6 further includes: writing the updated parameter value of the CNN upsampling filter into the code stream.

After obtaining the update parameter value of the CNN upsampling filter, the update parameter value of the CNN upsampling filter can also be written into the code stream, so that the decoder can also obtain the update parameter of the CNN upsampling filter by parsing the code stream. And according to the update parameters of the CNN upsampling filter, the preset parameter value of the CNN upsampling filter is updated, so that the decoding end can use the updated parameter value of the CNN upsampling filter at the encoding end to perform an upsampling operation.

Optionally, as an embodiment, before determining an upsampling filter whose corresponding encoding cost meets a preset requirement in the upsampling filter set as the target upsampling filter, the method shown in FIG. 6 further includes: according to the The coded image block jointly trains the CNN upsampling filter and the CNN downsampling filter online, and obtains the updated parameter value of the CNN upsampling filter and the updated parameter value of the CNN downsampling filter.

Optionally, before determining the target upsampling filter from the preset upsampling filter set according to the coding cost of the image block to be coded, the method shown in FIG. 6 further includes: according to the coding cost of the image block to be coded, from Determine the target encoding mode from the original resolution encoding mode and the variable resolution encoding mode; generate encoding mode indication information corresponding to the target encoding mode; write the encoding mode indication information into the code stream.

Optionally, the target upsampling filter is determined from the preset upsampling filter set according to the encoding cost of the image block to be encoded, which specifically includes: determining the encoding cost of the image block to be encoded in the original resolution encoding mode; determining the variable The encoding cost of the image block to be encoded in the resolution encoding mode; the encoding mode with the smallest encoding cost is selected from the original resolution encoding mode and the variable resolution encoding mode as the target encoding mode.

The above coding mode indication information is used to indicate which coding mode among the candidate coding modes is the target coding mode. For example, the above encoding mode indication information may indicate that the original resolution encoding mode is the target encoding mode, or the above encoding mode indication information may also indicate the variable resolution encoding mode as the target encoding mode.

According to the encoding cost of the image block to be encoded, an encoding mode with a lower encoding cost can be selected as the target encoding mode, which can reduce the loss of image encoding in the encoding process.

It should be understood that the target upsampling filter is determined from the preset upsampling filter set according to the coding cost of the image block when it is determined to use the variable resolution coding mode to encode the image block to be coded. When it is determined to use the variable resolution coding mode to encode the image block to be coded, the original resolution coding mode is directly used to encode the image block to be coded, and there is no need to determine the target upsampling filter, because in the original resolution coding mode, there is no Upsampling is required.

Fig. 7 is a schematic flowchart of an image encoding method according to an embodiment of the present application. The method shown in FIG. 7 can be executed by an encoding end device, and the encoding end device can specifically be a smart terminal, a PAD, a video player, an Internet of Things device, or other devices with a video image encoding function. The method shown in FIG. 7 includes steps 501 to 506, and the steps 501 to 506 will be described in detail below in conjunction with specific examples.

501. Determine a target downsampling filter from a preset set of downsampling filters according to an encoding cost of an image block to be encoded.

Wherein, the downsampling filter set includes at least an FIR downsampling filter and a CNN downsampling filter.

Determine the target downsampling filter from the preset downsampling filter set according to the encoding cost of the image block to be encoded The method of the sampling filter is substantially the same as the specific process, and will not be repeated here.

502. Use a target downsampling filter to downsample the image block to be coded to obtain a first image block.

503. Encode the first image block to obtain a code stream.

504. Determine an upsampling filter of the same type as the target downsampling filter in the preset upsampling filter set as the target upsampling filter, wherein the upsampling filter set includes at least a finite impulse response FIR upsampling filter and convolutional neural network CNN upsampling filter.

505. Generate upsampling filter indication information corresponding to the target upsampling filter.

506. Write the upsampling filter indication information into the code stream.

It should be understood that, for specific definitions and explanations of the steps of the method shown in FIG. 7 , reference may be made to the above explanations and definitions of corresponding steps in the methods shown in FIGS. 4 and 6 , and will not be described in detail here.

Optionally, in this application, both the target up-sampling filter and the target down-sampling filter may be determined from a preset up-sampling filter set and target down-sampling filter set according to the encoding cost of the image block to be encoded. Also, the processes of determining the target up-sampling filter and determining the target down-sampling filter may be independent of each other.

In this application, the target downsampling filter can be determined from the downsampling filter set according to the encoding cost, and the upsampling filter can be determined according to the filter type of the downsampling filter, and the upsampling filter directly using fixed parameter values Compared with the upsampling operation, the scheme of the present application fully considers the coding cost of the image block to be coded when selecting the target upsampling filter, and can achieve better coding effect.

The above describes the image encoding method of the embodiment of the present application in detail from the perspective of the encoding end with reference to FIGS. 4 to 7 . The image decoding method of the embodiment of the present application is described in detail below in conjunction with FIG. 8 . It should be understood that the process of the image decoding method shown in FIG. 8 corresponds to or is opposite to the process of the image coding method shown in FIG. 4 , FIG. 6 and FIG. 7 above.

Fig. 8 is a schematic flowchart of an image decoding method according to an embodiment of the present application. The method shown in FIG. 8 may be executed by a decoding end device, and the decoding end device may specifically be a smart terminal, a PAD, a video player, an Internet of Things device, or other devices with a video image decoding function. The method shown in FIG. 8 includes steps 401 to 408, and the steps 401 to 408 will be described in detail below with reference to specific examples.

601. Obtain a code stream.

The above code stream may be a code stream obtained through the image coding method of the present application. Specifically, the above code stream may be obtained by the image coding method shown in FIG. 4 , FIG. 6 and FIG. 7 above.

602. Perform entropy decoding, inverse quantization, and inverse transformation on the code stream to obtain a reconstructed residual signal of the image block to be decoded.

603. Acquire a prediction signal of an image block to be decoded.

604. Add the reconstruction residual signal and the prediction signal to obtain an initial reconstructed image block of the image block to be decoded.

605. Parse the code stream to acquire coding mode indication information.

The above encoding mode indication information is used to indicate whether the encoding end adopts the original resolution encoding mode or the variable resolution encoding mode during encoding. That is to say, the above coding mode indication information will indicate in which coding mode the above code stream is obtained. For example, the above coding mode indication information may indicate that the code stream is obtained by coding in the original resolution coding mode or the variable resolution coding mode.

606. Determine a target decoding mode from the original resolution decoding mode and the variable resolution decoding mode according to the encoding mode indication information.

Optionally, determining the target decoding mode from the original resolution decoding mode and the variable resolution decoding mode according to the coding mode indication information includes: when the coding mode is the variable resolution coding mode, determining that the target decoding mode is the variable resolution decoding mode ; When the encoding mode is the original resolution encoding mode, determine that the target decoding mode is the original resolution decoding mode.

Optionally, the above encoding mode indication information is specifically represented by the value of the variable resolution encoding mode flag Flag1, and different values of Flag1 are used to indicate different encoding modes. For example, when Flag1 is 0, it means that the above bit stream is obtained in the original resolution encoding mode, and when Flag1 is 1, it means that the above bit stream is obtained in the variable resolution encoding mode.

When the above coding mode indication information is represented by the value of the variable resolution coding mode flag Flag1, the decoder can determine the target decoding mode according to the value of Flag1. For example, when Flag1 is 0, the decoding end determines that the target decoding mode is the variable resolution decoding mode; when Flag1 is 1, the decoding end determines that the target decoding mode is the original resolution decoding mode.

In addition, the above encoding mode information may also be represented by the value of the original resolution encoding mode flag Flag1, and different values of Flag1 are used to indicate different encoding modes. For example, when Flag1 is 0, it indicates that the target encoding mode is a variable resolution encoding mode, and when Flag1 is 1, it indicates that the target encoding mode is an original resolution encoding mode.

The decoding end selects the decoding mode corresponding to the encoding mode of the encoding end as the target decoding mode, which can ensure the decoding effect of the decoding end.

607. When the target decoding mode is the variable resolution decoding mode, analyze the code stream of the image block to be encoded, and acquire upsampling filter indication information.

When the decoding end determines that the target decoding mode is the variable resolution decoding mode, it means that the above bit stream is also encoded in the variable resolution encoding mode, and the encoding end will write the upsampling filter indication into the bit stream, so that the decoding end can pass the analysis Code stream gets upsampling filter indication information.

The above upsampling filter indication information may be used to indicate that a certain upsampling filter in the upsampling filter set is a target upsampling filter.

For example, the upsampling filter indication information may indicate the FIR upsampling filter in the upsampling filter set as the target upsampling filter, or the upsampling filter indication information may also indicate the CNN upsampling filter in the upsampling filter set filter as the target upsampling filter.

608. Determine a target upsampling filter from a preset upsampling filter set according to the upsampling filter indication information, where the upsampling filter set includes at least an FIR upsampling filter and a CNN upsampling filter.

Optionally, the upsampling filter indication information may be an upsampling filter selection flag Flag2, and a value of Flag2 is used to indicate a target upsampling filter.

For example, when the value of Flag2 is 0, the FIR filter is the target upsampling filter, and when the value of Flag2 is 1, the CNN filter is the target upsampling filter 1.

Alternatively, when the value of Flag2 is 0, the CNN filter is the target upsampling filter, and when the value of Flag2 is 1, the FIR filter is the target upsampling filter 1.

The decoding end obtains the value of Flag2 by analyzing the code stream, and can directly determine the target upsampling filter according to the value of Flag2.

609. Upsample the initial reconstructed image block by using the target upsampling filter to obtain the target reconstructed image block.

In this application, when decoding an image block in the variable resolution decoding mode, the decoding end can select the corresponding upsampling filter from the preset upsampling filter set as the target upsampling filter according to the upsampling filter indication information Filter, and then perform upsampling operation. Compared with the way of directly using fixed parameter upsampling filter for upsampling operation, the matching upsampling filter can be selected according to the situation of the image block for upsampling operation, thereby improving the decoding effect .

Optionally, as an embodiment, the parameter value of the CNN upsampling filter is preset, and the parameter value of the CNN upsampling filter is obtained by performing offline training on a preset image training set.

For the offline training process of the CNN upsampling filter, refer to the specific process described in the image encoding method shown in Fig. 4, Fig. 6 and Fig. 7. For the sake of brevity, the description will not be repeated here.

Since the parameter values of the CNN upsampling filter are obtained through offline training, using the CNN upsampling filter for upsampling during the decoding process can reduce the information loss of the image during the upsampling process and improve the image decoding quality.

In a possible implementation, the above method further includes: parsing the code stream to obtain an updated parameter value of the CNN upsampling filter, where the updated parameter value of the CNN upsampling filter is used to replace the preset value of the CNN upsampling filter parameter value.

The encoding end can update the CNN upsampling filter regularly or irregularly, and write the obtained parameter value of the CNN upsampling filter into the code stream, so that the decoding end can also use the parameter value of the CNN upsampling filter Update the initial parameter values of the CNN upsampling filter.

By updating the parameter values of the CNN upsampling filter, filter parameters that better match the image content can be obtained, thereby further reducing image information loss and improving image decoding quality when using the CNN upsampling filter for upsampling operations .

Optionally, the updated parameter value of the CNN upsampling filter is obtained by the encoder through online training of the CNN upsampling network according to the image block to be encoded, wherein the image block to be decoded is obtained by encoding the image block to be encoded.

It should be understood that the image block to be encoded corresponds to the image block to be decoded, that is, the image block to be encoded is encoded to obtain a code stream of the image block to be decoded.

By training the CNN upsampling filter online, an upsampling filter with more matching filter parameters can be used for the upsampling operation according to the image texture characteristics. Compared with the preset CNN upsampling filter, the upsampling output can be further improved. The image quality of the image block.

Optionally, the updated parameter values of the above-mentioned CNN upsampling filter are obtained by performing offline training on a preset image training set.

The updated parameter value of the CNN upsampling filter is obtained through offline training, and the parameter value of the CNN upsampling filter can be updated, thereby improving the encoding quality.

In a possible implementation, before parsing the code stream and obtaining the update parameter value of the CNN upsampling filter, the method further includes: parsing the code stream, obtaining filter parameter update indication information, and the filter parameter update indication information is used for Indicates whether to update the parameter value of the target upsampling filter; parse the code stream to obtain the updated parameter value of the CNN upsampling filter, including: indicating that the parameters of the target upsampling filter are updated in the filter parameter update instruction information In this case, parse the code stream to obtain the updated parameter value of the CNN upsampling filter.

Optionally, the above filter update instruction information is specifically the value of Flag3, and different values of Flag3 are used to indicate whether to update the parameter value of the target upsampling filter.

For example, when the value of Flag3 is 0, it means that the parameter value of the CNN upsampling filter does not need to be updated; when the value of Flag3 is 1, it means that the parameter value of the CNN upsampling filter does not need to be updated;

Optionally, the above filter parameter update instruction information is carried in the SPS or the PPS.

The update value of the CNN upsampling filter is obtained from the code stream only when the filter parameter update indication information indicates that the target upsampling filter needs to be updated, which can reduce the decoding burden and improve decoding efficiency.

In this application, the filter parameter update indication information can be sent periodically, so that the parameter value of the CNN upsampling filter can be updated according to the update parameter value of the CNN upsampling filter, so as to improve the coding effect as much as possible .

The image encoding method and the image decoding method according to the embodiments of the present application are described in detail from the perspectives of the encoding end and the decoding end with reference to FIGS. 4 to 8 . In order to better understand the image encoding and decoding method of the embodiment of the present application, the specific process of the image encoding and decoding method of the embodiment of the present application will be described in detail below in conjunction with specific embodiments.

Embodiment 1: Both the up-sampling filter and the down-sampling filter are CNN filters or FIR filters.

In the first embodiment, the upsampling filter and the downsampling filter use the same type of filter, and the specific type of filter used by the upsampling filter and the downsampling filter can be filtered from the CNN according to the encoding cost of the image block filter and FIR filter to choose from.

In Embodiment 1, the specific architecture of the encoder may be as shown in FIG. 9 . When encoding the image block, there are original resolution encoding mode and variable resolution encoding mode to choose from. When selecting the encoding mode, you can choose from the original resolution encoding mode and variable resolution encoding mode according to the encoding cost of the image block. One encoding mode encodes image blocks. When it is determined to encode the image block in the original resolution encoding mode, directly use the original resolution encoding module shown in Figure 9 to encode the image block; and when it is determined to encode the image block in the variable resolution mode, The upsampling filter needs to be selected from the CNN upsampling filter and the FIR upsampling filter, and the downsampling filter is selected from the CNN downsampling filter and the FIR downsampling filter. In order to ensure the encoding effect of the image block, the downsampling filter and the upsampling filter use the same type of filter, as shown in Figure 9, when the upsampling filter is a CNN upsampling filter, the downsampling filter is a CNN downsampling filter A sampling filter, when the up-sampling filter is an FIR up-sampling filter, the down-sampling filter is an FIR down-sampling filter.

Wherein, the CNN upsampling filter, the CNN downsampling filter, the FIR upsampling filter and the FIR downsampling filter can be pre-designed (filter parameter values have been determined) and built into the encoder.

In Embodiment 1, the encoding end needs to select the encoding mode and filter. Therefore, the encoding end will generate two unique selection information when encoding the image block, that is, the original resolution or low resolution encoding mode selection information and the CNN or FIR filter selection information, and write the two selection information into the code stream. The decoder can identify whether an image block adopts the variable resolution decoding mode, and in the case of adopting the variable resolution decoding mode, select the corresponding upsampling filter to perform an upsampling operation on the decoded low resolution reconstructed image block, In this way, the reconstructed image block with the original resolution is obtained.

As shown in Figure 10, in Embodiment 1, the encoding process of the encoder is specifically as follows:

701. Acquire an image block.

Specifically, the above-mentioned image block may be a complete frame of image, or a part of a frame of image. When the above image block is part of a frame of image, the image may be divided before step 701 to obtain multiple image blocks, and then in step 701 the image block to be coded is obtained.

702. Determine the encoding cost of the current image block in the original resolution encoding mode.

Specifically, in step 702, it is necessary to calculate the encoding cost of encoding the current image block in the original resolution encoding mode.

The encoding cost of the current image block in the original resolution encoding mode can be calculated through the following process.

Firstly, the current image block is encoded in the original resolution mode according to the preset codec scheme to obtain the compressed code stream of the current image block and the reconstructed image block of the current image block;

Secondly, calculate the mean square error D _org between the reconstructed image block and the current image block, and determine the size R _org of the compressed code stream of the current image block;

Finally, calculate the rate-distortion cost cost _org for encoding the current image block in the original resolution encoding mode according to D _org and R _org , and determine the rate-distortion cost cost _org as the encoding cost of the current image block in the original resolution encoding mode.

It should be understood that the encoding cost has many specific forms. For example, the encoding cost may specifically be a rate-distortion cost or distortion, etc. The specific form of the encoding cost is not limited in this application.

703. Determine the encoding cost of the current image block when the FIR upsampling filter is used as the target upsampling filter.

Step 703 is essentially to calculate the encoding cost of encoding the current image block in the variable resolution encoding mode (the FIR upsampling filter is used as the upsampling filter).

Specifically, the encoding cost of the current image block when the FIR upsampling filter is used as the target upsampling filter may be calculated through the following process.

(1) adopt FIR down-sampling filter (typical FIR down-sampling filter is bicubic down-sampling filter) to carry out down-sampling operation to current image block, obtain the image block of low resolution;

(2) Encoding the low-resolution image block according to the selected encoding and decoding scheme, and outputting a compressed code stream;

(3) Reconstructing the low-resolution image block to obtain a low-resolution reconstructed image block;

(4) Upsampling the low-resolution reconstructed image block by using an FIR upsampling filter to obtain a reconstructed image block identical to the original resolution of the current image block;

(4) Calculate the mean square error D _FIR between the reconstructed image block and the current image block, and determine the size R _FIR of the compressed code stream of the current image block;

(5) Calculate the rate-distortion cost cost _FIR according to D _FIR and R _FIR , and determine the rate-distortion cost cost _FIR as the encoding cost of the current image block when the FIR filter is used as the target upsampling filter.

In the above process of calculating the encoding cost of the current image block when the FIR upsampling filter is used as the target upsampling filter, in order to obtain the same reconstructed image block as the decoding end, the FIR upsampling filter used here is the same as that used by the decoding end. The FIR upsampling filter must be the same filter (same filter parameters).

704. Determine the coding cost of the current image block when the CNN filter is used as the target upsampling filter.

Step 704 is essentially to calculate the encoding cost of encoding the current image block in the variable resolution encoding mode (the CNN upsampling filter is used as the upsampling filter).

Specifically, the encoding cost of the current image block when the CNN filter is used as the target upsampling filter can be calculated through the following process.

(1) Use the CNN downsampling filter to perform a downsampling operation on the current image block to obtain a low-resolution image block;

(2) Encoding the low-resolution image block according to the selected encoding and decoding scheme, and outputting a compressed code stream;

(3) Reconstructing the low-resolution image block to obtain a low-resolution reconstructed image block;

(4) Upsampling the low-resolution reconstructed image block by using a CNN upsampling filter to obtain a reconstructed image block with the same original resolution as the current image block;

(5) Calculate the mean square error D _CNN between the reconstructed image block and the current image block, and determine the size R _CNN of the compressed code stream of the current image block;

(6) Calculate the rate-distortion cost cost _CNN according to _DCNN and R- _CNN , and determine the rate-distortion cost cost _CNN as the coding cost of the current image block when the CNN filter is used as the target upsampling filter.

In the above process of calculating the encoding cost of the current image block when the CNN upsampling filter is used as the target upsampling filter, in order to obtain the same reconstructed image block as the decoder, the CNN upsampling filter used here is the same as that used by the decoder. The CNN upsampling filter must be the same filter (the filter parameters are the same).

Fig. 11 shows a specific structure of the CNN downsampling filter. The network structure of the CNN downsampling filter consists of 10 convolutional layers. The convolution kernel size of each convolutional layer is 3x3. Except for the last convolutional layer, the other convolutional layers are followed by linear Rectified Linear Unit (ReLU), where the linear rectification functions used in convolutional layer 1 to convolutional layer 9 are ReLU.1 to ReLU.9, and the step size (Stride) of the first layer of convolution is set to 2 . In addition, the network structure adds the low-resolution image obtained after the input image block is down-sampled by a FIR filter to the output of the last layer of the network, so that the network has residual learning characteristics.

In addition, the FIR filter in the network structure shown in Figure 11 can specifically be a bicubic filter, an interpolation filter based on discrete cosine transform (Discrete Cosine Transform Based Interpolation Filter, DCTIF), etc., wherein, the interpolation filter based on discrete cosine transform The filter can be directly referred to as a DCTIF filter for short.

The above step 703 and step 704 are essentially to determine the cost of encoding the current image block in the variable resolution encoding mode. Since different upsampling filters can be used for upsampling during variable resolution encoding, it is necessary to separately Calculate the encoding cost of the current image block when different filters (FIR upsampling filter and CNN upsampling filter) are used as the target upsampling filter.

705. Perform an encoding operation on the current image block in an encoding mode with the smallest encoding cost to generate a code stream.

Specifically, in step 705, a coding mode may be selected according to the size of the coding cost calculated in steps 702 to 704, and coding information may be generated. The encoding information may specifically include a variable resolution encoding mode indication flag Flag1 and an upsampling filter selection flag Flag2.

The above-mentioned setting of Flag1 and Flag2 according to the encoding cost specifically includes the following three situations.

(1) When cost _org <cost _FIR and cost _org <cost _CNN , it is determined to encode the current image block in the original resolution mode. At this time, Flag1 is set to 0, and Flag2 is meaningless, so there is no need to set it;

(2) When cost _FIR <cost _org and cost _FIR <cost _CNN , it is determined to encode the current image block in the variable resolution mode, and use the FIR filter to perform up-sampling and down-sampling operations, and set Flag1 at this time is 1, set Flag2 to 0;

(3) When cost _CNN <cost _org and cost _CNN <cost _FIR , determine to encode the current image block in the variable resolution mode, and use the CNN filter to perform up-sampling and down-sampling operations, and set Flag1 at this time is 1, set Flag2 to 1.

706. Write the encoding information of the current image block into the code stream.

In step 706, in addition to the variable resolution coding mode flag Flag1 and upsampling filter selection flag Flag2 determined in step 705, the coding information may also include the prediction mode when coding the current image block and the quantized coding coefficients and so on.

The decoding process of the decoding end in Embodiment 1 is shown in FIG. 12 . The decoding process shown in FIG. 12 corresponds to the encoding process shown in FIG. 10 , and FIG. 12 specifically includes the following steps.

801. Acquire a compressed code stream.

The compressed code stream acquired in step 801 may be a code stream finally generated by the encoding method shown in FIG. 10 . When obtaining the compressed code stream, the decoder can obtain it directly from the coder or from the server.

802. Analyze the variable resolution coding mode indicator Flag1 of the current image block.

The current image block may be the image block currently to be decoded, the compressed code stream obtained in step 801 may be the code stream of the current image block, and after the code stream of the current image block is acquired, it may be parsed from the code stream of the current image block The variable resolution encoding mode indicates Flag1, and obtains the value of Flag1.

803. Determine whether to adopt a variable resolution decoding mode according to Flag1.

The value of Flag1 can be used to indicate whether the variable resolution coding mode or the original resolution coding mode is adopted when generating the code stream of the current image block. Therefore, whether to decode the current image block in the variable resolution decoding mode can be determined according to the value of Flag1.

For example, when the value of Flag1 is 0, it means that the original resolution coding mode is used when generating the code stream of the current image block; when the value of Flag1 is 1, it means that the variable resolution coding is used when generating the code stream of the current image block model.

Therefore, when the value of Flag1 is 0, it can be determined that the current image block is decoded in the original resolution coding mode; when the value of Flag1 is 1, it can be determined that the current image block is decoded in the variable resolution coding mode.

804. Analyze the upsampling filter selection flag Flag2 of the current image block.

The value of Flag2 can be used to indicate whether to use the FIR upsampling filter or the CNN upsampling filter as the upsampling filter for upsampling during the encoding process.

805. Analyze the coding information of the low-resolution coding block, and obtain a low-resolution reconstructed image block.

Since the encoding end can select different encoding schemes to encode image blocks, it is also necessary to select a decoding scheme corresponding to the encoding scheme for decoding when analyzing and decoding information to obtain reconstructed image blocks.

For example, if an image block is encoded in a manner similar to the Joint Photographic Experts Group (JPEG) (an international image compression standard), then only the quantized transform coefficients of the image block need to be parsed from the code stream during decoding information, and perform inverse quantization and inverse transformation operations on the coefficients to obtain the original resolution image block reconstruction.

In addition, if an image block is encoded in a manner similar to H.264 or H.265 intra-frame encoding, then encoding information such as the encoding mode, prediction mode, and quantized transform coefficient of the current block needs to be decoded from the code stream during decoding. And after inverse quantization and inverse transformation of the quantized transform coefficients to obtain the reconstructed residual signal, it is necessary to perform a prediction operation according to the prediction mode to obtain the prediction signal, and finally add the two to obtain the reconstruction of the original resolution image block.

Specifically, as an optional implementation, obtaining low-resolution reconstruction blocks mainly includes the following processes:

First, entropy decoding is performed on the compressed code stream of the current image block to obtain quantized coefficients;

Secondly, dequantize and transform the quantization coefficients to obtain the reconstruction residual signal of the current image block;

Again, obtain the prediction signal of the current image block (or called the prediction image block);

Finally, according to the residual signal and the prediction signal of the current image block, a low-resolution reconstructed image block of the current image block is obtained.

806. Analyze the coding information of the coding block with the original resolution, and obtain the reconstructed image block with the original resolution.

The process of obtaining the reconstructed image block with original resolution in step 806 is the same as the specific process of obtaining the reconstructed image block with low resolution in step 805, except that the input signal in step 805 is the code stream obtained in low-resolution encoding mode, and the step The input signal in 806 is the code stream acquired in the original resolution encoding mode.

807. Determine whether to use a CNN upsampling filter according to Flag2.

Specifically, it may be specifically determined to use the CNN upsampling filter or the FIR upsampling filter for upsampling according to the value of Flag2.

For example, when the value of Flag2 is 0, it means that the CNN upsampling filter is used for upsampling, and when the value of Flag2 is 1, it means that the FIR upsampling filter is used for upsampling. Therefore, when the value of Flag2 is 1, it is determined to use the CNN upsampling filter, and when the value of Flag2 is 0, it is determined not to use the CNN upsampling filter (the FIR upsampling filter is used).

808. Use the CNN upsampling filter to upsample the low-resolution reconstruction block to obtain the original resolution reconstruction image block.

The CNN upsampling filter used in step 808 can be shown in Figure 13. Figure 13 shows a network structure of a CNN upsampling filter. The network is divided into five layers, the first two layers are convolutional layers, and multi-scale The feature extraction operation, the third layer is the deconvolution layer, which performs the upsampling operation of multi-scale features, and the last two layers are convolution layers, which complete the multi-scale reconstruction. The size of the convolution kernel of the first convolutional layer is 5x5, and the number of feature maps is 64; the second layer uses both 5x5 and 3x3 convolution kernels, and the number of feature maps is 16 and 32 respectively; the deconvolution of the third layer The size of the product kernel is 12x12, and the number of feature maps is 48; the fourth layer uses both 3x3 and 1x1 convolution kernels, and the corresponding number of feature maps are 16 and 32 respectively; the size of the convolution kernel of the fifth layer is 3x3, The number of feature maps is 1, which is the output original resolution image block. After each layer of the first 4 convolutional layers, a ReLU layer is added as an activation function to activate the convolutional output signal.

809. Use an FIR upsampling filter to upsample the low-resolution reconstruction block to obtain a reconstruction image block with the original resolution.

In step 809, the DCTIF interpolation filter in the H.265 standard solution may be used to interpolate the low-resolution reconstruction to obtain the original-resolution reconstruction.

In steps 808 and 809, the original resolution refers to the resolution of the image block to be encoded corresponding to the reconstructed image block at the encoding end, and the reconstructed image block with the same resolution as the original image to be encoded can be obtained through upsampling.

810. Determine whether the current image block is the last image block.

Specifically, it may be determined whether the current image block is the last image block to be decoded according to other encoding information carried in the code stream.

In the case that the current image block is not the last image block to be decoded, continue to execute step 802 to continue decoding the next image block; in the case that the current image block is the last image block to be decoded, execute step 811 , the decoding process of the current image ends.

811. The decoding of the current image ends.

In Embodiment 1, the CNN up-sampling filter and the CNN down-sampling filter can be obtained through pre-training and pre-installed in the encoder and the decoder. The encoder and decoder directly use the preset CNN upsampling filter and CNN upsampling filter in the process of encoding and decoding.

Specifically, the parameter values of the CNN up-sampling filter and the CNN down-sampling filter may be preset parameter values, and the preset parameter values may be obtained through offline training based on the image training set.

In the first embodiment, since the up-sampling filter and the down-sampling filter may respectively adopt the CNN up-sampling filter and the CNN down-sampling filter, the CNN up-sampling filter and the CNN down-sampling filter can be obtained through joint training. The parameter value of the sampling filter.

The joint training of CNN upsampling filter and CNN downsampling filter includes the following specific steps:

Step 1. Separately train the CNN upsampling network to obtain the initial parameters of the upsampling network.

The pictures in the training set are down-sampled, and the obtained down-sampled images are used as the input of the CNN upsampling network, and the original pictures in the training set are used as the training target. Specifically, the CNN upsampling network can be trained by formula (7), and then the initial parameters of the upsampling network can be obtained.

In the above formula (7), _xi represents the original image of the training set, y _i represents the low-resolution image obtained after downsampling the original image, n represents the number of training samples, and θ _g represents the CNN upsampling network. All parameters, g(y _i ; θ _g ) means that the upsampling network performs an upsampling operation on the input y _i , Indicates the initial parameters of the CNN upsampling network obtained after the training in step 1.

Step 2: Train the CNN downsampling network based on the CNN upsampling network trained in step 1 to obtain initial parameters of the CNN downsampling network.

The original image in the training set is used as the input of the CNN downsampling network, the output of the CNN downsampling network is used as the input of the CNN upsampling network obtained in the above step 1, and the image output by the CNN upsampling network is obtained, and the CNN upsampling network is calculated. The mean square error between the output image and the original image is used as the loss function to minimize the loss function, thereby obtaining the initial parameters of the CNN downsampling network.

Specifically, the CNN upsampling network may be trained through formula (8), and then initial parameters of the upsampling network may be obtained.

In formula (8), θ _f represents all the parameters in the CNN downsampling network, f( _xi ; θ _f ) represents the downsampling network performs downsampling operation on the input _xi , Indicates the initial parameters of the CNN downsampling network obtained after the training in step 2.

Step 3. Jointly train the up-sampling network and the down-sampling network to obtain updated network parameters.

Parameters trained in step 1 and step 2 are the initial parameters of the upsampling network and the downsampling network, respectively. Next, use the original resolution images in the training set as input and output, and train the parameters of the upsampling and downsampling networks at the same time.

Specifically, CNN upsampling network and downsampling network can be trained simultaneously according to formula (9).

In formula (9), the loss function consists of two terms, the first term The calculation is the error between the image output by the CNN downsampling network and the image output by a FIR filter downsampling, the second item The calculation is the error between the image output by the CNN upsampling network and the original resolution image. According to formula (9), the up-sampling network and down-sampling network are trained, and the optimal parameters of the CNN down-sampling network can be obtained The CNN downsampling network with the optimal parameters is the CNN downsampling filter used in the encoding process of this embodiment.

Step 4. Fine-tune the upsampling network based on the images in the training set

Use the optimal downsampling network obtained after the training in step 3 to downsample all the images in the training set, and then compress and encode these low-resolution images to obtain low-resolution reconstructed images. The low-resolution compressed and reconstructed image is used as input, and the corresponding original resolution image in the training set is used as the training target, and the upsampling network is retrained according to the method of step 1 to obtain the optimal parameters of the CNN upsampling network The CNN upsampling network with the optimal parameters is the CNN upsampling filter used in the encoding and decoding process of this embodiment. It should be understood that in step 4, the image training set is still used to train the upsampling network.

When training the CNN downsampling filter and CNN upsampling filter in the above steps 1 to 4, the process shown in Figure 14 can be used for joint training. When constructing the reconstruction loss function In addition, construct a regularization loss function The CNN upsampling filter and the CNN upsampling filter are preset through the image training set, so that the reconstruction loss function and the regularization loss function are minimized, and the corresponding parameter values when the reconstruction loss function and the regularization loss function are minimized are used as CNN downsampling Parameter values for filters and CNN upsampling filters.

In the first embodiment, when encoding each image block, there are two encoding modes for selection: the original resolution encoding mode and the low resolution encoding mode. And when the image block is encoded in the original resolution encoding mode, there are two options for the up-sampling filter and the down-sampling filter, one is that the up-sampling filter and the down-sampling filter both use CNN filters, and the other Both the up-sampling filter and the down-sampling filter use FIR filters. In the first embodiment, by adaptively selecting the up-sampling filter and the down-sampling filter, the compression coding efficiency of the existing block-level variable resolution coding scheme can be significantly enhanced. In addition, in Embodiment 1, the parameter values of the CNN upsampling filter and the CNN downsampling filter are obtained through joint offline training, and are preset in the encoder and decoder, which can reduce the image texture in the upsampling and downsampling processes. The loss of information brought about in the process improves the quality of the encoded image.

Embodiment 2: the downsampling filter is a preset FIR downsampling filter.

In the second embodiment, the filter parameters of the downsampling filter are fixed, but the upsampling filter is not fixed, and the upsampling filter can be determined from the CNN filter and the FIR filter according to the image content of the image block , and write the upsampling filter parameters into the code stream, so that the decoding end can know the parameters of the upsampling filter.

The system architecture of the encoder in Embodiment 2 may be shown in FIG. 15 . When the encoder uses the variable resolution coding mode to encode an image block, a preset FIR downsampling filter is used to downsample the original resolution image block to obtain a low resolution image block. When the encoder performs an upsampling operation on low-resolution image block reconstruction, one of the preset CNN upsampling filters and the preset FIR upsampling filters can be flexibly selected as an upsampling filter. In addition, the encoder needs to write variable resolution mode indication information and upsampling filter selection flag information into the compressed code stream, so that the decoder can use the corresponding upsampling filter to perform decoding operations in the corresponding decoding mode.

It should be understood that in Embodiment 2, the encoding process of the encoder is basically the same as that of the encoder in Embodiment 1. In Embodiment 2, the method of determining the upsampling filter in Embodiment 1 can be used to upsample from FIR The up-sampling filter is selected from the CNN up-sampling filter and the CNN up-sampling filter. The difference is that in the second embodiment, the down-sampling filter is fixed, so it is not necessary to determine the down-sampling filter in the second embodiment.

The decoding process of the decoder in the second embodiment is the same as the decoding process of the decoder in the first embodiment, and for the sake of brevity, no repeated description is given here.

In the second embodiment, by adaptively selecting the CNN upsampling filter or the FIR upsampling filter, the compression coding efficiency of the existing block-level variable resolution coding scheme can be significantly enhanced. In addition, the FIR upsampling filter is fixed, and the encoder does not need to perform calculation and comparison between multiple downsampling filters, thereby reducing the complexity of encoding operations and speeding up encoding.

In the second embodiment, the CNN upsampling filter can be obtained through pre-training and pre-installed in the encoder and decoder. The encoder and decoder directly use the preset CNN upsampling filter in the process of encoding and decoding.

Specifically, the CNN upsampling filter parameter value may be a preset parameter value, and the preset parameter value may be obtained through offline training based on an image training set.

In the second embodiment, when the CNN upsampling filter is trained offline, the preset FIR downsampling filter can be used to downsample the original image x _i in the training set, and then the image y _i obtained by downsampling can be used as The CNN upsamples the input of the filter and takes the original image _xi as the training target.

Next, the CNN upsampling network can be trained according to the general training method of the CNN network, so as to obtain the optimal parameters of the upsampling filter Specifically, the CNN upsampling network can be trained according to formula (10), and then the optimal parameters of the upsampling filter can be obtained

In formula (10), _xi represents the original image in the training set, y _i represents the low-resolution image obtained after downsampling the original image, n represents the number of training samples, θ _g represents all parameters in the CNN upsampling network, g (y _i ; θ _g ) means that the upsampling network performs an upsampling operation on the input y _i .

In addition, in the above training process, in addition to using the downsampled image _yi as the network input for training. It is also possible to encode y _i first to obtain the downsampled image reconstruction y′ _i , and then use y′ _i as network input for training. The upsampling network trained in this way can not only restore the details of the original resolution image, but also eliminate the image defects introduced by the encoding operation to a certain extent, and further improve the reconstruction quality of the original resolution image block output by the upsampling operation.

Embodiment 3: The upsampling filter is a preset FIR filter.

In the third embodiment, the filter parameters of the upsampling filter are fixed, but the downsampling filter is not fixed, and can be determined from the CNN downsampling filter and the FIR downsampling filter according to the image content of the image block For the downsampling filter, since the parameters of the upsampling filter are fixed, that is, the parameters of the upsampling filter are known to both the encoding end and the decoding end. Therefore, the encoding end does not need to write the parameter information of the upsampling filter Into the code stream, saving the code stream.

The system architecture of the encoder in Embodiment 3 may be shown in FIG. 16 . In the third embodiment, when the encoder encodes the image block, it can fixedly use a preset FIR upsampling filter for upsampling, and can select from the preset CNN downsampling filter and FIR downsampling filter Flexible choice of a filter as downsampling filter. In the third embodiment, since the decoder only needs to perform an upsampling operation, the encoder can flexibly select any downsampling filter without transmitting the selection information of the downsampling filter to the decoder.

The encoding process of the encoder in Embodiment 3 specifically includes the following processes.

Firstly, the encoding cost of encoding the image block to be encoded in the original resolution encoding mode and the variable resolution encoding mode is determined respectively.

Specifically, determining the encoding costs of the image blocks to be encoded in different encoding modes may include Step 1, Step 2 and Step 3.

Step 1. Determine the encoding cost of the image block to be encoded in the original resolution encoding mode.

It should be understood that when determining the encoding cost of the original resolution encoding in step 1, the specific process in step 702 can be referred to, and the rate-distortion cost cost _org of the current image block can be determined as the encoding cost of the current image block in the original resolution encoding mode cost.

When determining the encoding cost of low-resolution encoding, it is necessary to determine the encoding cost when the CNN downsampling filter is used as the downsampling filter and the encoding cost when the FIR downsampling filter is used as the downsampling filter, as shown in step 2 below and step 3.

Step 2. Calculating the coding cost of the image block to be coded when the FIR downsampling filter is used for downsampling.

The specific process of calculating the encoding cost of the image block to be encoded when using the FIR downsampling filter for downsampling is as follows:

(1) Use the FIR downsampling filter to perform a downsampling operation on the current image block to obtain a low-resolution image block;

(2) performing an encoding operation on the low-resolution image block to output a compressed code stream;

(3) Obtaining low-resolution reconstruction image blocks;

(4) performing an FIR upsampling filter operation on the reconstruction of the low-resolution image block to obtain a reconstructed image block of the original resolution;

(5) Calculate the mean square error D _FIR between the reconstructed image block and the current image block, and determine the size R _FIR of the compressed code stream of the current image block;

(6) Calculate the rate-distortion cost cost _downFIR according to D _FIR and R _FIR , and determine the rate-distortion cost cost _downFIR as the encoding cost of the current image block when the FIR filter is used as the target upsampling filter.

Step 3. Calculate the encoding cost of the image block to be encoded when the CNN downsampling filter is used for downsampling.

The specific process of calculating the variable resolution encoding cost when using the CNN downsampling filter is as follows:

(1) Use the CNN downsampling filter to perform a downsampling operation on the current image block to obtain a low-resolution image block to be encoded;

(2) performing an encoding operation on the low-resolution image block to output a compressed code stream and a low-resolution reconstructed image block;

(3) Perform a CNN upsampling filter operation on the reconstruction of the low-resolution image block to obtain the reconstructed image block of the original resolution;

(4) Calculate the mean square error D _CNN between the reconstructed image block and the current image block, and determine the size R _CNN of the compressed code stream of the current image block;

(5) Calculate the rate-distortion cost cost _{downCNN according to DCNN and R-CNN , and} determine the rate-distortion cost cost _downCNN as the encoding cost of the current image block when the CNN filter is used as the target upsampling filter.

Secondly, select the mode with low encoding cost as the preferred mode to perform encoding operation on the current image block.

Specifically, in step 902, a coding mode may be selected according to the magnitude of the coding cost calculated in step 901, and coding information may be generated. The encoding information may specifically include a variable resolution encoding mode indication flag Flag1.

The above-mentioned setting of Flag1 according to the encoding cost specifically includes the following three situations.

(1) When cost _org <cost _downFIR and cost _org <cost _downCNN , it is determined to encode the current image block in the original resolution mode, and Flag1 is set to 0 at this time;

(2) When cost _downFIR <cost _org and cost _downFIR <cost _downCNN , it is determined that the current image block is encoded in variable resolution mode, and the FIR downsampling filter is used for downsampling operation, and Flag1 is set to 1 at this time;

(3) When cost _downCNN <cost _org and cost _downCNN <cost _downFIR , it is determined to encode the current image block in the variable resolution mode, and the CNN downsampling filter is used for downsampling operation, and Flag1 is set to 1 at this time.

Finally, the encoding information of the current image block is determined, and the encoding information is written into the code stream.

In step 903, besides the variable resolution encoding mode flag Flag1 determined in step 902, the encoding information may also include the prediction mode when encoding the current image block, the quantized encoding coefficient, and so on.

The decoding process of the decoding end in Embodiment 3 is shown in FIG. 17 . The decoding process shown in FIG. 17 corresponds to the encoding process shown in steps 901 to 903 above. FIG. 17 specifically includes the following steps.

1001. Acquire a compressed code stream.

1002. Analyze the variable resolution coding mode indicator Flag1 of the current image block.

The current image block may be the image block currently to be decoded, the compressed code stream obtained in step 1001 may be the code stream of the current image block, and after obtaining the code stream of the current image block, it may be parsed from the code stream of the current image block The variable resolution encoding mode indicates Flag1, and obtains the value of Flag1.

1003. Determine whether to adopt a variable resolution decoding mode according to Flag1.

The value of Flag1 can be used to indicate whether the variable resolution coding mode or the original resolution coding mode is adopted when generating the code stream of the current image block. Therefore, whether to decode the current image block in the variable resolution decoding mode can be determined according to the value of Flag1.

For example, when the value of Flag1 is 0, it means that the original resolution coding mode is used when generating the code stream of the current image block; when the value of Flag1 is 1, it means that the variable resolution coding is used when generating the code stream of the current image block model.

Therefore, when the value of Flag1 is 0, it can be determined that the current image block is decoded in the original resolution coding mode; when the value of Flag1 is 1, it can be determined that the current image block is decoded in the variable resolution coding mode.

1004. Analyze the coding information of the low-resolution coding block, and obtain a low-resolution reconstruction image block.

Since the encoding end can select different encoding schemes to encode image blocks, it is also necessary to select a decoding scheme corresponding to the encoding scheme for decoding when analyzing and decoding information to obtain reconstructed image blocks.

For example, if you choose a method similar to JPEG to encode an image block, you only need to parse the quantized transformation coefficient information of the image block from the code stream during decoding, and perform inverse quantization and inverse transformation operations on the coefficients to obtain the original resolution image block rebuild.

In addition, if an image block is encoded in a manner similar to H.264 or H.265 intra-frame encoding, then encoding information such as the encoding mode, prediction mode, and quantized transform coefficient of the current block needs to be decoded from the code stream during decoding. And after inverse quantization and inverse transformation of the quantized transform coefficients to obtain the reconstructed residual signal, it is necessary to perform a prediction operation according to the prediction mode to obtain the prediction signal, and finally add the two to obtain the reconstruction of the original resolution image block.

Specifically, as an optional implementation, obtaining low-resolution reconstruction blocks mainly includes the following processes:

First, entropy decoding is performed on the compressed code stream of the current image block to obtain quantized coefficients;

Secondly, dequantize and transform the quantization coefficients to obtain the reconstruction residual signal of the current image block;

Again, obtain the prediction signal of the current image block (or called the prediction image block);

Finally, according to the residual signal and the prediction signal of the current image block, a low-resolution reconstructed image block of the current image block is obtained.

1005. Use an FIR upsampling filter to upsample the low-resolution reconstruction block to obtain a reconstruction image block with the original resolution.

In step 1005, the original resolution refers to the resolution of the image block to be encoded corresponding to the reconstructed image block at the encoding end, and the reconstructed image block with the same resolution as the original image to be encoded can be obtained by performing upsampling.

1006. Analyze the coding information of the coding block with the original resolution, and obtain the reconstructed image block with the original resolution.

The process of obtaining the original resolution reconstructed image block in step 1006 is the same as the specific process of obtaining the low-resolution reconstructed image block in step 1005, except that the input signal in step 1005 is the code stream obtained in the low-resolution encoding mode, and the step The input signal in 1006 is the code stream acquired in the original resolution encoding mode.

1007. Determine whether the current image block is the last image block.

Specifically, it may be determined whether the current image block is the last image block to be decoded according to other encoding information carried in the code stream.

In the case that the current image block is not the last image block to be decoded, continue to execute step 1002 to continue decoding the next image block; in the case that the current image block is the last image block to be decoded, execute step 1008 , the decoding process of the current image ends.

1008. The decoding of the current image ends.

In the third embodiment, the CNN downsampling filter can be obtained through pre-training and pre-installed in the encoder. The encoder directly uses the preset CNN downsampling filters during encoding.

Specifically, the CNN downsampling filter parameter value may be a preset parameter value, and the preset parameter value may be obtained through offline training based on an image training set.

In the third embodiment, the offline training of the CNN downsampling filter includes the following specific steps:

Offline training of CNN downsampling filters can include two steps of initial parameter training and parameter fine-tuning.

Step 1. Train the CNN downsampling filter to obtain the initial parameters of the CNN downsampling network.

The original image in the training set is used as the input of the CNN downsampling filter, the output of the CNN downsampling filter is used as the input of the FIR upsampling filter, and the mean square error between the image output by the FIR upsampling filter and the original image is calculated As a loss function, the filter parameter with the smallest loss function is used as the initial parameter of the CNN downsampling network. Specifically, offline training can be performed according to formula (11) to obtain the initial parameters of the CNN downsampling network.

In formula (11), Indicates all parameters in the CNN downsampling network, f( _xi ; θ _f ) indicates that the downsampling network performs a downsampling operation on the input _xi , g _FIR indicates that the FIR filter is used for upsampling operation, Indicates the initial parameters of the CNN downsampling network.

Step 2. Fine-tune the initial parameters of the CNN downsampling network.

Use the original resolution image in the training set as the input input and train the parameters of the upsampling and downsampling networks. Specifically, CNN upsampling network and downsampling network can be trained simultaneously according to formula (12).

In formula (12), the loss function consists of two terms, the first term The calculation is the error between the original image of the image output by the CNN upsampling network, the second item The calculation is the error between the image output by the CNN downsampling network and the bicubic downsampling output image. According to the formula (12), the up-sampling network and down-sampling network are trained, and the optimal parameters of the CNN down-sampling network can be obtained The CNN downsampling network with the optimal parameters is the CNN downsampling filter used in the encoding process of this embodiment.

It should be understood that if you consider saving the computation of the encoder, you can skip step 2 and directly use the result obtained in step 1 Set the CNN downsampling filter.

In the first to third embodiments above, the parameters of CNN upsampling filter and CNN downsampling filter are obtained through offline training. In order to obtain better encoding effect, CNN can also be obtained through online training. Parameters for upsampling filter and CNN downsampling filter.

The process of obtaining the parameters of the CNN filter through online training will be described in detail below in conjunction with Embodiment 4 to Embodiment 6. It should be understood that the online training method in Embodiment 4 to Embodiment 6 can be used to update the parameters of the CNN upsampling filter and the CNN downsampling filter, that is to say, the online training method in Embodiment 4 to Embodiment 6 can be applied to In the first to sixth embodiments above, it is used to update the parameters of the CNN upsampling filter or the CNN downsampling filter.

Embodiment 4: Online training is performed on the CNN downsampling filter.

In Embodiment 4, the encoder can train CNN downsampling filter parameters online according to the image block to be encoded, and use the trained CNN downsampling filter to perform downsampling operation on the image block to be encoded. Apparently, when the CNN downsampling filter is used in the technical solution of the encoder to perform the downsampling operation on the image block to be coded, the solution in this embodiment can be used to update the parameters of the CNN downsampling filter.

In Embodiment 4, the online training process shown in FIG. 5 can be used to update the CNN downsampling network parameters. As shown in Figure 5, I _O is the current image to be encoded, I _L represents the low-resolution image to be encoded after downsampling the input image I _O using the CNN downsampling filter, and the encoder emulator simulates the image encoding process A CNN network of , which can simulate the image encoding process and output low-resolution reconstructed image I′ _L and encoding bit overhead. Among them, the CNN upsampling network is obtained through offline training, and is preset in the encoder and decoder for upsampling operations on low-resolution image block reconstruction. The CNN downsampling network structure can be as shown in Figure 11 above, and the CNN upsampling network structure can be as shown in Figure 13 above.

According to the formula (13) to train the CNN downsampling network, the optimal parameters of the CNN downsampling network can be obtained

In Equation (13), f represents the downsampling operation mapping function, θ _f represents the downsampling network parameters, h represents the encoding emulator mapping function output encoding reconstructed image, g represents the mapping function of the upsampling network, Represents the preset upsampling filter parameters, r represents the encoding bit overhead output by the encoding simulator, and λ is the weighting coefficient.

In Embodiment 4, the information loss caused by downsampling can be reduced to the greatest extent by online training of CNN downsampling filter parameters, and the quality of the reconstructed image can be improved. In addition, since the CNN downsampling filter is only used in the encoder, there is no need to pass it to the decoder without increasing the encoding overhead, making the overall rate-distortion of the scheme less.

Embodiment 5: Online training is performed on the CNN upsampling filter.

In Embodiment 5, the encoder can train CNN upsampling filter parameters online according to the image block to be encoded, and use the trained CNN upsampling filter to perform an upsampling operation on low-resolution image block reconstruction. It should be understood that, in this application, when the encoder uses a CNN upsampling filter to perform an upsampling operation on a low-resolution reconstructed image block, the solution in Embodiment 5 may be used to update its CNN upsampling filter parameters. Moreover, since the decoding end needs to use the same CNN upsampling filter to perform upsampling operations on low-resolution image block reconstruction, the encoding end needs to write the parameters of the updated CNN upsampling filter into the code stream, so that the decoding The terminal can also obtain the parameters of the CNN upsampling filter.

In Embodiment 5, the online training process shown in FIG. 5 can be used to update the CNN upsampling filter parameters. As shown in Figure 5, I _O is the current image to be encoded, I _L represents the low-resolution image to be encoded after downsampling the input image I _O using the CNN downsampling filter, and the encoder emulator simulates the image encoding process A CNN network of , which can simulate the image encoding process and output low-resolution reconstructed image I′ _L and encoding bit overhead. The CNN downsampling network structure can be as shown in Figure 11 above, and the CNN upsampling network structure can be as shown in Figure 13 above. Among them, the CNN downsampling network is obtained through offline training, and is preset in the encoder for downsampling the original image block to be encoded.

The CNN upsampling network is trained by formula (14), and the optimal parameters of the CNN upsampling network can be obtained

In formula (14), g represents the upsampling operation mapping function, θ _g represents the upsampling filter parameter, and g represents the mapping function of the upsampling filter.

In Embodiment 5, through online training of CNN upsampling filter parameters, matching parameters can be used for upsampling operations according to image texture features, and compared with preset CNN upsampling filter parameters, the quality of upsampled output image blocks can be further improved. In addition, the encoder can also comprehensively consider the image quality improvement and parameter transfer cost brought about by updating the upsampling filter parameters, for example, by using RDO to improve the overall rate-distortion performance of the encoding scheme of this embodiment.

It should be understood that the method for online training CNN upsampling filter parameters in the fifth embodiment above can also be applied to the first embodiment above. When the solution in the fifth embodiment is applied to the first embodiment, the CNN upsampling filter uses the preset In addition to the filter parameters, the filter parameters can also be obtained through online training. Specifically, it can be set to only use CNN upsampling filters obtained from online training. It can also be set as a coded image or a coded block to choose between the preset filter and the online training filter.

Since the decoding end needs to use the same CNN upsampling filter to perform upsampling operations on low-resolution image block reconstruction, the encoding end needs to pass the trained CNN upsampling filter parameters to the decoder. Specifically, the encoder can write the filter parameters into parameter sets such as SPS and PPS in the video compression code stream, or write them into a slice header. The number of filter parameters is determined by the CNN upsampling filter structure, including the weights and biases of all convolutional layers in the CNN upsampling filter. In order to save the cost of parameter transmission, one of the existing lossless data compression methods can be selected to compress and encode this group of parameters, for example, Differential Pulse Code Modulation (DPCM) encoding, Huffman Coding (Huffman Coding) ), arithmetic coding, etc.

When Embodiment 5 is applied to Embodiment 1, the decoding process of the decoder is basically the same as that of Embodiment 1. The difference is that the decoder also needs to parse the code stream to obtain CNN upsampling filter parameters. Refer to Figure 18 for the specific decoding process. As shown in Figure 18, the decoding process of the decoder specifically includes:

1101. Acquire a compressed code stream.

1102. Analyze and acquire CNN upsampling filter parameters.

The CNN upsampling filter parameters may be obtained by the encoder based on online training of the CNN upsampling filter.

1103. Analyze the variable resolution coding mode indicator Flag1 of the current image block.

The current image block may be the image block currently to be decoded, the compressed code stream obtained in step 1101 may be the code stream of the current image block, and after the code stream of the current image block is acquired, it may be parsed from the code stream of the current image block The variable resolution encoding mode indicates Flag1, and obtains the value of Flag1.

1104. Determine whether to adopt a variable resolution decoding mode according to Flag1.

The value of Flag1 can be used to indicate whether the variable resolution coding mode or the original resolution coding mode is adopted when generating the code stream of the current image block. Therefore, whether to decode the current image block in the variable resolution decoding mode can be determined according to the value of Flag1.

For example, when the value of Flag1 is 0, it means that the original resolution coding mode is used when generating the code stream of the current image block; when the value of Flag1 is 1, it means that the variable resolution coding is used when generating the code stream of the current image block model.

Therefore, when the value of Flag1 is 0, it can be determined that the current image block is decoded in the original resolution coding mode; when the value of Flag1 is 1, it can be determined that the current image block is decoded in the variable resolution coding mode.

1105. Analyze the upsampling filter selection flag Flag2 of the current image block.

The value of Flag2 can be used to indicate whether to use the FIR upsampling filter or the CNN upsampling filter as the upsampling filter for upsampling during the encoding process.

1106. Analyze the coding information of the low-resolution coding block, and obtain a low-resolution reconstruction image block.

Since the encoding end can select different encoding schemes to encode image blocks, it is also necessary to select a decoding scheme corresponding to the encoding scheme for decoding when analyzing and decoding information to obtain reconstructed image blocks.

1107. Analyze the coding information of the coding block with the original resolution, and obtain the reconstructed image block with the original resolution.

1108. Determine whether to use a CNN upsampling filter according to Flag2.

Specifically, it may be specifically determined to use the CNN upsampling filter or the FIR upsampling filter for upsampling according to the value of Flag2.

For example, when the value of Flag2 is 0, it means that the CNN upsampling filter is used for upsampling, and when the value of Flag2 is 1, it means that the FIR upsampling filter is used for upsampling. Therefore, when the value of Flag2 is 1, it is determined to use the CNN upsampling filter, and when the value of Flag2 is 0, it is determined not to use the CNN upsampling filter (the FIR upsampling filter is used).

1109. Use the CNN upsampling filter to upsample the low-resolution reconstruction block to obtain the original resolution reconstruction image block.

1110. Use an FIR upsampling filter to upsample the low-resolution reconstruction block to obtain a reconstruction image block with the original resolution.

In steps 1108 and 1109, the original resolution refers to the resolution of the image block to be encoded corresponding to the reconstructed image block at the encoding end, and the reconstructed image block with the same resolution as the original image to be encoded can be obtained through upsampling.

1111. Determine whether the current image block is the last image block.

Specifically, it may be determined whether the current image block is the last image block to be decoded according to other encoding information carried in the code stream.

When the current image block is not the last image block to be decoded, continue to execute step 1103 to continue decoding the next image block; when the current image block is the last image block to be decoded, execute step 1112 , the decoding process of the current image ends.

1112. The decoding of the current image ends.

Embodiment 6: Carry out joint online training on the CNN downsampling filter and the CNN upsampling filter.

Specifically, the CNN network online training process in FIG. 5 can be used to simultaneously update CNN upsampling and downsampling network parameters. As shown in Figure 5, I _O is the current image to be encoded, I _L represents the low-resolution image to be encoded after downsampling the input image I _O using the CNN downsampling filter, and the encoder emulator simulates the image encoding process A CNN network of , which can simulate the image encoding process and output low-resolution reconstructed image I′ _L and encoding bit overhead. The CNN downsampling network structure may be as shown in FIG. 11 above, and the CNN upsampling network structure may be as shown in FIG. 13 above.

The CNN downsampling filter and the CNN upsampling filter are jointly trained online. Specifically, the CNN upsampling network and the CNN downsampling network can be trained according to formula (15) to obtain the optimal parameters of the CNN upsampling network and the optimal parameters of the CNN downsampling network

In formula (15), f represents the downsampling operation mapping function, g represents the mapping function of the upsampling network, h represents the encoding emulator mapping function output code reconstruction image, θ _f represents the downsampling network parameters, θ _g represents the upsampling filter parameter, r represents the encoding bit overhead output by the encoding simulator, and λ is the weighting coefficient.

In Embodiment 6, the parameters of the CNN upsampling filter and the CNN downsampling filter are jointly trained online, and compared with the method of training filter parameters alone, the image to be encoded can be obtained more accurately The parameter values of the CNN upsampling filter and CNN downsampling filter that match the block can reduce the information loss caused by the upsampling and downsampling process of the image texture during encoding, and improve the quality of the encoded image.

The image encoding method and the image decoding method of the embodiment of the present application are described in detail above in conjunction with FIGS. , the image coding device shown in FIG. 19 and FIG. 20 can execute each step of the image coding method of the embodiment of the application shown in FIG. 2 to FIG. 18, and the image decoding device shown in FIG. 21 can execute the steps in FIG. Each step of the image decoding method in the embodiment of the present application shown in . For the sake of brevity, repeated descriptions are appropriately omitted below.

In addition, the image encoding device and the image decoding device shown in Fig. 19 to Fig. 21 may specifically be a smart terminal, a PAD, a video player, an Internet of Things device or other devices having a video image encoding or decoding function.

Fig. 19 is a schematic block diagram of an image encoding device according to an embodiment of the present application. The image encoding device 1200 shown in FIG. 19 specifically includes:

A determining module 1201, configured to determine a target upsampling filter from a preset upsampling filter set according to the encoding cost of the image block to be encoded, wherein the upsampling filter set includes at least a finite impulse response FIR upsampling filter filter and convolutional neural network CNN upsampling filter;

A processing module 1202, the processing module 1202 is specifically configured to: generate upsampling filter indication information corresponding to the target upsampling filter; use a preset FIR downsampling filter to downsample the image block to be encoded , to obtain a first image block; encode the first image block to obtain a code stream; write the upsampling filter indication information into the code stream.

In this application, according to the coding cost of the image block, one filter can be selected from various candidate filters as the up filter, which can achieve better coding effect compared with the way of using a fixed filter as the up sampling filter .

Fig. 20 is a schematic block diagram of an image encoding device according to an embodiment of the present application. The image encoding device 1300 shown in FIG. 20 specifically includes:

An acquisition module 1301, configured to acquire a code stream;

A processing module 1302, the processing module is specifically configured to: perform entropy decoding, inverse quantization, and inverse transformation on the code stream to obtain a reconstructed residual signal of the image block to be decoded; obtain a prediction signal of the image block to be decoded; Adding the reconstruction residual signal and the prediction signal to obtain the initial reconstructed image block of the image block to be decoded; parsing the code stream to obtain coding mode indication information; according to the coding mode indication information from the original resolution Determining the target decoding mode in the decoding mode and the variable resolution decoding mode; in the case where the target decoding mode is the variable resolution decoding mode, parsing the code stream of the image block to be encoded, and obtaining the upsampling filter indication information; according to The upsampling filter indication information determines a target upsampling filter from a preset upsampling filter set, wherein the upsampling filter set includes at least a finite impulse response FIR upsampling filter and a convolutional neural network CNN upsampling filter;

In this application, the target upsampling filter can be determined from the set of upsampling filters according to the encoding cost. Compared with directly using the upsampling filter with fixed parameter values for the upsampling operation, this application scheme selects the target upsampling filter The encoding cost of the image block to be encoded is fully considered when the encoder is used, and better encoding effect can be achieved.

Fig. 21 is a schematic block diagram of an image decoding device according to an embodiment of the present application. The image encoding device 1400 shown in FIG. 21 specifically includes:

An acquisition module 1401, configured to acquire a code stream;

The processing module 1402 is specifically configured to: perform entropy decoding, inverse quantization, and inverse transformation on the code stream to obtain a reconstructed residual signal of the image block to be decoded; obtain a prediction signal of the image block to be decoded; Adding the reconstruction residual signal and the prediction signal to obtain the initial reconstructed image block of the image block to be decoded; parsing the code stream to obtain coding mode indication information; according to the coding mode indication information from the original resolution Determining the target decoding mode in the decoding mode and the variable resolution decoding mode; in the case where the target decoding mode is the variable resolution decoding mode, parsing the code stream of the image block to be encoded, and obtaining the upsampling filter indication information; according to The upsampling filter indication information determines a target upsampling filter from a preset upsampling filter set, wherein the upsampling filter set includes at least a finite impulse response FIR upsampling filter and a convolutional neural network A CNN upsampling filter; using the target upsampling filter to upsample the initial reconstructed image block to obtain a target reconstructed image block.

In this application, when decoding an image block in the variable resolution decoding mode, the decoding end can select the corresponding upsampling filter from the preset upsampling filter set as the target upsampling filter according to the upsampling filter indication information Filter, and then perform upsampling operation. Compared with the way of directly using fixed parameter upsampling filter for upsampling operation, the matching upsampling filter can be selected according to the situation of the image block for upsampling operation, thereby improving the decoding effect .

The specific implementation form of the above image encoding device 1200, image encoding device 1300 and image decoding device 1400 may be any of the following devices: desktop computer, mobile computing device, notebook (for example, laptop) computer, tablet computer, set-top box , smartphones, handhelds, televisions, cameras, display devices, digital media players, video game consoles, vehicle computers, or other similar devices.

Fig. 22 is a schematic block diagram of an encoder according to an embodiment of the present application. The encoder 2000 shown in FIG. 22 includes: an encoding end prediction module 2001 , a transform and quantization module 2002 , an entropy encoding module 2003 , an encoding reconstruction module 2004 and an encoding end filtering module.

The specific structures of the above-mentioned image coding device 1200 and the image coding device 1300 can be shown as the above-mentioned coder 2000, and the coder 2000 can execute various steps of the image coding method in the embodiment of the present application.

Fig. 23 is a schematic block diagram of a decoder according to an embodiment of the present application. The decoder 3000 shown in FIG. 23 includes: an entropy decoding module 3001 , an inverse transform and inverse quantization module 3002 , a decoding end prediction module 3003 , a decoding reconstruction module 3004 and a decoding end filtering module 3005 .

The specific structure of the above-mentioned image decoding apparatus 1400 can be shown as the above-mentioned decoder 3000, and the decoder 3000 can execute various steps of the image decoding method in the embodiment of the present application.

Fig. 24 is a schematic block diagram of a codec device according to an embodiment of the present application. The codec device 50 may be a device dedicated to encoding and/or decoding video images, or an electronic device with a video codec function, and further, the codec device 50 may be a mobile terminal of a wireless communication system or user equipment.

It should be understood that the codec device shown in FIG. 24 can be regarded as the specific structure of the above-mentioned image coding device 1200 , image coding device 1300 and image decoding device 1400 . FIG. 24 can execute various steps of the image encoding and decoding method of the embodiment of the present application.

Codec device 50 may include the following modules or units: controller 56 , codec 54 , radio interface 52 , antenna 44 , smart card 46 , card reader 48 , memory 58 , infrared port 42 , display 32 . In addition to the modules and units shown in Figure 24, the codec device 50 may also include a microphone or any suitable audio input module, which may be a digital or analog signal input, and the codec device 50 may also include an audio output module, the audio output module can be a headphone, a speaker, or an analog audio or digital audio output connection. The codec device 50 may also include a battery, which may be a solar cell, a fuel cell, or the like. The codec device 50 may also include an infrared port for short-range line-of-sight communication with other devices, and the codec device 50 may also communicate with other devices using any suitable short-range communication methods, such as Bluetooth wireless connection, USB /FireWire wired connection.

The memory 58 may store data in the form of images and audio, as well as instructions for execution on the controller 56 .

The codec 54 can implement encoding and decoding of audio and/or video data or implement auxiliary encoding and auxiliary decoding of audio and/or video data under the control of the controller 56 .

Smart card 46 and card reader 48 can provide user information, and can also provide authentication information for network authentication and authorized users. The specific implementation forms of the smart card 46 and the card reader 48 may be an integrated circuit card (Universal Integrated Circuit Card, UICC) and a UICC reader.

The radio interface circuit 52 can generate wireless communication signals, which can be communication signals generated during cellular communication network, wireless communication system or wireless local area network communication.

Antenna 44 is used to send the radio frequency signal generated in radio interface circuit 52 to other devices (the number of devices can be one or more), and can also be used to receive signals from other devices (the number of devices can be one or more). ) to receive radio frequency signals.

In some embodiments of the present application, the codec device 50 may receive video image data to be processed from another device before transmission and/or storage. In some other embodiments of the present application, the codec device 50 may receive images through a wireless or wired connection and encode/decode the received images.

It should be understood that the processing object of the image encoding and decoding method in the embodiment of the present application may be a video image, that is, the image encoding and decoding method of the present application may perform encoding and decoding on a video image. Therefore, the video codec system can also execute the image codec method of the embodiment of the present application. The video codec system of the embodiment of the present application will be described in detail below with reference to FIG. 17 .

Fig. 25 is a schematic block diagram of a video encoding and decoding system according to an embodiment of the present application.

As shown in FIG. 25 , the video codec system 7000 includes a source device 4000 and a destination device 5000 . The source device 4000 generates encoded video data. The source device 4000 may also be called a video encoding device or a video encoding device. The destination device 5000 can decode the encoded video data generated by the source device 4000. The destination device 5000 may also be called a video decoding device or a video decoding device.

The above source device 4000 is equivalent to the above image encoding device 1200 and the above image encoding device 1300 , and the destination device 5000 is equivalent to the above image decoding device 1400 .

Specific implementations of source device 4000 and destination device 5000 may be any of the following devices: desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, smartphones, handsets, Television, camera, display device, digital media player, video game console, vehicle computer, or other similar devices.

The destination device 5000 may receive encoded video data from the source device 4000 via the channel 6000 . Channel 6000 may include one or more media and/or devices capable of moving encoded video data from source device 4000 to destination device 5000 . In one example, channel 6000 may include one or more communication media that enable source device 4000 to transmit encoded video data directly to destination device 5000 in real time, in which case source device 4000 may transmit encoded video data according to a communication standard ( For example, a wireless communication protocol) to modulate the encoded video data, and the modulated video data may be transmitted to the destination device 5000. The aforementioned one or more communication media may include wireless and/or wired communication media, such as radio frequency (Radio Frequency, RF) spectrum or one or more physical transmission lines. One or more of the communication media described above may form part of a packet-based network such as a local area network, a wide area network, or a global network such as the Internet. The one or more communication media described above may include routers, switches, base stations, or other devices that enable communication from the source device 4000 to the destination device 5000 .

In another example, channel 6000 may include a storage medium that stores encoded video data generated by source device 4000 . In this example, destination device 5000 may access the storage medium via disk access or card access. The storage medium may include a variety of local access data storage media, such as Blu-ray discs, high-density digital video discs (Digital Video Disc, DVD), read-only discs (Compact Disc Read-Only Memory, CD-ROM), flash memory, or other suitable digital storage media for storing encoded video data.

In another example, channel 6000 may include a file server or another intermediate storage device that stores encoded video data generated by source device 4000 . In this example, destination device 5000 may access encoded video data stored at a file server or other intermediate storage device via streaming or download. The file server may be a type of server capable of storing encoded video data and transmitting the encoded video data to the destination device 5000 . For example, a file server may include a World Wide Web (Web) server (e.g., for a website), a File Transfer Protocol (File Transfer Protocol, FTP) server, a Network Attached Storage (NAS) device, and a local disk drive .

Destination device 5000 may access the encoded video data via a standard data connection (eg, an Internet connection). Example types of data connections include wireless channels suitable for accessing encoded video data stored on a file server, wired connections (eg, cable modem, etc.), or a combination of both. The transmission of encoded video data from the file server can be a streaming transmission, a download transmission, or a combination of both.

The image encoding and decoding method of the embodiment of the present application is not limited to the wireless application scenario. Exemplarily, the image encoding and decoding method of the embodiment of the present application can be applied to the video encoding and decoding of various multimedia applications such as the following applications: air TV broadcasting, cable TV transmission, satellite television transmission, streaming video transmission (eg, via the Internet), encoding of video data stored on a data storage medium, decoding of video data stored on a data storage medium, or other applications. In some examples, video codec system 7000 can be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and/or video telephony.

In FIG. 25 , a source device 4000 includes a video source 4001 , a video encoder 4002 and an output interface 4003 . In some examples, output interface 4003 may include a modulator/demodulator (modem) and/or a transmitter. Video source 4001 may include a video capture device (e.g., a video camera), a video archive containing previously captured video data, a video input interface for receiving video data from a video content provider, and/or a computer for generating video data Graphics system, or a combination of the above video data sources.

Video encoder 4002 may encode video data from video source 4001 . In some examples, source device 4000 transmits the encoded video data directly to destination device 5000 via output interface 4003 . The encoded video data may also be stored on a storage medium or file server for later access by the destination device 5000 for decoding and/or playback.

In the example of FIG. 25 , destination device 5000 includes input interface 5003 , video decoder 5002 and display device 5001 . In some examples, input interface 5003 includes a receiver and/or a modem. The input interface 5003 can receive encoded video data via the channel 6000 . The display device 5001 may be integrated with the destination device 5000 or may be external to the destination device 5000 . In general, the display device 5001 displays decoded video data. The display device 5001 may include various display devices, such as a liquid crystal display, a plasma display, an organic light emitting diode display, or other types of display devices.

The video encoder 4002 and the video decoder 5002 may operate according to a video compression standard (for example, the High Efficiency Video Coding (H.265) standard), and may comply with the High Efficiency Video Coding (HEVC) test model (HM ). The text description of the H.265 standard ITU-TH.265(V3)(04/2015) was released on April 29, 2015 and can be downloaded from http://handle.itu.int/11.1002/7000/12455, the The entire content of the document is incorporated herein by reference.

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other various media that can store program codes. .

The above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto. Anyone familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the application. Should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (42)

1. An image coding method, characterized in that, comprising: Determine the target upsampling filter from the preset upsampling filter set according to the encoding cost of the image block to be encoded, wherein the upsampling filter set includes at least a finite impulse response FIR upsampling filter and a convolutional neural network CNN upsampling filter; generating upsampling filter indication information corresponding to the target upsampling filter; Using a preset FIR downsampling filter to downsample the image block to be encoded to obtain a first image block; Encoding the first image block to obtain a code stream; Writing the upsampling filter indication information into the code stream. 2. The method according to claim 1, wherein said determining a target upsampling filter from a preset set of upsampling filters according to the encoding cost of the image block to be encoded comprises: determining the encoding cost of the image block to be encoded when each type of upsampling filter in the set of upsampling filters is used as the target upsampling filter; determining a first upsampling filter in the set of upsampling filters as the target upsampling filter, wherein, in the set of upsampling filters, the first upsampling filter serves as the target When the upsampling filter is used, the encoding cost of the image block to be encoded is the smallest. 3. the method as claimed in claim 1 or 2, is characterized in that, the parameter value of described CNN upsampling filter is preset, and the parameter value of described CNN upsampling filter is to preset image training set obtained from offline training. 4. The method according to any one of claims 1-3, wherein before the target upsampling filter is determined from a preset set of upsampling filters according to the encoding cost of the image block to be encoded, the The method also includes: Perform online training on the CNN upsampling filter according to the image block to be encoded to obtain an update parameter value of the CNN upsampling filter, and the update parameter value of the CNN upsampling filter is used to replace the CNN Sample filter preset parameter values. 5. The method of claim 4, further comprising: Write the updated parameter value of the CNN upsampling filter into the code stream. 6. The method according to any one of claims 1-5, wherein before the target upsampling filter is determined from the preset upsampling filter set according to the encoding cost of the image block to be encoded, the The method also includes: determining a target coding mode from the original resolution coding mode and the variable resolution coding mode according to the coding cost of the image block to be coded; generating coding mode indication information corresponding to the target coding mode; Writing the coding mode indication information into a code stream. 7. An image coding method, characterized in that, comprising: Determine the target upsampling filter from the preset upsampling filter set according to the encoding cost of the image block to be encoded, wherein the upsampling filter set includes at least a finite impulse response FIR upsampling filter and a convolutional neural network CNN upsampling filter; generating upsampling filter indication information corresponding to the target upsampling filter; Determining a downsampling filter of the same type as the target upsampling filter in the preset downsampling filter set as the target downsampling filter, wherein the downsampling filter set includes at least finite impulse response FIR downsampling Filters and CNN downsampling filters; Using the target downsampling filter to downsample the image block to be encoded to obtain a first image block; Encoding the first image block to obtain a code stream; Writing the upsampling filter indication information into the code stream. 8. The method according to claim 7, wherein said determining a target upsampling filter from a preset set of upsampling filters according to the encoding cost of the image block to be encoded comprises: determining the encoding cost of the image block to be encoded when each type of upsampling filter in the set of upsampling filters is used as the target upsampling filter; determining a first upsampling filter in the set of upsampling filters as the target upsampling filter, wherein, in the set of upsampling filters, the first upsampling filter in the set of upsampling filters When the sampling filter is used as the target up-sampling filter, the encoding cost of the image block to be encoded is the smallest. 9. the method as claimed in claim 7 or 8, is characterized in that, the parameter value of described CNN upsampling filter is preset, and the parameter value of described CNN upsampling filter is to preset image training set obtained from offline training. 10. The method according to any one of claims 7-9, wherein the parameter value of the CNN downsampling filter is preset, and the parameter value of the CNN downsampling filter is preset The image training set is obtained by offline training. 11. The method according to claim 7 or 8, characterized in that, the parameter values of the CNN upsampling filter and the CNN downsampling filter are preset, and the CNN upsampling filter The parameter value of the filter and the parameter value of the CNN downsampling filter are obtained through joint training on a preset image training set in an offline situation. 12. The method according to any one of claims 7-11, wherein before using the target downsampling filter to downsample the image block to be encoded, the method further comprises: Perform online training on the CNN downsampling filter according to the image block to be encoded to obtain an update parameter value of the CNN downsampling filter, and the update parameter value of the CNN downsampling filter is used to replace the CNN downsampling filter Sample filter preset parameter values. 13. The method according to any one of claims 7-12, wherein before the target upsampling filter is determined from a preset set of upsampling filters according to the encoding cost of the image block to be encoded, the The method also includes: Perform online training on the CNN upsampling filter according to the image block to be encoded to obtain an update parameter value of the CNN upsampling filter, and the update parameter value of the CNN upsampling filter is used to replace the CNN Sample filter preset parameter values. 14. The method according to any one of claims 7-11, wherein before using the target downsampling filter to downsample the image block to be encoded, the method further comprises: Carry out joint online training on the CNN downsampling filter and the CNN upsampling filter according to the image block to be encoded, to obtain an update parameter value of the CNN downsampling filter and an update of the CNN upsampling filter parameter value; Wherein, the update parameter value of the CNN downsampling filter is used to replace the preset parameter value of the CNN downsampling filter, and the update parameter value of the CNN upsampling filter is used to replace the CNN upsampling filter Preset parameter values. 15. The method of claim 13 or 14, further comprising: Write the updated parameter value of the CNN upsampling filter into the code stream. 16. The method according to any one of claims 7-15, wherein before the target upsampling filter is determined from a preset set of upsampling filters according to the encoding cost of the image block to be encoded, the The method also includes: determining a target coding mode from the original resolution coding mode and the variable resolution coding mode according to the coding cost of the image block to be coded; generating coding mode indication information corresponding to the target coding mode; Writing the coding mode indication information into a code stream. 17. An image decoding method, characterized in that, comprising: Get code stream; performing entropy decoding, inverse quantization and inverse transformation on the code stream to obtain a reconstructed residual signal of the image block to be decoded; Acquiring a prediction signal of the image block to be decoded; adding the reconstruction residual signal and the prediction signal to obtain an initial reconstructed image block of the image block to be decoded; Analyzing the code stream to obtain coding mode indication information; determining a target decoding mode from an original resolution decoding mode and a variable resolution decoding mode according to the encoding mode indication information; If the target decoding mode is a variable resolution decoding mode, analyze the code stream of the image block to be encoded, and acquire upsampling filter indication information; Determine the target upsampling filter from the preset upsampling filter set according to the upsampling filter indication information, wherein the upsampling filter set includes at least a finite impulse response FIR upsampling filter and a convolution neural network Network CNN upsampling filter; Upsampling the initial reconstructed image block by using the target upsampling filter to obtain a target reconstructed image block. 18. The method according to claim 17, wherein the parameter value of the CNN upsampling filter is preset, and the parameter value of the CNN upsampling filter is to carry out offline training to a preset image training set owned. 19. The method of claim 17 or 18, further comprising: Parsing the code stream to obtain an updated parameter value of the CNN upsampling filter, where the updated parameter value of the CNN upsampling filter is used to replace the preset parameter value of the CNN upsampling filter. 20. The method according to claim 19, wherein, before parsing the code stream and obtaining the update parameter value of the CNN upsampling filter, the method further comprises: Parsing the code stream to obtain filter parameter update indication information, where the filter parameter update indication information is used to indicate whether to update the parameter value of the target upsampling filter; The parsing of the code stream to obtain the update parameter value of the CNN upsampling filter includes: If the filter parameter update instruction information indicates to update the parameters of the target upsampling filter, parse the code stream to acquire the update parameter value of the CNN upsampling filter. 21. The method according to claim 19 or 20, wherein the update parameter value of the CNN upsampling filter is obtained by performing online training on the CNN upsampling network according to the image block to be encoded, wherein the to-be-coded The decoded image block is obtained by encoding the image block to be encoded. 22. An image encoding device, comprising: A determination module, configured to determine a target upsampling filter from a preset set of upsampling filters according to the encoding cost of the image block to be encoded, wherein the set of upsampling filters at least includes a finite impulse response FIR upsampling filter and convolutional neural network CNN upsampling filter; A processing module, the processing module is specifically used for: generating upsampling filter indication information corresponding to the target upsampling filter; Using a preset FIR downsampling filter to downsample the image block to be encoded to obtain a first image block; Encoding the first image block to obtain a code stream; Writing the upsampling filter indication information into the code stream. 23. The device according to claim 22, wherein the determining module is specifically configured to: determining the encoding cost of the image block to be encoded when each type of upsampling filter in the set of upsampling filters is used as the target upsampling filter; determining a first upsampling filter in the set of upsampling filters as the target upsampling filter, wherein, in the set of upsampling filters, the first upsampling filter serves as the target When the upsampling filter is used, the encoding cost of the image block to be encoded is the smallest. 24. The device according to claim 22 or 23, wherein the parameter value of the CNN upsampling filter is preset, and the parameter value of the CNN upsampling filter is performed on a preset image training set obtained from offline training. 25. The device according to any one of claims 22-24, wherein, in the determination module, the target upsampling filter is determined from the preset upsampling filter set according to the encoding cost of the image block to be encoded. Before the device, the determination module is also used to: Perform online training on the CNN upsampling filter according to the image block to be encoded to obtain an update parameter value of the CNN upsampling filter, and the update parameter value of the CNN upsampling filter is used to replace the CNN Sample filter preset parameter values. 26. The device according to claim 25, wherein the processing module is further used for: Write the updated parameter value of the CNN upsampling filter into the code stream. 27. The device according to any one of claims 22-26, wherein, in the determination module, the target upsampling filter is determined from the preset upsampling filter set according to the encoding cost of the image block to be encoded. Before the device, the determination module is also used to: determining a target coding mode from the original resolution coding mode and the variable resolution coding mode according to the coding cost of the image block to be coded; generating coding mode indication information corresponding to the target coding mode; Writing the coding mode indication information into a code stream. 28. An image encoding device, comprising: A determination module, configured to determine a target upsampling filter from a preset set of upsampling filters according to the encoding cost of the image block to be encoded, wherein the set of upsampling filters at least includes a finite impulse response FIR upsampling filter and convolutional neural network CNN upsampling filter; A processing module, the processing module is specifically used for: generating upsampling filter indication information corresponding to the target upsampling filter; Determining a downsampling filter of the same type as the target upsampling filter in the preset downsampling filter set as the target downsampling filter, wherein the downsampling filter set includes at least finite impulse response FIR downsampling Filters and CNN downsampling filters; Using the target downsampling filter to downsample the image block to be encoded to obtain a first image block; Encoding the first image block to obtain a code stream; Writing the upsampling filter indication information into the code stream. 29. The device according to claim 28, wherein the determining module is specifically configured to: determining the encoding cost of the image block to be encoded when each type of upsampling filter in the set of upsampling filters is used as the target upsampling filter; determining a first upsampling filter in the set of upsampling filters as the target upsampling filter, wherein, in the set of upsampling filters, the first upsampling filter in the set of upsampling filters When the sampling filter is used as the target up-sampling filter, the encoding cost of the image block to be encoded is the smallest. 30. The device according to claim 28 or 29, wherein the parameter value of the CNN upsampling filter is preset, and the parameter value of the CNN upsampling filter is performed on a preset image training set obtained from offline training. 31. The device according to any one of claims 28-30, wherein the parameter value of the CNN downsampling filter is preset, and the parameter value of the CNN downsampling filter is preset The image training set is obtained by offline training. 32. The device according to claim 28 or 29, wherein the parameter values of the CNN upsampling filter and the CNN downsampling filter are preset, and the CNN upsampling filter The parameter value of the filter and the parameter value of the CNN downsampling filter are obtained through joint training on a preset image training set in an offline situation. 33. The device according to any one of claims 28-32, wherein before the processing module uses the target downsampling filter to downsample the image block to be encoded, the processing module Also used for: Perform online training on the CNN downsampling filter according to the image block to be encoded to obtain an update parameter value of the CNN downsampling filter, and the update parameter value of the CNN downsampling filter is used to replace the CNN downsampling filter Sample filter preset parameter values. 34. The device according to any one of claims 28-33, wherein, in the determination module, the target upsampling filter is determined from the preset upsampling filter set according to the encoding cost of the image block to be encoded. Before the device, the determination module is also used to: Perform online training on the CNN upsampling filter according to the image block to be encoded to obtain an update parameter value of the CNN upsampling filter, and the update parameter value of the CNN upsampling filter is used to replace the CNN Sample filter preset parameter values. 35. The device according to any one of claims 28-32, wherein, before the processing module uses the target downsampling filter to downsample the image block to be encoded, the processing module Also used for: Carry out joint online training on the CNN downsampling filter and the CNN upsampling filter according to the image block to be encoded, to obtain an update parameter value of the CNN downsampling filter and an update of the CNN upsampling filter parameter value; Wherein, the update parameter value of the CNN downsampling filter is used to replace the preset parameter value of the CNN downsampling filter, and the update parameter value of the CNN upsampling filter is used to replace the CNN upsampling filter Preset parameter values. 36. The device according to claim 34 or 35, wherein the processing module is further used for: Write the updated parameter value of the CNN upsampling filter into the code stream. 37. The device according to any one of claims 28-36, wherein, in the determination module, the target upsampling filter is determined from the preset upsampling filter set according to the encoding cost of the image block to be encoded. Before the device, the determination module is also used to: determining a target coding mode from the original resolution coding mode and the variable resolution coding mode according to the coding cost of the image block to be coded; generating coding mode indication information corresponding to the target coding mode; Writing the coding mode indication information into a code stream. 38. An image decoding device, comprising: The acquisition module is used to acquire the code stream; A processing module, the processing module is specifically used for: performing entropy decoding, inverse quantization and inverse transformation on the code stream to obtain a reconstructed residual signal of the image block to be decoded; Acquiring a prediction signal of the image block to be decoded; adding the reconstruction residual signal and the prediction signal to obtain an initial reconstructed image block of the image block to be decoded; Analyzing the code stream to obtain coding mode indication information; determining a target decoding mode from an original resolution decoding mode and a variable resolution decoding mode according to the encoding mode indication information; If the target decoding mode is a variable resolution decoding mode, analyze the code stream of the image block to be encoded, and acquire upsampling filter indication information; Determine the target upsampling filter from the preset upsampling filter set according to the upsampling filter indication information, wherein the upsampling filter set includes at least a finite impulse response FIR upsampling filter and a convolution neural network Network CNN upsampling filter; Upsampling the initial reconstructed image block by using the target upsampling filter to obtain a target reconstructed image block. 39. The device according to claim 38, wherein the parameter value of the CNN upsampling filter is preset, and the parameter value of the CNN upsampling filter is to carry out offline training to a preset image training set owned. 40. The device according to claim 38 or 39, wherein the processing module is further used for: Parsing the code stream to obtain an updated parameter value of the CNN upsampling filter, where the updated parameter value of the CNN upsampling filter is used to replace the preset parameter value of the CNN upsampling filter. 41. The device according to claim 40, wherein the processing module is further configured to: Parsing the code stream to obtain filter parameter update indication information, where the filter parameter update indication information is used to indicate whether to update the parameter value of the target upsampling filter; The parsing of the code stream to obtain the update parameter value of the CNN upsampling filter includes: If the filter parameter update instruction information indicates to update the parameters of the target upsampling filter, parse the code stream to acquire the update parameter value of the CNN upsampling filter. 42. The device according to claim 40 or 41, wherein the update parameter value of the CNN upsampling filter is obtained by performing online training on the CNN upsampling network according to the image block to be encoded, wherein the to-be-coded The decoded image block is obtained by encoding the image block to be encoded.