CN114298922A - Image processing method, device and electronic device - Google Patents

Abstract

Embodiments of the present application provide an image processing method, apparatus, and electronic device. The method includes: first, acquiring a reconstructed image, and then performing image enhancement on the reconstructed image to obtain an intermediate image; then, determining the texture complexity weights corresponding to each region in the intermediate image, and the texture complexity weights are between 0 and 1. Then, according to the texture complexity weight corresponding to each area in the intermediate image, the texture intensity corresponding to each area in the intermediate image is respectively attenuated to obtain an enhanced image corresponding to the reconstructed image. In this way, the texture of the textured area in the intermediate image can be preserved, and the texture in the non-textured area can be attenuated at the same time, so that the texture of the textured area in the reconstructed image can be enhanced, and the false texture of the non-textured area can be avoided, thereby reducing the reconstructed image. Visual distortion, increase the realism of the image, and improve the subjective quality of the image.

Description

Image processing method, device and electronic device

technical field

The embodiments of the present application relate to the technical field of image processing, and in particular, to an image processing method, an apparatus, and an electronic device.

Background technique

Generally, before video is transmitted, the video is compressed to improve the efficiency of video transmission; wherein, the larger the video compression rate is, the smaller the data volume of the compressed video is. However, with the increase of the compression rate, visible visual distortion will appear in the video; in order to reduce the visual distortion of the reconstructed image, the reconstructed image can be post-processed to improve the quality when the bit rate remains unchanged.

Although post-processing of the reconstructed image can significantly improve the subjective quality of the reconstructed image, false textures are easily generated in the non-textured area, which affects the subjective quality of the image.

SUMMARY OF THE INVENTION

In order to solve the above technical problems, the present application provides an image processing method, apparatus and electronic device.

In a first aspect, an embodiment of the present application provides an image processing method. The method includes: first, acquiring a reconstructed image; then, performing image enhancement on the reconstructed image to obtain an intermediate image. Next, determine the texture complexity weight corresponding to each area in the intermediate image, and the texture complexity weight is a number between 0 and 1; then, according to the texture complexity weight corresponding to each area in the intermediate image, each area in the intermediate image The texture intensity corresponding to the region is respectively attenuated to obtain the enhanced image corresponding to the reconstructed image. In this way, the texture of the textured area in the intermediate image can be preserved, and the texture in the non-textured area can be attenuated at the same time, so that the texture of the textured area in the reconstructed image can be enhanced, and the false texture of the non-textured area can be avoided, thereby reducing the reconstructed image. Visual distortion, increase the realism of the image, and improve the subjective quality of the image.

In addition, the present application controls the texture complexity weight in the range of 0 to 1, so that the texture corresponding to the adjacent areas is softer, and the problem of inconsistent texture effects enhanced by adjacent areas can be avoided.

Exemplarily, the texture complexity weight is proportional to the texture complexity, that is, the greater the texture complexity, the greater the texture complexity weight. In this way, the texture effect of the textured area in the intermediate image can be maintained as much as possible, while the non-textured area is attenuated according to the texture complexity.

Exemplarily, the intermediate image is a texture-enhanced image or a residual image.

Exemplarily, the resolution of the intermediate image and the enhanced image are the same as the resolution of the reconstructed image.

Exemplarily, a GANEF (Generative Adversarial Network Enhancement Filter, an enhancement filter of a generative adversarial network) can be used to perform image enhancement on the reconstructed image to obtain an intermediate image.

Exemplarily, the image output by GANEF can be a texture-enhanced image or a residual image.

It should be noted that although the enhanced image is the attenuation of texture enhancement for each area in the intermediate image, only part of the texture intensity is attenuated for the intermediate image. Compared with the reconstructed image, the texture intensity of some areas of the enhanced image is still larger than the reconstructed image; that is, the enhanced image is still texture-enhanced.

According to the first aspect, the intermediate image is a residual image; according to the texture complexity weight corresponding to each area in the intermediate image, the texture intensity corresponding to each area in the intermediate image is respectively attenuated to obtain an enhanced image corresponding to the reconstructed image, including: The pixel value of each pixel in the residual image is multiplied by the texture complexity weight corresponding to the region to which each pixel belongs to obtain the residual updated image; the enhanced image is generated according to the residual updated image and the reconstructed image.

Exemplarily, the residual image is obtained by subtracting the texture-enhanced image and the reconstructed image.

According to the first aspect, or any implementation manner of the above first aspect, generating an enhanced image according to the residual updated image and the reconstructed image includes: adding the residual updated image and the reconstructed image to obtain the enhanced image.

Exemplarily, the residual update image and the reconstructed image may be added pixel by pixel to obtain an enhanced image.

According to the first aspect, or any implementation manner of the above first aspect, the intermediate image is a texture-enhanced image, and the method further includes: performing image fidelity on the reconstructed image to obtain a basic fidelity image.

Exemplarily, the base fidelity image is the same resolution as the reconstructed image.

Exemplarily, a non-GANEF can be used to perform image fidelity on the reconstructed image to obtain a basic fidelity image.

According to the first aspect, or any implementation manner of the above first aspect, determining the texture complexity weight corresponding to each region in the intermediate image includes: dividing the basic fidelity image and the texture-enhanced image into two groups according to preset partitioning rules. N regions, where N is a positive integer; determine the texture complexity of the N regions in the basic fidelity image respectively; determine the texture corresponding to the N regions in the texture enhanced image based on the texture complexity of the N regions in the basic fidelity image complexity weight.

According to the first aspect, or any one of the implementation manners of the first aspect above, according to the texture complexity weights corresponding to the respective regions in the intermediate image, the texture intensities corresponding to the regions in the intermediate image are respectively attenuated to obtain the corresponding texture intensity of the reconstructed image. Enhancing the image, including: according to the texture complexity weights corresponding to the N areas in the texture-enhanced image, determining the weighted calculation weights corresponding to the N areas in the texture-enhanced image and the weighted calculation weights corresponding to the N areas in the basic fidelity image respectively; According to the weighted calculation weights corresponding to the N regions in the texture-enhanced image and the weighted calculation weights corresponding to the N regions in the basic fidelity image, the weighted calculation is performed on the N regions in the texture-enhanced image and the N regions in the basic fidelity image. , to obtain the enhanced image corresponding to the reconstructed image.

According to the first aspect, or any implementation manner of the first aspect above, according to the weighted calculation weights corresponding to the N regions in the texture enhancement image and the weighted calculation weights corresponding to the N regions in the basic fidelity image, the texture enhancement Perform weighted calculation on the N areas in the image and the N areas in the basic fidelity image to obtain an enhanced image corresponding to the reconstructed image, including: weighting the weighted calculation weights corresponding to the N areas in the texture enhanced image respectively, with the N areas in the texture enhanced image. The regions are multiplied separately to obtain the first product; the weighted calculation weights corresponding to the N regions in the basic fidelity image are respectively multiplied by the N regions in the basic fidelity image to obtain the second product; the first product and the third The two products are added together to obtain the enhanced image corresponding to the reconstructed image.

According to the first aspect, or any one of the implementation manners of the first aspect above, according to the texture complexity weights corresponding to the N regions in the texture-enhanced image, respectively, the weighted calculation weights and the basic guarantees corresponding to the N regions in the texture-enhanced image are determined. The weighted calculation weights corresponding to the N areas in the real image respectively include: determining the texture complexity weights corresponding to the N areas in the texture-enhanced image as the weighted calculation weights corresponding to the N areas in the texture-enhanced image respectively; The difference between the texture complexity weights corresponding to the N regions in the texture-enhanced image is determined as the weighted calculation weights corresponding to the N regions in the basic fidelity image.

According to the first aspect, or any one of the implementation manners of the above first aspect, determining the texture complexity weights corresponding to the regions in the intermediate image respectively includes: decoding the corresponding N regions in the intermediate image from the received code stream. Texture complexity weight, N is a positive integer. In this way, the accuracy of determining the texture intensity weight corresponding to each region in the intermediate image can be improved, and the attenuation of the image texture intensity can be controlled more accurately, thereby further improving the image quality.

According to the first aspect, or any implementation manner of the first aspect above, determining the texture complexity weights corresponding to the regions in the intermediate image respectively includes: dividing the reconstructed image and the intermediate image into N pieces according to a preset partitioning rule. region, N is a positive integer; the texture complexity of N regions in the reconstructed image is determined respectively; based on the texture complexity of the N regions in the reconstructed image, the texture complexity weights corresponding to the N regions in the intermediate image are determined respectively.

In a second aspect, an embodiment of the present application provides an image processing apparatus, and the apparatus includes:

an image acquisition module for acquiring reconstructed images;

The image enhancement module is used to perform image enhancement on the reconstructed image to obtain an intermediate image;

The texture weight determination module is used to determine the texture complexity weight corresponding to each region in the intermediate image, and the texture complexity weight is a number between 0 and 1;

The texture attenuation module is used for attenuating the texture intensity corresponding to each region in the intermediate image according to the texture complexity weight corresponding to each region in the intermediate image, to obtain an enhanced image corresponding to the reconstructed image.

According to the second aspect, the intermediate image is a residual image; the texture attenuation module includes:

The residual update module is used to multiply the pixel value of each pixel in the residual image by the texture complexity weight corresponding to the region to which each pixel belongs to obtain the residual update image;

The image generation module is used to update the image and reconstruct the image based on the residual to generate an enhanced image.

According to the second aspect, or any implementation manner of the above second aspect, the image generation module is specifically configured to add the residual update image and the reconstructed image to obtain an enhanced image.

According to the second aspect, or any implementation manner of the above second aspect, the intermediate image is a texture-enhanced image, and the apparatus further includes: an image fidelity module configured to perform image fidelity on the reconstructed image to obtain a basic fidelity image.

According to the second aspect, or any implementation manner of the above second aspect, the texture weight determination module is specifically configured to divide the basic fidelity image and the texture enhancement image into N regions respectively according to preset partition rules, where N is positive Integer; determine the texture complexity of the N regions in the base fidelity image respectively; determine the texture complexity weights corresponding to the N regions in the texture enhancement image based on the texture complexity of the N regions in the base fidelity image.

According to the second aspect, or any implementation manner of the above second aspect, the texture attenuation module includes:

The weighted weight determination module is used to determine the weighted calculation weights corresponding to the N areas in the texture-enhanced image and the weighted calculation weights corresponding to the N areas in the basic fidelity image according to the texture complexity weights corresponding to the N areas in the texture-enhanced image. Weights;

The weighted calculation module is used to calculate the corresponding weights of N areas in the texture enhanced image and the corresponding weighted calculation weights of the N areas in the basic fidelity image respectively. The N regions are weighted to obtain an enhanced image corresponding to the reconstructed image.

According to the second aspect, or any implementation manner of the second aspect above, the weighted calculation module is specifically configured to multiply the weighted calculation weights corresponding to the N regions in the texture-enhanced image by the N regions in the texture-enhanced image, respectively. , obtain the first product; multiply the weighted calculation weights corresponding to the N regions in the basic fidelity image with the N regions in the basic fidelity image to obtain the second product; add the first product and the second product , to obtain the enhanced image corresponding to the reconstructed image.

According to the second aspect, or any implementation manner of the second aspect above, the weighted weight determination module is specifically configured to determine the texture complexity weights corresponding to the N regions in the texture enhanced image as the N regions in the texture enhanced image The corresponding weighted calculation weights respectively; the difference between 1 and the texture complexity weights corresponding to the N regions in the texture enhanced image is determined as the weighted calculation weights corresponding to the N regions in the basic fidelity image.

According to the second aspect, or any implementation manner of the second aspect above, the texture weight determination module is specifically configured to decode the texture complexity weights corresponding to the N regions in the intermediate image from the received code stream, where N is a positive Integer.

According to the second aspect, or any one of the implementation manners of the above second aspect, the texture weight determination module 803 is specifically configured to divide the reconstructed image and the intermediate image into N regions respectively according to the preset partitioning rule, where N is a positive integer; Determine the texture complexity of the N regions in the reconstructed image respectively; based on the texture complexity of the N regions in the reconstructed image, determine the texture complexity weights corresponding to the N regions in the intermediate image respectively.

The second aspect and any implementation manner of the second aspect correspond to the first aspect and any implementation manner of the first aspect, respectively. For the technical effects corresponding to the second aspect and any implementation manner of the second aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect, which will not be repeated here.

In a third aspect, embodiments of the present application provide an electronic device, including: a memory and a processor, where the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the first aspect or The image processing method in any possible implementation manner of the first aspect.

The third aspect and any implementation manner of the third aspect correspond to the first aspect and any implementation manner of the first aspect, respectively. For the technical effects corresponding to the third aspect and any implementation manner of the third aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect, which will not be repeated here.

In a fourth aspect, an embodiment of the present application provides a chip, including one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of an electronic device, and send a signal to the processor, and the signal includes the memory The computer instructions stored in the computer; when the processor executes the computer instructions, the electronic device is caused to perform the image processing method in the first aspect or any possible implementation manner of the first aspect.

The fourth aspect and any implementation manner of the fourth aspect correspond to the first aspect and any implementation manner of the first aspect, respectively. For the technical effects corresponding to the fourth aspect and any implementation manner of the fourth aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect, which will not be repeated here.

In a fifth aspect, embodiments of the present application provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, causes the computer or processor to execute the first aspect or the first aspect. The image processing method in any possible implementation of an aspect.

The fifth aspect and any implementation manner of the fifth aspect correspond to the first aspect and any implementation manner of the first aspect, respectively. For the technical effects corresponding to the fifth aspect and any implementation manner of the fifth aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect, which will not be repeated here.

In a sixth aspect, embodiments of the present application provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, causes the computer or processor to execute the first aspect or the first aspect. The image processing method in any possible implementation of an aspect.

The sixth aspect and any implementation manner of the sixth aspect correspond to the first aspect and any implementation manner of the first aspect, respectively. For the technical effects corresponding to the sixth aspect and any implementation manner of the sixth aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect, which will not be repeated here.

Description of drawings

Fig. 1a is a schematic diagram of an exemplary application scenario;

Fig. 1b is a schematic diagram of an exemplary processing process;

FIG. 2 is a schematic diagram of an exemplary processing process;

3a is a schematic diagram of an exemplary image enhancement processing process;

Figure 3b is a schematic diagram of an exemplary processing process;

Fig. 3c is a schematic diagram of an exemplary image enhancement effect;

4 is a schematic diagram of an exemplary processing process;

5 is a schematic diagram of an exemplary processing process;

6 is a schematic diagram of an exemplary image enhancement processing process;

7 is a schematic diagram of an exemplary processing process;

FIG. 8 is a schematic diagram of an exemplary image processing apparatus;

FIG. 9 is a schematic structural diagram of an exemplarily shown device.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

The term "and/or" in this article is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and A and B exist independently B these three cases.

The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order of the objects. For example, the first target object, the second target object, etc. are used to distinguish different target objects, rather than to describe a specific order of the target objects.

In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations or illustrations. Any embodiments or designs described in the embodiments of the present application as "exemplary" or "such as" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present the related concepts in a specific manner.

In the description of the embodiments of the present application, unless otherwise specified, the meaning of "plurality" refers to two or more. For example, multiple processing units refers to two or more processing units; multiple systems refers to two or more systems.

FIG. 1a is a schematic diagram of an exemplary application scenario.

Referring to FIG. 1a, exemplarily, the encoding end may encode the image to be encoded to obtain a code stream; and then transmit the code stream to the decoding end. After the decoding end obtains the code stream, it can decode the code stream to obtain a reconstructed image.

Exemplarily, after obtaining the reconstructed image, the decoding end may perform post-processing on the reconstructed image to enhance the reconstructed image to obtain the enhanced image, and then output the enhanced image to reduce visual distortion of the reconstructed image.

Exemplarily, for the post-processing stage, the present application proposes an image processing method, which can enhance the texture of the textured area in the reconstructed image and avoid false textures in the non-textured area, so as to reduce the visual distortion of the reconstructed image, thereby increasing the image quality. Realism, improving the subjective quality of the image.

FIG. 1b is a schematic diagram of an exemplary processing process.

S101, acquiring a reconstructed image.

Exemplarily, after acquiring the code stream, the decoding end may decode the code stream to obtain a reconstructed image.

S102, performing image enhancement on the reconstructed image to obtain an intermediate image.

Exemplarily, in order to reduce the visual distortion of the reconstructed image, image enhancement may be performed on the reconstructed image to obtain an intermediate image.

Exemplarily, an image enhancement model can be used to perform image enhancement on the reconstructed image. Exemplarily, the image enhancement model may be a GANEF (Generative Adversarial Network Enhancement Filter, an enhancement filter of a generative adversarial network), that is, an enhancement filter implemented based on a generative adversarial network. Among them, GAN (Generation Adversarial Network) can include generator and discriminator; in this way, GANEF also includes generator and discriminator. When training GANEF, the generator and discriminator can be trained alternately and iteratively. Among them, the goal of the discriminator is to distinguish the authenticity of the image generated by the generator and the original image. The goal of the generator is to generate an image that is close to the original image and deceive the discriminator. Through effective adversarial training, the generator can be generated as close as possible. Image of the original image.

Exemplarily, the training process of GANEF may be as follows: multiple sets of training data are generated, and a set of training data includes: an image to be encoded and a reconstructed image obtained by decoding a code stream of the image to be encoded. Exemplarily, the training data can be used to train the generator and the discriminator alternately, wherein when training the generator, the weight parameter of the discriminator can be fixed; when training the discriminator, the weight parameter of the generator can be fixed.

Exemplarily, the process of training the generator may be as follows: in advance, input the to-be-encoded image in the training data to the discriminator, and the discriminator performs forward calculation on the to-be-encoded image, and outputs the discrimination result. Then, the weight parameters of the discriminator are adjusted with the goal of making the probability of the discriminator output being "true" close to the preset probability. The preset probability may be set according to requirements, which is not limited in this application. When the difference between the probability that the discriminator outputs a “true” discrimination result and the preset probability according to the input image to be encoded is smaller than the probability threshold, the weight parameter of the discriminator can be fixed. The probability threshold can be set according to requirements, which is not limited in this application. At this time, the reconstructed image in the training data is input into the generator, and the generator performs forward calculation on the reconstructed image, and outputs the intermediate image to the discriminator. Then, on the one hand, the discriminator performs forward calculation on the intermediate image, and outputs the discrimination result; on the other hand, based on the intermediate image and the to-be-encoded image, a loss function value is generated. Then, the weight parameters of the generator are adjusted to maximize the probability that the discriminator output is "true" and to minimize the loss function value.

Exemplarily, the process of training the discriminator may be as follows: in advance, input the to-be-encoded image in the training data to the discriminator, and the discriminator performs forward calculation on the to-be-encoded image, and outputs the discrimination result. Then, the weight parameters of the discriminator are adjusted with the goal of making the probability of the discriminator output being "true" close to the preset probability. According to the input image to be encoded, the discriminator outputs the difference between the probability that the discriminant result is "true" and the preset probability is smaller than the probability threshold, and then input the reconstructed image in the training data into the generator, and the generator will reconstruct the reconstructed image. The image is forwarded and an intermediate image is output to the discriminator. Then, the discriminator performs forward calculation on the intermediate image, outputs the discriminant result, and then adjusts the weight parameters of the discriminator with the goal of maximizing the probability that the discriminator output's discrimination result is "false".

In this way, the generator and discriminator in GANEF are alternately and iteratively trained in the above-mentioned manner, until the discriminator cannot make a good discrimination between the intermediate image output by the generator and the to-be-encoded image.

Exemplarily, the intermediate image may be a residual image or a texture-enhanced image. Among them, in the training process, when the intermediate image output by the generator is a residual image, a texture-enhanced image can be generated based on the residual image and the reconstructed image; then the texture-enhanced image is input to the discriminator, and the image is enhanced according to the texture Calculate the loss function value with the image to be encoded. When the intermediate image output by the generator is a texture-enhanced image, the texture-enhanced image can be directly input into the discriminator.

Exemplarily, the residual image and the reconstructed image may be added to generate a texture-enhanced image. Exemplarily, the residual image and the reconstructed image may be added pixel by pixel, that is, the pixel values of the pixels corresponding to the residual image and the reconstructed image are added to obtain an enhanced image corresponding to the reconstructed image.

Exemplarily, after the GANEF training is completed, only the generator trained in the GANEF may be used to perform image enhancement on the reconstructed image to obtain an intermediate image.

Among them, the resolution of the intermediate image is the same as that of the reconstructed image.

S103: Determine the texture complexity weights corresponding to the regions in the intermediate image respectively.

In a possible manner, after the intermediate image is determined, the texture complexity weight corresponding to each area in the intermediate image may be determined according to the texture complexity of each area in the reconstructed image.

In a possible manner, after the intermediate image is determined, the texture complexity weight corresponding to each area in the intermediate image may be determined according to the texture complexity of each area in the image to be encoded corresponding to the reconstructed image.

Exemplarily, the texture complexity weight is proportional to the texture complexity, that is, the higher the texture complexity, the higher the texture complexity weight; the lower the texture complexity, the lower the texture complexity weight.

Exemplarily, the texture complexity weight is a decimal between 0 and 1.

S104, according to the texture complexity weight corresponding to each region in the intermediate image, attenuate the texture intensity corresponding to each region in the intermediate image respectively, to obtain an enhanced image corresponding to the reconstructed image.

Exemplarily, for each region in the intermediate image, the pixel value of each pixel in the region may be attenuated based on the texture complexity weight corresponding to the region, so as to adjust the texture intensity corresponding to the region. Since the texture complexity weight is proportional to the texture complexity, after the texture intensity is adjusted, the area with higher texture complexity in the intermediate image has less corresponding texture intensity attenuation, and the area with lower texture complexity corresponds to the corresponding texture enhancement attenuation bigger. In this way, while enhancing the texture of the textured region (with a higher texture complexity), it can avoid false streaks in the non-textured region (with a lower texture complexity). That is to say, the present application controls the texture intensity attenuation of the GANEF result according to the texture complexity of the local area of the image, that is, the texture area maintains the original GANEF filtering effect as much as possible, and the non-textured area attenuates the GANEF filtering effect according to the texture complexity. The smaller the texture complexity of the local area, the greater the attenuation, which can effectively remove the false texture of the non-textured area. In addition, the present application controls the texture complexity weight in the range of 0 to 1, so that the texture corresponding to the adjacent areas is softer, and the problem of inconsistent texture effects enhanced by adjacent areas can be avoided.

Exemplarily, for the specific implementation of FIG. 1 b, reference may be made to FIG. 2 , FIG. 4 , FIG. 5 , and FIG. 7 , and the corresponding descriptions.

FIG. 2 is a schematic diagram of an exemplary processing process. In the embodiment of Fig. 2, firstly, GANEF is used to filter the reconstructed image to obtain a residual image; then based on the texture complexity of each area in the reconstructed image, the texture complexity weight corresponding to each area in the residual image is determined; The residual image is updated according to the texture complexity weights corresponding to each region, and finally the updated residual image and the reconstructed image are added to obtain the final enhanced image.

S201, performing image enhancement on the reconstructed image to obtain a residual image.

Exemplarily, after obtaining the code stream, the decoding end can decode the code stream to obtain a reconstructed image; and then an image enhancement model can be used to enhance the reconstructed image to obtain an intermediate image.

Exemplarily, the intermediate image is a residual image.

Fig. 3a is an exemplary schematic diagram of an image enhancement processing process.

In one possible way, the output of the trained image augmentation model is a residual image. In this way, by inputting the reconstructed image to the image enhancement model, the residual image can be directly obtained, as shown in Figure 3a(1).

In one possible way, the output of the trained image augmentation model is a texture augmented image. In this way, after inputting the reconstructed image to the image enhancement model, the image enhancement model can output a texture-enhanced image; then, based on the texture-enhanced image and the reconstructed image, a residual image is determined. Optionally, pixel-by-pixel subtraction can be performed on the texture-enhanced image and the reconstructed image, that is, the pixel values of the corresponding pixels of the texture-enhanced image and the reconstructed image are subtracted to obtain a residual image, as shown in Figure 3a(2). .

Exemplarily, the residual image has the same resolution as the reconstructed image.

S202, based on the reconstructed image, determine the texture complexity weight corresponding to each region in the residual image.

Exemplarily, the texture complexity weight corresponding to each region in the residual image may be determined according to the texture complexity corresponding to each region in the reconstructed image. Exemplarily, S202 may include S2021-S2023:

S2021, according to a preset partition rule, divide the reconstructed image and the residual image into N regions respectively.

Exemplarily, the reconstructed image may be divided into N regions according to a preset preset partition rule; and the residual image may be divided into N regions according to the preset partition rule. That is to say, the areas of the reconstructed image and the residual image are divided in the same way. Since the resolutions of the reconstructed image and the residual image are the same, the N areas in the reconstructed image and the N areas in the residual image are in one-to-one correspondence. .

Exemplarily, the preset partitioning rule may be set as required, which is not limited in this application; wherein, N is a positive integer and is determined according to the preset partitioning rule. For example, the preset partition rule is: divide the image into N regions of size w*h. Wherein, w=W/N1, h=H/N2, N=N1*N2, where W and H are the width and height of the image, and N1 and N2 are positive integers.

It should be noted that the size of each of the N regions may be the same or different, which is not limited in this application. In addition, the shape of each of the N regions may be the same or different, which is not limited in the present application. Moreover, the present application does not limit the shapes of the N regions.

S2022: Determine the texture complexity of the N regions in the reconstructed image respectively.

The following is an example of determining the ith (i is an integer between 1 and N, and the value of i may be the texture complexity of 1, N) regions in the reconstructed image.

Exemplarily, for the ith region of the reconstructed image, the texture complexity of the ith region may be determined according to pixel values of pixels included in the ith region.

In a possible manner, the texture complexity of the ith region of the reconstructed image may be determined based on the co-occurrence matrix of the ith region in the reconstructed image. Exemplarily, the co-occurrence matrix of the ith region in the reconstructed image may be determined according to the pixel values of the pixels included in the ith region in the reconstructed image. Then extract the feature quantity (such as energy, contrast, entropy, inverse variance, etc.) corresponding to the co-occurrence matrix of the ith area in the reconstructed image, and then determine the ith area in the reconstructed image according to the feature quantity of the co-occurrence matrix of the ith area in the reconstructed image. Image texture complexity for i regions. For example, the feature quantity of the co-occurrence matrix of the ith region in the reconstructed image is taken as the texture complexity of the ith region in the reconstructed image.

In a possible manner, the texture complexity of the ith region in the reconstructed image may be determined based on the edge ratio of the ith region in the reconstructed image. Exemplarily, the gradient intensity corresponding to each pixel in the ith region in the reconstructed image may be calculated according to the pixel value of the pixel included in the ith region of the reconstructed image. Then, the proportion of pixels whose gradient intensity is greater than the gradient intensity threshold in the ith region of the reconstructed image is taken as the texture complexity of the ith region of the reconstructed image. The gradient intensity threshold may be set as required, which is not limited in this application.

It should be understood that the texture complexity of the ith region in the reconstructed image may also be determined in other manners, such as according to the grayscale histogram distribution of the ith region in the reconstructed image, which is not limited in this application.

In this way, in the above manner, the texture complexity of N regions in the reconstructed image can be determined.

S2023 , based on the texture complexity of the N regions in the reconstructed image, determine the texture complexity weights corresponding to the N regions in the residual image respectively.

The texture complexity weight corresponding to the ith (i is an integer between 1 and N, i may be equal to 1 and N) regions in the residual image is determined below for exemplary illustration.

In a possible way, the texture complexity of the ith region in the reconstructed image may be used as the texture complexity weight corresponding to the ith region in the residual image.

In a possible manner, the texture complexity of the ith region in the reconstructed image may be mapped according to a preset mapping rule, to obtain the texture complexity weight corresponding to the ith region in the residual image. Exemplarily, the preset mapping rules may be set according to requirements, such as normalization, which is not limited in this application. For example, the texture complexity of the ith region in the reconstructed image may be normalized to obtain the texture complexity weight corresponding to the ith region in the residual image.

Exemplarily, the texture complexity weight may be a decimal between 0 and 1.

Exemplarily, the texture complexity weight is proportional to the texture complexity, that is, the higher the texture complexity, the higher the texture complexity weight; the lower the texture complexity, the lower the texture complexity weight.

Exemplarily, after obtaining the texture complexity weights corresponding to N areas in the residual image, the texture intensity corresponding to each area in the residual image may be attenuated according to the texture complexity weights corresponding to each area in the residual image, To perform image enhancement on the reconstructed image to obtain the enhanced image corresponding to the reconstructed image, refer to S203-S204:

S203 , based on the texture complexity weights corresponding to each region in the residual image, perform residual update on each region in the residual image to obtain a residual update image.

Exemplarily, based on the texture complexity weights corresponding to the N regions of the residual image, the residual update may be performed on the N regions in the residual image, respectively, to obtain the residual updated image.

In a possible way, the residual image can be updated by multiplying the pixel value of each pixel in the residual image by the texture complexity weight corresponding to the region to which each pixel belongs to obtain the residual update. image.

The following is an example of performing residual update in the ith (i is an integer between 1 and N, and i may be equal to 1 and N) regions in the residual image as an example.

Exemplarily, assuming that the texture enhancement weight corresponding to the ith region in the residual image is ratio_i, for a pixel (k, j) in the ith region, the pixel value corresponding to the pixel (k, j) can be used. To multiply R1(k,j) and ratio_i, update the pixel value of the pixel point (k,j) to obtain a new pixel value R2(k,j)=R1(k,j)*ratio_i. Among them, (k, j) is the integer index of the pixel point coordinates of the residual image, k, j represent the coordinate index of the horizontal and vertical directions respectively, and the pixel index of the upper left corner of the residual image is (0, 0). That is to say, after the pixel value of each pixel in the residual image is updated, the residual updated image can be obtained, and the pixel value of the pixel in the residual updated image is R2(k, j).

Exemplarily, the residual update image and the reconstructed image have the same resolution.

Because the texture complexity weight is proportional to the texture complexity, in this way, the area with higher texture complexity of the residual image has a smaller corresponding texture intensity attenuation, and the area with lower texture complexity corresponds to a larger texture enhancement attenuation.

S204, adding the reconstructed image and the residual update image to obtain an enhanced image.

Exemplarily, the pixel values of the pixel points corresponding to the reconstructed image and the residual update image may be added to obtain an enhanced image corresponding to the reconstructed image.

FIG. 3b is a schematic diagram of an exemplary processing process.

Referring to Fig. 3b, exemplarily, A1 is a reconstructed image, and after filtering the reconstructed image with GANEF, a texture-enhanced image is obtained, as shown in A2. Then the texture-enhanced image A2 can be used to subtract the reconstructed image A1 to obtain a residual image, as shown in A4

Exemplarily, the gradient of each pixel in the reconstructed image A1 may be calculated, and then based on the gradient of each pixel in the reconstructed image A1, the reconstructed image is binarized to obtain a binary image, as shown in A3. Then, the binary image A3 is divided into N regions, and for each region, the texture complexity corresponding to the region is determined according to the proportion of black pixels in the region; and then the corresponding N regions in the binary image A3 can be obtained respectively. texture complexity. Then, according to the texture complexity corresponding to the N regions in the binary image, the texture complexity weights corresponding to the N regions in the residual image A4 are determined respectively. Then, based on the texture complexity weights corresponding to the N regions in the residual image A4, the residual image A4 is updated to obtain the residual updated image, as shown in A5.

Exemplarily, the reconstructed image A1 and the residual update image A5 may be added to obtain the enhanced image A6.

Fig. 3c is an exemplary schematic diagram of an image enhancement effect.

Referring to Fig. 3c, exemplarily, Fig. 3c(1) is a schematic diagram of a local area in the texture-enhanced image A2 of Fig. 3b, and Fig. 3c(2) is a schematic diagram of a local area in the enhanced image A6 of Fig. 3b.

Exemplarily, the road surface in Fig. 3c(1) and Fig. 3c(2) is a non-textured area, through the elliptical area in Fig. 3c(1) and Fig. 3c(2), and comparing Fig. 3c(1) and Fig. 3c (2) The rectangular area in the middle shows that there are no false streaks in the non-textured area in the enhanced image obtained by the present application.

In this way, by controlling the texture complexity weight in the range of 0 to 1 according to the texture complexity of the reconstructed image, the texture intensity corresponding to each area in the reconstructed image can be attenuated, so that the texture of the texture area in the intermediate image can be retained, and the non-texture texture can be preserved. The texture in the area is attenuated, so as to enhance the texture of the texture area (higher texture complexity) in the reconstructed image, while avoiding false streaks in the non-texture area (low texture complexity) in the reconstructed image; thus reducing the visual quality of the reconstructed image distortion. In addition, the present application controls the texture complexity weight in the range of 0 to 1, so that the texture corresponding to the adjacent area is softer, and the problem of inconsistent texture effects enhanced by the adjacent area can be avoided.

In addition, compared with the prior art using two models for post-processing, the present application uses only one model, which reduces the computational complexity.

FIG. 4 is a schematic diagram of an exemplary processing process. In the embodiment of FIG. 4 , firstly, GANEF is used to filter the reconstructed image to obtain a residual image; then based on the texture complexity of each area in the image to be encoded corresponding to the reconstructed image, the texture complexity corresponding to each area of the residual image is determined. Then, the residual image is updated according to the texture complexity weight corresponding to each region, and finally the updated residual image and the reconstructed image are added to obtain the final enhanced image.

S401: Decode the code stream to obtain texture complexity weights corresponding to regions in the reconstructed image and the image to be encoded.

Exemplarily, the encoding end may generate, based on the image to be encoded, the texture complexity weight corresponding to each region in the image to be encoded, which may include S4011 to S4013:

S4011, according to a preset partition rule, divide the image to be encoded into N regions.

S4012: Determine the texture complexity of the N regions in the to-be-coded image respectively.

Exemplarily, for S4011 to S4012, reference may be made to the descriptions of the foregoing S2021 to S2022, which will not be repeated here.

S4013 , based on the texture complexity of the N regions in the to-be-coded image, determine the texture complexity weights corresponding to the N-regions in the to-be-coded image respectively.

The following is an example of determining the texture complexity weight corresponding to the ith (i is an integer between 1 and N, and the value of i may be 1, N) regions in the image to be encoded.

In a possible manner, the texture complexity of the ith region in the image to be encoded may be used as the texture complexity weight corresponding to the ith region in the image to be encoded.

In a possible manner, the texture complexity of the ith region in the image to be encoded may be mapped according to a preset mapping rule to obtain the texture complexity weight corresponding to the ith region in the image to be encoded. For example, the texture complexity of the ith region in the to-be-coded image may be normalized to obtain the texture complexity weight corresponding to the ith region in the to-be-coded image.

Exemplarily, the texture complexity weight may be a decimal between 0 and 1.

Exemplarily, the texture complexity weight is proportional to the texture complexity, that is, the higher the texture complexity, the higher the texture complexity weight; the lower the texture complexity, the lower the texture complexity weight.

Exemplarily, the encoding end may encode the image to be encoded to obtain a corresponding code stream; and may encode texture complexity weights corresponding to N regions in the image to be encoded to obtain a corresponding code stream. Then, the code stream obtained by encoding the texture complexity weights corresponding to the N regions in the image to be encoded and the code stream obtained by encoding the image to be encoded may be sent to the decoding end. In this way, after acquiring the code stream, the decoding end decodes the texture complexity weights corresponding to the N regions in the reconstructed image and the to-be-encoded image respectively from the code stream.

S402, performing image enhancement on the reconstructed image to obtain a residual image.

Exemplarily, the decoding end may use an image enhancement model to perform image enhancement on the reconstructed image to obtain an intermediate image. Exemplarily, the intermediate image is a residual image.

Wherein, for S402, reference may be made to the description of the above-mentioned S201, and details are not repeated here.

Exemplarily, the resolution of the residual image is the same as the resolution of the reconstructed image and the resolution of the to-be-encoded image.

S403, according to the texture complexity weight corresponding to each area in the image to be encoded, generate the texture complexity weight corresponding to each area in the residual image.

Exemplarily, after the residual image is obtained, the residual image may be divided into N regions according to a preset preset partitioning rule. The area division method of the residual image is the same as that of the image to be encoded. Since the resolution of the residual image and the image to be encoded are the same, the N areas in the residual image are the same as the N areas in the image to be encoded. Regions are in one-to-one correspondence. Furthermore, the texture complexity weight corresponding to the ith region of the image to be encoded may be used as the texture complexity weight corresponding to the ith region of the residual image. In this way, the texture complexity weights corresponding to the N regions in the residual image can be determined.

Exemplarily, the texture complexity weight may be a decimal between 0 and 1.

Exemplarily, after obtaining the texture complexity weights corresponding to N regions in the residual image, the texture intensity corresponding to each region in the residual image can be adjusted according to the texture complexity weights corresponding to each region in the residual image, respectively, To obtain the enhanced image corresponding to the reconstructed image, refer to S404-S405:

S404 , performing a residual update on the residual image based on the texture complexity weights corresponding to each region in the residual image to obtain a residual updated image.

S405, adding the reconstructed image and the residual update image to obtain an enhanced image.

Exemplarily, for S404-S405, reference may be made to the descriptions of S203-S204 above, which will not be repeated here.

In this way, by controlling the texture complexity weight in the range of 0 to 1 according to the texture complexity of the image to be encoded, to adjust the texture enhancement corresponding to each area in the reconstructed image, it is possible to enhance the texture area (with high texture complexity) in the reconstructed image. At the same time of texture, it avoids false streaks in the non-textured area (with low texture complexity) in the reconstructed image; thereby reducing the visual distortion of the reconstructed image. In addition, the present application controls the texture complexity weight in the range of 0 to 1, so that the texture corresponding to the adjacent area is softer, and the problem of inconsistent texture effects enhanced by the adjacent area can be avoided.

In addition, compared with the prior art using two models for post-processing, the present application uses only one model, which reduces the computational complexity.

Thirdly, compared with the texture complexity weight determined according to the reconstructed image, the texture complexity weight determined according to the to-be-encoded image is more accurate, which can more accurately control the attenuation of the texture intensity of the image, thereby further improving the image quality.

FIG. 5 is a schematic diagram of an exemplary processing process. In the embodiment of FIG. 5 , the reconstructed image is first filtered by a non-generative adversarial network (non-GANEF) and a generative adversarial network (GANEF) to obtain a basic fidelity image and a texture-enhanced image, respectively, and then based on the reconstructed image or basic fidelity The texture complexity corresponding to each area in the image is determined, and the weighted calculation factors corresponding to the basic fidelity image and the texture-enhanced image are determined. Finally, the basic fidelity image and the texture-enhanced image are weighted and fused based on the weighted calculation factor to obtain the final enhanced image.

S501, performing image enhancement on the reconstructed image to obtain a texture-enhanced image.

Exemplarily, after obtaining the code stream, the decoding end can decode the code stream to obtain a reconstructed image; and then an image enhancement model can be used to enhance the reconstructed image to obtain an intermediate image. Exemplarily, the intermediate image is a texture-enhanced image.

FIG. 6 is an exemplary schematic diagram of an image enhancement processing process.

In one possible way, the output of the trained image augmentation model is a residual image. In this way, by inputting the reconstructed image into the image enhancement model, the residual image output by the image enhancement model can be obtained, and then the texture enhanced image can be determined according to the residual image and the reconstructed image. Optionally, the pixel values of the corresponding pixels of the residual image and the reconstructed image may be added to obtain a texture-enhanced image, as shown in FIG. 6(1).

In one possible way, the output of the trained image augmentation model is a texture augmented image. In this way, after inputting the reconstructed image to the image enhancement model, a texture-enhanced image can be directly obtained, as shown in Figure 6(2).

Exemplarily, the texture-enhanced image is of the same resolution as the reconstructed image.

S502, performing image fidelity on the reconstructed image to obtain a basic fidelity image.

Exemplarily, a preset image fidelity model can be used to reconstruct an image to perform image fidelity to obtain a basic fidelity image.

Exemplarily, the network used by the image fidelity model may be a non-generative adversarial network, such as a convolutional neural network, which is not limited in this application.

Exemplarily, the training process of the image fidelity model may be as follows: collect multiple sets of training data, where a set of training data includes an image to be encoded and a reconstructed image obtained by decoding a code stream of the image to be encoded. For a set of training data, a set of training data is input into the image fidelity model, and the image fidelity model performs forward calculation on the reconstructed image, and outputs the basic fidelity image. Calculate the loss function value based on the basic fidelity image output by the image fidelity model and the to-be-encoded image in the training data, and adjust the weight parameters of the image fidelity model with the goal of minimizing the loss function value. Then, the image fidelity model can be trained by using the collected training data in the above manner, until the number of training times of the image fidelity model is equal to the preset number of training times, or the loss function value of the image fidelity model is less than or equal to the loss function threshold , or the effect of the image fidelity model satisfies the preset effect condition, stop the training of the image fidelity model, and obtain the trained image fidelity model. Exemplarily, the preset training times, the loss function threshold, and the preset effect conditions can all be set as required, which is not limited in this application.

It should be noted that this application does not limit the execution order of S501 and S502.

Exemplarily, both the texture-enhanced image and the base fidelity image are of the same resolution as the reconstructed image.

S503 , based on the basic fidelity image, generate texture complexity weights corresponding to each region in the texture enhanced image.

Exemplarily, S503 may include S5031a to S5034a:

S5031a, according to preset partition rules, divide the basic fidelity image and the texture enhancement image into N regions respectively.

S5032a: Determine the texture complexity of the N regions in the basic fidelity image respectively.

S5033a, based on the texture complexity of the N regions in the basic fidelity image, determine the texture complexity weights corresponding to the N regions in the texture enhancement image respectively.

Exemplarily, for S5031a-S5033a, reference may be made to the descriptions of S2021-S2023 above, which will not be repeated here.

In a possible way, the texture complexity weight corresponding to each region in the texture-enhanced image can be generated based on the reconstructed image. Refer to S5031b to S5033b:

S5031b, according to a preset partition rule, divide the reconstructed image and the texture-enhanced image into N regions respectively.

S5032b, respectively determine the texture complexity of the N regions in the reconstructed image.

S5033b, based on the texture complexity of the N regions in the reconstructed image, determine the texture complexity weights corresponding to the N regions in the texture enhancement image respectively.

Exemplarily, for S5031b to S5033b, reference may be made to the descriptions of S2021 to S2023 above, which will not be repeated here.

S504: Perform weighted fusion of the basic fidelity image and the texture-enhanced image according to the texture complexity weights corresponding to each region in the texture-enhanced image to obtain an enhanced image.

Exemplarily, after both the texture-enhanced image and the basic fidelity image are divided into N regions, the regions of the texture-enhanced image and the basic fidelity image are in one-to-one correspondence. In this way, each area in the texture-enhanced image can be weighted and fused with the corresponding area in the basic fidelity image, and an enhanced image can be obtained.

Exemplarily, S504 may include S5041-S5042:

S5041 , according to the texture complexity weights corresponding to the N regions in the texture enhanced image respectively, determine the weighted calculation weights corresponding to the N regions in the texture enhanced image and the weighted calculation weights corresponding to the N regions in the basic fidelity image respectively.

Exemplarily, the texture complexity weights corresponding to the N regions in the texture-enhanced image may be used as the weighted calculation weights of the N regions in the texture-enhanced image. And the difference between 1 and the texture complexity weights corresponding to the N regions in the texture-enhanced image is used as the weighted calculation weight of the N regions in the basic fidelity image.

For example, for the ith area, the texture complexity weight corresponding to the ith area of the texture-enhanced image is ratio_i, then the weighted calculation weight of the ith area of the texture-enhanced image can be ratio_i, and the ith area of the basic fidelity image The weighted calculation weight of the region is 1-ratio_i.

S5042, according to the weighted calculation weights corresponding to the N areas in the texture enhancement image and the weighted calculation weights corresponding to the N areas in the basic fidelity image respectively, perform the calculation on the N areas in the texture enhancement image and the N areas in the basic fidelity image. Weighted calculation to get an enhanced image.

Exemplarily, for the ith area, according to the weighted calculation weight of the ith area in the texture-enhanced image and the weighted calculation weight of the ith area in the basic fidelity image, each pixel of the ith area in the texture-enhanced image is calculated. Points and each pixel point of the ith area in the basic fidelity image are weighted to obtain the enhanced image of the ith area.

Exemplarily, the weighted calculation weights corresponding to the N regions in the texture-enhanced image can be multiplied by the N regions in the texture-enhanced image to obtain the first product; the weights corresponding to the N regions in the basic fidelity image are respectively Calculate the weight and multiply it with N regions in the basic fidelity image to obtain the second product; add the first product and the second product to obtain the enhanced image corresponding to the reconstructed image.

For example, the weighted calculation weight of the ith area in the texture-enhanced image is ratio_i, and the pixel value of a pixel (j, k) in the ith area in the texture-enhanced image is E1(j, k); in the basic fidelity image The weighted calculation weight of the ith area is 1-ratio_i, and the pixel value of a pixel (j, k) of the ith area in the basic fidelity image is E2(j, k), then the ith area of the reconstructed image is The pixel value R(i,j)=ratio_i*E1(i,j)+(1-ratio_i)*E2(i,j) for the pixel point (j,k) of the region after image enhancement.

In this way, by adjusting the texture enhancement corresponding to each area in the reconstructed image by controlling the texture complexity weight in the range of 0 to 1 according to the texture complexity of the reconstructed image, the texture of the texture area (higher texture complexity) in the reconstructed image can be enhanced. At the same time, it avoids false streaks in the non-textured area (with low texture complexity) in the reconstructed image; thereby reducing the visual distortion of the reconstructed image. In addition, the present application controls the texture complexity weight in the range of 0 to 1, so that the texture corresponding to the adjacent area is softer, and the problem of inconsistent texture effects enhanced by the adjacent area can be avoided.

FIG. 7 is a schematic diagram of an exemplary processing process. In the embodiment of FIG. 7 , a non-generative adversarial network (non-GANEF) and a generative adversarial network (GANEF) are used to filter the reconstructed image, respectively, to obtain a basic fidelity image and a texture-enhanced image, and then based on the reconstructed image corresponding to the to-be-encoded image The texture complexity corresponding to each area in the image or the basic fidelity image, determine the weighted calculation factors corresponding to the basic fidelity image and the texture-enhanced image respectively, and finally perform weighted fusion of the basic fidelity image and the texture-enhanced image based on the weighted calculation factor to obtain The final enhanced image.

S701: Decode the code stream to obtain texture complexity weights corresponding to regions in the reconstructed image and the image to be encoded.

Exemplarily, the encoding end may generate the texture complexity weight based on the to-be-encoded image, which may include S7011 to S7013:

S7011, according to a preset partition rule, divide the image to be encoded into N regions.

S7012: Determine the texture complexity of the N regions in the to-be-coded image respectively.

S7013, based on the texture complexity of the N regions in the to-be-coded image, determine the texture complexity weights corresponding to the N-regions in the to-be-coded image respectively.

Exemplarily, for S7011-S7013, reference may be made to the descriptions of the above-mentioned S2021-S2023, which will not be repeated here.

S702, performing image enhancement on the reconstructed image to obtain a texture-enhanced image.

Exemplarily, for S702, reference may be made to the description of the foregoing S502, and details are not repeated here.

S703, according to the texture complexity weights corresponding to the regions in the image to be encoded, determine the texture enhancement weights corresponding to the regions in the texture enhancement image respectively.

Exemplarily, after the texture-enhanced image is obtained, the texture-enhanced image may be divided into N regions according to a preset preset partition rule. The area division method of the texture-enhanced image is the same as that of the image to be encoded, and the resolution of the texture-enhanced image and the image to be encoded is the same, so the area in the texture-enhanced image and the area in the image to be encoded are one-to-one corresponding. Furthermore, the texture complexity weight corresponding to the ith area of the image to be encoded may be used as the texture complexity weight corresponding to the ith area of the texture-enhanced image. In this way, the texture complexity weights corresponding to the N regions in the texture-enhanced image can be determined.

Exemplarily, the texture complexity weight may be a decimal between 0 and 1.

Exemplarily, after obtaining the texture complexity weights corresponding to N regions in the texture enhanced image, the texture intensity corresponding to each region in the texture enhanced image can be adjusted according to the texture complexity weights corresponding to each region in the texture enhanced image to obtain a reconstruction. For the enhanced image corresponding to the image, refer to S704-S705:

S704, performing image fidelity on the reconstructed image to obtain a basic fidelity image.

Exemplarily, for S704, reference may be made to the description of the foregoing S502, and details are not repeated here.

Exemplarily, the present application does not limit the execution order of S704 and S702.

S705, according to the texture complexity weight corresponding to each region in the texture enhanced image, perform weighted fusion of the basic fidelity image and the texture enhanced image to obtain an enhanced image.

Exemplarily, for S705, reference may be made to the description of the foregoing S504, and details are not repeated here.

In this way, by controlling the texture complexity weight in the range of 0 to 1 according to the texture complexity of the image to be encoded, to adjust the texture enhancement corresponding to each area in the reconstructed image, it is possible to enhance the texture area (with high texture complexity) in the reconstructed image. At the same time of texture, it avoids false streaks in the non-textured area (with low texture complexity) in the reconstructed image; thereby reducing the visual distortion of the reconstructed image. In addition, the present application controls the texture complexity weight in the range of 0 to 1, so that the texture corresponding to the adjacent area is softer, and the problem of inconsistent texture effects enhanced by the adjacent area can be avoided.

Thirdly, compared with the texture complexity weight determined according to the reconstructed image, the texture complexity weight determined according to the to-be-encoded image is more accurate, which can more accurately control the attenuation of the texture intensity of the image, thereby further improving the image quality.

FIG. 8 is a schematic diagram of an exemplary image processing apparatus.

8 , the image processing apparatus exemplarily includes: an image acquisition module 801, an image enhancement module 802, a texture weight determination module 803 and a texture attenuation module 804, wherein:

an image acquisition module 801, configured to acquire a reconstructed image;

an image enhancement module 802, configured to perform image enhancement on the reconstructed image to obtain an intermediate image;

The texture weight determination module 803 is used for determining the texture complexity weight corresponding to each region in the intermediate image respectively, and the texture complexity weight is a number between 0 and 1;

The texture attenuation module 804 is configured to attenuate the texture intensity corresponding to each region in the intermediate image according to the texture complexity weight corresponding to each region in the intermediate image, to obtain an enhanced image corresponding to the reconstructed image.

Exemplarily, the intermediate image is a residual image;

Texture attenuation module 804, including:

The residual update module is used to multiply the pixel value of each pixel in the residual image by the texture complexity weight corresponding to the region to which each pixel belongs to obtain the residual update image;

The image generation module is used to update the image and reconstruct the image based on the residual to generate an enhanced image.

Exemplarily, the image generation module is specifically configured to add the residual update image and the reconstructed image to obtain an enhanced image.

Exemplarily, the intermediate image is a texture-enhanced image, and the apparatus further includes:

The image fidelity module is used to perform image fidelity on the reconstructed image to obtain a basic fidelity image.

Exemplarily, the texture weight determination module 803 is specifically configured to divide the basic fidelity image and the texture-enhanced image into N regions respectively according to preset partition rules, where N is a positive integer; respectively determine N regions in the basic fidelity image. The texture complexity is determined based on the texture complexity of the N regions in the basic fidelity image, and the texture complexity weights corresponding to the N regions in the texture-enhanced image are determined.

Exemplarily, the texture attenuation module 804 includes:

The weighted weight determination module is used to determine the weighted calculation weights corresponding to the N areas in the texture-enhanced image and the weighted calculation weights corresponding to the N areas in the basic fidelity image according to the texture complexity weights corresponding to the N areas in the texture-enhanced image. Weights;

The weighted calculation module is used to calculate the corresponding weights of N areas in the texture enhanced image and the corresponding weighted calculation weights of the N areas in the basic fidelity image respectively. The N regions are weighted to obtain an enhanced image corresponding to the reconstructed image.

Exemplarily, the weighted calculation module is specifically used to multiply the weighted calculation weights corresponding to the N regions in the texture-enhanced image respectively with the N regions in the texture-enhanced image to obtain the first product; The weighted calculation weights corresponding to the respective regions are multiplied by the N regions in the basic fidelity image to obtain the second product; the first product and the second product are added to obtain the enhanced image corresponding to the reconstructed image.

Exemplarily, the weighted weight determination module is specifically configured to determine the texture complexity weights corresponding to the N regions in the texture enhanced image as the weighted calculation weights corresponding to the N regions in the texture enhanced image respectively; The difference between the texture complexity weights corresponding to the N regions in the base fidelity image is determined as the weighted calculation weights corresponding to the N regions in the basic fidelity image.

Exemplarily, the texture weight determination module 803 is specifically configured to decode, from the received code stream, texture complexity weights corresponding to N regions in the intermediate image, where N is a positive integer.

Exemplarily, the texture weight determination module 803 is specifically configured to divide the reconstructed image and the intermediate image into N regions respectively according to a preset partition rule, where N is a positive integer; and respectively determine the texture complexity of the N regions in the reconstructed image; Based on the texture complexity of the N regions in the reconstructed image, the texture complexity weights corresponding to the N regions in the intermediate image respectively are determined.

In an example, FIG. 9 shows a schematic block diagram of an apparatus 900 according to an embodiment of the present application. The apparatus 900 may include: a processor 901 , a transceiver/transceiver pin 902 , and optionally, a memory 903 .

Various components of the device 900 are coupled together through a bus 904, wherein the bus 904 includes a power bus, a control bus and a status signal bus in addition to a data bus. For clarity, however, the various buses are referred to as bus 904 in the figures.

Optionally, the memory 903 may be used for instructions in the foregoing method embodiments. The processor 901 can be used to execute the instructions in the memory 903, and control the receive pins to receive signals, and control the transmit pins to transmit signals.

The apparatus 900 may be the electronic device or the chip of the electronic device in the above method embodiments.

Wherein, all relevant contents of the steps involved in the above method embodiments can be cited in the functional descriptions of the corresponding functional modules, which will not be repeated here.

This embodiment also provides a computer-readable storage medium, where computer instructions are stored in the computer-readable storage medium, and when the computer instructions are executed on the electronic device, the electronic device executes the above-mentioned related method steps to realize the above-mentioned embodiments. image processing method.

This embodiment also provides a computer program product, which when the computer program product runs on a computer, causes the computer to execute the above-mentioned relevant steps, so as to realize the image processing method in the above-mentioned embodiment.

In addition, the embodiments of the present application also provide an apparatus, which may specifically be a chip, a component or a module, and the apparatus may include a connected processor and a memory; wherein, the memory is used to store computer execution instructions, and when the apparatus is running, The processor can execute the computer-executed instructions stored in the memory, so that the chip executes the image processing methods in the above method embodiments.

Wherein, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding method provided above. Therefore, for the beneficial effects that can be achieved, reference may be made to the above-provided method. The beneficial effects in the corresponding method will not be repeated here.

From the description of the above embodiments, those skilled in the art can understand that for the convenience and brevity of the description, only the division of the above functional modules is used as an example for illustration. In practical applications, the above functions can be allocated by different The function module is completed, that is, the internal structure of the device is divided into different function modules, so as to complete all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or May be integrated into another device, or some features may be omitted, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

Units described as separate components may or may not be physically separated, and components shown as units may be one physical unit or multiple physical units, that is, may be located in one place, or may be distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

Any content of each embodiment of the present application and any content of the same embodiment can be freely combined. Any combination of the above is within the scope of this application.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, which are stored in a storage medium , including several instructions to make a device (which may be a single chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a removable hard disk, a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk and other media that can store program codes.

The embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific embodiments, which are merely illustrative rather than restrictive. Under the inspiration of this application, without departing from the scope of protection of the purpose of this application and the claims, many forms can be made, which all fall within the protection of this application.

The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of this application may be implemented in a hardware manner, or may be implemented in a manner in which a processor executes software instructions. Software instructions can be composed of corresponding software modules, and software modules can be stored in random access memory (Random Access Memory, RAM), flash memory, read only memory (Read Only Memory, ROM), erasable programmable read only memory ( Erasable Programmable ROM, EPROM), Electrically Erasable Programmable Read-Only Memory (Electrically EPROM, EEPROM), registers, hard disks, removable hard disks, compact disks (CD-ROMs), or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, such that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium may reside in an ASIC.

Those skilled in the art should realize that, in one or more of the above examples, the functions described in the embodiments of the present application may be implemented by hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer-readable storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage medium can be any available medium that can be accessed by a general purpose or special purpose computer.

The embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific embodiments, which are merely illustrative rather than restrictive. Under the inspiration of this application, without departing from the scope of protection of the purpose of this application and the claims, many forms can be made, which all fall within the protection of this application.

Claims (24)

1. An image processing method, characterized in that the method comprises:

acquiring a reconstructed image;

carrying out image enhancement on the reconstructed image to obtain an intermediate image;

determining texture complexity weights respectively corresponding to all regions in the intermediate image, wherein the texture complexity weights are numbers between 0 and 1;

and respectively attenuating the texture intensity corresponding to each region in the intermediate image according to the texture complexity weight corresponding to each region in the intermediate image, so as to obtain an enhanced image corresponding to the reconstructed image.

2. The method of claim 1, wherein the intermediate image is a residual image;

the attenuating the texture intensity corresponding to each region in the intermediate image respectively according to the texture complexity weight corresponding to each region in the intermediate image to obtain an enhanced image corresponding to the reconstructed image includes:

multiplying the pixel value of each pixel point in the residual image by the texture complexity weight corresponding to the region to which each pixel point belongs to obtain a residual updated image;

and generating the enhanced image according to the residual updating image and the reconstructed image.

3. The method of claim 2, wherein generating the enhanced image from the residual updated image and the reconstructed image comprises:

and adding the residual updating image and the reconstructed image to obtain the enhanced image.

4. The method of claim 1, wherein the intermediate image is a texture enhanced image, the method further comprising:

and performing image fidelity on the reconstructed image to obtain a basic fidelity image.

5. The method of claim 4, wherein determining the texture complexity weight corresponding to each region in the intermediate image comprises:

according to a preset partition rule, dividing the basic fidelity image and the texture enhanced image into N regions respectively, wherein N is a positive integer;

respectively determining the texture complexity of N areas in the basic fidelity image;

and determining texture complexity weights corresponding to the N regions in the texture enhanced image respectively based on the texture complexity of the N regions in the basic fidelity image.

6. The method according to claim 5, wherein the attenuating the texture intensities corresponding to the regions in the intermediate image respectively according to the texture complexity weights corresponding to the regions in the intermediate image to obtain the enhanced image corresponding to the reconstructed image comprises:

determining weighting calculation weights respectively corresponding to the N regions in the texture enhanced image and weighting calculation weights respectively corresponding to the N regions in the basic fidelity image according to texture complexity weights corresponding to the N regions in the texture enhanced image;

and carrying out weighted calculation on the N regions in the texture enhanced image and the N regions in the basic fidelity image according to the weighted calculation weights respectively corresponding to the N regions in the texture enhanced image and the weighted calculation weights respectively corresponding to the N regions in the basic fidelity image to obtain an enhanced image corresponding to the reconstructed image.

7. The method according to claim 6, wherein the performing weighted computation on the N regions in the texture-enhanced image and the N regions in the basic fidelity image according to the weighted computation weights corresponding to the N regions in the texture-enhanced image and the weighted computation weights corresponding to the N regions in the basic fidelity image, respectively, to obtain an enhanced image corresponding to the reconstructed image comprises:

respectively multiplying the weighted calculation weights corresponding to the N areas in the texture enhanced image with the N areas in the texture enhanced image to obtain a first product;

respectively multiplying the weighting calculation weights respectively corresponding to the N areas in the basic fidelity image with the N areas in the basic fidelity image to obtain a second product;

and adding the first product and the second product to obtain an enhanced image corresponding to the reconstructed image.

8. The method according to claim 6, wherein the determining the weighted computation weights corresponding to the N regions in the texture enhanced image and the weighted computation weights corresponding to the N regions in the basic fidelity image according to the texture complexity weights corresponding to the N regions in the texture enhanced image respectively comprises:

determining texture complexity weights respectively corresponding to the N areas in the texture enhanced image as weighted calculation weights respectively corresponding to the N areas in the texture enhanced image;

and determining the difference value of the texture complexity weight corresponding to 1 and N areas in the texture enhanced image as the weighting calculation weight corresponding to the N areas in the basic fidelity image.

9. The method according to any one of claims 1 to 8, wherein the determining the texture complexity weight corresponding to each region in the intermediate image comprises:

and decoding texture complexity weights respectively corresponding to N areas in the intermediate image from the received code stream, wherein N is a positive integer.

10. The method according to claim 1, 2, 3, 4, 6, 7, 8 or 9, wherein the determining the texture complexity weight corresponding to each region in the intermediate image comprises:

according to a preset partition rule, dividing the reconstructed image and the intermediate image into N areas respectively, wherein N is a positive integer;

respectively determining the texture complexity of N areas in the reconstructed image;

and determining texture complexity weights corresponding to the N regions in the intermediate image respectively based on the texture complexity of the N regions in the reconstructed image.

11. An image processing apparatus, characterized in that the apparatus comprises:

the image acquisition module is used for acquiring a reconstructed image;

the image enhancement module is used for carrying out image enhancement on the reconstructed image to obtain an intermediate image;

a texture weight determining module, configured to determine a texture complexity weight corresponding to each region in the intermediate image, where the texture complexity weight is a number between 0 and 1;

and the texture attenuation module is used for respectively attenuating the texture intensity corresponding to each region in the intermediate image according to the texture complexity weight corresponding to each region in the intermediate image, so as to obtain an enhanced image corresponding to the reconstructed image.

12. The apparatus of claim 11, wherein the intermediate image is a residual image;

the texture attenuation module comprising:

the residual error updating module is used for multiplying the pixel value of each pixel point in the residual error image by the texture complexity weight corresponding to the region to which each pixel point belongs respectively to obtain a residual error updated image;

and the image generation module is used for generating the enhanced image according to the residual error updating image and the reconstructed image.

13. The apparatus of claim 12,

the image generation module is specifically configured to add the residual updated image and the reconstructed image to obtain the enhanced image.

14. The apparatus of claim 11, wherein the intermediate image is a texture enhanced image, the apparatus further comprising:

and the image fidelity module is used for performing image fidelity on the reconstructed image to obtain a basic fidelity image.

15. The apparatus of claim 14,

the texture weight determining module is specifically configured to divide the basic fidelity image and the texture enhanced image into N regions according to a preset partition rule, where N is a positive integer; respectively determining the texture complexity of N areas in the basic fidelity image; and determining texture complexity weights corresponding to the N regions in the texture enhanced image respectively based on the texture complexity of the N regions in the basic fidelity image.

16. The apparatus of claim 15, wherein the texture attenuation module comprises:

a weighted weight determining module, configured to determine, according to texture complexity weights corresponding to N regions in the texture enhanced image, weighted calculation weights corresponding to the N regions in the texture enhanced image and weighted calculation weights corresponding to the N regions in the basic fidelity image;

and the weighting calculation module is used for performing weighting calculation on the N areas in the texture enhanced image and the N areas in the basic fidelity image according to the weighting calculation weights respectively corresponding to the N areas in the texture enhanced image and the weighting calculation weights respectively corresponding to the N areas in the basic fidelity image to obtain an enhanced image corresponding to the reconstructed image.

17. The apparatus of claim 16,

the weighting calculation module is specifically configured to multiply the weighting calculation weights respectively corresponding to the N regions in the texture enhanced image by the N regions in the texture enhanced image, respectively, to obtain a first product; respectively multiplying the weighting calculation weights respectively corresponding to the N areas in the basic fidelity image with the N areas in the basic fidelity image to obtain a second product; and adding the first product and the second product to obtain an enhanced image corresponding to the reconstructed image.

18. The apparatus of claim 16,

the weighting weight determining module is specifically configured to determine texture complexity weights corresponding to the N regions in the texture enhanced image as weighting calculation weights corresponding to the N regions in the texture enhanced image; and determining the difference value of the texture complexity weight corresponding to 1 and N areas in the texture enhanced image as the weighting calculation weight corresponding to the N areas in the basic fidelity image.

19. The apparatus of any one of claims 11 to 18,

the texture weight determining module is specifically configured to decode texture complexity weights corresponding to N regions in the intermediate image from the received code stream, where N is a positive integer.

20. The apparatus of claim 11 or 12 or 13 or 14 or 16 or 17 or 18 or 19,

the texture weight determining module is specifically configured to divide the reconstructed image and the intermediate image into N regions according to a preset partition rule, where N is a positive integer; respectively determining the texture complexity of N areas in the reconstructed image; and determining texture complexity weights corresponding to the N regions in the intermediate image respectively based on the texture complexity of the N regions in the reconstructed image.

21. An electronic device, comprising:

a memory and a processor, the memory coupled with the processor;

the memory stores program instructions that, when executed by the processor, cause the electronic device to perform the image processing method of any one of claims 1 to 10.

22. A chip comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive signals from a memory of an electronic device and to transmit the signals to the processor, the signals including computer instructions stored in the memory; the computer instructions, when executed by the processor, cause the electronic device to perform the image processing method of any of claims 1 to 10.

23. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when run on a computer or a processor, causes the computer or the processor to execute an image processing method according to any one of claims 1 to 10.

24. A computer program product, characterized in that it contains a software program which, when executed by a computer or processor, causes the steps of the method of any one of claims 1 to 10 to be performed.

Images