CN116547696A - Image enhancement method and device - Google Patents

Abstract

An image enhancement method comprising: generating an input image by concatenating an original input image with a depth map. The method includes: encoding the input image by using an encoder to generate a bottleneck feature; injecting a disturbance vector into the bottleneck feature. The bottleneck features injected into the perturbation vector are fed to an image generator. The method further includes: at the image generator, generating an enhanced image according to the bottleneck feature and the disturbance vector. The method also includes receiving, at a discriminator, the enhanced image and a randomly selected clear image from the clear image dataset, and determining an image enhancement based on a comparison between the enhanced image and the randomly selected clear image Score. The method can provide multiple output images, reduce processing complexity, and improve the quality of the output images.

Description

A method and device for image enhancement

technical field

The present invention generally relates to the fields of computer vision and machine learning, and more particularly relates to a method and device for multimodal image enhancement in an unsupervised manner.

Background technique

Currently, processing images captured under different weather and lighting conditions (e.g., clear, rainy, foggy, overcast, blurry, or snowy conditions) for image enhancement is an outstanding technical challenge. For example, in autonomous driving, it is a great challenge to capture images under different weather and lighting conditions and perform image enhancement processing on such images in real-time or near real-time to facilitate safe autonomous driving. The reason is that due to different weather and lighting conditions, many features of such images that are helpful for perception often do not emerge.

Currently, certain methods have been proposed to process such images. For example, traditional methods use paired data of an input image and its corresponding output clear image to train a traditional model for mapping an input image to its corresponding output clear image. However, performing such supervised learning on traditional models using labeled datasets (e.g., input images and their corresponding output clean images) requires significant effort. Another traditional approach to image dehazing exists that is based on traditional atmospheric scattering models (i.e., physics-based models) instead of paired data used in supervised learning. However, traditional image dehazing methods are task-specific and difficult to generalize to other image enhancement tasks, such as image deblurring, low-light enhancement, etc. In addition, in traditional image enhancement methods, only one output image is obtained from the input image, which is less informative and less efficient, so its usefulness is limited. Since existing methods are complex, inefficient, and task-specific image enhancement methods with limited utility and unsuitable for overall image enhancement, there are studies on how sufficient, overall enhancements can be captured under different weather and lighting conditions for practical applications (such as Technical issues with images that promote safe autonomous driving).

Therefore, in light of the above discussion, there is a need to overcome the above-mentioned disadvantages associated with conventional image enhancement methods.

Contents of the invention

The present invention provides a multimodal image enhancement method and device to promote safe automatic driving. Since existing methods are complex, inefficient, and task-specific image enhancement methods with limited utility and unsuitable for overall image enhancement, the present invention provides a solution to the problem of how to adequately and overall enhance images captured under different weather and lighting conditions. Image of this existing problem scenario. The purpose of the present invention is to provide a solution to at least partially solve the problems encountered in the prior art, and provide a multi-modal image enhancement method and device, the multi-modal image enhancement can fully and overall enhance the image quality in different weather Images captured under and lighting conditions for use in various practical applications such as facilitating safe autonomous driving.

The objects of the invention are achieved by the solutions presented in the appended independent claims. Advantageous implementations of the invention are further defined in the dependent claims.

In one aspect, the present invention provides an image enhancement method, wherein the method comprises: generating an input image by concatenating the original input image with a sharpened attention map or a depth map. The method further includes generating bottleneck features by encoding the input image using an encoder. The method also includes injecting a perturbation vector into the bottleneck feature. The method also includes feeding the bottleneck feature injected into the perturbation vector to an image generator. The method further includes: at the image generator, generating an enhanced image according to the bottleneck feature and the disturbance vector. The method also includes receiving, at a discriminator, the enhanced image and a randomly selected clear image from the clear image dataset, and determining an image enhancement based on a comparison between the enhanced image and the randomly selected clear image Score.

The disclosed method generates a controllable solution space (or a set of multiple output images) for image enhancement, rather than just one non-optimal output image. By taking into account the uncertainty in the original input image, the controllable solution space can be more rationally created and allows searching for optimal solutions (eg, improved and enhanced output images). Thus, the disclosed method provides multiple output images and enables processing of an optimal output image, thereby exhibiting greater reliability and efficiency. The disclosed method uses either the sharpening attention map or the depth map and thus provides better guidance in order to generate a sharper and brighter output image. Furthermore, the disclosed method provides a unified approach for multiple image enhancement tasks (e.g., dehazing, deblurring, low-light enhancement, etc.), dealing with different weather and lighting conditions (such as foggy, clear, rainy, shady or snowy environments) for unsupervised, controllable overall image enhancement of such images. For example, in one exemplary practical application, the method makes distinct image features of such enhanced images that aid in perception, even under varying weather and lighting conditions, to facilitate safe automatic drive.

In an implementation manner, the method further includes: feeding back the discrimination score to the encoder and the image generator.

By feeding the discriminant score back to the encoder and the image generator, the image quality is gradually improved. In addition, based on a pre-trained convolutional neural network (Convolutional Neural Network, CNN) (such as a VGG neural network), the perceptual loss between the original input image and the enhanced image is calculated to keep the structure at the feature level.

In another implementation, the disturbance vector is sampled according to a Gaussian distribution.

Said sampling said perturbation vector according to a Gaussian distribution results in the generation of a multimodal output (or multiple output images), thereby creating a solution space of output images for improved image enhancement.

In another implementation manner, the perturbation vector is updated according to pre-trained network weights.

The perturbation vectors are used to determine the optimal output image (ie improved output) from the controllable solution space of the output images. During training, the perturbation vector has no predetermined network weights and acts as a random vector sampled according to a Gaussian distribution at each training step. However, after the training is complete, the perturbation vector is updated according to the pre-trained (i.e., fixed) network weights, and then adjusted (or fine-tuned) to search for an optimal solution (i.e., improved output image).

In another implementation, gradient descent is used to adjust the perturbation vector by reducing adversarial loss, Frechet Inception distance score or structural similarity index score.

Randomly initialize the perturbation vector, and then use gradient descent to adjust the perturbation vector by reducing the adversarial loss, the Frechet Inception distance score or the structural similarity index score, wherein the adjusted (or fine-tuned) The perturbation vector results in an improved or most visually pleasing image.

In another implementation, the discriminator is a gradient-based multi-tile discriminator.

Using the gradient-based multi-patch discriminator further improves the quality of the output image.

In another implementation, the discriminator includes at least the following three network branches: the first network branch is used to collect the output of the Gaussian blur generator; the second network branch is used to collect the identity image; the third network branch, It is used to collect the output of the Gaussian Laplacian fuzzy generator; wherein, after summing the outputs generated by the three branches after each convolutional layer, the result of the discriminator is obtained.

Using the three network branches of the discriminator provides improved lighting control and sharp edge information in the enhanced image.

In another aspect, the present invention provides an image enhancement apparatus, wherein the apparatus is configured to generate an input image by concatenating the original input image with a sharpened attention map or a depth map. The device is further configured to: generate bottleneck features by using an encoder to encode the input image. The apparatus is further configured to: inject a disturbance vector into the bottleneck feature. The apparatus is further configured to: feed the bottleneck feature injected into the perturbation vector to an image generator. The device is further configured to: at the image generator, generate an enhanced image according to the bottleneck feature and the disturbance vector. The apparatus is further configured to: at the discriminator, receive the enhanced image and a randomly selected clear image from the clear image dataset, and determine an image according to a difference between the enhanced image and the randomly selected clear image Boost score.

The device of the invention achieves all the advantages and effects of the method described.

In yet another aspect, the present invention provides a computer program comprising program code which, when executed by a computer, causes the computer to perform the method.

After executing the method described in the present invention, the computer realizes all the advantages and effects of the method.

In yet another aspect, the present invention provides an electronic component mounted on a vehicle, the electronic component operable to perform the method.

After carrying out the method according to the invention, the electronic component realizes all the advantages and effects of said method.

It should be understood that all above-described implementations may be combined.

It should be noted that all devices, elements, circuits, units and components described in this application can be implemented by software or hardware elements or any combination thereof. All steps described in this application as being performed by various entities and functions described to be performed by the various entities are intended to indicate that the corresponding entities are used to perform the corresponding steps and functions. Although in the description of the following specific embodiments, the specific functions or steps performed by the external entity are not reflected in the description of the specific detailed elements of the entity that performs the specific steps or functions, it should be clear to those skilled in the art that these methods and functions can be implemented through corresponding implemented by hardware or software elements or any combination thereof. It will be understood that various combinations of features of the invention are possible without departing from the scope of the invention as defined by the appended claims.

Additional aspects, advantages, features and objects of the present invention will become apparent from the accompanying drawings and detailed description of illustrative implementations interpreted in conjunction with the following appended claims.

Description of drawings

The foregoing summary, as well as the following detailed description of illustrative embodiments, may be better understood when read in conjunction with the accompanying drawings. In order to illustrate the present invention, exemplary structures of the present invention are shown in the drawings. However, the invention is not limited to the particular methodology and tools disclosed herein. Furthermore, those skilled in the art will appreciate that the drawings are not drawn to scale. Where possible, like elements have been given like numerals.

Embodiments of the present invention are now described, by way of example only, with reference to the following drawings, in which:

FIG. 1 shows a flowchart of an image enhancement method provided by an embodiment of the present invention;

Fig. 2 shows a block diagram of various exemplary components of a device provided by an embodiment of the present invention;

FIG. 3A shows a graphical representation of a learning (or training) image dehazing model provided by an embodiment of the present invention;

Fig. 3B shows a graphical representation of an encoder provided by an embodiment of the present invention;

Figure 3C shows a graphical representation of a densely connected block provided by an embodiment of the present invention;

FIG. 3D shows a graphical representation of an encoder-decoder structure with a Gaussian perturbation vector provided by an embodiment of the present invention;

FIG. 3E shows a graphical representation of a discriminator provided by an embodiment of the present invention;

Fig. 3F shows a pictorial representation of fine-tuning provided by an embodiment of the present invention to obtain an optimal output image;

Fig. 4 shows an illustration of an exemplary implementation scenario of the image enhancement method and apparatus provided by the embodiments of the present invention.

In the figures, an underlined number is used to denote an item on which the underlined number is located or an item adjacent to the underlined number. A non-underlined number refers to an item identified by a line connecting the non-underlined number with the item. When a number is not underlined and has an associated arrow, the ununderlined number is used to identify the general item that the arrow points to.

Detailed ways

The following detailed description illustrates embodiments of the invention and the manner in which these embodiments can be implemented. While certain modes of carrying out the invention have been disclosed, those skilled in the art will recognize that there may be other embodiments for carrying out or practicing the invention.

Fig. 1 shows a flowchart of an image enhancement method provided by an embodiment of the present invention. Referring to FIG. 1 , an image enhancement method 100 is shown. The method 100 includes steps 102 to 112 . The method 100 is performed by an apparatus such as that described in detail in FIG. 2 .

The present invention provides the image enhancement method 100, wherein the method 100 includes:

(i) Generate an input image by concatenating the original input image with a sharpened attention map or a depth map;

(ii) generating bottleneck features by encoding said input image using an encoder;

(iii) injecting a perturbation vector into the bottleneck feature;

(iv) feeding said bottleneck feature injected into said perturbation vector to an image generator;

(v) at the image generator, generate an enhanced image based on the bottleneck feature and the perturbation vector;

(vi) At a discriminator, receiving the enhanced image and a randomly selected clear image from the clear image dataset, and determining an image enhancement score based on a difference between the enhanced image and the randomly selected clear image.

The present invention provides the image enhancement method 100 . The method 100 is based on a generative adversarial network (GAN). The GAN can be used to generate a plurality of output images relative to an input image by using an image generator. The generated multiple output images exhibit higher visual quality and provide useful image features required for perception in real-time or near real-time. The GAN can also be configured to include a discriminator that enables the image generator to generate realistic outputs.

In step 102, the method 100 includes generating an input image by concatenating the original input image with a sharpened attention map or a depth map. The original input image corresponds to an image captured under one of the following conditions: foggy, foggy, clear, rainy, dark, snowy, or other adverse weather or lighting conditions. The original input image may also be referred to as a degraded image because the original input image does not reveal useful image features suitable for the actual application or use. For example, the raw input image may be captured by a camera mounted on an autonomous vehicle, which may not reveal the characteristics required for safe autonomous driving in different weather and lighting conditions. In another example, the raw input image includes one or more objects, which may be captured by a handheld device (such as a smartphone) in a low-light and rainy environment, where the one or more objects captured may not be clear . Therefore, in order to obtain useful features (such as object shapes and edges, etc.) from the original input image, the original input image is processed using the sharpened attention map or the depth map to generate the input image, thereby Acts as input to the encoder in the next operation. Compared to the original input image, the input image exhibits improved features, such as improved visual quality, object shape and edge detail. The sharpened attention map can be defined as a scalar matrix representing the relative importance of multi-layer activations at different two dimensional (2D) spatial locations with respect to a target task (eg, outputting a sharp image). The depth map may be defined as an image or image channel that provides information about the distance between the scene object surface and the viewpoint. In an implementation manner, the sharpening attention map or the depth map can be obtained from a pre-trained model, or can be jointly trained in a network in an unsupervised manner.

In step 104, the method 100 further includes: generating bottleneck features by using an encoder to encode the input image. The input image is sent to the encoder to generate the bottleneck feature. The bottleneck feature refers to an encoded feature map extracted from the input image, which has a smaller spatial size but more channels than the input image. For example, Figure 2 provides a detailed description of the encoder.

At step 106, the method 100 further includes: injecting a disturbance vector into the bottleneck feature. By injecting the perturbation vector into the bottleneck feature, a multimodal image output can be generated. The multimodal image output is generated by changing the appearance of the output image.

According to one embodiment, said perturbation vector is sampled according to a Gaussian distribution. In an implementation manner, the disturbance vector corresponds to a six-dimensional disturbance vector sampled according to the Gaussian distribution. In general, the Gaussian distribution can be defined as following a normal distribution with an equal number of measurements above and below the mean. The perturbation vectors are up-sampled by using a multi-layer perceptron (MLP) network. For example, Figure 3A details the MLP network. The upsampled perturbation vector is injected into the bottleneck feature by using an adaptive instance normalization (AdaIn) method. The adaptive instance normalization (AdaIn) method affects the image generation stage and generates multimodal output images. For example, FIG. 3A describes in detail the adaptive instance normalization (AdaIn) method.

According to one embodiment, said perturbation vector is updated according to pre-trained network weights. During training, the perturbation vector has no predetermined network weights and acts as a random vector sampled according to a Gaussian distribution at each training step. However, after the training is complete, the perturbation vector is updated according to the pre-trained (i.e., fixed) network weights, and then adjusted (or fine-tuned) to search for an optimal solution (i.e., improved output image). The pre-trained network weights are based on the pre-trained encoder, image generator, and discriminator network weights. After the training is completed, such network weights are fixed, and then the fine-tuning is performed on the perturbation vector.

According to one embodiment, the perturbation vector is adjusted by reducing the adversarial loss, the Frechet Inception distance score or the structural similarity index score using gradient descent. Randomly initialize the perturbation vector, and then use gradient descent to minimize the various loss, adjusts the perturbation vector. In general, the adversarial loss can be defined as the difference between the ground truth data (or raw or source data) and the generated data computed by using a generative adversarial network (GAN). The FID score can be defined as a metric that computes the distance between feature vectors computed for real images and generated images with the aid of a pre-trained Inception network. The SSI score may also be called a structural similarity index measure (SSIM). The SSIM can be defined as a method for predicting the perceived quality of digital television and movie pictures, as well as other types of digital images and video. The SSIM can be used to measure the similarity between two images. The updated perturbation vector (or resulting perturbation vector) eventually converges to a point where the updated perturbation vector can cooperate with a decoder (or image generator) to generate a visually pleasing output image. The updated perturbation vector (or resulting perturbation vector) may be applied to one or more images (eg, test images).

In step 108, the method 100 further includes: feeding the bottleneck feature injected into the perturbation vector to an image generator. The bottleneck feature injected into the perturbation vector is further fed to the image generator. The image generator can be used to generate multiple output images that improve visual quality and provide useful image features needed for perception in real-time or near real-time. In step 110, the method 100 further includes: at the image generator, generating an enhanced image according to the bottleneck feature and the disturbance vector. In one implementation manner, the image generator may be configured to use the bottleneck feature injected into the perturbation vector to generate the enhanced image (ie, the output image) in each iteration. In each iteration, using a different perturbation vector results in changes in aspects such as illumination or brightness control of the enhanced image. In this way, a Gaussian solution space for changing the appearance of the augmented image is created and enables control of the image augmentation output during the testing phase. Additionally, the enhanced image may be sent back to the encoder to force the encoder to generate the same encoded features (eg, another similar bottleneck feature) as the original input image. Therefore, there is an L1-norm feature reconstruction loss between these two encoded features (e.g., two bottleneck features).

In step 112, the method 100 further includes: at the discriminator, receiving the enhanced image and a randomly selected clear image from the clear image data set, and according to the enhanced image and the randomly selected clear image The comparison between them to determine the image enhancement score. The comparison is made in terms of the authenticity of the enhanced image relative to the randomly selected sharp image. According to the received enhanced image and a randomly selected clear image from the clear image dataset, the discriminator is used to determine whether the enhanced image is a fake image or a real clear image.

According to one embodiment, said discriminator is a gradient based multi-tile discriminator. The gradient-based multi-patch discriminator includes multiple network branches and thus improves output image quality.

According to an embodiment, the method 100 further includes: feeding back the discrimination score to the encoder and the image generator. By feeding the discriminant score back to the encoder and the image generator, the image quality is gradually improved. In addition, based on a pre-trained convolutional neural network (Convolutional Neural Network, CNN) (such as a VGG neural network), the perceptual loss between the original input image and the enhanced image is calculated to maintain the structure at the feature level.

According to one embodiment, the discriminator includes at least the following three network branches:

(a) a first network branch for collecting the Gaussian blur generator output;

(b) a second network branch for collecting identity images;

(c) the third network branch, used to collect the Gaussian Laplacian fuzzy generator output,

Wherein, after summing the outputs generated by the three branches after each convolutional layer, the result of the discriminator is obtained.

The Gaussian blur generator output is responsible for introducing an illumination distribution in the output image that is closer to the distribution of the target data (ie, the enhanced image). The identity image captured by the second network branch is similar to a standard discriminator (e.g., discriminator) generated images. A Laplacian of Gaussian (LoG) blur generator output is responsible for generating sharper edges in the output image that are closer to the distribution of the target data (ie, the enhanced image). The LoG can be defined as a two-dimensional isotropic measure of the second spatial derivative of an image. Or, in other words, the LoG highlights regions of the image with rapidly changing intensities, and is often used for edge detection. The sum of the outputs generated by the three branches after each convolutional layer is obtained from the discriminator to make true and false predictions for different tiles in the output image. In this way, the discriminator can improve the output image quality.

Thus, the method 100 generates a controllable solution space (or a set of multiple output images) for image enhancement, rather than generating only one non-optimal output image. By taking into account the uncertainty in the original input image, the controllable solution space can be more rationally created and allows searching for optimal solutions (eg, improved and enhanced output images). Thus, the method 100 provides multiple output images and enables processing of an optimal output image, thereby exhibiting greater reliability and efficiency. The method 100 uses the sharpening attention map or the depth map and thus provides better guidance in order to generate a sharper and brighter output image. In addition, the method 100 performs multiple image enhancement tasks (such as image defogging, image deblurring, and/or low-light enhancement) to overall enhance images under different weather and lighting conditions (such as foggy, clear, rainy, dark or images captured in a snowy environment). For example, in one exemplary practical application, the method 100 makes apparent various features of such enhanced images that facilitate perception, even under varying weather and lighting conditions, to facilitate safe automatic drive. In addition, the method 100 can reduce processing complexity and improve the quality of the output image.

Step 102 to step 112 are only illustrative, and other optional solutions can also be provided, without departing from the scope of claims herein, adding one or more steps, deleting one or more steps, or using different A sequence provides one or more steps.

Fig. 2 shows a block diagram of various exemplary components of a device provided by an embodiment of the present invention. Referring to FIG. 2 , a block diagram 200 of an apparatus 202 is shown. The apparatus 202 includes a sharpening attention map or depth map 204 , an encoder 206 , an image generator 208 , a memory 210 , a discriminator 212 and a processor 214 .

The apparatus 202 includes suitable logic, circuits, interfaces and/or codes for enhancing images. The device 202 is configured to execute the method 100 (in FIG. 1 ). Examples of the apparatus 202 include, but are not limited to, handheld devices or electronic devices or components that may be mounted on a vehicle (e.g., an autonomous vehicle or a semi-autonomous vehicle), a mobile device, a portable device, etc., that are operable to perform the described method 100.

The sharpening attention map or the depth map 204 is used to provide better guidance in order to generate output images with higher visual quality compared to traditional scattering models. The sharpening attention map or the depth map 204 may be a software program or a mathematical expression or application that may be installed in the device 202 .

The encoder 206 (also denoted Enc) includes appropriate logic, circuitry, interfaces and/or code that may be defined as a network (e.g., a convolutional neural network). network, CNN) or a recurrent neural network (RNN, etc.), which takes input data (e.g., images) and provides outputs based on feature maps or vectors or tensors representing latent information about said input data data (e.g. output image). Examples of the encoder 206 include, but are not limited to, recurrent neural networks, feedforward neural networks, deep belief networks and convolutional deep belief networks, self-organizing maps, deep Boltzmann machines, and stacked denoising autoencoders.

The image generator 208 (also denoted G) includes suitable logic, circuitry, interfaces, and/or code for generating one or more images that approximate the real data distribution. Enhanced clarity output image. In one implementation, the image generator 208 can also be defined as a network, and the network is used to reconstruct the input data (ie, the input image) according to the feature map or change the feature map to Different but related representations. The image generator 208 may also be called a decoder.

The memory 210 includes suitable logic, circuitry or interfaces for storing instructions executable by the processor 214 . The memory 210 can also be used to store sharp image data sets. Implementation examples of the memory 210 may include, but are not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Random Access Memory (Random Access Memory, RAM), Read Only Memory (Read Only Memory) , ROM), hard disk drive (Hard Disk Drive, HDD), flash memory, solid-state drive (Solid-State Drive, SSD) and/or CPU cache memory. The memory 210 may store an operating system or other program product (including one or more operating algorithms) to operate the apparatus 202 .

The discriminator 212 (also denoted D) includes suitable logic, circuitry, interface and/or code for An image enhancement score is determined by comparing the one or more enhanced sharp output images with a randomly selected sharp image from the sharp image dataset stored in the memory 210. In other words, checking or comparing the authenticity of said one or more enhanced sharp output images with respect to said randomly selected sharp images.

The processor 214 includes suitable logic, circuitry, interfaces and/or code for executing instructions stored in the memory 210 . In one example, the processor 214 may be a general purpose processor. Other examples of the processor 214 may include, but are not limited to, microprocessors, microcontrollers, complex instruction set computing (complex instruction set computing, CISC) processors, application-specific integrated circuit (application-specific integrated circuit, ASIC) processors , reduced instruction set (reduced instruction set, RISC) processor, very long instruction word (very long instruction word, VLIW) processor, central processing unit (central processing unit, CPU), state machine, data processing unit and other processors or Control circuit. Additionally, the processor 214 may refer to one or more individual processors, processing devices, or processing units that are part of a machine, such as the apparatus 202 .

In operation, said means 202 for image enhancement are provided, wherein said means 202 are configured to: generate an input image by concatenating an original input image with said sharpened attention map or said depth map 204 . The device 202 is further configured to: generate bottleneck features by using the encoder 206 to encode the input image. The device 202 is further configured to: inject a disturbance vector into the bottleneck feature. The device 202 is further configured to: feed the bottleneck feature injected into the disturbance vector to the image generator 208 . The device 202 is further configured to: generate an enhanced image at the image generator 208 according to the bottleneck feature and the disturbance vector. The device 202 is further configured to: at the discriminator 212, receive the enhanced image and a randomly selected clear image from the clear image data set, and according to the difference between the enhanced image and the randomly selected clear image difference to determine the image enhancement score.

According to one embodiment, said perturbation vector is sampled according to a Gaussian distribution. In an implementation manner, the disturbance vector is a six-dimensional disturbance vector sampled according to the Gaussian distribution.

According to one embodiment, said perturbation vector is updated according to pre-trained network weights. In another implementation manner, the perturbation vector is used in conjunction with the pre-trained network weights of the encoder 206 or the image generator 208 . In this implementation, said means 202 are configured to start processing from a random perturbation vector.

According to one embodiment, the perturbation vector is adjusted by reducing the adversarial loss, the Frechet Inception distance score or the structural similarity index score using gradient descent. In the case of the random perturbation vector, using the gradient descent, by minimizing the adversarial loss, the Frechet Inception Distance (FID) score or the Structural Similarity Index (SSI) Various losses such as scores update the value of the random perturbation vector.

According to one embodiment, the discriminator 212 is a gradient-based multi-tile discriminator. The discriminator 212 (or gradient-based multi-tile discriminator) includes multiple network branches and thus improves the output image quality.

According to one embodiment, the discriminator 212 includes three network branches:

(a) a first network branch for collecting the Gaussian blur generator output;

(b) a second network branch for collecting identity images;

(c) a third network branch for capturing the Laplacian of Gaussian of the output generated, wherein the discriminator is obtained after summing the outputs generated by the three branches after each convolutional layer 212 results. By using the first network branch, the second network branch and the third network branch, the discriminator 212 is used to make true and false predictions for different blocks in the output image.

According to one embodiment, a computer program is provided, including program code, which, when executed by a computer, causes the computer to execute the method 100 . Examples of the computer include, but are not limited to, a laptop computer, an electronic control unit (ECU) in a vehicle or onboard a vehicle, a desktop computer, a mainframe computer, a handheld computer, the processor 214 and other computing devices.

Thus, the apparatus 202 generates a controllable solution space (or a set of multiple output images) for image enhancement, rather than generating only one non-optimal output image. By taking into account the uncertainty in the original input image, the controllable solution space can be more rationally created and allows searching for optimal solutions (eg, improved and enhanced output images). Thus, the device 202 provides multiple output images and enables processing of an optimal output image, thereby exhibiting greater reliability and efficiency. The means 202 use the sharpening attention map or the depth map and thus provide better guidance in order to generate a sharper, brighter output image. In addition, the device 202 performs multiple image enhancement tasks (such as image defogging, image deblurring and/or low-light enhancement) to overall enhance images under different weather and lighting conditions (such as foggy, sunny, rainy, dark or images captured in a snowy environment). For example, in one exemplary practical application, the apparatus 202 makes distinct features of such enhanced images that facilitate perception, even under varying weather and lighting conditions, to facilitate safe automatic drive.

Fig. 3A shows a graphical representation of a learning (or training) image dehazing model provided by an embodiment of the present invention. FIG. 3A is described in conjunction with elements of FIGS. 1 and 2 . Referring to FIG. 3A, it is shown that the original input image 302, densely connected block 304, bottleneck feature 306, perturbation vector 308, multi-layer perceptron (multi-layer perceptron, MLP) network 310, residual block 312, output image (ie, enhanced sharp output image 314 ), another bottleneck feature 316 , and a graphical representation 300A of a convolutional network 318 .

The original input image 302 (also denoted as X _input ) corresponds to an image captured in a foggy environment during automatic driving. Therefore, the original input image 302 (ie, X _input ) may be referred to as a degraded image that does not reveal parameters or features for guiding safe automated driving. In the graphical representation 300A, an enhancement of the original input image 302 (eg, a blurred input image) is considered. However, the pictorial representation 300A is equally applicable to blurred images or images captured in foggy, clear, rainy, dark or snowy environments.

The sharpening attention map or the depth map 204 corresponds to a sharpening attention or depth estimation model that processes the original input image 302 (i.e., X _input ) and provides an input image (e.g., sharpening image). The input image (ie the sharpened image) exhibits high visual quality and thus provides better object shape and edge information present in the image. Thereafter, the input image (ie, the sharpened image) is encoded by using the encoder 206 (ie, Enc).

The densely connected block 304 (also denoted as DenseBlk) may be referred to as one or more convolutional layers in a convolutional neural network (CNN), which is used to utilize (ie, Enc) the received bottleneck features for image classification training. The one or more convolutional layers are connected (or convolved) to each other in a feed-forward manner via multiplication or dot product. The densely connected block 304 (ie, DenseBlk) is a fully connected layer, typically used for image classification tasks.

The bottleneck feature 306 (also denoted C) refers to the encoded feature map of the input image (encoded by the encoder 206), which has a smaller spatial size but with more channels. The bottleneck feature 306 (ie, C) may also be referred to as a bottleneck content feature, which represents the content feature of the input image in encoded form.

The disturbance vector 308 refers to a six-dimensional disturbance vector. In the graphical representation 300A, the perturbation vector 308 is sampled according to a Gaussian distribution. The perturbation vector 308 is up-sampled by using a multi-layer perceptron (MLP) network 310 . The MLP network 310 is defined as a type of feedforward artificial neural network (artificial neural network, ANN). The MLP network 310 includes multiple neural network layers (eg, an input layer, an output layer, and one or more hidden layers) of nonlinear activation nodes. The MLP network 310 is a fully connected network, so each node in one layer is connected with a certain weight to every other node in another layer. The perturbation vector 308 (ie, the upsampled perturbation vector) is injected into the bottleneck feature by using an adaptive instance normalization (AdaIn) method. The AdaIn method affects the image generation stage and generates a multimodal output image. The AdaIn method is commonly used in image style transfer and image generation tasks to change the appearance of the original input image 302 (ie, X _input ).

The residual block 312 (also denoted ResBlk) comprises two convolutional layers (eg, an input layer and an output layer) with skip connections between them allowing identity mapping. The residual block 312 (ie, ResBlk) is used to prevent vanishing gradients and provide better characterization of the bottleneck features.

Thereafter, the bottleneck features injected into the perturbation vector 308 (ie, the upsampled perturbation vector) are fed to the image generator 208 (ie, G). The image generator 208 (i.e., G) is configured to generate an enhanced clear output image 314 (also denoted as X _clear ) based on the bottleneck features injected into the upsampled perturbation vector 308 in each training iteration, and The enhanced clear output image 314 (ie, X _clear ) is added to the clear image dataset stored in the memory 210 .

Encoding the enhanced clear output image 314 (i.e., X _clear ) by using the encoder 206 (i.e., Enc ) and then processing the enhanced clear through the sharpening attention map or the depth map 204 Outputting an image 314 (ie, X _clear ), the further bottleneck feature 316 (also denoted C') is obtained. The other bottleneck feature 316 (ie, C') is used to be consistent with the bottleneck feature 306 (ie, C) through an L1 norm loss.

In addition, the discriminator 212 (ie, D) is configured to receive the enhanced clear output image 314 (ie, X _clear ) from the image generator 208 (ie, G) and to store in the memory 210 Randomly selected clear images from the clear image dataset of . The discriminator 212 (ie, D) is further configured to determine an image enhancement score based on a comparison between the enhanced clear output image 314 (ie, X _clear ) and the randomly selected clear image.

The convolutional network 318 corresponds to a VGG network (eg, VGG 16 or VGG 19 ), which is used for image classification and image feature detection. Furthermore, the original input image 302 (i.e., X _input ) and the enhanced clear output image 314 (i.e., X _clear ) are configured relative to the perceptual loss (i.e., X clear ) by using the pre-trained convolutional network. L _percep ) are kept consistent so that the image structure is kept at the feature level. As shown, the convolutional network 318 is independent of the overall framework and is only used to add additional losses to the overall framework.

In operation, the original input image 302 (i.e., X _input ) is concatenated with the sharpened attention map or the depth map 204 and passed to the encoder 206 (i.e., Enc) and the The dense connection block 304 (ie, DenseBlk) to obtain the bottleneck feature 306 (ie, C). The perturbation vector 308 is up-sampled by the MLP network 310 and integrated with the bottleneck feature 306 (ie, C) by the AdaIn method used in the MLP network 310 . After integrating the perturbation vector 308 with the bottleneck feature 306 (ie, C), the bottleneck feature 306 (ie, C) is fed to the image generator through the residual block 312 (ie, ResBlk) 208 (ie, G). The image generator 208 (ie, G) is configured to generate the enhanced clear output image 314 (ie, X _clear ) in each iteration. In each training (or learning) iteration of the image dehazing model, some parameters will be changed based on different perturbation vectors 308 sampled according to the Gaussian distribution, such as the enhanced clear output image 314 (i.e., X _clear ) lighting or brightness contrast. The different perturbation vectors 308 result in the creation of a Gaussian solution space for multiple output images for image enhancement, rather than just one output image, and enable processing of optimal output images such as the enhanced sharp output image 314 . In each training iteration, the enhanced clear output image 314 (i.e., X _clear ) is fed to the discriminator 212 (i.e., D) together with a randomly selected clear image from the clear image dataset to A determination is made as to whether the enhanced sharp output image 314 is a fake image or a real sharp image. Thereafter, the image generator 208 (ie, G) is used to improve itself by minimizing the adversarial loss of the discriminator 212 (ie, D). Furthermore, the enhanced clear output image 314 (i.e., X _clear ) is further concatenated with the sharpened attention map or the depth map 204 , and by using the encoder 206 (i.e., Enc ) and the The dense connection block 304 (ie, DenseBlk) encodes it to obtain the other bottleneck feature 316 (ie, C'). The other bottleneck feature 316 (ie, C′) is used to be consistent with the bottleneck feature 306 (ie, C) through an L1 norm loss (ie, C _recon ). Forcing the original input image 302 (ie, X _input ) and the enhanced clear output image 314 (ie, X _clear ) with respect to the perceptual loss (eg, L _percep ) to maintain the features of the original input image 302 (ie, X _input ) in the enhanced clear output image 314 (ie, X _clear ).

After performing multiple iterations to train the image defogging model, generate the output image (or enhanced image) controllable solution space, and search for the optimal output image from the output image (or enhanced image) controllable solution space . To fine-tune the optimal output image, two methods are used, which are described in detail, for example, in Fig. 3F.

In this embodiment, the graph representation 300A is used to learn (or train) the image dehazing model. In another embodiment, the graphical representation 300A may also be used for image deblurring or low light enhancement, or for enhancing images captured in foggy, clear, rainy, dark or snowy environments.

Fig. 3B shows a graphical representation of an encoder provided by an embodiment of the present invention. Figure 3B is described in conjunction with elements of Figures 1, 2 and 3A. Referring to FIG. 3B , a pictorial representation 300B of the encoder 206 (of FIG. 2 ) is shown. The encoder 206 includes an Inception residual block 320 . The Inception residual block 320 includes multiple 1×1 convolutional blocks 320A and multiple 3×3 convolutional blocks 320B.

The Inception residual block 320 can be defined as a convolution block, and the convolution block combines a plurality of convolution branches, and the plurality of convolution branches can capture various features of different block sizes of the image (as shown in the figure represents the bottleneck feature used in table 300A).

In contrast to traditional residual blocks, the encoder 206 includes the Inception residual block 320 and thus provides an improved encoded feature map combining local image content and global image content. Since the Inception residual block 320 of the encoder 206 includes the plurality of 1×1 convolutional blocks 320A and the plurality of 3×3 convolutional blocks 320B, the improved encoding feature map is obtained.

Figure 3C shows a graphical representation of a densely connected block provided by an embodiment of the present invention. Figure 3C is described in conjunction with elements of Figures 1, 2, 3A and 3B. Referring to FIG. 3C , a graphical representation 300C of the densely connected block 304 (of FIG. 3A ) is shown. The densely connected block 304 includes the plurality of 3x3 convolutional blocks 320B.

In order to obtain the bottleneck feature 306 (i.e., C) with a more meaningful and robust feature representation of the bottleneck feature received from the encoder 206 (i.e., Enc), the conventional residual The poor block is replaced with the densely connected block 304 (ie, DenseBlk). In the densely connected block 304 (ie, DenseBlk), each of the plurality of 3x3 convolutional blocks 320B is connected to each other and thus provides an improved feature representation of the bottleneck feature.

FIG. 3D shows a graphical representation of an encoder-decoder structure with a Gaussian perturbation vector provided by an embodiment of the present invention. Figure 3D is described in conjunction with elements of Figures 1, 2, 3A, 3B and 3C. Referring to FIG. 3D , a graphical representation 300D of an encoder-decoder structure with a Gaussian perturbation vector 322 is shown. The encoder-decoder structure with Gaussian perturbation vector 322 includes the encoder 206 and the image generator 208 (also referred to as a decoder). The encoder-decoder structure with Gaussian perturbation vector 322 also includes the densely connected block 304 , the bottleneck feature 306 , the perturbation vector 308 , the MLP network 310 and the residual block 312 .

The conventional encoder-decoder structure comprises the conventional residual block and provides only one non-optimal output image, and is therefore not the preferred structure in comparison. However, the encoder-decoder structure with Gaussian perturbation vector 322 includes the densely connected block 304 (i.e., DenseBlk), the perturbation vector 308 and the MLP network 310, and provides multiple The output image (or multimodal output image) Gaussian solution space, rather than providing only one output image, and enables processing of an optimal output image (such as the enhanced sharp output image 314).

Fig. 3E shows a graphical representation of the discriminator provided by the embodiment of the present invention. Figure 3E is described in conjunction with elements of Figures 1, 2, 3A, 3B, 3C and 3D. Referring to FIG. 3E , a graphical representation of the discriminator 212 (of FIG. 2 ) is shown. The discriminator 212 (ie, D) includes a first network branch 212A, a second network branch 212B and a third network branch 212C. The discriminator 212 (ie, D) also includes a first convolutional layer 324A, a second convolutional layer 324B, a third convolutional layer 324C, and an output image 328 .

Each of the first convolutional layer 324A, the second convolutional layer 324B, and the third convolutional layer 324C may also be called a convolutional neural network (CNN). Generally, the convolutional neural network (CNN) can be defined as a network of highly interconnected processing elements. Optionally, each element is associated with a local memory (ie, the memory 210 ) and is used for image recognition and processing (eg, for processing the enhanced sharp output image 314 ).

The first network branch 212A is used to acquire a Gaussian blur generator output 326A responsible for introducing an illumination distribution in the enhanced sharp output image 314 that is closer to the target data distribution.

The second network branch 212B is used to acquire an identity image 326B similar to a standard discriminator (e.g., discriminator) generated images.

The third network branch 212C is used to acquire a Laplacian of Gaussian (Laplacian of Gaussian, LoG) blur generator output 326C, which is responsible for generating in the enhanced sharp output image 314 The sharper edges of the data distribution closer to the target.

Thereafter, after each convolutional layer (ie, the first convolutional layer 324A, the second convolutional layer 324B and the third convolutional layer 324C), the three network branches (ie, the After summing the outputs (ie, 326A, 326B, 326C) generated by the first network branch 212A, the second network branch 212B, and the third network branch 212C), the discriminator 212 (ie, D) The output image 328 is obtained in . Based on the enhanced clear output image 314 and the different tiles formed in the output image 328 , it is perceived whether the output image 328 is a real image or a fake image. The real image refers to an image that is similar to an input image such as the enhanced sharp output image 314 (also referred to as an enhanced image) in all image features. The fake image can be described as an image that can be artificially generated using software tools. In this way, the discriminator 212 can improve the output image quality compared to conventional discriminators that use only one network branch to generate the output image, thus exhibiting lower image quality.

FIG. 3F shows a pictorial representation of fine-tuning to obtain an optimal output image provided by an embodiment of the present invention. Figure 3F is described in conjunction with elements of Figures 1, 2, 3A, 3B, 3C, 3D, and 3E. Referring to FIG. 3F , there is shown a pictorial representation 300F of fine-tuning to obtain an optimal output image from a set of multiple sharp output images.

In said graphical representation 300F, two methods may be used to obtain said optimal output image from said set of multiple sharp output images which may be obtained by using said graphical representation 300A generated.

In a first method, the perturbation vector 308 is sampled according to the Gaussian distribution. In this method, a grid search is employed and the image with the best visual quality is checked by interpolating the values of each two dimensions in the Gaussian perturbation. The corresponding perturbation vectors are applied to all test images.

In a second method, the perturbation vector is updated 308 according to pre-trained network weights. In this approach, the weights associated with the encoder 206, the image generator 208, the densely connected block 304, the bottleneck features 306, the residual block 312 and the discriminator 212 have Fixed value. Thereafter, starting from a random perturbation vector, and using the gradient descent, by minimizing the adversarial loss, the Frechet Inception Distance (FID) score or the Structural Similarity Index (SSI) Various losses such as scores update the value of the random perturbation vector. The updated perturbation vector (or resulting perturbation vector) eventually converges to a point where the updated perturbation vector can cooperate with the image generator 208 to generate a visually pleasing output image. The updated perturbation vector (or resulting perturbation vector) may also be applied to one or more test images.

Fig. 4 shows an illustration of an exemplary implementation scenario of the image enhancement method and apparatus provided by the embodiments of the present invention. Figure 4 is described in conjunction with elements in Figures 1, 2, 3A-3F. Referring to FIG. 4 , there is shown an exemplary scenario 400 describing a practical application of the disclosed method and apparatus ( FIGS. 1 and 2 ) in the field of autonomous driving. In the exemplary scene 400, a vehicle 402 is shown moving along a road section. Also shown are electronic components 404 and one or more image capture devices, such as image capture device 406 mounted on the vehicle 402 . It should be appreciated that the vehicle 402 may include many other known components commonly used in autonomous vehicles, which are omitted here for the sake of brevity. For example, the vehicle 402 may include a battery for powering the image capture device 406 and the electronic components 404 .

In the example scenario 400, the vehicle 402 may be an autonomous vehicle or a semi-autonomous vehicle. The electronic components 404 may include suitable logic, circuits, interfaces and/or codes for performing multimodal image enhancement that is substantially, Overall enhancement of images captured under different weather and lighting conditions. For example, the electronic components 404 are configured to process and enhance the image capture device in real-time or near-real-time by performing various image enhancement tasks in a holistic manner, such as image defogging, image deblurring, and overall low-light enhancement. 406 Images captured under different weather and lighting conditions, such as foggy, clear, rainy, shady or snowy environments, to facilitate safe autonomous driving. Or, in other words, the electronics 404 make apparent many of the features of such enhanced images that facilitate perception of the real environment surrounding the vehicle 40, even under varying weather and lighting conditions . This in turn enables the vehicle 402 to safely drive itself. Examples of the electronic component 404 include, but are not limited to, an electronic control unit (ECU), an on-board device, an on-board computer, or other electronic components for the vehicle 402 . The electronic component 404 may correspond to the apparatus 202 (in FIG. 2 ), wherein the electronic component 404 is configured to perform the method 100 (in FIG. 1 ).

The electronics 404 may be used to perform various image enhancement tasks while the vehicle 402 is driving. Said electronic components 404 may be adapted to use said perturbation vector 308 sampled according to said Gaussian distribution, said perturbation vector 308 further resulting in the generation of sharp output images with respect to one original input image (i.e. degraded image), And thus an optimal output image (eg enhanced and improved image) may be selected from said plurality of clear output images. Accordingly, the electronics 404 provide the optimal output image (ie, the enhanced and improved image) for perception and enable the vehicle 402 to drive reliably and safely. In addition, the electronic component 404 can be configured to use the discriminator 212 (ie, the gradient-based multi-tile discriminator), and the discriminator 212 can further improve the visual quality of the plurality of clear output images.

In another implementation scenario, the apparatus 202 may be implemented as a handheld device operable to execute the method 100 (in FIG. 1 ). In one example, the handheld device may be a smartphone capable of substantially processing one or more images captured under different weather and lighting conditions using the method 100 to generate an enhanced image, as perceived by the human eye as high Quality, real, photo-like sharp output images. The method 100 enables the handheld device to predict the "realness" or "fakeness" of different tiles in the output image, thereby improving output image quality.

Modifications may be made to the embodiments of the invention described above without departing from the scope of the invention as defined in the appended claims. Expressions such as "comprises", "combines", "has", "is", etc. used to describe and claim the present invention are intended to be interpreted in a non-exclusive manner, ie to allow the presence of items, components or elements not explicitly described. References to the singular should also be construed as relating to the plural. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments and/or to exclude the incorporation of features of other embodiments. The word "optionally" is used herein to mean "provided in some embodiments and not provided in other embodiments". It is to be appreciated that features of the invention which are, for clarity, described in the context of separate embodiments may also be provided in combination in a single embodiment. Conversely, various features of the invention which are, for brevity, described in the context of a single embodiment, may also be provided separately, in any suitable combination or as any other described embodiment of the invention.

Claims (15)

1. An image enhancement method (100), the method comprising (100):

(i) Generating an input image by concatenating an original input image (302) with a sharpening attention map or depth map (204);

(ii) -generating bottleneck characteristics by encoding the input image using an encoder (206);

(iii) Injecting a disturbance vector (308) into the bottleneck feature;

(iv) Feeding the bottleneck characteristics injected into the disturbance vector (308) to an image generator (208);

(v) Generating, at the image generator (208), an enhanced image from the bottleneck characteristics and the disturbance vector (308);

(vi) At a arbiter (212), the enhanced image and a randomly selected sharp image from a sharp image dataset are received, and an image enhancement score is determined from a difference between the enhanced image and the randomly selected sharp image.

2. The method (100) of claim 1, wherein the method (100) further comprises: the discrimination score is fed back to the encoder (206) and the image generator.

3. The method (100) of claim 1, wherein the disturbance vector (308) is sampled according to a gaussian distribution.

4. The method (100) of claim 1, wherein the perturbation vector (308) is updated according to pre-trained network weights.

5. The method (100) of claim 4, wherein the disturbance vector (308) is adjusted using gradient descent by reducing a challenge loss, a Frechet Inception distance score, or a structural similarity index score.

6. The method (100) of claim 1, wherein the arbiter (212) is a gradient-based multi-tile arbiter.

7. A method (100) according to claim 3, wherein the arbiter (212) comprises at least three network branches:

(a) A first network branch (212A) for collecting gaussian blur generator output (326A);

(b) -a second network branch (212B) for acquiring an identity image (326B);

(c) A third network branch (212C) for collecting a Gaussian Laplacian blur generator output (326C),

wherein the result of the arbiter (212) is obtained after summing the outputs generated by the three branches after each convolution layer.

8. An image enhancement device (202), characterized in that the device (202) is adapted to:

(i) Generating an input image by concatenating an original input image (302) with a sharpening attention map or depth map (204);

(ii) -generating bottleneck characteristics by encoding the input image using an encoder (206);

(iii) Injecting a disturbance vector (308) into the bottleneck feature;

(iv) Feeding the bottleneck characteristics injected into the disturbance vector (308) to an image generator (208);

(v) Generating, at the image generator (208), an enhanced image from the bottleneck characteristics and the disturbance vector (308);

(vi) At a arbiter (212), the enhanced image and a randomly selected sharp image from a sharp image dataset are received, and an image enhancement score is determined from a comparison between the enhanced image and the randomly selected sharp image.

9. The apparatus (202) of claim 8, wherein the disturbance vector (308) is sampled according to a gaussian distribution.

10. The apparatus (202) of claim 8, wherein the disturbance vector (308) is updated according to pre-trained network weights.

11. The apparatus (202) of claim 10, wherein the disturbance vector (308) is adjusted using gradient descent by reducing a challenge loss, a Frechet Inception distance score, or a structural similarity index score.

12. The apparatus (202) of claim 8, wherein the arbiter (212) is a gradient-based multi-tile arbiter.

13. The apparatus (202) of claim 8, wherein the arbiter (212) comprises three network branches:

(a) A first network branch (212A) for collecting gaussian blur generator output (326A);

(b) -a second network branch (212B) for acquiring an identity image (326B);

(c) A third network branch (212C) for collecting the Gaussian Laplacian of the generated output (326C),

wherein the result of the arbiter (212) is obtained after summing the outputs generated by the three branches after each convolution layer.

14. A computer program characterized by comprising program code which, when executed by a computer, causes the computer to perform the method (100) according to claim 1.

15. An electronic component (404) mounted on a vehicle (402), characterized in that the electronic component (404) is operable to perform the method (100) according to claim 1.