CN111724309B - Image processing method and device, neural network training method, storage medium - Google Patents

Abstract

An image processing method, an image processing apparatus, a training method of a neural network, and a storage medium. The image processing method comprises the following steps: receiving a first feature image; performing at least one multi-scale cyclic sampling process on the first characteristic image; the multi-scale cyclic sampling process comprises a first hierarchical sampling process and a second hierarchical sampling process, the first hierarchical sampling process comprises a first downsampling process, a first upsampling process and a first residual link addition process which are sequentially executed, the second hierarchical sampling process is nested between the first downsampling process and the first upsampling process, and the second hierarchical sampling process comprises a second downsampling process, a second upsampling process and a second residual link addition process which are sequentially executed. The image processing method can carry out image enhancement processing on the input image with low quality, and can greatly improve the quality of the output image by repeatedly sampling on a plurality of scales to obtain higher image fidelity.

Description

Image processing method and device, neural network training method, storage medium

technical field

Embodiments of the present disclosure relate to an image processing method, an image processing device, a neural network training method, and a storage medium.

Background technique

Currently, deep learning techniques based on artificial neural networks have made great progress in areas such as image classification, image capture and search, facial recognition, age, and speech recognition. The strength of deep learning is that it can take advantage of general structures to solve very different technical problems with relatively similar systems. Convolutional Neural Network (CNN) is an artificial neural network developed in recent years and has attracted widespread attention. CNN is a special image recognition method, which is a very effective network with forward feedback. Now, the application range of CNN is not limited to the field of image recognition, but can also be applied in face recognition, text recognition, image processing and other application directions.

Contents of the invention

At least one embodiment of the present disclosure provides an image processing method, including: receiving a first feature image; and performing at least one multi-scale circular sampling process on the first feature image;

Wherein, the multi-scale circular sampling processing includes nested first-level sampling processing and second-level sampling processing, and the first-level sampling processing includes first down-sampling processing, first up-sampling processing and first residual linking Addition processing, wherein the first down-sampling processing performs down-sampling processing based on the input of the first-level sampling processing to obtain a first down-sampling output, and the first up-sampling processing performs up-sampling based on the first down-sampling output processing to obtain a first up-sampling output, the first residual chain addition process performs first residual chain addition on the input of the first-level sampling process and the first up-sampling output, and then adds the first A residual chain addition result is used as the output of the first-level sampling process; the second-level sampling process is nested between the first down-sampling process and the first up-sampling process, receiving the first The down-sampling output is used as an input of the second-level sampling process, and the output of the second-level sampling process is provided as an input of the first up-sampling process, so that the first up-sampling process performs up-sampling processing based on the first down-sampling output ; The second-level sampling process includes a second down-sampling process, a second up-sampling process, and a second residual chain addition process, wherein the second down-sampling process is performed based on the input of the second-level sampling process The down-sampling process obtains a second down-sampling output, the second up-sampling process performs up-sampling processing based on the second down-sampling output to obtain a second up-sampling output, and the second residual chain addition process combines the first The input of the two-level sampling process and the output of the second upsampling are subjected to a second residual link addition, and then the result of the second residual link addition is used as an output of the second level sampling process.

For example, in the image processing method provided by an embodiment of the present disclosure, the size of the output of the first up-sampling process is the same as the size of the input of the first down-sampling process; the output of the second up-sampling process is The size is the same as that of the input to the second downsampling process.

For example, in the image processing method provided by an embodiment of the present disclosure, the multi-scale circular sampling process further includes a third-level sampling process, and the third-level sampling process is nested in the second down-sampling process and the Between the second up-sampling process, the second down-sampling output is received as the input of the third-level sampling process, and the output of the third-level sampling process is provided as the input of the second up-sampling process, so that the second up-sampling processing and performing upsampling processing based on the second downsampling output; the third level sampling processing includes third downsampling processing, third upsampling processing and third residual link addition processing, wherein the third downsampling The sampling process performs down-sampling processing based on the input of the third-level sampling processing to obtain a third down-sampling output, and the third up-sampling process performs up-sampling processing based on the third down-sampling output to obtain a third up-sampling output, so The third residual link addition process performs third residual link addition on the input of the third-level sampling process and the third upsampling output, and then uses the third residual link addition result as The output of the third-level sampling process.

For example, in the image processing method provided by an embodiment of the present disclosure, the multi-scale cyclic sampling process includes the second-level sampling process executed sequentially multiple times, and the first time the second-level sampling process receives the first The down-sampling output is used as the input of the second-level sampling process for the first time, and each second-level sampling process except the first-time second-level sampling process receives the previous second-level sampling process The output of the processing is used as the input of the second-level sampling process this time, and the output of the last second-level sampling process is used as the input of the first up-sampling process.

For example, in the image processing method provided by an embodiment of the present disclosure, the at least one multi-scale cyclic sampling process includes multiple sequential executions of the multi-scale cyclic sampling process, and the input of each multi-scale cyclic sampling process is taken as The input of the first-level sampling process in the multi-scale circular sampling process this time, the output of the first-level sampling process in the multi-scale circular sampling process each time is used as the multi-scale circular sampling process this time The output of the processing; the first multi-scale cyclic sampling process receives the first feature image as the input of the first multi-scale cyclic sampling process, and each process except the first multi-scale cyclic sampling process The multi-scale circular sampling process receives the output of the previous multi-scale circular sampling process as the input of the multi-scale circular sampling process this time, and the output of the last multi-scale circular sampling process is used as the at least one multi-scale circular sampling process. The output of the scaling cycle sampling process.

For example, in the image processing method provided in an embodiment of the present disclosure, the multi-scale circular sampling processing further includes: during the first down-sampling processing, the first up-sampling processing, the second down-sampling processing, and After the second upsampling process, perform instance normalization processing or layer normalization processing on the first downsampling output, the first upsampling output, the second downsampling output, and the second upsampling output respectively .

For example, the image processing method provided by an embodiment of the present disclosure further includes: using a first convolutional neural network to perform the multi-scale circular sampling process; wherein, the first convolutional neural network includes: a first meta-network for Executing the first-level sampling process; a second meta-network configured to perform the second-level sampling process.

For example, in the image processing method provided in an embodiment of the present disclosure, the first meta-network includes: a first sub-network configured to perform the first down-sampling process; a second sub-network configured to perform the first sub-network An up-sampling process; the second meta-network includes: a third sub-network configured to perform the second down-sampling process; a fourth sub-network configured to perform the second up-sampling process.

For example, in the image processing method provided in an embodiment of the present disclosure, each of the first subnetwork, the second subnetwork, the third subnetwork, and the fourth subnetwork includes a convolutional layer, One of residual network and dense network.

For example, in the image processing method provided by an embodiment of the present disclosure, each of the first sub-network, the second sub-network, the third sub-network and the fourth sub-network includes instance normalization A layer or a layer normalization layer for performing an instance normalization process, the layer normalization layer for performing a layer normalization process.

For example, the image processing method provided by an embodiment of the present disclosure further includes: acquiring an input image; using an analysis network to convert the input image into the first feature image; and using a synthesis network to process the output of the at least one multi-scale cyclic sampling process Convert to output image.

At least one embodiment of the present disclosure also provides a neural network training method, wherein the neural network includes: an analysis network, a first sub-neural network, and a synthesis network, and the analysis network processes an input image to obtain a first feature image, the first sub-neural network performs at least one multi-scale cyclic sampling process on the first feature image to obtain a second feature image, and the synthesis network processes the second feature image to obtain an output image;

The training method includes: obtaining a training input image; using the analysis network to process the training input image to provide a first training feature image; using the first sub-neural network to process the first training feature image. The at least one multi-scale cyclic sampling process to obtain a second training feature image; use the synthesis network to process the second training feature image to obtain a training output image; based on the training output image, calculate the A loss value of the neural network; and modifying parameters of the neural network according to the loss value;

Wherein, the multi-scale circular sampling processing includes nested first-level sampling processing and second-level sampling processing, and the first-level sampling processing includes sequentially executed first down-sampling processing, first up-sampling processing, and first Residual link addition processing, wherein the first down-sampling processing is based on the input of the first-level sampling processing to perform down-sampling processing to obtain a first down-sampling output, and the first up-sampling processing is based on the first down-sampling output performing an upsampling process to obtain a first upsampling output, the first residual linking addition process performs a first residual linking addition on the input of the first level sampling process and the first upsampling output, and then The result of the first residual chain addition is used as the output of the first-level sampling process; the second-level sampling process is nested between the first down-sampling process and the first up-sampling process, receiving the The first down-sampling output is used as the input of the second-level sampling process, and the output of the second-level sampling process is provided as the input of the first up-sampling process, so that the first up-sampling process is performed based on the first down-sampling output Up-sampling processing; the second-level sampling processing includes second down-sampling processing, second up-sampling processing, and second residual chain addition processing performed in sequence, wherein the second down-sampling processing is based on the second The input of the hierarchical sampling process is subjected to down-sampling processing to obtain a second down-sampling output, and the second up-sampling processing is performed based on the second down-sampling output to perform up-sampling processing to obtain a second up-sampling output, and the second residual link is related to The addition process performs the second residual link addition on the input of the second level sampling process and the second upsampling output, and then uses the second residual link addition result as the second level The output of the sample processing.

For example, in the training method provided by an embodiment of the present disclosure, the size of the output of the first up-sampling process is the same as the size of the input of the first down-sampling process; the size of the output of the second up-sampling process is Same size as the input to the second downsampling process.

For example, in the training method provided in an embodiment of the present disclosure, the first sub-neural network includes: a first meta-network, configured to perform the first-level sampling process; a second meta-network, configured to perform the second Two-level sampling processing.

For example, in the training method provided in an embodiment of the present disclosure, the first meta-network includes: a first sub-network for performing the first down-sampling process; a second sub-network for performing the first Up-sampling processing; the second meta-network includes: a third sub-network configured to execute the second down-sampling processing; a fourth sub-network configured to execute the second up-sampling processing.

For example, in the training method provided in an embodiment of the present disclosure, each of the first sub-network, the second sub-network, the third sub-network and the fourth sub-network includes a convolutional layer, One of residual network and dense network.

For example, in the training method provided by an embodiment of the present disclosure, each of the first sub-network, the second sub-network, the third sub-network and the fourth sub-network includes an instance normalization layer or a layer normalization layer, wherein the instance normalization layer is used to perform instance normalization on the first downsampled output, the first upsampled output, the second downsampled output, and the second upsampled output, respectively, The layer normalization layer is used to respectively perform layer normalization processing on the first downsampled output, the first upsampled output, the second downsampled output, and the second upsampled output.

At least one embodiment of the present disclosure further provides an image processing apparatus, including: a memory for non-transitory storage of computer-readable instructions; and a processor for executing the computer-readable instructions, the computer-readable instructions being The processor executes the image processing method provided by any embodiment of the present disclosure when running.

At least one embodiment of the present disclosure further provides a storage medium that stores computer-readable instructions in a non-transitory manner. When the computer-readable instructions are executed by a computer, the instructions of the image processing method provided by any embodiment of the present disclosure can be executed.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present disclosure more clearly, the accompanying drawings of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description only relate to some embodiments of the present disclosure, rather than limiting the present disclosure .

In order to illustrate the technical solutions of the embodiments of the present disclosure more clearly, the accompanying drawings of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description only relate to some embodiments of the present disclosure, rather than limiting the present disclosure .

Fig. 1 is a schematic diagram of a convolutional neural network;

FIG. 2A is a schematic structural diagram of a convolutional neural network;

2B is a schematic diagram of a working process of a convolutional neural network;

FIG. 3 is a flowchart of an image processing method provided by an embodiment of the present disclosure;

FIG. 4A is a schematic flowchart corresponding to multi-scale circular sampling processing in the image processing method shown in FIG. 3 provided by an embodiment of the present disclosure;

FIG. 4B is a schematic flowchart corresponding to multi-scale circular sampling processing in the image processing method shown in FIG. 3 provided by another embodiment of the present disclosure;

FIG. 4C is a schematic flowchart corresponding to multi-scale circular sampling processing in the image processing method shown in FIG. 3 provided by another embodiment of the present disclosure;

FIG. 4D is a schematic flowchart corresponding to multi-scale circular sampling processing in the image processing method shown in FIG. 3 provided by another embodiment of the present disclosure;

FIG. 5 is a flowchart of an image processing method provided by another embodiment of the present disclosure;

FIG. 6A is a schematic diagram of an input image;

FIG. 6B is a schematic diagram of an output image obtained by processing the input image shown in FIG. 6A according to an image processing method provided by an embodiment of the present disclosure;

FIG. 7A is a schematic structural diagram of a neural network provided by an embodiment of the present disclosure;

FIG. 7B is a flowchart of a neural network training method provided by an embodiment of the present disclosure;

FIG. 7C is a schematic architectural block diagram of training the neural network shown in FIG. 7A corresponding to the training method shown in FIG. 7B provided by an embodiment of the present disclosure;

FIG. 8 is a schematic block diagram of an image processing device provided by an embodiment of the present disclosure; and

Fig. 9 is a schematic diagram of a storage medium provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present disclosure. Apparently, the described embodiments are some of the embodiments of the present disclosure, not all of them. Based on the described embodiments of the present disclosure, all other embodiments obtained by persons of ordinary skill in the art without creative effort fall within the protection scope of the present disclosure.

Unless otherwise defined, the technical terms or scientific terms used in the present disclosure shall have the usual meanings understood by those skilled in the art to which the present disclosure belongs. "First", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Comprising" or "comprising" and similar words mean that the elements or items appearing before the word include the elements or items listed after the word and their equivalents, without excluding other elements or items. Words such as "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "Down", "Left", "Right" and so on are only used to indicate the relative positional relationship. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

The present disclosure is described below through several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and known components may be omitted. When any part of an embodiment of the present disclosure appears in more than one drawing, that part is represented by the same or similar reference numeral in each drawing.

Image enhancement is one of the research hotspots in the field of image processing. Due to the limitations of various physical factors in the image acquisition process (for example, the size of the image sensor of the mobile phone camera is too small and other software and hardware limitations, etc.) and the interference of environmental noise, the image quality will be greatly reduced. The purpose of image enhancement is to improve the gray histogram of the image and the contrast of the image through image enhancement technology, so as to highlight the details of the image and improve the visual effect of the image.

Image enhancement using deep neural network is an emerging technology with the development of deep learning technology. For example, based on convolutional neural networks, low-quality photos (input images) taken by mobile phones can be processed to obtain high-quality output images, which can be close to the quality of digital single-lens reflex cameras (Digital Single LensReflex Camera, Often referred to as DSLR, also referred to as digital single-lens reflex camera) the quality of photos taken. For example, a peak signal to noise ratio (Peak Signal to Noise Ratio, PSNR) index is commonly used to characterize image quality, wherein a higher PSNR value indicates that the image is closer to a real photo taken by a digital single-lens reflex camera.

For example, Andrey Ignatov et al. proposed a convolutional neural network image enhancement method, see literature, Andrey Ignatov, Nikolay Kobyshev, Kenneth Vanhoey, Radu Timofte, Luc VanGool, DSLR-Quality Photos on Mobile Devices with Deep ConvolutionalNetworks. arXiv: 1704.02470v2[cs.CV], 5 September 2017. This document is hereby incorporated by reference in its entirety as part of this application. This method mainly uses convolutional layers, batch normalization layers and residual connections to construct a single-scale convolutional neural network, which can be used to convert input low-quality images (for example, low contrast, underexposed images or overexposed Excessive, the entire image is too dark or too bright, etc.) processed into a higher quality image. Using color loss, texture loss and content loss as the loss function in training can achieve better processing results.

At least one embodiment of the present disclosure provides an image processing method, an image processing device, a neural network training method, and a storage medium. This image processing method proposes a multi-scale cyclic sampling method based on convolutional neural network. By repeatedly sampling on multiple scales to obtain higher image fidelity, the quality of the output image can be greatly improved. It is suitable for image processing. Offline applications such as batch processing with high quality requirements.

Initially, Convolutional Neural Network (CNN) was mainly used to recognize two-dimensional shapes, which are highly invariant to translation, scaling, tilting or other forms of deformation of the image. CNN mainly simplifies the complexity of the neural network model and reduces the number of weights through local perceptual fields and weight sharing. With the development of deep learning technology, the scope of application of CNN is not limited to the field of image recognition, it can also be used in face recognition, text recognition, animal classification, image processing and other fields.

Fig. 1 shows a schematic diagram of a convolutional neural network. For example, this convolutional neural network can be used for image processing, which uses images as input and output, and replaces scalar weights by convolution kernels. FIG. 1 only shows a convolutional neural network with a 3-layer structure, which is not limited by embodiments of the present disclosure. As shown in FIG. 1 , the convolutional neural network includes an input layer 101 , a hidden layer 102 and an output layer 103 . The input layer 101 has 4 inputs, the hidden layer 102 has 3 outputs, and the output layer 103 has 2 outputs, and finally the convolutional neural network finally outputs 2 images.

For example, the four inputs of the input layer 101 may be four images, or four feature images of one image. The three outputs of the hidden layer 102 may be feature images of images input through the input layer 101 .

和偏置/>

权重/>

表示卷积核，偏置/>

是叠加到卷积层的输出的标量，其中，k是表示输入层101的标签，i和j分别是输入层101的单元和隐藏层102的单元的标签。例如，第一卷积层201包括第一组卷积核(图1中的/>

)和第一组偏置(图1中的/>

)。第二卷积层202包括第二组卷积核(图1中的/>

)和第二组偏置(图1中的/>

)。通常，每个卷积层包括数十个或数百个卷积核，若卷积神经网络为深度卷积神经网络，则其可以包括至少五层卷积层。For example, as shown in Figure 1, a convolutional layer has weights

and bias />

Weight />

Indicates the convolution kernel, bias />

is a scalar superimposed to the output of the convolutional layer, where k is the label representing the input layer 101, and i and j are the labels of the units of the input layer 101 and hidden layer 102, respectively. For example, the first convolution layer 201 includes a first set of convolution kernels (> in FIG. 1

) and the first set of biases (/> in Figure 1

). The second convolutional layer 202 includes a second set of convolution kernels (> in FIG. 1

) and a second set of biases (/> in Figure 1

). Usually, each convolutional layer includes dozens or hundreds of convolutional kernels, and if the convolutional neural network is a deep convolutional neural network, it may include at least five convolutional layers.

For example, as shown in FIG. 1 , the convolutional neural network further includes a first activation layer 203 and a second activation layer 204 . The first activation layer 203 is located after the first convolutional layer 201 , and the second activation layer 204 is located after the second convolutional layer 202 . The activation layer (for example, the first activation layer 203 and the second activation layer 204) includes an activation function, and the activation function is used to introduce nonlinear factors into the convolutional neural network, so that the convolutional neural network can better solve more complex problems . The activation function may include a rectified linear unit (ReLU) function, a Sigmoid function (Sigmoid function), or a hyperbolic tangent function (tanh function), and the like. The ReLU function is an unsaturated nonlinear function, and the Sigmoid function and tanh function are saturated nonlinear functions. For example, the activation layer can be used alone as a layer of the convolutional neural network, or the activation layer can also be included in the convolutional layer (for example, the first convolutional layer 201 can include the first activation layer 203, and the second convolutional layer 202 can be included in the second active layer 204).

和第一组偏置中的若干偏置/>

以得到第一卷积层201的输出；然后，第一卷积层201的输出可以通过第一激活层203进行处理，以得到第一激活层203的输出。在第二卷积层202中，首先，对输入的第一激活层203的输出应用第二组卷积核中的若干卷积核/>

和第二组偏置中的若干偏置/>

以得到第二卷积层202的输出；然后，第二卷积层202的输出可以通过第二激活层204进行处理，以得到第二激活层204的输出。例如，第一卷积层201的输出可以为对其输入应用卷积核/>

后再与偏置/>

相加的结果，第二卷积层202的输出可以为对第一激活层203的输出应用卷积核/>

后再与偏置/>

相加的结果。For example, in the first convolutional layer 201, first, several convolution kernels in the first set of convolution kernels are applied to each input

and some of the biases in the first set of biases />

to obtain the output of the first convolutional layer 201; then, the output of the first convolutional layer 201 can be processed by the first activation layer 203 to obtain the output of the first activation layer 203. In the second convolutional layer 202, first, several convolution kernels in the second set of convolution kernels are applied to the output of the input first activation layer 203 />

and some of the biases in the second set of biases />

to obtain the output of the second convolutional layer 202; then, the output of the second convolutional layer 202 can be processed by the second activation layer 204 to obtain the output of the second activation layer 204. For example, the output of the first convolutional layer 201 may be the application of a convolution kernel to its input />

then bias with />

As a result of the addition, the output of the second convolutional layer 202 can be the application of a convolution kernel to the output of the first activation layer 203 />

then bias with />

The result of the addition.

Before using the convolutional neural network for image processing, the convolutional neural network needs to be trained. After training, the kernels and biases of a convolutional neural network remain constant during image processing. During the training process, each convolution kernel and bias are adjusted through multiple sets of input/output example images and an optimization algorithm to obtain an optimized convolutional neural network model.

FIG. 2A shows a schematic structural diagram of a convolutional neural network, and FIG. 2B shows a schematic diagram of a working process of a convolutional neural network. For example, as shown in FIGS. 2A and 2B , after the input image is input to the convolutional neural network through the input layer, it goes through several processing processes (each layer in FIG. 2A ) in turn to output a category identifier. The main components of a convolutional neural network can include multiple convolutional layers, multiple downsampling layers, and fully connected layers, among others. In this disclosure, it should be understood that each of these layers, such as multiple convolutional layers, multiple downsampling layers, and fully connected layers, refers to a corresponding processing operation, that is, convolution processing, downsampling processing, fully connected processing, etc. , the described neural network also refers to the corresponding processing operations, and the example normalization layer or layer normalization layer described below is also similar to this, and will not be repeated here. For example, a complete convolutional neural network can be composed of these three layer stacks. For example, FIG. 2A only shows three layers of a convolutional neural network, namely, the first layer, the second layer and the third layer. For example, each layer can include a convolutional block and a downsampling layer. For example, each convolutional module may include a convolutional layer. Thus, the processing at each level may include: performing convolution and down-sampling (sub-sampling/down-sampling) on the input image. For example, according to actual needs, each convolution module may also include an instance normalization layer or a layer normalization layer, so that the processing at each level may also include instance normalization or layer normalization.

For example, the instance normalization layer is used to perform instance normalization processing on the feature image output by the convolution layer, so that the gray value of the pixel of the feature image changes within a predetermined range, thereby simplifying the image generation process and improving the effect of image enhancement. For example, the predetermined range may be [-1, 1]. The instance normalization layer performs instance normalization on each feature image according to its own mean and variance. For example, an instance normalization layer can also be used to perform instance normalization on a single image.

For example, assuming that the size of the mini-batch gradient descent method (mini-batch gradient decent) is T, the number of feature images output by a certain convolutional layer is C, and each feature image is a matrix of H rows and W columns, then the feature The model of the image is denoted as (T,C,H,W). Thus, the instance normalization formula of the instance normalization layer can be expressed as follows:

Among them, x _tijk is the value of the t-th feature block (patch), the i-th feature image, the j-th row, and the k-th column in the feature image set output by the convolutional layer. y _tijk represents the result obtained after processing x _tijk by the instance normalization layer. e ₁ is a very small integer to avoid the denominator being 0.

For example, the layer normalization layer is similar to the instance normalization layer, and it is also used to perform layer normalization processing on the feature image output by the convolution layer, so that the gray value of the pixel of the feature image changes within a predetermined range, thereby simplifying the image generation process and improving The effect of image enhancement. For example, the predetermined range may be [-1, 1]. Different from the example normalization layer, the layer normalization layer performs layer normalization processing on each column of each feature image according to the mean and variance of each column of each feature image, thereby realizing layer normalization processing on the feature image. For example, the layer normalization layer can also be used to perform layer normalization on a single image.

For example, still taking the above mini-batch gradient descent method as an example, the model of the feature image is expressed as (T, C, H, W). Thus, the layer normalization formula of the layer normalization layer can be expressed as follows:

Among them, x _tijk is the value of the t-th feature block (patch), the i-th feature image, the j-th row, and the k-th column in the feature image set output by the convolutional layer. y′ _tijk represents the result obtained after processing x _tijk by the layer normalization layer. e ₂ is a small integer to avoid 0 in the denominator.

The convolutional layer is the core layer of the convolutional neural network. In the convolutional layers of a convolutional neural network, a neuron is only connected to some neurons in adjacent layers. A convolutional layer can apply several convolution kernels (also known as filters) to an input image to extract various types of features of the input image. Each convolution kernel can extract one type of feature. The convolution kernel is generally initialized in the form of a random decimal matrix. During the training process of the convolutional neural network, the convolution kernel will be learned to obtain reasonable weights. The result obtained after applying a convolution kernel to the input image is called a feature map, and the number of feature images is equal to the number of convolution kernels. Each feature image is composed of some rectangularly arranged neurons, and the neurons of the same feature image share weights, and the shared weights here are convolution kernels. The feature image output by one level of convolutional layer can be input to the adjacent next level of convolutional layer and processed again to obtain a new feature image. For example, as shown in FIG. 2A , the first-level convolutional layer may output the first-level feature image, which is input to the second-level convolutional layer for further processing to obtain the second-level feature image.

For example, as shown in Figure 2B, the convolution layer can use different convolution kernels to convolve the data of a certain local receptive field of the input image, and the convolution result is input into the activation layer, which is calculated according to the corresponding activation function To get the feature information of the input image.

For example, as shown in FIGS. 2A and 2B , a downsampling layer is disposed between adjacent convolutional layers, and the downsampling layer is a form of downsampling. On the one hand, the downsampling layer can be used to reduce the size of the input image, simplify the computational complexity, and reduce the phenomenon of overfitting to a certain extent; on the other hand, the downsampling layer can also perform feature compression to extract the input image. main features. Downsampling layers are able to reduce the size of feature images without changing the number of feature images. For example, an input image with a size of 12×12 is sampled by a 6×6 convolution kernel, then a 2×2 output image can be obtained, which means that 36 pixels on the input image are merged into 1 pixel. The last downsampling layer or convolutional layer can be connected to one or more fully connected layers, which are used to connect all the extracted features. The output of the fully connected layer is a one-dimensional matrix, that is, a vector.

Some embodiments and examples of the present disclosure will be described in detail below with reference to the accompanying drawings.

Fig. 3 is a flowchart of an image processing method provided by an embodiment of the present disclosure. For example, as shown in Figure 3, the image processing method includes:

Step S110: receiving the first feature image;

Step S120: Perform at least one multi-scale circular sampling process on the first feature image.

For example, in step S110, the first feature image may include a feature image obtained after the input image is processed by one of a convolutional layer, a residual network, and a dense network (for example, refer to FIG. 2B ). For example, a residual network keeps a certain proportion of its input in its output by means of, for example, the addition of residual connections. For example, a dense network includes a bottleneck layer and a convolutional layer. For example, in some examples, the bottleneck layer is used to reduce the dimensionality of the data to reduce the number of parameters in subsequent convolution operations, such as the convolution kernel of the bottleneck layer. is a 1x1 convolution kernel, for example, a convolution kernel of a convolution layer is a 3x3 convolution kernel; the present disclosure includes but is not limited thereto. For example, the input image is processed by convolution, down-sampling, etc. to obtain the first feature image. It should be noted that, in this embodiment, there is no limitation on the manner of acquiring the first feature image. For example, the first characteristic image may include a plurality of characteristic images, but is not limited thereto.

For example, the first feature image received in step S110 is used as the input of the multi-scale circular sampling process in step S120. For example, the multi-scale round-robin sampling process may have various forms, including but not limited to the three forms shown in FIGS. 4A-4C which will be described below.

FIG. 4A is a schematic flowchart corresponding to multi-scale circular sampling processing in the image processing method shown in FIG. 3 provided by an embodiment of the present disclosure. As shown in FIG. 4A , multi-scale circular sampling processing includes nested first-level sampling processing and second-level sampling processing.

For example, as shown in FIG. 4A , the input of the multi-scale circular sampling process is used as the input of the first-level sampling process, and the output of the first-level sampling process is used as the output of the multi-scale circular sampling process. For example, the output of the multi-scale cyclic sampling process is called the second feature image. For example, the size of the second feature image (values of rows and columns of the pixel array) may be the same as that of the first feature image.

For example, as shown in FIG. 4A , the first-level sampling processing includes a first down-sampling process, a first up-sampling process, and a first residual chained addition process performed in sequence. The first down-sampling process performs down-sampling processing based on the input of the first-level sampling process to obtain the first down-sampling output, for example, the first down-sampling process can directly down-sample the input of the first-level sampling process to obtain the first down-sampling output. The first up-sampling process performs up-sampling processing based on the first down-sampling output to obtain the first up-sampling output, for example, after the first down-sampling output undergoes second-level sampling processing, then performs up-sampling processing to obtain the first up-sampling output , that is, the first up-sampling process may indirectly perform up-sampling processing on the first down-sampling output. The first residual chain addition process performs the first residual chain addition on the input of the first-level sampling process and the first upsampling output, and then uses the result of the first residual chain addition as the output of the first-level sampling process . For example, the size of the output of the first upsampling process (that is, the first upsampling output) is the same as the size of the input of the first-level sampling process (that is, the input of the first downsampling process), so that they are added via the first residual link After that, the size of the output of the first-level sampling process is the same as the size of the input of the first-level sampling process.

For example, as shown in Figure 4A, the second-level sampling process is nested between the first down-sampling process and the first up-sampling process of the first-level sampling process, and receives the first down-sampling output as the input of the second-level sampling process , providing the output of the second-level sampling processing as an input of the first up-sampling processing, so that the first up-sampling processing performs up-sampling processing based on the first down-sampling output.

For example, as shown in FIG. 4A , the second-level sampling process includes a second down-sampling process, a second up-sampling process, and a second residual chained addition process performed in sequence. The second down-sampling process performs down-sampling processing based on the input of the second-level sampling process to obtain the second down-sampling output. For example, the second down-sampling process can directly down-sample the input of the second-level sampling process to obtain the second down-sampling output. The second up-sampling process performs up-sampling processing based on the second down-sampling output to obtain the second up-sampling output. For example, the second up-sampling process may directly perform up-sampling on the second down-sampling output to obtain the second up-sampling output. The second residual link addition process performs the second residual link addition on the input of the second-level sampling process and the second upsampling output, and then uses the result of the second residual link addition as the output of the second-level sampling process . For example, the size of the output of the second up-sampling process (ie, the second up-sampling output) is the same as the size of the input of the second-level sampling process (ie, the input of the second down-sampling process), so that they are added via the second residual link After that, the output of the second-level sampling process has the same size as the input to the second-level sampling process.

It should be noted that, in some embodiments of the present disclosure (not limited to this embodiment), each level of sampling processing (for example, the first level sampling processing, the second level sampling processing, and the embodiment shown in FIG. 4B will The processes of the third-level sampling processing, etc.) introduced are similar, including down-sampling processing, up-sampling processing, and residual link addition processing. In addition, taking the feature image as an example, the residual link addition process may include correspondingly adding the values of each row and each column of the matrix of the two feature images, but it is not limited thereto.

In the present disclosure, "nesting" means that an object includes another object that is similar or identical to the object, and the object includes but not limited to a process or a network structure.

It should be noted that, in some embodiments of the present disclosure, the size of the output of the up-sampling process (for example, the output of the up-sampling process is a feature image) in the sampling process of each level is the same as the size of the input of the down-sampling process (for example, The input of the downsampling process is the feature image) size is the same, so after the residual link is added, the output of the sampling process of each level (for example, the output of the sampling process of each level can be the feature image) size and each The input of the sampling process of the levels (for example, the input of the sampling process of each level may be a feature image) has the same size.

It should be noted that, in some embodiments of the present disclosure, multi-scale cyclic sampling processing may be implemented through a convolutional neural network. For example, in some embodiments of the present disclosure, the first convolutional neural network may be used to perform multi-scale cyclic sampling processing. For example, in some examples, the first convolutional neural network may include a nested first meta-network and a second meta-network, the first meta-network is used to perform a first-level sampling process, and the second meta-network is used to perform a second meta-network. Hierarchical sampling processing.

For example, in some examples, the first meta-network may include a first sub-network and a second sub-network, the first sub-network is used to perform the first down-sampling process, and the second sub-network is used to perform the first up-sampling process. The second meta-network is nested between the first sub-network and the third sub-network of the first meta-network. For example, in some examples, the second meta-network may include a third sub-network and a fourth sub-network, the third sub-network is used to perform the second down-sampling process, and the fourth sub-network is used to perform the second up-sampling process. For example, both the first meta-network and the second meta-network are similar to the form of the aforementioned residual network.

For example, in some examples, each of the first sub-network, the second sub-network, the third sub-network and the fourth sub-network includes one of a convolutional layer, a residual network, a dense network, and the like. Specifically, the first sub-network and the third sub-network can include a convolutional layer (down-sampling layer) with a down-sampling function, and can also include one of a residual network and a dense network with a down-sampling function; the second sub-network and the fourth sub-network may include a convolutional layer (upsampling layer) with an upsampling function, and may also include one of a residual network, a dense network, etc. with an upsampling function. It should be noted that the first subnetwork and the third subnetwork may have the same structure or different structures; the second subnetwork and the fourth subnetwork may have the same structure or different structures; The disclosed embodiments are not limited in this regard.

Downsampling is used to reduce the size of the feature image, thereby reducing the amount of data of the feature image. For example, downsampling may be performed through a downsampling layer, but not limited thereto. For example, the downsampling layer can use max pooling, average pooling, strided convolution, undersampling (decimation, such as selecting fixed pixels), demultiplexing output (demuxout, Downsampling methods such as splitting the input image into multiple smaller images) implement downsampling processing.

Upsampling is used to increase the size of the feature image, thereby increasing the data volume of the feature image. For example, an upsampling layer may be used to perform upsampling processing, but is not limited thereto. For example, the upsampling layer may implement upsampling processing by using upsampling methods such as strided transposed convolution and interpolation algorithms. The interpolation algorithm may include, for example, algorithms such as interpolation, bilinear interpolation, and bi-cubic interpolation (Bicubic Interprolation).

It should be noted that, in some embodiments of the present disclosure, the downsampling factor of the downsampling process at the same level corresponds to the upsampling factor of the upsampling process, that is, when the downsampling factor of the downsampling process is 1/y , then the upsampling factor of the upsampling process is y, where y is a positive integer, and y is usually greater than 2. Therefore, it can be ensured that the output of the up-sampling process at the same level is the same as the input size of the down-sampling process.

It should be noted that, in some embodiments of the present disclosure (not limited to this embodiment), the parameters of the downsampling process at different levels (that is, the parameters of the network corresponding to the downsampling process) may be the same or different; different levels The parameters of the upsampling processing (that is, the parameters of the network corresponding to the upsampling processing) can be the same or different; the parameters added by residual connections at different levels can be the same or different. This disclosure does not limit this.

For example, in some embodiments of the present disclosure (not limited to this embodiment), in order to improve global features such as brightness and contrast of the feature image, the multi-scale circular sampling process may also include: in the first down-sampling process, the first up-sampling After processing, second downsampling and second upsampling, instance normalization or layer normalization is performed on the first downsampling output, first upsampling output, second downsampling output and second upsampling output respectively. It should be noted that, the first downsampling output, the first upsampling output, the second downsampling output and the second upsampling output may adopt the same normalization processing method (instance normalization processing or layer normalization processing), or different Standardized processing methods, which are not limited in the present disclosure.

Correspondingly, the first sub-network, the second sub-network, the third sub-network and the fourth sub-network further include an instance normalization layer or a layer normalization layer respectively, the instance normalization layer is used to perform instance normalization processing, and the layer normalization layer is used to execute the layer Standardized processing. For example, the instance normalization layer may perform instance normalization processing according to the aforementioned instance normalization formula, and the layer normalization layer may perform layer normalization processing according to the aforementioned layer normalization formula, which is not limited in the present disclosure. It should be noted that the first sub-network, the second sub-network, the third sub-network and the fourth sub-network may include the same normalization layer (instance normalization layer or layer normalization layer), or may include different normalization layers. There is no restriction on this either.

FIG. 4B is a schematic flowchart corresponding to multi-scale circular sampling processing in the image processing method shown in FIG. 3 provided by another embodiment of the present disclosure. As shown in FIG. 4B , on the basis of the multi-scale circular sampling process shown in FIG. 4A , the multi-scale circular sampling process further includes a third-level sampling process. It should be noted that other processes of the multi-scale cyclic sampling process shown in FIG. 5 are basically the same as the process of the multi-scale cyclic sampling process shown in FIG. 4A , and repeated descriptions will not be repeated here.

For example, as shown in FIG. 4B, the third-level sampling process is nested between the second down-sampling process and the second up-sampling process of the second-level sampling process, and receives the second down-sampling output as the input of the third-level sampling process. , providing the output of the third-level sampling processing as an input of the second up-sampling processing, so that the second up-sampling processing performs up-sampling processing based on the second down-sampling output. It should be noted that, at this time, similar to the first upsampling process that indirectly performs upsampling processing on the first downsampling output, the second upsampling process also indirectly performs upsampling processing on the second downsampling output.

The third-level sampling process includes a third down-sampling process, a third up-sampling process, and a third residual linking addition process performed in sequence. The third down-sampling process performs down-sampling processing based on the input of the third-level sampling process to obtain the third down-sampling output. For example, the third down-sampling process can directly down-sample the input of the third-level sampling process to obtain the third down-sampling process output. The third up-sampling process performs up-sampling processing based on the third down-sampling output to obtain a third up-sampling output. For example, the third up-sampling process may directly perform up-sampling on the third down-sampling output to obtain a third up-sampling output. The third residual link addition process performs the third residual link addition on the input of the third-level sampling process and the third upsampling output, and then uses the result of the third residual link addition as the output of the third-level sampling process . For example, the size of the output of the third up-sampling process (ie, the third up-sampling output) is the same as the size of the input of the third-level sampling process (ie, the input of the third down-sampling process), so that they are added via the third residual link After that, the output of the third-level sampling process has the same size as the input of the third-level sampling process.

It should be noted that for more details and implementations (i.e., network structure) of the third-level sampling processing, reference may be made to the description of the first-level sampling processing and the second-level sampling processing in the embodiment shown in FIG. Let me repeat.

It should be noted that, based on this embodiment, those skilled in the art should understand that the multi-scale cyclic sampling processing may also include more levels of sampling processing, for example, may also include a fourth level nested in the third level of sampling processing Sampling processing, fifth-level sampling processing nested in fourth-level sampling processing, etc., are nested in a similar manner to the above-mentioned second-level sampling processing and third-level sampling processing, and this disclosure does not make any limit.

FIG. 4C is a schematic flowchart corresponding to multi-scale circular sampling processing in the image processing method shown in FIG. 3 provided by yet another embodiment of the present disclosure. As shown in FIG. 4C , on the basis of the multi-scale circular sampling process shown in FIG. 4A , the multi-scale circular sampling process includes multiple second-level sampling processes performed sequentially. It should be noted that other processes of the multi-scale cyclic sampling process shown in FIG. 5 are basically the same as the process of the multi-scale cyclic sampling process shown in FIG. 4A , and repeated descriptions will not be repeated here. It should also be noted that the inclusion of two second-level sampling processes in FIG. 4C is exemplary. In an embodiment of the present disclosure, the multi-scale circular sampling process may include two or more second-level sampling processes performed sequentially. deal with. It should be noted that, in the embodiments of the present disclosure, the number of sampling processes at the second level may be selected according to actual needs, which is not limited in the present disclosure. For example, in some examples, the inventors of the present application found that, compared with the image processing method with one or three second-level sampling processes, the image enhancement process using the image processing method with two second-level sampling processes The effect is better, but this should not be considered as a limitation of the present disclosure.

For example, the first level 2 sampling process receives the first downsampled output as input to the first level 2 sampling process, and each level 2 sampling process other than the first level 2 sampling process receives the previous The output of the second-level sampling process is used as the input of the current second-level sampling process, and the output of the last second-level sampling process is used as the input of the first up-sampling process.

It should be noted that, for more details and implementation manners of each second-level sampling process, reference may be made to the description about the second-level sampling process in the embodiment shown in FIG. 4A , which will not be repeated in this disclosure.

It should be noted that, in some embodiments of the present disclosure (not limited to this embodiment), the parameters of the downsampling process of the same level in different orders may be the same or different; the parameters of the upsampling process of the same level in different orders It can be the same or different; the parameters added by residual connections of the same level in different orders can be the same or different. This disclosure does not limit this.

It should be noted that, based on this embodiment, those skilled in the art should understand that in the multi-scale cyclic sampling process, the first-level sampling process can nest multiple second-level sampling processes that are executed sequentially; further, at least partly The second-level sampling process can nest one or more third-level sampling processes that are executed sequentially, and the number of third-level sampling processes nested in at least part of the second-level sampling process can be the same or different; further, The third-level sampling process can be nested with the fourth-level sampling process, and the specific nesting method can be the same as that of the second-level sampling process nesting the third-level sampling process; and so on.

It should be noted that FIGS. 4A-4C show a situation where the image processing method provided by the embodiment of the present disclosure includes one multi-scale circular sampling process. In the image processing method provided by the embodiment shown in FIGS. 4A-4C , at least one multi-scale circular sampling process includes one multi-scale circular sampling process. The multi-scale cyclic sampling process receives the first feature image as the input of the multi-scale cyclic sampling process, the input of the multi-scale cyclic sampling process is used as the input of the first level sampling process in the multi-scale cyclic sampling process, and the multi-scale cyclic sampling process in the multi-scale cyclic sampling process The output of the first-level sampling process is used as the output of the multi-scale circular sampling process, and the output of the multi-scale circular sampling process is used as the output of at least one multi-scale circular sampling process. This disclosure includes but is not limited to this.

FIG. 4D is a schematic flowchart corresponding to multi-scale circular sampling processing in the image processing method shown in FIG. 3 provided by another embodiment of the present disclosure. As shown in Figure 4D, in the image processing method provided in this embodiment, at least one multi-scale circular sampling process includes multiple sequentially executed multi-scale circular sampling processes, for example, at least one multi-scale circular sampling process may include two or three A multi-scale cyclic sampling process performed sequentially, but not limited thereto. It should be noted that, in the embodiments of the present disclosure, the number of multi-scale cyclic sampling processes may be selected according to actual needs, and the present disclosure does not limit this. For example, in some examples, the inventors of the present application found that, compared to the image processing method with one or three multi-scale cyclic sampling processes, the image enhancement process using the image processing method with two multi-scale cyclic sampling processes The effect is better, but this should not be considered as a limitation of the present disclosure.

For example, the input of each multi-scale cyclic sampling process is used as the input of the first-level sampling process in this multi-scale cyclic sampling process, and the output of the first-level sampling process in each multi-scale cyclic sampling process is used as the current multi-scale cyclic sampling process. The output of the loop sampling process.

For example, as shown in Figure 4D, the first multi-scale round-robin sampling process receives the first feature image as the input of the first multi-scale round-robin sampling process, and each multi-scale round-robin sampling process except the first multi-scale round-robin sampling process The sampling process receives the output of the previous multi-scale cyclic sampling process as the input of this multi-scale cyclic sampling process, and the output of the last multi-scale cyclic sampling process as the output of at least one multi-scale cyclic sampling process.

It should be noted that, for more details and implementation manners of each multi-scale cyclic sampling process, reference may be made to the description of the multi-scale cyclic sampling process in the embodiment shown in FIGS. 4A-4C , which will not be repeated in this disclosure. It should also be noted that the implementation manners (that is, network structures) and parameters of the multi-scale cyclic sampling processing in different orders may be the same or different, which is not limited in the present disclosure.

Fig. 5 is a flowchart of an image processing method provided by another embodiment of the present disclosure. As shown in FIG. 5, the image processing method includes step S210 to step S250. It should be noted that steps S230 to S240 of the image processing method shown in FIG. 5 are correspondingly the same as steps S110 to S120 of the image processing method shown in FIG. 3 , that is, the image processing method shown in FIG. The image processing method shown in FIG. 5, therefore step S230 to step S240 of the image processing method shown in FIG. The method of the embodiment shown in 4D and the like. Hereinafter, step S210 to step S250 of the image processing method shown in FIG. 5 will be described in detail.

Step S210: Obtain an input image.

For example, in step S210, the input image may include photos collected by a camera of a smartphone, a camera of a tablet computer, a camera of a personal computer, a lens of a digital camera, a surveillance camera or a network camera, etc., which may include images of people, moving pictures, etc. Plant images or landscape images, etc., are not limited in this disclosure. For example, the quality of the input image is lower than that of a photo taken by a real digital SLR camera, that is, the input image is a low-quality image. For example, in some examples, the input image may include 3-channel RGB images; in other examples, the input image may include 3-channel YUV images. Hereinafter, the input image includes an RGB image as an example for description, but embodiments of the present disclosure are not limited thereto.

Step S220: Using the analysis network to convert the input image into a first feature image.

For example, in step S220, the analysis network may be a convolutional neural network including one of a convolutional layer, a residual network, a dense network, and the like. For example, in some examples, the analysis network may convert the 3-channel RGB image (ie, the input image) into a plurality of first feature images, for example, 64 first feature images, which the present disclosure includes but is not limited to.

It should be noted that the embodiments of the present disclosure do not limit the structure and parameters of the analysis network, as long as it can convert the input image to the convolutional feature dimension (ie, convert it to the first feature image).

Step S230: receiving the first feature image;

Step S240: Perform at least one multi-scale circular sampling process on the first feature image.

It should be noted that, for steps S230 to S240, reference may be made to the foregoing description of steps S110 to S120, which will not be repeated in this disclosure.

Step S250: Convert the output of at least one multi-scale cyclic sampling process into an output image using a synthesis network.

For example, in step S250, the synthetic network may be a convolutional neural network including one of a convolutional layer, a residual network, a dense network, and the like. For example, the output of at least one multi-scale cyclic sampling process may be referred to as a second feature image. For example, the number of second feature images may be multiple, but not limited thereto. For example, in some examples, the synthesis network may convert a plurality of second feature images into an output image, for example, the output image may include a 3-channel RGB image, the present disclosure includes but is not limited thereto.

Fig. 6A is a schematic diagram of an input image, and Fig. 6B is obtained by processing the input image shown in Fig. 6A according to an image processing method provided by an embodiment of the present disclosure (for example, the image processing method shown in Fig. 5 ). Schematic of the output image.

For example, as shown in Figure 6A and Figure 6B, the output image retains the content of the input image, but improves the contrast of the image and improves the problem of the input image being too dark, so that compared with the input image, the quality of the output image can be close to Based on the quality of photos taken by a real DSLR camera, the output image is a high-quality image.

It should be noted that the embodiments of the present disclosure do not limit the structure and parameters of the synthesis network, as long as it can convert the convolutional feature dimension (ie, the second feature image) into an output image.

The image processing method provided by the embodiments of the present disclosure can perform image enhancement processing on low-quality input images, and obtain higher image fidelity through repeated sampling on multiple scales, which can greatly improve the quality of output images. Suitable for offline applications such as batch processing that require high image quality. Specifically, the PSNR of the image output by the image enhancement method proposed in the literature of Andrey Ignatov et al. is 20.08, while the PSNR of the output image obtained based on the image processing method shown in the embodiment shown in FIG. 4C of the present disclosure can reach 23.35, that is The image obtained by the image processing method provided by the embodiments of the present disclosure can be closer to the photo taken by a real digital single-lens reflex camera.

At least one embodiment of the present disclosure also provides a neural network training method. Fig. 7A is a schematic structural diagram of a neural network provided by an embodiment of the present disclosure, Fig. 7B is a flowchart of a training method of a neural network provided by an embodiment of the present disclosure, and Fig. 7C is a schematic diagram of a neural network provided by an embodiment of the present disclosure A schematic architectural block diagram of training the neural network shown in FIG. 7A corresponding to the training method shown in FIG. 7B.

For example, as shown in FIG. 7A , the neural network 300 includes an analysis network 310 , a first sub-neural network 320 and a synthesis network 330 . For example, the analysis network 310 processes the input image to obtain the first feature image, the first sub-neural network 320 performs at least one multi-scale circular sampling process on the first feature image to obtain the second feature image, and the synthesis network 330 processes the second feature image The image is processed to obtain an output image.

For example, the structure of the analysis network 310 can refer to the description of the analysis network in the aforementioned step S220, and this disclosure is not limited; the structure of the first sub-neural network 320 can refer to the multi-scale loop in the aforementioned step S120 (ie, step S240). The description of the implementation of the sampling process, for example, the first sub-neural network may include but not limited to the aforementioned first convolutional neural network, and this disclosure is not limited to this; description, the present disclosure is not limited thereto.

For example, for the input image and the output image, reference may also be made to the description about the input image and the output image in the image processing method provided in the foregoing embodiments, which will not be repeated in this disclosure.

For example, as shown in FIG. 7B and FIG. 7C , the neural network training method includes steps S410 to S460.

Step S410: Obtain training input images.

For example, similar to the input image in the aforementioned step S210, the training input image may also include photos collected by a camera of a smartphone, a camera of a tablet computer, a camera of a personal computer, a lens of a digital camera, a monitoring camera or a network camera, etc. It may include images of people, images of animals and plants, or images of landscapes, etc., which are not limited in the present disclosure. For example, the quality of training input images is lower than the quality of photos taken by real digital SLR cameras, that is, the training input images are low-quality images. For example, in some examples, the training input images may include 3-channel RGB images.

Step S420: Use the analysis network to process the training input image to provide a first training feature image.

For example, similar to the analysis network in the aforementioned step S220, the analysis network 310 may be a convolutional neural network including one of a convolutional layer, a residual network, and a dense network. For example, in some examples, the analysis network may convert the 3-channel RGB image (ie, the training input image) into a plurality of first training feature images, for example, 64 first training feature images, the present disclosure includes but is not limited thereto.

Step S430: Use the first sub-neural network to perform at least one multi-scale circular sampling process on the first training feature image to obtain a second training feature image.

For example, in step S430, the multi-scale circular sampling process may be implemented as the multi-scale circular sampling process in any of the embodiments shown in FIGS. 4A-4D , but is not limited thereto. Hereinafter, the multi-scale circular sampling process in step S430 is implemented as the multi-scale circular sampling process shown in FIG. 4A as an example for illustration.

For example, as shown in FIG. 4A , the multi-scale circular sampling process nests the first-level sampling process and the second-level sampling process.

For example, as shown in Figure 4A, the input of the multi-scale circular sampling process (ie, the first training feature image) is used as the input of the first-level sampling process, and the output of the first-level sampling process is used as the output of the multi-scale circular sampling process (ie, the first 2 training feature images). For example, the size of the second training feature image may be the same as the size of the first training feature image.

For example, as shown in FIG. 4A , the first-level sampling processing includes a first down-sampling process, a first up-sampling process, and a first residual chained addition process performed in sequence. The first down-sampling process performs down-sampling processing based on the input of the first-level sampling process to obtain the first down-sampling output, for example, the first down-sampling process can directly down-sample the input of the first-level sampling process to obtain the first down-sampling output. The first up-sampling process performs up-sampling processing based on the first down-sampling output to obtain the first up-sampling output, for example, after the first down-sampling output undergoes second-level sampling processing, then performs up-sampling processing to obtain the first up-sampling output , that is, the first up-sampling process may indirectly perform up-sampling processing on the first down-sampling output. The first residual chain addition process performs the first residual chain addition on the input of the first-level sampling process and the first upsampling output, and then uses the result of the first residual chain addition as the output of the first-level sampling process . For example, the size of the output of the first upsampling process (that is, the first upsampling output) is the same as the size of the input of the first-level sampling process (that is, the input of the first downsampling process), so that they are added via the first residual link After that, the size of the output of the first-level sampling process is the same as the size of the input of the first-level sampling process.

For example, as shown in Figure 4A, the second-level sampling process is nested between the first down-sampling process and the first up-sampling process of the first-level sampling process, and receives the first down-sampling output as the input of the second-level sampling process , providing the output of the second-level sampling processing as an input of the first up-sampling processing, so that the first up-sampling processing performs up-sampling processing based on the first down-sampling output.

For example, as shown in FIG. 4A , the second-level sampling process includes a second down-sampling process, a second up-sampling process, and a second residual chained addition process performed in sequence. The second down-sampling process performs down-sampling processing based on the input of the second-level sampling process to obtain the second down-sampling output. For example, the second down-sampling process can directly down-sample the input of the second-level sampling process to obtain the second down-sampling output. The second up-sampling process performs up-sampling processing based on the second down-sampling output to obtain the second up-sampling output. For example, the second up-sampling process may directly perform up-sampling on the second down-sampling output to obtain the second up-sampling output. The second residual link addition process performs the second residual link addition on the input of the second-level sampling process and the second upsampling output, and then uses the result of the second residual link addition as the output of the second-level sampling process . For example, the size of the output of the second up-sampling process (ie, the second up-sampling output) is the same as the size of the input of the second-level sampling process (ie, the input of the second down-sampling process), so that they are added via the second residual link After that, the output of the second-level sampling process has the same size as the input to the second-level sampling process.

For example, correspondingly, the first sub-neural network 320 may be implemented as the aforementioned first convolutional neural network. For example, the first sub-neural network 320 may include a nested first meta-network and a second meta-network, the first meta-network is used to perform first-level sampling processing, and the second meta-network is used to perform second-level sampling processing.

For example, the first meta-network may include a first sub-network and a second sub-network, the first sub-network is used to perform the first down-sampling process, and the second sub-network is used to perform the first up-sampling process. The second meta-network is nested between the first sub-network and the third sub-network of the first meta-network. For example, the second meta-network may include a third sub-network and a fourth sub-network, the third sub-network is used to perform the second down-sampling process, and the fourth sub-network is used to perform the second up-sampling process.

For example, each of the first sub-network, the second sub-network, the third sub-network and the fourth sub-network includes one of a convolutional layer, a residual network, a dense network, and the like. Specifically, the first sub-network and the third sub-network can include one of a convolutional layer (down-sampling layer), a residual network, a dense network, etc. with a down-sampling function; the second sub-network and the fourth sub-network can include a One of the convolutional layer (upsampling layer), residual network, dense network, etc. of the upsampling function. It should be noted that the first subnetwork and the third subnetwork may have the same structure or different structures; the second subnetwork and the fourth subnetwork may have the same structure or different structures; There is no limit to this publicly.

For example, in an embodiment of the present disclosure, in order to improve global features such as brightness and contrast of the feature image, the multi-scale circular sampling process may also include: the first down-sampling process, the first up-sampling process, the second down-sampling process and After the second upsampling process, instance normalization processing or layer normalization processing is performed on the first downsampling output, the first upsampling output, the second downsampling output, and the second upsampling output respectively. It should be noted that, the first downsampling output, the first upsampling output, the second downsampling output and the second upsampling output may adopt the same normalization processing method (instance normalization processing or layer normalization processing), or different Standardized processing methods, which are not limited in the present disclosure.

Correspondingly, the first sub-network, the second sub-network, the third sub-network and the fourth sub-network further include an instance normalization layer or a layer normalization layer respectively, the instance normalization layer is used to perform instance normalization processing, and the layer normalization layer is used to execute the layer Standardized processing. For example, the instance normalization layer may perform instance normalization processing according to the aforementioned instance normalization formula, and the layer normalization layer may perform layer normalization processing according to the aforementioned layer normalization formula, which is not limited in the present disclosure. It should be noted that the first sub-network, the second sub-network, the third sub-network and the fourth sub-network may include the same normalization layer (instance normalization layer or layer normalization layer), or may include different normalization layers. There is no restriction on this either.

It should be noted that, for more implementation methods and more details of the multi-scale circular sampling processing in step S430, please refer to the above-mentioned step S120 (that is, step S240) and the multi-scale circular sampling processing in the embodiment shown in FIGS. 4A-4D description, which will not be repeated in this disclosure. It should also be noted that when the multi-scale cyclic sampling processing in step S430 is implemented in other forms, the first sub-neural network 320 should be changed accordingly to realize other forms of multi-scale cyclic sampling processing, which is not discussed in this disclosure. repeat.

For example, in step S430, the number of second training feature images may be multiple, but not limited thereto.

Step S440: Use the synthesis network to process the second training feature image to obtain a training output image.

For example, similar to the synthesis network in the aforementioned step S250, the synthesis network 330 may be a convolutional neural network including one of a convolutional layer, a residual network, a dense network, and the like. For example, in some examples, the synthesis network may convert a plurality of second training feature images into training output images, for example, the training output images may include 3-channel RGB images, the present disclosure includes but is not limited thereto.

Step S450: Based on the training output image, calculate the loss value of the neural network through a loss function.

For example, the parameters of the neural network 300 include parameters of the analysis network 310 , parameters of the first sub-neural network 320 and parameters of the synthesis network 330 . For example, the initial parameters of the neural network 300 may be random numbers, such as random numbers conforming to Gaussian distribution, which is not limited in the embodiments of the present disclosure.

For example, the loss function in this embodiment may refer to the loss function in the literature provided by Andrey Ignatov et al. For example, similar to the loss function in this document, the loss function can include a color loss function, a texture loss function, and a content loss function; correspondingly, the specific process of calculating the loss value of the parameters of the neural network 300 through the loss function can also refer to this description in the literature. It should be noted that the embodiment of the present disclosure does not limit the specific form of the loss function, which includes but not limited to the form of the loss function in the above-mentioned documents.

Step S460: Correct the parameters of the neural network according to the loss value.

For example, an optimization function (not shown in FIG. 7C ) may also be included in the training process of the neural network 300. The optimization function may calculate the error value of the parameters of the neural network 300 according to the loss value calculated by the loss function, and based on the error value The parameters of the neural network 300 are modified. For example, the optimization function may use a stochastic gradient descent (SGD) algorithm, a batch gradient descent (BGD) algorithm, etc. to calculate error values of parameters of the neural network 300 .

For example, the training method of the neural network may also include: judging whether the training of the neural network satisfies the predetermined condition, if the predetermined condition is not satisfied, then repeatedly execute the above training process (ie step S410 to step S460); if the predetermined condition is satisfied, then stop the above-mentioned During the training process, the trained neural network is obtained. For example, in one example, the aforementioned predetermined condition is that the loss values corresponding to two consecutive (or more) training output images no longer decrease significantly. For example, in another example, the aforementioned predetermined condition is that the training times or training cycles of the neural network reach a predetermined number. This disclosure does not limit this.

For example, the training output image output by the trained neural network 300 retains the content of the training input image, but the quality of the training output image can be close to the quality of a photo taken by a real digital single-lens reflex camera, that is, the training output image is high quality image.

It should be noted that, the foregoing embodiments are only schematically illustrating the training process of the neural network. Those skilled in the art should know that in the training phase, a large number of sample images need to be used to train the neural network; at the same time, in the training process of each sample image, multiple iterations may be included to correct the parameters of the neural network. For another example, the training phase also includes fine-tuning the parameters of the neural network to obtain more optimized parameters.

The neural network training method provided by the embodiment of the present disclosure can train the neural network used in the image processing method of the embodiment of the present disclosure, and the neural network trained by the training method can perform image enhancement on the low-quality input image Processing, through repeated sampling on multiple scales to obtain higher image fidelity, can greatly improve the quality of the output image, suitable for offline applications such as batch processing that require high image quality.

At least one embodiment of the present disclosure further provides an image processing device. Fig. 8 is a schematic block diagram of an image processing device provided by an embodiment of the present disclosure. For example, as shown in FIG. 8 , the image processing device 500 includes a memory 510 and a processor 520 . For example, the memory 510 is used for non-transitory storage of computer-readable instructions, and the processor 520 is used for running the computer-readable instructions. When the computer-readable instructions are executed by the processor 520, the image processing method provided by the embodiments of the present disclosure is executed.

For example, the memory 510 and the processor 520 may directly or indirectly communicate with each other. For example, components such as the memory 510 and the processor 520 may communicate through a network connection. A network may include a wireless network, a wired network, and/or any combination of a wireless network and a wired network. The network may include a local area network, the Internet, a telecommunication network, an Internet of Things (Internet of Things) based on the Internet and/or a telecommunication network, and/or any combination of the above networks. The wired network can use twisted pair, coaxial cable or optical fiber transmission for example to communicate, and the wireless network can use 3G/4G/5G mobile communication network, Bluetooth, Zigbee or WiFi and other communication methods. The present disclosure does not limit the type and function of the network here.

For example, the processor 520 may control other components in the image processing apparatus to perform desired functions. The processor 520 may be a device with data processing capabilities and/or program execution capabilities, such as a central processing unit (CPU), a tensor processing unit (TPU), or a graphics processing unit (GPU). The central processing unit (CPU) may be an X86 or ARM architecture or the like. The GPU can be integrated directly on the motherboard alone, or built into the north bridge chip of the motherboard. A GPU can also be built into a central processing unit (CPU).

For example, memory 510 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include random access memory (RAM) and/or cache memory (cache), etc., for example. Non-volatile memory may include, for example, read only memory (ROM), hard disks, erasable programmable read only memory (EPROM), compact disc read only memory (CD-ROM), USB memory, flash memory, and the like.

For example, one or more computer instructions may be stored on memory 510 and executed by processor 520 to implement various functions. Various application programs and various data may also be stored in the computer-readable storage medium, such as training input images, and various data used and/or generated by the application programs.

For example, when some computer instructions stored in the memory 510 are executed by the processor 520, one or more steps in the image processing method described above may be performed. For another example, when other computer instructions stored in the memory 510 are executed by the processor 520, one or more steps in the above-mentioned neural network training method may be executed.

For example, for a detailed description of the processing process of the image processing method, refer to the relevant descriptions in the above-mentioned embodiment of the image processing method, and for a detailed description of the processing process of the neural network training method, refer to the above-mentioned embodiment of the neural network training method. Relevant descriptions will not be repeated here.

It should be noted that the image processing device provided by the above-mentioned embodiments of the present disclosure is exemplary rather than limiting. According to actual application requirements, the image processing device may also include other conventional components or structures, for example, to realize image For the necessary functions of the processing device, those skilled in the art may configure other conventional components or structures according to specific application scenarios, which are not limited by the embodiments of the present disclosure.

For the technical effects of the image processing device provided by the above embodiments of the present disclosure, reference may be made to the corresponding descriptions about the image processing method and the training method of the neural network in the above embodiments, which will not be repeated here.

At least one embodiment of the present disclosure further provides a storage medium. Fig. 9 is a schematic diagram of a storage medium provided by an embodiment of the present disclosure. For example, as shown in FIG. 9 , the storage medium 600 stores computer-readable instructions 601 non-transitory. When the non-transitory computer-readable instructions 601 are executed by a computer (including a processor), any embodiment of the present disclosure can be executed. Instructions for image processing methods.

For example, one or more computer instructions may be stored on storage medium 600 . Some computer instructions stored on the storage medium 600 may be, for example, instructions for implementing one or more steps in the above image processing method. Other computer instructions stored on the storage medium may be, for example, instructions for implementing one or more steps in the above neural network training method.

For example, the storage medium may include a memory component of a tablet computer, a hard disk of a personal computer, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM), compact disc read only memory (CD -ROM), flash memory, or any combination of the above-mentioned storage media, or other applicable storage media.

For the technical effect of the storage medium provided by the embodiments of the present disclosure, reference may be made to the corresponding descriptions about the image processing method and the training method of the neural network in the foregoing embodiments, and details are not repeated here.

For this disclosure, the following points need to be explained:

(1) In the drawings of the embodiments of the present disclosure, only the structures related to the embodiments of the present disclosure are involved, and other structures may refer to general designs.

(2) In the case of no conflict, features in the same embodiment and different embodiments of the present disclosure can be combined with each other.

The above is only a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope of the present disclosure, and should cover all within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be determined by the protection scope of the claims.

Claims (15)

1. An image processing method, comprising: receiving a first feature image; and performing at least one multi-scale circular sampling process on the first feature image; Wherein, the multi-scale circular sampling processing includes nested first-level sampling processing and second-level sampling processing, The first-level sampling process includes a first down-sampling process, a first up-sampling process, and a first residual chain addition process, wherein the first down-sampling process performs down-sampling processing based on an input of the first-level sampling process Obtain a first down-sampling output, the first up-sampling process performs up-sampling processing based on the first down-sampling output to obtain a first up-sampling output, and the first residual chain addition process samples the first level performing a first residual link addition on the processed input and the first upsampling output, and then using the result of the first residual link addition as an output of the first-level sampling process; The second-level sampling process is nested between the first down-sampling process and the first up-sampling process, receives the first down-sampling output as the input of the second-level sampling process, and provides a second-level sampling The processed output is used as an input to a first upsampling process, so that the first upsampling process performs an upsampling process based on the first downsampling output; The second-level sampling process includes a second down-sampling process, a second up-sampling process, and a second residual chain addition process, wherein the second down-sampling process performs downsampling based on an input of the second-level sampling process. Sampling processing to obtain a second down-sampling output, the second up-sampling processing to perform up-sampling processing based on the second down-sampling output to obtain a second up-sampling output, and the second residual chain addition processing to combine the second The input of the level sampling process and the output of the second upsampling are subjected to a second residual link addition, and then the result of the second residual link addition is used as an output of the second level sampling process. 2. The image processing method according to claim 1, wherein the size of the output of the first up-sampling process is the same as the size of the input of the first down-sampling process; The size of the output of the second upsampling process is the same as the size of the input of the second downsampling process. 3. The image processing method according to claim 1 or 2, wherein the multi-scale circular sampling process further comprises a third-level sampling process, The third-level sampling process is nested between the second down-sampling process and the second up-sampling process, receives the second down-sampling output as an input of the third-level sampling process, and provides third-level sampling The processed output is used as an input to a second upsampling process, so that the second upsampling process performs an upsampling process based on the second downsampling output; The third-level sampling process includes a third down-sampling process, a third up-sampling process, and a third residual chain addition process, wherein the third down-sampling process performs downsampling based on an input of the third-level sampling process. The sampling process obtains a third down-sampling output, the third up-sampling process performs up-sampling processing based on the third down-sampling output to obtain a third up-sampling output, and the third residual chain addition process combines the third The input of the hierarchical sampling process and the output of the third upsampling are subjected to third residual link addition, and then the result of the third residual link addition is used as the output of the third level sampling process. 4. The image processing method according to claim 1 or 2, wherein the multi-scale cyclic sampling process includes the second-level sampling process performed sequentially multiple times, receiving the first downsampling output as an input of the second-level sampling process for the first time, Each second-level sampling process except the first second-level sampling process receives the output of the previous second-level sampling process as the input of this second-level sampling process, The output of the last second-level sampling process is used as the input of the first up-sampling process. 5. The image processing method according to claim 1 or 2, wherein the at least one multi-scale cyclic sampling process comprises multiple sequentially executed multi-scale cyclic sampling processes, The input of each multi-scale circular sampling process is used as the input of the first-level sampling process in the multi-scale circular sampling process this time, and each time the first-level sampling process in the multi-scale circular sampling process The output of the processing is used as the output of the multi-scale circular sampling processing described herein; The first multi-scale circular sampling process receives the first feature image as an input of the first multi-scale circular sampling process, Each multi-scale cyclic sampling process except the first multi-scale cyclic sampling process receives the output of the previous multi-scale cyclic sampling process as the input of this multi-scale cyclic sampling process, The output of the last multi-scale circular sampling process is used as the output of the at least one multi-scale circular sampling process. 6. The image processing method according to claim 1 or 2, wherein the multi-scale circular sampling process further comprises: After the first downsampling process, the first upsampling process, the second downsampling process, and the second upsampling process, the first downsampling output, the first upsampling output, said second downsampled output and said second upsampled output undergo instance normalization or layer normalization. 7. The image processing method according to claim 1 or 2, further comprising: using a first convolutional neural network to perform the multi-scale circular sampling process; Wherein, the first convolutional neural network includes: A first meta-network, configured to perform the first-level sampling process; The second meta-network is configured to perform the second-level sampling process. 8. The image processing method according to claim 7, wherein, The first meta-network includes: a first sub-network, configured to perform the first down-sampling process; a second subnetwork, configured to perform the first upsampling process; The second meta-network includes: a third sub-network, configured to perform the second down-sampling process; The fourth sub-network is configured to perform the second up-sampling process. 9. The image processing method according to claim 8, wherein each of the first sub-network, the second sub-network, the third sub-network and the fourth sub-network comprises a convolutional layer, One of residual network and dense network. 10. The image processing method according to claim 9, wherein each of the first sub-network, the second sub-network, the third sub-network and the fourth sub-network includes instance normalization layer or layer normalization layer, The instance normalization layer is used to perform instance normalization processing, and the layer normalization layer is used to perform layer normalization processing. 11. The image processing method according to claim 1 or 2, further comprising: get the input image; converting an input image into said first feature image using an analysis network; and: The output of the at least one multi-scale cyclic sampling process is converted into an output image using a synthesis network. 12. A training method for a neural network, wherein, The neural network includes: an analysis network, a first sub-neural network and a synthetic network, The analysis network processes the input image to obtain a first feature image, the first sub-neural network performs at least one multi-scale circular sampling process on the first feature image to obtain a second feature image, and the synthesis network The second feature image is processed to obtain an output image; The training methods include: Get the training input image; processing the training input image using the analysis network to provide a first training feature image; Using the first sub-neural network to perform the at least one multi-scale circular sampling process on the first training feature image to obtain a second training feature image; processing the second training feature image using the synthesis network to obtain a training output image; calculating a loss value of the neural network through a loss function based on the training output image; and Correcting the parameters of the neural network according to the loss value; Wherein, the multi-scale circular sampling processing includes nested first-level sampling processing and second-level sampling processing, The first-level sampling processing includes sequentially executed first down-sampling processing, first up-sampling processing, and first residual chain addition processing, wherein the first down-sampling processing is performed based on the input of the first-level sampling processing The down-sampling process obtains a first down-sampling output, the first up-sampling process performs up-sampling processing based on the first down-sampling output to obtain a first up-sampling output, and the first residual chain addition process combines the first performing a first residual link addition on the input of the first level sampling process and the first upsampling output, and then using the result of the first residual link addition as an output of the first level sampling process; The second-level sampling process is nested between the first down-sampling process and the first up-sampling process, receives the first down-sampling output as the input of the second-level sampling process, and provides a second-level sampling The processed output is used as an input to a first upsampling process, so that the first upsampling process performs an upsampling process based on the first downsampling output; The second-level sampling process includes a second down-sampling process, a second up-sampling process, and a second residual chain addition process performed in sequence, wherein the second down-sampling process is based on the second-level sampling process The input is subjected to down-sampling processing to obtain a second down-sampling output, the second up-sampling processing is performed based on the second down-sampling output to perform up-sampling processing to obtain a second up-sampling output, and the second residual link addition processing combines the obtained The input of the second-level sampling process and the output of the second up-sampling are subjected to a second residual chain addition, and then the result of the second residual chain addition is used as an output of the second-level sampling process. 13. The training method according to claim 12, wherein the size of the output of the first upsampling process is the same as the size of the input of the first downsampling process; The size of the output of the second upsampling process is the same as the size of the input of the second downsampling process. 14. An image processing device, comprising: memory for non-transitory storage of computer readable instructions; and A processor, configured to run the computer-readable instructions, and execute the image processing method according to any one of claims 1-11 when the computer-readable instructions are executed by the processor. 15. A storage medium, storing computer-readable instructions in a non-transitory manner, and when the computer-readable instructions are executed by a computer, the instructions of the image processing method according to any one of claims 1-11 can be executed.

Images