CN114078168B - Image processing model training method, image processing method and electronic device - Google Patents
Image processing model training method, image processing method and electronic device Download PDFInfo
- Publication number
- CN114078168B CN114078168B CN202010839463.2A CN202010839463A CN114078168B CN 114078168 B CN114078168 B CN 114078168B CN 202010839463 A CN202010839463 A CN 202010839463A CN 114078168 B CN114078168 B CN 114078168B
- Authority
- CN
- China
- Prior art keywords
- image
- image processing
- processing model
- processing
- loss
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 247
- 238000000034 method Methods 0.000 title claims abstract description 81
- 238000012549 training Methods 0.000 title claims abstract description 56
- 238000003672 processing method Methods 0.000 title claims abstract description 20
- 230000006870 function Effects 0.000 claims description 143
- 238000013507 mapping Methods 0.000 claims description 57
- 238000004590 computer program Methods 0.000 claims description 11
- 230000006399 behavior Effects 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000012937 correction Methods 0.000 description 34
- 238000010586 diagram Methods 0.000 description 16
- 230000001360 synchronised effect Effects 0.000 description 9
- BANFOXDROWIUGE-UHFFFAOYSA-N deniose Natural products O1C(C)C(O)C(OC)CC1OC1C(C)OC(OC2C(CC(O)OC2C)OC)CC1O BANFOXDROWIUGE-UHFFFAOYSA-N 0.000 description 7
- 238000003705 background correction Methods 0.000 description 5
- 230000005291 magnetic effect Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000003068 static effect Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 3
- 239000003086 colorant Substances 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000005294 ferromagnetic effect Effects 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
- Facsimile Image Signal Circuits (AREA)
Abstract
本申请公开了一种图像处理模型训练方法、图像处理方法及电子设备。图像处理模型训练方法包括：将至少一个第一图像输入至图像处理模型，得到所述图像处理模型输出的至少一个第一图像中每个第一图像对应的第二图像；所述图像处理模型用于对RAW图像进行设定的图像处理；所述第一图像和所述第二图像均为RAW图像；采用配置有至少一个设定的影响因子的损失函数，计算所述至少一个第一图像中的每个第一图像对应的损失值；所述设定的影响因子表征图像信号处理ISP对图像的质量的影响；所述损失值基于对应的第二图像与对应的第三图像计算得到；根据所述至少一个第一图像中的每个第一图像对应的损失值，更新所述图像处理模型的权重参数。
The present application discloses an image processing model training method, an image processing method and an electronic device. The image processing model training method comprises: inputting at least one first image into the image processing model to obtain a second image corresponding to each first image in at least one first image output by the image processing model; the image processing model is used to perform set image processing on the RAW image; the first image and the second image are both RAW images; using a loss function configured with at least one set influence factor, calculating the loss value corresponding to each first image in the at least one first image; the set influence factor represents the influence of the image signal processing ISP on the quality of the image; the loss value is calculated based on the corresponding second image and the corresponding third image; according to the loss value corresponding to each first image in the at least one first image, updating the weight parameter of the image processing model.
Description
Technical Field
The present application relates to the field of image processing, and in particular, to an image processing model training method, an image processing method, and an electronic device.
Background
The related art provides an image Processing method, which processes a RAW image by using an image Processing model, and performs image signal Processing (ISP, image Signal Processing) on the processed RAW image to obtain a three-primary-color RGB image. In the above image processing, there may be a problem in that the image quality of the RGB image is deteriorated. The RAW image data is RAW data obtained by converting the captured light source signals into digital signals by the image sensor.
Disclosure of Invention
In view of the above, an embodiment of the present application is expected to provide an image processing model training method, an image processing device and an electronic device, so as to solve the technical problem that the image quality of an RGB image may be degraded in the related art.
In order to achieve the above purpose, the technical scheme of the application is realized as follows:
The embodiment of the application provides an image processing model training method, which comprises the following steps:
inputting at least one first image into an image processing model to obtain a second image corresponding to each first image in the at least one first image output by the image processing model, wherein the image processing model is used for setting an RAW image;
Calculating a loss value corresponding to each first image in at least one first image by adopting a loss function configured with at least one set influence factor, wherein the set influence factor characterizes the influence of an Image Signal Processing (ISP) on the quality of the image, the ISP is used for converting a RAW image into a three-primary-color RGB image, the loss value is calculated based on a corresponding second image and a corresponding third image, and the third image is a calibration image of the corresponding first image;
And updating the weight parameters of the image processing model according to the loss value corresponding to each first image in the at least one first image.
In the above scheme, the set influencing factors include at least one of the following:
Each gain function of the at least one set gain function corresponds to a class of image gain behavior in the ISP;
and each mapping function in the at least one set mapping function corresponds to one type of image mapping behavior in the ISP.
In the above scheme, the loss function is configured with at least two set influence factors, and the method further comprises:
Determining the execution sequence of each influence factor in the at least two set influence factors when calculating the loss value;
Based on the determined execution order, a loss function configured with the at least two set influencing factors is generated.
In the above solution, the determining an execution sequence of each of the at least two set influence factors when calculating the loss value includes:
determining an order of execution of each of the at least two set influence factors in calculating a penalty value based on the order of execution of each of the at least two set influence factors in the ISP.
In the above solution, when calculating the loss value corresponding to each first image in the at least one first image, the method includes:
Processing the corresponding second image by adopting the loss function to obtain a fourth image;
processing the corresponding third image by adopting the loss function to obtain a fifth image;
and calculating a loss value corresponding to the first image based on the fourth image and the fifth image.
In the above aspect, the calculating, based on the fourth image and the fifth image, a loss value corresponding to the first image includes:
calculating a gray difference value corresponding to each pixel in the first image based on the first gray value of each pixel in the fourth image and the second gray value of each pixel in the fifth image;
And calculating a loss value of the gray value corresponding to the first image based on the gray difference value corresponding to each pixel in the first image.
The embodiment of the application also provides an image processing method, which comprises the following steps:
processing the first RAW image by using a first model to obtain a second RAW image, wherein,
The first model is an image processing model obtained by training by adopting any one of the image processing model training methods.
The embodiment of the application also provides an image processing method, which comprises the following steps:
processing the first RAW image by adopting a first model to obtain a second RAW image;
Performing image signal processing on the second RAW image to obtain a three-primary-color RGB image corresponding to the second RAW image,
The first model is an image processing model obtained by training by adopting any one of the image processing model training methods.
The embodiment of the application also provides electronic equipment, which comprises:
The image processing unit is used for inputting at least one first image into an image processing model to obtain a second image corresponding to each first image in the at least one first image output by the image processing model, wherein the image processing model is used for carrying out image processing on RAW images;
The system comprises a calculating unit, a calculating unit and a calculating unit, wherein the calculating unit is used for calculating a loss value corresponding to each first image in at least one first image by adopting a loss function configured with at least one set influence factor, wherein the set influence factor characterizes the influence of image signal processing ISP on the quality of the image, the ISP is used for converting a RAW image into a three-primary-color RGB image, the loss value is calculated based on a corresponding second image and a corresponding third image, and the third image is a calibration image of the corresponding first image;
and the updating unit is used for updating the weight parameters of the image processing model according to the loss value.
The embodiment of the application also provides electronic equipment, which comprises:
the processing unit is used for processing the first RAW image by adopting the first model to obtain a second RAW image, wherein the first model is an image processing model trained by adopting the image processing model training method in any embodiment.
The embodiment of the application also provides electronic equipment, which comprises:
the first image processing unit is used for processing the first RAW image by adopting the first model to obtain a second RAW image;
The first image processing unit is used for performing image signal processing on the first RAW image to obtain a three-primary-color RGB image corresponding to the first RAW image, wherein the first model is an image processing model trained by the image processing model training method in any embodiment.
The embodiment of the application also provides an electronic device comprising a processor and a memory for storing a computer program capable of running on the processor,
Wherein the processor is configured to, when executing the computer program, either:
executing any one of the image processing model training methods;
A step of executing any one of the above image processing methods.
The embodiment of the application also provides a storage medium having stored thereon a computer program which, when executed by a processor, performs any one of the following:
The step of any one of the image processing model training methods;
A step of any one of the above image processing methods.
In the scheme provided by the embodiment of the application, at least one first image is input into an image processing model, a second image corresponding to each first image in at least one first image output by the image processing model is obtained, a loss function configured with at least one set influence factor is adopted, a loss value corresponding to each first image in at least one first image is calculated, and a weight parameter of the image processing model is updated according to the calculated loss value. Since at least one set influence factor is configured in the loss function, and the set influence factor characterizes the influence of ISP on the quality of the image, the influence of ISP on the quality of the image is considered when the image processing model is trained, therefore, the image processing performance of the image processing model obtained based on the loss function training can be improved, the RAW image output by the image processing model is processed based on the ISP processing flow, and the quality of the obtained RGB image is improved.
Drawings
Fig. 1 is a schematic diagram of an image processing flow provided in the related art;
FIG. 2 is a schematic diagram of an ISP processing flow provided in the related art;
FIG. 3 is a schematic diagram of an implementation flow of an image processing model training method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of an implementation flow of calculating a loss value in an image processing model training method according to an embodiment of the present application;
FIG. 5 is a schematic diagram of a gamma curve according to an embodiment of the present application;
FIG. 6 is a schematic diagram of an implementation flow of calculating a loss value in an image processing model training method according to another embodiment of the present application;
Fig. 7 is a schematic diagram of an implementation flow of an image processing method according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application;
Fig. 9 is a schematic structural diagram of an electronic device according to another embodiment of the present application;
fig. 10 is a schematic diagram of a hardware composition structure of an electronic device according to an embodiment of the application.
Detailed Description
Fig. 1 shows a schematic diagram of an image processing flow provided by the related art. As shown in fig. 1, in the related art, a training-completed image processing model is generally used to process a RAW image acquired by a photographing device, so as to reduce image noise in the RAW image, improve resolution of the RAW image, and the like, obtain a processed RAW image, and perform image processing on the processed RAW image according to an ISP processing flow, so as to convert the RAW image into an RGB image that is convenient for transmission, storage and viewing.
Because the image processing model is trained based on the RAW image and the corresponding RAW image is processed based on the ISP processing flow, the influence of subsequent ISP on the image quality is not considered in the process of training the image processing model based on the RAW image, and the quality of the RGB image obtained after the RAW image output by the image processing model is poor due to the adoption of the ISP processing flow.
Referring to fig. 2 together, fig. 2 is a schematic diagram showing an ISP processing flow provided in the related art. As shown in fig. 2, the ISP processing flow may include lens shading Correction (LENS SHADING Correction), dead point Correction (bad pixel Correction), denoising, automatic white balance Correction, color Correction (Color Correction), gamma Correction, and the like. In practical applications, the ISP processing flow may be more complex (i.e., include more functional items), and the order of execution of the functional items in the ISP processing flow may also be different. Here, the functional items include the above-described lens shading correction, dead pixel correction, denoising, automatic white balance correction, color correction, gamma correction, and the like, and may further include demosaicing (Demosaic). Wherein,
The lens shading correction is used for eliminating the influence of vignetting phenomenon existing in the RAW image data on the image. Vignetting refers to the phenomenon that the middle of an image is brighter, the edges are darker, and the brightness is darker the farther from the center of the image.
The bad pixels refer to pixel points with information errors on the image due to the defects of the corresponding array of the pixel points on the image sensor in the process of converting the optical signals into the electrical signals or due to errors.
White balance is an index describing the accuracy of white color after mixing of three primary colors red (R), green (G), and blue (B) in a display. White balance correction is used for restoring the normal color of an image, and the basic principle is to restore a white object to a white object in any environment. White balance correction may be achieved by white balance gain.
The demosaicing is used for recovering true colors which are consistent with the display of the display device from the RAW image data.
Color correction is used to correct the color of an image and may be achieved by a tone function.
Gamma correction is used to adjust the gray level of the image so that the processed image appears more comfortable to the human eye. The gamma correction is achieved by a gamma mapping function.
In order to solve the above-mentioned problems, an embodiment of the present application provides an image processing model training method, where at least one set influence factor is configured in a loss function, where the set influence factor characterizes the quality influence of an ISP on an image. In training the image processing model, corresponding loss values are calculated using the influence factors configured with at least one setting, so that the weight parameters of the image processing model are updated based on the calculated loss values. Since the influence of ISP on the quality of the image has been considered in training the image processing model, the image processing performance of the image processing model trained based on the loss function can be improved. And processing the RAW image output by the image processing model based on the ISP processing flow, so that the quality of the obtained RGB image is improved.
The technical scheme of the application is further elaborated below by referring to the drawings in the specification and the specific embodiments.
Fig. 3 is a schematic implementation flow chart of an image processing model training method according to an embodiment of the present application. In the embodiment of the present application, the execution subject of the image processing model training method may be electronic devices such as a terminal and a server.
Referring to fig. 3, the image processing model training method provided by the embodiment of the application includes:
s101, inputting at least one first image into an image processing model to obtain a second image corresponding to each first image in at least one first image output by the image processing model, wherein the image processing model is used for setting an RAW image, and the first image and the second image are RAW images.
Here, the image processing model includes, but is not limited to, a model constructed based on a U-Net network model or a convolutional neural network (CNN, convolutional Neural Network) model.
Each of the at least one first image corresponds to a respective third image. The third image is a calibration image of the corresponding first image. Wherein the image quality of the first image is inferior to the image quality of the corresponding third image, and the first image and the corresponding third image are both obtained based on the same shooting scene. That is, the first image and the corresponding third image differ in image quality. Here, the evaluation index of the image quality includes, but is not limited to, resolution, image noise, color difference, and the like.
In an embodiment, the electronic device may acquire at least one first image, input the acquired at least one first image to the image processing model, perform image processing on each first image in the at least one first image by using the image processing model, and acquire a second image corresponding to each first image in the at least one first image output by the image processing model.
In one embodiment, the electronic device may obtain at least one batch (batch) of image samples. Wherein each of the at least one lot includes at least one image sample pair, each image sample pair including one first image and a corresponding third image.
When training is performed each time, the electronic equipment takes a batch as a unit, inputs first images in all image sample pairs in one batch into an image processing model, adopts the image processing model to perform image processing on the input first images, and acquires second images corresponding to each first image in the batch output by the image processing model.
In the present embodiment, the image processing model may be used to perform denoising processing, super-resolution processing, and the like on the input first image. Wherein the super resolution processing refers to a process of restoring an image of a second resolution based on an image or an image sequence of a first resolution, where the first resolution is smaller than the second resolution.
S102, calculating a loss value corresponding to each first image in at least one first image by adopting a loss function configured with at least one set influence factor, wherein the set influence factor characterizes influence of ISP on image quality, the ISP is used for converting a RAW image into a three-primary-color RGB image, the loss value is calculated based on a corresponding second image and a corresponding third image, and the third image is a calibration image of the corresponding first image.
In the case that the image processing model outputs a second image corresponding to each of the at least one first image, the electronic device calculates a loss value corresponding to each of the at least one first image using a loss function configured with at least one set influence factor based on each of the second images output by the image processing model and based on a third image corresponding to each of the corresponding first images.
The loss function (loss function) is used to measure the degree of inconsistency between the image output by the model and the corresponding calibration image. In practical applications, model training is achieved by minimizing the loss function.
In practical applications, factors that have a greater influence on the quality of the image in the ISP processing flow may be preferentially selected and configured in the loss function.
And S103, updating the weight parameters of the image processing model according to the loss value corresponding to each first image in the at least one first image.
And the electronic equipment updates the weight parameters of the image processing model according to the loss value corresponding to each first image in at least one first image so as to improve the quality of the image output by the image processing model. The electronic device carries out back propagation on the loss value corresponding to each first image in the at least one first image in the image processing model, calculates the gradient of the loss function according to the loss value corresponding to each first image in the at least one first image in the process of back propagation of the loss value corresponding to each first image in the at least one first image to each layer of the image processing model, and updates the weight parameter back propagated to the current layer along the descending direction of the gradient.
The electronic device uses the updated weight parameters as weight parameters used in the image processing model when the image processing model is trained next time.
Here, the update stop condition may be set, and when the update stop condition is satisfied, the weight parameter obtained by the last update is determined as the weight parameter used by the trained image processing model. The stopping condition is updated, such as a set training round (epoch), a training round being a process of training the image processing model once based on at least one first image and a corresponding third image. Of course, the update stop condition is not limited thereto, and for example, a loss value corresponding to each of the at least one first image may be less than or equal to a set loss threshold value or the like.
Back propagation is relative to forward propagation, which refers to the feed-forward processing of the model, and the direction of back propagation is opposite to that of forward propagation. The back propagation refers to updating the weight parameters of each layer of the model according to the result output by the model. For example, if the model includes a first convolution layer, a second convolution layer, and a third convolution layer, then forward propagation refers to processing in the order of the first convolution layer-the second convolution layer-the third convolution layer, and reverse propagation refers to updating the weight parameters of each layer in the order of the third convolution layer-the second convolution layer-the first convolution layer.
In the scheme provided by the embodiment, at least one first image is input into an image processing model, a second image corresponding to each first image in at least one first image output by the image processing model is obtained, a loss function configured with at least one set influence factor is adopted, a loss value corresponding to each first image in at least one first image is calculated, and a weight parameter of the image processing model is updated according to the calculated loss value. Since at least one set influence factor is configured in the loss function, and the set influence factor characterizes the influence of ISP on the quality of the image, the influence of ISP on the quality of the image is considered when the image processing model is trained, therefore, the image processing performance of the image processing model obtained based on the loss function training can be improved, the RAW image output by the image processing model is processed based on the ISP processing flow, and the quality of the obtained RGB image is improved.
As another embodiment of the present application, the set influence factor includes at least one of:
Each gain function of the at least one set gain function corresponds to a class of image gain behavior in the ISP;
and each mapping function in the at least one set mapping function corresponds to one type of image mapping behavior in the ISP.
Here, the at least one set gain function includes, but is not limited to, a white balance gain function, an automatic gain function. The white balance gain function corresponds to automatic white balance correction in the ISP and the automatic gain function corresponds to automatic gain control in the ISP. The at least one set mapping function includes, but is not limited to, a gamma mapping function, a tone mapping function. The gamma mapping function corresponds to gamma correction in the ISP and the tone mapping function corresponds to color correction in the ISP.
The gain function is used for performing gain processing on the color value corresponding to each pixel in the RAW image. Each pixel in the RAW image corresponds to a color. The data in the RAW image is arranged in the RGRG/BGBG format. The expression of the gain function may be f (x) =x×gain x, where x represents a color value of an x channel corresponding to one pixel in the RAW image, and x is an R channel, a G channel, or a B channel. gain x characterizes the gain value corresponding to the x-channel in the RAW image.
In some gain functions, the gain values corresponding to R, G, B are all set gain values. In the white balance gain function, the gain value gain G of the G channel is 1, the gain value gain R of the R channel may be a ratio between the color average value of the G channel and the color average value of the R channel, and the gain value gain B of the B channel may be a ratio between the color average value of the G channel and the color average value of the B channel.
The gamma mapping function is used for performing nonlinear operation on the gray value of the input image, so that the gray value of the output image and the gray value of the input image are in an exponential relationship. In practical application, in the case that the data bit width corresponding to each pixel in the RAW image is 8 bits (bit), and the data range corresponding to each pixel is [0,255], the expression of the gamma mapping function may be: wherein i represents an input gray value of an ith pixel in the RAW image, i is a positive integer, gamma (i) represents an output gray value of the ith pixel in the RAW image, and gamma represents a set gamma value. Gamma may be a number of 1/10, 1/5, 1/2, 1,2, 5 or 10, etc.
It should be noted that in other embodiments, the data bit width may be other values, such as 16 bits.
As another embodiment of the present application, the loss function is configured with at least two set influencing factors, and the image processing model training method further comprises:
Determining the execution sequence of each influence factor in the at least two set influence factors when calculating the loss value;
Based on the determined execution order, a loss function configured with the at least two set influencing factors is generated.
Here, at least two set influence factors configured in the loss function may be determined according to the degree of influence of each set influence factor on the quality of the RGB image in the ISP processing flow. For example, based on the degree of influence of all the set influence factors on the quality of the RGB image, at least two set influence factors are determined from among the plurality of set influence factors in order from high to low.
In practical applications, the execution order of each of the at least two set influence factors in calculating the loss value may be random. The electronic device may also determine an order of execution of each of the at least two set influence factors in calculating the loss value based on the set condition.
Here, the set condition may be a degree of influence of each of at least two set influence factors on the quality of the RGB image. For example, the execution order of each of the at least two set influence factors in calculating the loss value is determined in order from high to low based on the degree of influence of each of the at least two set influence factors on the quality of the RGB image. The higher the degree of influence of a set influence factor, the earlier the execution order of the set influence factor in calculating the loss value.
The set condition may also be an execution order of the function item corresponding to each of the at least two set influence factors when calculating the loss value. The function item corresponding to the white balance gain function is automatic white balance correction. The function item corresponding to the gamma mapping function is gamma correction. The function term corresponding to the tone function is color correction.
In practical application, when the at least two set influence factors include a white balance gain function and a gamma mapping function, the expression of the generated loss function is as follows:
Loss=gamma(y)-gamma(f(x))
And the y represents a gain image obtained after gain processing is carried out on a third image corresponding to the first image by adopting a gain function. And gamma (y) represents a mapping image obtained after gamma mapping is carried out on the gain image corresponding to the third image by adopting a gamma mapping function. f (x) represents a gain image obtained after gain processing is performed on a second image corresponding to the first image by using a gain function. gamma (f (x)) represents a mapping image obtained after gamma mapping is performed on the gain image corresponding to the second image by using a gamma mapping function.
Or the generated loss function is expressed as follows:
Loss=gain(yi)-gain(gamma(i))
Wherein y i represents a mapping image obtained after gamma mapping is performed on the third image corresponding to the first image by using a gamma mapping function. gain (y i) represents a gain image obtained by performing gain processing on the mapping image corresponding to the third image by using a gain function. gamma (i) represents a mapping image obtained after gamma mapping is carried out on a second image corresponding to the first image by adopting a gamma mapping function. gain (gamma (i)) represents a gain image obtained by performing gain processing on the mapping image corresponding to the second image by using a gain function.
It should be noted that, in some embodiments, when the at least two set influence factors include a white balance gain function and a gamma mapping function, the expression of the generated loss function may also be:
Loss= { gamma (y) -gamma (f (x)) } 2, or loss= { gamma (y) i)-gain(gamma(i))}2
Here, the expression of the loss function in the present embodiment differs from that in the above embodiment in that the loss function in the present embodiment is used to characterize a variance value calculated based on the third image and the first image as a loss value, and the loss function in the above embodiment is used to characterize a difference value calculated based on the third image and the first image as a loss value. The meaning of the parameters in the expression of the loss function in the present embodiment is the same as the meaning of the parameters in the expression of the loss function in the above-described embodiment.
As another embodiment of the present application, the determining an execution order of each of the at least two set influence factors when calculating the loss value includes:
determining an order of execution of each of the at least two set influence factors in calculating a penalty value based on the order of execution of each of the at least two set influence factors in the ISP.
Here, it is assumed that the ISP processing flow sequentially includes lens shading correction, dead pixel correction, automatic white balance correction, denoising, demosaicing (Demosaic), and gamma correction, and the determined at least two set influence factors include a white balance gain function and a gamma mapping function under the condition that both the automatic white balance correction and the gamma correction affect the quality of the RGB image, and at this time, based on the ISP processing flow, in the execution sequence of each influence factor of the determined at least two set influence factors when calculating the loss value, the influence factor corresponding to the automatic white balance correction is executed first, and then the influence factor corresponding to the gamma correction is executed. At this time, the expression of the generated loss function may be:
Loss=gamma (y) -gamma (f (x)), or loss= { gamma (y) -gamma (f (x)) } 2.
In some embodiments, under the condition that the automatic white balance correction, the denoising point and the gamma correction all affect the quality of the RGB image, the determined at least two set influence factors comprise a white balance gain function, a denoising function and a gamma mapping function, and at the moment, based on the ISP processing flow, in the execution sequence of each influence factor in the determined at least two set influence factors when calculating the loss value, firstly, the influence factor corresponding to the automatic white balance correction is executed, and finally, the influence factor corresponding to the gamma correction is executed after the influence factor corresponding to the denoising point is executed. The expression of the generated loss function may be:
Loss=gamma(Deniose(y))-gamma(Deniose(f(x)))。
The denose (y) represents a denoising image obtained after denoising a gain image corresponding to a third image corresponding to the first image by adopting a denoising function. gamma (Deniose (y)) represents a mapped image obtained after gamma mapping is performed on a denoised image corresponding to a third image corresponding to the first image by using a gamma mapping function. Deniose (f (x)) represents a denoised image obtained after denoising a gain image corresponding to a second image corresponding to the first image using a denoising function. gamma (Deniose (f (x))) represents a mapped image obtained after gamma mapping is performed on a denoised image corresponding to a second image corresponding to the first image using a gamma mapping function.
Or in the execution sequence of each of the determined at least two set influence factors when calculating the loss value, firstly executing the influence factor corresponding to automatic white balance correction, then executing the influence factor corresponding to denoising point, and finally executing the influence factor corresponding to gamma correction, the expression of the generated loss function may be:
Loss={gamma(Deniose(y))-gamma(Deniose(f(x)))}2。
As another embodiment of the present application, fig. 4 is a schematic diagram illustrating an implementation flow of calculating a loss value in the image processing model training method according to the embodiment of the present application. As shown in fig. 4, in calculating a loss value corresponding to each of the at least one first image, the method includes:
And S201, processing the corresponding second image by adopting the loss function to obtain a fourth image.
And S202, processing the corresponding third image by adopting the loss function to obtain a fifth image.
And S203, calculating a loss value corresponding to the first image based on the fourth image and the fifth image.
In practical application, under the condition that at least one set influence factor configured in the Loss function comprises a white balance mapping function and a gamma mapping function, and the Loss function is loss=gamma (y) -gamma (f (x)), the electronic device performs gain processing on a second image corresponding to each first image in at least one first image output by the image processing model by adopting a gain function in the Loss function, so as to obtain a gain image corresponding to the second image corresponding to each first image in at least one first image, and performs gamma mapping on a gain image corresponding to the second image corresponding to the gamma mapping function in the Loss function, so as to obtain a corresponding fourth image of the corresponding second image.
The electronic equipment adopts a gain function in the loss function to perform gain processing on a third image corresponding to each first image in at least one first image to obtain a gain image corresponding to the third image corresponding to each first image in at least one first image, and adopts a gamma mapping function in the loss function to perform gamma mapping on the gain image corresponding to the third image to obtain a fifth image corresponding to the third image.
The electronic device calculates a loss value corresponding to each first image in the at least one first image based on a fourth image corresponding to a second image corresponding to each first image in the at least one first image and a fifth image corresponding to a third image corresponding to each first image in the at least one first image, and calculates an average loss value corresponding to the at least one first image based on the loss value corresponding to each first image in the at least one first image.
In practical application, under the condition that the Loss function is loss= { gamma (y) -gamma (f (x)) } 2, after calculating the Loss value corresponding to each first image in at least one first image, the electronic device performs variance operation based on the Loss value corresponding to each first image in at least one first image, and calculates the variance value of the Loss value corresponding to at least one first image.
It should be noted that, in the process of performing gain processing on the third image corresponding to each first image in the at least one first image by using the gain function in the loss function, the gain value gain G of the G channel in the third image is 1, the gain value gain R of the R channel in the third image may be a ratio between the color average value of all the G channels in the third image and the color average value of all the R channels in the third image, and the gain value gain B of the B channel in the third image may be a ratio between the color average value of all the G channels in the third image and the color average value of all the B channels in the third image.
Gain' G =1 of the gain of the G channel in the corresponding second image during gain processing of the corresponding second image of each of the at least one first image with the gain function in the loss function. Gain' R of the corresponding R-channel in the second image may be the same as gain value gain R of the R-channel in the third image, or may be a ratio between the color average of all G-channels in the corresponding second image and the color average of all R-channels in the second image. Gain' B of the B-channel gain in the corresponding second image may be the same as gain value gain B of the B-channel in the third image, or may be a ratio between the color average value of all G-channels in the corresponding second image and the color average value of all B-channels in the second image.
It should be noted that, for the sake of calculation, when the gamma mapping function in the loss function is used to perform mapping processing on the corresponding image, the gray value of each pixel in the corresponding image may be normalized, so as to convert the gray value of each pixel in the corresponding image into a value between 0 and 1. Referring to fig. 5, fig. 5 is a schematic diagram illustrating a gamma curve according to an embodiment of the application. In fig. 5, X represents an input gray value of each pixel in the corresponding image, and y represents an output gray value of each pixel in the corresponding image. The input gray value refers to the gray value before the image is mapped by adopting the gamma mapping curve, and the output gray value refers to the gray value obtained after the image is mapped by adopting the gamma mapping curve.
Fig. 6 is a schematic flow chart of an implementation of calculating a loss value in an image processing model training method according to another embodiment of the present application. As shown in fig. 6, the calculating, based on the fourth image and the fifth image, a loss value corresponding to the first image includes:
S301, calculating a gray level difference value corresponding to each pixel in the first image based on the first gray level value of each pixel in the fourth image and the second gray level value of each pixel in the fifth image.
S302, calculating a loss value of the gray value corresponding to the first image based on the gray difference value corresponding to each pixel in the first image.
Here, when the Loss function is loss=gamma (y) -gamma (f (x)), the electronic device calculates a gray difference value corresponding to each pixel in the first image based on the first gray value of each pixel in the fourth image corresponding to the corresponding first image and the second gray value of each pixel in the fifth image corresponding to the first image, and calculates a mean value of the gray difference values based on the gray difference values corresponding to each pixel in at least one first image, to obtain a Loss value of the gray value corresponding to the first image.
Under the condition that the Loss function is loss= { gamma (y) -gamma (f (x)) } 2, after the gray level difference value corresponding to each first image in at least one first image is calculated, calculating the variance value of the gray level difference value based on the gray level difference value corresponding to each pixel in each first image in at least one first image, and obtaining the Loss value of the gray level value corresponding to the first image.
As another embodiment of the present application, the image processing model may be put into use after the image processing model has been trained. The electronic device in the embodiment corresponding to the training image processing model may be different from the electronic device in the embodiment that processes the RAW image by using the trained image processing model. For example, the first electronic device trains the image processing model, the second electronic device transfers the trained image processing model to the second electronic device, and the trained image processing model is adopted to process the first RAW image to obtain a second RAW image, wherein the processing capacity of the first electronic device is stronger than that of the second electronic device. In practical application, the first electronic device may be a server, and the second electronic device may be a terminal device such as a mobile phone, a tablet computer, a wearable device, and the like.
The implementation process of the electronic equipment for processing the RAW image by adopting the trained image processing model is as follows:
And the electronic equipment processes the first RAW image by adopting the trained image processing model to obtain a second RAW image. Wherein the resolution of the second RAW image is greater than the resolution of the first RAW image.
Here, the first RAW image may be acquired by the electronic device, or may be acquired by another device and transmitted to the electronic device.
It should be noted that, in the scene of performing super-resolution reconstruction on the RAW image, the electronic device may perform super-resolution processing on the RAW image by using the image processing model trained by the foregoing embodiment to obtain the second RAW image. At this time, the resolution of the second RAW image is greater than that of the first RAW image.
In the context of denoising the RAW image, the electronic device may perform denoising processing on the first RAW image using the trained image processing model, to obtain a second RAW image. At this time, the noise signal in the second RAW image is smaller than the noise signal in the first RAW image.
In practical application, when the electronic device cannot execute the ISP processing flow or the ISP processing performance is poor, the electronic device inputs the acquired first RAW image to the trained image processing model to obtain the second RAW image output by the trained image processing model, and the second RAW image may be sent to the device for executing the ISP processing flow, so that the device for executing the ISP processing flow processes the second RAW image to obtain an RGB image corresponding to the second RAW image.
In the scheme provided by the embodiment, the RAW image is processed by adopting the image processing model obtained by training the loss function provided with at least one set influence factor, and because the influence of ISP on the quality of the image is considered in the training process of the image processing model, the image effect of the RAW image processed by adopting the trained image processing model is better.
As another embodiment of the present application, fig. 7 is a schematic diagram illustrating an implementation flow of an image processing method according to an embodiment of the present application. The electronic device in the embodiment corresponding to the training image processing model may be different from the electronic device in the embodiment that processes the RAW image by using the trained image processing model. For example, the first electronic device trains the image processing model, the third electronic device transfers the trained image processing model to the third electronic device, and processes the RAW image by using the trained image processing model to obtain an RGB image, where the processing capability of the first electronic device is stronger than that of the third electronic device. In practical application, the first electronic device may be a server, and the third electronic device may be a terminal device such as a mobile phone, a tablet computer, or a wearable device.
Referring to fig. 7, an image processing method provided by an embodiment of the present application includes:
s401, processing the first RAW image by adopting a first model to obtain a second RAW image.
S402, performing image signal processing on the second RAW image to obtain a three-primary-color RGB image corresponding to the second RAW image, wherein the first model is an image processing model trained by the image processing model training method according to any embodiment.
In the scheme provided by the embodiment, the image processing model obtained by training the loss function configured with at least one set influence factor is adopted to process the RAW image, and because the influence of ISP on the quality of the image is considered in the training process of the image processing model, the RAW image output by the trained image processing model is processed based on the ISP processing flow, the quality of the obtained RGB image is better, and the quality of the RGB image can be improved.
In order to implement the method of the embodiment of the present application, the embodiment of the present application further provides an electronic device, where the electronic device includes a terminal, a server, and the like, as shown in fig. 8, and the electronic device includes:
The image processing unit 81 is configured to input at least one first image into an image processing model, and obtain a second image corresponding to each first image in at least one first image output by the image processing model;
a calculating unit 82, configured to calculate a loss value corresponding to each first image in the at least one first image by using a loss function configured with at least one set influence factor, where the set influence factor characterizes an influence of an image signal processing ISP on quality of the image, the ISP is configured to convert the RAW image into a trichromatic RGB image, the loss value is calculated based on a corresponding second image and a corresponding third image, and the third image is a calibration image of the corresponding first image;
an updating unit 83, configured to update the weight parameter of the image processing model according to the loss value corresponding to each of the at least one first image.
In an embodiment, the set influence factor includes at least one of:
Each gain function of the at least one set gain function corresponds to a class of image gain behavior in the ISP;
and each mapping function in the at least one set mapping function corresponds to one type of image mapping behavior in the ISP.
In one embodiment, the loss function is configured with at least two set influencing factors, and the image processing model training device further comprises:
A determining unit for determining an execution order of each of the at least two set influence factors when calculating the loss value;
And the generating unit is used for generating a loss function configured with the at least two set influence factors based on the determined execution sequence.
In an embodiment the determining unit is specifically configured to determine an order of execution of each of the at least two set influence factors when calculating the penalty value based on the order of execution of each of the at least two set influence factors in the ISP.
In an embodiment, the calculating unit 82 is specifically configured to, when calculating the loss value of each of the at least one first image when calculating the loss value:
Processing the corresponding second image by adopting the loss function to obtain a fourth image;
processing the corresponding third image by adopting the loss function to obtain a fifth image;
and calculating a loss value corresponding to the first image based on the fourth image and the fifth image.
In one embodiment, the computing unit 82 is specifically configured to:
calculating a gray difference value corresponding to each pixel in the first image based on the first gray value of each pixel in the fourth image and the second gray value of each pixel in the fifth image;
And calculating a loss value of the gray value corresponding to the first image based on the gray difference value corresponding to each pixel in the first image.
In practical application, each unit included in the electronic device may be implemented by a processor in the electronic device. Of course, the processor needs to execute the program stored in the memory to realize the functions of the program modules.
It should be noted that, when the electronic device provided in the above embodiment trains the image processing model, only the division of the program modules is used for illustration, in practical application, the processing allocation may be completed by different program modules according to the need, that is, the internal structure of the electronic device is divided into different program modules, so as to complete all or part of the processing described above. In addition, the electronic device provided in the above embodiment and the image processing model training method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which is not repeated here.
In order to implement the method of the embodiment of the present application, the embodiment of the present application further provides another electronic device, where the electronic device includes a terminal, a server, and the like, and the electronic device includes:
the processing unit is used for processing the first RAW image by adopting the first model to obtain a second RAW image, wherein the first model is an image processing model trained by adopting the image processing model training method in any embodiment.
In practical applications, the processing unit included in the electronic device may be implemented by a processor in the electronic device. Of course, the processor needs to execute the program stored in the memory to realize the functions of the program modules.
In order to implement the method of the embodiment of the present application, another embodiment of the present application further provides an electronic device, where the electronic device includes a terminal, a server, and the like, as shown in fig. 9, and the electronic device includes:
a first image processing unit 91, configured to process the first RAW image by using the first model, so as to obtain a second RAW image;
And a second image processing unit 92, configured to perform image signal processing on the second RAW image to obtain a tricolor RGB image corresponding to the second RAW image, where the first model is an image processing model trained by using the image processing model training method in any of the foregoing embodiments.
In practical application, each unit included in the electronic device may be implemented by a processor in the electronic device. Of course, the processor needs to execute the program stored in the memory to realize the functions of the program modules.
It should be noted that, when the electronic device provided in the above embodiment processes the RAW image, only the division of the program modules is used for illustration, in practical application, the processing allocation may be performed by different program modules according to needs, that is, the internal structure of the electronic device is divided into different program modules, so as to complete all or part of the processing described above. In addition, the electronic device provided in the above embodiment and the image processing method embodiment belong to the same concept, and specific implementation processes of the electronic device are detailed in the method embodiment, which is not described herein again.
Based on the hardware implementation of the program modules, and in order to implement the method of the embodiment of the present application, the embodiment of the present application further provides an electronic device. Fig. 10 is a schematic diagram of a hardware composition structure of an electronic device according to an embodiment of the present application, as shown in fig. 10, the electronic device includes:
a communication interface 1 capable of information interaction with other devices such as electronic devices and the like;
the processor 2 is connected with the communication interface 1 to realize information interaction with other devices, and is used for executing any one of the following when running a computer program:
the image processing model training method provided by the one or more technical schemes;
the image processing method provided by one or more of the above technical solutions.
And the computer program is stored on the memory 3.
Of course, in practice, the various components in the electronic device are coupled together by a bus system 4. It will be appreciated that the bus system 4 is used to enable connected communications between these components. The bus system 4 comprises, in addition to a data bus, a power bus, a control bus and a status signal bus. But for clarity of illustration the various buses are labeled as bus system 4 in fig. 10.
The memory 3 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include any computer program for operating on an electronic device.
It will be appreciated that the memory 3 may be either volatile memory or nonvolatile memory, and may include both volatile and nonvolatile memory. The non-volatile Memory may be, among other things, a Read Only Memory (ROM), a programmable Read Only Memory (PROM, programmable Read-Only Memory), erasable programmable Read-Only Memory (EPROM, erasable Programmable Read-Only Memory), electrically erasable programmable Read-Only Memory (EEPROM, ELECTRICALLY ERASABLE PROGRAMMABLE READ-Only Memory), Magnetic random access Memory (FRAM, ferromagnetic random access Memory), flash Memory (Flash Memory), magnetic surface Memory, optical disk, or compact disk-Only Memory (CD-ROM, compact Disc Read-Only Memory), which may be disk Memory or tape Memory. The volatile memory may be random access memory (RAM, random Access Memory) which acts as external cache memory. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM, static Random Access Memory), synchronous static random access memory (SSRAM, synchronous Static Random Access Memory), dynamic random access memory (DRAM, dynamic Random Access Memory), synchronous dynamic random access memory (SDRAM, synchronous Dynamic Random Access Memory), and, Double data rate synchronous dynamic random access memory (DDRSDRAM, double Data Rate Synchronous Dynamic Random Access Memory), enhanced synchronous dynamic random access memory (ESDRAM, enhanced Synchronous Dynamic Random Access Memory), synchronous link dynamic random access memory (SLDRAM, sync Link Dynamic Random Access Memory), direct memory bus random access memory (DRRAM, direct Rambus Random Access Memory). The memory 3 described in the embodiments of the present application is intended to comprise, without being limited to, these and any other suitable types of memory.
The method disclosed in the above embodiment of the present application may be applied to the processor 2 or implemented by the processor 2. The processor 2 may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in the processor 2 or by instructions in the form of software. The processor 2 described above may be a general purpose processor, DSP, or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. The processor 2 may implement or perform the methods, steps and logic blocks disclosed in embodiments of the present application. The general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method disclosed in the embodiment of the application can be directly embodied in the hardware of the decoding processor or can be implemented by combining hardware and software modules in the decoding processor. The software modules may be located in a storage medium in the memory 3 and the processor 2 reads the program in the memory 3 to perform the steps of the method described above in connection with its hardware.
The corresponding flow in each method of the embodiments of the present application is implemented when the processor 2 executes the program, and for brevity, will not be described in detail herein.
In an exemplary embodiment, the present application also provides a storage medium, i.e. a computer storage medium, in particular a computer readable storage medium, for example comprising a memory 3 storing a computer program executable by the processor 2 for performing the steps of the method described above. The computer readable storage medium may be FRAM, ROM, PROM, EPROM, EEPROM, flash Memory, magnetic surface Memory, optical disk, or CD-ROM.
In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is merely a logical function division, and there may be additional divisions of actual implementation, such as multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. In addition, the various components shown or discussed may be coupled or directly coupled or communicatively coupled to each other via some interface, whether indirectly coupled or communicatively coupled to devices or units, whether electrically, mechanically, or otherwise.
The units described as separate components may or may not be physically separate, and components displayed as units may or may not be physical units, may be located in one place, may be distributed on a plurality of network units, and may select some or all of the units according to actual needs to achieve the purpose of the embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing module, or each unit may be separately used as a unit, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of hardware plus a form of software functional unit.
It will be appreciated by those of ordinary skill in the art that implementing all or part of the steps of the above method embodiments may be implemented by hardware associated with program instructions, where the above program may be stored in a computer readable storage medium, where the program when executed performs the steps comprising the above method embodiments, where the above storage medium includes various media that may store program code, such as a removable storage device, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic or optical disk, etc.
The technical schemes described in the embodiments of the present application may be arbitrarily combined without any collision.
The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. An image processing model training method, comprising:
The method comprises the steps of inputting at least one first image into an image processing model to obtain a second image corresponding to each first image in at least one first image output by the image processing model, wherein the image processing model is used for setting an RAW image, and the set image processing at least comprises denoising processing and/or super-resolution processing;
Calculating a loss value corresponding to each first image in at least one first image by adopting a loss function configured with at least one set influence factor, wherein the set influence factor characterizes the influence of an Image Signal Processing (ISP) on the quality of the image, the ISP is used for converting a RAW image into a three-primary-color RGB image, the loss value is calculated based on a corresponding second image and a corresponding third image, and the third image is a calibration image of the corresponding first image;
updating the weight parameter of the image processing model according to the loss value corresponding to each first image in the at least one first image, wherein,
The set influence factors include at least one of:
Each gain function of the at least one set gain function corresponds to a class of image gain behavior in the ISP;
and each mapping function in the at least one set mapping function corresponds to one type of image mapping behavior in the ISP.
2. The image processing model training method of claim 1, wherein the loss function is configured with at least two set influencing factors, the method further comprising:
Determining the execution sequence of each influence factor in the at least two set influence factors when calculating the loss value;
Based on the determined execution order, a loss function configured with the at least two set influencing factors is generated.
3. The image processing model training method according to claim 2, wherein determining an execution order of each of the at least two set influence factors in calculating the loss value comprises:
determining an order of execution of each of the at least two set influence factors in calculating a penalty value based on the order of execution of each of the at least two set influence factors in the ISP.
4. The image processing model training method according to claim 1, wherein in calculating the loss value corresponding to each of the at least one first image, the method comprises:
Processing the corresponding second image by adopting the loss function to obtain a fourth image;
processing the corresponding third image by adopting the loss function to obtain a fifth image;
and calculating a loss value corresponding to the first image based on the fourth image and the fifth image.
5. The image processing model training method of claim 4, wherein calculating a loss value corresponding to a first image based on the fourth image and the fifth image comprises:
calculating a gray difference value corresponding to each pixel in the first image based on the first gray value of each pixel in the fourth image and the second gray value of each pixel in the fifth image;
And calculating a loss value of the gray value corresponding to the first image based on the gray difference value corresponding to each pixel in the first image.
6. An image processing method, comprising:
processing the first RAW image by using a first model to obtain a second RAW image, wherein,
The first model is an image processing model trained by the image processing model training method according to any one of claims 1 to 5.
7. An image processing method, comprising:
processing the first RAW image by adopting a first model to obtain a second RAW image;
Performing image signal processing on the second RAW image to obtain a three-primary-color RGB image corresponding to the second RAW image,
The first model is an image processing model trained by the image processing model training method according to any one of claims 1 to 5.
8. An electronic device, comprising:
The image processing unit is used for inputting at least one first image into an image processing model to obtain a second image corresponding to each first image in at least one first image output by the image processing model, wherein the image processing model is used for performing set image processing on a RAW image, and the set image processing at least comprises denoising processing and/or super-resolution processing;
The system comprises a calculation unit, a calculation unit and a display unit, wherein the calculation unit is used for calculating a loss value corresponding to each first image in at least one first image by adopting a loss function configured with at least one set influence factor, the set influence factor characterizes the influence of Image Signal Processing (ISP) on the quality of the image, the ISP is used for converting a RAW image into a three-primary-color RGB image, the loss value is calculated based on a corresponding second image and a corresponding third image, the third image is a calibration image of the corresponding first image, and the set influence factor comprises at least one set gain function, each gain function in the at least one set gain function corresponds to one type of image gain behavior in the ISP, and at least one set mapping function, each mapping function in the at least one set mapping function corresponds to one type of image mapping behavior in the ISP;
and the updating unit is used for updating the weight parameter of the image processing model according to the loss value corresponding to each first image in the at least one first image.
9. An electronic device comprising a processor and a memory for storing a computer program capable of running on the processor,
Wherein the processor is configured to execute any one of the following when executing the computer program:
the image processing model training method of any one of claims 1 to 5;
The image processing method of claim 6;
the image processing method of claim 7.
10. A storage medium having a computer program stored thereon, wherein the computer program when executed by a processor performs any one of:
the image processing model training method of any one of claims 1 to 5;
The image processing method of claim 6;
the image processing method of claim 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010839463.2A CN114078168B (en) | 2020-08-19 | 2020-08-19 | Image processing model training method, image processing method and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010839463.2A CN114078168B (en) | 2020-08-19 | 2020-08-19 | Image processing model training method, image processing method and electronic device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114078168A CN114078168A (en) | 2022-02-22 |
| CN114078168B true CN114078168B (en) | 2025-03-14 |
Family
ID=80281820
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010839463.2A Active CN114078168B (en) | 2020-08-19 | 2020-08-19 | Image processing model training method, image processing method and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114078168B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110365949A (en) * | 2018-03-26 | 2019-10-22 | 展讯通信(天津)有限公司 | A kind of bearing calibration of image color cast, device and electronic equipment |
| CN110555877A (en) * | 2018-05-31 | 2019-12-10 | 杭州海康威视数字技术股份有限公司 | Image processing method, device and equipment and readable medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10796200B2 (en) * | 2018-04-27 | 2020-10-06 | Intel Corporation | Training image signal processors using intermediate loss functions |
| US11265446B2 (en) * | 2018-10-18 | 2022-03-01 | Sony Corporation | Frame handling for ML-based upscaling |
| WO2020126023A1 (en) * | 2018-12-21 | 2020-06-25 | Huawei Technologies Co., Ltd. | Image processor |
| US10817991B2 (en) * | 2019-01-14 | 2020-10-27 | Advanced New Technologies Co., Ltd. | Methods for deep-learning based super-resolution using high-frequency loss |
| CN110766610B (en) * | 2019-10-28 | 2023-06-09 | 维沃移动通信有限公司 | Reconstruction method of super-resolution image and electronic equipment |
| CN111127336B (en) * | 2019-11-18 | 2023-05-02 | 复旦大学 | An Image Signal Processing Method Based on Adaptive Selection Module |
-
2020
- 2020-08-19 CN CN202010839463.2A patent/CN114078168B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110365949A (en) * | 2018-03-26 | 2019-10-22 | 展讯通信(天津)有限公司 | A kind of bearing calibration of image color cast, device and electronic equipment |
| CN110555877A (en) * | 2018-05-31 | 2019-12-10 | 杭州海康威视数字技术股份有限公司 | Image processing method, device and equipment and readable medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114078168A (en) | 2022-02-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3609177B1 (en) | Control method, control apparatus, imaging device, and electronic device | |
| CN108632537B (en) | Control method and device, imaging device, computer device, and readable storage medium | |
| US10630906B2 (en) | Imaging control method, electronic device and computer readable storage medium | |
| CN110930301A (en) | Image processing method, image processing device, storage medium and electronic equipment | |
| CN108270977A (en) | Control method and device, imaging device, computer device and readable storage medium | |
| WO2020034702A1 (en) | Control method, device, electronic equipment and computer readable storage medium | |
| CN109040607B (en) | Imaging control method, imaging control device, electronic device and computer-readable storage medium | |
| WO2020010638A1 (en) | Method and device for detecting defective pixel in image | |
| CN111885312A (en) | HDR image imaging method, system, electronic device and storage medium | |
| CN102023456B (en) | Light metering weight adjustment method and device | |
| CN115942128B (en) | A design and implementation method of ISP system based on heterogeneous platform | |
| JP2001103497A (en) | Image processor | |
| CN113781958B (en) | Optical compensation method and device for display panel | |
| US9715720B1 (en) | System and method for reducing image noise | |
| CN105007474A (en) | Image processing device and image processing method | |
| CN114078168B (en) | Image processing model training method, image processing method and electronic device | |
| CN113658050B (en) | Image denoising method, denoising device, mobile terminal and storage medium | |
| CN116744125B (en) | Image color data processing method, device, equipment and storage medium | |
| CN117579944A (en) | Overexposed image processing method and system based on saturation correction and detail enhancement | |
| CN113891081A (en) | Video processing method, device and equipment | |
| CN114169380A (en) | Image processing model training method, device, electronic device and storage medium | |
| CN113194267B (en) | Image processing method and device and photographing method and device | |
| JP2021114186A (en) | Image processing device, image processing method, and program | |
| US20240070826A1 (en) | Image processing method, image processing apparatus, and storage medium | |
| CN111242087A (en) | Object recognition method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |