CN115660950B - Color migration method, device and medium for generating LUT based on neural network - Google Patents

Abstract

The invention discloses a color migration method, device and medium for generating LUT based on a neural network, belonging to the field of computer graphics and computer vision, comprising the steps of: selecting a content map and a reference map, and sending both the content map and the reference map to the neural network The network generates a LUT; then the generated LUT is applied to the content image to generate a color-migrated image, and the color migration is realized by constraining the style loss and content loss. The invention improves the speed and stability of color migration.

Description

Color migration method, device and medium for generating LUT based on neural network

technical field

The present invention relates to the fields of computer graphics and computer vision, and more specifically, relates to a color migration method, device and medium for generating LUT based on a neural network.

Background technique

With the rapid development of digital media, especially in the process of digital creative production, color migration has become a very important task in video editing. The common color migration method, the given reference image and content image, is realized by calculating the mean and variance between the reference image and the content image. This method is time-consuming to calculate larger images, and can only achieve global color transfer. Some studies also use local methods for color transfer, but the time consumption will increase with the increase of the processed area.

Contents of the invention

The purpose of the present invention is to overcome the deficiencies of the prior art, to provide a color migration method, device and medium based on a neural network to generate a LUT, to realize the generation of a color LUT between a reference image and content based on a neural network, so as to transfer the color to the LUT Applied in video editing, it provides intelligent technical support for digital creative production, and improves the speed and stability of color migration.

The purpose of the present invention is achieved by the following scheme:

A method for color migration based on neural network generation LUT, comprising the following steps:

Select the content image and reference image, and send both the content image and reference image to the neural network to generate LUT; then apply the generated LUT to the content image to generate a color-migrated image, and achieve color migration by constraining style loss and content loss .

Further, before selecting the content map and the reference map, the steps include: constructing a color data set, and selecting the content map and the reference map on the constructed color data set.

Further, the selection of the content map and the reference map includes the steps of: given a style map and a piece of content video, selecting a frame of the content video as the content map, and using the style map as the reference map;

Applying the generated LUT to the content map includes the step of: applying the generated LUT to frames in the video.

Further, the content image and the reference image are both sent to the neural network to generate LUT; then the generated LUT is applied to the content image to generate a color-migrated image, and the color migration is realized by constraining the style loss and content loss, including sub- step:

Randomly select the reference image and the content image on the constructed color data set, and randomly crop the reference image and the content image in the image space according to the same resolution, and the cropped reference image and content map As the input of the neural network, it is used to train the neural network to generate LUT;

When starting to train the neural network, apply the LUT generated by the neural network to the content image to obtain the image after color migration , this process is expressed as:

in, ( , ) means the reference image and content map The LUT obtained by inputting into the neural network; Indicates the content map The color of each pixel is searched through the LUT generated by the neural network to obtain the color of the corresponding position;

During the training process, the reference image is extracted , content map And the graph obtained by color migration deep features of , and , respectively calculate the corresponding style loss and loss of content ,Expressed as:

=mse(mean( )-mean( ))+(std( )-std( ))

=mse(mean( )-mean( ))+(std( )-std( ))

Among them, mean represents the mean value of image features; std represents the variance of image features; mse represents the mean square error of the matrix.

Further, the neural network performs convolutional feature extraction by inputting style maps and content maps, and then fuses the extracted features to finally generate weight values of N basic LUT linear combinations, where N is an integer.

Further, the neural network includes N initialized basic LUTs, values in these basic LUTs are updated according to gradient backpropagation during network training, and N is an integer.

Further, after the generated LUT is applied to the frames in the video, the generating the color-shifted image includes a step of: migrating the color of the content image according to the color of the style image.

Further, the selecting the frame of the content video as the content map includes the steps of: taking the first frame of the video as the content map;

After the generated LUT is applied to the first frame in the video, the generating the image after color migration includes the step of performing color migration on all frames of the video through the color migration LUT generated by the neural network according to the style image.

A computer device, the computer device includes a processor and a memory, a computer program is stored in the memory, when the computer program is loaded by the processor and executes the method as described above.

A readable storage medium stores a program on the readable storage medium, and when the program is loaded by a processor, the above method is implemented.

The beneficial effects of the present invention include:

The technical solution of the present invention can use the LUT for color migration of the style map and the content map generated based on the neural network, so that the LUT can be used in any video color migration task.

The technical solution of the present invention makes video color migration more stable by generating a color migration LUT.

The technical solution of the present invention combines the neural network to generate the color migration LUT of the reference image and the content image, and uses the LUT in the video color migration, which can not only use the neural network method to automatically generate the LUT, but also take advantage of the advantages of the LUT for rapid color correction. Quickly migrate the style transfer of the reference image to the content video, improving the speed and stability of color transfer.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained according to these drawings without any creative effort.

Fig. 1 is the flowchart of the embodiment method of the present invention;

Fig. 2 is a schematic diagram of color migration by the method of the embodiment of the present invention;

FIG. 3 is a schematic diagram of video color migration by using a generated LUT in a method according to an embodiment of the present invention.

Detailed ways

All features disclosed in all embodiments in this specification, or steps in all implicitly disclosed methods or processes, except for mutually exclusive features and/or steps, can be combined and/or extended and replaced in any way.

In view of the problems in the background, the inventors of the present invention found that in recent years, using LUTs to stylize shots or captured videos has become a common method in film production because LUTs have faster computing power. At the same time, LUT is more stable in video than other methods of color transfer. However, the production of these LUTs is usually the result of careful tuning by colorists.

In the existing methods, there is no use of the reference image and the content image to generate the LUT for color migration. After creative thinking, the technical solution of the present invention combines the neural network to generate the color migration LUT of the reference image and the content image, and uses the LUT in the video In terms of color migration, it can not only use the method of neural network to automatically generate LUT, but also take advantage of the advantages of fast color correction of LUT to quickly transfer the style of the reference picture to the content video, improving the speed and stability of color migration.

In a further inventive concept, in order to quickly generate LUTs through reference images and content images, and then realize color migration of videos, the technical solution of the present invention innovatively designs a color migration scheme for neural network generated LUTs to provide intelligent digital creative production. Technical Support.

In the specific implementation process, as shown in Figure 1, a color migration method based on neural network generation LUT provided by the embodiment of the present invention includes the following steps:

S1, build a large-scale color data set: the data set needs to be diverse in color and rich in content;

S2, by training the neural network to generate the LUT for color migration: randomly select the content image and reference image on the large data set, send them to the network to generate the LUT, apply the LUT to the content image to generate the image after color migration, and constrain the style loss and content loss to achieve color transfer;

S3, use the trained neural network for video color migration: Given a style image and a content video, select the first frame of the content video as the content image, send it to the neural network to generate a color migration LUT, and then The LUT acts on the video to realize the color transfer of the video.

In the actual application process, in step S1, the construction of a large-scale color data set requires various colors and rich content.

In the actual application process, in step S2, as shown in Figure 2, the LUT used to generate the color migration by training the neural network includes the following sub-steps:

S21: Randomly select the reference image and the content image from the constructed large-scale color data set, and randomly crop the reference image and the content image in the image space with the same resolution, and the cropped reference image and content map As the input of the neural network, it is used to train the neural network to generate LUT;

S22: When training the neural network, apply the LUT generated by the neural network to the content image to obtain the image after color migration , this process can be expressed as:

in, ( , ) means the reference image and content map The LUT obtained by inputting into the neural network; Indicates the content map The color of each pixel is searched through the LUT generated by the neural network to obtain the color of the corresponding position;

S23: During the training process, extract the reference image , content map And the graph obtained by color migration deep features of ,and , respectively calculate the corresponding style loss and loss of content , which can be expressed as:

=mse(mean( )-mean( ))+(std( )-std( ))

=mse(mean( )-mean( ))+(std( )-std( ))

Among them, mean represents the mean value of image features; std represents the variance of image features; mse represents the mean square error of the matrix.

In the actual application process, in step S2, the neural network contains N initialized basic LUTs, and the values in these LUTs are updated according to the gradient backpropagation during the network training process.

In the actual application process, in step S2, the neural network is designed as a convolutional feature extraction through the input style map and content map, and then the extracted features are fused to finally generate the weight value of the linear combination of N basic LUTs.

In the actual application process, in step S3, given a style map and a content map, the trained neural network is used to generate a color-transferred LUT, and the color of the content map is transferred according to the color of the style map.

In the actual application process, in step S3, as shown in Figure 3, the first frame of the video is used as the content map, and the color transfer LUT generated by the neural network is used to perform color transfer on all frames of the video according to the given style map.

It should be noted that within the scope of protection defined in the claims of the present invention, the following embodiments can be obtained from the above specific implementation methods, such as disclosed technical principles, disclosed technical features or implicitly disclosed technical features, etc., Combining and/or extending, replacing in any logical way.

Example 1

A method for color migration based on neural network generation LUT, comprising the following steps:

Select the content image and reference image, and send both the content image and reference image to the neural network to generate LUT; then apply the generated LUT to the content image to generate a color-migrated image, and achieve color migration by constraining style loss and content loss .

Example 2

On the basis of Embodiment 1, before selecting the content map and the reference map, the steps include: constructing a color data set, and selecting the content map and the reference map on the constructed color data set.

Example 3

On the basis of Embodiment 1, the selection of the content map and the reference map includes the steps of: given a style map and a section of content video, selecting frames of the content video as the content map, and using the style map as the reference map;

Applying the generated LUT to the content map includes the step of: applying the generated LUT to frames in the video.

Example 4

On the basis of Embodiment 2, the content map and the reference map are sent to the neural network to generate the LUT; then the generated LUT is applied to the content map to generate a color-shifted map, which is realized by constraining the style loss and content loss Color migration, including substeps:

Randomly select the reference image and the content image on the constructed color data set, and randomly crop the reference image and the content image in the image space according to the same resolution, and the cropped reference image and content map As the input of the neural network, it is used to train the neural network to generate LUT;

When starting to train the neural network, apply the LUT generated by the neural network to the content image to obtain the image after color migration , this process is expressed as:

in, ( , ) means the reference image and content map The LUT obtained by inputting into the neural network;

During the training process, the reference image is extracted , content map And the graph obtained by color migration deep features of , and , respectively calculate the corresponding style loss and loss of content ,Expressed as:

=mse(mean( )-mean( ))+(std( )-std( ))

=mse(mean( )-mean( ))+(std( )-std( ))

Among them, mean represents the mean value of image features; std represents the variance of image features; mse represents the mean square error of the matrix.

Example 5

On the basis of Embodiment 1 or Embodiment 4, the neural network performs convolution feature extraction by inputting style maps and content maps, and then fuses the extracted features to finally generate weight values of N basic LUT linear combinations, N is an integer.

Example 6

On the basis of Embodiment 1 or Embodiment 4, the neural network includes N initialized basic LUTs, and values in these basic LUTs are updated according to gradient backpropagation during network training, and N is an integer.

Example 7

On the basis of Embodiment 3, after the generated LUT is applied to the frames in the video, the generating the color-shifted map includes the step of: migrating the color of the content map according to the color of the style map.

Example 8

On the basis of embodiment 3, said selecting the frame of the content video as the content map includes the steps of: taking the first frame of the video as the content map;

After the generated LUT is applied to the first frame in the video, the generating the image after color migration includes the step of performing color migration on all frames of the video through the color migration LUT generated by the neural network according to the style image.

Example 9

A computer device. The computer device includes a processor and a memory, and a computer program is stored in the memory. When the computer program is loaded by the processor and executed as described in any one of Embodiment 1 to Embodiment 4, method.

Example 10

A readable storage medium, in which a program is stored, and when the program is loaded by a processor, the method described in any one of Embodiment 1 to Embodiment 4 is implemented.

The units described in the embodiments of the present invention may be implemented by software or by hardware, and the described units may also be set in a processor. Wherein, the names of these units do not constitute a limitation of the unit itself under certain circumstances.

According to an aspect of embodiments of the present invention, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the various optional implementation manners above.

As another aspect, an embodiment of the present invention also provides a computer-readable medium, which may be included in the electronic device described in the above-mentioned embodiments; in electronic equipment. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by an electronic device, the electronic device is made to implement the methods described in the above-mentioned embodiments.

The parts not involved in the present invention are the same as the prior art or can be realized by adopting the prior art.

The above-mentioned technical solution is only an embodiment of the present invention. For those skilled in the art, on the basis of the application methods and principles disclosed in the present invention, it is easy to make various types of improvements or deformations, and is not limited to The methods described in the above specific embodiments of the present invention, therefore, the above-described methods are only preferred and not limiting.

In addition to the above examples, those skilled in the art obtain inspiration from the above disclosure or use knowledge or technology in the relevant field to make changes to obtain other embodiments. The features of each embodiment can be interchanged or replaced. The changes and changes made by those skilled in the art If they do not depart from the spirit and scope of the present invention, they should all be within the protection scope of the appended claims of the present invention.

Claims (10)

1. A color migration method for generating a LUT based on a neural network, comprising the steps of:

selecting a content graph and a reference graph, and sending the content graph and the reference graph into a neural network to generate an LUT; and then the generated LUT is acted on the content graph to generate a graph after color migration, and color migration is realized by restricting style loss and content loss.

2. The color migration method based on neural network generated LUT of claim 1, comprising the steps of, before extracting the content map and the reference map: and constructing a color data set, and selecting a content image and a reference image on the constructed color data set.

3. The method for color migration based on neural network generated LUT of claim 1, wherein the selecting the content map and the reference map comprises the steps of: giving a style sheet and a segment of content video, selecting frames of the content video as a content sheet, and taking the style sheet as a reference sheet;

said acting the generated LUT on the content map comprises the steps of: the generated LUT is applied to frames in the video.

4. The color migration method based on neural network generation LUT of claim 2, wherein the feeding both the content map and the reference map into the neural network generation LUT; and then the generated LUT is acted on the content graph to generate a graph after color migration, and the color migration is realized by restricting style loss and content loss, and the method comprises the following substeps:

randomly selecting a reference image and a content image on the constructed color data set, randomly cutting the reference image and the content image in an image space according to the same resolution, and cutting the cut reference image

And content graph

As the input of the neural network, the LUT is generated by training the neural network;

when the training of the neural network is started, the LUT generated by the neural network is acted on the content graph to obtain the picture after color migration

This process is represented as:

wherein,

(

,

) Show the reference diagram

And content graph

Inputting the LUT obtained in the neural network;

content presentation graph

Searching the color of each pixel point through an LUT generated by a neural network to obtain the color of the corresponding position;

extracting reference picture in training process

Content map

And images obtained by color migration

Depth feature of

、

And

respectively calculating corresponding style losses

And content loss

Expressed as:

=mse(mean(

)-mean(

))+(std(

)-std(

))

=mse(mean(

)-mean(

))+(std(

)-std(

))

wherein mean represents the mean of the image features; std represents the variance of the image feature; mse represents the mean square error of the matrix.

5. The color migration method according to any one of claims 1 or 4, wherein the neural network performs convolution feature extraction by inputting a style diagram and a content diagram, and then fuses the extracted features to finally generate weight values of linear combinations of N basic LUTs, where N is an integer.

6. The method for color migration based on neural network generated LUT of any one of claims 1 or 4, wherein the neural network comprises N initialized basic LUTs, values in the basic LUTs are updated according to gradient back propagation during network training, and N is an integer.

7. The neural network generated LUT based color migration method of claim 3, wherein after said applying the generated LUT on a frame in a video, said generating a color migrated map comprises the steps of: and migrating the color of the content graph according to the color of the style graph.

8. The method of claim 3, wherein the selecting the frame of the content video as the content map comprises: taking the first frame of the video as a content graph;

after applying the generated LUT to the first frame in the video, said generating the color shifted map comprises the steps of: and performing color migration on all frames of the video through a color migration LUT generated by a neural network according to the style diagram.

9. A computer arrangement comprising a processor and a memory, in which a computer program is stored which, when loaded by the processor, carries out the method according to any one of claims 1 to 4.

10. A readable storage medium, in which a program is stored which, when loaded by a processor, carries out the method according to any one of claims 1 to 4.

Images