CN111667399B - Method for training style transfer model, method and device for video style transfer - Google Patents

Abstract

This application discloses a training method for a style transfer model in the field of artificial intelligence, a method and a device for video style transfer, including: obtaining training data; performing image style transfer on N frames of sample content images according to the sample style images through a neural network model Processing to obtain N frames of predicted composite images; according to the image loss function between N frames of sample content images and N frames of predicted composite images, determine the parameters of the neural network model. The image loss function includes a low-rank loss function, and the low-rank loss function is used for Indicates the difference between the first low-rank matrix and the second low-rank matrix. The first low-rank matrix is obtained based on N frames of sample content images and optical flow information, and the second low-rank matrix is based on N frames of predicted composite images and optical flow information. The optical flow information is obtained from the flow information, and the optical flow information is used to represent the position difference of corresponding pixel points between two adjacent frames of images in the N frame sample content images. The technical solution of the present application can improve the stability of the video after style transfer processing.

Description

Method for training style transfer model, method and device for video style transfer

technical field

The present application relates to the field of artificial intelligence, and more specifically, to a training method for a style transfer model in the field of computer vision, a method and a device for video style transfer.

Background technique

Artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is the branch of computer science that attempts to understand the nature of intelligence and produce a new class of intelligent machines that respond in ways similar to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

Image rendering tasks such as image style transfer have a wide range of application requirements on terminal devices. With the rapid improvement of the performance of terminal equipment and network performance, the entertainment needs of terminal equipment have gradually changed from image level to video level, that is, from the image style transfer processing of single images to the image style transfer processing of videos; Compared with the style transfer task, the video style transfer task not only considers the stylization effect of the image, but also considers the stability between the multiple frames included in the video, so as to ensure the fluency of the video after the image style transfer process.

Therefore, how to improve the stability of video after image migration processing has become an urgent problem to be solved.

Contents of the invention

This application provides a training method for a style transfer model, a method and a device for video style transfer, by introducing a low-rank loss function in the process of training a style transfer model for video, the video after style transfer can be synchronized with the original video The stability of the method can improve the stability of the video after style transfer processing obtained by the target transfer model.

In the first aspect, a method for training a style transfer model is provided, including: obtaining training data, wherein the training data includes N frames of sample content images, sample style images, and N frames of composite images, and the N frames of composite images are An image obtained by performing image style transfer processing on the N frames of sample content images according to the sample style image, where N is an integer greater than or equal to 2; using a neural network model to transfer the N frames of sample content according to the sample style image The image is subjected to image style transfer processing to obtain N frames of predicted composite images; according to the image loss function between the N frames of sample content images and the N frames of predicted composite images, the parameters of the neural network model are determined,

Wherein, the image loss function includes a low-rank loss function, and the low-rank loss function is used to represent the difference between the first low-rank matrix and the second low-rank matrix, and the first low-rank matrix is based on the N frame sample content image and optical flow information, the second low-rank matrix is obtained based on the N-frame predicted composite image and the optical flow information, and the optical flow information is used to represent the N-frame sample content The position difference of corresponding pixels between two adjacent frames of images in the image.

It should be understood that, for a matrix composed of multiple frames of images, a low-rank matrix can be used to represent regions that appear in all N frames of images and are not motion boundaries. A sparse matrix can be used to represent regions that appear intermittently in N frames of images; for example, a sparse matrix can refer to regions that newly appear or disappear at image borders due to camera movement, or border regions of moving objects.

In the embodiment of the present application, when training the target style transfer model for video style transfer processing, a low-rank loss function is introduced. By introducing a low-rank loss function, it is possible to make the images that appear in adjacent multiple frames of the video to be processed and are not The region of the motion boundary remains the same after the style transfer process, that is, the rank of the region in the style transfer processed video is close to the rank of the region in the video to be processed, thereby improving the stability of the style transfer processed video.

It should be understood that the image style transfer process refers to the process of fusing an image that requires style transfer, that is, the image content in the content image A, with the image style of a style image B, so as to generate an image with the content of image A and A composite image C in the style of image B, or a fused image C.

Wherein, the style image may refer to a reference image for style transfer processing, and the style in the image may include the texture features of the image and the artistic expression of the image; for example, the style of a famous painting, the artistic expression of the image may include cartoons, comics, Image styles such as oil painting, watercolor, and ink; the content image can refer to the image that needs style transfer, and the content in the image can refer to the semantic information in the image, that is, it can include high-frequency information and low-frequency information in the content image.

In a possible implementation, the first low-rank matrix is obtained based on N frames of sample content images and optical flow information; for example, the first low-rank matrix may refer to the calculation of adjacent image frames in N frames of sample content images The optical flow information between them; the mask information can be obtained according to the optical flow information, wherein the optical flow information is used to represent the running information of the pixels corresponding to the adjacent frame images, and the mask information can be used to represent the The change area in two consecutive frames of images; further, according to the optical flow information and mask information, N frames of sample content images are mapped to a fixed frame of images, and the mapped N frames of sample content images are respectively developed into vectors and arranged in columns Combined into a matrix, the matrix is the first low-rank matrix. Similarly, the second low-rank matrix can be obtained based on N frames of predicted composite images and optical flow information. According to the optical flow information and mask information, N frames of predicted composite images are mapped to a fixed frame of images, and the mapped N The predicted synthetic images are respectively developed into vectors and combined into a matrix by column, then the matrix is the second low-rank matrix, where the optical flow information is used to represent the corresponding pixel points between two adjacent frames of images in the N frame sample content image location difference.

With reference to the first aspect, in some implementations of the first aspect, the image loss function further includes a residual loss function, and the residual loss function is the difference between the composite image based on the first sample and the composite image based on the second sample. The first sample composite image refers to the image obtained by performing image style transfer processing on the N-frame sample content images through the first model, and the second sample composite image refers to the image obtained through the second The model is an image obtained by performing image style transfer processing on the N frames of sample content images, the first model and the second model are image style transfer models pre-trained according to the sample style images, and the second model includes An optical flow module, the first model does not include the optical flow module, and the optical flow module is used to determine the optical flow information.

In the embodiment of this application, the goal of introducing the residual loss function when training the target style transfer model is to enable the neural network model to learn the style transfer model including the optical flow module and the style transfer model without the optical flow module during the training process. The difference of the synthesized image output by the model can improve the stability of the style transfer processed video obtained by the target transfer model.

It should be understood that the difference between the first sample composite image and the second sample composite image may refer to a difference between pixel values corresponding to the first sample composite image and the second sample composite image.

In a possible implementation, the first model and the second model may use the same sample content image and sample style image in the training phase; for example, the first model and the second model may refer to the same model in the training phase; However, in the test phase, the second model also needs to calculate the optical flow information between multiple frames of sample content images; while the first model does not need to calculate the optical flow information between multiple frames of images.

With reference to the first aspect, in some implementations of the first aspect, the first model and the second model are pre-trained teacher models, and the target style transfer model refers to the residual loss function and The knowledge distillation algorithm is the target student model obtained by training the student model to be trained.

In a possible implementation, the target style transfer model may refer to the target student model, and the target student model may be trained according to the pre-trained first teacher model (excluding the optical flow module), the pre-trained second teacher model (including the optical flow module), the pre-trained basic model trains a student model to be trained to obtain the target student model; wherein, the network structure of the student model to be trained, the pre-trained basic model, and the target student model are the same , train the student model to be trained through the above low-rank loss function, residual loss function and perceptual loss function, so as to obtain the target student model.

Wherein, the above-mentioned pre-trained basic model may refer to a style transfer model that does not include an optical flow module in the testing stage obtained through perceptual loss function training in advance; or, the pre-trained style transfer model may refer to Loss function pre-trained style transfer model that does not include the optical flow module in the test phase; perceptual loss function is used to represent the content loss between the synthetic image and the content image and the style loss between the synthetic image and the style image; the optical flow loss function It is used to represent the difference between the corresponding pixels of the composite image of adjacent frames.

In a possible implementation, in the process of training the student model to be trained, the above-mentioned residual loss function is used to make the output migration result (also called a synthetic image) between the student model to be trained and the pre-trained basic model The difference of is constantly approaching the difference of the transfer results output between the second model and the first model.

In the embodiment of the present application, the target style transfer model may refer to the target student model, by adopting the knowledge distillation method of teacher-student model learning, the relationship between the student model to be trained and the style transfer result output by the pre-trained basic model is The difference is constantly approaching the difference between the style transfer results output by the teacher model that includes the optical flow module and the teacher model that does not include the optical flow module. This training method can effectively avoid the ghosting caused by the inconsistent styles of the teacher model and the student model. Phenomenon.

With reference to the first aspect, in some implementations of the first aspect, the residual loss function is obtained according to the following equation,

Wherein, L _res represents the residual loss function; N _T represents the second model; Represents the first model; N _S represents the student model to be trained; /> Represents a pre-trained basic model, which has the same network structure as the student model to be trained; ^xi represents the i-th frame sample content image included in the sample video, and i is a positive integer.

With reference to the first aspect, in some implementations of the first aspect, the image loss function further includes a perceptual loss function, wherein the perceptual loss function includes a content loss and a style loss, and the content loss is used to represent the Image content differences between the N frames of predicted composite images and their corresponding N frames of sample content images, the style loss is used to represent the image style differences between the N frames of predicted composite images and the sample style images.

With reference to the first aspect, in some implementation manners of the first aspect, the image loss function is obtained by weighting the low-rank loss function, the residual loss function, and the perceptual loss function.

With reference to the first aspect, in some implementation manners of the first aspect, the parameters of the target style transfer model are obtained based on the image loss function through multiple iterations of a backpropagation algorithm.

In the second aspect, a method for video style transfer includes: acquiring a video to be processed, wherein the video to be processed includes N frames of content images to be processed, and N is an integer greater than or equal to 2; Perform image style transfer processing on the content image to be processed to obtain N frames of composite images; according to the N frames of composite images, obtain the video after the style transfer processing corresponding to the video to be processed,

Wherein, the parameters of the target style transfer model are determined according to the image loss function that performs style transfer processing on N frames of sample content images according to the target style transfer model, and the image loss function includes a low-rank loss function, and the low-rank The loss function is used to represent the difference between the first low-rank matrix and the second low-rank matrix, the first low-rank matrix is obtained based on the N frames of sample content images and optical flow information, and the second low-rank matrix The matrix is obtained based on N frames of predicted composite images and the optical flow information, the optical flow information is used to represent the position difference of corresponding pixels between two adjacent frames of images in the N frames of sample content images, and the N The frame prediction composite image refers to an image obtained by performing image style transfer processing on the N frames of sample content images according to the sample style images through the target style transfer model.

It should be noted that image style transfer refers to merging the image content of a content image A with the image style of a style image B, so as to generate a composite image C with the image content of A and the image style of B; The style in the image may include information such as texture features of the image; the content in the image may refer to semantic information in the image, that is, it may include high-frequency information, low-frequency information, etc. in the content image.

It should be understood that, for a matrix composed of multiple frames of images, a low-rank matrix can be used to represent regions that appear in all N frames of images and are not motion boundaries. A sparse matrix can be used to represent regions that appear intermittently in N frames of images; for example, a sparse matrix can refer to regions that newly appear or disappear at image borders due to camera movement, or border regions of moving objects.

In the embodiment of the present application, when training the target style transfer model for video style transfer processing, a low-rank loss function is introduced. By introducing a low-rank loss function, it is possible to make the images that appear in adjacent multiple frames of the video to be processed and are not The region of the motion boundary remains the same after the style transfer process, that is, the rank of the region in the style transfer processed video is close to the rank of the region in the video to be processed, thereby improving the stability of the style transfer processed video.

On the other hand, the target style transfer model provided by the embodiment of the present application does not need to calculate the optical flow information between the multiple frames of images included in the video to be processed during the style transfer process of the video to be processed, so the embodiment of the present application The provided target style transfer model can not only improve the stability, but also shorten the processing time of the model's style transfer, and improve the operating efficiency of the target style transfer model.

In a possible implementation manner, the video to be processed may be a video captured by the electronic device through a camera, or the video to be processed may also be a video obtained from inside the electronic device (for example, a video stored in the photo album of the electronic device , or, video captured by the electronic device from the cloud).

It should be understood that the above-mentioned video to be processed may be a video that requires style transfer, and this application does not set any limitation on the source of the video to be processed.

With reference to the second aspect, in some implementations of the second aspect, the image loss function further includes a residual loss function, and the residual loss function is the difference between the composite image based on the first sample and the composite image based on the second sample. The first sample composite image refers to the image obtained by performing image style transfer processing on the N-frame sample content images through the first model, and the second sample composite image refers to the image obtained through the second The model is an image obtained by performing image style transfer processing on the N frames of sample content images, the first model and the second model are image style transfer models pre-trained according to the sample style images, and the second model includes An optical flow module, the first model does not include the optical flow module, and the optical flow module is used to determine the optical flow information.

In the embodiment of this application, the goal of introducing the residual loss function when training the target style transfer model is to enable the neural network model to learn the style transfer model including the optical flow module and the style transfer model without the optical flow module during the training process. The difference of the synthesized image output by the model can improve the stability of the style transfer processed video obtained by the target transfer model.

It should be understood that the difference between the first sample composite image and the second sample composite image may refer to a difference between pixel values corresponding to the first sample composite image and the second sample composite image. In a possible implementation, the first model and the second model may use the same sample content image and sample style image in the training phase; for example, the first model and the second model may refer to the same model in the training phase; However, in the test phase, the second model also needs to calculate the optical flow information between multiple frames of sample content images; while the first model does not need to calculate the optical flow information between multiple frames of images.

With reference to the second aspect, in some implementations of the second aspect, the first model and the second model are pre-trained teacher models, and the target style transfer model refers to the residual loss function and The knowledge distillation algorithm is the target student model obtained by training the student model to be trained.

It should be noted that the network structure of the above student model and the target student model can be the same, that is, the student model can refer to a pre-trained style transfer model that does not need to input optical flow information in the test phase; while the target student model refers to the Based on the student model, the model obtained by further training through the above residual loss function and low rank loss function.

In a possible implementation, the pre-trained student model can be a student model pre-trained by a perceptual loss function, and the perceptual loss function is used to represent the effect of video stylization, that is, it can be used to represent the sample composite image and the sample style Content differences between images and style differences between sample composite images and sample content images.

In a possible implementation manner, the pre-trained student model may be a student model obtained by pre-training through a perceptual loss function, and the optical flow loss function is used to represent the difference between corresponding pixels of the synthesized images of adjacent frames.

With reference to the second aspect, in some implementations of the second aspect, the residual loss function is obtained according to the following equation,

Wherein, L _res represents the residual loss function; N _T represents the second model; Represents the first model; N _S represents the student model to be trained; /> Represents a pre-trained basic model, which has the same network structure as the student model to be trained; ^xi represents the i-th frame sample content image included in the sample video, and i is a positive integer.

In a possible implementation, the target style transfer model may refer to the target student model, and the target student model may be trained according to the pre-trained first teacher model (excluding the optical flow module), the pre-trained second teacher model (including the optical flow module), the pre-trained basic model trains a student model to be trained to obtain the target student model; wherein, the network structure of the student model to be trained, the pre-trained basic model, and the target student model are the same , train the student model to be trained through the above low-rank loss function, residual loss function and perceptual loss function, so as to obtain the target student model.

Wherein, the above-mentioned pre-trained basic model may refer to a style transfer model that does not include an optical flow module in the testing stage obtained through perceptual loss function training in advance; or, the pre-trained style transfer model may refer to Loss function pre-trained style transfer model that does not include the optical flow module in the test phase; perceptual loss function is used to represent the content loss between the synthetic image and the content image and the style loss between the synthetic image and the style image; the optical flow loss function It is used to represent the difference between the corresponding pixels of the composite image of adjacent frames.

In a possible implementation, in the process of training the student model to be trained, the above-mentioned residual loss function is used to make the output migration result (also called a synthetic image) between the student model to be trained and the pre-trained basic model The difference of is constantly approaching the difference between the migration results output by the second model and the first model.

In the embodiment of the present application, the target style transfer model may refer to the target student model, by adopting the knowledge distillation method of teacher-student model learning, the relationship between the student model to be trained and the style transfer result output by the pre-trained basic model is The difference is constantly approaching the difference between the style transfer results output by the teacher model that includes the optical flow module and the teacher model that does not include the optical flow module. This training method can effectively avoid the ghosting caused by the inconsistent styles of the teacher model and the student model. Phenomenon.

With reference to the second aspect, in some implementations of the second aspect, the image loss function further includes a perceptual loss function, wherein the perceptual loss function includes a content loss and a style loss, and the content loss is used to represent the Image content differences between the N frames of predicted composite images and their corresponding N frames of sample content images, the style loss is used to represent the image style differences between the N frames of predicted composite images and the sample style images.

With reference to the second aspect, in some implementation manners of the second aspect, the image loss function is obtained by weighting the low-rank loss function, the residual loss function, and the perceptual loss function.

With reference to the second aspect, in some implementation manners of the second aspect, the parameters of the target style transfer model are obtained through multiple iterations of a backpropagation algorithm based on the image loss function.

In a third aspect, a training device for a style transfer model is provided, including: an acquisition unit configured to acquire training data, wherein the training data includes N frames of sample content images, sample style images, and N frames of synthetic images, the N frames of synthetic images are images obtained by performing image style transfer processing on the N frames of sample content images according to the sample style images, and N is an integer greater than or equal to 2; the processing unit is used to use the neural network model according to the The sample style image performs image style transfer processing on the N frames of sample content images to obtain N frames of predicted composite images; according to the image loss function between the N frames of sample content images and the N frames of predicted composite images, determine the The parameters of the neural network model, wherein the image loss function includes a low-rank loss function, the low-rank loss function is used to represent the difference between the first low-rank matrix and the second low-rank matrix, and the first low-rank The matrix is obtained based on the N frames of sample content images and optical flow information, the second low-rank matrix is obtained based on the N frames of predicted composite images and the optical flow information, and the optical flow information is used to represent Position differences of corresponding pixel points between two adjacent frames of images in the N frames of sample content images.

In a possible implementation manner, the above-mentioned training device includes a functional unit/module for executing the first aspect and the method in any one of the implementation manners of the first aspect.

It should be understood that the extensions, limitations, explanations and descriptions of relevant content in the first aspect above are also applicable to the same content in the third aspect.

In a fourth aspect, an apparatus for video style transfer is provided, including: an acquisition unit configured to acquire a video to be processed, wherein the video to be processed includes N frames of content images to be processed, and N is an integer greater than or equal to 2; A processing unit, configured to perform image style transfer processing on the N frames of content images to be processed according to the target style transfer model to obtain N frames of composite images; according to the N frames of composite images, obtain the corresponding style transfer processing of the video to be processed after the video,

Wherein, the parameters of the target style transfer model are determined according to the image loss function that performs style transfer processing on N frames of sample content images according to the target style transfer model, and the image loss function includes a low-rank loss function, and the low-rank The loss function is used to represent the difference between the first low-rank matrix and the second low-rank matrix, the first low-rank matrix is obtained based on the N frames of sample content images and optical flow information, and the second low-rank matrix The matrix is obtained based on N frames of predicted composite images and the optical flow information, the optical flow information is used to represent the position difference of corresponding pixels between two adjacent frames of images in the N frames of sample content images, and the N The frame prediction composite image refers to an image obtained by performing image style transfer processing on the N frames of sample content images according to the sample style images through the target style transfer model.

In a possible implementation manner, the above apparatus includes a functional unit/module further configured to execute the second aspect and the method in any one implementation manner of the second aspect.

It should be understood that the extensions, limitations, explanations and descriptions of related content in the second aspect above are also applicable to the same content in the fourth aspect.

In a fifth aspect, a training device for a style transfer model is provided, including: a memory for storing a program; a processor for executing the program stored in the memory, and when the program stored in the memory is executed, the processor is used for Execution: Acquire training data, wherein the training data includes N frames of sample content images, sample style images, and N frames of composite images, and the N frames of composite images are performed on the N frames of sample content images according to the sample style images For the image obtained after the image style transfer processing, N is an integer greater than or equal to 2; the image style transfer processing is performed on the N frames of sample content images according to the sample style image through the neural network model, and N frames of predicted composite images are obtained; An image loss function between the N frames of sample content images and the N frames of predicted composite images to determine the parameters of the neural network model, wherein the image loss function includes a low-rank loss function, and the low-rank loss function Used to represent the difference between the first low-rank matrix and the second low-rank matrix, the first low-rank matrix is obtained based on the N-frame sample content images and optical flow information, and the second low-rank matrix is It is obtained based on the N frames of predicted composite images and the optical flow information, where the optical flow information is used to represent the position difference of corresponding pixels between two adjacent frames of images in the N frames of sample content images.

In a possible implementation manner, the training device includes a processor further configured to execute the first aspect and the method in any one implementation manner of the first aspect.

It should be understood that the extensions, limitations, explanations and descriptions of relevant content in the first aspect above are also applicable to the same content in the fifth aspect.

In the sixth aspect, there is provided a device for video style migration, including: a memory for storing programs; a processor for executing the programs stored in the memory, and when the programs stored in the memory are executed, the processor is used for executing : Acquiring the video to be processed, wherein the video to be processed includes N frames of content images to be processed, and N is an integer greater than or equal to 2; performing image style transfer processing on the N frames of content images to be processed according to the target style transfer model, Obtain N frames of composite images; according to the N frames of composite images, obtain the style transfer-processed video corresponding to the video to be processed, wherein the parameters of the target style transfer model are based on the target style transfer model for N frames The sample content image is determined by an image loss function for style transfer processing, the image loss function includes a low-rank loss function, and the low-rank loss function is used to represent the difference between the first low-rank matrix and the second low-rank matrix, The first low-rank matrix is obtained based on the N frames of sample content images and optical flow information, the second low-rank matrix is obtained based on the N frames of predicted composite images and the optical flow information, and the The optical flow information is used to represent the position difference of corresponding pixels between two adjacent frames of images in the N frames of sample content images, and the N frames of predicted composite images refer to the pairing of the target style images according to the sample style images by the target style transfer model. An image obtained after performing image style transfer processing on the N frames of sample content images.

In a possible implementation manner, the processor included in the above device is further configured to execute the training method in any two implementation manners in the second aspect.

It should be understood that the extensions, limitations, explanations and descriptions of related content in the second aspect above are also applicable to the same content in the sixth aspect.

In a seventh aspect, a computer-readable medium is provided, the computer-readable medium stores program code for execution by a device, and the program code includes a program code for executing the above-mentioned first aspect to the second aspect and the first aspect to the second aspect A method in any of the implementations.

In an eighth aspect, a computer program product containing instructions is provided, and when the computer program product is run on a computer, it causes the computer to execute any one of the above-mentioned first to second aspects and the first to second aspects method in the implementation.

A ninth aspect provides a chip, the chip includes a processor and a data interface, the processor reads instructions stored on the memory through the data interface, and executes the above-mentioned first to second aspects and the first aspect to the method in any one of the implementation manners in the second aspect.

Optionally, as an implementation manner, the chip may further include a memory, the memory stores instructions, the processor is configured to execute the instructions stored in the memory, and when the instructions are executed, the The processor is configured to execute the method in any one implementation manner of the first aspect to the second aspect and any one of the first aspect to the second aspect.

Description of drawings

Fig. 1 is a schematic diagram of an artificial intelligence subject framework provided by an embodiment of the present application;

FIG. 2 is a schematic diagram of an application scenario provided by an embodiment of the present application;

Fig. 3 is a system architecture provided by the embodiment of the present application;

FIG. 4 is a schematic structural diagram of a convolutional neural network provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of a chip hardware structure provided by an embodiment of the present application;

FIG. 6 shows a system architecture provided by an embodiment of the present application;

Fig. 7 is a schematic flow chart of the training method of the style transfer model provided by the embodiment of the present application;

Fig. 8 is a schematic diagram of the training process of the style transfer model provided by the embodiment of the present application;

FIG. 9 is a schematic flowchart of a method for video style transfer provided by an embodiment of the present application;

Fig. 10 is a schematic diagram of the training phase and the testing phase provided by the embodiment of the present application;

FIG. 11 is a schematic block diagram of an apparatus for video style migration provided by an embodiment of the present application;

FIG. 12 is a schematic block diagram of a training device for a style transfer model provided by an embodiment of the present application;

FIG. 13 is a schematic block diagram of an apparatus for video style migration provided by an embodiment of the present application;

Fig. 14 is a schematic block diagram of a training device for a style transfer model provided by an embodiment of the present application.

Detailed ways

The following will describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application; obviously, the described embodiments are only a part of the embodiments of the application, not all the embodiments. According to the embodiments in the present application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present application.

Figure 1 shows a schematic diagram of an artificial intelligence main framework, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence field requirements.

The artificial intelligence theme framework 100 will be described in detail below from the two dimensions of "intelligent information chain" (horizontal axis) and "information technology (information technology, IT) value chain" (vertical axis).

"Intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensed process of "data-information-knowledge-wisdom".

"IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of artificial intelligence, information (provided and processed by technology) to the systematic industrial ecological process.

(1) Infrastructure 110

The infrastructure can provide computing power support for the artificial intelligence system, realize communication with the outside world, and realize support through the basic platform.

The infrastructure can communicate with the outside through sensors, and the computing power of the infrastructure can be provided by smart chips.

The smart chip here can be a central processing unit (central processing unit, CPU), a neural network processor (neural-network processing unit, NPU), a graphics processing unit (graphics processing unit, GPU), an application-specific integrated circuit (application specific integrated circuit, ASIC) and field programmable gate array (field programmable gate array, FPGA) and other hardware acceleration chips.

The basic platform of infrastructure can include related platform guarantees and supports such as distributed computing framework and network, and can include cloud storage and computing, interconnection and interworking network, etc.

For example, for infrastructure, data can be obtained through sensors and external communication, and then these data can be provided to smart chips in the distributed computing system provided by the basic platform for calculation.

(2) Data 120

Data from the upper layer of the infrastructure is used to represent data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional equipment, including business data of existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.

(3) Data processing 130

The above data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other processing methods.

Among them, machine learning and deep learning can symbolize and formalize intelligent information modeling, extraction, preprocessing, training, etc. of data.

Reasoning refers to the process of simulating human intelligent reasoning in a computer or intelligent system, and using formalized information to carry out machine thinking and solve problems according to reasoning control strategies. The typical functions are search and matching.

Decision-making refers to the process of decision-making after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

(4) General ability 140

After the data is processed by the data mentioned above, some general capabilities can be formed according to the results of data processing, such as algorithms or a general system, such as translation, text analysis, computer vision processing, speech recognition, image processing identification, etc.

(5) Smart products and industry applications 150

Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. It is the packaging of the overall solution of artificial intelligence, which commercializes intelligent information decision-making and realizes landing applications. Its application fields mainly include: intelligent manufacturing, intelligent transportation, Smart home, smart medical care, smart security, automatic driving, safe city, smart terminals, etc.

FIG. 2 is a schematic diagram of an application scenario provided by an embodiment of the present application.

As shown in Figure 2, the video style transfer method of the embodiment of the present application can be applied to the smart terminal; In the target style transfer model provided by the embodiment of the application, the video after the style transfer process is obtained; by adopting the target style transfer model provided by the embodiment of the present application, the stability of the video after the style transfer process can be ensured, that is, the video after the style transfer process can be ensured. Get the fluency of the video.

In an example, the video style transfer method provided in the embodiment of the present application can be applied in an offline scenario.

For example, by acquiring the video to be processed, the video to be processed is input into the target style transfer model, so as to obtain the video processed by the style transfer, that is, the output stable stylized video.

In an example, the video style transfer method provided in the embodiment of the present application can be applied in an online scene.

For example, obtain the real-time recorded video of the smart terminal, and input the real-time recorded video into the target style transfer model, so as to obtain the real-time output style-transferred video; for example, it can be used in scenarios such as real-time display of booths

For example, when making an online video call through a smart terminal, the user video captured by the camera in real time can be input into the target style transfer model, so as to obtain the output video after style transfer processing. For example, a stable stylized video can be shown to others in real time to enhance the fun.

Wherein, the above-mentioned target style transfer model is a pre-trained model obtained by training through the training method of the style transfer model provided in the embodiment of the present application.

Exemplarily, the above-mentioned smart terminal may be mobile or fixed; for example, the smart terminal may be a mobile phone with an image processing function, a tablet personal computer (tablet personal computer, TPC), a media player, a smart TV, a notebook computer ( laptop computer (LC), personal digital assistant (personal digital assistant, PDA), personal computer (personal computer, PC), camera, video camera, smart watch, wearable device (wearable device, WD) or self-driving vehicles, etc. This embodiment of the present application does not limit it.

It should be understood that the foregoing is an illustration of an application scenario, and does not limit the application scenario of the present application in any way.

Since the embodiment of the present application involves the application of a large number of neural networks, for ease of understanding, the following first introduces the related terms and concepts of the neural network that may be involved in the embodiment of the present application.

1. Neural network

A neural network can be composed of neural units, and a neural unit can refer to an operation unit that takes x _s and an intercept 1 as input, and the output of the operation unit can be:

Among them, s=1, 2, ... n, n is a natural number greater than 1, W _s is the weight of x _s , and b is the bias of the neuron unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple above-mentioned single neural units, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected with the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

2. Deep neural network

A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple hidden layers. DNN is divided according to the position of different layers, and the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in the middle are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

Although DNN looks complicated, it is actually not complicated in terms of the work of each layer. In simple terms, it is the following linear relationship expression: where, /> is the input vector, /> is the output vector, /> Is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector /> After such a simple operation to get the output vector /> Due to the large number of DNN layers, the coefficient W and the offset vector /> The number is also higher. The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the fourth neuron of the second layer to the second neuron of the third layer is defined as /> . The superscript 3 represents the layer number of the coefficient W, and the subscript corresponds to the output third layer index 2 and the input second layer index 4.

In summary, the coefficient from the kth neuron of the L-1 layer to the jth neuron of the L layer is defined as

It should be noted that the input layer has no W parameter. In deep neural networks, more hidden layers make the network more capable of describing complex situations in the real world. Theoretically speaking, a model with more parameters has a higher complexity and a greater "capacity", which means that it can complete more complex learning tasks. Training the deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by the vector W of many layers).

3. Convolutional neural network

Convolutional neural network (CNN) is a deep neural network with a convolutional structure. The convolutional neural network contains a feature extractor composed of a convolutional layer and a subsampling layer, which can be regarded as a filter. The convolutional layer refers to the neuron layer that performs convolution processing on the input signal in the convolutional neural network. In the convolutional layer of a convolutional neural network, a neuron can only be connected to some adjacent neurons. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units of the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as a way to extract image information that is independent of location. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

4. Loss function

In the process of training the deep neural network, because it is hoped that the output of the deep neural network is as close as possible to the value you really want to predict, you can compare the predicted value of the current network with the target value you really want, and then according to the difference between the two to update the weight vector of each layer of the neural network (of course, there is usually an initialization process before the first update, that is, to pre-configure parameters for each layer in the deep neural network); for example, if the predicted value of the network If it is high, adjust the weight vector to make it predict lower, and keep adjusting until the deep neural network can predict the real desired target value or a value very close to the real desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function (loss function) or objective function (objective function), which are used to measure the difference between the predicted value and the target value important equation. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Then the training of the deep neural network becomes a process of reducing the loss as much as possible.

5. Back propagation algorithm

The neural network can use the error back propagation (back propagation, BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller.

Specifically, passing the input signal forward until the output will generate an error loss, and updating the parameters in the initial neural network model by backpropagating the error loss information, so that the error loss converges. The backpropagation algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

6. Image style transfer

Image style transfer refers to fusing the image content of a content image A with the image style of a style image B to generate a composite image C with the content of image A and the style of image B.

Exemplarily, image style transfer is performed on the content image 1 according to the style image 1 to obtain a composite image 1, wherein the composite image 1 includes the content in the content image 1 and the style in the style image 1; similarly, according to the style image 2 Perform image style transfer on the content image 1 to obtain a composite image 2, wherein the composite image 2 includes the content in the content image 1 and the style in the style image 2.

Among them, the style image can refer to a reference image for style transfer, and the style in the image can include the texture features of the image and the artistic expression of the image; for example, the style of a famous painting, the artistic expression of the image can include cartoons, comics, oil paintings, etc. , watercolor, ink and other image styles; the content image can refer to the image that needs style transfer, and the content in the image can refer to the semantic information in the image, that is, it can include high-frequency information and low-frequency information in the content image.

7. Optical flow information

Optical flow (optical flow or optic flow) is used to represent the instantaneous velocity of the pixel motion of a space moving object on the observation imaging plane, which is found by using the change of pixels in the image sequence in the time domain and the correlation between adjacent frames. A method to calculate the motion information of objects between adjacent frames by the corresponding relationship between the previous frame and the current frame.

8. Knowledge Distillation

Knowledge distillation refers to the key technology to make the deep learning model miniaturized and meet the deployment requirements of terminal equipment. Compared with compression techniques such as quantization and sparsification, it does not require specific hardware support to achieve the purpose of compressing the model. The knowledge distillation technology adopts the strategy of teacher-student model learning. Among them, the teacher model can refer to large model parameters, which generally cannot meet the deployment requirements; while the student model has few parameters and can be directly deployed. By designing an effective knowledge distillation algorithm, the student model can learn to imitate the behavior of the teacher model, and carry out effective knowledge transfer, so that the student model can finally perform the same processing ability as the teacher model.

First, the system architecture of the video style transfer method and the style transfer model training method provided by the embodiment of the present application is introduced.

FIG. 3 shows a system architecture 200 provided by an embodiment of the present application.

As shown in the system architecture 200 in FIG. 3 , the data collection device 260 is used to collect training data. Regarding the method for training a style transfer model in the embodiment of the present application, the target style transfer model can be further trained with training data, that is, the training data collected by the data collection device 260 .

Exemplarily, in the embodiment of the present application, the training data for training the target style transfer model can be composed of N frames of sample content images, sample style images, and N frames of sample composite images, wherein the composite images of N frames of samples are based on the pairing of N frames of sample style images The sample content image is an image obtained after image style transfer processing, and N is an integer greater than or equal to 2.

After collecting the training data, the data acquisition device 260 stores the training data in the database 230, and the training device 220 obtains the target model/rule 201 according to the training data maintained in the database 230 (that is, the target style transfer model in the embodiment of the present application) ). The training device 220 inputs the training data into the target style transfer model until the difference between the output data of the training target style transfer model and the sample data satisfies a preset condition (for example, the difference between the predicted data and the sample data is less than a certain threshold, or the predicted data The difference with the sample data remains unchanged or no longer decreases), so as to complete the training of the target model/rule 201 .

Wherein, the output data may refer to N frames of predicted composite images output by the target style transfer model; the sample data may refer to N frames of sample composite images.

In the embodiment provided in the present application, the target model/rule 201 is obtained by training a target style transfer model, and the target style transfer model can be used to perform style transfer processing on the video to be processed. It should be noted that, in practical applications, the training data maintained in the database 230 may not all be collected by the data collection device 260, but may also be received from other devices.

It should be noted that the training device 220 does not necessarily perform the training of the target model/rule 201 completely based on the training data maintained by the database 230, and it is also possible to obtain training data from the cloud or other places for model training. Example limitations.

It should also be noted that at least part of the training data maintained in the database 230 may also be used to execute the process of the processing to be processed by the device 210 .

The target model/rule 201 trained according to the training device 220 can be applied to different systems or devices, such as the execution device 210 shown in FIG. Laptops, AR/VR, vehicle terminals, etc., can also be servers or clouds.

In FIG. 3 , the execution device 210 is configured with an input/output (input/output, I/O) interface 212 for data interaction with external devices, and the user can input data to the I/O interface 212 through the client device 240, the described In this embodiment of the application, the input data may include: a video to be processed input by the client device.

Wherein, the preprocessing module 213 and the preprocessing module 214 are used to perform preprocessing according to the input data received by the I/O interface 212 (such as video to be processed). In the embodiment of the present application, there may be no preprocessing module 213 and preprocessing module 214 (or only one of the preprocessing modules), and the calculation module 211 may be used directly to process the input data.

When the execution device 210 preprocesses the input data, or in the execution device 210 computing module 211 performs calculations and other related processing, the execution device 210 can call the data, codes, etc. in the data storage system 250 for the corresponding processing, and may also store the data and instructions obtained through the corresponding processing into the data storage system 250 .

Finally, the I/O interface 212 returns the processing results, such as the video to be processed as described above, that is, the obtained video after the style transfer process, to the client device 240, so as to provide it to the user.

It is worth noting that the training device 220 can generate corresponding target models/rules 201 according to different training data for different goals or different tasks, and the corresponding target models/rules 201 can be used to achieve the above goals or complete above tasks, thereby providing the desired result to the user.

In the situation shown in FIG. 3 , in one case, the user can manually specify the input data, and the manual specification can be operated through the interface provided by the I/O interface 212 .

In another case, the client device 240 can automatically send the input data to the I/O interface 212 . If the client device 240 is required to automatically send the input data to obtain the user's authorization, the user can set the corresponding authority in the client device 240 . The user can view the results output by the execution device 210 on the client device 240, and the specific presentation form may be specific ways such as display, sound, and action. The client device 240 can also be used as a data collection terminal, collecting input data from the input I/O interface 212 and output results from the output I/O interface 212 as new sample data, and storing them in the database 230 . Of course, the client device 240 may not be used for collection, but the I/O interface 212 directly uses the input data input to the I/O interface 212 as shown in the figure and the output result of the output I/O interface 212 as a new sample. The data is stored in database 230 .

It should be noted that FIG. 3 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship among devices, devices, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG. 3 , the data storage system 250 is an external memory relative to the execution device 210 , and in other cases, the data storage system 250 may also be placed in the execution device 210 .

As shown in FIG. 3 , the target model/rule 201 is trained according to the training device 220. The target model/rule 201 may be the target style transfer model in the embodiment of the present application. Specifically, the target style transfer model provided in the embodiment of the present application It can be a deep neural network, a convolutional neural network, or a deep convolutional neural network, etc.

The following is a detailed introduction to the structure of the convolutional neural network in combination with Figure 4. As mentioned in the introduction to the basic concepts above, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. Multiple levels of learning are performed at the abstraction level. As a deep learning architecture, a convolutional neural network is a feed-forward artificial neural network in which individual neurons can respond to images fed into it.

The network structure of the style transfer model in the embodiment of the present application may be shown in FIG. 4 . In FIG. 4 , a convolutional neural network 300 may include an input layer 310 , a convolutional layer/pooling layer 320 (where the pooling layer is optional), and a neural network layer 330 . Wherein, the input layer 310 can obtain the image to be processed, and pass the obtained image to be processed by the convolution layer/pooling layer 320 and the subsequent neural network layer 330 to obtain the processing result of the image. The following is a detailed introduction to the internal layer structure of CNN300 in FIG. 4 .

Convolutional/pooling layer 320:

As shown in Figure 4, the convolutional layer/pooling layer 320 may include layers 321-326 as examples; for example: in one implementation, layer 321 is a convolutional layer, layer 322 is a pooling layer, and layer 323 is a volume Layer, 324 is a pooling layer, 325 is a convolutional layer, and 326 is a pooling layer; in another implementation, 321, 322 are convolutional layers, 323 are pooling layers, 324, 325 are convolutional layers Layer 326 is a pooling layer, that is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or can be used as the input of another convolutional layer to continue the convolution operation.

The following will take the convolutional layer 321 as an example to introduce the inner working principle of one convolutional layer.

The convolution layer 321 can include many convolution operators, which are also called kernels, and their role in image processing is equivalent to a filter that extracts specific information from the input image matrix. The convolution operator is essentially It can be a weight matrix. This weight matrix is usually pre-defined. During the convolution operation on the image, the weight matrix is usually one pixel by one pixel (or two pixels by two pixels) along the horizontal direction on the input image. etc., depending on the value of the stride), so as to complete the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image. During the convolution operation, the weight matrix will be extended to The entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolutional output with a single depth dimension, but in most cases instead of using a single weight matrix, multiple weight matrices of the same size (row×column) are applied, That is, multiple matrices of the same shape. The output of each weight matrix is stacked to form the depth dimension of the convolution image, where the dimension can be understood as determined by the "multiple" mentioned above.

Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract image edge information, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to extract unwanted features in the image. Noise blurring etc. The multiple weight matrices have the same size (row×column), and the convolutional feature maps extracted by the multiple weight matrices of the same size are also of the same size, and then the extracted multiple convolutional feature maps of the same size are combined to form The output of the convolution operation.

The weight values in these weight matrices need to be obtained through a lot of training in practical applications, and each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network 300 can make correct predictions .

When the convolutional neural network 300 has multiple convolutional layers, the initial convolutional layer (for example, 321) often extracts more general features, which can also be referred to as low-level features; as the convolutional neural network With the deepening of 300 depth, the features extracted by the later convolutional layers (such as 326) become more and more complex, for example, features such as high-level semantics, and features with higher semantics are more suitable for the problem to be solved.

Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolutional layer. In the layers 321-326 as shown in 320 in Figure 4, it can be a convolutional layer followed by a layer The pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers. In the image processing process, the purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling an input image to obtain an image of a smaller size. The average pooling operator can calculate the pixel values in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value within a specific range as the result of maximum pooling. Also, just like the size of the weight matrix used in the convolutional layer should be related to the size of the image, the operators in the pooling layer should also be related to the size of the image. The size of the image output after being processed by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average or maximum value of the corresponding sub-region of the image input to the pooling layer.

Neural Network Layer 330:

After being processed by the convolutional layer/pooling layer 320, the convolutional neural network 300 is not enough to output the required output information. Because as mentioned earlier, the convolutional layer/pooling layer 320 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (required class information or other relevant information), the convolutional neural network 300 needs to use the neural network layer 330 to generate one or a group of outputs with the required number of classes. Therefore, the neural network layer 330 may include multiple hidden layers (331, 332 to 33n as shown in FIG. 4 ) and an output layer 340, and the parameters contained in the multi-layer hidden layers may be based on specific task types. Relevant training data are pre-trained. For example, the task type can include image recognition, image classification, image detection, and image super-resolution reconstruction.

After the multi-layer hidden layer in the neural network layer 330, that is, the last layer of the entire convolutional neural network 300 is the output layer 340, which has a loss function similar to the classification cross-entropy, and is specifically used to calculate the prediction error, Once the forward propagation of the entire convolutional neural network 300 (as shown in Figure 4, the propagation from 310 to 340 is forward propagation), the reverse propagation (as shown in Figure 4, the propagation from 340 to 310 is backward propagation) will Start to update the weight values and biases of the aforementioned layers to reduce the loss of the convolutional neural network 300 and the error between the output of the convolutional neural network 300 through the output layer and the ideal result.

It should be noted that the convolutional neural network shown in FIG. 4 is only an example of the structure of the target style transfer model of the embodiment of the present application. In a specific application, the video style transfer method of the embodiment of the present application adopts Style transfer models can also exist in the form of other network models.

FIG. 5 is a hardware structure of a chip provided by an embodiment of the present application, and the chip includes a neural network processor 400 (neural-network processing unit, NPU). The chip can be set in the execution device 210 shown in FIG. 3 to complete the computing work of the computing module 211 . The chip can also be set in the training device 220 shown in FIG. 3 to complete the training work of the training device 220 and output the target model/rule 201 . The algorithms of each layer in the convolutional neural network shown in FIG. 4 can be implemented in the chip shown in FIG. 5 .

The NPU 400 is mounted on a main central processing unit (central processing unit, CPU) as a coprocessor, and the main CPU assigns tasks. The core part of the NPU 400 is the operation circuit 403, and the controller 404 controls the operation circuit 403 to extract data in the memory (weight memory or input memory) and perform operations.

In some implementations, the operation circuit 403 includes multiple processing units (process engine, PE). In some implementations, arithmetic circuit 403 is a two-dimensional systolic array. The arithmetic circuit 403 may also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 403 is a general-purpose matrix processor.

For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 403 fetches the data corresponding to the matrix B from the weight memory 402 and caches it in each PE in the operation circuit 403 . The operation circuit 403 takes the data of matrix A from the input memory 401 and performs matrix operation with matrix B, and the obtained partial or final results of the matrix are stored in the accumulator 408 (accumulator).

The vector calculation unit 407 can perform further processing on the output of the operation circuit 403, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison and so on. For example, the vector calculation unit 407 can be used for network calculations of non-convolution/non-FC layers in neural networks, such as pooling (pooling), batch normalization (batch normalization), local response normalization (local response normalization), etc. .

In some implementations, the vector computation unit can 407 store the vector of the processed output to the unified memory 406 . For example, the vector calculation unit 407 may apply a non-linear function to the output of the operation circuit 403, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 407 generates normalized values, merged values, or both.

In some implementations, the vector of processed outputs can be used as an activation input to operational circuitry 403, eg, for use in subsequent layers in a neural network.

The unified memory 406 is used to store input data and output data.

The weight data directly stores the input data in the external memory into the input memory 401 and/or the unified memory 406 through the storage unit access controller 405 (direct memory access controller, DMAC), and stores the weight data in the external memory into the weight memory 402, And store the data in the unified memory 406 into the external memory.

A bus interface unit 410 (bus interface unit, BIU) is configured to realize the interaction between the main CPU, DMAC and instruction fetch memory 409 through the bus.

An instruction fetch buffer 409 (instruction fetch buffer) connected to the controller 404 is used to store instructions used by the controller 404 .

The controller 404 is configured to call the instruction cached in the instruction fetch memory 409 to control the operation process of the operation accelerator.

Generally, the unified memory 406, the input memory 401, the weight memory 402, and the instruction fetch memory 409 are all on-chip (On-Chip) memories, and the external memory is a memory outside the NPU, and the external memory can be a double data rate synchronous dynamic random Memory (double data rate synchronous dynamic random access memory, DDR SDRAM), high bandwidth memory (high bandwidth memory, HBM) or other readable and writable memory.

Wherein, the operations of each layer in the convolutional neural network shown in FIG. 4 can be performed by the operation circuit 403 or the vector calculation unit 407 .

The executing device 210 in FIG. 3 described above can execute each step of the video style transfer method of the embodiment of the present application. The CNN model shown in FIG. 4 and the chip shown in FIG. 5 can also be used to execute the embodiment of the present application The various steps of the video style transfer method.

FIG. 6 shows a system architecture 500 provided by the embodiment of the present application. The system architecture includes a local device 520, a local device 530, an execution device 510, and a data storage system 550, wherein the local device 520 and the local device 530 are connected to the execution device 510 through a communication network.

Execution device 510 may be implemented by one or more servers. Optionally, the execution device 510 may be used in cooperation with other computing devices, such as data storage, routers, load balancers and other devices. Execution device 510 may be arranged on one physical site, or distributed on multiple physical sites. The execution device 510 may use the data in the data storage system 550, or call the program code in the data storage system 550 to implement the method for video style transfer in the embodiment of the present application.

It should be noted that the execution device 510 may also be called a cloud device, and in this case, the execution device 510 may be deployed in the cloud.

Specifically, the execution device 510 may perform the following process:

Obtain training data, wherein the training data includes N frames of sample content images, sample style images, and N frames of composite images, and the N frames of composite images are image styled on the N frames of sample content images according to the sample style images. For the image obtained after migration processing, N is an integer greater than or equal to 2; performing image style migration processing on the N frames of sample content images according to the sample style image through a neural network model to obtain N frames of predicted composite images; according to the An image loss function between the N frames of sample content images and the N frames of predicted composite images to determine the parameters of the neural network model, wherein the image loss function includes a low-rank loss function, and the low-rank loss function is used for Indicates the difference between the first low-rank matrix and the second low-rank matrix, the first low-rank matrix is obtained based on the N frames of sample content images and optical flow information, and the second low-rank matrix is obtained based on the The N frames of predicted composite images and the optical flow information are used to represent the position differences of corresponding pixels between two adjacent frames of images in the N frames of sample content images.

Alternatively, the execution device 510 may perform the following process:

Obtain a video to be processed, wherein the video to be processed includes N frames of content images to be processed, and N is an integer greater than or equal to 2; perform image style transfer processing on the N frames of content images to be processed according to the target style transfer model, and obtain N-frame composite image; according to the N-frame composite image, obtain the style-transfer-processed video corresponding to the video to be processed, wherein the parameters of the target style-transfer model are N-frame samples according to the target style-transfer model The content image is determined by an image loss function for style transfer processing, the image loss function includes a low-rank loss function, and the low-rank loss function is used to represent the difference between the first low-rank matrix and the second low-rank matrix, so The first low-rank matrix is obtained based on the N frames of sample content images and optical flow information, the second low-rank matrix is obtained based on N frames of predicted composite images and the optical flow information, and the optical flow information It is used to indicate the position difference of corresponding pixels between two adjacent frames of images in the sample content image of the N frames, and the predicted composite image of the N frames means that the target style transfer model is used to transform the N frames according to the sample style image The sample content image is the image obtained after image style transfer processing.

Users can operate respective user devices (for example, local device 520 and local device 530 ) to interact with execution device 510 . Each local device can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

Each user's local device can interact with the execution device 510 through any communication mechanism/communication standard communication network, and the communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

In one implementation, the local device 520 and the local device 530 can obtain relevant parameters of the target style transfer model from the execution device 510, deploy the target style transfer model on the local device 520 and the local device 530, and use the target style transfer model to The model performs video style transfer processing, etc.

In another implementation, the target style transfer model can be directly deployed on the execution device 510, and the execution device 510 obtains the video to be processed from the local device 520 and the local device 530, and performs style transfer processing on the video to be processed according to the target style transfer model, etc. .

At present, there are mainly two types of video style transfer models that use optical flow information to stabilize stylized videos. The first one uses optical flow in the training process of the style transfer model, but does not introduce optical flow information in the testing phase; the second One is to integrate the optical flow module into the structure of the style transfer model; however, the first method can guarantee the operation efficiency of the style transfer model in the test phase, but the stability of the video after the style transfer process is poor; the second method The method can ensure the stability of the output video after style transfer processing, but due to the introduction of the optical flow module, it is necessary to calculate the optical flow information between the image frames included in the video and the image frames during the test phase, so the style transfer model cannot be used in the test phase. Operational efficiency.

In view of this, the embodiment of the present application proposes a method for training a style transfer model and a method for video style transfer. In the process of training a style transfer model for videos, a low-rank loss function is introduced. Through the learning of low-rank information , can synchronize the stability of the style-transferred video and the original video, thereby improving the stability of the style-transferred video obtained by the target transfer model; in addition, the style transfer model proposed in the embodiment of the present application is the video to be processed in the testing phase In the process of style transfer processing, it is not necessary to calculate the optical flow information between multiple frames of images included in the video, so the target transfer style transfer model provided by the embodiment of the present application can shorten the style transfer processing of the model while improving stability. Time to improve the operating efficiency of the target style transfer model.

FIG. 7 shows a schematic flow chart of a method 600 for training a style transfer model provided by an embodiment of the present application. The method can be executed by a device capable of image style transfer; for example, the training method can be performed by the execution device in FIG. 6 510, or may also be executed by the local device 520. Wherein, the training method 600 includes S610 to S630, and these steps will be described in detail below respectively.

S610. Obtain training data.

Wherein, the training data may include N frames of sample content images, sample style images, and N frames of composite images. The N frames of composite images are images obtained after performing image style transfer processing on N frames of sample content images according to the sample style images, and N is greater than or An integer equal to 2.

Exemplarily, the N frames of sample content images may refer to N frames of continuous sample content images included in the sample video; the N frames of composite images may refer to the samples included in the video obtained after performing style transfer processing on the sample video according to the sample style image N consecutive composite images.

It should be understood that for the style transfer processing of a single-frame image, that is, image style transfer, only the content in the content image and the style in the style image need to be considered; Style transfer should not only consider the stylized effect of the image, but also consider the stability between multiple frames of images; that is, it is necessary to ensure the smoothness of the video after the style transfer process and avoid noise such as splash screens and artifacts.

It should be noted that the above N frames of sample content images refer to N frames of adjacent images in the video; N frames of composite images refer to images corresponding to N frames of sample content images.

S620. Perform image style transfer processing on N frames of sample content images according to the sample style images through the neural network model, to obtain N frames of predicted composite images.

Exemplarily, N frames of sample content images and N frames of sample style images included in the sample video may be input to the neural network model.

For example, N frames of sample content images may be input to the neural network model frame by frame, and the neural network model may perform image style transfer processing on a frame of sample content images according to the sample style image, thereby obtaining the frame of sample content One frame of predicted composite images corresponding to the images; after performing the above process N times, N frames of predicted composite images corresponding to N frames of sample content images can be obtained.

For example, multiple frames of N frames of sample content images may be input to the neural network model once, and the neural network model may perform image style transfer processing on the multiple sample content images according to the sample style image.

S630. Determine the parameters of the neural network model according to the image loss function between the N frames of sample content images and the N frames of predicted composite images.

Wherein, the image loss function includes a low-rank loss function, and the low-rank loss function is used to represent the difference between the first low-rank matrix and the second low-rank matrix, and the first low-rank matrix is based on N frame samples The content image and optical flow information are obtained, the second low-rank matrix is obtained based on N frames of predicted composite images and the optical flow information, and the optical flow information is used to represent the adjacent The position difference of corresponding pixel points between two frames of images.

It should be noted that, for a matrix composed of multiple frames of images, a low-rank matrix can be used to represent regions that appear in all N frames of images and are not motion boundaries. A sparse matrix can be used to represent regions that appear intermittently in N frames of images; for example, a sparse matrix can refer to regions that newly appear or disappear at image borders due to camera movement, or border regions of moving objects.

For example, N frames of sample content images can refer to images in which the user is moving, and then the low-rank matrix formed by N frames of sample content images can be used to represent areas that appear in N frames of sample content images and are not motion boundaries; for example, N frames The low-rank matrix formed by the sample content can be used to represent the background area where no motion occurs, or the user appears in all the N frames of sample content images and is not a motion boundary area.

Exemplarily, it is assumed that the video includes 5 consecutive frames of images, that is, 5 consecutive frames of sample content images; 5 frames of style transfer results obtained from 5 frames of sample content images for style transfer processing are 5 frames of sample composite images; the low-rank matrix can be Calculate as follows:

Step 1. Calculate optical flow information between image frames and image frames according to 5 frames of images in the video, that is, 5 frames of sample content images;

Step 2. Calculate the mask information according to the optical flow information, wherein the mask information can be used to represent the change area in two consecutive frames of images obtained according to the optical flow information;

Step 3. Calculate the low-rank part after aligning the 1st, 2nd, 4th, and 5th frame images with the 3rd frame image according to the optical flow information and mask information, and set the sparse part to 0;

Step 4. Expand the aligned 5 frames of images into vectors and combine them into a matrix by column, then the matrix can be a low-rank matrix.

It should be understood that when calculating the low-rank loss function, the goal is to approximate the rank of the low-rank part of the image matrix composed of 5 frames of sample content images to the rank of the low-rank part of the image matrix composed of 5 frames of sample synthetic images; wherein, by optimizing The nuclear norm can continuously optimize the rank of the low-rank part, and the nuclear norm is obtained by performing singular value decomposition on the matrix.

Exemplarily, for continuous K frames of images And its corresponding optical flow information (for example, may include forward optical flow information and reverse optical flow information) /> Composite image of student model output />

First, K frames of synthetic images can be mapped to a fixed frame according to the forward optical flow information, backward optical flow information and mask information, generally speaking, τ=[K/2] frames; that is, for the tth frame synthetic image , after mapping it to the synthesized image at frame τ, its low-rank matrix can be expressed as:

R _t ＝M _t-τ ⊙W[N _s (x _t ), f _t-τ ];

Among them, M _t-τ is used to represent the mask information calculated from the forward optical flow information and reverse optical flow information of K frames of images; W is used to represent the mapping operation (warp).

According to step 4, the matrix X obtained after vectorization and column combination can be obtained, X=[vec(R ₀ ),...,vec(R _K )] ^T ∈ ^{R K*L} , where L=H*W *3, K is used to indicate the number of rows of matrix X, which is the number of frames of the image; L is used to indicate the number of columns of matrix X; H is used to indicate the height of each frame of image; W is used to indicate the width of each frame of image.

Perform singular value decomposition on X, hoping to get the nuclear norm, the decomposition process is X=u∑v ^T , where the size of the matrix X is K*L, u∈R ^K*K , v∈R ^L*L , and Kernel norm ||X|| _* ＝tr(∑). tr is used to represent the trace of the matrix; for example, for an n×n matrix A, the sum of the elements on the main diagonal (diagonal from the upper left to the lower right) is called the trace of the matrix A, tr(A ).

The low-rank loss function is:

L＝(||X _input || _* -||X _s || _* ) ² ;

Among them, X _input represents the vectorized matrix obtained from the input K frames of images; X _s represents the vectorized matrix obtained from the synthesized images of K frames output by the target student network.

In the embodiment of the present application, the goal of introducing a low-rank loss function when training the style transfer model is to keep the same area after the style transfer process that appears in the original video in multiple adjacent frames of content images that are not motion boundaries. , that is, the rank of the region in the style-transferred video is close to the rank of the region in the original video, so that the stability of the style-transferred video can be improved.

Further, in the embodiment of the present application, the above image loss function also includes a residual loss function, and the residual loss function is obtained according to the difference between the composite image of the first sample and the composite image of the second sample, wherein, The first sample composite image is obtained by performing image style transfer processing on the N frames of sample content images by the first model, and the second sample composite image is obtained by performing image style transfer processing on the N frames of sample content images by the second model, The first model and the second model are image style transfer models pre-trained according to sample style images, the second model includes an optical flow module, the first model does not include an optical flow module, and the optical flow module is used to determine the optical flow of N frames of sample content images stream information.

It should be noted that the first model and the second model above refer to the style transfer model trained by the same style image, wherein the difference between the first model and the second model is that the first model does not include an optical flow module; the second The model includes an optical flow module; that is, the first model and the second model can use the same sample content image and sample style image during training, for example, the first model and the second model can refer to the same model in the training phase; but , in the test phase, the second model also needs to calculate the optical flow information between multiple frames of sample content images; while the first model does not need to calculate the optical flow information between multiple frames of images.

In the embodiment of this application, the goal of introducing the residual loss function when training the target style transfer model is to enable the neural network model to learn the style transfer model including the optical flow module and the style transfer model without the optical flow module during the training process. The difference of the synthesized image output by the model can improve the stability of the style transfer processed video obtained by the target transfer model.

Further, in the embodiments of the present application, in order to meet the deployment requirements of mobile terminals, a teacher-student model learning strategy can be adopted, that is, the style transfer model for training can be the target student model; during the training process, the image loss function can Update the parameters of the student model to obtain the target student model.

It should be noted that the network structure of the student model is the same as that of the target student model, and the student model may refer to a pre-trained style transfer model that does not require input of optical flow information during the test phase.

Optionally, in a possible implementation, the first model and the second model can be pre-trained teacher models, and the target style transfer model refers to training the student model to be trained according to the residual loss function and the knowledge distillation algorithm Get the target student model.

It should be understood that knowledge distillation refers to the key technology to make the deep learning model miniaturized and meet the deployment requirements of terminal equipment. Compared with compression techniques such as quantization and sparsification, it does not require specific hardware support to achieve the purpose of compressing the model. The knowledge distillation technology adopts the strategy of teacher-student model learning. Among them, the teacher model can mean that the model parameters are large, which generally cannot meet the deployment requirements; while the student model has few parameters and can be deployed directly. By designing an effective knowledge distillation algorithm, the student model can learn to imitate the behavior of the teacher model, and carry out effective knowledge transfer, so that the student model can finally perform the same processing ability as the teacher model.

In the embodiment of the present application, knowledge distillation can be performed on the model without the optical flow module during the test by using the model including the optical flow module during the test; in the style transfer of the video, due to the different structures of the teacher model and the student model and Depending on the training method, the stylization effects of the student model and the teacher model may not be exactly the same; if the student model directly learns the output information of the teacher model at the pixel level, the output of the student model may appear ghosting or blurred. In the embodiment of the present application, the target style transfer model may refer to the target student model, by adopting the knowledge distillation method of teacher-student model learning, the relationship between the student model to be trained and the style transfer result output by the pre-trained basic model is The difference is constantly approaching the difference between the style transfer results output by the teacher model including the optical flow module and the teacher model not including the optical flow module. This training method can effectively avoid the ghosting caused by the inconsistent styles of the teacher model and the student model. Phenomenon.

Optionally, in a possible implementation, the residual loss function is obtained according to the following equation,

Wherein, L _res represents the residual loss function; N _T represents the second model; Represents the first model; N _S represents the student model to be trained; /> Represents a pre-trained basic model, which has the same network structure as the student model to be trained; ^xi represents the i-th frame sample content image included in the sample video, and i is a positive integer.

In one example, the target style transfer model may refer to the target student model, and the target student model may be trained according to the pre-trained first teacher model (excluding the optical flow module), the pre-trained second teacher model (including the optical flow module), the pre-trained basic model trains a student model to be trained to obtain the target student model; among them, the network structure of the student model to be trained, the pre-trained basic model and the target student model are all the same, through the above-mentioned low The rank loss function, residual loss function, and perceptual loss function train the student model to be trained to obtain the target student model.

Wherein, the above-mentioned pre-trained basic model may refer to a style transfer model that does not include an optical flow module in the testing stage obtained through perceptual loss function training in advance; or, the pre-trained style transfer model may refer to Loss function pre-trained style transfer model that does not include the optical flow module in the test phase; perceptual loss function is used to represent the content loss between the synthetic image and the content image and the style loss between the synthetic image and the style image; the optical flow loss function It is used to represent the difference between the corresponding pixels of the composite image of adjacent frames.

In a possible implementation, in the process of training the student model to be trained, the above-mentioned residual loss function is used to make the output migration result (also called a synthetic image) between the student model to be trained and the pre-trained basic model The difference of is constantly approaching the difference of the transfer results output between the second model and the first model.

In the embodiment of the present application, the target style transfer model may refer to the target student model, by adopting the knowledge distillation method of teacher-student model learning, the relationship between the student model to be trained and the style transfer result output by the pre-trained basic model is The difference is constantly approaching the difference between the style transfer results output by the teacher model including the optical flow module and the teacher model not including the optical flow module. This training method can effectively avoid the ghosting caused by the inconsistent styles of the teacher model and the student model. Phenomenon.

In one example, the parameters of the neural network model are determined according to the image loss function between N frames of sample composite images and N frames of predicted composite images, wherein the image loss function includes the above-mentioned low-rank loss function, and the low-rank loss function is used to represent The difference between the low-rank matrix formed by N frames of sample content images and the low-rank matrix formed by N frames of sample composite images.

In one example, the parameters of the neural network model are determined according to the image loss function between N frames of sample composite images and N frames of predicted composite images, wherein the image loss function includes the above-mentioned low-rank loss function and the above-mentioned residual loss function, low The rank loss function is used to represent the difference between the low-rank matrix composed of N frames of sample content images and the low-rank matrix composed of N frames of sample composite images; the residual loss function is based on the composite image of the first sample and the composite image of the second sample The first sample composite image refers to the image obtained by performing image style transfer processing on the N-frame sample content images through the first model, and the second sample composite image refers to the N-frame sample content image through the second model An image obtained by performing image style transfer on a sample content image. The first model and the second model are pre-trained image style transfer models based on the same sample style image. The second model includes an optical flow module, and the first module does not include an optical flow module. , the optical flow module is used to determine the optical flow information of N frames of sample content images.

Optionally, in a possible implementation manner, the image loss function further includes a perceptual loss function, wherein the perceptual loss function includes a content loss and a style loss, and the content loss is used to represent the N frame prediction The image content difference between the synthesized image and its corresponding N-frame sample content images, the style loss is used to represent the image style difference between the N-frame predicted composite image and the sample style image.

Among them, the perceptual loss function can be used to represent the content similarity between the sample content image and the corresponding composite image; and to represent the style similarity between the sample style image and the corresponding composite image.

In one example, the parameters of the neural network model are determined according to the image loss function between N frames of sample composite images and N frames of predicted composite images, wherein the image loss function includes the above-mentioned low-rank loss function, the above-mentioned residual loss function, and the above-mentioned Perceptual loss function.

For example, the image loss is obtained by weighting the aforementioned low-rank loss function, the aforementioned residual loss function, and the aforementioned perceptual loss function.

Optionally, in a possible implementation manner, the parameters of the target style transfer model are obtained through multiple iterations of a backpropagation algorithm based on the image loss function.

Exemplarily, FIG. 8 is a schematic diagram of the training process of the style transfer model provided by the embodiment of the present application.

As shown in Figure 8, the first teacher model may refer to the above second model, that is, the pre-trained style transfer model including the optical flow module; the second teacher model may refer to the above first model, that is, the pre-trained style transfer model that does not include the optical flow module. The style transfer model of the flow module; the pre-trained basic model has the same network structure as the student model to be trained and the target student model; wherein, the input data of the first teacher model can include the T-th frame content image, processed with optical flow information The composite image of the T-1th frame, and the change information calculated by using the optical flow information, the change information may refer to the different regions in the two frames of content images obtained according to the content image of the T-1th frame and the content image of the T-th frame; The optical flow information may refer to the motion information of corresponding pixels in the T-1th content image and the Tth frame content image, and the output data of the first teacher model is the Tth frame composite image (#1). For the second teacher model, since the model does not include the optical flow module, the change information in the input data of the second teacher model can be all set to 1; the T-1 frame synthetic image processed with the optical flow information can be all Set to 0, the output data of the second teacher model is the T frame composite image (#2); the input data of the student model to be trained is the T frame content image, and the output data is the T frame composite image (#3); During the training process, the input data of the pre-trained basic model can be the content image of the Tth frame, and the output data is the predicted synthetic image of the Tth frame (#4); sequentially input the Tth frame to the T~T+N-1 A total of N frames of sample content images, the pre-trained basic model can obtain a total of N frames of T~T+N-1 frames to predict the composite image, according to the image loss function, namely the low-rank loss function, residual loss function and perceptual loss function The parameters of the student model to be trained are continuously updated through the backpropagation algorithm, so as to obtain the trained target student model.

Wherein, the above-mentioned pre-trained basic model may refer to a style transfer model that does not include an optical flow module in the testing stage obtained through perceptual loss function training in advance; or, the pre-trained style transfer model may refer to Loss function pre-trained style transfer model that does not include the optical flow module in the test phase; perceptual loss function is used to represent the content loss between the synthetic image and the content image and the style loss between the synthetic image and the style image; the optical flow loss function It is used to represent the difference between the corresponding pixels of the composite image of adjacent frames.

In the embodiment of the present application, a low-rank loss function is introduced in the process of training the style transfer model for video. Through the learning of low-rank information, the stability of the style-transferred video and the original video can be synchronized, so that Improve the stability of style transfer processed videos obtained by the target transfer model.

In addition, the style transfer model for videos trained in the embodiment of the present application may be the target student model obtained by using the teacher-learning model learning strategy, which can meet the needs of deploying the style transfer model on mobile devices on the one hand; on the other hand, When training the target student model, the learning model learns the difference between the output information of the teacher model that includes the optical flow module and the teacher model that does not include the optical flow module, so as to effectively avoid the ghosting caused by the inconsistent styles of the teacher model and the student model phenomenon, which can improve the stability of the video after style transfer processing obtained by the target transfer model.

FIG. 9 shows a schematic flow chart of a video style transfer method 700 provided by an embodiment of the present application. The method can be executed by an apparatus capable of image style transfer; for example, the method can be executed by the execution device 510 in FIG. 6 , or may also be executed by the local device 520 . Wherein, the method 700 includes S710 to S730, and these steps are described in detail below respectively.

S710. Obtain a video to be processed.

Wherein, the video to be processed includes N frames of content images to be processed, and N is an integer greater than or equal to 2.

Exemplarily, the video to be processed may be a video captured by the electronic device through a camera, or the video to be processed may also be a video obtained from inside the electronic device (for example, a video stored in the photo album of the electronic device, or a video stored in the electronic device video fetched from the cloud).

It should be understood that the above-mentioned video to be processed may be a video that requires style transfer, and this application does not set any limitation on the source of the video to be processed.

S720. Perform image style transfer processing on N frames of content images to be processed according to the target style transfer model to obtain N frames of composite images.

S730. Synthesize images according to the N frames, and obtain a style-transfer-processed video corresponding to the video to be processed.

Among them, the parameters of the target style transfer model are determined according to the image loss function that performs style transfer processing on N frames of sample content images according to the target style transfer model. The image loss function includes a low-rank loss function, and the low-rank loss function is used to represent the first lowest The difference between the rank matrix and the second low-rank matrix, the first low-rank matrix is obtained based on N frames of sample content images and optical flow information, and the second low-rank matrix is based on N frames of predicted composite images and the The optical flow information is obtained from the above optical flow information, and the optical flow information is used to represent the position difference of corresponding pixels between two adjacent frames of images in the N frames of sample content images. N frames of predicted composite images refer to the target style transfer model according to The sample style image is an image obtained by performing image style transfer processing on N frames of sample content images.

It should be noted that the above-mentioned N frames of sample content images refer to images adjacent to N frames in the video; N frames of synthetic images refer to images corresponding to N frames of sample content images; the above-mentioned target style transfer network can refer to The shown training method results in a pre-trained style transfer model.

It should be understood that for the style transfer processing of a single-frame image, that is, image style transfer, only the content in the content image and the style in the style image need to be considered; Style transfer should not only consider the stylized effect of the image, but also consider the stability between multiple frames of images; that is, it is necessary to ensure the smoothness of the video after the style transfer process and avoid noise such as splash screens and artifacts.

For example, assuming that the video includes 5 consecutive frames of images, that is, 5 consecutive frames of sample content images; the style transfer results of 5 frames of sample content images obtained through style transfer processing are 5 frames of sample composite images; the low-rank matrix can be used as follows way to calculate:

Step 1. Calculate optical flow information between image frames and image frames according to 5 frames of images in the video, that is, 5 frames of sample content images;

Step 2. Calculate the mask information according to the optical flow information, wherein the mask information can be used to represent the change area in two consecutive frames of images obtained according to the optical flow information;

Step 3. Calculate the low-rank part after aligning the 1st, 2nd, 4th, and 5th frame images with the 3rd frame image according to the optical flow information and mask information, and set the sparse part to 0;

Step 4. Expand the aligned 5 frames of images into vectors and combine them into a matrix by column, then the matrix can be a low-rank matrix.

It should be understood that when calculating the low-rank loss function, the goal is to approximate the rank of the low-rank part of the image matrix composed of 5 frames of sample content images to the rank of the low-rank part of the image matrix composed of 5 frames of sample synthetic images; wherein, by optimizing The nuclear norm can continuously optimize the rank of the low-rank part, and the nuclear norm is obtained by performing singular value decomposition on the matrix.

In the embodiment of the present application, the goal of introducing a low-rank loss function when training the style transfer model is that the regions that appear in adjacent multiple frames of images in the original video and are not motion boundaries remain the same after the style transfer process. That is, the rank of the region in the style-transferred video is close to the rank of the original video, so that the stability of the style-transferred video can be improved.

Further, in the embodiment of the present application, the above image loss function also includes a residual loss function, and the residual loss function is obtained according to the difference between the composite image of the first sample and the composite image of the second sample, wherein, The first sample composite image is obtained by performing image style transfer processing on the N frames of sample content images by the first model, and the second sample composite image is obtained by performing image style transfer processing on the N frames of sample content images by the second model, The first model and the second model are image style transfer models pre-trained according to sample style images, and the second model includes an optical flow module, which is used to determine optical flow information of N frames of sample content images.

It should be understood that the difference between the first sample composite image and the second sample composite image may refer to a position difference between pixel values corresponding to the first sample composite image and the second sample composite image.

It should be noted that the first model and the second model above refer to the style transfer model trained by the same style image, wherein the difference between the first model and the second model is that the first model does not include an optical flow module; the second The model includes an optical flow module; that is, the first model and the second model can use the same sample content image and sample style image during training; for example, the first model and the second model can refer to the same model in the training phase; but , in the test phase, the second model also needs to calculate the optical flow information between multiple frames of sample content images; while the first model does not need to calculate the optical flow information between multiple frames of images.

In the embodiment of this application, the goal of introducing the residual loss function when training the target style transfer model is to enable the neural network model to learn the style transfer model including the optical flow module and the style transfer model without the optical flow module during the training process. The difference of the synthesized image output by the model can improve the stability of the style transfer processed video obtained by the target transfer model.

Further, in the embodiments of the present application, in order to meet the deployment requirements of mobile terminals, a teacher-student model learning strategy can be adopted, that is, the style transfer model for training can be the target student model; during the training process, the image loss function can Update the parameters of the student model to obtain the target student model.

It should be noted that the learning model has the same network structure as the target student model, and the student model can refer to a pre-trained style transfer model that does not require input of optical flow information during the test phase.

Optionally, in a possible implementation, the first model and the second model can be pre-trained teacher models, and the target style transfer model refers to training the student model to be trained according to the residual loss function and the knowledge distillation algorithm Get the target student model.

It should be understood that knowledge distillation refers to the key technology to make the deep learning model miniaturized and meet the deployment requirements of terminal equipment. Compared with compression techniques such as quantization and sparsification, it does not require specific hardware support to achieve the purpose of compressing the model. The knowledge distillation technology adopts the strategy of teacher-student model learning. Among them, the teacher model can refer to large model parameters, which generally cannot meet the deployment requirements; while the student model has few parameters and can be directly deployed. By designing an effective knowledge distillation algorithm, the student model can learn to imitate the behavior of the teacher model, and carry out effective knowledge transfer, so that the student model can finally perform the same processing ability as the teacher model.

In the embodiment of the present application, knowledge distillation can be performed on the model without the optical flow module during the test by using the model including the optical flow module during the test; in the style transfer of the video, due to the different structures of the teacher model and the student model and Depending on the training method, the stylization effects of the student model and the teacher model may not be exactly the same; if the student model directly learns the output information of the teacher model at the pixel level, the output of the student model may appear ghosting or blurred. In the embodiment of the present application, by learning the difference between the style transfer results output by the teacher model that includes the optical flow module during the test and the teacher model that does not include the optical flow module during the test, it is possible to effectively avoid the inconsistency between the teacher model and the student model. The ghosting phenomenon caused by the target transfer model can improve the stability of the video after style transfer processing obtained by the target transfer model.

Optionally, in a possible implementation, the residual loss function is obtained according to the following equation,

Wherein, L _res represents the residual loss function; N _T represents the second model; Represents the first model; N _S represents the student model to be trained; /> Represents a pre-trained basic model, which has the same network structure as the student model to be trained; ^xi represents the i-th frame sample content image included in the sample video, and i is a positive integer.

In one example, the parameters of the neural network model are determined according to the image loss function between N frames of sample composite images and N frames of predicted composite images, wherein the image loss function includes the above-mentioned low-rank loss function, and the low-rank loss function is used to represent The difference between the low-rank matrix formed by N frames of sample content images and the low-rank matrix formed by N frames of sample composite images.

In one example, the parameters of the neural network model are determined according to the image loss function between N frames of sample composite images and N frames of predicted composite images, wherein the image loss function includes the above-mentioned low-rank loss function and the above-mentioned residual loss function, low The rank loss function is used to represent the difference between the low-rank matrix composed of N frames of sample content images and the low-rank matrix composed of N frames of sample composite images; the residual loss function is based on the composite image of the first sample and the composite image of the second sample The first sample composite image refers to the image obtained by performing image style transfer processing on the N-frame sample content images through the first model, and the second sample composite image refers to the N-frame sample content image through the second model An image obtained by performing image style transfer processing on a sample content image. The first model and the second model are pre-trained image style transfer models based on the same sample style image. The second model includes an optical flow module, and the first model does not include an optical flow module. , the optical flow module can be used to determine the optical flow information of N frames of sample content images.

Optionally, in a possible implementation manner, the image loss function further includes a perceptual loss function, wherein the perceptual loss function includes a content loss and a style loss, and the content loss is used to represent the N frame prediction The image content difference between the synthesized image and its corresponding N-frame sample content images, the style loss is used to represent the image style difference between the N-frame predicted composite image and the sample style image.

Among them, the perceptual loss function can be used to represent the content similarity between the sample content image and the corresponding composite image; and to represent the style similarity between the sample style image and the corresponding composite image.

In one example, the parameters of the neural network model are determined according to the image loss function between N frames of sample composite images and N frames of predicted composite images, wherein the image loss function includes the above-mentioned low-rank loss function, the above-mentioned residual loss function, and the above-mentioned Perceptual loss function.

For example, the image loss is obtained by weighting the aforementioned low-rank loss function, the aforementioned residual loss function, and the aforementioned perceptual loss function.

Optionally, in a possible implementation manner, the parameters of the target style transfer model are obtained through multiple iterations of a backpropagation algorithm based on the image loss function.

In the embodiment of this application, a low-rank loss function is introduced in the process of the target style transfer model. Through the learning of low-rank information, the stability of the style-transferred video and the original video can be synchronized, thereby improving the target transfer model. Stabilization of videos after the resulting style transfer process.

In addition, in the embodiment of the present application, the target style transfer model can be a target student model obtained by adopting the strategy of teacher-learning model learning, which can meet the needs of deploying the style transfer model on mobile devices on the one hand; on the other hand, when training the target student model When the learning model learns the difference between the output information of the teacher model that includes the optical flow module and the teacher model that does not include the optical flow module, it can effectively avoid the ghosting phenomenon caused by the inconsistent styles of the teacher model and the student model, thereby improving Stabilization of videos after style transfer processing obtained by the object transfer model.

Further, the target style transfer model provided in the embodiment of the present application does not need to calculate the optical flow information between the multiple frames of images included in the video during the test phase, that is, during the style transfer process of the video to be processed, so the embodiment of the present application provides The target style transfer model can not only improve the stability, but also shorten the processing time of the model's style transfer, and improve the operation efficiency of the target style transfer model.

Exemplarily, FIG. 10 is a schematic diagram of the training phase and the testing phase provided by the embodiment of the present application.

Training phase:

For example, in the embodiment of the present application, the Flownet2 network can be used and a data set with optical flow data can be generated based on the Hollywood2 data set, and the network can be trained by using the training method shown in the embodiment of the present application.

For example, the specific implementation steps include: first, only use the perceptual loss function or use the perceptual loss function and the optical flow loss function to train a style transfer model, that is, the pre-trained basic model Using video data and optical flow data to train a teacher model _NT including the optical flow module and a teacher model not including the optical flow module /> then according to _NT , /> And the above-mentioned low-rank loss function and residual loss function train the student model _NS to be trained, and finally obtain the target student model after training, in which the network structures of the pre-trained basic model, the student model to be trained and the target student model are all same.

It should be noted that, for the specific implementation manner of the training phase, reference may be made to the foregoing descriptions in FIG. 7 and FIG. 8 , which will not be repeated here.

Testing phase:

Exemplarily, during the test, test data may be input into the target student model, and test results, that is, data after style transfer processing, may be obtained through the target student model.

It should be noted that, for the specific implementation manner of the testing phase, reference may be made to the description in FIG. 9 above, which will not be repeated here.

Table 1

Among them, the teacher model in Table 1 may refer to the second model in the above embodiment, that is, the style transfer model including the optical flow module in the test phase; the first type of student model may refer to the pre-trained model obtained by perceptual loss function training student model; the second type of student model can refer to the pre-trained student model obtained through perceptual loss function and optical flow loss function training; loss function 1 can refer to the residual loss function in this application; loss function 2 can be Refers to the low-rank loss function in this application; Alley_2, Ambush_5, Bandage_2, Market_6, and Temple_2 respectively represent the names of the five video data in the MPI-Sintel dataset; All represents the previous five videos. Table 1 shows the test results of the stability of different models by using the MPI-Sintel data set; wherein, the calculation method of the stability index can use the following formula:

Among them, T represents the number of image frames included in the video; D=c*w*d, M _t ∈ R ^(w*d) represents the mask information, O _t represents the style transfer result of t frame; O _(t-1) represents The style transfer result of t-1 frame; W _t represents the optical flow information from t-1 frame to t frame; W _t (O _t-1 ) represents the style transfer result of t-1 frame and the style transfer result of t frame align.

As shown in Table 1, the smaller the result of the stability index, the better the stability of the output data after the migration process of the model; from the test results shown in Table 1, it can be seen that the target migration model provided by the embodiments of the present application The stability of the output data after style transfer processing is significantly better than other models.

It should be understood that the above illustrations are intended to help those skilled in the art understand the embodiments of the present application, rather than to limit the embodiments of the present application to the illustrated specific values or specific scenarios. Those skilled in the art can obviously make various equivalent modifications or changes based on the above illustrations given, and such modifications or changes also fall within the scope of the embodiments of the present application.

The above describes in detail the style transfer training method and the video style transfer method provided by the embodiment of the present application with reference to FIG. 1 to FIG. 10 ; the device embodiment of the present application will be described in detail below in conjunction with FIG. 11 to FIG. 14 . It should be understood that the image processing apparatus in the embodiment of the present application can execute the various methods in the foregoing embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the foregoing method embodiments.

Fig. 11 is a schematic block diagram of an apparatus for video style transfer provided by an embodiment of the present application.

It should be understood that the apparatus 800 for video style transfer may execute the method shown in FIG. 9 , or the method in the testing phase shown in FIG. 10 . The apparatus 800 includes: an acquisition unit 810 and a processing unit 820 .

Wherein, the obtaining unit 810 is used to obtain the video to be processed, wherein the video to be processed includes N frames of content images to be processed, and N is an integer greater than or equal to 2; the processing unit 820 is used to transfer the N frames according to the target style transfer model Perform image style transfer processing on the content image to be processed to obtain N frames of composite images; according to the N frames of composite images, obtain the video after the style transfer processing corresponding to the video to be processed,

Wherein, the parameters of the target style transfer model are determined according to the image loss function that performs style transfer processing on N frames of sample content images according to the target style transfer model, and the image loss function includes a low-rank loss function, and the low-rank The loss function is used to represent the difference between the first low-rank matrix and the second low-rank matrix. The first low-rank matrix is obtained based on N frames of sample content images and optical flow information. The second low-rank matrix is Obtained based on N frames of predicted composite images and the optical flow information, the optical flow information is used to represent the position difference of corresponding pixels between two adjacent frames of images in the N frames of sample content images, and the N frames of predicted The synthetic image refers to an image obtained by performing image style transfer processing on the N frames of sample content images according to the sample style images through the target style transfer model.

Optionally, as an embodiment, the image loss function further includes a residual loss function, and the residual loss function is obtained according to the difference between the composite image of the first sample and the composite image of the second sample,

Wherein, the first sample synthetic image refers to an image obtained by performing image style transfer processing on the N frames of sample content images through the first model, and the second sample synthetic image refers to the image obtained by performing image style transfer processing on the N frames of sample content images through the second model. An image obtained by performing image style transfer processing on a frame sample content image, the first model and the second model are image style transfer models pre-trained according to the sample style image, the second model includes an optical flow module, and The first module does not include the optical flow module, and the optical flow module is used to determine the optical flow information of the N frames of sample content images.

Optionally, as an embodiment, the first model and the second model are pre-trained teacher models, and the target style transfer model refers to the students to be trained according to the residual loss function and knowledge distillation algorithm The target student model obtained by training the model.

Optionally, as an embodiment, the residual loss function is obtained according to the following equation,

Wherein, L _res represents the residual loss function; N _T represents the second model; Represents the first model; N _S represents the student model to be trained; /> Represents a pre-trained basic model, which has the same network structure as the student model to be trained; ^xi represents the i-th frame sample content image included in the sample video, and i is a positive integer.

Optionally, as an embodiment, the image loss function further includes a perceptual loss function, and the image loss function further includes a perceptual loss function, wherein the perceptual loss function includes a content loss and a style loss, and the content loss uses To represent the image content difference between the N frames of predicted composite images and their corresponding N frames of sample content images, the style loss is used to represent the images between the N frames of predicted composite images and the sample style images style difference.

Optionally, as an embodiment, the image loss function is obtained by weighting the low-rank loss function, the residual loss function, and the perceptual loss function.

Optionally, as an embodiment, the parameters of the target style transfer model are obtained through multiple iterations of a backpropagation algorithm based on the image loss function.

Fig. 12 is a schematic block diagram of a training device for a style transfer model provided by an embodiment of the present application.

It should be understood that the training device 900 may implement the style transfer model training method shown in FIG. 7 , FIG. 8 or FIG. 10 . The training device 900 includes: an acquisition unit 910 and a processing unit 920 .

Wherein, the obtaining unit 910 is used to obtain training data, wherein the training data includes N frames of sample content images, sample style images and N frames of composite images, and the N frames of composite images are based on the comparison of the N frames of sample style images. An image obtained after the frame sample content image is subjected to image style transfer processing, N is an integer greater than or equal to 2; the processing unit 920 is used to perform image style transfer on the N frames of sample content images according to the sample style image through a neural network model Processing to obtain N frames of predicted composite images; according to the image loss function between the N frames of sample content images and the N frames of predicted composite images, determine the parameters of the neural network model,

Wherein, the image loss function includes a low-rank loss function, and the low-rank loss function is used to represent the difference between the first low-rank matrix and the second low-rank matrix, and the first low-rank matrix is based on N frame samples The content image and optical flow information are obtained, the second low-rank matrix is obtained based on N frames of predicted composite images and the optical flow information, and the optical flow information is used to represent the adjacent The position difference of corresponding pixel points between two frames of images.

Optionally, as an embodiment, the image loss function further includes a residual loss function, and the residual loss function is obtained according to the difference between the composite image of the first sample and the composite image of the second sample,

Wherein, the first sample synthetic image refers to an image obtained by performing image style transfer processing on the N frames of sample content images through the first model, and the second sample synthetic image refers to the image obtained by performing image style transfer processing on the N frames of sample content images through the second model. An image obtained by performing image style transfer processing on a frame sample content image, the first model and the second model are image style transfer models pre-trained according to the sample style image, the second model includes an optical flow module, and The first model does not include the optical flow module, and the optical flow module is used to determine the optical flow information of the N frames of sample content images.

Optionally, as an embodiment, the first model and the second model are pre-trained teacher models, and the target style transfer model refers to the students to be trained according to the residual loss function and knowledge distillation algorithm The target student model obtained by training the model.

Optionally, as an embodiment, the residual loss function is obtained according to the following equation,

Wherein, L _res represents the residual loss function; N _T represents the second model; Represents the first model; N _S represents the student model to be trained; /> Represents a pre-trained basic model, which has the same network structure as the student model to be trained; ^xi represents the i-th frame sample content image included in the sample video, and i is a positive integer.

Optionally, as an embodiment, the image loss function further includes a perceptual loss function, wherein the perceptual loss function includes a content loss and a style loss, and the content loss is used to indicate that the N frames of predicted composite images correspond to The image content differences between the N frames of sample content images, the style loss is used to represent the image style differences between the N frames of predicted composite images and the sample style images.

Optionally, as an embodiment, the image loss function is obtained by weighting the low-rank loss function, the residual loss function, and the perceptual loss function.

Optionally, as an embodiment, the parameters of the target style transfer model are obtained through multiple iterations of a backpropagation algorithm based on the image loss function.

It should be noted that the above-mentioned device 800 and the training device 900 are embodied in the form of functional units. The term "unit" here may be implemented in the form of software and/or hardware, which is not specifically limited.

For example, a "unit" may be a software program, a hardware circuit or a combination of both to realize the above functions. The hardware circuitry may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group processor) for executing one or more software or firmware programs. etc.) and memory, incorporating logic, and/or other suitable components to support the described functionality.

Therefore, the units of each example described in the embodiments of the present application can be realized by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

FIG. 13 is a schematic diagram of a hardware structure of an apparatus for video style migration provided by an embodiment of the present application. The apparatus 1000 shown in FIG. 13 (the apparatus 1000 may specifically be a computer device) includes a memory 1010 , a processor 1020 , a communication interface 1030 and a bus 1040 . Wherein, the memory 1010 , the processor 1020 , and the communication interface 1030 are connected to each other through the bus 1040 .

The memory 1010 may be a read only memory (read only memory, ROM), a static storage device, a dynamic storage device or a random access memory (random access memory, RAM). The memory 1010 may store a program, and when the program stored in the memory 1010 is executed by the processor 1020, the processor 1020 is configured to execute each step of the method for video style transfer in the embodiment of the present application; for example, execute each step shown in FIG. 9 .

It should be understood that the apparatus for video style transfer shown in the embodiment of the present application may be a server, for example, a server in the cloud, or a chip configured in the server in the cloud.

The processor 1020 may be a general-purpose central processing unit (central processing unit, CPU), a microprocessor, an application specific integrated circuit (application specific integrated circuit, ASIC), a graphics processing unit (graphics processing unit, GPU) or one or more The integrated circuit is used to execute related programs to realize the image classification method of the method embodiment of the present application.

The processor 1020 may also be an integrated circuit chip, which has a signal processing capability. During implementation, each step of the image classification method of the present application may be completed by an integrated logic circuit of hardware in the processor 1020 or instructions in the form of software.

The above-mentioned processor 1020 can also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application-specific integrated circuit (ASIC), a ready-made programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates Or transistor logic devices, discrete hardware components. Various methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory 1010, and the processor 1020 reads the information in the memory 1010, and combines its hardware to complete the functions required by the units included in the device shown in Figure 11 in the implementation of the application, or execute the method embodiment of the application Figure 9 shows the video style transfer method.

The communication interface 1030 implements communication between the apparatus 1000 and other devices or communication networks by using a transceiver device such as but not limited to a transceiver.

Bus 1040 may include pathways for communicating information between various components of device 1000 (eg, memory 1010, processor 1020, communication interface 1030).

Fig. 14 is a schematic diagram of the hardware structure of the training device for the style transfer model provided by the embodiment of the present application. The training device 1100 shown in FIG. 14 (the training device 1100 may specifically be a computer device) includes a memory 1110 , a processor 1120 , a communication interface 1130 and a bus 1140 . Wherein, the memory 1110 , the processor 1120 , and the communication interface 1130 are connected to each other through the bus 1140 .

The memory 1110 may be a read only memory (read only memory, ROM), a static storage device, a dynamic storage device or a random access memory (random access memory, RAM). The memory 1110 may store a program. When the program stored in the memory 1110 is executed by the processor 1120, the processor 1120 is used to execute each step of the method for training the style transfer model in the embodiment of the present application; each step shown.

It should be understood that the training device shown in the embodiment of the present application may be a server, for example, it may be a server in the cloud, or it may also be a chip configured in the server in the cloud.

Exemplarily, the processor 1120 may be a general-purpose central processing unit (central processing unit, CPU), a microprocessor, an application specific integrated circuit (application specific integrated circuit, ASIC), a graphics processing unit (graphics processing unit, GPU) or One or more integrated circuits are used to execute related programs to implement the image classification model training method of the method embodiment of the present application.

Exemplarily, the processor 1120 may also be an integrated circuit chip having a signal processing capability. In the implementation process, each step of the style transfer model training method of the present application can be completed by an integrated logic circuit of hardware in the processor 1120 or instructions in the form of software.

The above-mentioned processor 1120 may also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates Or transistor logic devices, discrete hardware components. Various methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory 1110, and the processor 1120 reads the information in the memory 1110, and combines its hardware to complete the functions required by the units included in the training device shown in Figure 12, or execute the method shown in Figure 7 of the method embodiment of the present application Or the training method of the style transfer model shown in FIG. 8 .

The communication interface 1130 implements communication between the training device 1100 and other devices or communication networks using a transceiver device such as but not limited to a transceiver.

Bus 1140 may include pathways for communicating information between various components of exercise device 1100 (eg, memory 1110 , processor 1120 , communication interface 1130 ).

It should be noted that although the above-mentioned device 1000 and training device 1100 only show a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the device 1000 and the training device 1100 may also include other necessary devices. At the same time, those skilled in the art should understand according to specific needs that the above-mentioned device 1000 and training device 1100 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the above-mentioned device 1000 and training device 1100 may only include the devices necessary to realize the embodiment of the present application, instead of all the devices shown in FIG. 13 or FIG. 14 .

Exemplarily, an embodiment of the present application further provides a chip, where the chip includes a transceiver unit and a processing unit. Wherein, the transceiver unit may be an input/output circuit or a communication interface; the processing unit may be a processor, a microprocessor or an integrated circuit integrated on the chip; the chip may execute the video style transfer method in the above method embodiment.

Exemplarily, an embodiment of the present application further provides a chip, where the chip includes a transceiver unit and a processing unit. Wherein, the transceiver unit may be an input/output circuit or a communication interface; the processing unit may be a processor, a microprocessor or an integrated circuit integrated on the chip; the chip may execute the training method of the style transfer model in the above method embodiment.

Exemplarily, an embodiment of the present application further provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed, the method for video style transfer in the foregoing method embodiments is executed.

Exemplarily, the embodiment of the present application further provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed, the method for training the style transfer model in the above method embodiment is executed.

Exemplarily, an embodiment of the present application further provides a computer program product including an instruction, and when the instruction is executed, the video style transfer method in the above method embodiment is executed.

Exemplarily, an embodiment of the present application further provides a computer program product including instructions, and when the instructions are executed, the method for training the style transfer model in the above method embodiment is executed.

It should be understood that the processor in the embodiment of the present application may be a central processing unit (central processing unit, CPU), and the processor may also be other general processors, digital signal processors (digital signal processor, DSP), application specific integrated circuits (application specific integrated circuit (ASIC), off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the non-volatile memory can be read-only memory (read-only memory, ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically programmable Erases programmable read-only memory (electrically EPROM, EEPROM) or flash memory. Volatile memory can be random access memory (RAM), which acts as external cache memory. By way of illustration and not limitation, many forms of random access memory (RAM) are available such as static random access memory (static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (RAM), Access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory (synchlink DRAM, SLDRAM) and direct memory bus random access memory (direct rambus RAM, DR RAM).

The above-mentioned embodiments may be implemented in whole or in part by software, hardware, firmware or other arbitrary combinations. When implemented using software, the above-described embodiments may be implemented in whole or in part in the form of computer program products. The computer program product comprises one or more computer instructions or computer programs. When the computer instruction or computer program is loaded or executed on the computer, the processes or functions according to the embodiments of the present application will be generated in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center that includes one or more sets of available media. The available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media. The semiconductor medium may be a solid state drive.

It should be understood that the term "and/or" in this article is only an association relationship describing associated objects, indicating that there may be three relationships, for example, A and/or B may mean: A exists alone, and A and B exist at the same time , there are three cases of B alone, where A and B can be singular or plural. In addition, the character "/" in this article generally indicates that the related objects are an "or" relationship, but it may also indicate an "and/or" relationship, which can be understood by referring to the context.

In this application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one item (piece) of a, b, or c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c can be single or multiple .

It should be understood that, in various embodiments of the present application, the sequence numbers of the above-mentioned processes do not mean the order of execution, and the execution order of the processes should be determined by their functions and internal logic, and should not be used in the embodiments of the present application. The implementation process constitutes any limitation.

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other various media that can store program codes. .

The above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto. Anyone familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the application. Should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (30)

1. A method for training a style migration model, comprising:

acquiring training data, wherein the training data comprises N frames of sample content images, sample style images and N frames of synthesized images, the N frames of synthesized images are images obtained by performing image style migration processing on the N frames of sample content images according to the sample style images, and N is an integer greater than or equal to 2;

performing image style migration processing on the N frames of sample content images according to the sample style images through a neural network model to obtain N frames of predicted composite images;

determining parameters of the neural network model according to an image loss function between the N frames of sample content images and the N frames of predicted composite images,

the image loss function comprises a low-rank loss function, the low-rank loss function is used for representing the difference between a first low-rank matrix and a second low-rank matrix, the first low-rank matrix is obtained based on the N-frame sample content image and optical flow information, the second low-rank matrix is obtained based on the N-frame prediction synthesized image and the optical flow information, and the optical flow information is used for representing the position difference of corresponding pixel points between two adjacent frame images in the N-frame sample content image.

2. The training method of claim 1 wherein the image loss function further comprises a residual loss function derived from a difference between the first sample composite image and the second sample composite image,

the first sample synthesized image is an image obtained by performing image style migration processing on the N frames of sample content images through a first model, the second sample synthesized image is an image obtained by performing image style migration processing on the N frames of sample content images through a second model, the first model and the second model are image style migration models trained in advance according to the sample style images, the second model comprises an optical flow module, the first model does not comprise the optical flow module, and the optical flow module is used for determining optical flow information.

3. The training method of claim 2, wherein the first model and the second model are pre-trained teacher models, and the target style migration model is a target student model obtained by training a student model to be trained according to the residual loss function and a knowledge distillation algorithm.

4. The training method of claim 3, wherein the residual loss function is obtained according to the following equation,

wherein L is _res Representing the residual loss function; n (N) _T Representing the secondA model;representing the first model; n (N) _S Representing the student model to be trained; />Representing a pre-trained base model, the pre-trained base model being the same as the network structure of the student model to be trained; x is x ⁱ And representing an ith frame of sample content image included in the N frames of sample content images, wherein i is a positive integer.

5. Training method according to any of the claims 2-4, wherein the image loss function further comprises a perceptual loss function, wherein the perceptual loss function comprises a content loss representing an image content difference between the N-frame predictive composite image and the N-frame sample content image corresponding thereto and a style loss representing an image style difference between the N-frame predictive composite image and the sample style image.

6. The training method of claim 5 wherein said image loss function is obtained by weighting said low rank loss function, said residual loss function, and said perceptual loss function.

7. Training method according to claim 3 or 4, wherein the parameters of the target style migration model are obtained by a number of iterations of a back propagation algorithm based on the image loss function.

8. A method of video style migration, comprising:

acquiring a video to be processed, wherein the video to be processed comprises N frames of content images to be processed, and N is an integer greater than or equal to 2;

performing image style migration processing on the N frames of content images to be processed according to a target style migration model to obtain N frames of synthesized images;

obtaining a video after style migration processing corresponding to the video to be processed according to the N frames of synthesized images,

the parameters of the target style migration model are determined according to an image loss function of performing style migration processing on N frames of sample content images by the target style migration model, the image loss function comprises a low-rank loss function, the low-rank loss function is used for representing the difference between a first low-rank matrix and a second low-rank matrix, the first low-rank matrix is obtained based on the N frames of sample content images and optical flow information, the second low-rank matrix is obtained based on N frames of predicted synthesized images and the optical flow information, the optical flow information is used for representing the position difference of corresponding pixel points between two adjacent frames of images in the N frames of sample content images, and the N frames of predicted synthesized images are images obtained after performing image style migration processing on the N frames of sample content images according to the sample style images by the target style migration model.

9. The method of claim 8, wherein the image loss function further comprises a residual loss function derived from a difference between the first sample composite image and the second sample composite image,

the first sample synthesized image is an image obtained by performing image style migration processing on the N frames of sample content images through a first model, the second sample synthesized image is an image obtained by performing image style migration processing on the N frames of sample content images through a second model, the first model and the second model are image style migration models trained in advance according to the sample style images, the second model comprises an optical flow module, the first model does not comprise the optical flow module, and the optical flow module is used for determining optical flow information.

10. The method of claim 9, wherein the first model and the second model are pre-trained teacher models, and the target style migration model is a target student model obtained by training a student model to be trained according to the residual loss function and a knowledge distillation algorithm.

11. The method of claim 10, wherein the residual loss function is derived from the equation,

wherein L is _res Representing the residual loss function; n (N) _T Representing the second model;representing the first model; n (N) _S Representing the student model to be trained; />Representing a pre-trained style migration model, wherein the pre-trained style migration model has the same network structure as the student model to be trained; x is x ⁱ And representing an ith frame of sample content image included in the N frames of sample content images, wherein i is a positive integer.

12. The method of any of claims 9 to 11, wherein the image loss function further comprises a perceptual loss function, wherein the perceptual loss function comprises a content loss representing an image content difference between the N-frame predicted composite image and the N-frame sample content image corresponding thereto and a style loss representing an image style difference between the N-frame predicted composite image and the sample style image.

13. The method of claim 12, wherein the image loss function is obtained by weighting the low rank loss function, the residual loss function, and the perceptual loss function.

14. The method according to any one of claims 8 to 11, wherein the parameters of the target style migration model are obtained by a plurality of iterations of a back propagation algorithm based on the image loss function.

15. A training device for a style migration model, comprising:

the training device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring training data, the training data comprises N frames of sample content images, sample style images and N frames of synthesized images, the N frames of synthesized images are images obtained by performing image style migration processing on the N frames of sample content images according to the sample style images, and N is an integer greater than or equal to 2;

the processing unit is used for performing image style migration processing on the N frames of sample content images according to the sample style images through a neural network model to obtain N frames of predicted composite images; determining parameters of the neural network model according to an image loss function between the N frames of sample content images and the N frames of predicted composite images,

the image loss function comprises a low-rank loss function, the low-rank loss function is used for representing the difference between a first low-rank matrix and a second low-rank matrix, the first low-rank matrix is obtained based on the N-frame sample content image and optical flow information, the second low-rank matrix is obtained based on the N-frame prediction synthesized image and the optical flow information, and the optical flow information is used for representing the position difference of corresponding pixel points between two adjacent frame images in the N-frame sample content image.

16. The training device of claim 15 wherein the image loss function further comprises a residual loss function derived from a difference between the first sample composite image and the second sample composite image,

the first sample synthesized image is an image obtained by performing image style migration processing on the N frames of sample content images through a first model, the second sample synthesized image is an image obtained by performing image style migration processing on the N frames of sample content images through a second model, the first model and the second model are image style migration models trained in advance according to the sample style images, the second model comprises an optical flow module, the first model does not comprise the optical flow module, and the optical flow module is used for determining optical flow information.

17. The training apparatus of claim 16 wherein the first model and the second model are pre-trained teacher models and the target style migration model is a target student model obtained by training a student model to be trained according to the residual loss function and a knowledge distillation algorithm.

18. The training apparatus of claim 17 wherein said residual loss function is derived from the following equation,

wherein L is _res Representing the residual loss function; n (N) _T Representing the second model;representing the first model; n (N) _S Representing the student model to be trained; />Representing a pre-trained base model, the pre-trained base model being the same as the network structure of the student model to be trained; x is x ⁱ And representing an ith frame of sample content image included in the N frames of sample content images, wherein i is a positive integer.

19. The training apparatus of any of claims 16 to 18 wherein the image loss function further comprises a perceptual loss function, wherein the perceptual loss function comprises a content loss representing an image content difference between the N-frame predicted composite image and the N-frame sample content image corresponding thereto and a style loss representing an image style difference between the N-frame predicted composite image and the sample style image.

20. The training apparatus of claim 19 wherein the image loss function is obtained by weighting the low rank loss function, the residual loss function, and the perceptual loss function.

21. Training apparatus according to claim 17 or 18, wherein the parameters of the target style migration model are derived by a number of iterations of a back propagation algorithm based on the image loss function.

22. An apparatus for video style migration, comprising:

the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a video to be processed, the video to be processed comprises N frames of content images to be processed, and N is an integer greater than or equal to 2;

the processing unit is used for carrying out image style migration processing on the N frames of content images to be processed according to the target style migration model to obtain N frames of synthesized images; obtaining a video after style migration processing corresponding to the video to be processed according to the N frames of synthesized images,

the parameters of the target style migration model are determined according to an image loss function of performing style migration processing on N frames of sample content images by the target style migration model, the image loss function comprises a low-rank loss function, the low-rank loss function is used for representing the difference between a first low-rank matrix and a second low-rank matrix, the first low-rank matrix is obtained based on the N frames of sample content images and optical flow information, the second low-rank matrix is obtained based on N frames of predicted synthesized images and the optical flow information, the optical flow information is used for representing the position difference of corresponding pixel points between two adjacent frames of images in the N frames of sample content images, and the N frames of predicted synthesized images are images obtained after performing image style migration processing on the N frames of sample content images according to the sample style images by the target style migration model.

23. The apparatus of claim 22, wherein the image loss function further comprises a residual loss function derived from a difference between the first sample composite image and the second sample composite image,

the first sample synthesized image is an image obtained by performing image style migration processing on the N frames of sample content images through a first model, the second sample synthesized image is an image obtained by performing image style migration processing on the N frames of sample content images through a second model, the first model and the second model are image style migration models trained in advance according to the sample style images, the second model comprises an optical flow module, the first model does not comprise the optical flow module, and the optical flow module is used for determining optical flow information.

24. The apparatus of claim 23, wherein the first model and the second model are pre-trained teacher models, and the target style migration model is a target student model obtained by training a student model to be trained according to the residual loss function and a knowledge distillation algorithm.

25. The apparatus of claim 24, wherein the residual loss function is derived from the following equation,

wherein L is _res Representing the residual loss function; n (N) _T Representing the second model;representing the first model; n (N) _S Representing the student model to be trained; />Representing a pre-trained base model, the pre-trained base model being the same as the network structure of the student model to be trained; x is x ⁱ And representing an ith frame of sample content image included in the N frames of sample content images, wherein i is a positive integer.

26. The apparatus of any of claims 23 to 25, wherein the image loss function further comprises a perceptual loss function, wherein the perceptual loss function comprises a content loss representing an image content difference between the N-frame predicted composite image and the N-frame sample content image corresponding thereto and a style loss representing an image style difference between the N-frame predicted composite image and the sample style image.

27. The apparatus of claim 26, wherein the image loss function is obtained by weighting the low rank loss function, the residual loss function, and the perceptual loss function.

28. The apparatus of any one of claims 22 to 25, wherein the parameters of the target style migration model are derived by a plurality of iterations of a back propagation algorithm based on the image loss function.

29. A training device for a style migration model, comprising a processor and a memory, the memory for storing program instructions, the processor for invoking the program instructions to perform the method of any of claims 1-7 or 8-14.

30. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein program instructions, which when executed by a processor, implement the method of any of claims 1 to 7 or 8 to 14.