CN114793300A - Virtual video customer service robot synthesis method and system based on generation countermeasure network - Google Patents

Abstract

The invention relates to the technical field of human face video synthesis, and discloses a virtual video customer service robot synthesis method and system based on a generation countermeasure network. The method and the system for synthesizing the virtual video customer service robot based on the generated countermeasure network have the innovativeness that two schemes for synthesizing the virtual video customer service robot are provided, so that a user can select the virtual video customer service robot independently according to the requirements; the synthesis scheme can enable a user to realize the synthesis of various languages, the arbitrary selection of customer service images and the application of various scenes, and the emotion of a speaker is integrated into the video synthesis process, so that the voice synthesis method has good reality; a set of system based on a Web end is integrated, and a user is supported to directly log in a website, upload audio and video materials, synthesize the audio and video materials on line and produce the audio and video materials in batches quickly.

Description

A synthetic method and system of virtual video customer service robot based on generative adversarial network

technical field

The invention relates to the technical field of face video synthesis, in particular to a virtual video customer service robot synthesis method and system based on a generative confrontation network.

Background technique

Face video synthesis is an emerging and challenging problem in computer vision, and virtual video robots based on this technology are gaining more and more attention. The virtual video customer service robot includes modules such as lip shape generation, expression generation, and speech synthesis. It is expected to truly imitate the lip movements, voices and facial expressions of people when they speak.

Inspired by the successful application of deep learning in the field of computer vision, deep learning-based face video synthesis has achieved excellent performance and good visual effects. At present, some important benchmark datasets have been proposed in the field of face video synthesis, such as GRID [1], TIMIT [2] and LRW [3]. These datasets provide a large number of audio-video data pairs, which greatly promote the development of the field of face-to-video synthesis. Based on the above datasets, a large number of excellent algorithms have emerged, such as ObamaNet [4], LipGAN [5], ExprGAN [6], Wav2Lip [7], etc. Taking LipGAN as an example, it extracts audio and video features through the encoder-decoding structure of the generator in the generative adversarial network, and uses the discriminator to compare the generated video with the real video to achieve end-to-end training. The video has achieved good performance. These algorithms have played an important role in promoting the development of face video synthesis. In recent years, based on face video synthesis technology, companies such as Baidu, Sohu, and iFLYTEK have designed corresponding virtual video robots to complete simple tasks such as news broadcasts and customer service questions, which has promoted the implementation and development of strong artificial intelligence.

The prior art has the following defects and deficiencies:

However, most of the existing virtual video customer service synthesis methods and systems cannot achieve a true and reliable integrated synthesis from text to video. Specifically, it is not possible to align the lip shape with the voice well, the language of the speaker cannot be switched according to the user's needs, and the corresponding facial expressions and voice intonation cannot be generated according to the emotion of the expressed words and sentences. Although these systems have the primary functions of video customer service, they cannot better approach the habit of talking with real people, and the traces of manual processing are more obvious.

SUMMARY OF THE INVENTION

In view of the deficiencies of the prior art, the present invention proposes a method and system for synthesizing a virtual video customer service robot based on a generative adversarial network. ; The synthesis scheme allows users to realize the synthesis of various languages, arbitrary selection of customer service images, application of various scenarios, and integrate the speaker's emotion into the process of video synthesis, which has good authenticity; The system on the web side supports users to log in directly to the website, upload audio and video materials, synthesize online, and quickly produce in batches.

In order to realize the above-mentioned synthetic method and system purpose of virtual video customer service robot based on generative adversarial network, the present invention provides the following technical solutions: a synthetic method and system of virtual video customer service robot based on generative adversarial network, including a lip generator module , Expression Generator Module, Text Sentiment Analysis Module, Text-to-Speech Synthesis Module.

The method for synthesizing a virtual video customer service robot based on a generative adversarial network includes the following steps:

Step 1: Collect 1000 CCTV news broadcast videos with a duration of 15 seconds as the corresponding Chinese corpus-video dataset. The Wav2Lip and First Order Motion Model models are trained on this dataset to make them more in line with the characteristics of Chinese pronunciation as a lip generator.

Step 2: Train the ExprGAN model on the Oulu-CASIA NIR&VIS facial expression dataset as an expression generator, train a bidirectional LSTM model as a text sentiment analysis module, and call the Baidu TTS interface to synthesize speech with emotion.

Step 3: Integrate the above four modules and develop based on the web side. Use the VUE framework to build the front-end, use Python's flask and django packages to encapsulate the interface, build the back-end, use nginx for reverse proxy, and integrate a virtual video customer service robot synthesis website and platform with two schemes.

Step 4: The user selects two corresponding synthesis schemes according to their own needs.

Step 5: Log in to the website and submit the above-mentioned original materials to synthesize the facial video of the virtual customer service.

Among them, the solution described in step 1 is migration synthesis, which is more suitable for scenes with high requirements for lip alignment, and can provide clear and real face videos;

Among them, the solution described in step 2 is text synthesis, which is more suitable for large-scale commercial-level application scenarios, and can directly synthesize real lips, expressions, and voices according to the text. The synthesized video has good timing stability, rapid synthesis, and good effect. lifelike.

Further, if the user chooses the solution described in step 1, it is necessary to provide the platform server with a source video that has read the corresponding text aloud in advance and an image picture of the video customer service;

Further, if the user chooses the solution described in step 2, the platform server needs to provide any video representing the virtual customer service image and the text that the customer service will read aloud.

Among them, the Wav2Lip model specifically performs feature extraction on video and audio of consecutive frames, introduces synthesis loss, and synthesizes lip motion video with good smoothness through generative adversarial network. The First Order Motion Model model specifically does not need to use any labels or prior information to perform animation processing of pictures. That is, by training on a set of videos depicting facial features, the model can be used for lip shape transfer.

The ExprGAN model is specifically an expression editing algorithm with controllable expression intensity, which can change facial images to target expressions with various styles, and the expression intensity can also be continuously controlled. The bidirectional LSTM model specifically uses the bidirectional LSTM model to analyze text sentiment, so as to better process degree words and capture bidirectional semantic dependencies.

Further, if the user selects the scheme described in step 1, then directly generate a video with precise lip movement and natural facial expressions through the trained First OrderMotion model;

Further, if the user selects the scheme described in step 2, then the text is input into the sentiment analysis module to analyze the corresponding emotion; the audio of the corresponding voice intonation is generated by calling TTS; the video is input into the lip generator, and the audio generated by the TTS is jointly synthesized. Video with lip movement; input the above video into the expression generator, adjust the facial expression according to the analyzed emotion, and get the result.

The virtual video customer service robot synthesis system based on generative adversarial network, the device includes: a cloud server, a memory, a processor, and a computer program stored in the memory and running on the processor, the processor executing the The described integration scheme and method steps are implemented when the described program is executed.

Compared with the prior art, the present invention provides a virtual video customer service robot synthesis method and system based on a generative confrontation network, which has the following beneficial effects:

1. This method and system for synthesizing virtual video customer service robots based on generative adversarial networks proposes two solutions for synthesizing virtual video customer service robots, that is, the solution described in step 1 is to pre-record a video of reading the corresponding text aloud, and combine the The facial features are migrated to the image picture of the virtual customer service; the solution described in step 2 is to input text and the image video of the virtual customer service, and directly synthesize the virtual customer service video with emotions and real lip movements;

2. This method and system for synthesizing virtual video customer service robots based on generative adversarial networks, the proposed synthesis scheme allows users to realize the synthesis of various languages, the arbitrary selection of customer service images, the application of various scenarios, and the integration of the speaker's Emotion is integrated into the process of video synthesis, which has better authenticity and good scalability;

3. This method and system for synthesizing virtual video customer service robots based on generative adversarial networks integrates a web-based system, which supports users to directly log in to the website, upload audio and video materials, synthesize online, and quickly produce in batches.

Description of drawings

Fig. 1 is the flow chart of the synthetic method of virtual video customer service robot based on generative adversarial network of the present invention;

FIG. 2 is a schematic diagram of the overall network structure of the solution Wav2Lip described in step 2 of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Please refer to Figure 1-2, a virtual video customer service robot synthesis system based on generative adversarial network, including a lip generator module, an expression generator module, a text sentiment analysis module, and a text-to-speech synthesis module.

Example 1:

An embodiment of the present invention provides a method for synthesizing a virtual video customer service robot based on a generative adversarial network, the method comprising the following steps:

101: Use the you-get tool to collect 1,000 CCTV news broadcast videos of different characters as the corresponding Chinese corpus-video dataset, and organize them in the format of the LRS2 dataset.

Further, extract the audio from the video with the ffmpeg tool, and convert the audio file into mel blocks for network reading through the python library librosa, and crop the video into MP4 format files with a resolution of 256*256 and a duration of 15 seconds to complete the data. Preprocessing of the set.

102: Train the Wav2Lip network model on the collected Chinese dataset. The model can extract the mapping relationship between sound and lip shape through the face decoder and audio decoder, generate a synthetic lip shape, and pass the pre-trained lip shape synthesis discriminator and the visual effect discriminator jointly trained with the generator. Constantly modifying the synthetic effect as a lip generator for the scheme described in step 2.

In the specific implementation, training is performed on the pre-training model of the original network, so that the network can take into account the characteristics of Chinese pronunciation on the basis of maintaining the original performance, and improve the effect of lip synthesis.

103: Train the three models of First Order Motion, ExprGAN, and Bi-LSTM as the lip synthesizer of the above scheme 1, the expression generator of the scheme 2, and the text sentiment analysis module respectively.

104: Invoke the Baidu speech synthesis API interface and integrate the models of the two trained schemes above. Use the vue framework to build the front end, use python's flask and django packages to encapsulate the model interface, build the back end, and build a website based on the web side.

105 : The user can choose to synthesize the scheme described in step 1 and the scheme described in step 2 according to his own needs. If you choose the solution in step 1, you need to prepare a pre-read video and an image of the virtual customer service; if you choose the solution in step 2, you need to prepare the image video (or image) of the virtual customer service and the text to be read by the robot.

106: The user logs in to the website and submits the above materials, and the synthesis result can be obtained.

To sum up, the present invention proposes a method for synthesizing virtual video customer service robots based on generative adversarial networks. It allows users to realize the synthesis of different languages, arbitrary selection of customer service images, application of various scenarios, and integrates the speaker's emotions into the process of video synthesis, which has good authenticity; a set of web-based systems are integrated to support Users directly log in to the website, upload audio and video materials, and quickly synthesize them in batches.

Example 2:

The scheme in Embodiment 1 is further introduced below in conjunction with specific examples and calculation formulas, and is described in detail below:

1. Data preparation

The invention uses the you-get tool to collect 1,000 CCTV news broadcast videos of different characters as the corresponding Chinese corpus-video data set, and organizes them according to the format of the LRS2 data set. Further, the dataset is preprocessed with ffmpeg and librosa tools.

The dataset consists of corresponding audio and video. The video part contains the broadcast content of 5 different male anchors and 5 different female anchors, the frame rate is 25 fps, the resolution is all cropped to 256*256, the duration is 25 seconds, and the format is MP4; the audio part is from the video The extracted Mel block is used for the network to obtain sound information directly.

2. Model training

The present invention includes four modules: a lip shape generation module, an expression generation module, a text emotion analysis module and a speech synthesis module, the details are as follows.

(1) Lip shape generation module:

The lip shape generation module in the solution described in step 1 adopts the First Order Motion model, and does not need to use any labels or prior information for animation processing of pictures. The model was trained on a set of videos depicting facial features, and the model was then used for lip shape transfer. The specific implementation is to use a generative adversarial network approach to separate appearance information from motion information. To support the robustness of the model to complex motion, the model extracts facial keypoints and local affine transformations in the source video, and the generator network models the motion of the target object, i.e. extracts static appearance information from the source image and drives The motion information obtained in the video is combined to obtain a composite video.

The lip generation module in the solution described in step 2 adopts the Wav2Lip model, which consists of a generator and two discriminators. The generator can be divided into face information encoder, voice information encoder, and face information decoder. The facial information encoder consists of a series of skip-connected residual convolutional blocks. It takes the lip information masking of a set of random video frames R as the prior pose P, which is cascaded with R according to the number of channels as the encoder input. The encoder extracts the lip shape information in the input as a facial feature map for the subsequent decoding and reconstruction of the network; the speech encoder is encoded by a series of two-dimensional convolution blocks, extracts the speech information in the input Mel block S, and then combines the facial feature map with the facial feature map. Cascading; the facial information decoder decodes the features encoded by the above two encoders, and reconstructs a lip video that matches the audio through a series of upsampling and deconvolution operations. The specific lip reconstruction L1 loss is as follows :

Among them, Lg is the lip shape reconstructed by the generator, LG is the real image, and N is the number of input images

The lip-sync discriminator is used to penalize lip generation that is out of sync with the audio. When the generated video frames are cascaded into the pre-trained lip-sync discriminator according to the time dimension, the discriminator will discriminate the lower half of the generated face to minimize the synchronization loss, as follows:

The weight of the lip sync generator remains unchanged during the training of the GAN network, and it has a 91% accuracy rate for judging whether the lip audio is synchronized, which can well constrain the training of the generator.

The visual effects discriminator is jointly trained with the generator network, which is used to constrain distorted face generation. The discriminator D consists of a series of convolutional blocks. Each block contains a convolutional layer and a Leaky ReLU activation layer. During the discriminative training process, the network minimizes the loss Ldisc as follows:

Ultimately, the total loss of the network is

Among them, sw and sg are preset parameters, Lrecon is the reconstruction loss mentioned above, Esync is the synchronization loss mentioned above, and Lgen is the generator loss.

(2) Expression generation module:

The expression generation module consists of the ExprGAN model. ExprGAN is an expression editing algorithm with controllable expression intensity, which can change facial images to target expressions with various styles, and expression intensity can also be continuously controlled. The generator of ExprGAN consists of an encoder and a decoder, the input of the encoder is the face image, and the output of the decoder is the reconstructed image; the discriminator of ExprGAN is used to constrain the strength and authenticity of the expression. The whole network can be divided into three stages: controller learning stage, image reconstruction stage and image refinement stage. After three stages of generation, facial videos with specified expressions can be obtained.

(3) Text sentiment analysis module:

Text sentiment analysis is to analyze the sentiment tendency of sentences. The present invention uses a two-way LSTM model to analyze text sentiment, which can better process degree words and capture two-way semantic dependencies. The bidirectional LSTM model is synthesized from the forward LSTM and the backward LSTM. The LSTM model consists of the input word at time t, the cell state, the temporary cell state, the hidden layer state, the forgetting gate, the memory gate, and the output gate. The calculation process can be summarized as: gating the cell state to forget and memorize new information, so that the information useful for subsequent calculations can be transmitted, and the useless information is discarded; the hidden layer state and new input of the previous step are involved. The operation of each step determines the content of forgetting and memory in each step. By synthesizing the forward and reverse LSTMs, that is, by splicing the output results of the hidden layer states of the two LSTMs, the required judgment of emotional tendencies can be obtained. In the system of the present invention, emotions are divided into six kinds of emotions: neutral, happy, angry, sad, surprise, and fear.

(4) Speech synthesis module:

The present invention calls Baidu's TTS interface in this module. This technology can well complete Chinese speech synthesis, and prosody processing can naturally deal with text segmentation, polyphonic words and other problems, the effect is more realistic, and can serve the whole system well.

3. Model integration

The integration method of the four modules is that if the user selects the solution described in step 1, the trained FirstOrder Motion model will directly generate a video with precise lip movements and natural facial expressions; if the user chooses the solution described in step 2, the text will be Input the emotion analysis module to analyze the corresponding emotions; generate the audio of the corresponding voice intonation by calling TTS; input the video into the lip generator, and synthesize the lip motion video synchronized with the audio and picture together with the audio generated by the TTS; input the above video into the expression generator , according to the analyzed emotion, adjust the facial expression to get the result.

This model embodiment has the following three key creation points:

1. Two schemes for synthesizing virtual video customer service robots are proposed;

Technical effect: The solution described in step 1 is migration synthesis, and the synthesis effect is realistic, which is suitable for scenes with high requirements for video authenticity; the solution described in step 2 is text synthesis, which can quickly synthesize real lips according to the text in one stop. , expressions, voices, and synthetic videos have good timing stability and are more suitable for large-scale commercial-level application scenarios.

2. Propose a method to make the synthesized Chinese video have precise lip movements and natural expressions;

Technical effect: The trained model has excellent performance, the lip-sync error LSE-D is reduced from 10.33 to 6.39, the lip-sync confidence LSE-C is increased from 3.199 to 7.789, and the visual quality is improved from 3.91 to 4.12. At the same time, the model is composed of simple synthetic lips and expressions.

3. A system integrating a synthetic virtual video customer service robot.

Technical effect: Integrating the four modules into a system and building a website can well realize the one-stop synthesis of the two schemes.

To sum up, this method realizes the synthesis of virtual video customer service robots through four modules and two schemes, which can accurately drive the lip shape, naturally synthesize expressions and voices, and has a good visual effect. At the same time, the integrated system enables users to quickly mass-produce virtual video customer service robots.

Example 3:

The embodiments of the present invention can be used not only in the generation of virtual video customer service, but also in the following application scenarios.

For example, let historical figures and static pictures complete specific actions such as singing and saying holiday wishes, such as importing the question corpus in advance, the system of virtual video customer service robot can be applied to campus welcome robots, psychological counseling robots, etc., enabling students and robots to achieve face-to-face Real communication, to achieve better human-computer interaction.

Example 4:

A virtual video customer service robot synthesis system based on generative adversarial network, the system includes: a website domain name, a cloud server, a memory, a processor and a computer program stored in the memory and running on the processor, when the processor executes the program The method steps in Examples 1 and 2 were carried out.

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus.

Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, and substitutions can be made in these embodiments without departing from the principle and spirit of the invention and modifications, the scope of the present invention is defined by the appended claims and their equivalents.

Claims (11)

1. a virtual video customer service robot synthesis method and system based on generative confrontation network, it is characterized in that: described a kind of virtual video customer service robot synthesis system based on generative confrontation network, comprises lip shape generator module, expression generator module, Text sentiment analysis module, text-to-speech synthesis module. 2. a kind of virtual video customer service robot synthesis method and system based on generative confrontation network according to claim 1, is characterized in that: described a kind of virtual video customer service robot synthesis method based on generative confrontation network comprises the following steps: Step 1: Collect 1000 CCTV news broadcast videos with a duration of 15 seconds as the corresponding Chinese corpus-video data set, and train the Wav2Lip and First Order Motion Model models on this data set to make it more in line with the characteristics of Chinese pronunciation. lip generator; Step 2: Train the ExprGAN model as an expression generator on the Oulu-CASIA NIR&VIS facial expression dataset, train the bidirectional LSTM model as a text sentiment analysis module, and call the Baidu TTS interface to synthesize speech with emotion; Step 3: Integrate the above four modules, develop based on the web, use the VUE framework to build the front-end, use Python's flask and django packages to encapsulate the interface, build the back-end, use nginx for reverse proxy, and integrate a virtual system with two solutions. Video customer service robot synthesis website and platform; Step 4: The user selects the corresponding two synthesis schemes according to their own needs; Step 5: Log in to the website and submit the above-mentioned original materials to synthesize the facial video of the virtual customer service. 3. a kind of virtual video customer service robot synthesis method and system based on generative adversarial network according to claim 2 step 1, is characterized in that: the scheme described in step 1 is migration synthesis, and is more suitable for lip-shaped alignment requirements high Scene, can clear and real face video. 4. a kind of virtual video customer service robot synthesis method and system based on generative adversarial network according to claim 2, it is characterized in that: the scheme described in step 2 is text synthesis, which is more suitable for large-scale commercial application scenarios , which can directly synthesize real lips, expressions, and sounds according to the text. The synthetic video has good timing stability, rapid synthesis, and realistic effects. 5. a kind of virtual video customer service robot synthesis method and system based on generative adversarial network according to claim 2 step 1, is characterized in that: if the user selects the scheme described in step 1, it is necessary to provide a section of pre-reading corresponding to the platform server. The source video of the text and the image picture of the video customer service. 6. a kind of virtual video customer service robot synthesis method and system based on generative adversarial network according to claim 2 step 2, it is characterized in that: if the user selects the scheme described in step 2, it is necessary to provide the platform server with the image representing the virtual customer service image. Any video and text that the agent will read aloud. 7. a kind of virtual video customer service robot synthesis method and system based on generative adversarial network according to claim 2 step 1, is characterized in that: described Wav2Lip model specifically is to carry out feature extraction to the video and audio of continuous frame, introduce synthetic Loss, synthesizing a lip motion video with good smoothness through a generative adversarial network, the First OrderMotion Model model specifically does not need to use any labels or prior information to animate pictures, that is, by performing on a set of videos depicting facial features. After training, the model can be used for lip transfer. 8. a kind of virtual video customer service robot synthesis method and system based on generative adversarial network according to claim 2 step 2, it is characterized in that: described ExprGAN model specifically has the expression editing algorithm of controllable expression intensity, can be facial expression editing algorithm. The image is changed to a target expression with multiple styles, and the expression intensity can also be continuously controlled. The bidirectional LSTM model specifically uses a bidirectional LSTM model to analyze text sentiment, which is used to better process degree words and capture bidirectional semantic dependencies. 9. a kind of virtual video customer service robot synthesis method and system based on generative adversarial network according to claim 2 step 1, is characterized in that: if the user selects the scheme described in step 1, then directly generates by trained First OrderMotion model Videos with precise lip movements, natural facial expressions. 10. A kind of virtual video customer service robot synthesis method and system based on generative adversarial network according to claim 2 step 2, it is characterized in that: if the user selects the scheme described in step 2, then the text is input into the sentiment analysis module, and the result is analyzed. Corresponding emotion, generate the audio of the corresponding voice tone by calling TTS, input the video into the lip generator, synthesize the video with lip movement together with the audio generated by TTS, input the above video into the expression generator, and adjust the emotion according to the analyzed emotion. Facial expressions, get results. 11. The method and system for synthesizing a virtual video customer service robot based on a generative adversarial network according to claim 2, characterized in that: the virtual video customer service robot synthesizing system based on a generative adversarial network, the device comprises: A cloud server, a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implements the integration scheme and method steps when the processor executes the program.

Images