CN116469372A - Speech synthesis method, speech synthesis device, electronic device, and storage medium - Google Patents

Abstract

The embodiment of the application provides a voice synthesis method, a voice synthesis device, electronic equipment and a storage medium, and belongs to the technical field of financial science and technology. The method comprises the following steps: acquiring target text data and reference voice data; vectorizing the reference voice data to obtain a reference embedded voice vector; extracting features of the target text data to obtain a target text representation vector; performing style marking on the reference embedded voice vector to obtain a target style embedded vector corresponding to the reference embedded voice vector; performing voice synthesis based on the target style embedded vector and the target text representation vector to obtain synthesized spectrum data; and performing frequency spectrum conversion on the synthesized frequency spectrum data to obtain synthesized voice data. The embodiment of the application can improve the accuracy of voice synthesis.

Description

Speech synthesis method, speech synthesis device, electronic device and storage medium

technical field

The present application relates to the technical field of financial technology, and in particular to a speech synthesis method, a speech synthesis device, electronic equipment, and a storage medium.

Background technique

With the rapid development of artificial intelligence technology, intelligent voice interaction is widely used in finance, logistics, customer service and other fields. Through intelligent marketing, intelligent collection, content navigation and other functions, the service level of enterprise customer service has been improved.

At present, dialogue robots are often used in financial service scenarios such as intelligent customer service and shopping guidance to provide corresponding service support for each object. The dialogue voices used by these dialogue robots are often generated based on speech synthesis.

Most of the speech synthesis methods in the related art are based on the convolutional neural network to extract the text content information in the text data, and rely on the extracted text content information and fixed prosody templates to perform speech synthesis. This method often results in poor emotional expression ability of the synthesized speech data, which affects the accuracy of speech synthesis. Therefore, how to improve the accuracy of speech synthesis has become a technical problem to be solved urgently.

Contents of the invention

The main purpose of the embodiment of the present application is to provide a speech synthesis method, a speech synthesis device, an electronic device and a storage medium, aiming at improving the accuracy of speech synthesis.

In order to achieve the above purpose, the first aspect of the embodiment of the present application proposes a speech synthesis method, the method comprising:

Obtain target text data and reference voice data;

Carrying out vectorization processing on the reference voice data to obtain a reference embedded voice vector;

Carrying out feature extraction to the target text data to obtain a target text representation vector;

Performing style marking on the reference embedded speech vector to obtain a target style embedding vector corresponding to the reference embedded speech vector;

performing speech synthesis based on the target style embedding vector and the target text representation vector to obtain synthesized spectrum data;

Spectrum conversion is performed on the synthesized spectrum data to obtain synthesized speech data.

In some embodiments, performing style marking on the reference embedded speech vector to obtain a target style embedding vector corresponding to the reference embedded speech vector includes:

Obtaining a plurality of preset style tag vectors, wherein the style tag vector is a vector representation corresponding to a preset style tag;

performing attention calculation on each of the style label vectors and the reference embedded speech vector, to obtain the style similarity between each of the style label vectors and the reference embedded speech vector;

Obtaining a style label weight of each style label vector based on a plurality of style similarities;

The style label vectors are weighted and summed based on the style label weights to obtain the target style embedding vector.

In some embodiments, the attention calculation is performed on each of the style label vectors and the reference embedded speech vector to obtain the style similarity between each of the style label vectors and the reference embedded speech vector, including:

performing matrix multiplication on each of the style tag vectors and the reference embedded speech vector to obtain a query vector, a key vector, and a value vector corresponding to each of the style tag vectors;

Attention calculation is performed on the query vector, the key vector, and the value vector based on a preset function to obtain a style similarity between the style label vector and the reference embedded speech vector.

In some embodiments, the obtaining the style label weight of each style label vector based on the multiple style similarities includes:

summing the multiple style similarities to obtain a comprehensive style similarity;

The style similarity of each style label vector is divided by the comprehensive style similarity to obtain the style label weight of the style label vector.

In some embodiments, performing speech synthesis based on the target style embedding vector and the target text representation vector to obtain synthesized spectral data includes:

splicing the target style embedding vector and the target text representation vector to obtain a combined embedding vector;

performing feature alignment on the combined embedding vector based on an attention mechanism to obtain a target phoneme vector;

Decoding the target phoneme vector to obtain the synthesized spectrum data.

In some embodiments, the attention-based mechanism performs feature alignment on the combined embedding vector to obtain a target phoneme vector, including:

performing time prediction on the combined embedding vector based on a preset duration prediction model to obtain phoneme duration;

The phoneme length adjustment is performed on the combined embedding vector based on the attention mechanism and the phoneme duration to obtain the target phoneme vector.

In some embodiments, performing spectral conversion on the synthesized spectral data to obtain synthesized speech data includes:

Inputting the synthesized spectral data into a preset vocoder, wherein the vocoder includes a deconvolution layer and a multi-receptive field fusion layer;

performing upsampling processing on the synthesized spectral data based on the deconvolution layer to obtain target spectral data;

performing multi-scale feature fusion on the target spectrum data based on the multi-receptive field fusion layer to obtain the synthesized speech data.

In order to achieve the above purpose, the second aspect of the embodiment of the present application proposes a speech synthesis device, the device includes:

A data acquisition module, configured to acquire target text data and reference voice data;

A vectorization module, configured to perform vectorization processing on the reference speech data to obtain a reference embedded speech vector;

A feature extraction module, configured to perform feature extraction on the target text data to obtain a target text representation vector;

A style labeling module, configured to perform style labeling on the reference embedded speech vector to obtain a target style embedding vector corresponding to the reference embedded speech vector;

Speech synthesis module, for performing speech synthesis based on the target style embedding vector and the target text representation vector, to obtain synthesized spectrum data;

The spectrum conversion module is configured to perform spectrum conversion on the synthesized spectrum data to obtain synthesized speech data.

To achieve the above object, the third aspect of the embodiments of the present application provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect above when executing the computer program.

To achieve the above object, the fourth aspect of the embodiments of the present application provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the above-mentioned first aspect is implemented.

The speech synthesis method, speech synthesis device, electronic equipment, and storage medium proposed by the present application obtain target text data and reference speech data; perform vectorization processing on the reference speech data to obtain reference embedded speech vectors; perform feature extraction on the target text data to obtain target text representation vectors, and can obtain target text representation vectors representing the semantic content of the text, so that speech synthesis can be performed using the target text representation vectors. Furthermore, the reference embedded speech vector is style-marked to obtain the target style embedding vector corresponding to the reference embedded speech vector, which can realize the control of speech prosodic style, so that the target style embedding vector can be used for speech synthesis, and improve the accuracy of speech synthesis emotion and speech synthesis effect. Furthermore, speech synthesis is performed based on the target style embedding vector and the target text representation vector to obtain synthetic spectral data, which can better improve the data quality of the synthetic spectral data. Finally, spectrum conversion is performed on the synthesized spectrum data to obtain synthesized speech data, and the synthesized speech data in the form of a waveform can be obtained more conveniently. The synthesized speech data contains both the text content characteristics of the target text data and the style characteristics of the reference speech data. It has better emotional expression ability and improves the accuracy of speech synthesis. In turn, in the process of intelligent dialogues such as insurance products and wealth management products, the synthetic speech expressed by the dialogue robot can be more suitable for the dialogue style preference of the dialogue object. By using the dialogue style and dialogue style that the dialogue object is more interested in for conversation communication, the dialogue quality and effectiveness can be improved, and intelligent voice dialogue can be realized. service, improve customer service quality and customer satisfaction.

Description of drawings

Fig. 1 is the flowchart of the speech synthesis method that the embodiment of the present application provides;

Fig. 2 is the flowchart of step S104 in Fig. 1;

Fig. 3 is the flowchart of step S202 in Fig. 2;

Fig. 4 is the flowchart of step S203 in Fig. 2;

Fig. 5 is the flowchart of step S105 in Fig. 1;

Fig. 6 is the flowchart of step S502 in Fig. 5;

Fig. 7 is the flowchart of step S106 in Fig. 1;

FIG. 8 is a schematic structural diagram of a speech synthesis device provided by an embodiment of the present application;

FIG. 9 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, not to limit the present application.

It should be noted that although the functional modules are divided in the schematic diagram of the device, and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the flow chart. The terms "first", "second" and the like in the specification and claims and the above drawings are used to distinguish similar objects, and not necessarily used to describe a specific sequence or sequence.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

First, analyze some nouns involved in this application:

Artificial intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, artificial intelligence attempts to understand the essence of intelligence, and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

Natural language processing (natural language processing, NLP): NLP uses computers to process, understand and use human languages (such as Chinese, English, etc.). NLP belongs to a branch of artificial intelligence and is an interdisciplinary subject of computer science and linguistics. Natural language processing includes syntax analysis, semantic analysis, text understanding, etc. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

Information Extraction (Information Extraction, NER): A text processing technology that extracts specified types of factual information such as entities, relationships, and events from natural language texts, and forms structured data output. Information extraction is a technique to extract specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters, and text information is composed of some small specific units, such as words, words, phrases, sentences, paragraphs, or combinations of these specific units. Extracting noun phrases, personal names, and place names in text data is all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

Mel-Frequency Cepstral Coefficients (MFCC): It is a set of key coefficients used to establish Mel-Frequency Cepstral Coefficients. A set of cepstrums sufficient to represent the music signal can be obtained from the segments in the music signal, and the Mel cepstral coefficients are the cepstrums (that is, the frequency spectrum) derived from the cepstrums. Different from the general cepstrum, the biggest feature of the Mel cepstrum is that the frequency bands on the Mel cepstrum are evenly distributed on the Mel scale. That is to say, compared with the generally seen, linear cepstrum representation method, such frequency bands will be closer to the human nonlinear auditory system (Audio System). For example: In audio compression technology, Mel cepstrum is often used for processing.

Vocoder (vocoder): that is, a speech analysis and synthesis system of a certain model of the speech signal. Only model parameters are used in transmission. It is a speech signal codec that uses model parameter estimation and speech synthesis technology when encoding and decoding. It is also a coder and decoder that analyzes and synthesizes speech.

Multi-head attention: (multi-head attention) is to use multiple queries to calculate and select multiple information from input information in parallel. Each attention focuses on a different part of the input information. Hard attention, i.e. the expectation of all input information based on the attention distribution.

Embedding: embedding is a kind of vector representation, which refers to using a low-dimensional vector to represent an object, which can be a word, a commodity, or a movie, etc.; the nature of this embedding vector is to enable objects corresponding to vectors with similar distances to have similar meanings. Embedding is essentially a mapping from semantic space to vector space, while maintaining the relationship of the original sample in the semantic space as much as possible in the vector space. Embedding can encode an object with a low-dimensional vector and retain its meaning. It is often used in machine learning. In the process of building a machine learning model, the object is encoded as a low-dimensional dense vector and then passed to DNN to improve efficiency.

Encoder: Converts the input sequence into a fixed-length vector.

Decoding (Decoder): It is to convert the previously generated fixed vector into an output sequence; where the input sequence can be text, voice, image, video; the output sequence can be text, image.

BERT (Bidirectional Encoder Representation from Transformers) model: The BERT model further increases the generalization ability of the word vector model, fully describes character-level, word-level, sentence-level and even inter-sentence relationship features, and is built based on Transformer. There are three kinds of embeddings in BERT, namely Token Embedding, SegmentEmbedding, and Position Embedding; among them, Token Embeddings is a word vector, and the first word is a CLS mark, which can be used for subsequent classification tasks; Segment Embeddings are used to distinguish two kinds of sentences, because pre-training is not only for LM but also for classification tasks with two sentences as input; learned in training. However, BERT directly trains a position embedding to retain position information. Each position randomly initializes a vector, joins the model training, and finally obtains an embedding containing position information. In the final combination of position embedding and word embedding, BERT chooses to splice directly. Phoneme (Phone): It is the smallest unit of speech divided according to the natural properties of speech. It is analyzed according to the pronunciation actions in syllables. An action constitutes a phoneme.

Softmax function: The Softmax function is a normalized exponential function.

Speech synthesis refers to synthesizing understandable and natural speech from text, also known as Text-To-Speech (TTS). Speech synthesis systems are widely used in various scenarios in life, including voice dialogue systems; intelligent voice assistants; telephone information query systems; auxiliary applications such as car navigation and audio e-books; language learning; real-time information broadcasting systems at airports and stations;

With the rapid development of artificial intelligence technology, intelligent voice interaction is widely used in finance, logistics, customer service and other fields. Through intelligent marketing, intelligent collection, content navigation and other functions, the service level of enterprise customer service has been improved.

At present, dialogue robots are often used in financial service scenarios such as intelligent customer service and shopping guidance to provide corresponding service support for each object. The dialogue voices used by these dialogue robots are often generated based on speech synthesis.

Taking the insurance service robot as an example, it is often necessary to integrate the description text of the insurance product with the speaking style of the fixed object to generate a voice describing the insurance product by the fixed object. When the insurance service robot has a conversation with some interested objects, it will automatically call this description voice to introduce insurance products for the objects.

Most of the current speech synthesis methods are based on the convolutional neural network to extract the text content information in the text data, and rely on the extracted text content information and fixed prosody templates for speech synthesis. This method often results in poor emotional expression ability of the synthesized speech data, which affects the accuracy of speech synthesis. Therefore, how to improve the accuracy of speech synthesis has become a technical problem to be solved urgently.

Based on this, an embodiment of the present application provides a speech synthesis method, a speech synthesis device, an electronic device, and a storage medium, aiming at improving the accuracy of speech synthesis.

The speech synthesis method, speech synthesis device, electronic equipment, and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the speech synthesis method in the embodiments of the present application is described.

The embodiments of the present application may acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning/deep learning.

The speech synthesis method provided in the embodiment of the present application relates to the technical field of artificial intelligence. The speech synthesis method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running on the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, or can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms;

It should be noted that, in each specific implementation of this application, when it comes to relevant processing based on user information, user voice data, user behavior data, user historical data, user location information and other data related to user identity or characteristics, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of this application needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or separate consent through a pop-up window or jump to a confirmation page, etc., and obtain the necessary user-related data for the normal operation of the application embodiment after the user's separate permission or separate consent is clearly obtained.

The application can be used in numerous general purpose or special purpose computer system environments or configurations. Examples: personal computers, server computers, handheld or portable devices, tablet-type devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. This application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

Fig. 1 is an optional flow chart of the speech synthesis method provided by the embodiment of the present application. The method in Fig. 1 may include but not limited to steps S101 to S106.

Step S101, acquiring target text data and reference voice data;

Step S102, vectorizing the reference speech data to obtain a reference embedded speech vector;

Step S103, performing feature extraction on the target text data to obtain a target text representation vector;

Step S104, performing style marking on the reference embedded speech vector to obtain a target style embedding vector corresponding to the reference embedded speech vector;

Step S105, performing speech synthesis based on the target style embedding vector and the target text representation vector to obtain synthesized spectrum data;

Step S106, performing spectrum conversion on the synthesized spectrum data to obtain synthesized speech data.

From step S101 to step S106 shown in the embodiment of the present application, by acquiring target text data and reference voice data; performing vectorization processing on the reference voice data to obtain reference embedded voice vectors; performing feature extraction on the target text data to obtain the target text representation vector, the target text representation vector representing the semantic content of the text can be obtained, so that speech synthesis can be performed using the target text representation vector. Furthermore, the reference embedded speech vector is style-marked to obtain the target style embedding vector corresponding to the reference embedded speech vector, which can realize the control of speech prosodic style, so that the target style embedding vector can be used for speech synthesis, and improve the accuracy of speech synthesis emotion and speech synthesis effect. Furthermore, speech synthesis is performed based on the target style embedding vector and the target text representation vector to obtain synthetic spectral data, which can better improve the data quality of the synthetic spectral data. Finally, spectrum conversion is performed on the synthesized spectrum data to obtain synthesized voice data, and the synthesized voice data in the form of a waveform can be obtained more conveniently. The synthesized voice data contains both the text content characteristics of the target text data and the style characteristics of the reference voice data, which has better emotional expression ability and improves the accuracy of speech synthesis.

In step S101 of some embodiments, the target text data may be obtained from public datasets, or the target text data to be processed may be obtained from existing text databases or network platforms, without limitation. For example, the public dataset can be THCHS30 dataset or LJSpeech dataset, etc.

It should be noted that the target text data can be text data containing proper nouns in the financial field, financial business template vocabulary, product descriptions of insurance products, product descriptions of wealth management products, and common dialogue terms in the financial field.

At the same time, by writing a web crawler, setting up the data source and crawling data in a targeted manner, the reference voice data of the reference speaker can be obtained. The data source can be various types of network platforms, social media, or certain specific audio databases. Wherein, the reference speech data may be an audio signal, and the audio signal may be composed of phonemes, where a phoneme is the smallest unit or the smallest speech segment constituting a syllable. The reference speech data and the target text data can be obtained more conveniently through the above method, and the data acquisition efficiency is improved.

In step S102 of some embodiments, the reference speech data can be input into a preset reference encoder, and the reference speech data is compressed into a variable-length prosodic audio signal by the reference encoder, thereby realizing vectorization processing of the reference speech data, converting the reference speech data into a fixed-length speech vector, and obtaining a reference embedded speech vector, wherein the reference embedded speech vector includes speech prosody feature information of the reference speech data. This method can more conveniently convert the reference speech data from the time-domain signal to the frequency-domain feature, and extract the prosodic feature information in the reference speech data, so that the prosodic feature information of the reference speech data can be integrated into the synthesized speech in the subsequent speech synthesis process, thereby improving the accuracy of speech synthesis.

In step S103 of some embodiments, the target text data may be input into a preset text model, and the preset text model may be a text encoding model such as a Bert model, without limitation. The feature extraction of the target text data is carried out through the preset text model, the text embedding representation corresponding to each text character in the target text data is obtained, and all the text embedding representations corresponding to the target text data are spliced to obtain the target text representation vector. For example, firstly, the text model is used to segment the target text data to obtain multiple text characters, and then the text embedding representation of each text character is queried through the preset character dictionary, and all the text embedding representations are vector-connected to obtain the target text representation vector corresponding to the target text data. This method can more accurately realize the conversion of target text data from data space to vector space, obtain the target text representation vector representing the semantic content of the text, make it possible to use the target text representation vector for speech synthesis, and obtain the synthesized speech corresponding to the target text data, which helps to improve the accuracy of speech synthesis.

Referring to FIG. 2, in some embodiments, step S104 may include but not limited to include steps S201 to S204:

Step S201, obtaining a plurality of preset style label vectors, wherein the style label vector is a vector representation corresponding to the preset style label;

Step S202, perform attention calculation on each style label vector and the reference embedded speech vector, and obtain the style similarity between each style label vector and the reference embedded speech vector;

Step S203, based on multiple style similarities, obtain the style label weight of each style label vector;

Step S204, weighting and summing the style label vectors based on the style label weights to obtain the target style embedding vector.

In step S201 of some embodiments, a plurality of preset style tags can be extracted from a preset tag database, and the extracted style tags are encoded to obtain a vector representation of each style tag, thereby obtaining a plurality of style tag vectors. Among them, different style tags contain different prosody styles. For example, the style tags include various types of prosody styles such as deep voice, high voice, and rapid voice. There is no limitation. The preset label database can be constructed based on expert experience and other reasoning.

In step S202 of some embodiments, a multi-head attention mechanism can be introduced to calculate the style similarity between each style tag vector and the reference embedded speech vector, and the multi-head attention mechanism is used to project the reference embedded speech vector onto different style tag vectors to obtain the proportion of the reference embedded speech vector on different style tag vectors, and determine the style tag weight of each style tag vector according to the proportion. For example, first, matrix multiplication can be performed on each style label vector and the reference embedded speech vector, and then the attention calculation can be performed on the query vector, key vector, and value vector corresponding to each style label vector based on a preset function (such as softmax function, etc.), so as to obtain the style similarity between each style label vector and the reference embedded speech vector.

In step S203 of some embodiments, multiple style similarities can be summed to obtain a comprehensive style similarity, and then the style similarity of each style label vector and the comprehensive style similarity are divided to obtain the style label weight of each style label vector.

In step S204 of some embodiments, the style label vectors may be weighted and summed according to the style label weights, so as to implement style labeling on the reference embedded speech vector, and obtain a target style embedding vector corresponding to the reference embedded speech vector. For example, the style label vector includes style label vector A, style label vector B and style label vector C, and the style label weights of these three style label vectors are 0.1, 0.3 and 0.6 respectively, then the target style embedding vector M is M=0.1*A+0.3*B+0.6*C. The target style embedding vector contains prosodic style information of the reference speech data.

Through the above steps S201 to S204, it is possible to more conveniently determine the proportion of the reference embedded speech vector on different style tag vectors, determine the style tag weight of each style tag vector according to the ratio, and generate a weighted style embedding representation of the reference embedded speech vector based on the different style tag weights. The style tag weighting can be used to represent the spatial characteristics of speech prosody, and the control of speech rhythm can be realized, so that the target style embedding vector can be used for speech synthesis, improving the accuracy of speech synthesis emotion and speech synthesis effect.

Referring to FIG. 3, in some embodiments, step S202 may include but not limited to include steps S301 to S302:

Step S301, performing matrix multiplication on each style label vector and the reference embedded speech vector to obtain the query vector, key vector, and value vector corresponding to each style label vector;

Step S302, perform attention calculation on the query vector, key vector, and value vector based on a preset function, and obtain the style similarity between the style label vector and the reference embedded speech vector.

In step S301 of some embodiments, the multi-head attention mechanism can be used to perform matrix multiplication on each style label vector and the reference embedded speech vector, and calculate the query vector, key vector, and value vector of each style label vector. The key vector can be expressed as K=N*W1, the value vector can be expressed as V=N*W2, and the query vector can be expressed as Q=N*W3, where N is the style label vector, and W1, W2, and W3 are trainable parameters.

In step S302 of some embodiments, the preset function can be an activation function such as a softmax function. Taking the softmax function as an example, the softmax function is used to perform attention calculation on the query vector, key vector, and value vector to obtain the style similarity Z between the style label vector and the reference embedded speech vector. The style similarity Z can be expressed as shown in formula (1), where d is the feature dimension of the style label vector, and T represents the transposition operation on the key vector K:

Through the above steps S301 to S302, the proximity of the reference embedded speech vector to different style label vectors can be determined more conveniently, and the style similarity can be obtained, so that the prosodic style bias of the reference embedded speech vector can be judged based on the style similarity, and the accuracy of prosodic style configuration can be improved.

Referring to FIG. 4, in some embodiments, step S203 may include but not limited to include steps S401 to S402:

Step S401, performing a summation process on multiple style similarities to obtain a comprehensive style similarity;

In step S402, the style similarity of each style label vector is divided by the comprehensive style similarity to obtain the style label weight of the style label vector.

In step S401 of some embodiments, a sum function or other statistical functions or statistical tools may be used to sum the multiple style similarities to obtain a comprehensive style similarity.

In step S402 of some embodiments, the style similarity of each style label vector can be divided by the comprehensive style similarity to obtain the style label weight of the style label vector.

For example, the style tag vector includes style tag vector A, style tag vector B, and style tag vector C, and the style similarities between these three style tag vectors and the reference speech embedding vector are 0.37, 0.66, and 0.8 respectively, then the multiple style similarities are summed to obtain a comprehensive style similarity, that is, the comprehensive style similarity is 0.37+0.66+0.8=1.83. The style tag weight of style tag vector A is 0.37/1.83=0.2, the style tag weight of style tag vector B is 0.66/1.83=0.36, and the style tag weight of style tag vector C is 0.8/1.83=0.44.

Through the above steps S401 to S402, it is possible to more conveniently determine the proportion of the reference embedded speech vector on different style label vectors, determine the style label weight of each style label vector according to the proportion, and generate a weighted style embedding representation of the reference embedded speech vector based on the different style label weights, so as to improve the accuracy of prosody control in speech synthesis.

Referring to FIG. 5, in some embodiments, step S105 may include but not limited to include steps S501 to S503:

Step S501, splicing the target style embedding vector and the target text representation vector to obtain a combined embedding vector;

Step S502, performing feature alignment on the combined embedding vector based on the attention mechanism to obtain the target phoneme vector;

Step S503, decoding the target phoneme vector to obtain synthesized spectrum data.

In step S501 of some embodiments, when splicing the target style embedding vector and the target text representation vector, the target style embedding vector and the target text representation vector may be directly vector-connected to obtain a combined embedding vector.

In step S502 of some embodiments, first, a preset duration prediction model may be used to perform time prediction on the combined embedding vector to obtain the phoneme duration. And the attention mechanism and phoneme duration are used to adjust the phoneme length of the combined embedding vector to obtain the target phoneme vector. It should be noted that this process is mainly for the alignment of phonemes and Mel cepstrum frames. Since the length of a phoneme sequence is usually smaller than the length of its Mel cepstrum sequence. Therefore, it is necessary to calculate the length of the Mel cepstrum sequence that aligns each phoneme, which is the phoneme duration. Based on the length regulator and phoneme duration, the phoneme sequence can be tiled more conveniently, so that the phoneme sequence matches the length of the Mel cepstrum sequence.

In some other embodiments, the phoneme duration can also be extended or shortened proportionally to realize the control of the sound speed of the synthesized speech during the speech synthesis process. Alternatively, the prosody of the synthesized speech can also be adjusted by adjusting the duration of the space characters that combine the embedding vectors to control the length of pauses between words in the synthesized speech.

In step S503 of some embodiments, the target phoneme vector may be decoded by a decoder to convert the target phoneme vector into a form of Mel cepstrum sequence to obtain synthesized spectrum data.

Through the above steps S501 to S503, speech synthesis can be performed based on the text content features of the target text data and the prosodic style features of the reference speech data, so that the synthesized spectral data contains text content information and prosodic style features that meet the requirements, which can better improve the data quality of the synthesized spectral data, thereby synthesizing speech data containing the target prosody, and improving the accuracy of speech synthesis.

Referring to FIG. 6, in some embodiments, step S502 includes but is not limited to steps S601 to S602:

Step S601, performing time prediction on the combined embedding vector based on the preset duration prediction model to obtain the phoneme duration;

Step S602, based on the attention mechanism and the duration of the phoneme, the phoneme length is adjusted on the combined embedding vector to obtain the target phoneme vector.

In step S601 of some embodiments, the preset duration prediction model includes a two-layer one-dimensional convolutional network and a linear layer, the one-dimensional convolutional network is used to extract the spectrum-time feature information in the combined embedded vector, and the linear layer is mainly used to output scalars to predict the duration of phonemes. The duration prediction model may use a mean square error function (MSE) as a loss function. The one-dimensional convolutional network of the duration prediction model is used to combine the spectral time feature information embedded in the vector to obtain the spectral time feature, and then use the prediction function (such as softmax function, sigmiod function, etc.) in the linear layer to predict the spectral time feature to obtain the corresponding time length of the spectral time feature, and use the time length as the phoneme duration.

In step S602 of some embodiments, when adjusting the phoneme length of the combined embedding vector based on the attention mechanism and phoneme duration, the phoneme duration is used as the length of the combined embedding vector to align the Mel cepstrum sequence, and the attention mechanism is used to perform phoneme tiling on the phoneme sequence in the combined embedding vector, so that the phoneme sequence of the combined embedding vector can be consistent with the length of the Mel cepstrum sequence, and the target phoneme vector is obtained.

Through the above steps S601 to S602, the phoneme and frame alignment of the combined embedding vector can be realized more conveniently, so that the phoneme sequence of the combined embedding vector can match the length of the Mel cepstrum sequence to be synthesized, and the prosodic rationality of the synthesized speech and the speech quality of the synthesized speech can be improved.

Referring to FIG. 7, in some embodiments, step S106 may include but not limited to include steps S701 to S702:

Step S701, input the synthesized spectral data into a preset vocoder, wherein the vocoder includes a deconvolution layer and a multi-receptive field fusion layer;

Step S702, performing upsampling processing on the synthesized spectrum data based on the deconvolution layer to obtain the target spectrum data;

Step S703, performing multi-scale feature fusion on the target spectrum data based on the multi-receptive field fusion layer to obtain synthesized speech data.

In step S701 of some embodiments, a preset computer program or script program can be used to input the synthesized spectral data into a preset vocoder, wherein the vocoder can be HiFi-Gan or MelGan, etc., without limitation, the vocoder includes a deconvolution layer and a multi-receptive field fusion layer, and the vocoder is used to convert the synthesized spectrum data in the form of a Mel cepstrum sequence into synthesized voice data in the form of a waveform.

In step S702 of some embodiments, upsampling processing is performed on the synthesized spectrum data based on the deconvolution layer to implement convolution and transposition of the synthesized spectrum data, and obtain target spectrum data with richer spectrum feature content.

In step S703 of some embodiments, the multi-receptive field fusion layer includes a plurality of residual blocks. When performing multi-scale feature fusion on the target spectral data based on the multi-receptive field fusion layer, each residual block can be used to reconstruct the features of the target spectral data to obtain speech waveform features of multiple scales, and fuse the speech waveform features of all scales to obtain synthesized speech data.

In a specific example, the synthesized voice data is a description voice about insurance products and wealth management products, including the speaking style and emotion of a certain animation character. Dialogue robots can attract potential objects with the unique speaking style and emotion of animated characters by synthesizing voice data, making potential objects more interested in the insurance products or financial products recommended by the synthetic voice data.

Through the above steps S701 to S703, the synthesized target speech data can simultaneously contain the text content features of the target text data and the prosody features of the reference speech data, thereby effectively improving the accuracy of speech synthesis.

The speech synthesis method of the embodiment of the present application obtains target text data and reference speech data; vectorizes the reference speech data to obtain a reference embedded speech vector; performs feature extraction on the target text data to obtain a target text representation vector, and can obtain a target text representation vector representing the semantic content of the text, so that speech synthesis can be performed using the target text representation vector. Furthermore, the reference embedded speech vector is style-marked to obtain the target style embedding vector corresponding to the reference embedded speech vector, which can realize the control of speech prosodic style, so that the target style embedding vector can be used for speech synthesis, and improve the accuracy of speech synthesis emotion and speech synthesis effect. Furthermore, speech synthesis is performed based on the target style embedding vector and the target text representation vector to obtain synthetic spectral data, which can better improve the data quality of the synthetic spectral data. Finally, spectrum conversion is performed on the synthesized spectrum data to obtain synthesized speech data, and the synthesized speech data in the form of a waveform can be obtained more conveniently. The synthesized speech data contains both the text content characteristics of the target text data and the style characteristics of the reference speech data. It has better emotional expression ability and improves the accuracy of speech synthesis. In turn, in the process of intelligent dialogues such as insurance products and wealth management products, the synthetic speech expressed by the dialogue robot can be more suitable for the dialogue style preference of the dialogue object. By using the dialogue style and dialogue style that the dialogue object is more interested in for conversation communication, the dialogue quality and effectiveness can be improved, and intelligent voice dialogue can be realized. services, improve customer service quality and customer satisfaction, and increase the transaction rate of insurance products and wealth management products.

Please refer to FIG. 8, the embodiment of the present application also provides a speech synthesis device, which can realize the above speech synthesis method, and the device includes:

Data acquiring module 801, for acquiring target text data and reference voice data;

A vectorization module 802, configured to vectorize the reference voice data to obtain a reference embedded voice vector;

Feature extraction module 803, is used for carrying out feature extraction to target text data, obtains target text representation vector;

The style marking module 804 is used to carry out style marking on the reference embedded speech vector, and obtain the target style embedding vector corresponding to the reference embedded speech vector;

Speech synthesis module 805, for carrying out speech synthesis based on the target style embedding vector and the target text representation vector, to obtain synthetic spectral data;

The spectrum conversion module 806 is configured to perform spectrum conversion on the synthesized spectrum data to obtain synthesized speech data.

The specific implementation manner of the speech synthesis device is basically the same as the specific embodiment of the above speech synthesis method, and will not be repeated here.

The embodiment of the present application also provides an electronic device, the electronic device includes: a memory, a processor, a program stored in the memory and operable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the above speech synthesis method is implemented when the program is executed by the processor. The electronic device may be any intelligent terminal including a tablet computer, a vehicle-mounted computer, and the like.

Please refer to FIG. 9. FIG. 9 illustrates a hardware structure of an electronic device in another embodiment. The electronic device includes:

The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit, central processing unit), microprocessor, application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application;

The memory 902 may be implemented in the form of a read-only memory (ReadOnlyMemory, ROM), a static storage device, a dynamic storage device, or a random access memory (RandomAccessMemory, RAM). The memory 902 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 902, and the processor 901 invokes and executes the speech synthesis method of the embodiments of the present application;

The input/output interface 903 is used to realize information input and output;

The communication interface 904 is used to realize the communication and interaction between the device and other devices, and the communication can be realized through a wired method (such as USB, network cable, etc.), or can be realized through a wireless method (such as a mobile network, WIFI, Bluetooth, etc.);

bus 905, for transferring information between various components of the device (such as processor 901, memory 902, input/output interface 903 and communication interface 904);

The processor 901 , the memory 902 , the input/output interface 903 and the communication interface 904 are connected to each other within the device through the bus 905 .

The embodiment of the present application also provides a computer-readable storage medium, where one or more programs are stored in the computer-readable storage medium, and the one or more programs can be executed by one or more processors, so as to implement the above speech synthesis method.

As a non-transitory computer-readable storage medium, memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes memory located remotely from the processor, and these remote memories may be connected to the processor via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The speech synthesis method, speech synthesis device, electronic equipment, and computer-readable storage medium provided by the embodiments of the present application obtain target text data and reference speech data; perform vectorization processing on the reference speech data to obtain reference embedded speech vectors; perform feature extraction on the target text data to obtain target text representation vectors, and can obtain target text representation vectors representing the semantic content of the text, so that speech synthesis can be performed using the target text representation vectors. Furthermore, the reference embedded speech vector is style-marked to obtain the target style embedding vector corresponding to the reference embedded speech vector, which can realize the control of speech prosodic style, so that the target style embedding vector can be used for speech synthesis, and improve the accuracy of speech synthesis emotion and speech synthesis effect. Furthermore, speech synthesis is performed based on the target style embedding vector and the target text representation vector to obtain synthetic spectral data, which can better improve the data quality of the synthetic spectral data. Finally, spectrum conversion is performed on the synthesized spectrum data to obtain synthesized speech data, and the synthesized speech data in the form of a waveform can be obtained more conveniently. The synthesized speech data contains both the text content characteristics of the target text data and the style characteristics of the reference speech data. It has better emotional expression ability and improves the accuracy of speech synthesis. In turn, in the process of intelligent dialogues such as insurance products and wealth management products, the synthetic speech expressed by the dialogue robot can be more suitable for the dialogue style preference of the dialogue object. By using the dialogue style and dialogue style that the dialogue object is more interested in for conversation communication, the dialogue quality and effectiveness can be improved, and intelligent voice dialogue can be realized. services, improve customer service quality and customer satisfaction, and increase the transaction rate of insurance products and wealth management products.

The embodiments described in the embodiments of the present application are to illustrate the technical solutions of the embodiments of the present application more clearly, and do not constitute a limitation to the technical solutions provided by the embodiments of the present application. Those skilled in the art know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

Those skilled in the art can understand that the technical solutions shown in FIGS. 1-7 do not limit the embodiments of the present application, and may include more or fewer steps than those shown in the illustrations, or combine some steps, or different steps.

The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, the functional modules/units in the system, and the device can be implemented as software, firmware, hardware, and an appropriate combination thereof.

The terms "first", "second", "third", "fourth" and the like (if any) in the description of the present application and the above drawings are used to distinguish similar objects and not necessarily to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not expressly listed or inherent to the process, method, product or device.

It should be understood that in this application, "at least one (item)" means one or more, and "multiple" means two or more. "And/or" is used to describe the association relationship of associated objects, indicating that there can be three types of relationships, for example, "A and/or B" can mean: there are only A, only B, and both A and B, where A and B can be singular or plural. The character "/" generally indicates that the contextual objects are an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one item (piece) of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c can be single or multiple.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, the computer software product is stored in a storage medium, and includes multiple instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method in each embodiment of the application. The above-mentioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM for short), random access memory (Random Access Memory, RAM for short), magnetic disk or optical disk, and various media that can store programs.

The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings, which does not limit the scope of rights of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall fall within the scope of rights of the embodiments of the present application.

Claims (10)

1. A method of speech synthesis, the method comprising:

acquiring target text data and reference voice data;

vectorizing the reference voice data to obtain a reference embedded voice vector;

extracting features of the target text data to obtain a target text representation vector;

performing style marking on the reference embedded voice vector to obtain a target style embedded vector corresponding to the reference embedded voice vector;

performing voice synthesis based on the target style embedded vector and the target text representation vector to obtain synthesized spectrum data;

and performing frequency spectrum conversion on the synthesized frequency spectrum data to obtain synthesized voice data.

2. The method of claim 1, wherein the performing style marking on the reference embedded speech vector to obtain a target style embedded vector corresponding to the reference embedded speech vector comprises:

acquiring a plurality of preset style tag vectors, wherein the style tag vectors are vector representations corresponding to the preset style tags;

performing attention calculation on each style tag vector and each reference embedded voice vector to obtain style similarity between each style tag vector and each reference embedded voice vector;

Based on the style similarity, style tag weight of each style tag vector is obtained;

and carrying out weighted summation on the style tag vector based on the style tag weight to obtain the target style embedded vector.

3. The method of speech synthesis according to claim 2, wherein performing attention computation on each of the style tag vectors and the reference embedded speech vectors to obtain style similarities between each of the style tag vectors and the reference embedded speech vectors comprises:

performing matrix multiplication on each style tag vector and the reference embedded voice vector to obtain a query vector, a key vector and a value vector corresponding to each style tag vector;

and performing attention calculation on the query vector, the key vector and the value vector based on a preset function to obtain the style similarity between the style tag vector and the reference embedded voice vector.

4. The method of claim 2, wherein the deriving style tag weights for each of the style tag vectors based on the plurality of style similarities comprises:

Summing the style similarities to obtain comprehensive style similarity;

and dividing the style similarity of each style tag vector and the comprehensive style similarity to obtain the style tag weight of the style tag vector.

5. The method according to claim 1, wherein the performing speech synthesis based on the target style embedding vector and the target text representation vector to obtain synthesized spectrum data includes:

performing splicing processing on the target style embedded vector and the target text representation vector to obtain a combined embedded vector;

performing feature alignment on the combined embedded vector based on an attention mechanism to obtain a target phoneme vector;

and decoding the target phoneme vector to obtain the synthesized spectrum data.

6. The method of claim 5, wherein feature alignment of the combined embedded vectors based on an attention mechanism to obtain a target phoneme vector comprises:

performing time prediction on the combined embedded vector based on a preset duration prediction model to obtain a phoneme duration;

And carrying out phoneme length adjustment on the combined embedded vector based on the attention mechanism and the phoneme duration to obtain the target phoneme vector.

7. The method according to any one of claims 1 to 6, wherein the performing spectral conversion on the synthesized spectral data to obtain synthesized speech data includes:

inputting the synthesized spectrum data into a preset vocoder, wherein the vocoder comprises a deconvolution layer and a multi-receptive field fusion layer;

up-sampling the synthesized spectrum data based on the deconvolution layer to obtain target spectrum data;

and carrying out multi-scale feature fusion on the target frequency spectrum data based on the multi-receptive field fusion layer to obtain the synthesized voice data.

8. A speech synthesis apparatus, the apparatus comprising:

the data acquisition module is used for acquiring target text data and reference voice data;

the vectorization module is used for vectorizing the reference voice data to obtain a reference embedded voice vector;

the feature extraction module is used for extracting features of the target text data to obtain a target text expression vector;

The style marking module is used for performing style marking on the reference embedded voice vector to obtain a target style embedded vector corresponding to the reference embedded voice vector;

the voice synthesis module is used for carrying out voice synthesis based on the target style embedded vector and the target text representation vector to obtain synthesized spectrum data;

and the spectrum conversion module is used for performing spectrum conversion on the synthesized spectrum data to obtain synthesized voice data.

9. An electronic device comprising a memory storing a computer program and a processor implementing the speech synthesis method according to any of claims 1 to 7 when the computer program is executed by the processor.

10. A computer readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the speech synthesis method of any one of claims 1 to 7.