CN114283782A - Speech synthesis method and apparatus, electronic device, and storage medium - Google Patents

Abstract

The application discloses a voice synthesis method and device, electronic equipment and a storage medium, wherein the voice synthesis method comprises the following steps: extracting pronunciation attribute features of a text to be synthesized, and acquiring target attribute features of various voice attributes based on target categories of the text to be synthesized on a plurality of voice attributes respectively; and synthesizing to obtain the synthesized voice of the text to be synthesized based on the pronunciation attribute characteristics and the target attribute characteristics. According to the scheme, the voice synthesis freedom degree can be improved, and meanwhile the cost is reduced.

Description

Speech synthesis method and device, electronic device and storage medium

technical field

The present application relates to the technical field of speech synthesis, and in particular, to a speech synthesis method and apparatus, electronic equipment and storage medium.

Background technique

Speech synthesis technology is one of the core technologies for realizing human-computer interaction. With the continuous development and improvement of speech synthesis technology, speech synthesis has been widely used in all aspects of social life, including public services (information broadcasting, intelligent customer service, etc.), intelligent hardware (smart speakers, intelligent robots, etc.), intelligent transportation (voice navigation, intelligent vehicle equipment, etc.), education (smart classroom, foreign language learning, etc.), pan-entertainment (audio reading, video dubbing, avatar, etc.) and other fields.

At present, speech synthesis technology has been able to reach the level of human voice, and improving the degree of freedom of synthesized speech and reducing the cost of building a synthesis system have become one of the current research hotspots of speech synthesis technology. Although the problem can be solved by letting the same speaker record voice data of different emotions and styles for modeling, this method requires a high degree of ability of the speaker, and it is difficult to find the speaker and record the audio. In view of this, how to reduce the cost of speech synthesis while improving the freedom of speech synthesis has become an urgent problem to be solved.

SUMMARY OF THE INVENTION

The main technical problem to be solved by this application is to provide a speech synthesis method and device, an electronic device and a storage medium, which can reduce the cost while improving the freedom of speech synthesis.

In order to solve the above technical problems, a first aspect of the present application provides a speech synthesis method, including: extracting pronunciation attribute features of text to be synthesized; and obtaining various speech attributes based on target categories of the text to be synthesized on several speech attributes The target attribute feature of the attribute; and then based on the pronunciation attribute feature and each target attribute feature, the synthesized speech of the text to be synthesized is synthesized.

In order to solve the above technical problems, a second aspect of the present application provides a speech synthesis method, which includes: extracting pronunciation attribute features of the speech to be processed; obtaining various speech attributes based on the target categories of the speech to be processed on several speech attributes respectively Then, based on the pronunciation attribute feature and each target attribute feature, the synthesized speech of the to-be-processed speech is synthesized.

In order to solve the above technical problems, a third aspect of the present application provides a speech synthesis device, including an extraction module, an acquisition module and a synthesis module, the extraction module is used to extract the pronunciation attribute features of the text to be synthesized; Obtain the target attribute features of various voice attributes from the target categories on several voice attributes respectively; the synthesis module is used to synthesize the synthesized voice of the text to be synthesized based on the pronunciation attribute features and each target attribute feature.

In order to solve the above technical problems, a fourth aspect of the present application provides a speech synthesis device, including an extraction module, an acquisition module and a synthesis module, the extraction module is used to extract the pronunciation attribute features of the text to be synthesized; Obtain the target attribute features of various voice attributes from the target categories on several voice attributes respectively; the synthesis module is used to synthesize the synthesized voice of the text to be synthesized based on the pronunciation attribute features and each target attribute feature.

In order to solve the above technical problem, a fifth aspect of the present application provides an electronic device, comprising a memory and a processor coupled to each other, the memory stores program instructions, and the processor is configured to execute the program instructions to implement the above first aspect or the second Aspects of speech synthesis methods.

In order to solve the above technical problems, a sixth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, and the program instructions are used to implement the voice in the first aspect or the second aspect. resolve resolution.

In the above scheme, the pronunciation attribute features of the text to be synthesized are extracted, and then the target attribute features of various speech attributes are obtained based on the target categories of the to-be-synthesized texts respectively on several kinds of speech attributes; finally, based on the pronunciation attribute features and each target attribute feature, Synthesized to obtain the synthesized speech of the text to be synthesized. On the one hand, it saves time and helps to reduce the cost of speech synthesis because there is no need to perform a large number of voice recordings. On the basis of the target attribute features of various speech attributes, speech synthesis is carried out, and speech data with any type of speech attributes can be synthesized, which improves the freedom of speech synthesis. Therefore, the cost of speech synthesis can be reduced while the degree of freedom of speech synthesis is improved.

Description of drawings

1 is a schematic flowchart of an embodiment of a speech synthesis method of the present application;

FIG. 2 is a schematic diagram of attribute feature extraction in an embodiment of step S12 in FIG. 1;

3 is a schematic diagram of attribute feature synthesis in an embodiment of step S13 in FIG. 1;

4 is a schematic flowchart of another embodiment of the speech synthesis method of the present application;

5 is a schematic diagram of a framework of an embodiment of a speech synthesis apparatus of the present application;

6 is a schematic diagram of a framework of another embodiment of the speech synthesis apparatus of the present application;

7 is a schematic diagram of a framework of an embodiment of an electronic device of the present application;

FIG. 8 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application.

Detailed ways

The solutions of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

In the following description, for purposes of illustration and not limitation, specific details such as specific system structures, interfaces, techniques, etc. are set forth in order to provide a thorough understanding of the present application.

The terms "system" and "network" are often used interchangeably herein. The term "and/or" in this article is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and A and B exist independently B these three cases. In addition, the character "/" in this document generally indicates that the related objects are an "or" relationship. Also, "multiple" herein means two or more than two.

Please refer to FIG. 1. FIG. 1 is a schematic flowchart of an embodiment of a speech synthesis method of the present application.

Specifically, the following steps can be included:

Step S11: Extract the pronunciation attribute features of the text to be synthesized.

In an implementation scenario, the pronunciation attribute may include, but is not limited to, pronunciation content information, etc., which is not limited herein. Specifically, the pronunciation sequence of the text to be synthesized can be extracted first, and the pronunciation sequence includes several pronunciation marks, and then feature encoding is performed on each pronunciation mark, so as to obtain the pronunciation attribute features of the to-be-synthesized text. Exemplarily, International Phonetic Alphabet (IPA) may be used to mark the pronunciation content information of the synthesized text to obtain a pronunciation sequence; in addition, one-hot encoding may be used to perform feature encoding on each pronunciation mark. For the specific process, you can refer to technical details such as IPA and one-hot, which will not be repeated here.

Step S12: Obtain target attribute features of various voice attributes based on the target categories of the text to be synthesized on several voice attributes respectively.

In an implementation scenario, the several speech attributes include at least one of channel attributes, dialect attributes, timbre attributes, emotional attributes, and style attributes. In the above method, by distinguishing the speech attributes, that is, distinguishing the coupled speech information into multiple speech attributes by decoupling, it is helpful to improve the diversity of speech synthesis in the process of speech synthesis, and improve the performance of speech synthesis. efficiency.

In an implementation scenario, the difficulty in controlling speech attributes is that speech contains a lot of information coupling. In order to freely combine and control the synthesis of attribute information contained in speech, it is necessary to distinguish speech attributes, and combine the coupled information. Decoupling, so as to achieve free combination control of the content of speech synthesis. Voice includes rich attribute information. In order to control the execution information, the category of the voice attribute needs to be determined first. The specific category of the voice attribute can be set according to the specific application scenario, and there is no specific limitation here. This application takes the channel attribute, dialect attribute, timbre attribute, emotion attribute, and style attribute among the speech attributes as examples to describe the speech synthesis process.

In a specific implementation scenario, the channel attribute refers to the environment in which the voice is located, such as a recording studio, conference room, vehicle, etc. In different recording environments, the channel attributes of the voice are different, and the sense of hearing will also be different. You can control the voice channel attribute by controlling the voice channel attribute. Performing voice generation can make the synthesized voice more consistent with the scene and the synthesized voice more realistic; the dialect attribute refers to different dialects under a language, such as Mandarin under Chinese, Northeast dialect, etc. or British English, American under English English etc. The dialect category is determined according to the language scene used; the timbre attribute refers to the human voice color information contained in the speech to distinguish the identity of the speaker; the emotion attribute refers to the emotional category contained in the voice, such as neutral, happy, sad, angry, etc. etc.; the style attribute refers to the speech style of the speech, such as the speech style in different scenarios such as news, interaction, customer service, and novels. After the voice attributes are determined, several voice attributes can also be subdivided into categories. For example, for the timbre attribute, each speaker can be regarded as a category, gender can also be regarded as a category, and age range can be regarded as a category. The specific classification of timbre attributes can be set according to the actual situation, which is not specifically limited here. For attributes that have mature categories such as emotion, they are determined according to application requirements. For attributes that have mature categories such as emotion, they can be determined according to application requirements.

In an implementation scenario, the speech attributes include multiple categories. To ensure the accuracy of speech synthesis, voice data needs to be collected, and the voice data collection is performed according to the multiple categories of voice attributes to cover these voice attribute categories. In order to ensure the correct rate of decoupling of speech attributes between different categories, it should be ensured that each attribute category of several speech attributes collects at least two people's speech data. In this process, the voice data can be recorded by the speaker; the voice data can also be obtained through collection and other means; the voice data can also be obtained by processing the recorded data in the recording equipment of the announcement place. The specific voice data collection method can be selected according to the actual application scenario, which is not specifically limited here.

In an implementation scenario, acquiring target attribute features of various voice attributes according to target categories of the text to be synthesized on several voice attributes, at least including: performing feature construction based on voice data related to the attribute categories of various voice attributes Then, based on the target category of the text to be synthesized on the voice attribute, sampling is performed in the attribute space probability distribution model of the corresponding voice attribute to obtain the target attribute feature of the corresponding voice attribute. In the above manner, by performing feature modeling on the speech data related to the attribute category of the voice attribute, the attribute spatial probability distribution model of the voice attribute can be obtained. In the spatial probability distribution model, different attribute categories can be divided into regions to obtain the same voice attribute The distribution space of different attribute categories is helpful for data sampling during speech synthesis and improves the efficiency of speech synthesis.

In an implementation scenario, each voice attribute has at least one attribute category; then feature modeling is performed based on the voice data related to the attribute categories of various voice attributes to obtain attribute space probability distribution models of various voice attributes, including: For various attribute categories under the voice attribute, extract the sample attribute features of the voice data related to the attribute category, and then construct the attribute space probability distribution model of the voice attribute based on the sample attribute features of the various attribute categories under the voice attribute. . In the above manner, by constructing an attribute space probability distribution model of speech attributes, in the process of speech synthesis, corresponding attribute features can be directly sampled in the distribution model, thereby improving the efficiency of speech synthesis.

In an implementation scenario, in the process of acquiring the target attribute features of various voice attributes, the target attributes can be extracted by manual annotation, and the feature modeling can also be performed on the voice data related to each voice attribute category to obtain The attribute space probability distribution model of various speech attributes obtains the target attribute feature by sampling the attribute space probability distribution of the target voice attribute. The specific acquisition method of the target attribute feature can be selected according to the actual application scenario, which is not specifically limited here.

In a specific implementation scenario, before feature modeling is performed on the speech data related to the attribute categories of various speech attributes, it is necessary to extract the features of each attribute category in the voice data, and then model the attribute features to obtain various Attribute space probability distribution model for speech attributes. In the process of extracting the target attribute, the target attribute can be extracted by establishing a feature extraction model, and the target attribute can also be extracted by manual annotation. The specific method of extracting the target attribute can be set according to the actual situation. This is not specifically limited.

In a specific implementation scenario, the target attribute category features are extracted by manual labeling. The manual labeling method means that people mark the corresponding attribute features after listening to the speech, and then encode the labeling information through a certain encoding method, and the The encoded feature is used as the characteristic feature of the attribute information. For example, if the pronunciation attribute is included in the pronunciation attribute, the pronunciation attribute can use the International Phonetic Alphabet (IPA) to mark the pronunciation content in the speech, obtain the pronunciation sequence annotation of the speech, and then use one-hot encoding to label each pronunciation sequence in the pronunciation sequence. encoding the utterances to obtain a sequence of encoded representations. Or, manually label the emotional attributes. The emotional attributes include neutral, happy, sad, angry and other categories. For the emotional attributes, one-hot encoding can be used to label them. Neutral can be labeled as 1000, happy can be labeled as 0100, and sad can be labeled as 0010, angry can be marked as 0001; emotional attributes can also be encoded by Arabic numerals, neutral can be marked as 1, happy can be marked as 2, sadness can be marked as 3, angry can be marked as 4. In the process of manually labeling the target attributes, different coding methods can be selected for labeling according to the actual situation, which is not specifically limited here.

In a specific implementation scenario, the target attribute category features are extracted by establishing features and then using model training methods to extract models. Model training methods may include but are not limited to supervised learning, self-supervised learning, modeling methods, and the like. Among them, in the learning method of the supervised learning method, the existing attribute labeling data is used as training data to train a sentiment classification model. For example, taking the emotion attribute as an example, the training data of the emotion attribute can be recorded voice data or additionally collected voice data with emotional annotations. The emotion classification model can be used to predict the emotion of the speech data, and the predicted emotion category of the speech data can be obtained, and based on the difference between the sample emotion category marked by the speech data and the predicted emotion category, the network parameters of the emotion classification model can be adjusted to achieve Training of sentiment classification models. It should be noted that the emotion classification model may adopt a deep neural network, a convolutional neural network, etc., which is not limited here. In addition, the model can be continuously updated and optimized by minimizing the cross-entropy loss function to obtain a training-converged sentiment classification model. In this case, the last hidden layer feature of the emotion classification model can be used as the emotion attribute feature of the current speech. In addition, in order to make the extracted emotional attribute features do not contain the feature information of other attributes, in the training process of the emotion classification model, additional information constraint criteria can be designed, such as minimizing the mutual information with the timbre and other criteria to constrain the extracted information representation Not affected by other attribute characteristics. Other voice attributes can be deduced in the same way, and will not be listed one by one here.

Please refer to FIG. 2. FIG. 2 is a schematic diagram of attribute feature extraction in an embodiment of step S12 in FIG. 1. As shown in FIG. 2, the voice data 20 is audio information collected in various ways, and several attribute modules may include a channel attribute module. 21. The dialect attribute module 22, the timbre attribute module 23, the emotion attribute module 24 and the style attribute module 25, and each attribute module can be obtained based on manual annotation or model learning. Certainly, several attribute modules may also include pronunciation attribute modules (not shown), through which pronunciation attribute features can be extracted. For the specific extraction process of the pronunciation attribute module, you can refer to the above-mentioned related descriptions about extracting pronunciation attribute features, which are not described here. Repeat. Corresponding to each attribute module, the attribute characteristics may include channel attribute characteristics 210, dialect attribute characteristics 220, timbre attribute characteristics 230, emotional attribute characteristics 240 and style attribute characteristics 250, and the attribute characteristics are obtained according to the corresponding attribute modules. After the attribute feature extraction corresponding to a large amount of speech data, different attribute category features corresponding to different attribute categories in each attribute can be obtained. According to the sample attribute characteristics of various attribute categories under the voice attribute, the attribute space probability distribution model of the voice attribute is constructed. According to the attribute space probability distribution model of speech attributes, corresponding attribute features of different attribute categories can be obtained.

In an implementation scenario, feature modeling is performed according to the speech data related to the attribute categories of various speech attributes to obtain attribute space probability distribution models of various speech attributes. The method for modeling the speech attribute with the attribute space probability distribution model may include, but is not limited to, the maximum likelihood criterion modeling method, etc., wherein the adopted model structure includes but is not limited to a Gaussian mixture model, a flow model, and the like. After the attribute space probability distribution model of speech attributes is established, in the process of speech synthesis, the probability distribution can be used to sample different attribute categories to obtain the target attribute category features. In addition, in order to achieve controlled sampling of the attribute space probability distribution, other fine control information is input to model the conditional probability distribution of the attribute feature space during modeling. For example, for emotion space, when modeling, the emotion category information is input into the model for modeling. Therefore, in the generation stage, the specified emotional attribute category features can be sampled, so that more refined sampling prediction can be achieved.

In a specific implementation scenario, the attribute features of the attribute space probability distribution model for speech attributes need to be obtained from the speech in the model learning method, and when building the model for the attribute space probability distribution of speech attributes, there can be selected A model is established for the speech attribute, and the specific selection method can be selected according to the specific situation.

Step S13: Synthesize the synthesized speech of the text to be synthesized based on the pronunciation attribute feature and each target attribute feature.

In an implementation scenario, the pronunciation attribute feature is the pronunciation attribute feature of the text to be synthesized, and each target attribute feature is the target attribute feature sampled from the attribute space probability distribution model of various speech attributes. Synthesized to obtain synthesized speech.

In an implementation scenario, as mentioned above, the speech attributes include timbre attributes, and before synthesizing the synthesized speech of the text to be synthesized based on the pronunciation attribute feature and each target attribute feature, it can also be based on the timbre attribute of the to-be-synthesized text. The target category is to obtain the reference attribute feature of the timbre attribute; and the reference attribute feature of the timbre attribute is extracted based on the speech data related to the target category of the timbre attribute; and then the adjustment processing is performed based on the reference attribute feature of the timbre attribute to obtain the target of the timbre attribute. Attribute features, so that a timbre that is different from the speaker in the training set can be generated. On this basis, the synthesized speech of the text to be synthesized can be synthesized based on the pronunciation attribute features and each target attribute feature. In the above manner, in the process of speech synthesis, the reference attribute can also be adjusted and processed to generate a timbre that is different from that of the speaker in the training set, which is conducive to quickly creating a synthesized speech containing rich timbre.

In a specific implementation scenario, during the adjustment process based on the reference attribute feature of the timbre attribute, weighting processing may be performed on multiple reference attribute features of the timbre attribute to obtain the target attribute feature of the timbre attribute. Exemplarily, by sampling the timbre space probability distribution, a synthesized speech with a timbre different from that of the speaker in the training set can be generated, thereby generating a synthesized speech with a new timbre. For two different timbre attribute features S _A and S _B , a new timbre attribute feature S _new can be generated by interpolation, by adding a weight λ, where λ is the interpolation weight between the two features, and the value range is 0 <λ<1. The interpolated feature S _new is synthesized into new speech data by speech, which can be expressed as S _new =λ*S _A +(1-λ)*S _B . The above method, using this method to construct a synthetic voice, saves the work of finding a speaker and audio recording, and can generate a synthetic voice that is different from the voice of the speaker in the training set, thereby generating a synthetic voice with a new tone, and has the ability to avoid copyright risks. advantage.

In a specific implementation scenario, during the adjustment process based on the reference attribute feature of the timbre attribute, the reference attribute feature of the timbre attribute may also be scaled to obtain the target attribute feature of the timbre attribute. Exemplarily, a timbre attribute feature S _C can be stretched to generate a new timbre attribute feature S _new , and by adding a weight λ, the stretched feature S _new can be expressed as S _new =λ*S _C . In the above manner, the synthetic speech constructed in this way can generate a synthesized speech with a timbre different from that of the speaker in the training set, thereby generating a synthesized speech with a new timbre, which can reduce the cost of speech synthesis.

Please refer to FIG. 3 . FIG. 3 is a schematic diagram of attribute feature synthesis in an embodiment of step S13 in FIG. 1 . As shown in FIG. 3 , the speech attribute features include pronunciation attribute feature 30 , channel attribute feature 31 , dialect attribute feature 32 , and timbre attribute feature 33. The emotional attribute feature 34 and the style attribute feature 35, the pronunciation attribute feature and the above-mentioned various speech attribute features can be input into the speech synthesis module 36 for synthesis processing, and the synthesized speech can be obtained. Specifically, the speech synthesis module 36 can be obtained by training sample speech. In this training process, the pronunciation attribute features of the sample speech and the above-mentioned various speech attribute characteristics can be obtained, and the speech synthesis module 36 is used to synthesize the pronunciation attribute characteristics of the sample speech and the above-mentioned various speech attribute characteristics, so as to obtain the synthesis corresponding to the sample speech. speech, so that the network parameters of the speech synthesis module 36 can be adjusted based on the difference between the sample speech and its corresponding synthesized speech (for example, the difference between the two Mel spectra), so that the speech synthesis module 36 can be trained to achieve Each attribute feature controls the generation of speech. It should be noted that the input of the speech synthesis module 36 can be the attribute features obtained by the aforementioned extraction, and the output is synthesized speech 37. The hidden layer network is composed of a deep neural network module, such as a deep neural network, a recurrent neural network, a convolutional neural network. One or a combination of network types such as neural networks. Then the network is trained by setting certain training criteria. The training criterion includes, but is not limited to, the minimum mean square error criterion, the maximum likelihood criterion, and the like. According to the speech synthesis module 36, the control speech synthesis can be performed, and then the characteristics of the synthesized speech 37 can be better controlled.

In the above scheme, the pronunciation attribute features of the text to be synthesized are extracted, and then the target attribute features of various speech attributes are obtained based on the target categories of the to-be-synthesized texts respectively on several kinds of speech attributes; finally, based on the pronunciation attribute features and each target attribute feature, Synthesized to obtain the synthesized speech of the text to be synthesized. On the one hand, it saves time and helps to reduce the cost of speech synthesis because there is no need to perform a large number of voice recordings. On the basis of the target attribute features of various speech attributes, speech synthesis can be performed, and speech data with any type of speech attribute can be synthesized, which improves the freedom of speech synthesis. Therefore, the cost of speech synthesis can be reduced while the degree of freedom of speech synthesis is improved.

Please refer to FIG. 4 , which is a schematic flowchart of another embodiment of the speech synthesis method of the present application. It should be noted that, in the embodiment of the present disclosure, any attribute transfer can be performed on the basis of the to-be-processed speech, so as to synthesize a synthesized speech having some of the same attributes and features as the to-be-processed speech. In addition, the embodiments of the present disclosure only focus on the differences between the embodiments of the present disclosure and the foregoing disclosed embodiments. For the same or similar points, reference may be made to the foregoing disclosed embodiments, which will not be repeated here. Specifically, the embodiments of the present disclosure may specifically include the following steps:

Step S41: Extract the pronunciation attribute features of the speech to be processed.

In an implementation scenario, as mentioned above, the pronunciation attribute may include pronunciation content information in the speech. Similar to the above-mentioned extraction of the pronunciation attribute features of the text to be synthesized, for the speech to be processed, the International Phonetic Alphabet (IPA) can also be used to mark the pronunciation content in the speech to obtain the pronunciation sequence labeling of the speech, and then use One-hot encoding encodes each pronunciation in the pronunciation sequence to obtain pronunciation attribute features.

Step S42: Obtain target attribute features of various voice attributes based on target categories of the speech to be processed on several voice attributes respectively.

In one implementation scenario, target attribute features of various speech attributes are obtained based on the target categories of the speech to be processed on several speech attributes. Please refer to the aforementioned disclosed embodiments for details on the specific speech attributes based on the to-be-synthesized text. The relevant descriptions of the target attribute features of various speech attributes are obtained, which will not be repeated here.

In an implementation scenario, before acquiring the target attribute features of various voice attributes based on the target categories of the speech to be processed on several voice attributes, at least one voice attribute can be selected as the first voice attribute, and the unselected voice attribute can be selected as the first voice attribute. The voice attribute of the voice attribute is used as the second voice attribute; based on the target categories of the to-be-processed voices on several voice attributes, the target attribute features of various voice attributes can be obtained, and the target attribute features of the voice to be processed can be obtained based on the target categories of the first voice attributes respectively. , acquiring the target attribute features of each first voice attribute; and extracting the voice attribute features of each second voice attribute of the to-be-processed voice, respectively, as the target attribute features corresponding to the second voice attributes. In the above-mentioned manner, by migrating the speech attribute features, the attribute feature migration of the speech is completed, and it is possible to control the synthesis of a variety of synthesized speeches with different attribute feature categories.

In a specific implementation scenario, taking the emotion attribute as the first voice attribute as an example, the channel attribute, dialect attribute, timbre attribute, and style attribute can be used as the second voice attribute, and each attribute in the voice to be processed has its own attribute category , and the target attribute feature of the first voice attribute is happy, at this time, the attribute category of the emotional attribute can be extracted from the existing voice data as the attribute feature of happy, and the attribute feature of the extracted emotional attribute as happy and the remaining attributes have The attribute features are synthesized to obtain the final synthesized speech. That is to say, for the speech to be processed, only the emotional attribute features are changed, and the other attribute features are maintained to obtain the final synthesized speech.

In a specific implementation scenario, taking the style attribute as the first voice attribute as an example, the channel attribute, dialect attribute, timbre attribute, and emotion attribute can be used as the second voice attribute, and each attribute in the to-be-processed voice has its own attribute category , and the target attribute feature of the first voice attribute is interaction, at this time, the attribute category of the style attribute can be extracted from the existing voice data as the interactive attribute feature, and the extracted style attribute is the interactive attribute feature and the remaining attributes have The attribute features are synthesized to obtain the final synthesized speech.

It should be noted that the above only takes emotional attributes and style attributes as examples to illustrate the specific process of attribute feature migration. In the actual application process, it can be carried out by selecting any one of the aforementioned several speech attributes or at least a combination of the two. Migration is not limited here.

In addition, exemplarily, the pronunciation attribute features of the speech to be processed can be extracted, and based on the target categories of the to-be-processed speech on channel attributes, timbre attributes, dialect attributes, emotional attributes, and style attributes, the targets of the above-mentioned several speech attributes can be obtained. attribute feature, and the target category of the timbre attribute of the voice to be processed is the original timbre category, that is to say, the timbre attribute feature can be extracted from the to-be-processed voice as the target attribute feature of the timbre attribute, so that the subsequent can be based on the pronunciation attribute feature, The timbre attribute features, as well as the target attribute features corresponding to the target categories specified respectively on the channel attributes, dialect attributes, emotional attributes, style attributes and other phonetic attributes, are used for speech synthesis to preserve the original pronunciation attributes and original timbre of the speech to be processed. On the premise of properties, synthetic speech with different channels, different dialects, different emotions, and different styles can be freely synthesized.

Step S43: Based on the pronunciation attribute feature and each target attribute feature, synthesize the synthesized speech to obtain the speech to be processed.

In an implementation scenario, according to the pronunciation attribute feature and each target attribute feature, the synthesized speech of the speech to be processed can be synthesized, and the target attribute feature can be set according to the actual application scenario. It can effectively control the speech attribute characteristics of the synthesized speech. For the specific synthesis process, reference may be made to the relevant description of the synthesized speech to obtain the text to be synthesized based on the pronunciation attribute feature and each target attribute feature in the aforementioned disclosed embodiments, which will not be repeated here.

In the above scheme, by extracting the pronunciation attribute features of the speech to be processed, and then based on the target categories of the to-be-processed speech on several kinds of speech attributes, the target attribute features of various speech attributes are obtained; finally, based on the pronunciation attribute features and each target attribute feature, Synthesized to obtain the synthesized speech of the speech to be processed. On the one hand, it saves time and helps to reduce the cost of speech synthesis because there is no need to perform a large number of voice recordings. On the basis of the target attribute features of various speech attributes, speech synthesis can be performed, and speech data with any type of speech attribute can be synthesized, which improves the freedom of speech synthesis. Therefore, the cost of speech synthesis can be reduced while the degree of freedom of speech synthesis is improved.

Please refer to FIG. 5 , which is a schematic diagram of a framework of an embodiment of a speech synthesis apparatus of the present application. The speech synthesis apparatus 50 includes an extraction module 51 , an acquisition module 52 and a synthesis module 53 . Among them, the extraction module 51 is used to extract the pronunciation attribute features of the text to be synthesized; the acquisition module 52 is used to obtain the target attribute features of various voice attributes based on the target categories of the text to be synthesized on several kinds of voice attributes; the synthesis module 53 uses Based on the pronunciation attribute feature and each target attribute feature, the synthesized speech of the text to be synthesized is obtained by synthesizing.

In the above scheme, the pronunciation attribute features of the text to be synthesized are extracted, and then the target attribute features of various speech attributes are obtained based on the target categories of the to-be-synthesized texts respectively on several kinds of speech attributes; finally, based on the pronunciation attribute features and each target attribute feature, Synthesized to obtain the synthesized speech of the text to be synthesized. On the one hand, it saves time and helps to reduce the cost of speech synthesis because there is no need to perform a large number of voice recordings. On the basis of the target attribute features of various speech attributes, speech synthesis can be performed, and speech data with any type of speech attribute can be synthesized, which improves the freedom of speech synthesis. Therefore, the cost of speech synthesis can be reduced while the degree of freedom of speech synthesis is improved.

In some disclosed embodiments, the several speech attributes include at least one of channel attributes, dialect attributes, timbre attributes, emotional attributes, and style attributes.

Therefore, by distinguishing the speech attributes, that is, distinguishing the coupled speech information into multiple speech attributes by decoupling, it is helpful to improve the diversity of speech synthesis in the process of speech synthesis, and improve the efficiency of speech synthesis .

In some disclosed embodiments, the acquisition module 52 includes a feature modeling sub-module for performing feature modeling based on speech data related to the attribute categories of various speech attributes to obtain attribute space probability distribution models of various speech attributes; The module 52 includes a model sampling sub-module for sampling in the attribute space probability distribution model corresponding to the speech attribute based on the target category of the speech attribute of the text to be synthesized to obtain the target attribute feature corresponding to the speech attribute.

Therefore, by performing feature modeling on the speech data related to the attribute categories of the voice attributes, the attribute spatial probability distribution model of the voice attributes can be obtained. The distribution space of attribute categories is helpful for data sampling during speech synthesis and improves the efficiency of speech synthesis.

In some disclosed embodiments, each voice attribute has at least one attribute category; the feature modeling sub-module includes an extraction unit for extracting, for various attribute categories under the voice attribute, the voice data related to the attribute category with respect to the voice attribute The sample attribute feature; the feature modeling sub-module includes a construction unit for constructing an attribute space probability distribution model of the voice attribute based on the sample attribute features of various attribute categories under the voice attribute.

Therefore, by constructing an attribute space probability distribution model of speech attributes, the corresponding attribute features can be directly sampled in the distribution model in the process of speech synthesis, which improves the efficiency of speech synthesis.

In some disclosed embodiments, the synthesis module 53 includes an acquisition sub-module for acquiring reference attribute features of the timbre attribute based on the target category of the text to be synthesized on the timbre attribute; and the reference attribute feature of the timbre attribute is based on the target of the timbre attribute Class-related speech data is extracted; the synthesis module 53 includes an adjustment sub-module for performing adjustment processing based on the reference attribute feature of the timbre attribute to obtain the target attribute feature of the timbre attribute.

Therefore, in the process of speech synthesis, the target attribute feature can also be obtained by adjusting the reference attribute, which can generate a synthesized speech different from the timbre of the speaker in the training set, thereby generating a synthesized speech with a new timbre, and has the ability to avoid copyright The advantage of risk.

In some disclosed embodiments, the adjustment sub-module includes any one of a weighting processing unit and a scale adjusting unit: the weighting processing unit is configured to perform weighting processing on a plurality of reference attribute features of the timbre attribute to obtain a target attribute feature of the timbre attribute; the scale The adjustment unit is used to perform scale adjustment on the reference attribute feature of the timbre attribute to obtain the target attribute feature of the timbre attribute.

Therefore, the timbre attributes can be adjusted by means of interpolation, stretching, etc., and the target attribute characteristics of the timbre attributes can be obtained, and then in the process of speech synthesis, the voice data of a new timbre different from the voice of the speaker in the training set can be obtained. The advantage of avoiding copyright risks.

Please refer to FIG. 6 , which is a schematic diagram of a framework of another embodiment of the speech synthesis apparatus of the present application. The speech synthesis device 60 includes an extraction module 61 , an acquisition module 62 and a synthesis module 63 . Among them, the extraction module 61 is used to extract the pronunciation attribute features of the speech to be processed; the acquisition module 62 is used to obtain the target attribute features of various speech attributes based on the target categories of the speech to be processed on several speech attributes; the synthesis module 63 uses Based on the pronunciation attribute feature and each target attribute feature, the synthesized speech is obtained by synthesizing the speech to be processed.

In the above scheme, by extracting the pronunciation attribute features of the speech to be processed, and then based on the target categories of the to-be-processed speech on several kinds of speech attributes, the target attribute features of various speech attributes are obtained; finally, based on the pronunciation attribute features and each target attribute feature, Synthesized to obtain the synthesized speech of the speech to be processed. On the one hand, it saves time and helps to reduce the cost of speech synthesis because there is no need to perform a large number of voice recordings. On the basis of the target attribute features of various speech attributes, speech synthesis can be performed, and speech data with any type of speech attribute can be synthesized, which improves the freedom of speech synthesis. Therefore, the cost of speech synthesis can be reduced while the degree of freedom of speech synthesis is improved.

In some disclosed embodiments, the speech synthesis apparatus 60 further includes a selection module for selecting at least one speech attribute as the first speech attribute, and using the unselected speech attribute as the second speech attribute; the obtaining module 62 includes an obtaining sub-module and extraction sub-module, the acquisition sub-module is used to obtain the target attribute feature of each first voice attribute based on the target category of the to-be-processed voice respectively on each first voice attribute; The voice attribute feature of the second voice attribute is used as the target attribute feature corresponding to the second voice attribute.

Therefore, by migrating the speech attribute features to complete the speech attribute feature migration, it is possible to control the synthesis of a variety of synthesized speeches with different attribute feature categories.

Please refer to FIG. 7 , which is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 70 includes a memory 71 and a processor 72 coupled to each other, the memory 71 stores program instructions, and the processor 72 is configured to execute the program instructions to implement the steps in any of the foregoing speech synthesis method embodiments. Specifically, the electronic device 70 may include, but is not limited to, a desktop computer, a notebook computer, a server, a mobile phone, a tablet computer, etc., which are not limited herein.

Specifically, the processor 72 is used to control itself and the memory 71 to implement the steps in any of the above-mentioned speech synthesis method embodiments. The processor 72 may also be referred to as a CPU (Central Processing Unit, central processing unit). The processor 72 may be an integrated circuit chip with signal processing capability. The processor 72 may also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other Programming logic devices, discrete gate or transistor logic devices, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. In addition, the processor 72 may be jointly implemented by an integrated circuit chip.

The above solution, on the one hand, saves time and helps to reduce the cost of speech synthesis because there is no need to perform a large number of voice recordings by humans; Class, obtain the target attribute features of various speech attributes, and then perform speech synthesis on this basis, which can synthesize speech data with any type of speech attribute, which improves the degree of freedom of speech synthesis. Therefore, the cost of speech synthesis can be reduced while the degree of freedom of speech synthesis is improved.

Please refer to FIG. 8 , which is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 80 stores program instructions 81 that can be executed by the processor, and the program instructions 81 are used to implement the steps in any of the foregoing speech synthesis method embodiments.

The above solution, on the one hand, saves time and helps to reduce the cost of speech synthesis because there is no need to perform a large number of voice recordings by humans; Class, obtain the target attribute features of various speech attributes, and then perform speech synthesis on this basis, which can synthesize speech data with any type of speech attribute, which improves the degree of freedom of speech synthesis. Therefore, the cost of speech synthesis can be reduced while the degree of freedom of speech synthesis is improved.

In some embodiments, the functions or modules included in the apparatuses provided in the embodiments of the present disclosure may be used to execute the methods described in the above method embodiments. For specific implementation, reference may be made to the descriptions of the above method embodiments. For brevity, here No longer.

The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, details are not repeated herein.

In the several embodiments provided in this application, it should be understood that the disclosed method and apparatus may be implemented in other manners. For example, the apparatus implementations described above are only illustrative, for example, the division of modules or units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.

Units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this implementation manner.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented as a software functional unit and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

Claims (12)

1. A method of speech synthesis, comprising:

extracting pronunciation attribute characteristics of a text to be synthesized;

acquiring target attribute characteristics of various voice attributes based on target categories of the text to be synthesized on the plurality of voice attributes respectively;

and synthesizing to obtain the synthesized voice of the text to be synthesized based on the pronunciation attribute characteristics and the target attribute characteristics.

2. The method of claim 1, wherein the plurality of voice attributes comprises: at least one of channel attribute, dialect attribute, timbre attribute, emotion attribute and style attribute.

3. The method according to claim 1 or 2, wherein the obtaining of the target attribute feature of each voice attribute based on the target category of the text to be synthesized respectively on several voice attributes at least comprises:

performing feature modeling based on voice data related to attribute types of various voice attributes to obtain attribute space probability distribution models of various voice attributes;

and sampling an attribute space probability distribution model corresponding to the voice attribute based on the target category of the text to be synthesized on the voice attribute to obtain target attribute characteristics corresponding to the voice attribute.

4. The method of claim 3, wherein each of said voice attributes has at least one of said attribute categories; the performing feature modeling based on the voice data related to the attribute types of the various voice attributes to obtain an attribute space probability distribution model of the various voice attributes includes:

for each attribute category under the voice attribute, extracting sample attribute features of the voice data related to the attribute category and related to the voice attribute;

and constructing an attribute space probability distribution model of the voice attributes based on the sample attribute characteristics of various attribute categories under the voice attributes.

5. The method according to claim 1, wherein the voice attribute comprises a tone attribute, and the obtaining of the target attribute feature of the tone attribute comprises:

acquiring reference attribute characteristics of the tone attribute based on the target category of the text to be synthesized on the tone attribute; wherein the reference attribute feature of the timbre attribute is extracted based on the voice data related to the target category of the timbre attribute;

and adjusting based on the reference attribute characteristics of the tone attributes to obtain target attribute characteristics of the tone attributes.

6. The method according to claim 5, wherein the adjusting process based on the reference attribute feature of the tone color attribute to obtain the target attribute feature of the tone color attribute comprises any one of:

weighting the multiple reference attribute characteristics of the tone attributes to obtain target attribute characteristics of the tone attributes;

and carrying out scale adjustment on the reference attribute characteristics of the tone attributes to obtain target attribute characteristics of the tone attributes.

7. A method of speech synthesis, comprising:

extracting pronunciation attribute characteristics of the voice to be processed;

acquiring target attribute characteristics of various voice attributes based on target categories of the voice to be processed on a plurality of voice attributes respectively;

and synthesizing to obtain the synthesized voice of the voice to be processed based on the pronunciation attribute characteristics and the target attribute characteristics.

8. The method according to claim 7, before said obtaining target attribute features of various voice attributes based on target classes of the voice to be processed on several voice attributes respectively, comprising:

selecting at least one voice attribute as a first voice attribute, and using the unselected voice attribute as a second voice attribute;

the obtaining of the target attribute characteristics of various voice attributes based on the target categories of the voice to be processed on the plurality of voice attributes respectively comprises:

acquiring target attribute characteristics of each first voice attribute based on the target category of the voice to be processed on each first voice attribute; and the number of the first and second groups,

and extracting voice attribute characteristics of the voice to be processed respectively related to the second voice attributes as target attribute characteristics corresponding to the second voice attributes.

9. A speech synthesis apparatus, comprising:

the extraction module is used for extracting pronunciation attribute characteristics of the text to be synthesized;

the acquisition module is used for acquiring target attribute characteristics of various voice attributes based on target categories of the text to be synthesized on the voice attributes respectively;

and the synthesis module is used for synthesizing the synthesized voice of the text to be synthesized based on the pronunciation attribute characteristics and the target attribute characteristics.

10. A speech synthesis apparatus, comprising:

the extraction module is used for extracting pronunciation attribute characteristics of the voice to be processed;

the acquisition module is used for acquiring target attribute characteristics of various voice attributes based on target categories of the voice to be processed on the voice attributes respectively;

and the synthesis module is used for synthesizing the synthesized voice of the voice to be processed based on the pronunciation attribute characteristics and the target attribute characteristics.

11. An electronic device comprising a memory and a processor coupled to each other, the memory having stored therein program instructions, the processor being configured to execute the program instructions to implement the speech synthesis method of any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that program instructions executable by a processor for implementing the speech synthesis method of any one of claims 1 to 8 are stored.

Images