CN102982809A - Conversion method for sound of speaker - Google Patents

Abstract

The invention discloses a speaker voice conversion method, which includes a training stage and a conversion stage. The training stage includes: extracting fundamental frequency features, speaker features and content features from the training voice signals of the source speaker and the target speaker; The fundamental frequency feature constructs a fundamental frequency transfer function; constructs a speaker transfer function according to the speaker feature. The conversion stage includes: extracting fundamental frequency features and spectral features from the source speaker's speech signal to be converted; The features and speaker features are converted to obtain the converted fundamental frequency features and speaker features; the speech of the target speaker is synthesized according to the obtained converted fundamental frequency features, speaker features and content features in the speech signal to be converted. The invention is easy to implement and has higher sound quality and similarity after conversion.

Description

A method of speaker's voice conversion

technical field

The invention belongs to the technical field of signal processing, and specifically relates to converting a speaker's voice signal into a voice signal that can be perceived as another speaker's voice signal without changing the content information in the voice signal. A speaker voice conversion method for separating speaker information and content information in a voice signal.

Background technique

In today's information age, human-computer interaction has always been a research hotspot in the computer field, and an efficient and intelligent human-computer interaction environment has become an urgent need for the application and development of current information technology. As we all know, speech is one of the most important and convenient ways for human communication. Voice interaction will be the most "friendly" among human interactions. Human-machine speech dialogue technology based on speech recognition, speech synthesis and natural language understanding is recognized as a very difficult and challenging high-tech field in the world, but its application prospect is very bright.

As one of the core technologies of human-computer interaction, speech synthesis has made great progress in both technology and application in recent years. At present, the speech synthesized by the synthesis system based on a large corpus has achieved good results in terms of sound quality and naturalness, so everyone puts forward more demands on the speech synthesis system - diversified speech synthesis, including multiple speakers, Multiple pronunciation styles, multiple emotions, multiple languages, etc. However, most of the existing speech synthesis systems are simplistic. A synthesis system generally only includes one or two speakers, adopts the style of reading aloud or news broadcast, and is aimed at a specific language. This simplification of synthesized speech greatly limits the practical applications of speech synthesis systems, including education, entertainment and toys. For this reason, research on diverse speech synthesis has gradually become one of the mainstream directions in the field of recent speech synthesis research.

The most direct way to realize a multi-speaker, multi-pronunciation style, and multi-emotion speech synthesis system is to record multiple people and multi-style sound banks, and build individual speech synthesis systems for each speaker and each style . This approach is not feasible in practice because it would be too much work to produce a specific speech library for each speaker, each style, and each emotion. In this background, speaker voice conversion technology is proposed. The speaker voice conversion technology is to try to convert the speech (voice) of a person (source speaker) (adjust the fundamental frequency, duration, spectral parameters and other parameters containing speaker characteristic information) to make it sound like another person (source speaker) The same as what a person (the target speaker) says. At the same time, the meaning expressed by the source speaker is kept unchanged. The speaker voice conversion technology trains by recording a small amount of speaker's voice signals, adjusts the voice of the source speaker to obtain the synthesized voice of the target speaker, and quickly realizes the personalized speech synthesis system.

To implement a speaker voice conversion system, the main challenge lies in the similarity and sound quality of the converted voice. As a current mainstream speaker voice conversion method-the speaker voice conversion method based on the joint space Gaussian mixture model, due to the use of a statistical modeling framework, it is relatively robust and generalizable. However, this method is only a typical feature mapping method in machine learning, and does not take advantage of some characteristics unique to speech signals (coexistence of speaker information and content information), and statistical modeling brings many problems, such as the amount of data. Dependence, insufficient modeling accuracy, and the destruction of the original information of the acoustic parameters by the statistical model all lead to a sharp decline in the effect of converting speech. Another mainstream speech synthesis technology, the formant-based spectrum bending method, utilizes the speaker formant structure in the speech signal, which mainly reflects the speaker's information, and preserves the speech signal as much as possible during the conversion. However, due to the difficulty in extracting and modeling formants, this type of method requires a lot of manual intervention and is less robust.

In general, the traditional speaker-to-speech conversion method lacks effective expression and effective modeling of the voice information of a specific speaker in the speech signal, and has high requirements for modeling data. The conversion of signal content, so the voice quality and similarity after conversion cannot reach a satisfactory level at present.

Contents of the invention

(1) Technical problems to be solved

The technical problem to be solved by the invention is that the existing speaker voice conversion method has poor voice quality and low similarity.

(2) Technical solutions

The present invention proposes a speaker voice conversion method, which is used to convert the voice signal of the source speaker's speech, so that the converted voice sounds different from the target speaker's speech of the source speaker, and is characterized in that , the method includes a training phase and a conversion phase, where,

The training phase includes:

Step A1, extract fundamental frequency features and spectral features respectively from the training voice signals of the source speaker and the target speaker, the spectral features include speaker features and content features;

Step A2, according to the fundamental frequency characteristics of the training voice signals of the source speaker and the target speaker, construct a fundamental frequency conversion function from the voice of the source speaker to the voice of the target speaker;

Step A3, constructing a speaker conversion function according to the speaker characteristics of the source speaker and the target speaker extracted in step A1;

The conversion phase includes:

Step B1, extracting fundamental frequency features and spectral features from the source speaker's speech signal to be converted, the spectral features include speaker features and content features;

Step B2, using the base frequency conversion function and the speaker conversion function obtained in the training stage to convert the base frequency feature and the speaker feature extracted from the speech signal to be converted in step B1 to obtain the converted base frequency frequency features and speaker features;

Step B3: Synthesize the speech of the target speaker according to the converted fundamental frequency features and speaker features obtained in step B2, and the content features in the speech signal to be converted extracted in step B1.

According to a specific embodiment of the present invention, the method for extracting the fundamental frequency feature and spectral feature of the speech signal in the steps A1 and B1 includes:

Step a1, based on the source-filter structure of the speech signal, the speech signal is segmented by 20-30 ms, each segment is regarded as a frame, and the fundamental frequency and spectrum parameters are extracted from the speech signal of each frame;

Step a2, using a neural network to separate the speaker features and content features in the spectral parameters, the neural network structure adopts a symmetrical up-down 2K-1 layer multi-layer (K is a natural number) network structure, including: the lowest layer is The input layer, from which the acoustic features to be separated are input; the uppermost layer is the output layer, which outputs the reconstructed acoustic features; the middle 2K-3 hidden layers, each with several nodes, simulates the processing process of the neural unit. The Kth hidden layer from the input layer to the top is the encoding network, which is used to extract high-level information from the input speech acoustic features; the Kth hidden layer from the bottom to the top is the encoding layer; the network node of the encoding layer It is divided into two parts, one part is related to the speaker, and the other part is related to the content. Their outputs correspond to the speaker features and content features respectively; Acoustic spectral parameters are reconstructed from the speaker features and content features of the system.

According to a specific embodiment of the present invention, step a2 includes training the neural network on a speech signal database, so that it has the ability to extract and separate speaker features and content features from acoustic features, the pair The steps of training the neural network include:

Step b1, initialize the network weights of the neural network through pre-training;

Step b2, for the output features of each node of the coding layer of the neural network, use a discrimination criterion to count its discrimination among different speakers and between different contents, and distinguish between different speakers and Nodes with little discrimination between different contents are regarded as speaker-related nodes, and the remaining nodes are regarded as content-related nodes;

Step b3, designing a specific discriminative objective function to fine-tune the weights of the neural network, so that the neural network has the ability to separate speaker information and content information from acoustic features.

According to a specific embodiment of the present invention, the speech signal database is made through the following steps:

Step c1, set up a corpus so that multiple sentences are included in the corpus;

Step c2, recording voice signals of multiple speakers reading sentences in the corpus, constructing a voice signal database, and preprocessing the voice signals in the voice signal database to remove abnormal parts in the voice signals;

Step c3, using the hidden Markov model to segment the speech signal in the preprocessed speech signal database, each segment after the segmentation is used as a frame, and the frame-level speaker of each speech signal is obtained Labeling information and content labeling information;

Step c4. Randomly combine the speech signals in the speech database to construct training data for the neural network.

(3) Beneficial effects

The speaker voice conversion method of the present invention has the following advantages:

1. The present invention proposes for the first time the use of a deep neural network to separate speaker information and content information in speech signals, so as to meet the needs of different speech signal processing tasks, such as speech recognition, speaker recognition and conversion.

2. When the present invention converts the speaker's voice, only the factor of the speaker is considered, and the interference of the content factor is eliminated, so that the conversion of the speaker's voice is easier to realize, and the converted sound quality and similarity can be greatly improved.

3. The separator used in the present invention only needs to be trained once. After training, it can extract speaker features and content features from any speaker's voice, and it can be used for multiple times after one training session without repeated training of the model.

Description of drawings

Fig. 1 is the flowchart of the speaker's voice conversion method of the present invention;

Fig. 2 is the block diagram of feature extraction step of the present invention;

Fig. 3 is a schematic diagram of the neural network structure for feature separation of the present invention;

Fig. 4 is the neural network training flowchart of the present invention;

Fig. 5 is the flowchart that database is made among the present invention;

Fig. 6 is a schematic diagram of the discrimination of cepstral features between different speakers and different contents in the present invention;

Fig. 7 is a schematic diagram of the distinction between different speakers and different contents of the speaker features and content features extracted in the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

From a physiological point of view, the work of scholars has confirmed that when the human brain perceives speech signals, the perception of speaker information and the perception of speech content are completed in different areas of the cerebral cortex. This shows that the human brain decomposes the speaker and content information at a high level. The information in the speech signal is separable. The separation of speaker information and content information is of great significance to speech signal processing. The separated information can be used separately. For speaker recognition, speech recognition and other targeted applications.

The present invention starts from the essence of the speaker's voice conversion, that is, keeps the content of the speaker's words unchanged, and only changes the information of the speaker who said the sentence. Based on this consideration, the information in the speech signal is separated to obtain speaker features and content features in order to operate on speaker components. The "speaker feature" in the present invention refers to the feature that reflects the characteristics of the speaker and distinguishes different speakers, and the "content feature" refers to the feature that reflects the meaning to be expressed by the speech signal.

In this regard, the present invention uses a deep neural network-based technology to decompose the acoustic features of the speech signal into speaker features and content features at a high level, so that the speaker's voice conversion can be realized more perfectly and simply, achieving sound quality and The converted speech signal with greatly improved similarity.

FIG. 1 is a flow chart of the speaker voice conversion method of the present invention. As shown in the figure, the method of the present invention generally includes two phases: a training phase and a conversion phase. The following are introduced in turn:

(1) Training stage

The training phase mainly includes three steps:

Step A1: Feature Extraction.

This step extracts features from the training speech signals of the source speaker and the target speaker respectively, and the features include fundamental frequency features and spectral features, and the spectral features are divided into speaker features and content features in the present invention.

Step A2: Fundamental frequency conversion function training.

This step constructs a fundamental frequency conversion function from the voice of the source speaker to the voice of the target speaker according to the fundamental frequency features of the training speech signals of the source speaker and the target speaker.

According to a specific implementation manner, this step counts the mean and variance of the fundamental frequency features of the training speech signals of the source speaker and the target speaker in the logarithmic domain, and constructs the speech from the source speaker to The fundamental frequency transfer function of the speech of the target speaker.

Since the fundamental frequency characteristic parameter of each speaker is Gaussian distributed in the logarithmic domain, the fundamental frequency conversion is preferably performed only by simple linear transformation in the logarithmic domain in the present invention.

Step A3: spectral conversion function training.

This step constructs a speaker transfer function according to the speaker features in the spectral features extracted from the training speech signals of the source speaker and the target speaker.

The aforementioned requirement of speaker switching keeps the speech content unchanged and only changes the speaker information. Therefore, the present invention only needs to train the conversion function of speaker characteristics (speaker conversion function).

Because when recording the voice signals of the source speaker and the target speaker, it is impossible to keep the same duration when different speakers record the same sentence, so some regular means are needed to regularize the sentences of different durations to the same duration In order to carry out supervised feature conversion learning (feature alignment), the present invention uses a dynamic time warping (dynamic time warping) algorithm to carry out time length regularization, and the modeling of speaker feature conversion can use methods such as linear regression model or joint space Gaussian mixture model to fulfill.

(2) Conversion stage

The conversion phase consists of three steps:

Step B1: Feature extraction.

Similar to the training stage, this step extracts features from the source speaker's speech signal to be converted, the features include fundamental frequency features and spectral features, and the spectral features are divided into speaker features and content features.

Step B2: Feature Transformation.

Using the base frequency conversion function and the speaker conversion function obtained in the training stage respectively, the base frequency feature and the speaker feature extracted from the speech signal to be converted in step B1 are converted to obtain the converted base frequency feature and speaker characteristics.

基频转换时转换函数形如下式所示：For fundamental frequency conversion, specifically, in the training phase, the mean value μ _x , μ _y and variance of the fundamental frequency of the source and target speaker’s speech signals in the logarithmic domain are calculated on the training set

The form of the conversion function when the base frequency is converted is as follows:

$\mathrm{<span\; class="notranslate">\; log</span>}{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="0"\; label="f"\; filenames="HDA00002561067300051.png"\; state="\{\{state\}\}">f</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}^{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; +</span>\frac{{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}}{{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; log</span>}{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}^{\mathrm{<span\; class="notranslate">\; x</span>}}{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; )</span>$

And for the transformation of speaker features, it is assumed that the source and target speakers correspond to time-aligned speaker features X={x ₁ , x ₂ ,...x _T } and Y={y ₁ , y ₂ ,... y _T } as training data. The present invention adopts two kinds of schemes. One solution is to use the linear regression model F(x _t )=Ax _t +b as the spectrum conversion function, where the parameters can be calculated by the following formula:

[A, b] = YX ^T (XX ^T ) ^-1

Another solution, the method based on the joint space Gaussian mixture model, needs to use the joint feature Z=[X ^T , Y ^T ] ^T to train a Gaussian mixture model, which describes the distribution of the joint feature space in the following form:

P(z)= _∑mwmN (z; _μm , _∑m ) _,

in

${\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; =</span>\left[\begin{array}{c}{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span>}\\ {\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}\end{array}\right]<span\; class="notranslate">\; ,</span>$ ${\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; =</span>\left[\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; xxx</span>}<span\; class="notranslate">\; )</span>}& {\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; xy</span>}<span\; class="notranslate">\; )</span>}\\ {\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; yx</span>}<span\; class="notranslate">\; )</span>}& {\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; yy</span>}<span\; class="notranslate">\; )</span>}\end{array}\right]$

From it, export the conversion function:

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; yx</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; xxx</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ]</span>$

为后验概率。In the formula

is the posterior probability.

Step B3: speech synthesis.

In this step, the speech of the target speaker is synthesized according to the converted fundamental frequency features and speaker features obtained in step B2, and the content features in the speech signal to be converted extracted in step B1.

The present invention uses a synthesizer based on a source-filter structure that requires an input stimulus (ie fundamental frequency) and a vocal tract response (spectral parameters) to generate the speech to be converted. Therefore, it is first necessary to reconstruct the converted speaker spectral parameters from the converted speaker features and the content features of the speaker's voice signal to be converted (the spectral parameter reconstruction process is described below), and then the converted speech is generated by a synthesizer. The present invention uses a STRAIGHT analysis synthesizer for speech generation.

(3) Feature extraction

The method of the present invention has been introduced as a whole above, and the feature extraction steps adopted in the method will be described in detail below.

As mentioned above, the feature extraction in the present invention includes the extraction of fundamental frequency features, speaker features and content features. The fundamental frequency feature extraction in the present invention adopts the traditional fundamental frequency extraction method. The feature extraction method of speaker features and content features is the core of the present invention.

3.1 Basic steps

Fig. 2 is a block diagram of the feature extraction steps of the present invention. As shown in Figure 2, the feature extraction step is specifically divided into two steps:

Step a1: Acoustic feature extraction.

Based on the source-filter structure of the speech signal, considering the short-term stationary and long-term non-stationary of the speech signal, the speech signal is segmented by 20-30 ms, and each segment is called a frame in the present invention. For each frame of speech signal, use the existing speech analysis algorithm (such as STRAIGHT, etc.) to extract fundamental frequency and spectral parameters (such as line spectrum pair, Mel cepstrum, etc.) from the speech signal.

Step a2: Speaker feature and content feature extraction.

Considering that the difference between speakers is mainly reflected in the structure of the vocal tract, in terms of acoustic characteristics, that is, it is mainly reflected in the spectral parameters. Therefore, the present invention mainly considers separating speaker-dependent features and content-dependent features from spectral features. In addition, the present invention considers that the speaker feature is a long-term feature of the suprasegment, in order to effectively extract the speaker-related features in the speech signal, so that it can be better separated from the content-related features, the present invention combines the continuous multi-frame The features are concatenated into a so-called suprasegment feature input to the feature separator. The specific feature separation method is as follows:

3.2 Feature Separation Algorithm

The present invention uses a deep neural network to separate speaker features and content features in acoustic spectrum parameters. Fig. 3 is a schematic diagram of the structure of the neural network used for feature separation in the present invention. As shown in Figure 3, the neural network structure adopts a symmetric 2K-1 layer multi-layer (K is a natural number) network structure, including: the lowermost layer is the input layer, from which the acoustic features to be separated are input; the uppermost layer is The output layer, which outputs the reconstructed acoustic features; the middle 2K-3 hidden layers, each layer includes several nodes, simulating the processing process of the neural unit.

The Kth hidden layer from the input layer to the top is the encoding network (or encoder), which is used to extract high-level information from the input speech acoustic features, and the Kth hidden layer from the bottom to the top is the encoding layer ; The network nodes of the encoding layer are divided into two parts, one part is related to the speaker, and the other part is related to the content, and their outputs correspond to the speaker features and content features respectively. The hidden layer above the Kth hidden layer from bottom to top is the decoding network (or decoder), its function is opposite to that of the encoding network, and it is used to reconstruct the acoustic spectrum parameters from the high-level speaker features and content features.

The deep neural network shown in Figure 3 that the present invention adopts is a simulation of the processing of speech signals to the human nervous system, and it needs to be trained so that it has the required ability to extract and separate speaker features from acoustic features and the specific ability to characterize content. The training of the deep neural network shown in Fig. 3 is carried out on the speech signal database designed by the database production method proposed by the present invention, and the database production method proposed by the present invention is shown in the database production part of the present invention.

Fig. 4 is a specific flowchart of neural network training in the present invention. The training process is divided into three steps:

Step b1: Pre-training.

Since the optimization of deep neural networks is difficult, it is necessary to initialize the network weights through pre-training before training. The present invention adopts an unsupervised learning mode, uses a greedy algorithm to train the network layer by layer, and quickly obtains the initial parameters of the model. In the training of each layer, a De-noising auto-encoder can be used to initialize the network weights, that is, a certain amount of noise is added to the input features to make the training of the neural network more efficient. Robust and resistant to overtraining. Specifically, in the input layer, if the input features obey the Gaussian distribution, an appropriate amount of Gaussian noise is added to each dimension of the input, and the minimum mean square error criterion is used for training. In the layers above the first layer, the input features obey the binary distribution, so with a certain probability, some dimensions of the input features are set to zero, and the minimum cross-entropy (cross-entropy) criterion is used for training. After pre-training, a K-layer superimposed autoencoder is obtained, and it is turned upside down to obtain a vertically symmetrical autoencoder structure.

Step b2: Coding layer adjustment.

After pre-training, the neural network already has a certain high-level information extraction capability. In the coding layer, some nodes can reflect a strong ability to distinguish speakers, while others can reflect a strong ability to distinguish content. In this step, some objective criteria will be used to select these nodes, and their outputs will be used as corresponding features. Here you can use some distinguishing criteria, such as Fisher's ratio, to select. Specifically, on the training set of the speech signal database, for the output features of each node of the coding layer, this criterion is used to count its discrimination among different speakers and between different contents, and the different speech Nodes with high inter-personal distinction but little distinction between different contents are regarded as speaker-related nodes, and the rest are regarded as content-related nodes.

Step b3: Fine adjustment.

和因此，训练网络的目标函数中包含如下的三部分：The present invention needs to separate the speaker-related and content-related features from the input acoustic spectrum parameters, and apply them to the speaker's voice conversion. For this, it is necessary to design a specific discriminative objective function to train the network so that it has the ability expected by the present invention. To meet this requirement, it is necessary to introduce a means of comparison and competition in the input training samples. In the network structure shown in Figure 3, in the input layer, two samples x ₁ and x ₂ are input in parallel each time, and they generate speaker features c _s1 , c _s2 and content features c _c1 , c _c2 , and then through the decoding network, the input acoustic features are reconstructed

and Therefore, the objective function of training the network contains the following three parts:

Reconstruction error: On the one hand, due to the needs of the speaker's voice conversion application, to reconstruct and restore the acoustic spectrum parameters from the high-level features, the decoding network needs to have a good ability to restore and reconstruct, which will directly affect the quality of synthesized speech . Therefore, it is necessary to limit the reconstruction error in the training objective function. On the other hand, the limitation of reconstruction error is also added to ensure the integrity of information in the encoded output speaker features and content features. The error form of following form is adopted in the present invention:

${\mathrm{<span\; class="notranslate">\; L</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; 0,1</span>}<span\; class="notranslate">\; \}</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span>{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$

Speaker feature cost: In order to make the speaker feature highly distinguishable to the speaker but not to the content, such a criterion can be designed so that the error of the speaker feature between the same speakers is as small as possible, while The error between different speakers should be as large as possible. This criterion can be expressed as the following formula:

L _sc ＝δ _s *E _s +(1-δ _s )*exp(-λ _s E _s )

Among them, E _s ＝|c _s1 -c _s2 | ² , δ _s is the speaker annotation of the two input samples, δ _s =1 means that the two inputs come from the same speaker, and δ _s =0 means from two different speakers.

Content feature cost: Similar to the speaker feature error, a discriminative cost function for content features can be constructed:

L _cc ＝δ _c *E _s +(1-δ _c )*exp(-λ _c E _c )

Combining the above three costs, the final fine-tuned objective function can be obtained:

L _cc =αL _r +βL _sc +ζL _cc

α, β and ζ adjust the weights of these three kinds of cost proportions. The training goal of the neural network is to adjust the network weights so that the objective function is as small as possible. During training, the present invention uses the error backpropagation algorithm and utilizes the gradient descent algorithm with impulse to update the network weights.

(4) Production of speaker speech signal library

The neural network used in the present invention requires a large amount of training data, needs to include many speakers, and each speaker also needs to record a corpus with sufficient content.

It should be pointed out that the large amount of training data required by the neural network is not the source speaker or target speaker data in the training process shown in Figure 1. In practical applications, it is impractical or too demanding to obtain a large amount of data of the source speaker or the target speaker in the training process shown in Figure 1, but it is feasible to obtain the large amount of training data required by the neural network described here. practical requirements.

Fig. 5 is a flowchart of database creation in the present invention. Divided into four steps:

Step c1: Establish a corpus, so that the corpus includes multiple sentences.

Considering that to design a robust separation network, it needs to be able to handle all people and all content. In the present invention, a phoneme-balanced corpus is designed, and the number of sentences should not be too many, usually within 100 sentences, so as to collect a large number of speaker data. The so-called phoneme balance means that the corpus contains all phonemes, and the number of each phoneme is relatively balanced.

Step c2: recording voice signals of multiple speakers reading sentences in the corpus, constructing a voice signal database, and preprocessing the voice signals in the voice signal database to remove abnormal parts in the voice signals.

Considering that in order for the network to have the ability to distinguish speakers, it is necessary to record a large number of speaker data to train the network. In the recording stage, due to cost and other reasons, it is impossible to find so many announcers to record the sound library, and only the recordings of amateurs can be collected, which makes the quality of recorded voices uneven. Therefore, after the recording is completed, it is necessary to The recorded speech is pre-processed, such as energy regularization, channel equalization, processing of microphone spraying, etc., to ensure the quality of the training corpus.

Step c3: Use the hidden Markov model to segment the speech signal in the preprocessed speech signal database, and each segment after segmentation is used as a frame, and the frame-level speaker of each speech signal is obtained Annotation information and content annotation information.

It can be seen from the above that in the fine adjustment stage of neural network training, it is a supervised learning process, and it is necessary to know the speaker annotation information and content annotation information of each frame of training data input. Therefore, it is necessary to mark the speech signals in the speech signal database at the frame level, that is, to segment the speech segments. Specifically, an existing context-dependent hidden Markov model used for speech synthesis can be used to realize segment segmentation. Before segmentation, use the maximum likelihood linear regression algorithm to adapt the model to the speaker's acoustic space with the recording data of each speaker, and then use the adaptive model to use the Werther's recording data for the speaker. The ratio algorithm is decoded to obtain the boundary information of each state of the model.

Step c4: Randomly combine the speech signals in the speech database to construct training data for the neural network.

According to the above description, there are four types of training data for neural networks: the same content for the same speaker, different content for the same speaker, the same content for different speakers, and different content for different speakers. Since there are many speaker features and content feature attributes, in the training phase, the present invention randomly selects combinations from the training data and inputs them to the network for training.

(5) Specific examples

According to the method described above, as an example of the implementation of the present invention, the present invention builds a speaker voice conversion system. First, the present invention designs a phoneme-balanced corpus containing 100 sentences, recruits 81 speakers (including 40 male and 41 female speakers) to record, and forms the final training corpus after processing. The recorded voice files are mono, 16kHz sampling rate. Among the 81 speaker data, we randomly selected the data of 60 people (30 males, 30 females) as the training set for training the neural network, and the data of the other 10 people (5 males and 5 females) as the training set. The verification set of the neural network training, and the data of the remaining 11 people are used as the test set to test the effect of the speaker's voice conversion. When extracting acoustic features, we use a Hamming window of 25ms to process the waveform signal into frames, and move the short time window with a frame shift of 5ms. Each frame extracts a fundamental frequency and a set of 24-dimensional Mel cepstrum parameters as acoustic features.

In the stage of training the neural network for feature separation, the input vector of the network is the supersegment features composed of 11 frames of the current frame and 5 frames before and after each, with a total of 264 dimensions. Since the output only needs to reconstruct the current input frame, so , the output layer is 24-dimensional. In addition, the network contains 7 hidden layers, in which the number of nodes is 500, 400, 300, 200, 300, 400, 500. In the middle layer, we make the output of the first 100 nodes the speaker features, and the remaining The output of 100 nodes is the content feature. In the pre-training stage, we use 4 stacked autoencoders to initialize the network weights, the number of nodes are: 264-500, 500-400, 400-300 and 300-200, from bottom to top, each automatic The output of the encoder is used as the input of the next autoencoder. The network weights are initialized in the form of unsupervised learning, and finally the network weights are flipped to obtain the initialization weights of the entire network. It should be noted that the first layer is flipped to the entire At the top layer of the network, since the output is only 24-dimensional, it is only necessary to flip the weight corresponding to the current frame of the input layer. In addition, before the middle layer is flipped, it is necessary to calculate the difference between different speakers and different content of each node output (Fisher's ratio mentioned above), and use this to re-weight the node and network weights. Row. After pre-training, fine-tuning is carried out according to the method described above. In this process, the weight of the objective function needs to be adjusted on the verification set to obtain the optimal value.

After the feature separator is trained, the speaker voice conversion system can be built. We randomly select two speakers from the test set, select 50 sentences as the training data, and extract the required features according to the above. The training base The conversion function of the frequency and speaker features (a direct linear regression model is used as an example in this embodiment), and the remaining 50 sentences are used as test data to verify the effect of speaker voice conversion.

We use Fisher's ratio to measure the discriminativeness of different extracted features between different speakers and different content. Fisher's ratio measures the ratio of feature intra-class distance to inter-class distance. The larger the ratio, the more discriminative the feature is under this classification method. Figure 6 and Figure 7 are the discriminativeness of Mel cepstral coefficients and separated features between different speakers (solid line) and different content (dashed line), respectively. It can be seen that among the input acoustic features, except for the low-dimensions that show strong discrimination in terms of content, the other dimensions are not very differentiated. The extracted features (the first 100 dimensions are speaker features, and the remaining 100 dimensions are content features) have been trained to show the desired distinction for different categories. In the speaker conversion experiment, the cepstrum error is 4.39dB for the speech synthesized by directly using the speaker characteristics of the target speaker plus the content characteristics of the source speaker, while the linearly transformed speaker characteristics of the source speaker The speech cepstrum error synthesized with its content features is 5.64dB, which is close to the target speaker's speech from the subjective sense of hearing.

The specific embodiments described above have further described the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above descriptions are only specific embodiments of the present invention, and are not intended to limit the present invention. Within the spirit and principles of the present invention, any modifications, equivalent replacements, improvements, etc., shall be included in the protection scope of the present invention.

Claims (11)

1. A speaker's voice conversion method, which is used to convert the voice signal of the source speaker's spoken words, so that the converted voice sounds different from what the source speaker's target speaker said, it is characterized in that, The method includes a training phase and a conversion phase, where, The training phase includes: Step A1, extract fundamental frequency features and spectral features respectively from the training voice signals of the source speaker and the target speaker, the spectral features include speaker features and content features; Step A2, according to the fundamental frequency characteristics of the training voice signals of the source speaker and the target speaker, construct a fundamental frequency conversion function from the voice of the source speaker to the voice of the target speaker; Step A3, constructing a speaker conversion function according to the speaker characteristics of the source speaker and the target speaker extracted in step A1; The conversion phase includes: Step B1, extracting fundamental frequency features and spectral features from the source speaker's speech signal to be converted, the spectral features include speaker features and content features; Step B2, using the base frequency conversion function and the speaker conversion function obtained in the training stage to convert the base frequency feature and the speaker feature extracted from the speech signal to be converted in step B1 to obtain the converted base frequency frequency features and speaker features; Step B3: Synthesize the speech of the target speaker according to the converted fundamental frequency features and speaker features obtained in step B2, and the content features in the speech signal to be converted extracted in step B1. 2. the speaker's voice conversion method as claimed in claim 1, is characterized in that, the fundamental frequency feature of the training speech signal of the training speech signal of described step A2 statistics source speaker and target speaker is in the mean value and the variance of logarithmic domain distribution, according to The calculated mean and variance construct the fundamental frequency transfer function from the source speaker's speech to the target speaker's speech. 3. The speaker voice conversion method according to claim 2, wherein the fundamental frequency conversion function is a linear conversion function. 4. the speaker's voice conversion method as claimed in claim 1, is characterized in that, the fundamental frequency characteristic and the spectral characteristic method of the extracting voice signal of described step A1 and step B1 comprise: Step a1, based on the source-filter structure of the speech signal, the speech signal is segmented by 20-30 ms, each segment is regarded as a frame, and the fundamental frequency and spectrum parameters are extracted from the speech signal of each frame; Step a2, using a neural network to separate the speaker features and content features in the spectral parameters, the neural network structure adopts a symmetrical up-down 2K-1 layer multi-layer (K is a natural number) network structure, including: the lowest layer is The input layer, from which the acoustic features to be separated are input; the uppermost layer is the output layer, which outputs the reconstructed acoustic features; the middle 2K-3 hidden layers, each with several nodes, simulates the processing process of the neural unit. The Kth hidden layer from the input layer to the top is the encoding network, which is used to extract high-level information from the input speech acoustic features; the Kth hidden layer from the bottom to the top is the encoding layer; the network node of the encoding layer It is divided into two parts, one part is related to the speaker, and the other part is related to the content. Their outputs correspond to the speaker features and content features respectively; Acoustic spectral parameters are reconstructed from the speaker features and content features of the system. 5. The speaker's voice conversion method as claimed in claim 4, wherein said step a2 includes training said neural network on a speech signal database, so that it has the ability to extract and separate speech from acoustic features. Ability to characterize people and characterize content. 6. The speaker's voice conversion method as claimed in claim 5, wherein the described step of training the neural network comprises: Step b1, initialize the network weights of the neural network through pre-training; Step b2, for the output features of each node of the coding layer of the neural network, use a discrimination criterion to count its discrimination among different speakers and between different contents, and distinguish between different speakers and Nodes with little discrimination between different contents are regarded as speaker-related nodes, and the remaining nodes are regarded as content-related nodes; Step b3, designing a specific discriminative objective function to fine-tune the weights of the neural network, so that the neural network has the ability to separate speaker information and content information from acoustic features. 7. the speaker voice conversion method as claimed in claim 5, is characterized in that, described step b1 adopts unsupervised learning pattern, uses greedy algorithm to train this neural network layer by layer; 8. the speaker's voice conversion method as claimed in claim 7, is characterized in that, described step b1 comprises: In the input layer, if the input features obey the Gaussian distribution, add an appropriate amount of Gaussian noise to each dimension of the input, and use the minimum mean square error criterion for training; in the layers above the first layer, the input features obey the binary distribution, so the With a certain probability, some dimensions of the input features are set to zero, and the minimum cross-entropy criterion is used for training; after pre-training, a K-layer superimposed auto-encoder is obtained, and it is flipped upwards to obtain an up-down symmetrical auto-encoder device structure. 9. speaker's voice conversion method as claimed in claim 6, is characterized in that, described step b2 adopts Fisher's ratio criterion as distinguishing criterion. 10. the speaker's voice conversion method as claimed in claim 9, is characterized in that, described step b3 comprises: A discriminative objective function with a contrastive competition mechanism is designed, and the error backpropagation algorithm is used to fine-tune the network weights of the neural network, so that the neural network has the ability to separate speaker information and content information from acoustic features. 11. the speaker's voice conversion method as claimed in claim 5, is characterized in that, wherein said voice signal database is made through the following steps: Step c1, set up a corpus so that multiple sentences are included in the corpus; Step c2, recording voice signals of multiple speakers reading sentences in the corpus, constructing a voice signal database, and preprocessing the voice signals in the voice signal database to remove abnormal parts in the voice signals; Step c3, using the hidden Markov model to segment the speech signal line in the preprocessed speech signal database, each segment after the segmentation is used as a frame, and the frame-level speaker of each speech signal is obtained Labeling information and content labeling information; Step c4. Randomly combine the speech signals in the speech database to construct training data for the neural network.

Images