KR102298901B1 - Method and Apparatus for Emotional Voice Conversion using Multitask Learning with Text-to-Speech - Google Patents

Abstract

A method and apparatus for emotional speech conversion using multitask learning along with text-to-speech conversion are presented. The emotional speech conversion method using multi-task learning together with text-to-speech conversion proposed in the present invention is a log Mel spectrogram of a language in which a pair of input speech delivers language content and a log Mel spectrogram of a style reference speech ( Mel spectrogram), performing voice conversion (Voice Conversion; VC), when the pair of input voices is a log Mel spectrogram of a one-hot representative text and style reference voice, The step of performing text-to-speech (TTS), the log Mel spectrogram of the language conveying the language content and the one-hot representative text are all mapped in the same space and then converted to a Mel spectrogram. and obtaining a linear spectrum from the decoded Mel spectrogram through a preprocessor.

Description

{Method and Apparatus for Emotional Voice Conversion using Multitask Learning with Text-to-Speech}

The present invention relates to a method and apparatus for emotional speech conversion using multitask learning together with text-to-speech conversion.

Speech can be considered as a construct of what is to be conveyed - the content of the language - and the method of delivery - of the style. Voice Conversion (VC) is the operation of changing the voice style while maintaining the language content. This is a difficult task because language content may be lost or style information may not change during the conversion process.

In the prior art, speech conversion was performed using a frame-based approach. When considering the source and target voices, an alignment between the two voices is obtained, and then the acoustic properties of the source voice are transformed into the target voice. Various methods have been applied to the model acoustic function, such as Gaussian Mixture Model (GMM), Deep Neural Network (DNN), and Recurrent Neural Network (RNN). The success of the sequence-to-sequence (seq2seq) model, which is noteworthy in the recent prior art, is also applied to speech conversion. Problems such as incorrect pronunciation and learning instability were observed while training the seq2seq speech transformation model. In this prior art, the quality of speech conversion is improved by adding text supervision to each step of the decoder output. However, this approach has limitations because it requires explicit alignment by humans or Dynamic Time Wrapping (DTW).

Traditionally, frame-based models have been used to solve VCs. Given the source and target voices, their alignment is found by DTW. We then model the transformation between the acoustic properties of the aligned frames. Recently, the seq2seq model has been proposed in VC. In such a model, the model jointly learns the alignment and frame transformation by the attention mechanism without clear temporal alignment. However, the performance of VC is not sufficient because the converted voice may lose language information. For example, the same word may be repeatedly generated, or some words may be deleted or mispronounced. To prevent this speech phenomenon, we add text supervision to the VC, but we need explicit alignment. Unlike previous work, TTS was used to guide language information to the VC. This approach does not require explicit alignment.

On the other hand, emotional voice transformation mainly performed frame-based transformation or rule-based approach. This approach is limited because DTW does not guarantee that the exact alignment and rule-based approaches have limitations on speech transformation modeling. This can be improved by using a model that does not rely on explicit alignment.

An object of the present invention is to provide a speech conversion method and apparatus using multi-task learning using text-to-speech (TTS). Through the proposed emotional speech conversion method and apparatus, the whole network is trained to minimize the loss of VC and TTS, and the VC tries to capture more language information and preserve learning stability by multi-task learning.

In one aspect, the emotional speech conversion method using multi-task learning together with text-to-speech conversion proposed in the present invention is a log Mel spectrogram of a language in which a pair of input speech conveys language content and a style reference speech. In case of log Mel spectrogram, performing Voice Conversion (VC), the input voice pair is one-hot representative text and log Mel spectrogram of the style reference voice (Mel) spectrogram), performing text-to-speech (TTS), log Mel spectrogram and one-hot representative text of the language conveying language content, all mapped in the same space Then, decoding into a Mel spectrogram and obtaining a linear spectrum from the decoded Mel spectrogram through a preprocessor.

If the pair of input speech is a log Mel spectrogram of a language conveying language content and a log Mel spectrogram of a style reference voice, the step of performing speech conversion is a log Mel spectrogram of the language conveying the language content using a content encoder. and embedding the log Mel spectrogram of the style reference voice through the style encoder.

When the pair of input voices is a log melt spectrogram of a one-hot representative text and a style reference voice, the step of performing text-to-speech is to embed the one-hot representative text through a text encoder, and log Mel spectrogram of the style reference voice. Embed the gram through the style encoder

The log Mel spectrogram and one-hot representative text of the language conveying the language content are both mapped to the same space and then decoded into the Mel spectrogram in each decoding step from the log Mel spectrogram of the style reference voice. The extracted style vector is connected to the attention RNN and the decoder RNN.

The log Mel spectrogram and one-hot representative text of the language conveying the language content are both mapped to the same space and then decoded into a Mel spectrogram through a text encoder, attention RNN, decoder RNN and preprocessor. When performing speech transformation, the context vector is utilized for every iteration of the attention RNN.

When considering the style reference voice, the style encoder only extracts emotion information and removes the linguistic content, is designed to extract the emotion regardless of the linguistic content, processes multiple input style domains, and when the extracted emotion is injected into the decoder It handles many-to-many emotional speech transformations by generating a variety of emotions.

In another aspect, the emotional speech conversion apparatus using multi-task learning together with text-to-speech conversion proposed in the present invention refers to a log Mel spectrogram and style of a language in which a pair of input speech conveys language content. Speech conversion through style encoder, content encoder and text encoder depending on whether it is a log Mel spectrogram of speech or a log Mel spectrogram of a one-hot representative text and style reference speech. Alternatively, a conversion unit that performs text-to-speech conversion, a neural network network that is decoded into a Mel spectrogram after all of the log Mel spectrogram and one-hot representative text of the language that delivers the language content are mapped in the same space and decoded It includes a preprocessor that acquires a linear spectrum from the Mel spectrogram.

According to embodiments of the present invention, through the emotional speech conversion method and apparatus using multitask learning using text-to-speech conversion, the entire network learns to minimize the loss of VC and TTS, and the VC captures more language information and multitasks. Learning stability can be preserved by task learning.

1 is a flowchart illustrating an emotional speech conversion method using multi-task learning together with text-to-speech conversion according to an embodiment of the present invention.
2 is a diagram showing the configuration of an emotional speech conversion apparatus using multi-task learning together with text-to-speech conversion according to an embodiment of the present invention.
3 is a diagram illustrating a congestion matrix of a style encoder according to an embodiment of the present invention.
4 is a diagram illustrating an example of a voice conversion result according to an embodiment of the present invention.

The present invention adopts an approach using text-to-speech (TTS). TTS is the task of converting text or voice information into voice waveforms, and abundant seq2seq-based research has been actively conducted. TTS is a work closely related to voice conversion (VC). VC and TTS differ only in the input domain, but the role of the decoder for converting voice information into an acoustic shape is very the same. The embedding space of TTS has a high correlation with voice information, and VC is expected to learn an embedding space close to TTS through multi-task learning. In the present invention, TTS is used to provide voice information to VC to improve performance.

According to an embodiment of the present invention, this task is extended to emotional speech transformation. When considering the style reference voice, the style encoder only extracts emotional information and removes the linguistic content. Style encoders are designed to extract emotions regardless of their linguistic content, so they can handle multiple input style domains. In addition, when the extracted emotions are injected into the decoder, various emotions can be generated. Therefore, the proposed model can handle many-to-many emotional speech transformation.

The contribution of the model according to the embodiment of the present invention is as follows: Multitasking learning using TTS can improve the performance of VC. A number of emotional speech transformations were initially performed by the seq2seq model. The style reference voice may determine a target region for voice transformation.

The many-to-many VC indicates the number of source domains and the number of target domains. In the prior art, Cycle-GAN converts a non-dataset speaker into a target speaker, and vice versa. An i-vector based VC system is proposed to generate linguistic features of speakers that are not in the training set. Compared with other many-to-many VC schemes, the proposed model can convey the emotional knowledge of the speaker and enable conversion between different emotions.

TTS is one of the most active research areas in the speech domain. Various kinds of Seq2seq models have been proposed and their expressive synthesis has also been studied. The style vector extracted from the style encoder generates emotional voice by injecting the input emotion into the network. In the present invention, emotion labels are not utilized during learning, and no emotion labels are explicitly accepted as input to the network. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a flowchart illustrating an emotional speech conversion method using multi-task learning together with text-to-speech conversion according to an embodiment of the present invention.

Voice transformation (VC) is the operation of changing a human voice into a different style while preserving the linguistic content. The prior state-of-the-art technology for VC is based on a sequence-to-sequence (seq2seq) model that can mislead linguistic information. Attempts have been made to overcome this by using text monitoring, which requires a clear alignment that loses the benefits of using the seq2seq model. In the present invention, speech conversion using multi-task learning using text-to-speech (TTS) is presented. The embedding of seq2seq-based TTS has rich information about the text. The role of the TTS decoder is to convert the built-in space into VC-like speech. In the proposed model, the entire network is trained to minimize the loss of VC and TTS. VC is expected to capture more language information and preserve learning stability by multi-task learning. Experiments in VC were performed on the male emotional text-to-speech dataset in Korea, and show that multi-task learning helps preserve verbal content in VC.

The proposed method for emotional speech conversion using multitask learning with text-to-speech conversion is the log Mel spectrogram of the language in which the pair of input speech conveys the language content and the log Mel spectrogram of the style reference speech. If , performing voice conversion (Voice Conversion; VC) step 110, if the input voice pair is a one-hot representative text and a log Mel spectrogram of a style reference voice, Step 120 of performing text-to-speech (TTS), the log-mel spectrogram of the language conveying the language content, and the one-hot representative text are all mapped in the same space, and then It includes a step 130 of decoding the spectrogram and a step 140 of obtaining a linear spectrum from the decoded Mel spectrogram through a preprocessor.

In step 110, when the pair of input voices is a log Mel spectrogram of a language that conveys language content and a log Mel spectrogram of a style reference voice, Voice Conversion (VC) carry out The log Mel spectrogram of the language that delivers the language content is embedded through the content encoder, and the log Mel spectrogram of the style reference voice is embedded through the style encoder.

In step 120, when the pair of input speech is a log Mel spectrogram of a one-hot representative text and a style reference speech, text-to-speech (TTS) is performed. carry out The one-hot representative text is embedded through the text encoder, and the log Mel spectrogram of the style reference voice is embedded through the style encoder.

In step 130, both the log Mel spectrogram and the one-hot representative text of the language conveying the language content are mapped to the same space and then decoded into the Mel spectrogram. In each decoding step, the style vector extracted from the log Mel spectrogram of the style reference voice is connected to the attention RNN and the decoder RNN. In this case, when text-to-speech conversion is performed through the text encoder, attention RNN, decoder RNN, and preprocessor, a context vector is utilized for all repetitions of the attention RNN.

In step 140, a linear spectrum is obtained from the decoded Mel spectrogram through a preprocessor.

According to an embodiment of the present invention, when considering a style reference voice, the style encoder is designed to extract only emotion information and remove linguistic content, and is designed to extract emotions regardless of linguistic content to process a plurality of input style domains, When the extracted emotion is injected into the decoder, it processes many-to-many emotional speech conversion by generating various emotions.

More specifically, the proposed emotional speech transformation method can perform VC and TTS in a single model. The network is played with _{VC when the input pair is (x c} , x _s ) or TTS when the input pair is (y _t , x _{s ).} Here, x _c , y _t , and x _s are the log Mel spectrogram of the language that conveys the language content, the one-hot representative text, and the log Mel spectogram of the style reference voice. Both x _c and y _t are mapped to the same space h _l and then decoded into Mel spectogram m. In each decoding step, the style vector h _{s extracted from x s} is connected to the attention RNN and the decoder RNN. A linear spectrum l is obtained by the preprocessor. A detailed network architecture will be described with reference to FIG. 2 and the equation below.

2 is a diagram showing the configuration of an emotional speech conversion apparatus using multi-task learning together with text-to-speech conversion according to an embodiment of the present invention.

The apparatus for emotional speech conversion using multitask learning together with the proposed text-to-speech conversion includes a conversion unit 210 , a neural network 220 , and a preprocessor 230 .

The conversion unit 210 determines whether the pair of input voices is a log Mel spectrogram (x _c ) of a language that transmits language content and a log Mel spectrogram of a style reference voice or one-hot (one). -hot) Speech conversion through the style encoder 211, the content encoder 212 and the text encoder 213 depending on whether it is a log Mel spectrogram (x _s ) of the representative text (y _{t ) and the style reference voice} Or perform text-to-speech conversion.

When the pair of input voices is a log Mel spectrogram (x _c ) of a language transmitting language content and a log Mel spectrogram (x _s ) of a style reference voice, it performs voice conversion. The log Mel spectrogram (x _c ) of the language delivering the language content is embedded through the content encoder 212 , and the log Mel spectrogram (x _s ) of the style reference voice is embedded through the style encoder 211 .

The conversion unit 210 performs text-to-speech conversion when the pair of input voices is the one-hot representative text (y _t ) and the log Mel spectrogram (x _{s) of the style reference voice.} The one-hot representative text (y _t ) is embedded through the text encoder 213 , and the log Mel spectrogram (x _s ) of the style reference voice is embedded through the style encoder 211 .

The neural network 220 is a log Mel spectrogram (x _c ) and a one-hot representative text (y _t ) of the language conveying the language content after being mapped in the same space as a Mel spectrogram (m). decoded The neural network 220 connects the style vector extracted from the log Mel spectrogram of the style reference voice to the attention RNN 221 and the decoder RNN 222 in each decoding step.

The preprocessor 230 obtains a linear spectrum l from the decoded Mel spectrogram m. The linear spectrum (l) outputs a _{converted voice (x o ) through a vocoder (240).}

According to an embodiment of the present invention, when considering a style reference voice, the style encoder is designed to extract only emotion information and remove linguistic content, and is designed to extract emotions regardless of linguistic content to process a plurality of input style domains, When the extracted emotion is injected into the decoder, it processes many-to-many emotional speech conversion by generating various emotions.

More specifically, VC and TTS can be performed in a single model of the proposed emotional speech conversion device. The network is played with _{VC when the input pair is (x c} , x _s ) or TTS when the input pair is (y _t , x _{s ).} Here, x _c , y _t , and x _s are the log Mel spectrogram of the language that conveys the language content, the one-hot representative text, and the log Mel spectogram of the style reference voice. Both x _c and y _t are mapped to the same space h _l and then decoded into Mel spectogram m. In each decoding step, the style vector h _{s extracted from x s} is connected to the attention RNN and the decoder RNN. A linear spectrum l is obtained by the preprocessor.

Here, h _t , h _c , and h _s are embeddings of the text encoder 213 , the content encoder 212 , and the style encoder 211 , respectively. h _att ^(t) and h _dec ^(t) are the hidden representations of the attention RNN and the decoder RNN at time step t. m, l, c ^(t) and o ^(t) are the log Mel spectrogram at time step t, the log linear spectrogram of the object, the context vector achieved by the attention mechanism, and the attention RNN output at time step t. XOR stands for an exclusive OR operator.

For the TTS part including the text encoder, decoder, attention RNN and preprocessor, the overall architecture is based on the tacotron, the context vector c ^(t) is utilized for every iteration of the attention RNN, and the residual concatenation is the Convolution Bank + Highway + bi-GRU (CBHG) connections are added.

The text encoder consists of a character embedding layer, prenet, and CBHG, where the prenet consists of two FC ReLU-Dropout layers. The LSTM stack is used for content encoders, and there was no decrease in temporal resolution because temporal reduction could result in loss of local temporal information. This means that _{the length of h c} is equal to x _{c .} For the style encoder, x _{s is mapped to h s} with fixed dimensions by embedding a fully connected layer following the last step in LSTM to reduce dimensions. The attention module consists of the attention RNN and the attention mechanism. Note RNNs take h _s , c ^(t-1) , m ^(t-1) as inputs. Then its outputs o ^(t) and h _l are used in the attention mechanism to generate the context vector c ^(t).

The learning loss is Im-mgtI+II-IgtI, where mgt and lgt are the ground truth log Mel spectrogram and log linear spectrogram.

According to an embodiment of the present invention, a Korean Emotional Text-to-speech (mKETTS) data set was constructed. For example, a 30-year-old man measured for the experiment pronounced 3,000 sentences with 7 different emotions (neutral, happy, sad, anger, fear, surprise, disgust), so the total number of utterances was 21 1 up to a thousand The text of every sentence is the same throughout the emotion. All recordings were made in a quiet, silent studio and recorded at a sampling rate of 44.1 kHz. The total time after refinement of the stillness is 29.2 hours.

For pre-processing, the static at the beginning and end of each waveform is trimmed using a voice activity detection (VAD) algorithm and resampled at 16 kHz. Then, the log Mel spectrogram is extracted with a window size of 50 ms, a shift of 12.5 ms, nfft 2048, 80 Mel bins and Hanning windows. The size was normalized to [0, 1]. Since the text encoder uses a character-based representation, Korean characters are decomposed into onset, nucleus, and coda, and then converted into a one-hot representation.

Detailed parameter settings are as follows. According to an embodiment of the present invention, 256-character embeddings, 32-dimensional hs were used. The initial learning rate was 1e-3 for the Adam optimum. Gradient clipping was used as 1, and m was created with a reduction factor of 5. The batch size was 32, and Bahdanau attention was used. The content encoder uses two layers for the bidirectional LSTM and the style encoder, and the content encoder consists of two layers for the unidirectional LSTM. The output of the last step is only used by the content encoder. The parameters of the content encoder and the style encoder are not shared. In the learning phase, teacher forcing was used to prevent the cumulative loss of prediction products. When making a mini-batch, each sample is zero-padded to the longest length of the sample. Then, the loss of the zero-faded region is also reverse transcribed for speculation stability. For every iteration, the network's operation is randomly determined by VC or TTS. For each sample, the source and target emotions are chosen differently.

To verify the linguistic consistency of the proposed model, three different models were trained and evaluated. VCTS is a model that combines TTS and VC as described above. VC refers to a voice converter without a TTS path. TTS refers to a TTS model that does not include a VC path. VCTTS-V is a speech transformation inference model using VCTTS, and VCTTS-T is a model when the TTS pathway of VCTTS is activated.

The word error rate (WER) was calculated to measure how the proposed model improves the consistency of the transformed language. In fact, morphemes are used in place of words because they are considered the recognition units of Korean. The Google Cloud speech-to-text API recorded the converted speeches, and the transcripts were broken down into a series of morphemes by KoNLPy's Komoran morphological analyzer. Then, the average WER between the two morphological sequences of the actual record and the automatically recognized record was calculated and displayed in Table 1. As a result, VCTTS-V was superior to VC, and WER of TTS was superior to VCTTS-T.

<Table 1>

After learning, 8 native speakers participated in the subjective evaluation. Twenty sentences were generated by the VCTTS-V model. The emotion of x _c was set to neutral, the target emotion was set to happiness, and the sentence of _{x s was fixed.} Blindly test that subjects never know which model produces which voice. Subjects were asked to rate their clarity on a scale of 1 to 5. In addition, a preference ABX test between the two models was conducted. Given two voices with no information about which model generated which sample, subjects were asked to choose the clearer voice. Subjects are not allowed to choose either one if the two samples are perceived to be similar. The results are shown in Table 2. It shows that VCTTV-V has high MOS and ABX preference scores, which means that multitask learning of VC and TTS helps retain language information.

<Table 2>

3 is a diagram illustrating a congestion matrix of a style encoder according to an embodiment of the present invention.

To investigate emotional speech transformation, the above-mentioned VCTTS model is used for inference. After training, 20 samples were randomly selected for each emotion, and the samples were fed to the model. Then, h _s per sample can be obtained, and the cosine similarity between each sample is measured and shown in FIG. 3 . The average value of cosine similarity between all emotion pairs is also displayed.

In FIG. 3 , it is shown that the sample of the same emotion has a very high cosine similarity, whereas the emotion pair other than the diagonal line has a low similarity. It implies that the style encoder can extract emotional styles regardless of linguistic content. Even with the exception of the diagonal emotion pair, the emotion pair (disgust-anger) shows a relatively high degree of similarity, which means that the immanence of these two emotions is closer than the other emotions. The same voice is observed in the (sad-fear) pair.

4 is a diagram illustrating an example of a voice conversion result according to an embodiment of the present invention.

Emotional speech transformations must reflect emotions while preserving the verbal content. 4 shows an example of voice conversion. When given a neutral voice, a given x _s transforms into six different emotions. The content of x _s is fixed during this experiment. The first row of FIG. 4 is the log Mel spectrogram of the input voice, and the second to the seventh row is the log Mel spectrogram of the converted voice. While the overall shape of the spectrogram is similar to the input, it can be seen that some changes such as time shift, frequency shift, and pause time have been made. Within a single model, it can create multiple language domains.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (12)

performing voice conversion (Voice Conversion; VC) when the pair of input voices is a log Mel spectrogram of a language transmitting language content and a log Mel spectrogram of a style reference voice;
performing text-to-speech (TTS) when the pair of input speech is a log Mel spectrogram of a one-hot representative text and a style reference speech;
A log Mel spectrogram and a one-hot representative text of a language conveying language content are both mapped to the same space and then decoded into a Mel spectrogram; and
Obtaining a linear spectrum from the decoded Mel spectrogram through a preprocessor
including,
When the pair of input voices is a log Mel spectrogram of a language conveying language content and a log Mel spectrogram of a style reference voice, the step of performing voice conversion includes:
Embedding the log Mel spectrogram of the language that conveys the language content through the content encoder, and embedding the log Mel spectrogram of the style reference voice through the style encoder.
Emotional speech transformation method. delete According to claim 1,
When the pair of input speech is a log Mel spectrogram of one-hot representative text and style reference speech, the step of performing text-to-speech conversion includes:
Embedding the one-hot representative text through the text encoder and embedding the log Mel spectrogram of the style reference voice through the style encoder
Emotional speech transformation method. According to claim 1,
The log Mel spectrogram and one-hot representative text of the language conveying language content are both mapped to the same space and then decoded into a Mel spectrogram,
At each decoding step, the style vector extracted from the log Mel spectrogram of the style reference voice is connected to the attention RNN and the decoder RNN.
Emotional speech transformation method. According to claim 1,
The log Mel spectrogram and one-hot representative text of the language conveying language content are both mapped to the same space and then decoded into a Mel spectrogram,
When text-to-speech is performed through a text encoder, attention RNN, decoder RNN, and preprocessor, the context vector is utilized for every iteration of the attention RNN.
Emotional speech transformation method. According to claim 1,
When considering the style reference voice, the style encoder only extracts emotion information and removes the linguistic content, is designed to extract the emotion regardless of the linguistic content, processes multiple input style domains, and when the extracted emotion is injected into the decoder processing many-to-many emotional speech by generating a variety of emotions
Emotional speech transformation method. Whether a pair of input voices is a log Mel spectrogram of a language and a log Mel spectrogram of a style-referenced voice conveying the linguistic content, or a one-hot representative text and a log-mel spectrogram of a style-referenced voice. a conversion unit that performs speech-to-speech or text-to-speech conversion through a style encoder, a content encoder, and a text encoder according to whether it is a log Mel spectrogram;
A neural network in which both log Mel spectrogram and one-hot representative text of a language conveying language content are mapped to the same space and then decoded into Mel spectrogram; and
A preprocessor that acquires a linear spectrum from the decoded Mel spectrogram
including,
conversion unit,
If the pair of input voices is a log Mel spectrogram of a language conveying language content and a log Mel spectrogram of a style reference voice, perform voice conversion;
Embedding the log Mel spectrogram of the language that conveys the language content through the content encoder, and embedding the log Mel spectrogram of the style reference voice through the style encoder.
Emotional Speech Transducer. delete 8. The method of claim 7,
conversion unit,
If the pair of input speech is a log Mel spectrogram of one-hot representative text and style reference speech, perform text-to-speech conversion;
Embedding the one-hot representative text through the text encoder and embedding the log Mel spectrogram of the style reference voice through the style encoder
Emotional Speech Transducer. 8. The method of claim 7,
The neural network is
At each decoding step, the style vector extracted from the log Mel spectrogram of the style reference voice is connected to the attention RNN and the decoder RNN.
Emotional Speech Transducer. 8. The method of claim 7,
The neural network is
When text-to-speech is performed through a text encoder, attention RNN, decoder RNN, and preprocessor, the context vector is utilized for every iteration of the attention RNN.
Emotional Speech Transducer. 8. The method of claim 7,
When considering the style reference voice, the style encoder only extracts emotion information and removes the linguistic content, is designed to extract the emotion regardless of the linguistic content, processes multiple input style domains, and when the extracted emotion is injected into the decoder processing many-to-many emotional speech by generating a variety of emotions
Emotional Speech Transducer.

Images