KR102568145B1 - Method and tts system for generating speech data using unvoice mel-spectrogram - Google Patents

Abstract

A method of generating voice data using speaker information and text includes generating a plurality of sub-mel-spectrograms based on the speaker information and text, and generating voice data from the plurality of sub-mel-spectrograms. can include
The step of generating voice data includes generating a final mel-spectrogram by adding a silent mel-spectrogram between a plurality of sub mel-spectrograms, and generating voice data from the final mel-spectrogram. can do.

Description

METHOD AND TTS SYSTEM FOR GENERATING SPEECH DATA USING UNVOICE MEL-SPECTROGRAM

The present disclosure relates to a method and a speech synthesis system for generating speech data using a silent mel-spectrogram.

Recently, with the development of artificial intelligence technology, an interface using a voice signal is becoming common. Accordingly, research on a speech synthesis technology capable of uttering a synthesized voice according to a given situation is being actively conducted.

Speech synthesis technology is applied to many fields, such as virtual assistants, audio books, automatic interpretation and translation, and virtual voice actors, in conjunction with artificial intelligence-based voice recognition technology.

Conventional speech synthesis methods include various methods such as Unit Selection Synthesis (USS) and HMM-based Speech Synthesis (HTS). The USS method is a method of cutting and storing voice data in phoneme units and searching for and attaching sound pieces suitable for speech during speech synthesis. way to reconstruct it. However, the conventional voice synthesis method described above has many limitations in synthesizing a natural voice reflecting a speaker's speech style or emotional expression.

Accordingly, recently, a speech synthesis method for synthesizing speech from text based on an artificial neural network is attracting attention.

It is an object of the present invention to provide an artificial intelligence-based voice synthesis technology capable of realizing a natural voice as if a real speaker is speaking. In addition, it is an object of the present invention to provide an artificial intelligence-based voice synthesis technology capable of realizing a natural voice as if a real speaker is speaking. In addition, it is to provide a high-efficiency artificial intelligence-based voice synthesis technology using a small amount of learning data.

The technical problem to be solved is not limited to the technical problems as described above, and other technical problems can be inferred.

As a technical means for achieving the above-described technical problem, a first aspect of the present disclosure is a method for generating voice data using speaker information and text, wherein a plurality of sub-mel-spectrograms are provided based on speaker information and text. generating; and generating voice data from the plurality of sub-mel-spectrograms, wherein the generating of voice data comprises adding a silent mel-spectrogram between the plurality of sub-mel-spectrograms to obtain a final mel-spectrogram. - generating a spectrogram; It is possible to provide a method including; generating voice data from the final mel-spectrogram.

Also, in the generating of the final mel-spectrogram, identifying the last character of the text corresponding to the sub mel-spectrogram; If the last character is a first group character, a silent mel-spectrogram having a first time is added to the sub-mel-spectrogram, and if the last character is a second-group character, to the sub-mel-spectrogram generating a final mel-spectrogram by adding a silent mel-spectrogram having a second time;

The method further includes obtaining breath data as the speaker information, and the generating of the voice data includes the breath sound generated from the breath data between the plurality of sub-mel-spectrograms. adding spectrograms to generate a final mel-spectrogram; and generating voice data from the final mel-spectrogram.

According to the above-mentioned problem solving means of the present disclosure, by dividing the mel-spectrogram into sub-mel-spectrograms based on the silent part of the mel-spectrogram and generating voice data from the sub-mel-spectrogram, more accurate Voice data can be generated.

According to one of the means for solving other problems of the present disclosure, more accurate voice data can be generated by generating voice data using a silent mel-spectrogram.

1 is a diagram schematically illustrating the operation of a speech synthesis system.
2 is a diagram illustrating an embodiment of a speech synthesis system.
3 is a diagram illustrating an embodiment of a synthesizer of a speech synthesis system.
4 is a diagram illustrating an embodiment of outputting a Mel spectrogram through a synthesizer.
5 is a diagram illustrating a volume graph corresponding to a mel-spectrogram according to an embodiment.
6A and 6B are diagrams for explaining a process of dividing a mel-spectrogram into a plurality of sub mel-spectrograms according to an embodiment.
7 is a diagram for explaining a process of generating voice data from a plurality of sub-MEL-spectrograms according to an embodiment.
8 is a diagram for explaining an example of segmenting a text sequence according to an exemplary embodiment.
9 is a diagram for explaining a process of generating a final mel-spectrogram by adding a silent mel-spectrogram between sub mel-spectrograms according to an embodiment.
10 is a flowchart of a method for determining a silent portion of a Mel-spectrogram according to an embodiment.

The terms used in the present embodiments have been selected from general terms that are currently widely used as much as possible while considering the functions in the present embodiments, but these may vary depending on the intention of a person skilled in the art or a precedent, the emergence of new technologies, etc. there is. In addition, in a specific case, there are also terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the relevant part. Therefore, the term used in the present embodiments should be defined based on the meaning of the term and the general content of the present embodiment, not a simple name of the term.

Since the present embodiments can have various changes and various forms, some embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present embodiments to a specific disclosure, and should be understood to include all changes, equivalents, or substitutes included in the spirit and scope of the present embodiments. Terms used in this specification are only used for description of the embodiments, and are not intended to limit the embodiments.

Terms used in the present embodiments have the same meaning as commonly understood by a person of ordinary skill in the art to which the present embodiments belong, unless otherwise defined. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present embodiments, in an ideal or excessively formal meaning. should not be interpreted.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description of the present invention which follows refers to the accompanying drawings which illustrate, by way of illustration, specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable any person skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different from each other but are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented from one embodiment to another without departing from the spirit and scope of the present invention. It should also be understood that the location or arrangement of individual components within each embodiment may be changed without departing from the spirit and scope of the present invention. Therefore, the detailed description to be described later is not performed in a limiting sense, and the scope of the present invention should be taken as encompassing the scope claimed by the claims and all scopes equivalent thereto. Like reference numbers in the drawings indicate the same or similar elements throughout the various aspects.

Meanwhile, technical features individually described in one drawing in this specification may be implemented individually or simultaneously.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily practice the present invention.

1 is a diagram schematically illustrating the operation of a speech synthesis system.

A speech synthesis device is a device that converts text into human voice.

For example, the voice synthesizing system 100 of FIG. 1 may be an artificial neural network based voice synthesizing system. An artificial neural network refers to a model in general that has problem-solving ability by changing synaptic coupling strength through learning of artificial neurons that form a network through synaptic coupling.

The voice synthesis system 100 may be implemented in various types of devices such as a personal computer (PC), a server device, a mobile device, and an embedded device, and as a specific example, a smartphone or a tablet that performs voice synthesis using an artificial neural network. Devices, Augmented Reality (AR) devices, Internet of Things (IoT) devices, self-driving cars, robotics, medical devices, e-book readers, and navigation devices may be used, but are not limited thereto.

Furthermore, the voice synthesis system 100 may correspond to a dedicated hardware accelerator (HW accelerator) mounted in the above device. Alternatively, the speech synthesis system 100 may be a hardware accelerator such as a neural processing unit (NPU), a tensor processing unit (TPU), or a neural engine, which are dedicated modules for driving an artificial neural network, but is not limited thereto.

Referring to FIG. 1 , the speech synthesis system 100 may receive text input and specific speaker information. For example, the speech synthesis system 100 may receive “Have a good day!” as a text input as shown in FIG. 1 and may receive “speaker 1” as a speaker information input.

“Speaker 1” may correspond to a voice signal or a voice sample representing a preset speech characteristic of speaker 1. For example, speaker information may be received from an external device through a communication unit included in the voice synthesis system 100 . Alternatively, speaker information may be input from a user through the user interface of the speech synthesis system 100 and may be selected from among various types of speaker information pre-stored in the database of the speech synthesis system 100, but is not limited thereto. no.

The speech synthesis system 100 may output speech based on the received text input and specific speaker information. For example, the speech synthesis system 100 may "Have a good day!" and “Speaker 1” are received as an input, and a voice for “Have a good day!” reflecting the speech characteristics of Speaker 1 may be output. The speech characteristics of speaker 1 may include at least one of various factors such as voice, prosody, pitch, and emotion of speaker 1. That is, the output voice may be a voice as if speaker 1 naturally pronounces "Have a good day!" A detailed operation of the voice synthesis system 100 will be described later with reference to FIGS. 2 to 4 .

2 is a diagram illustrating an embodiment of a speech synthesis system. The voice synthesis system 200 of FIG. 2 may be the same as the voice synthesis system 100 of FIG. 1 .

Referring to FIG. 2 , a speech synthesis system 200 may include a speaker encoder 210 , a synthesizer 220 and a vocoder 230 . Meanwhile, in the voice synthesizing system 200 shown in FIG. 2, only components related to one embodiment are shown. Accordingly, it is obvious to those skilled in the art that the voice synthesis system 200 may further include other general-purpose components in addition to the components shown in FIG. 2 .

The voice synthesis system 200 of FIG. 2 may output speech by receiving speaker information and text as inputs.

For example, the speaker encoder 210 of the speech synthesis system 200 may receive speaker information as an input and generate a speaker embedding vector. Speaker information may correspond to a speaker's voice signal or voice sample. The speaker encoder 210 may receive the speaker's voice signal or voice sample, extract the speaker's speech characteristics, and represent the speech characteristics as an embedding vector.

The speaker's speech characteristics may include at least one of various factors such as speech speed, pause period, pitch, timbre, prosody, intonation, or emotion. That is, the speaker encoder 210 may represent discontinuous data values included in speaker information as vectors composed of consecutive numbers. For example, the speaker encoder 210 includes a pre-net, a CBHG module, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), and a BRDNN ( A speaker embedding vector may be generated based on at least one or a combination of two or more of various artificial neural network models such as Bidirectional Recurrent Deep Neural Network).

For example, the synthesizer 220 of the speech synthesis system 200 may output a spectrogram by receiving text and an embedding vector representing speech characteristics of a speaker as inputs.

3 is a diagram illustrating an embodiment of a synthesizer of a speech synthesis system. The synthesizer 300 of FIG. 3 may be the same as the synthesizer 220 of FIG. 2 .

Referring to FIG. 3 , the synthesizer 300 of the speech synthesis system 200 may include an encoder and a decoder. Meanwhile, it is obvious to those skilled in the art that the synthesizer 300 may further include other general-purpose components.

The embedding vector representing the speech characteristics of the speaker may be generated by the speaker encoder 210 as described above, and the encoder or decoder of the synthesizer 300 may receive the embedding vector representing the speech characteristics of the speaker from the speaker encoder 210. can

The encoder of synthesizer 300 may receive text as an input and generate a text embedding vector. Text may contain sequences of characters in a particular natural language. For example, a sequence of characters may include alphabetic characters, numbers, punctuation marks, or other special characters.

The encoder of the synthesizer 300 may divide input text into consonant units, character units, or phoneme units, and may input the separated text into an artificial neural network model. For example, the encoder of the synthesizer 300 may generate a text embedding vector based on at least one or a combination of two or more of various artificial neural network models such as pre-net, CBHG module, DNN, CNN, RNN, LSTM, and BRDNN. .

Alternatively, the encoder of the synthesizer 300 may divide the input text into a plurality of short texts and generate a plurality of text embedding vectors for each of the short texts.

The decoder of the synthesizer 300 may receive the speaker embedding vector and the text embedding vector from the speaker encoder 210 as inputs. Alternatively, the decoder of the synthesizer 300 may receive the speaker embedding vector from the speaker encoder 210 as an input and the text embedding vector from the encoder of the synthesizer 300 as an input.

The decoder of the synthesizer 300 may generate a spectrogram corresponding to the input text by inputting the speaker embedding vector and the text embedding vector to the artificial neural network model. That is, the decoder of the synthesizer 300 may generate a spectrogram of the input text in which speech characteristics of the speaker are reflected. For example, the spectrogram may correspond to a mel-spectrogram, but is not limited thereto.

A spectrogram is a graph that visualizes the spectrum of a voice signal. The x-axis of the spectrogram represents time, and the y-axis represents frequency, and the value of each frequency per time can be expressed in color according to the size of the value. The spectogram may be a result of performing short-time Fourier transform (STFT) on a continuously given audio signal.

STFT is a method of dividing a speech signal into sections of a certain length and applying a Fourier transform to each section. At this time, since the result of performing the STFT on the voice signal is a complex value, it is possible to generate a spectrogram including only magnitude information by taking an absolute value of the complex value to lose phase information.

Meanwhile, the mel spectrogram is obtained by re-adjusting the frequency interval of the spectrogram to a mel scale. The human auditory organ is more sensitive in the low frequency band than the high frequency band, and the Mel scale reflects this characteristic and expresses the relationship between the physical frequency and the frequency perceived by the actual person. The Mel spectrogram may be generated by applying a filter bank based on the Mel scale to the spectrogram.

Meanwhile, although not shown in FIG. 3 , the synthesizer 300 may further include an attention module for generating an attention alignment. The attention module is a module that learns which output of a specific time-step of the decoder is most related to an output of all time-steps of the encoder. A higher quality spectrogram or MEL spectrogram can be output by using the attention module.

4 is a diagram illustrating an embodiment of outputting a Mel spectrogram through a synthesizer. The synthesizer 400 of FIG. 4 may be the same as the synthesizer 300 of FIG. 3 .

Referring to FIG. 4 , the synthesizer 400 may receive a list including input texts and speaker embedding vectors corresponding thereto. For example, the synthesizer 400 inputs the input text 'first sentence', the corresponding speaker embedding vector embed_voice1, the input text 'second sentence' and the corresponding speaker embedding vector embed_voice2, and the corresponding input text 'third sentence' and a list 410 including embed_voice3, which is a speaker embedding vector corresponding thereto, may be received as an input.

The combiner 400 may generate as many mel spectrograms 420 as the number of input texts included in the received list 410 . Referring to FIG. 4 , it can be seen that Mel spectrograms corresponding to each input text of 'first sentence', 'second sentence', and 'third sentence' are generated.

Alternatively, the synthesizer 400 may generate as many mel spectrograms 420 and attention alignment as the number of input texts. Although not shown in FIG. 4 , for example, attention alignments corresponding to input texts of 'first sentence', 'second sentence', and 'third sentence' may be additionally generated. Alternatively, the synthesizer 400 may generate a plurality of MEL spectrograms and a plurality of attention alignments for each of the input texts.

Returning to FIG. 2 again, the vocoder 230 of the voice synthesis system 200 may generate a spectrogram output from the synthesizer 220 as actual speech. As described above, the output spectrogram may be a MEL spectrogram.

In one embodiment, the vocoder 230 may generate a spectrogram output from the synthesizer 220 as an actual voice signal using an inverse short-time Fourier transform (ISFT). Since the spectrogram or mel spectrogram does not include phase information, the phase information of the spectrogram or mel spectrogram is not considered when a voice signal is generated using the ISFT.

In another embodiment, the vocoder 230 may generate a spectrogram output from the synthesizer 220 as an actual voice signal using a Griffin-Lim algorithm. The Griffin-Rim algorithm is an algorithm for estimating phase information from magnitude information of a spectrogram or a Mel spectrogram.

Alternatively, the vocoder 230 may generate a spectrogram output from the synthesizer 220 as an actual voice signal based on, for example, a neural vocoder.

A neural vocoder is an artificial neural network model that generates a voice signal by receiving a spectrogram or a mel spectrogram as an input. The neural vocoder can learn a relationship between a spectrogram or a MEL spectrogram and a voice signal through a large amount of data, and through this, it can generate a real voice signal of high quality.

The neural vocoder may correspond to a vocoder based on an artificial neural network model such as WaveNet, Parallel WaveNet, WaveRNN, WaveGlow, or MelGAN, but is not limited thereto.

For example, a WaveNet vocoder is composed of several dilated causal convolution layers and is an autoregressive model using sequential features between voice samples. WaveRNN vocoder is an autoregressive model that replaces multiple dilated causal convolution layers of WaveNet with GRU (Gated Recurrent Unit). The WaveGlow vocoder can learn to produce a simple distribution such as a Gaussian distribution from the spectrogram dataset (x) using an invertible transform function. After learning, the WaveGlow vocoder can output a voice signal from Gaussian distribution samples using the inverse function of the conversion function.

5 is a diagram illustrating a volume graph corresponding to a mel-spectrogram according to an embodiment.

The Mel-spectrogram 520 may include a plurality of frames. Referring to FIG. 5 , a Mel-spectrogram 520 may include 400 frames. The processor may generate the volume graph 510 by calculating the average energy of each frame. Among the frames of the Mel-spectrogram 520, a dark-colored portion (eg, a yellow portion) has a high volume value. The volume graph 510 generated from the mel-spectrogram 520 has a maximum value of 4.0 and a minimum value of -4.0.

The larger the average energy of a frame is, the larger the volume value is, and the smaller the average energy of a frame is, the smaller the volume value is. That is, a frame having a low average energy may correspond to a silent part.

The processor may determine a silent portion in the mel-spectrogram 520 . The processor may generate the volume graph 510 by calculating a volume value for each of a plurality of frames constituting the mel-spectrogram 520 .

The processor may select at least one frame having a volume value less than or equal to the first threshold value 511 from among the plurality of frames as the first intervals 521a to 521f.

In one embodiment, the processor may determine the first period 521a to 521f as a silent part of the Mel-spectrogram 520 . For example, the first threshold value 511 may be -3.0, -3.25, -3.5, -3.75, etc., but is not limited thereto. The first threshold 511 may be set differently depending on how much noise is included in the mel-spectrogram 520 . In the case of a noisy mel-spectrogram 520, the first threshold 511 may be set to a larger value.

In another embodiment, the processor may select, as the second intervals 521c and 521e, intervals in which the number of frames is equal to or greater than the second threshold value among the first intervals 521a to 521f. The processor may determine the second sections 521c and 521e of the Mel-spectrogram 520 as the silent part. For example, the second threshold may be 3, 4, 5, 6, 7, etc., but is not limited thereto. When generating a voice with the Mel-spectrogram 520, a second threshold value may be determined based on an overlap value and a hop size set in WaveRNN, which is one of the vocoders. Overlap means the length of crossfading between batches when creating voice data in WaveRNN. For example, when the overlap value is 1200 and the hop size is 300, the second threshold value may be set to 4 or 5 because it is desirable that the volume value of four consecutive frames be less than the first threshold value 511.

Referring to FIG. 5 , the processor may determine sections [[123, 132, 141], [280, 283, 286]] of the Mel-spectrogram 520 as the silent part. The above list means [silent part start point, middle value, silent part end point]. On the other hand, even if the volume value of the frames included in the first section 521a is less than the first threshold value 511 and the number of frames is greater than or equal to the second threshold value, the first section 521a starts with a voice. Since it is a section, it can be excluded from the silent part. However, when generating voice data from the Mel-spectrogram 520 later, the processor may set the starting point of the voice to after the first section 521a.

6A and 6B are diagrams for explaining a process of dividing a mel-spectrogram into a plurality of sub mel-spectrograms according to an embodiment.

The Mel-spectrogram 620 of FIG. 6A may correspond to the Mel-spectrogram 520 of FIG. 5 .

The processor may divide the mel-spectrogram 620 into a plurality of sub mel-spectrograms 631 , 632 , and 633 based on the second sections 521c and 521e determined as the silent parts in FIG. 5 . When the first second interval 521c is [123, 132, 141] and the second second interval 521e is [280, 283, 286], the processor determines the middle value of the first second interval 521c. may be determined as the first splitting point 641, and the middle value of the second section 521e may be determined as the second splitting point 642.

The processor may generate a plurality of sub mel-spectrograms 631, 632, and 633 by dividing the mel-spectrogram 620 based on the first dividing point 641 and the second dividing point 642.

The processor may calculate the length of each of the plurality of sub-MEL-spectrograms 631, 632, and 633. Since the processor divides the Mel-spectrogram 620 into a plurality of sub-Mel-spectrograms 631, 632, and 633 based on the silent part of the Mel-spectrogram 620, a plurality of sub-Mel-spectrograms 631 , 632, 633) may have different lengths.

Referring to FIG. 6A, the first sub-Mel-spectrogram 631 has a length of 132 corresponding to the interval [0, 132], and the second sub-Mel-spectrogram 632 has a length of [133, 283]. It has a length of 150, and the third sub-Mel-spectrogram 633 has a length of 114, corresponding to [284, 398].

Referring to FIG. 6B, the processor generates a plurality of sub-MEL-spectrograms 631, 632, and 633 such that the lengths of the plurality of sub-MEL-spectrograms 631, 632, and 633 are equal to the length of a reference batch. can be post-processed. In one embodiment, the reference batch length may be a preset value.

In another embodiment, the reference batch length may be set to the length of the sub-Mel-spectrogram having the longest length among the plurality of sub-MEL-spectrograms 631, 632, and 633. For example, if the length of the first sub-Mel-spectrogram 631 is 132, the length of the second sub-Mel-spectrogram 632 is 150, and the length of the third sub-Mel-spectrogram 633 is 114. In this case, the reference batch length may be set to 150.

The processor performs zero-padding on sub-Mel-spectrograms having a length less than the reference batch length so that the lengths of the plurality of sub-MEL-spectrograms 631, 632, and 633 are equal to the reference batch length. -padding) can be applied. For example, when the reference batch length is set to 150, the processor may apply zero-padding to the first sub-MEL-spectrogram 631 and the third sub-MEL-spectrogram 633.

Referring to FIG. 6B, the plurality of post-processed sub-MEL-spectrograms 651, 652, and 653 all have the same length (eg, 150).

7 is a diagram for explaining a process of generating voice data from a plurality of sub-MEL-spectrograms according to an embodiment.

Referring to FIG. 7 , a plurality of post-processed sub-MEL-spectrograms 751, 752, and 753 all have the same length (eg, 150). Each of the plurality of post-processed sub-MEL-spectrograms 751, 752, and 753 of FIG. 7 may correspond to the plurality of post-processed sub-MEL-spectrograms 751, 752, and 753 of FIG.

The processor may generate a plurality of sub voice data 761 , 762 , and 763 from each of the plurality of post-processed sub mel-spectrograms 751 , 752 , and 753 . For example, the processor may generate a plurality of sub-voice data 761, 762, and 763 from each of the plurality of sub-mel-spectrograms 751, 752, and 753 post-processed using ISFT or Griffin-Lim algorithm. there is.

In one embodiment, the processor determines the reference interval for each of the plurality of sub-speech data 761, 762, and 763 based on the length of each of the plurality of sub-mel-spectrograms 631, 632, and 633 before post-processing ( 771, 772, 773) can be determined.

For example, the plurality of post-processed sub-Mel-spectrograms 751, 752, and 753 all have a length of 150, but the first sub-MEL-spectrogram corresponding to the first post-processed sub-MEL-spectrogram 751 has a length of 150. Since the spectrogram 631 has a length of 132, the first post-processed sub-MEL-spectrogram 751 includes valid data only up to a length of 132. For the same reason, the third post-processed sub-Mel-spectrogram 753 contains valid data only up to a length of 114, while the second post-processed sub-Mel-spectrogram 752 contains valid data for all lengths of 150. can include

The processor determines the length of the first reference interval 771 of the first sub-speech data 761 generated from the first post-processed sub-Mel-spectrogram 751 to be 132, and uses the second post-processed sub-Mel-spectrogram as The length of the second reference interval 772 of the second sub-speech data 762 generated from the second sub-speech data 762 is determined to be 150, and the third sub-speech generated from the third post-processed sub-mel-spectrogram 753 is determined. The length of the third reference section 773 of the data 763 may be determined to be 114.

The processor may generate voice data 780 by connecting the first reference interval 771 , the second reference interval 772 , and the third reference interval 773 .

The processor may generate voice data from a plurality of sub-mel-spectrograms based on a length of each of the plurality of sub-mel-spectrograms and a preset hop size. Specifically, the processor multiplies the length of each of the plurality of sub mel-spectrograms by a hop size (eg, 300) corresponding to the length of voice data covered by one frame of the mel-spectrogram, thereby obtaining a plurality of Reference intervals 771 , 772 , and 773 for each of the sub voice data 761 , 762 , and 763 may be determined.

Meanwhile, the processors of FIGS. 5, 6a, 6b, and 7 may be hardware included in a vocoder of a voice synthesis system and/or separate hardware.

8 is a diagram for explaining an example of segmenting a text sequence according to an exemplary embodiment.

8 shows an example of a text sequence composed of a specific natural language. Also, text sequences include punctuation marks. For example, punctuation marks '.', ',', '?', '!' etc.

The speech synthesis system 100 or 200 identifies characters and punctuation marks included in a text sequence. Then, the speech synthesis system 100 or 200 checks whether punctuation marks included in the text sequence are predetermined punctuation marks.

If the punctuation marks are predetermined punctuation marks, the speech synthesis system 100 or 200 may divide the text sequence based on the punctuation marks. For example, when the punctuation marks '.', ',', '?', and '!' are predetermined punctuation marks, the speech synthesis system 100 or 200 divides the text sequence based on the predetermined punctuation marks. , subsequences can be created.

9 is a diagram for explaining a process of generating a final mel-spectrogram by adding a silent mel-spectrogram between sub mel-spectrograms according to an embodiment.

Referring to FIG. 9 , the speech synthesis systems 100 and 200 may divide a text sequence 910 into a plurality of subsequences 911 , 912 and 913 . Since the method of dividing the text sequence 910 into a plurality of subsequences 911, 912, and 913 has been described in detail with reference to FIG. 8, it will be omitted here.

The speech synthesis system 100 or 200 may generate a plurality of sub-Mel-spectrograms 921, 922, or 923 using the plurality of sub-sequences 911, 912, or 913. Specifically, the speech synthesis system 100 or 200 may generate a plurality of sub-mel-spectrograms 921, 922, or 923 based on a plurality of sub-sequences 911, 912, or 913 corresponding to text and speaker information. there is. In addition, the voice synthesis system 100 or 200 may generate voice data from the plurality of sub-mel-spectrograms 921 , 922 , or 923 . Since detailed information about this has been described above with reference to FIGS. 1 to 4 , it will be omitted here.

In one embodiment, the speech synthesis system 100, 200 adds silence mel-spectrograms 931, 932 between a plurality of sub mel-spectrograms 921, 922, 923 to form a final mel-spectrogram 940. can create The voice synthesis system 100 or 200 may generate voice data from the final mel-spectrogram 940 .

Specifically, the speech synthesis system 100, 200 identifies the last character of a plurality of subsequences 911, 912, and 913 (ie, text) corresponding to a plurality of submel-spectrograms 921, 922, and 923. can do. When the last character is a first group character, the speech synthesis system 100 or 200 may generate a final mel-spectrogram by adding a silent mel-spectrogram having a first time to the sub mel-spectrogram. In addition, when the last character is a second group character, the speech synthesis system 100, 200 may generate a final mel-spectrogram by adding a silent mel-spectrogram having a second time to the sub mel-spectrogram. .

For example, the first group characters may include ',' and ' ' as characters corresponding to a short rest period. Also, the second group characters may include '.', '?', and '!' as characters corresponding to a long pause period. In this case, the first time may be set to the reference time, and the second time may be set to three times the reference time.

On the other hand, the voice synthesis system 100, 200 may classify the characters of the plurality of subsequences 911, 912, and 913 into two or more groups, and the time of the silent mel-spectrogram corresponding to each group is also described as the above example. Not limited.

In another embodiment, the speech synthesis system 100 or 200 may generate a final mel-spectrogram by adding a breath sound mel-spectrogram between the plurality of sub mel-spectrograms 921 , 922 , and 923 . To this end, the speech synthesis systems 100 and 200 may obtain breath sound data as speaker information.

In another embodiment, the speech synthesis system 100, 200 converts the last character of a plurality of subsequences 911, 912, and 913 (ie, text) corresponding to the plurality of submel-spectrograms 921, 922, and 923. can be identified. When the last character is a first group character, the voice synthesis system 100 or 200 may generate a final mel-spectrogram by adding a silent mel-spectrogram having a predetermined time to the sub mel-spectrogram. In addition, when the last character is a second group character, the voice synthesis system 100 or 200 may generate a final mel-spectrogram by adding a breath sound mel-spectrogram to the sub mel-spectrogram. For example, the first group characters may include ',' and ' ' as characters corresponding to a short rest period. Also, the second group characters may include '.', '?', and '!' as characters corresponding to a long pause period.

In another embodiment, the speech synthesis system 100, 200 may generate a final Mel-spectrogram by adding silent Mel-spectrograms having a random time between the plurality of sub-Mel-spectrograms 921, 922, and 923. may be

10 is a flowchart of a method for determining a silent portion of a Mel-spectrogram according to an embodiment.

Referring to FIG. 10 , in step 1010, the processor may receive speaker information and generate a speaker embedding vector based on the speaker information.

Speaker information may correspond to a speaker's voice signal or voice sample. The processor may receive the speaker's voice signal or voice sample, extract the speaker's speech characteristics, and represent the speech characteristics as an embedding vector.

The speaker's speech characteristics may include at least one of various factors such as speech speed, pause period, pitch, timbre, prosody, intonation, or emotion. That is, the processor may represent discontinuous data values included in speaker information as vectors composed of consecutive numbers. For example, the processor has pre-net, CBHG module, deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), long short-term memory network (LSTM), bidirectional recurrent deep neural network (BRDNN) A speaker embedding vector may be generated based on at least one or a combination of two or more of various artificial neural network models such as a network). In one embodiment, step 1010 may be performed by the speaker encoder 210 of FIG. 2 .

In step 1020, the processor may receive text and generate a text embedding vector based on the text.

Text may contain sequences of characters in a particular natural language. For example, a sequence of characters may include alphabetic characters, numbers, punctuation marks, or other special characters.

The processor may separate input text into consonant units, character units, or phoneme units, and input the separated text into an artificial neural network model. For example, the processor may generate a text embedding vector based on at least one or a combination of two or more of various artificial neural network models such as pre-net, CBHG module, DNN, CNN, RNN, LSTM, and BRDNN.

Alternatively, the processor may divide the input text into a plurality of short texts and generate a plurality of text embedding vectors for each of the short texts. In one embodiment, step 1020 may be performed by synthesizer 220 of FIG.

In step 1030, the processor may generate a mel-spectrogram based on the speaker embedding vector and the text embedding vector.

The processor may receive a speaker embedding vector and a text embedding vector as inputs. The processor may generate a spectrogram corresponding to the input text by inputting the speaker embedding vector and the text embedding vector to the artificial neural network model. That is, the processor may generate a spectrogram of the input text in which the speech characteristics of the speaker are reflected. For example, the spectrogram may correspond to a mel-spectrogram, but is not limited thereto.

A spectrogram is a graph that visualizes the spectrum of a voice signal. The x-axis of the spectrogram represents time, and the y-axis represents frequency, and the value of each frequency per time can be expressed in color according to the size of the value. The spectogram may be a result of performing short-time Fourier transform (STFT) on a continuously given audio signal. The mel spectrogram is obtained by re-adjusting the frequency interval of the spectrogram to a mel scale. In one embodiment, step 1030 may be performed by synthesizer 220 of FIG.

In step 1040, the processor may determine a silence part in the mel-spectrogram.

The processor may generate a volume graph by calculating a volume value for each of a plurality of frames constituting the mel-spectrogram. The processor may select at least one frame having a volume value equal to or less than a first threshold value from among the plurality of frames as the first interval. In one embodiment, the processor may determine the first period as a silent part of the Mel-spectrogram.

In another embodiment, the processor may select, as the second period, a period in which the number of frames is greater than or equal to the second threshold among the first period. The processor may determine the second section of the mel-spectrogram as the silent part. For example, when a voice is generated using the mel-spectrogram 520, the second threshold may be determined based on an overlap value and a hop size set in WaveRNN, which is one of the vocoders.

In step 1050, the processor may divide the mel-spectrogram into a plurality of sub mel-spectrograms based on the silent part.

The processor may calculate the length of each of the plurality of sub mel-spectrograms. The processor may post-process the plurality of sub-MEL-spectrograms such that the length of the plurality of sub-MEL-spectrograms is equal to the reference batch length. In one embodiment, the reference batch length may be a preset value. In another embodiment, the reference batch length may be set to the length of a sub mel-spectrogram having the longest length among a plurality of sub mel-spectrograms.

The processor may generate the voice data from a plurality of post-processed sub-Mel-spectrograms. The processor applies zero-padding to sub-Mel-spectrograms having a length less than the reference batch length so that the length of each of the plurality of sub-MEL-spectrograms is equal to the preset batch length, thereby forming a plurality of sub-MEL-spectrograms. The sub-Mel-spectrogram can be post-processed.

In step 1060, the processor may generate voice data from a plurality of sub-MEL-spectrograms.

The processor may generate a plurality of sub voice data from each of the plurality of post-processed sub mel-spectrograms. The processor may determine a reference interval for each of the plurality of sub-speech data based on the length of each of the plurality of sub-Mel-spectrograms.

The processor may generate voice data from a plurality of sub-mel-spectrograms based on a length of each of the plurality of sub-mel-spectrograms and a preset hop size. Specifically, the processor multiplies the length of each of the plurality of sub-mel-spectrograms by the hop size corresponding to the length of speech data covered by one frame of the mel-spectrogram, thereby obtaining a reference for each of the plurality of sub-speech data section can be determined.

The processor may generate voice data by connecting the reference intervals.

In this specification, “unit” may be a hardware component such as a processor or a circuit, and/or a software component executed by a hardware component such as a processor.

The above description of this specification is for illustrative purposes, and those skilled in the art to which the contents of this specification pertain will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. You will be able to. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present embodiment is indicated by the appended claims rather than the detailed description above, and should be construed as including all changes or modifications derived from the meaning and scope of the claims and equivalent concepts thereof.

Claims (3)

A method for generating voice data using speaker information and text,
generating a mel-spectrogram based on the speaker embedding vector and the text embedding vector;
determining a silence part in the mel-spectrogram;
dividing the mel-spectrogram into a plurality of sub mel-spectrograms based on the silent part; and
generating voice data from the plurality of sub mel-spectrograms;
Generating the voice data,
identifying the last character of text corresponding to the plurality of sub mel-spectrograms;
adding a silent mel-spectrogram to each of the plurality of sub mel-spectrograms based on the last character of the text;
post-processing the plurality of sub-mel-spectrograms to which the silent mel-spectrograms are added such that the length of each of the plurality of sub-mel-spectrograms to which the silent mel-spectrograms are added is equal to a reference batch length;
generating a plurality of sub voice data from each of the plurality of post-processed sub mel-spectrograms;
A reference interval for each of the plurality of sub-speech data based on the length of each of the plurality of sub-Mel-spectrograms before being post-processed - the reference interval is data valid in each of the plurality of sub-Mel-spectrograms that have been post-processed Is an interval included in - Determining that; and
generating the voice data by connecting the reference sections;
including,
Determining the silent part,
calculating a volume value for each of a plurality of frames constituting the mel-spectrogram;
selecting at least one frame having a volume value equal to or less than a first threshold value from among the plurality of frames as a first interval; and
determining the first section as a silent part;
Including, method. According to claim 1,
The step of adding the silent mel-spectrogram,
If the last character is a first group character, add a silent mel-spectrogram having a first time to the sub mel-spectrogram;
if the last character is a second group character, generating a final mel-spectrogram by adding a silent mel-spectrogram having a second time to the sub mel-spectrogram;
Including, method. delete

Images