KR102757438B1 - Method and computer readable storage medium for performing text-to-speech synthesis using machine learning based on sequential prosody feature - Google Patents

Abstract

The present disclosure relates to a text-to-speech synthesis method using machine learning based on sequential prosody features. The text-to-speech synthesis using machine learning based on sequential prosody features includes a step of receiving an input text, a step of receiving sequential prosody features, and a step of inputting the input text and the received sequential prosody features into an artificial neural network text-to-speech synthesis model, thereby generating output voice data for the input text in which the received sequential prosody features are reflected.

Description

{METHOD AND COMPUTER READABLE STORAGE MEDIUM FOR PERFORMING TEXT-TO-SPEECH SYNTHESIS USING MACHINE LEARNING BASED ON SEQUENTIAL PROSODY FEATURE}

The present disclosure relates to a method and system for text-to-speech synthesis using machine learning based on sequential prosodic features. More specifically, the present disclosure relates to a method and system for generating an output speech text for an input text in which sequential prosodic features are reflected by inputting sequential prosodic features into an artificial neural network text-to-speech synthesis model.

Voice synthesis technology, commonly called text-to-speech (TTS) technology, is a technology used to reproduce the required voice without recording the actual voice in advance in applications that require human voice such as announcements, navigation, and artificial intelligence secretaries. Typical methods of voice synthesis include concatenative TTS, which cuts the voice into very short units such as phonemes in advance and stores it, and synthesizes voice by combining the phonemes that make up the sentence to be synthesized, and parametric TTS, which expresses the features of the voice as parameters and synthesizes the parameters representing the voice features that make up the sentence to be synthesized into a voice corresponding to the sentence using a vocoder.

Meanwhile, voice synthesis methods based on artificial neural networks have been actively studied recently, and voices synthesized using such voice synthesis methods contain more natural voice features than conventional methods. However, in conventional voice synthesis methods, only fixed-length prosody features were applied regardless of the length of the input text or the length of the reference voice, and thus the prosody at a specific point in time of the synthesized voice could not be controlled. This is because there is a high probability that information will be lost in time when fixed-length features are forcibly applied to the reference voice. Accordingly, conventional voice synthesis methods could not provide fine prosody control for synthesized voices in order to accurately express people's intentions or emotions.

In addition, if there is a large difference between the pitch range of the source speaker and the pitch range of the target speaker, it may be difficult to reflect the prosodic features of the source speaker to the target speaker. For example, if the source speaker is a female and the target speaker is a male, when the prosody of the source speaker is synthesized into the voice of the target speaker, the synthesized voice of the target speaker may have a pitch higher than the normal pitch. Considering these circumstances, it may be required to preprocess the prosodic features before applying the prosodic features to the artificial neural network model in order to improve the quality of the synthesized voice reflecting the prosodic features.

The method and device according to the present disclosure can generate output speech data for input text reflecting sequential prosodic features having time-dependent prosodic features to solve the above problems.

In addition, the method and device according to the present disclosure may input sequential prosody features to at least one of an encoder and a decoder of an artificial neural network text-to-speech synthesis model, and an attention module may be used to match the sequential prosody features of variable length to the length of the input text and/or the length of the synthesized speech.

In addition, the method and device according to the present disclosure can normalize a plurality of embedding vectors corresponding to sequential prosody features and apply the normalized plurality of embedding vectors to an artificial neural network text-to-speech synthesis model.

The present disclosure can be implemented in various ways, including as a computer-readable storage medium storing methods, systems, devices, or instructions.

A text-to-speech synthesis method using machine learning based on sequential prosody features according to one embodiment of the present disclosure may include a step of receiving an input text, a step of receiving sequential prosody features, and a step of inputting the input text and the received sequential prosody features into an artificial neural network text-to-speech synthesis model to generate output speech data for the input text in which the received sequential prosody features are reflected.

An artificial neural network text-to-speech synthesis model of a text-to-speech synthesis method using machine learning based on sequential prosodic features according to one embodiment of the present disclosure is generated by performing machine learning on the basis of data representing a plurality of learning texts and learning voices corresponding to the plurality of learning texts, and the data representing the learning voices may include sequential prosodic features of the learning voices.

The sequential prosodic feature of a text-to-speech synthesis method using machine learning based on the sequential prosodic feature according to one embodiment of the present disclosure includes prosodic information corresponding to at least one unit among a frame, a character, a phoneme, a syllable, or a word in chronological order, and the prosodic information may include at least one of information about a size of a sound, information about a pitch of a sound, information about a length of a sound, information about a pause period of a sound, or information about a style of a sound.

The step of receiving the sequential prosody feature of the text-to-speech synthesis method using machine learning based on the sequential prosody feature according to one embodiment of the present disclosure includes the step of receiving a plurality of embedding vectors representing the sequential prosody feature, and each of the plurality of embedding vectors may correspond to prosody information included in time order.

According to one embodiment of the present disclosure, an artificial neural network text-to-speech synthesis model of a text-to-speech synthesis method using machine learning based on sequential prosodic features includes an encoder and a decoder, and the text-to-speech synthesis method using machine learning based on sequential prosodic features further includes a step of inputting a plurality of received embedding vectors into an attention module, thereby generating a plurality of transformed embedding vectors corresponding to respective parts of an input text provided to the encoder, wherein the lengths of the plurality of transformed embedding vectors are variable according to the length of the input text, and the step of generating output speech data for the input text may include a step of inputting the plurality of generated transformed embedding vectors into an encoder of the artificial neural network text-to-speech synthesis model, and a step of generating output speech data for the input text to which the plurality of transformed embedding vectors are reflected.

According to one embodiment of the present disclosure, a text-to-speech synthesis method using machine learning based on sequential prosodic features includes an artificial neural network text-to-speech synthesis model including an encoder and a decoder, and a step of generating output speech data for an input text may include a step of inputting a plurality of received embedding vectors into a decoder of the artificial neural network text-to-speech synthesis model, and a step of generating output speech data for the input text to which the plurality of embedding vectors are reflected.

A text-to-speech synthesis method using machine learning based on sequential prosodic features according to one embodiment of the present disclosure may further include a step of receiving a speaker's pronunciation features, and the step of generating output speech data for an input text may include a step of generating output speech data for the input text in which a plurality of embedding vectors that mimic the speaker's voice and represent sequential prosodic features are reflected.

In one embodiment of the present disclosure, the step of receiving the speaker's pronunciation features in the text-to-speech synthesis method using machine learning based on the sequential prosodic features may include the step of receiving the speaker's sequential prosodic features, the step of extracting the plurality of embedding vectors may include the step of normalizing the plurality of embedding vectors extracted based on the speaker's sequential prosodic features, and the step of generating output speech data for the input text may include the step of generating output speech data for the input text in which the speaker's speech is simulated and the normalized plurality of embedding vectors are reflected.

The step of normalizing the extracted plurality of embedding vectors of the text-to-speech synthesis method using machine learning based on sequential prosodic features according to one embodiment of the present disclosure may include the step of calculating an average value of the embedding vectors representing the sequential prosodic features of the speaker at each time step, and the step of subtracting the extracted plurality of embedding vectors by the average value of the embedding vectors calculated at each time step.

The step of receiving the sequential prosodic features of the text-to-speech synthesis method using machine learning based on the sequential prosodic features according to one embodiment of the present disclosure may include the step of receiving prosodic information for at least a part of an input text through a user interface, and the step of generating output speech data for the input text to which the received sequential prosodic features are reflected may include the step of generating output speech data for the input text to which the prosodic information for at least a part of the input text is reflected.

According to one embodiment of the present disclosure, prosody information for at least a portion of input text can be input via tags provided in a speech synthesis markup language.

A text-to-speech synthesis method using machine learning based on sequential prosodic features according to one embodiment of the present disclosure may further include a step of receiving prosodic information for at least a portion of an input text through a user interface and a step of changing the received sequential prosodic features based on the prosodic information for at least a portion of the received input text, and the step of generating output speech data for the input text to which the received sequential prosodic features are reflected may include a step of generating output speech data for the input text to which the changed sequential prosodic features are reflected.

According to one embodiment of the present disclosure, prosody information for at least a portion of an input text, which is used to modify a received sequential prosody feature, may be input via a tag provided in a speech synthesis markup language.

In addition, a program for implementing a text-to-speech synthesis method using machine learning based on the sequential prosodic features described above can be recorded on a computer-readable recording medium.

In addition, devices and technical means related to a text-to-speech synthesis method using machine learning based on the sequential prosodic features described above can also be disclosed.

According to some embodiments of the present disclosure, text-to-speech synthesis using machine learning is provided based on sequential prosody features that include prosody information over time and have variable lengths, thereby enabling fine prosody control for synthesized speech, thereby enabling more accurate conveyance of human intention or emotion through speech synthesis.

According to some embodiments of the present disclosure, when applying sequential prosody features of variable length to at least one of an encoder and a decoder of an artificial neural network text-to-speech synthesis model, attention can be used to adjust the sequential prosody features to correspond to the length of an input text and/or a synthesized speech, so that the sequential prosody features of variable length can be effectively applied to the input text and/or the synthesized speech regardless of their lengths.

According to some embodiments of the present disclosure, since preprocessing is performed to normalize a plurality of embedding vectors corresponding to the sequential prosody features before applying the sequential prosody features to the artificial neural network text-to-speech synthesis model, when the prosody features of one person are applied to the synthetic speech of another person, the quality of the synthetic speech reflecting the prosody features can be further improved.

Embodiments of the present disclosure will be described below with reference to the accompanying drawings, in which like reference numerals represent similar elements, but are not limited thereto.
FIG. 1 is an exemplary diagram showing a process of receiving input text and sequential prosody features and outputting synthesized speech by a speech synthesizer according to one embodiment of the present disclosure.
FIG. 2 is an exemplary diagram showing a process of outputting a synthesized speech using sequential prosody features extracted from a sequential prosody feature extractor and an input text by a speech synthesizer according to one embodiment of the present disclosure.
FIG. 3 is an exemplary diagram showing a process of applying sequential prosodic features and speaker's pronunciation features to input text and outputting synthesized speech by a speech synthesizer according to one embodiment of the present disclosure.
FIG. 4 is a block diagram of a text-to-speech synthesis system according to one embodiment of the present disclosure.
FIG. 5 is a flowchart illustrating a text-to-speech synthesis method using machine learning based on sequential prosody features according to one embodiment of the present disclosure.
FIG. 6 is an exemplary diagram showing the configuration of an artificial neural network-based text-to-speech synthesis system according to one embodiment of the present disclosure.
FIG. 7 is an exemplary diagram showing a process of generating a synthetic voice by inputting sequential prosody features into a decoder of a text-to-speech synthesis system based on an artificial neural network according to one embodiment of the present disclosure.
FIG. 8 is an exemplary diagram showing a process of generating a synthetic voice by inputting sequential prosody features into an encoder of a text-to-speech synthesis system based on an artificial neural network according to one embodiment of the present disclosure.
FIG. 9 is an exemplary diagram illustrating a network of sequential prosodic feature extraction units configured to extract a plurality of embedding vectors representing sequential prosodic features from a speech signal or sample according to one embodiment of the present disclosure.
FIG. 10 is a schematic diagram of a text-to-speech synthesis system that applies tags provided in a markup language according to one embodiment of the present disclosure to input text to output synthesized speech.
FIG. 11 is a block diagram of a text-to-speech synthesis system according to one embodiment of the present disclosure.

The advantages and features of the disclosed embodiments, and the methods for achieving them, will become clear with reference to the embodiments described below together with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, but may be implemented in various different forms, and these embodiments are only provided to make the present disclosure complete and to fully inform a person having ordinary skill in the art to which the present disclosure belongs of the scope of the invention.

The terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail.

The terms used in this specification are selected from the most widely used general terms possible while considering the functions in the present disclosure, but they may vary depending on the intention of engineers working in the relevant field, precedents, the emergence of new technologies, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meanings thereof will be described in detail in the description of the relevant invention. Therefore, the terms used in this disclosure should be defined based on the meanings of the terms and the overall contents of the present disclosure, rather than simply the names of the terms.

In this specification, singular expressions include plural expressions unless the context clearly specifies that they are singular. In addition, plural expressions include singular expressions unless the context clearly specifies that they are plural.

When a part of a specification is said to 'include' a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise stated.

Also, the term 'part' or 'module' used in the specification means a software or hardware component, and the 'part' or 'module' performs certain roles. However, the 'part' or 'module' is not limited to software or hardware. The 'part' or 'module' may be configured to be on an addressable storage medium and may be configured to execute one or more processors. Thus, as an example, the 'part' or 'module' includes components such as software components, object-oriented software components, class components, and task components, and processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functionality provided in the components and 'parts' or 'modules' may be combined into a smaller number of components and 'parts' or 'modules' or may be further separated into additional components and 'parts' or 'modules'.

According to one embodiment of the present disclosure, a 'unit' or a 'module' may be implemented as a processor and a memory. The term 'processor' should be interpreted broadly to include a general purpose processor, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and the like. In some environments, a 'processor' may also refer to an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), and the like. The term 'processor' may also refer to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or any other such configuration.

The term 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. The term memory may also refer to various types of processor-readable media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, etc. A memory is said to be in electronic communication with the processor if the processor can read information from and/or write information to the memory. Memory integrated in a processor is in electronic communication with the processor.

In the present disclosure, the 'sequential prosodic feature' may include prosodic information corresponding to at least one unit of a frame, a phoneme, a letter, a syllable, or a word in chronological order. Here, the prosodic information may include at least one of information on the loudness of a sound, information on the pitch of a sound, information on the length of a sound, information on a pause period of a sound, or information on a style of a sound. In addition, the style of a sound may include any form, manner, or nuance expressed by a sound or a voice, and may include, for example, tone, intonation, emotion, etc. inherent in a sound or a voice. In addition, the sequential prosodic feature may be expressed by a plurality of embedding vectors, and each of the plurality of embedding vectors may correspond to prosodic information included in chronological order.

Below, with reference to the attached drawings, an embodiment is described in detail so that a person having ordinary skill in the art to which the present disclosure pertains can easily practice the present disclosure. In addition, in order to clearly explain the present disclosure in the drawings, parts that are not related to the description are omitted.

FIG. 1 is an exemplary diagram showing a process of receiving an input text (120) and sequential prosody features (130) and outputting a synthesized voice (140) by a voice synthesizer (110) according to one embodiment of the present disclosure. The voice synthesizer (110) may be configured to output a synthesized voice corresponding to the input text using an artificial neural network text-to-speech synthesis model. Here, the artificial neural network text-to-speech synthesis model may be a single artificial neural network text-to-speech synthesis model. In one embodiment, the voice synthesizer (110) may correspond to the data recognition unit (455) of FIG. 4 and/or the data recognition unit (1020) of FIG. 10. In addition, the voice synthesizer (110) may be included in or provided to a user terminal or a text-to-speech synthesis system.

According to one embodiment, the text input to the speech synthesizer (110) may include text received through any interface (not shown). According to another embodiment, the speech recognizer (not shown) may receive a specific voice, convert it into a character corresponding to the input voice, and provide the converted character as an input text to the speech synthesizer (110). Accordingly, as illustrated in FIG. 1, the speech synthesizer (110) may receive the character 'HELLO' as a text input through the interface or the speech recognizer.

In one embodiment, the speech synthesizer (110) may be configured to receive sequential prosodic features. Here, the sequential prosodic features may include prosodic information for each time unit according to a predetermined time unit. As illustrated in FIG. 1, the sequential prosodic features may include information about the pitch of a sound, for example, information indicating a pitch according to time, such as '11113'. In one embodiment, such sequential prosodic features may be extracted or determined from any extractor capable of extracting prosodic features for a sound, for example, extracted from a pitch tracker. In another embodiment, the speech synthesizer (110) may receive any information indicating sequential prosodic information for a sound, for example, information indicated by a musical score. According to another embodiment, the speech synthesizer (110) can receive sequential prosodic features corresponding to attribute values expressed in speech synthesis markup language over time for text input from any device. These attribute values are described in detail below with reference to FIG. 9.

The speech synthesizer (110) may be configured to generate output data for an input text in which the received sequential prosody feature is reflected. To this end, the speech synthesizer (110) may apply the prosody information according to the time order indicated by the sequential prosody feature to the input text. For example, as illustrated in FIG. 1, the speech synthesizer (110) may generate output speech data by reflecting '11113', which is information indicating the pitch according to the time received in the input text 'HELLO'. That is, the speech synthesizer (110) may generate an output speech corresponding to 'HELLO?', which is an interrogative text in which the pitch of 'o', which is the last character of the input text, is raised higher than that of other characters. The speech generated in this way may be output through an output device such as a speaker or transmitted to another device having an I/O device.

FIG. 2 is an exemplary diagram showing a process of outputting a synthesized speech (240) using sequential prosody features (210) extracted from a sequential prosody feature extractor (230) and an input text (120) by a speech synthesizer (110) according to one embodiment of the present disclosure. In one embodiment, the sequential prosody feature extractor (230) may correspond to the sequential prosody feature extractor (410) of FIG. 4. Since the input text (120) and the speech synthesizer (110) have been described with reference to FIG. 1, a redundant description will be omitted.

According to one embodiment, the sequential prosody feature extractor (230) may receive a speech signal or a speech sample (220) and extract sequential prosody features (210) from the received speech signal or sample. Here, the received speech signal or sample may include speech spectrum data representing information related to the sequential prosody features (210), and may include, for example, a melody, a specific speaker's voice, etc.

According to one embodiment, in extracting the sequential prosody feature (210), any known suitable feature extraction method capable of extracting the sequential prosody feature (210) from a speech signal or sample (220) may be used. According to one embodiment, an artificial neural network or a machine learning model may be used to extract the sequential prosody feature. For example, the artificial neural network or machine learning model used in the sequential prosody feature extractor (230) may be composed of one or a combination of various artificial neural network models including a recurrent neural network (RNN), a long short-term memory model (LSTM), a deep neural network (DNN), a convolution neural network (CNN), etc.

The sequential prosody feature extractor (230) can input a received speech signal or speech sample into an artificial neural network prosody feature model to extract a plurality of feature vectors (embedding vectors) representing sequential prosody features (210). Here, each of the plurality of embedding vectors may correspond to a predetermined time unit (e.g., a frame, a phoneme, a letter, a syllable, a word, etc.). For example, such a vector may include one of various speech feature vectors such as MFCC (mel frequency cepstral coefficient), LPC (linear predictive coefficients), PLP (perceptual linear prediction), etc., but is not limited thereto. In addition, since the plurality of embedding vectors extracted in this way include prosody features or information in time order, the lengths of such vectors may be variable or different depending on the length of the input speech sample.

The speech synthesizer (110) can generate speech output data reflecting the sequential prosody features (210) extracted from the sequential prosody feature extractor (230) to the received text (120). For example, the speech synthesizer (110) can input embedding information corresponding to the input 'HELLO' text and a plurality of embedding vectors extracted by the sequential prosody feature extractor (230) into an artificial neural network text-to-speech synthesis model to generate 'HELLO' speech data reflecting the sequential prosody features. The speech generated in this way can be output through an output device such as a speaker or transmitted to another device having an I/O device. FIG. 3 is a schematic diagram of a speech synthesizer (110) that applies the sequential prosody features (210) and the speaker's pronunciation features (330) to the input text (120) and outputs a synthesized speech according to one embodiment of the present disclosure. As illustrated in FIG. 3, the speech synthesizer (110) may receive an input text (120), sequential prosodic features (210), and speaker's speech features (330). Here, the sequential prosodic features (210) may be extracted from a sequential prosodic feature extractor (230) based on a speech signal or a speech sample (220), and the speaker's speech features (330) may be extracted from a speaker's speech feature extractor (310) based on a speech signal or a speech sample (320). In one embodiment, the speech signal or the speech sample (220) input to the sequential prosodic feature extractor (230) may be different from the speech signal or the speech sample (320) input to the speech feature extractor (310). In another embodiment, the two speech signals or speech samples (220, 320) may be identical to each other. The speech synthesizer (110), input text (120), sequential prosody feature (210), speech signal or speech sample (220), and sequential prosody feature extractor (230) have been described with reference to FIGS. 1 and 2, and therefore, any redundant description will be omitted.

The speech feature extractor (310) may be configured to extract the speech feature of a speaker from speech data. The speech feature of a speaker may include at least one of various elements that may constitute the speech, such as style, prosody, emotion, timbre, and pitch, in addition to replicating the speaker's voice. According to one embodiment, the speech feature of a speaker may include a one-hot speaker ID-vector representing the speaker. According to another embodiment, the speech feature of a speaker may include an embedding vector representing the speech feature of the speaker. In one embodiment, the speech feature extractor (310) may correspond to the speech feature extractor (415) of FIG. 4.

A speech synthesizer (110) can input an input text (120), sequential prosodic features (210), and a speaker's pronunciation features (330) into an artificial neural network text-to-speech synthesis model to generate an output speech (340). The output speech (340) can include output speech data for the input text (120) in which the sequential prosodic features (210) and the speaker's pronunciation features (330) are reflected. That is, the output speech (340) can be data synthesized as a speech in which a speaker speaks an input text (120) with the input sequential prosodic features (210) by simulating the speaker's voice based on the speaker's pronunciation features and reflecting the sequential prosodic features (210). For example, if sequential prosody features (210) and speaker's vocalization features (330) are extracted from the voices of a first speaker and a second speaker who is different from the first speaker, a voice of the second speaker saying 'HELLO' can be output based on the second speaker's time-dependent prosody information. The voice thus generated can be output through an output device such as a speaker or transmitted to another device having an I/O device.

In FIGS. 2 and 3, the speech synthesizer (110) is illustrated as receiving a plurality of time-dependent embedding vectors representing sequential prosodic features (210) extracted from the sequential prosodic feature extractor (230), but is not limited thereto, and the speech synthesizer (110) may receive input values for the plurality of time-dependent embedding vectors representing sequential prosodic features (210) through an I/O device (not shown). Alternatively, the plurality of time-dependent embedding vectors representing sequential prosodic features (210) may be stored in advance in a storage medium (not shown), and the speech synthesizer (110) may access the storage medium to receive the plurality of embedding vectors. In addition, correction information for the plurality of embedding vectors extracted or stored in this manner may be received through the I/O device, and the plurality of embedding vectors may be corrected according to the received correction information, and the corrected plurality of embedding vectors may be received by the speech synthesizer (110).

In addition, in FIG. 3, the speaker's pronunciation feature (330) is illustrated as being extracted from the pronunciation feature extractor (310) and provided to the speech synthesizer (110), but is not limited thereto, and the speech synthesizer (110) may receive an input value for an embedding vector representing the pronunciation feature through an I/O device (not shown). Alternatively, the embedding vector representing the pronunciation feature may be stored in advance in a storage medium (not shown), and the speech synthesizer (110) may access the storage medium to receive the embedding vector representing the pronunciation feature. In addition, modification information for the pronunciation feature extracted or stored in this manner may be received through the I/O device, and the embedding vector representing the pronunciation feature may be modified according to the received information, and the embedding vector representing the modified pronunciation feature may be received by the speech synthesizer (110). FIG. 4 is a block diagram of a text-to-speech synthesis system (400) according to one embodiment of the present disclosure. The text-to-speech synthesis system (400) may include a communication unit (405), a sequential prosodic feature extraction unit (410), a pronunciation feature extraction unit (415), a normalizer (420), a voice database (425), an attention module (430), an encoder (435), a decoder (440), a post-processing processor (445), a data learning unit (450), and a data recognition unit (455). The communication unit (405) may be configured to allow the text-to-speech synthesis system (400) to transmit and receive signals or data with an external device. The external device may include a user terminal that provides a text-to-speech synthesis service. Alternatively, the external device may include another text-to-speech synthesis system. Alternatively, the external device may be any device including a voice database.

According to one embodiment, the communication unit (405) may be configured to receive text from an external device. Here, the text may include training text to be used for training an artificial neural network text-to-speech synthesis model. Alternatively, the text may include input text to be used for generating a synthetic voice through the artificial neural network text-to-speech synthesis model. Such text may be provided to at least one of the voice database (425), the encoder (435), the decoder (440), the data learning unit (450), and the data recognition unit (455).

The communication unit (405) may be configured to receive a speech signal or a speech sample via an external device. According to one embodiment, the speech signal or sample may be transmitted to a sequential prosody feature extraction unit (410), so that sequential prosody features may be extracted from the speech signal or sample. According to another embodiment, the speech signal or sample may be transmitted to a speech feature extraction unit (415), so that a speaker's speech features may be extracted from the speech signal or sample. The sequential prosody features and/or the speaker's speech features thus extracted may be transmitted to an encoder (435) and/or a decoder (440) via a data learning unit (450) and used to learn an artificial neural network text-to-speech synthesis model. Alternatively, the sequential prosodic features and/or speaker's vocalization features extracted in this manner may be transmitted to an encoder (435) and/or a decoder (440) via a data recognition unit (455) and used to generate a synthetic voice from an artificial neural network text-to-speech synthesis model.

In one embodiment, the communication unit (405) can receive sequential prosodic features from an external device. For example, the text-to-speech synthesis system (400) can receive sequential prosodic features extracted by the sequential prosodic feature extractor (230) of FIG. 2 through the communication unit (405). In another embodiment, the communication unit (405) can receive speaker's speech features from an external device. For example, the communication unit (405) can transmit and receive speaker's speech features (330) from the speaker's speech feature extractor (310) of FIG. 3. The sequential prosodic features and/or speaker's speech features received in this manner can be provided to at least one of the normalizer (420), the speech database (425), the attention module (430), the encoder (435), the decoder (440), the data learning unit (450), or the data recognition unit (455).

In one embodiment, the communication unit (405) may receive prosody information for input text from an external device as sequential prosody features. Here, the prosody information may include attribute values input through tags provided in a speech synthesis markup language for each part (e.g., phoneme, letter, syllable, word, etc.) of the input text.

According to one embodiment, the communication unit (405) can transmit information related to the generated output voice, i.e., output voice data, to an external device. In addition, the generated artificial neural network text-to-speech synthesis model can be transmitted to a user terminal or another text-to-speech synthesis system via the communication unit (405).

In FIG. 4, the text-to-speech synthesis system (400) is illustrated as receiving text, voice signals or samples, sequential prosody features, and speaker's pronunciation features through the communication unit (405), or outputting voice data and an artificial neural network text-to-speech synthesis model through the communication unit (405). However, the text-to-speech synthesis system (400) may further include an input/output device (I/O device; not shown). Accordingly, the text-to-speech synthesis system (400) may directly receive input from a user, and output at least one of text, voice, and image to the user.

The sequential prosody feature extraction unit (410) may be configured to receive a speech signal or sample through the communication unit (405) or the input/output device, and extract sequential prosody features from the received speech signal or sample. In one embodiment, the sequential prosody feature extraction unit (410) may correspond to the sequential prosody feature extractor (230) of FIGS. 2 and 3. For example, the sequential prosody feature extraction unit (410) may extract sequential prosody features from the received speech signal or sample using a speech processing method such as Mel-frequency cepstral (MFC). Alternatively, the sequential prosody features may be extracted by inputting the prosody feature extraction model (e.g., an artificial neural network) learned using the speech sample. For example, the sequential prosody features may be represented by a plurality of embedding vectors corresponding to a certain unit over time. Here, the certain unit may correspond to at least one unit such as a frame, a phoneme, a letter, a syllable, a word, or a phrase. According to another embodiment, the sequential prosody feature extraction unit (410) may receive at least one of information about a video, music, or sheet music, and may be configured to extract sequential prosody features from the received video, music, and/or sheet music. According to one embodiment, modification information for a plurality of embedding vectors representing sequential prosody features may be received via an I/O device (not shown), and the plurality of embedding vectors may be modified through the received information.

The extracted or modified sequential prosody features may be provided to the data learning unit (450) and/or the data recognition unit (455) and then provided to at least one of the encoder (414) and/or the decoder (440). According to one embodiment, the sequential prosody features may be provided to the normalizer (420) and/or the attention module (430) before being provided to the data learning unit (450) and/or the data recognition unit (455). According to one embodiment, the sequential prosody features extracted from the sequential prosody feature extraction unit (410) may be stored in a storage medium (e.g., a voice database (425)) or an external storage device. Accordingly, when synthesizing speech for an input text, one or more of a plurality of sequential prosody features pre-stored in the storage medium may be selected or designated, and the selected or designated sequential prosody features may be used for speech synthesis.

The speech feature extraction unit (415) may be configured to receive a speech signal of a speaker (e.g., a speech sample) and extract the speech feature of the speaker from the received speech signal. Here, the extracted speech feature may mimic the speaker and include any feature included in the speaker's speech, and may be expressed as, for example, a plurality of embedding vectors. In extracting the speech feature of the speaker, any known appropriate feature extraction method capable of extracting the speech feature from the speaker's speech signal may be used. For example, the speech feature extraction unit (415) may extract the speech feature of the speaker from the speech sample using an artificial neural network or a machine learning model. In one embodiment, the speech feature extraction unit (415) of the speaker may correspond to the speech feature extractor (310) of FIG. 3. The speech feature of the speaker extracted in this way may be provided to at least one of the data learning unit (450), the data recognition unit (455), the encoder (435), or the decoder (440).

According to one embodiment, the speaker's speech characteristics extracted from the speech characteristic extraction unit (415) may be stored in a speech database (425) or an external storage device. Accordingly, when synthesizing speech for an input text, one or more of the plurality of speaker's speech characteristics pre-stored in the storage medium may be selected or designated, and the selected or designated speaker's speech characteristics may be used for speech synthesis.

According to one embodiment, the speaker's pronunciation characteristics may include the speaker's sequential prosodic characteristics. For this purpose, for example, the speaker's pronunciation characteristics extraction unit (415) may be configured to extract the speaker's sequential prosodic characteristics from a speech sample. As another example, the speaker's pronunciation characteristics extraction unit (415) may provide a speech sample to the sequential prosodic characteristics extraction unit (410) to receive the speaker's sequential prosodic characteristics extracted from the speech sample. The speaker's sequential prosodic characteristics extracted in this way may be provided to the normalizer (420). In FIG. 4, the sequential prosodic characteristics extraction unit (410) and the speaker's pronunciation characteristics extraction unit (415) are illustrated as being configured as separate units, but are not limited thereto, and may be configured as one unit.

The normalizer (420) can receive the sequential prosody features received from the sequential prosody feature extraction unit (410) and the speaker's sequential prosody features (multiple embedding vectors) from the speech feature extraction unit (415) as the speaker's speech features. For the sake of explanation, the sequential prosody features received from the sequential prosody feature extraction unit (410) may be referred to as the first sequential prosody features, and the speaker's sequential prosody features from the speech feature extraction unit (415) may be referred to as the second sequential prosody features.

The normalizer (420) can be configured to normalize the first sequential prosodic feature (e.g., the plurality of embedding vectors) based on the second sequential prosodic feature (e.g., the plurality of embedding vectors). Here, the first sequential prosodic feature can be a feature extracted from a different speaker than a speaker associated with the second sequential prosodic feature. In one embodiment, the normalizer (420) can be configured to calculate an average value of the plurality of embedding vectors corresponding to the second sequential prosodic feature at each time step. In addition, the normalizer (420) can normalize the plurality of embedding vectors representing the first sequential prosodic feature by subtracting the plurality of embedding vectors representing the first sequential prosodic feature by an average value of the embedding vectors produced at each time step. The plurality of embedding vectors normalized in this way can be provided to at least one of the speech database (425), the attention module (430), the encoder (435), the decoder (440), the data learning unit (450), or the data recognition unit (455). Since the plurality of embedding vectors corresponding to the first sequential prosody feature are normalized using the average value of the plurality of embedding vectors corresponding to the second sequential prosody feature, when a synthetic speech corresponding to an arbitrary text is generated by simulating a speaker associated with the second sequential prosody feature using an artificial neural network text-to-speech synthesis model and reflecting the first sequential prosody feature extracted from another speaker, the first sequential prosody feature can be applied more naturally to the speech of a different speaker.

The voice database (425) can store learning texts and voices corresponding to a plurality of learning texts, and the learning texts and the voice sound data corresponding thereto can be accessed by the learning unit (450). The learning texts can be written in at least one language and can include at least one of words, phrases, and sentences that can be understood by humans. In addition, the voices stored in the voice database (425) can include voice data in which a plurality of speakers read the learning texts. The learning texts and voice data can be stored in advance in the voice database (425) or received from the communication unit (405). Based on the learning texts and voices stored in the voice database (425), the data learning unit (450) can generate or learn an artificial neural network text-to-speech synthesis model. For example, the artificial neural network text-synthesis model can include an encoder (435) and a decoder (440). As another example, an artificial neural network text-synthesis model may include an encoder (435), a decoder (440), and a post-processing processor (445).

In one embodiment, the speech database (425) may be configured to store one or more sequential prosodic features. For example, the one or more sequential prosodic features may include sequential prosodic features normalized from the normalizer (420). In another embodiment, the speech database (425) may be configured to store one or more speaker's speech features extracted from the speech feature extractor (415). The stored sequential prosodic features may be provided to at least one of the encoder (435) or the decoder (440) during speech synthesis by the data learning unit (450) and/or the data recognition unit (455). Additionally, the stored speaker's speech features may be provided to at least one of the encoder (435) or the decoder (440) during speech synthesis by the data learning unit (450) and/or the data recognition unit (455).

The attention module (430) can receive sequential prosody features or normalized sequential prosody features from the sequential prosody feature extractor (410) or the normalizer (420). According to one embodiment, the attention module (430) can be configured to receive a plurality of embedding vectors representing sequential prosody features and generate a plurality of transformed embedding vectors corresponding to each portion of the input text provided to the encoder (435). For example, the attention module (430) can be configured to determine which portion of the plurality of embedding vectors over time corresponds to which portion of the input text at the current time-step. The plurality of transformed embedding vectors generated by the attention module (430) can be provided to the encoder (435) for speech synthesis.

The encoder (435) can receive an input text and can be configured to convert the input text into a character embedding and generate it. For example, the encoder (435) can be configured as a part of an artificial neural network text-to-speech synthesis model. The character embedding can be input to a first artificial neural network text-to-speech synthesis model (e.g., pre-net, CBHG module, DNN, CNN+DNN, etc.) corresponding to the encoder (435) to generate hidden states of the encoder (435). The first artificial neural network text-to-speech synthesis model can be included in the artificial neural network text-to-speech synthesis model. According to one embodiment, the encoder (435) can further receive sequential prosody features from the sequential prosody feature extraction unit (410), the normalizer (420), or the attention module (430). The character embedding and sequential prosody features can be input into the first artificial neural network text-to-speech synthesis model to generate hidden states of the encoder (435). In another embodiment, the encoder (435) can further receive speaker's speech features from the speech feature extraction unit (415). The speaker's speech features can be input into the first artificial neural network text-to-speech synthesis model together with the character embedding and sequential prosody features to generate hidden states of the encoder (435). The hidden states of the encoder (435) generated in this way can be provided to the decoder (440).

The decoder (440) may be configured as a part of an artificial neural network text-to-speech synthesis model. In one embodiment, the decoder (440) may be configured to receive sequential prosody features. The decoder (440) may receive the sequential prosody features from at least one of the sequential prosody feature extractor (410) or the normalizer (420). The decoder (440) may receive hidden states corresponding to the input text from the encoder (435). Additionally, the decoder (440) may include an attention module configured to determine which part of the input text to generate speech from at a current time-step. Accordingly, the sequential prosody features and/or hidden states corresponding to the input text may be input to a second artificial neural network text-to-speech synthesis model (e.g., attention module, decoder RNN, attention RNN, Pre-net, DNN, etc.) corresponding to the decoder (440) to generate output speech data corresponding to the input text. The second artificial neural network text-to-speech synthesis model may be included in the artificial neural network text-to-speech synthesis model.

In another embodiment, the decoder (440) may be configured to further receive speaker's speech features from the speech feature extraction unit (415). The sequential prosodic features, the hidden states corresponding to the input text, and/or the speaker's speech features may be input to a second artificial neural network text-to-speech synthesis model corresponding to the decoder (440) to generate output speech data corresponding to the input text. The output speech data may include output speech data for the input text that mimics the speaker's speech and in which the sequential prosodic features are reflected.

The output voice data generated in this way can be expressed as a Mel spectrogram. However, it is not limited thereto, and the output voice data can be expressed as a linear spectrogram. The output voice data can be output to at least one of a speaker, a post-processing processor (445), or a communication unit (405).

According to one embodiment, the post-processing processor (445) may be configured to convert output speech data generated by the decoder (440) into speech that can be output from a speaker. For example, the converted output speech may be represented as a waveform. The post-processing processor (445) may be configured to operate only when the output speech data generated by the decoder (440) is unsuitable for output from the speaker. That is, when the output speech data generated by the decoder (440) is unsuitable for output from the speaker, the output speech data may be output directly to the speaker without going through the post-processing processor (445). Accordingly, although the post-processing processor (445) is illustrated as being included in the text-to-speech synthesis system (400) in FIG. 4, the post-processing processor (445) may be configured not to be included in the text-to-speech synthesis system (400).

According to one embodiment, the post-processing processor (445) may be configured to convert output speech data represented by a Mel Spectrogram generated by the decoder (440) into a waveform in the time domain. In addition, the post-processing processor (445) may amplify the size of the output speech data when the size of a signal of the output speech data does not reach a predetermined reference size. The post-processing processor (445) may output the converted output speech data to at least one of a speaker or a communication unit (405).

The data learning unit (450) may correspond to the data learning unit (1010) of FIG. 10. The data learning unit (450) may receive data representing a plurality of learning texts and learning voices corresponding thereto through the voice database (425) or the communication unit (405). The data representing the learning texts may include information on at least one letter. For example, the data representing the learning texts may include a phoneme sequence corresponding to the learning texts using a G2P (Grapheme-to-phoneme) algorithm. The data representing the learning voices may be data of a voice recorded by a person reading the learning texts, sound features extracted from such recording data, a spectrogram, etc. In one embodiment, the data representing the learning voices may include sequential prosodic features of the learning voices. In another embodiment, the data representing the learning voices may further include vocalization features of a speaker who uttered the learning voices. The data learning unit (450) can perform machine learning based on pairs of learning data corresponding to a plurality of learning texts and their corresponding learning voices, thereby generating an artificial neural network text-to-speech synthesis model. During such learning, the learning text can be provided to a first artificial neural network text-to-speech synthesis model corresponding to an encoder of the artificial neural network text-to-speech synthesis model, and sequential prosody features can be input to a second artificial neural network text-to-speech synthesis model corresponding to the first artificial neural network text-to-speech synthesis model and/or the decoder.

According to one embodiment, the data recognition unit (455) may be configured to receive an input text and receive sequential prosody features. The input text and the sequential prosody features may be input into an artificial neural network text-to-speech synthesis model to generate output speech data for the input text in which the received prosody features are reflected. Here, the input text may be provided to the first artificial neural network text-to-speech synthesis model, and the sequential prosody features may be input to the first artificial neural network text-to-speech synthesis model and/or the second artificial neural network text-to-speech synthesis model. As a result, output speech data corresponding to the input text in which the sequential prosody features are reflected may be generated from the artificial neural network text-to-speech synthesis model. According to another embodiment, the data recognition unit (455) may be configured to further receive a speaker's pronunciation features. The received speaker's pronunciation features may be provided to the second artificial neural network text-to-speech synthesis model, similarly to the sequential prosody features. Under these operations, output speech data corresponding to input text that mimics the speaker's voice and reflects sequential prosodic features can be generated from an artificial neural network text-to-speech synthesis model.

FIG. 5 is a flowchart illustrating a text-to-speech synthesis method using machine learning based on sequential prosody features according to one embodiment of the present disclosure. First, in S510, the text-to-speech synthesis system (400) may perform a step of generating an artificial neural network text-to-speech synthesis model generated by performing machine learning based on a plurality of learning texts and voice data corresponding to the plurality of learning texts. Here, the artificial neural network text-to-speech synthesis model may be a single artificial neural network text-to-speech synthesis model.

The text-to-speech synthesis system (400) can perform a step of receiving an input text at step S520. At step S530, the text-to-speech synthesis system (400) A step of receiving sequential prosody features can be performed. The text-to-speech synthesis system (400) can perform a step of inputting input text and sequential prosody features into a pre-learned text-to-speech synthesis model to generate output voice data for the input text in which the sequential prosody features are reflected in S540.

FIG. 6 is an exemplary diagram showing the configuration of an artificial neural network-based text-to-speech synthesis system according to one embodiment of the present disclosure. In one embodiment, each of the encoder (610), the decoder (620), and the post-processing processor (630) may correspond to each of the encoder (435), the decoder (440), and the post-processing processor (445) of FIG. 4.

According to one embodiment, the encoder (610) may receive a character embedding for an input text, as illustrated in FIG. 6. According to another embodiment, the input text may include at least one of a word, phrase, or sentence used in one or more languages. For example, a text such as a sentence such as 'HELLO' may be input. When the input text is received, the encoder (610) may divide the received input text into character units, character units, and phoneme units. According to another embodiment, the encoder (610) may receive the input text divided into character units, character units, and phoneme units. Then, the encoder (610) may convert the input text into a character embedding and generate it.

The encoder (610) may be configured to generate text as pronunciation information. In one embodiment, the encoder (610) may pass the generated character embedding through a pre-net including a fully-connected layer. Additionally, the encoder (610) may provide output from the pre-net to the CBHG module to output encoder hidden states e _i , as illustrated in FIG. 6 . For example, the CBHG module may include a 1D convolution bank, max pooling, a highway network, and a bidirectional gated recurrent unit (GRU).

In another embodiment, when the encoder (610) receives an input text or a separated input text, the encoder (610) may be configured to generate at least one embedding layer. According to one embodiment, the at least one embedding layer of the encoder (610) may generate a character embedding based on the input text separated into a letter unit, a character unit, and a phoneme unit. For example, the encoder (610) may use a machine learning model (e.g., a probabilistic model or an artificial neural network, etc.) that has already been learned to obtain a character embedding based on the separated input text. Furthermore, the encoder (610) may update the machine learning model while performing the machine learning. When the machine learning model is updated, the character embedding for the separated input text may also be changed. The encoder (610) may pass the character embedding to a DNN (Deep Neural Network) module consisting of a fully-connected layer. The DNN may include a general feedforward layer or a linear layer. The encoder (610) may provide the output of the DNN to a module including at least one of a convolutional neural network (CNN) or a recurrent neural network (RNN), and may generate hidden states of the encoder (610). While the CNN may capture local characteristics according to the size of the convolution kernel, the RNN may capture long term dependencies. The hidden states of the encoder (610), i.e., pronunciation information for the input text, may be provided to a decoder (620) including an attention module, and the decoder (620) may be configured to generate the pronunciation information as speech.

The decoder (620) can receive hidden states e _i of the encoder from the encoder (610). In one embodiment, as illustrated in FIG. 6, the decoder (620) can include a decoder RNN including an attention module, a pre-net consisting of a fully-connected layer and a gated recurrnt unit (GRU), and an attention recurrent neural network (RNN), and a residual GRU. Here, the attention RNN can output information to be used in the attention module. In addition, the decoder RNN can receive position information of the input text from the attention module. That is, the position information can include information about which position of the input text the decoder (620) is converting into speech. The decoder RNN can receive information from the attention RNN. Information received from the attention RNN may include information about what speech the decoder (620) has generated up to the previous time-step. The decoder RNN may generate the next output speech that follows the speech generated so far. For example, the output speech may have a Mel Spectrogram form, and the output speech may include r frames.

In another embodiment, the pre-net included in the decoder (620) may be replaced with a DNN consisting of a fully-connected layer. Here, the DNN may include at least one of a general feedforward layer or a linear layer.

In one embodiment, the decoder (620) may be configured to receive sequential prosody features. For example, as illustrated in FIG. 6, the sequential prosody feature extraction unit (410) may extract a plurality of embedding vectors p1, p2, ... pn (where n is proportional to the length of the speech sample) representing sequential prosody features from a speech signal or sample. Each of the plurality of embedding vectors may include prosody features or information per unit time. The manner in which the sequential prosody feature extraction unit (410) inputs the plurality of embedding vectors p1, p2, ... pn from the speech signal or sample to the decoder is described in detail below with reference to FIG. 7.

In one embodiment, the decoder (620) may be configured to further receive speaker's speech characteristics. For example, as illustrated in FIG. 6, the speaker's speech characteristics may be generated by inputting a speaker ID into the speech characteristics extraction unit (415) so that a speaker embedding vector e corresponding to the speaker's speech characteristics may be generated as the speaker's speech characteristics. As another example, the speaker's speech characteristics may be generated by extracting the speaker's embedding vector from a speech signal or sample rather than the speaker ID.

In addition, the attention module of the decoder (620) can receive information from the attention RNN. The information received from the attention RNN can include information about what speech the decoder (620) generated up to the previous time-step. In addition, the attention module of the decoder (620) can output a context vector based on the information received from the attention RNN and the information of the encoder. The information of the encoder (610) can include information about the input text from which speech is to be generated. The context vector can include information for determining which part of the input text to generate speech from at the current time-step. For example, the attention module of the decoder (620) can output information to generate speech based on the front part of the input text at the beginning of speech generation, and to generate speech based on the back part of the input text as the speech is generated.

The decoder (620) can input each of p1, p2, ... pn, which are sequential prosodic features, as well as the speaker embedding vector e, for each time step of the attention RNN and the decoder RNN, and configure the structure of the artificial neural network to decode differently for each speaker and each part of the input text. In Fig. 6, a plurality of embedding vectors p1, p2, ... pn are illustrated as being extracted from the sequential prosodic feature extraction unit (410), but the present invention is not limited thereto, and the decoder (620) can receive a plurality of embedding vectors corresponding to the normalized sequential prosodic features from the normalizer (420), and the plurality of embedding vectors corresponding to the normalized sequential prosodic features can be input for each time step of the attention RNN and the decoder RNN together with the speaker's embedding vector e.

Dummy frames are frames that are input to the decoder (620) when a previous time-step does not exist. RNN can perform machine learning by autoregression. That is, the r frame output from the previous time-step (622) can be the input of the current time-step (623). Since there cannot be a previous time-step in the first time-step (621), the decoder (620) can input a dummy frame to the machine learning network of the first time-step.

For text-to-speech synthesis, the operations of DNN, attention RNN, and decoder RNN can be performed repeatedly. For example, r frames obtained in the first time step (621) can be inputs for the next time step (622). Also, r frames output in the time step (622) can be inputs for the next time step (623).

Through the process described above, not only can the input text be synthesized into speech for each speaker, but also the prosody characteristics of each part of the input text can be reflected in chronological order. In other words, it is possible to control the prosody characteristics of the synthesized voice corresponding to the input text at a specific point in time. Accordingly, the text-to-speech synthesis system can control the fine prosody of the synthesized voice in order to more accurately convey people's intentions or emotions.

According to one embodiment, the decoder (620) can concatenate the Mel Spectrograms generated at each time step in time order to obtain the voice of the Mel Spectrogram for the entire text. The voice of the Mel Spectrogram for the entire text can be output to the post-processing processor (630). For example, the post-processing processor (630) can correspond to the post-processing processor (445) of FIG. 4.

According to one embodiment, the CBHG of the post-processing processor (630) may be configured to transform the mel-scale spectrogram of the decoder (620) into a linear-scale spectrogram, as illustrated in FIG. 6. For example, the output signal of the CBHG of the post-processing processor (630) may include a magnitude spectrogram. The phase of the output signal of the CBHG of the post-processing processor (630) may be restored through the Griffin-Lim algorithm and may be inverse short-time Fourier transform. The post-processing processor (630) may output a speech signal in the time domain.

In another embodiment, when the encoder (610) is configured to include a CNN or an RNN, the post-processing processor (630) is configured to include a CNN or an RNN, and this CNN or RNN can perform operations similar to the CNN or the RNN of the encoder (610). That is, the CNN or the RNN of the post-processing processor (630) can capture local characteristics and long-term dependencies. For example, the post-processing processor (630) can be a vocoder. Accordingly, the CNN or the RNN of the post-processing processor (630) can output a linear-scale spectrogram. For example, the linear-scale spectrogram can include a magnitude spectrogram. The post-processing processor (630) can predict the phase of the spectrogram through the Griffin-Lim algorithm. The post-processing processor (630) can output a time domain voice signal using an inverse short-time Fourier transform.

According to another embodiment, the post-processing processor (630) can generate a speech signal from the Mel Spectrogram based on a machine learning model. The machine learning model can include a model that machine-learns the correlation between the Mel Spectrogram and the speech signal. For example, an artificial neural network model such as WaveNet or WaveGlow can be used.

Such an artificial neural network-based text-to-speech synthesis system can be trained using a large database that exists as a pair of training text and speech signals. According to one embodiment, a speech synthesis device can receive a text, compare an output speech signal with a correct speech signal, and define a loss function. The speech synthesis device can learn the loss function through an error back propagation algorithm, and finally obtain an artificial neural network that produces a desired speech output when an arbitrary text is input.

In such an artificial neural network-based speech synthesis device, text, speaker's pronunciation characteristics, sequential prosody characteristics, etc. can be input into an artificial neural network text-to-speech synthesis model to output a speech signal. The text-to-speech synthesis system learns by comparing the output speech signal with a correct speech signal, so that when it receives the text, speaker's pronunciation characteristics, and sequential prosody characteristics, it can generate output speech data that reads the text with the sequential prosody characteristics reflected in the speaker's voice.

FIG. 7 is an exemplary diagram showing a process of generating a synthesized speech by inputting sequential prosody features into a decoder (720) of the text-to-speech synthesis system based on an artificial neural network according to one embodiment of the present disclosure. Here, each of the encoder (710), the decoder (720), the sequential prosody feature extraction unit (730), and the attention module (740) may correspond to each of the encoder (435), the decoder (440), the sequential prosody feature extraction unit (410), and the attention module (430) of FIG. 4. In addition, the encoder (710) and the decoder (720) may correspond to each of the encoder (610) and the decoder (620) of FIG. 6. In Fig. 7, it is assumed and illustrated that the length of the voice, N, is 4 and the length of the text, T, is 3, but this is not limited to the present invention, and the length of the voice, N, and the length of the text, T, may be any positive numbers that are different from each other.

According to one embodiment, as illustrated in FIG. 7, the sequential prosody feature extraction unit (730) may be configured to receive a spectrogram (y ₁ , y ₂ , y ₃ , y _n ) and extract a plurality of embedding vectors (P ₁ , P ₂ , P ₃ , P _n ) representing sequential prosody features. The plurality of embedding vectors (P ₁ , P ₂ , P ₃ , P _n ) extracted in this manner may be provided to the decoder (720). For example, the plurality of extracted embedding vectors (P ₁ , P ₂ , P ₃ , P _n ) may be provided to N decoder RNNs and attention RNNs of the decoder (720). In addition, the hidden states (e ₁ , e ₂ , e _T ) provided from the encoder (710) can be provided to the attention module (740), and the attention module (740) can generate transformed hidden states (e' ₁ , e' ₂ , e' 3 , e' _N ) such that the hidden states (e 1 , e 2 , e _T ) correspond to the lengths of the spectrograms (P ₁ , P ₂ , P ₃ , P _n ). The generated transformed hidden states (e' ₁ , e' ₂ , e' _{3 ,} e' _N ) can _{be connected with the extracted plurality of embedding vectors (P 1 ,} P ₂ , P ₃ , P _n ) and input to and processed by _each of the N decoder RNNs and the attention RNN. Since the processing process within the decoder (720) is a process overlapping with the processing process described in FIG. 6 , a detailed description thereof will be omitted. Through this process, the artificial neural network text-to-speech synthesis model included in the encoder (710) and decoder (720) can be trained so that sequential prosody features are reflected more naturally.

In FIG. 7, a process is described in which a spectrogram (y ₁ , y ₂ , y ₃ , y _n ) representing a specific voice is provided to a sequential prosody feature extraction unit (730), and the same spectrogram (y ₁ , y ₂ , y ₃ , y _n ) is output through a decoder (620). However, the present invention is not limited thereto, and a voice of a different length from the voice output through the decoder (720) may be input to the sequential prosody feature extraction unit (730). In this case, an additional attention module (not shown) may receive a plurality of embedding vectors extracted from the sequential prosody feature extraction unit, and may convert the lengths of the received plurality of embedding vectors so that they correspond to the length of the voice output through the decoder (720). Then, the converted plurality of embedding vectors may be provided to the decoder (720).

FIG. 8 is an exemplary diagram showing a process of generating a synthesized speech by inputting sequential prosody features into an encoder (820) of a text-to-speech synthesis system based on an artificial neural network according to an embodiment of the present disclosure. Here, each of the attention module (810), the encoder (820), the decoder (830), and the sequential prosody feature extraction unit (840) may correspond to each of the attention module (430), the encoder (435), the decoder (440), and the sequential prosody feature extraction unit (410) of FIG. 4. In addition, the encoder (810) and the decoder (820) may correspond to each of the encoder (610) and the decoder (620) of FIG. 6. In Fig. 8, it is assumed and illustrated that the length of the voice, N, is 4 and the length of the text, T, is 3, but this is not limited to the present invention, and the length of the voice, N, and the length of the text, T, may be any positive numbers that are different from each other.

According to one embodiment, as illustrated in FIG. 8, the sequential prosodic feature extraction unit (840) may be configured to receive a spectrogram (y ₁ , y ₂ , y ₃ , y _n ) and extract a plurality of embedding vectors (P ₁ , P ₂ , P ₃ , P _n ) representing sequential prosodic features. The plurality of embedding vectors (P ₁ , P ₂ , P ₃ , P _n ) extracted in this manner may be provided to an attention module (810). The attention module (810) may be configured to generate a plurality of transformed embedding vectors (P _{1 '} , P ₂ ', P _T ') such that the input plurality of embedding vectors (P ₁ , P ₂ , P ₃ , P _n ) correspond to a length (T) of a phoneme sequence corresponding to an encoder (820). According to another embodiment, the sequential prosodic feature extraction unit (840) may generate multiple transformed embedding vectors corresponding to words, not phonemes. For example, if the i-th phoneme and the j-th phoneme belong to the same word (v), they may have the value of P _i '=P _j ', and the attention module (810) may generate multiple transformed embedding vectors corresponding to words ( , , ..., ) can be configured to generate the length of the word. can be any positive integer different from N and T. As an example of obtaining , the average of the transformed embedding vectors of phonemes belonging to the same word can be used, but is not limited to this, and additional attention modules, etc. can be used.

As illustrated in FIG. 8, each of the generated plurality of transformation embedding vectors (P ₁ ', P ₂ ', P _T ') can be connected to each of the hidden states (e ₁ , e ₂ , e _T ) corresponding to the phoneme sequence of the input text. The connected hidden states (e ₁ , e ₂ , e _T ) and the plurality of transformation embedding vectors (P ₁ ', P ₂ ', P _T ') can be provided to the decoder (830). In contrast, the decoder (830) can generate phoneme sequences y ₁ , y ₂ , y 3 , y n using the received hidden states (e ₁ , e ₂ , e _T ) and transformed embedding vectors (P ₁ ', P ₂ ', P _{T ' )} using the attention module of the decoder (830), Pre-net, N decoder RNNs, and attention RNNs. In contrast, multiple transformed embedding vectors corresponding to words ( , , ,,,, ) is generated, multiple transformed embedding vectors ( , , ..., ) can be connected to correspond to each of the hidden states corresponding to the word sequence of the input text. Since the processing process in the decoder overlaps with the processing process described in Fig. 6, a detailed description is omitted. Through this process, the artificial neural network text-to-speech synthesis model included in the encoder (820) and the decoder (830) can be trained so that the sequential prosody features can be reflected more naturally.

FIG. 9 is an exemplary diagram showing a network of a sequential prosody feature extraction unit (920) configured to extract a plurality of embedding vectors (930) representing sequential prosody features from a speech signal or sample (910) according to one embodiment of the present disclosure. In one embodiment, the network of the sequential prosody feature extraction unit (920) may include a convolutional neural network (CNN), a batch-normalization (BN), a rectifier linear unit (ReLU), and a gated recurrent unit (GRU). When the CNN, the BN, and the ReLU receive a speech signal or sample and input their output values into the gated recurrent unit (GRU), they may output a plurality of embedding vectors representing sequential prosody features. For example, the speech signal or sample may be received in the form of a log-Mel-spectrogram.

In one implementation, when the data recognition unit (455) infers speech synthesis, the speech signal or sample need not be speech data corresponding to the input text, and a randomly selected speech signal may be used. In contrast, when the data learning unit (450) learns speech synthesis, the speech signal or sample may include speech data corresponding to the input text.

In this network, since there is no restriction on the use of spectrograms, any spectrogram can be inserted into this network. In addition, through this, an embedding vector (930) representing sequential prosodic features can be generated through immediate adaptation of the network. A spectrogram input as a speech signal or sample can have a variable length, and the lengths of multiple embedding vectors can vary depending on the length. Although Fig. 9 illustrates a network including CNN, BN, ReLU, and GRU, a network including various layers can be constructed in order to extract sequential prosodic features.

FIG. 10 is a schematic diagram of a text-to-speech synthesis system (1000) that applies an attribute value input to a tag provided in a markup language according to one embodiment of the present disclosure to an input text and outputs a synthesized voice. In one embodiment, the text-to-speech synthesis system (1000) may correspond to the text-to-speech synthesis system (400) of FIG. 4 and/or the text-to-speech synthesis system (1100) of FIG. 11.

In order to generate, adjust or change sequential prosody information, the text-to-speech synthesis system (1000) can receive prosody information for at least a portion of the text through an interface device. Here, the interface device can include any interface device that is directly connected to the text-to-speech synthesis system (1000) or connected via wired and/or wireless communication, and can include, for example, an interface of a user terminal. Additionally, the prosody information for at least a portion of the text can be received through any document editor or voice editor capable of inputting and editing any text.

According to one embodiment, the speech synthesis system (1000) may receive attribute values corresponding to each portion of the input text as prosody information using tags of any speech synthesis markup language provided in any document editor. For example, tags provided in the speech synthesis markup language may include any tag for indicating attributes included in sequential prosody features. Prosody information corresponding to a portion of text between a start tag and an end tag may be input. For example, as illustrated in FIG. 10, '1. <speed=1.5>I'm a boy.</speed>' may include prosody information indicating speed in the portion I'm a boy between the start tag and the end tag. As another example, as illustrated in FIG. 10, '2. This is what <style=emphasis>I</style>have.' may include prosody information indicating emphasis on the character I between the start tag and the end tag.

The text-to-speech synthesis system (1000) may generate sequential prosody information based on prosody information for at least a portion of the received input text, or change the prosody information corresponding to the input text among the sequential prosody information corresponding to the input text, and generate a synthesized voice corresponding to the input text to which the generated or changed sequential prosody information is reflected. According to one embodiment, the text-to-speech synthesis system (1000) may apply prosody information (e.g., attribute values) corresponding to each portion of the input text input to a reference embedding vector corresponding to the reference sequential prosody information. Here, the reference embedding vector may include a plurality of embedding vectors representing predetermined sequential prosody feature information. For example, the reference embedding vector may include a prosody feature vector according to time, and each prosody feature information may be represented as a weighted sum of a plurality of sub-embedding vectors (e.g., height, size, length, pause duration, style vector, etc.) that are orthogonal to each other. The text-to-speech synthesis system (1000) can separate the inherent elements of the reference embedding vector. For example, the text-to-speech synthesis system (1000) can obtain a plurality of unit embedding vectors that are orthogonal to each other based on the reference embedding vector. According to one embodiment, there may be various methods for separating the inherent elements in the embedding vector, such as independent component analysis (ICA), independent vector analysis (IVA), sparse coding, independent factor analysis (IFA), independent subspace analysis (ISA), and nonnegative matrix factorization (NMF). In addition, the text-to-speech synthesis system (1000) can perform regularization when learning the embedding vector for the sequential prosodic feature so that the inherent elements in the embedding vector can be separated. This regularization can be performed through the regularizer (420) of FIG. 4. When the text-to-speech synthesis system (1000) performs machine learning by performing normalization during learning, the reference embedding vector can be learned as a sparse vector. Accordingly, the text-to-speech synthesis system (900) can accurately separate inherent elements from the embedding vector learned as a sparse vector by using PCA (principle component analysis). Under this configuration, the text-to-speech synthesis system (1000) can modify the reference embedding vector based on the attribute value in the tag provided in the speech synthesis markup language. For example, the text-to-speech synthesis system (1000) can change the weights for a plurality of unit embedding vectors based on the attribute value in the received tag.

In one embodiment, the text-to-speech synthesis system (1000) can be configured to modify a reference embedding vector based on attribute values in tags provided in a received speech synthesis markup language. For example, the text-to-speech synthesis system (1000) can resynthesize an embedding vector corresponding to a sequential prosody feature by multiplying and adding a plurality of unit embedding vectors by weights changed according to the received attribute values. The text-to-speech synthesis system (1000) can output an embedding vector for the changed sequential prosody feature information. The text-to-speech synthesis system (1000) can input the modified embedding vector into an artificial neural network text-to-speech synthesis model to convert output speech data into speech data for an input text in which information included in attribute values in tags provided in a speech synthesis markup language is reflected.

FIG. 11 is a block diagram of a text-to-speech synthesis system (1100) according to one embodiment of the present disclosure.

Referring to FIG. 11, a text-to-speech synthesis system (1100) according to one embodiment may include a data learning unit (1110) and a data recognition unit (1120). Each of the data learning unit (1110) and the data recognition unit (1120) of the text-to-speech synthesis system of FIG. 11 may correspond to each of the data learning unit (450) and the data recognition unit (455) used by the text-to-speech synthesis system (400) of FIG. 4.

The data learning unit (1110) can input data to obtain a machine learning model. In addition, the data recognition unit (1120) can apply data to the machine learning model to generate an output voice. The text-to-speech synthesis system (1100) as described above can include a processor and a memory.

The data learning unit (1110) can learn voice for text. The data learning unit (1110) can learn criteria regarding which voice to output according to the text. In addition, the data learning unit (1110) can learn criteria regarding which voice feature to use to output the voice. The voice feature can include at least one of pronunciation of phonemes, the user's tone, intonation, or stress. The data learning unit (1110) can learn voice according to the text by acquiring data to be used for learning and applying the acquired data to a data learning model to be described later.

The data recognition unit (1120) can output a voice for the text based on the text. The data recognition unit (1120) can output a voice from a predetermined text using a learned data learning model. The data recognition unit (1120) can obtain a predetermined text (data) according to a preset criterion through learning. In addition, the data recognition unit (1120) can output a voice based on the predetermined data by using the data learning model with the obtained data as an input value. In addition, a result value output by the data learning model with the obtained data as an input value can be used to update the data learning model.

At least one of the data learning unit (1110) or the data recognition unit (1120) may be manufactured in the form of at least one hardware chip and mounted on an electronic device. For example, at least one of the data learning unit (1110) or the data recognition unit (1120) may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as a part of an existing general-purpose processor (e.g., CPU or application processor) or a graphics-only processor (e.g., GPU) and mounted on various electronic devices as described above.

In addition, the data learning unit (1110) and the data recognition unit (1120) may be mounted on separate electronic devices, respectively. For example, one of the data learning unit (1110) and the data recognition unit (1120) may be included in the electronic device, and the other may be included in the server. In addition, the data learning unit (1110) and the data recognition unit (1120) may provide model information constructed by the data learning unit (1110) to the data recognition unit (1120) via wired or wireless means, and data input to the data recognition unit (1120) may be provided to the data learning unit (1110) as additional learning data.

Meanwhile, at least one of the data learning unit (1110) or the data recognition unit (1120) may be implemented as a software module. If at least one of the data learning unit (1110) and the data recognition unit (1120) is implemented as a software module (or a program module including instructions), the software module may be stored in a memory or a non-transitory computer readable media that can be read by a computer. In addition, in this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS) and the remaining some may be provided by a predetermined application.

A data learning unit (1110) according to one embodiment of the present disclosure may include a data acquisition unit (1111), a preprocessing unit (1112), a learning data selection unit (1113), a model learning unit (1114), and a model evaluation unit (1115).

The data acquisition unit (1111) can acquire data required for machine learning. Since a lot of data is required for learning, the data acquisition unit (1111) can receive multiple texts and their corresponding voices.

The preprocessing unit (1112) can preprocess the acquired data so that the acquired data can be used for machine learning to determine the user's psychological state. The preprocessing unit (1112) can process the acquired data into a preset format so that the model learning unit (1114) described later can use it. For example, the preprocessing unit (1112) can morphologically analyze text and voice to obtain morphological embedding.

The learning data selection unit (1113) can select data required for learning from among the preprocessed data. The selected data can be provided to the model learning unit (1114). The learning data selection unit (1113) can select data required for learning from among the preprocessed data according to preset criteria. In addition, the learning data selection unit (1113) can select data according to preset criteria through learning by the model learning unit (1114) described below.

The model learning unit (1114) can learn criteria on what kind of voice to output according to the text based on the learning data. In addition, the model learning unit (1114) can learn a learning model that outputs voice according to the text using the learning data. In this case, the data learning model can include a pre-built model. For example, the data learning model can include a pre-built model by inputting basic learning data (e.g., sample images, etc.).

The data learning model can be constructed by considering the application field of the learning model, the purpose of learning, or the computer performance of the device. The data learning model can include, for example, a model based on a neural network. For example, models such as a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), a Long Short-Term Memory models (LSTM), a Bidirectional Recurrent Deep Neural Network (BRDNN), and Convolutional Neural Networks (CNN) can be used as the data learning model, but are not limited thereto.

According to various embodiments, when there are multiple pre-built data learning models, the model learning unit (1114) may determine a data learning model with a high correlation between the input learning data and the basic learning data as a data learning model to be learned. In this case, the basic learning data may be pre-classified by data type, and the data learning model may be pre-classified by data type. For example, the basic learning data may be pre-classified by various criteria such as the region where the learning data was generated, the time when the learning data was generated, the size of the learning data, the genre of the learning data, the creator of the learning data, the type of object in the learning data, etc.

Additionally, the model learning unit (1114) can learn a data learning model using a learning algorithm, such as error back-propagation or gradient descent, for example.

In addition, the model learning unit (1114) can learn the data learning model through, for example, supervised learning using learning data as input values. In addition, the model learning unit (1114) can learn the data learning model through, for example, unsupervised learning that discovers a criterion for situation judgment by learning the types of data necessary for situation judgment on its own without any special guidance. In addition, the model learning unit (1114) can learn the data learning model through, for example, reinforcement learning that uses feedback on whether the result of situation judgment according to learning is correct.

In addition, when the data learning model is learned, the model learning unit (1114) can store the learned data learning model. In this case, the model learning unit (1114) can store the learned data learning model in the memory of the electronic device including the data recognition unit (1120). Alternatively, the model learning unit (1114) can store the learned data learning model in the memory of a server connected to the electronic device via a wired or wireless network.

In this case, the memory in which the learned data learning model is stored may store, for example, commands or data related to at least one other component of the electronic device together. In addition, the memory may store software and/or programs. The programs may include, for example, a kernel, middleware, an application programming interface (API), and/or an application program (or 'application').

The model evaluation unit (1115) inputs evaluation data into the data learning model, and if the result output from the evaluation data does not satisfy a predetermined standard, it can cause the model learning unit (1114) to learn again. In this case, the evaluation data can include preset data for evaluating the data learning model.

For example, the model evaluation unit (1115) may evaluate that the learning data learning model for the evaluation data does not meet a predetermined criterion if the number or ratio of the evaluation data for which the recognition result is incorrect exceeds a preset threshold among the results of the learning data learning model for the evaluation data. For example, if the predetermined criterion is defined as a ratio of 2%, if the learning data learning model outputs incorrect recognition results for more than 20 evaluation data out of a total of 1,000 evaluation data, the model evaluation unit (1115) may evaluate that the learning data learning model is not suitable.

Meanwhile, if there are multiple learned data learning models, the model evaluation unit (1115) can evaluate whether each learned video learning model satisfies a predetermined criterion and determine a model satisfying the predetermined criterion as the final data learning model. In this case, if there are multiple models satisfying the predetermined criterion, the model evaluation unit (1115) can determine one or a predetermined number of models in order of high evaluation scores as the final data learning model.

Meanwhile, at least one of the data acquisition unit (1111), the preprocessing unit (1112), the learning data selection unit (1113), the model learning unit (1114), or the model evaluation unit (1115) in the data learning unit (1110) may be manufactured in the form of at least one hardware chip and mounted on an electronic device. For example, at least one of the data acquisition unit (1111), the preprocessing unit (1112), the learning data selection unit (1113), the model learning unit (1114), or the model evaluation unit (1115) may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as a part of an existing general-purpose processor (e.g., a CPU or an application processor) or a graphics-only processor (e.g., a GPU) and mounted on the various electronic devices described above.

In addition, the data acquisition unit (1111), the preprocessing unit (1112), the learning data selection unit (1113), the model learning unit (1114), and the model evaluation unit (1115) may be mounted on one electronic device, or may be mounted on separate electronic devices, respectively. For example, some of the data acquisition unit (1111), the preprocessing unit (1112), the learning data selection unit (1113), the model learning unit (1114), and the model evaluation unit (1115) may be included in the electronic device, and the remaining some may be included in the server.

In addition, at least one of the data acquisition unit (1111), the preprocessing unit (1112), the learning data selection unit (1113), the model learning unit (1114), or the model evaluation unit (1115) may be implemented as a software module. When at least one of the data acquisition unit (1111), the preprocessing unit (1112), the learning data selection unit (1113), the model learning unit (1114), or the model evaluation unit (1115) is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer readable media that can be read by a computer. In addition, in this case, at least one software module may be provided by an operating system (OS) or provided by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS) and the remaining some may be provided by a predetermined application.

A data recognition unit (1120) according to one embodiment of the present disclosure may include a data acquisition unit (1121), a preprocessing unit (1122), a recognition data selection unit (1123), a recognition result providing unit (1124), and a model update unit (1125).

The data acquisition unit (1121) can acquire text required for outputting voice. Conversely, the data acquisition unit (1121) can acquire voice required for outputting text. The preprocessing unit (1122) can preprocess acquired data so that the acquired data can be used for outputting voice or text. The preprocessing unit (1122) can process acquired data into a preset format so that the recognition result providing unit (1124) described below can use the acquired data for outputting voice or text.

The recognition data selection unit (1123) can select data required for outputting voice or text from among the preprocessed data. The selected data can be provided to the recognition result provision unit (1124). The recognition data selection unit (1123) can select some or all of the preprocessed data according to preset criteria for outputting voice or text. In addition, the recognition data selection unit (1123) can select data according to preset criteria through learning by the model learning unit (1114).

The recognition result providing unit (1124) can apply the selected data to the data learning model and output voice or text. The recognition result providing unit (1124) can apply the selected data to the data learning model by using the data selected by the recognition data selecting unit (1123) as an input value. In addition, the recognition result can be determined by the data learning model.

The model update unit (1125) can update the data learning model based on the evaluation of the recognition result provided by the recognition result provider (1124). For example, the model update unit (1125) can provide the recognition result provided by the recognition result provider (1124) to the model learning unit (1114), thereby causing the model learning unit (1114) to update the data learning model.

Meanwhile, at least one of the data acquisition unit (1121), the preprocessing unit (1122), the recognition data selection unit (1123), the recognition result provision unit (1124), or the model update unit (1125) in the data recognition unit (1120) may be manufactured in the form of at least one hardware chip and mounted on an electronic device. For example, at least one of the data acquisition unit (1121), the preprocessing unit (1122), the recognition data selection unit (1123), the recognition result provision unit (1124), or the model update unit (1125) may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as a part of an existing general-purpose processor (e.g., a CPU or an application processor) or a graphics-only processor (e.g., a GPU) and mounted on the various electronic devices described above.

In addition, the data acquisition unit (1121), the preprocessing unit (1122), the recognition data selection unit (1123), the recognition result providing unit (1124), and the model update unit (1125) may be mounted on one electronic device, or may be mounted on separate electronic devices, respectively. For example, some of the data acquisition unit (1121), the preprocessing unit (1122), the recognition data selection unit (1123), the recognition result providing unit (1124), and the model update unit (1125) may be included in the electronic device, and the remaining some may be included in the server.

In addition, at least one of the data acquisition unit (1121), the preprocessing unit (1122), the recognition data selection unit (1123), the recognition result provision unit (1124), or the model update unit (1125) may be implemented as a software module. When at least one of the data acquisition unit (1121), the preprocessing unit (1122), the recognition data selection unit (1123), the recognition result provision unit (1124), or the model update unit (1125) is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer readable medium that can be read by a computer. In addition, in this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS) and the remaining some may be provided by a predetermined application.

In general, the user terminal providing the text-to-speech synthesis system and the text-to-speech synthesis service described herein may represent various types of devices, such as a cordless telephone, a cellular telephone, a laptop computer, a wireless multimedia device, a wireless communication personal computer (PC) card, a PDA, an external or internal modem, a device communicating over a wireless channel, and the like. The device may have various names, such as an access terminal (AT), an access unit, a subscriber unit, a mobile station, a mobile device, a mobile unit, a mobile telephone, a mobile, a remote station, a remote terminal, a remote unit, a user device, user equipment, a handheld device, and the like. Any of the devices described herein may have a memory for storing instructions and data, as well as hardware, software, firmware, or combinations thereof.

The techniques described herein may be implemented by a variety of means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure of this specification may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may be implemented within one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, a computer, or a combination thereof.

Accordingly, the various illustrative logical blocks, modules, and circuits described in connection with the disclosure of this specification may be implemented or performed by any combination of a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or those designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

In a firmware and/or software implementation, the techniques may be implemented as instructions stored on a computer-readable medium, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, a compact disc (CD), a magnetic or optical data storage device, etc. The instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described herein.

If implemented in software, the functions may be stored on or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a computer. By way of non-limiting example, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium.

For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, digital subscriber line, or wireless technologies such as infrared, radio, and microwave are included within the definition of media. Disk and disc, as used herein, includes compact discs, laser discs, optical discs, digital versatile discs (DVD), floppy disks, and Blu-ray discs, where disks usually reproduce data magnetically, whereas discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in the user terminal.

The previous description of the present disclosure is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to the present disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to various modifications without departing from the spirit or scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Although the exemplary implementations may refer to utilizing aspects of the presently disclosed subject matter in the context of one or more standalone computer systems, the present subject matter is not so limited, but rather may be implemented in conjunction with any computing environment, such as a network or distributed computing environment. Furthermore, aspects of the presently disclosed subject matter may be implemented in or across multiple processing chips or devices, and storage may be similarly affected across multiple devices. Such devices may include PCs, network servers, and handheld devices.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are described as exemplary forms of implementing the claims.

Although the method described in this specification has been described through specific embodiments, it is possible to implement it as computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording devices that store data that can be read by a computer system. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, etc. In addition, the computer-readable recording medium can be distributed over network-connected computer systems, so that the computer-readable code can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the embodiments can be easily inferred by programmers in the technical field to which the present invention belongs.

Although the present disclosure has been described in connection with some embodiments herein, it should be understood that various modifications and variations may be made therein without departing from the scope of the present disclosure as understood by those skilled in the art to which the present disclosure pertains. Furthermore, such modifications and variations should be considered to fall within the scope of the claims appended hereto.

110: Speech synthesizer 120: Input text
130: Sequential rhyme features 140: Synthetic speech
210: Sequential prosodic features 220: Speech signal or speech sample
230: Sequential prosodic feature extractor 240: Synthetic speech
310: Voice feature extractor 320: Voice signal or voice sample
330: Speaker's vocal characteristics 340: Output voice

Claims (13)

A text-to-speech synthesis method using machine learning based on sequential prosodic features, performed by a text-to-speech synthesis system.
Step of receiving input text;
A step of receiving sequential prosody features including prosody information corresponding to at least one unit of a frame, letter, phoneme, syllable or word included in the received input text in chronological order; and
A step of inputting the above input text and the received sequential prosody features into an artificial neural network text-to-speech synthesis model, thereby generating output voice data for the input text in which the received sequential prosody features are reflected.
Including,
The above sequential prosody features include a plurality of embedding vectors representing prosody information included in the time order,
Each of the plurality of embedding vectors representing the prosody information included in the above time order is sequentially applied to each of the plurality of time-steps of the artificial neural network text-to-speech synthesis model,
The method further comprises a step of receiving the speaker's vocal characteristics,
The step of generating output speech data for the input text includes the step of simulating the speaker's voice and generating output speech data for the input text in which the plurality of embedding vectors are reflected,
The step of receiving the speaker's vocalization characteristics includes the step of receiving the speaker's sequential prosodic characteristics,
The above method,
Further comprising a step of normalizing the plurality of embedding vectors based on the sequential prosodic features of the speaker,
A text-to-speech synthesis method, wherein the step of generating output speech data for the input text includes the step of simulating the speaker's voice and generating output speech data for the input text in which the normalized plurality of embedding vectors are reflected.
In the first paragraph,
The above artificial neural network text-to-speech synthesis model is generated by performing machine learning based on data representing a plurality of learning texts and learning voices corresponding to the plurality of learning texts.
A text-to-speech synthesis method, wherein the data representing the above learning speech includes sequential prosodic features of the above learning speech.
In the first paragraph,
A text-to-speech synthesis method, wherein the prosody information includes at least one of information about the size of the sound, information about the pitch of the sound, information about the length of the sound, information about the pause period of the sound, or information about the style of the sound.
In the first paragraph,
The above artificial neural network text-to-speech synthesis model includes an encoder and a decoder,
A step of inputting the plurality of embedding vectors into an attention module to generate a plurality of transformed embedding vectors corresponding to each part of the input text provided to the encoder, wherein the length of the plurality of transformed embedding vectors is variable according to the length of the input text,
The step of generating output voice data for the above input text is:
A step of inputting the generated plurality of transformation embedding vectors into the encoder of the artificial neural network text-to-speech synthesis model; and
A step of generating output speech data for the input text in which the above plurality of transformed embedding vectors are reflected.
A text-to-speech synthesis method comprising:
In the first paragraph,
The above artificial neural network text-to-speech synthesis model includes an encoder and a decoder,
The step of generating output voice data for the above input text is:
A step of inputting the above plurality of embedding vectors into a decoder of the artificial neural network text-to-speech synthesis model; and
A step of generating output speech data for the input text in which the above multiple embedding vectors are reflected.
A text-to-speech synthesis method comprising:
delete delete In the first paragraph,
The step of normalizing the above multiple embedding vectors is:
A step of calculating the average value of the embedding vector representing the sequential prosodic features of the speaker at each time step; and
A step of subtracting the above multiple embedding vectors by the average value of the embedding vectors produced at each time step.
A text-to-speech synthesis method comprising:
In the first paragraph,
The step of receiving the sequential prosody features comprises the step of receiving prosody information for at least a portion of the input text through a user interface,
A text-to-speech synthesis method, wherein the step of generating output speech data for the input text in which the received sequential prosody features are reflected includes the step of generating output speech data for the input text in which prosody information for at least a part of the input text is reflected.
In Article 9,
A text-to-speech synthesis method, wherein prosody information for at least a portion of the input text is input via tags provided in a speech synthesis markup language.
In the first paragraph,
A step of receiving prosody information for at least a portion of the input text through a user interface; and
Further comprising a step of modifying the received sequential prosody feature based on prosody information for at least a portion of the received input text,
A text-to-speech synthesis method, wherein the step of generating output speech data for the input text in which the received sequential prosody features are reflected includes the step of generating output speech data for the input text in which the changed sequential prosody features are reflected.
In Article 11,
A text-to-speech synthesis method, wherein prosody information for at least a portion of the input text, which is used to change the received sequential prosody features, is input via tags provided in a speech synthesis markup language.
A computer-readable storage medium having recorded thereon a program including commands for performing each step of a text-to-speech synthesis method using machine learning utilizing the sequential prosody features of any one of claims 1 to 5 and claims 8 to 12.