JP3706112B2 - Speech synthesizer and computer program - Google Patents

Description

発話スタイルについては本実施の形態では簡略化し、「家族」、「友人」及び「他人」、さらに自分に対する発話という分類の各々について「丁寧」、「親しい」、及び「くだけた」というグループに分けた。全部で２４の発話カテゴリがあったが、ここではそのうちの次の５つについて論じる。すなわち「情報の提供」、「あいづち」、「情報の要求」、「つぶやき」、及び「繰返しの要求」である。
【００４２】
−発話の韻律とＮＡＱ−
正規化前には、ＡＱは基本周波数ｆ０とｒ＝−０．４０６の相関を有していた。正規化（ＮＡＱ＝ｌｏｇ（ＡＱ）＋ｌｏｇ（ｆ０））により得られたＮＡＱは基本周波数ｆ０とｒ＝０．１８２の相関を有していた。
【００４３】
図２は、家族に対する発話についてのＮＡＱと基本周波数ｆ０とを示す。図２において、ｍ１、ｍ２、ｍ３、ｍ４、ｍ５、ｍ６、及びｍ８は、それぞれ母、父、娘、夫、姉、姉の子、及び叔母を示す。図２から、いくつか興味ある傾向がわかる。すなわち、話者（女性）の娘（１歳）に対する発話が、基本周波数ｆ０及び気息性のいずれにおいても最も高い値を示している。気息性から、家族の序列が次の様に定まる。すなわち、娘＞姉の子＞父＞母＝姉＞叔母＞夫という順序である。この順序が、家族内での対話において、「気配り」をされている程度を示すという事が可能かも知れない。ラベル付け作業者も、この結果は発話を聞いているときの印象と一致している事を確認した。
【００４４】
図３は、対話の相手によるＮＡＱと基本周波数ｆ０とを示す。図３において「ｆ」は友人を示す。「ｍ」は家族、「ｔ」は他人を示す。興味深いのは、友人に対する「ａ」（注意深い発話）に関するＮＡＱの値は高く「ｂ」（親しい会話）及び「ｃ」（くだけた会話）の間では違いが見られないのに対して、家族間の会話ではこの関係が逆転している事である。すなわち、注意深い会話と親しい会話との間では違いが見られないのに対して、くだけた会話ではＮＡＱの値はかなり低くなっている。他人との会話については、くだけた会話はないが、注意深い会話及び親しい会話は予想した通りのＮＡＱの相違を示した。
【００４５】
図４は、発話とその目的についての相違について論ずる。既に述べた事から、注意深い会話においては、より「手ごろな」会話と比較してＮＡＱの値が高くなる事が予測される。図４は、この予想が正しい事を示す。図４は５つのカテゴリ（つぶやき（「？」）、間投詞（「Ｉ」）、情報の提供（「ｅ」）、情報の要求（「ｒｅ」）、及び繰返しの要求（「ｒｚ」））についてのＮＡＱと基本周波数ｆ０とを示す。
【００４６】
図４を参照して、情報の提供のＮＡＱの値は、情報の要求についての値よりもかなり低い。また、繰返しの要求のＮＡＱの値が最も高い。「つぶやき」については他とは別カテゴリであると考えられるが、それは図４によっても裏打ちされる。すなわち、つぶやきについてはｆ０がきわだって低く、気息性（高ＮＡＱ値）の声質を示している。
【００４７】
以上から、ＮＡＱにより測定した声質が、会話の相手、発話スタイル、及び発話の目的と大きな相関を持っている事が分かる。ＮＡＱは、会話においてはらう「注意」の程度によって一定の変化をし、基本周波数とは独立に変化する。従って、この声質を、基本周波数ｆ０、発話の長さ、及び振幅とともに韻律的特徴と考える事ができ、意味上の非言語的な相違を示すために音声合成において制御すべきものと考える。
【００４８】
−音声合成装置の構成−
上に述べた考え方に従い、ＮＡＱにより測定した声質を制御することにより、意味上の非言語的な相違が反映された音声合成を行なう音声合成装置の実施の形態について以下説明する。
【００４９】
図５に、この一実施の形態に係る音声合成装置のブロック図を示す。図５を参照して、この音声合成装置は、入力される音声合成の対象となるテキスト及び非言語情報を表す属性などを含む入力ＸＭＬ（Ｅｘｔｅｎｄｅｄ Ｍａｒｋ−ＵｐＬａｎｇｕａｇｅ）文３０を前処理し、音声合成のターゲットとなるテキストを作成する前処理部３２と、予め準備された特定の話者のバランス文音声ＤＢ３４と、前処理部３２により生成されたターゲットテキストに対し、バランス文音声ＤＢから適切な音素列を選択し連結する事により、入力ＸＭＬ３０に対する音声波形データを生成するための波形生成部３６と、波形生成部３６により生成された音声波形データに基づいて音声信号を合成するための音声信号合成部３８とを含む。
【００５０】
波形生成部３６及び音声信号合成部３８にはいずれも従来の音声合成技術を用いる事ができる。バランス文音声ＤＢ３４の音声は自然な音声ではないので、生成される音声は生硬で、自然とはいえない音声となる。ただし、バランス文音声ＤＢ３４に含まれる各音素については、音素バランス文の朗読文から得られたものなので、適切にラベル付けをする事が可能である。その結果、音声信号合成部３８から出力される音声信号は、生硬ではあるが、入力ＸＭＬ３０で指定された非言語情報に比較的よくあった音声信号となる。
【００５１】
本実施の形態に係る装置は、この様に音声信号合成部３８の出力として得られた音声信号を、自然な音声合成のための音響的ターゲット４０としてさらに自然発話音声データを用いて音声合成を行ない、自然な発話に近い合成音声信号５４を得る点にある。そのために本実施の形態の装置は、上記した各構成要素に加えて、バランス文音声ＤＢ３４の話者と同じ話者（又はよく似た声を出す人）の自然な発話を集める事により予め準備された自然発話音声ＤＢ４２を用いる。自然発話音声ＤＢ４２は、上記した話者の自然発話を収集する事により得られたもので、様々な状況での音声データを集めてある。ただし、この自然発話音声ＤＢ４２内の音声データには、上記した非言語情報に合わせて音声を抽出するためのラベル付けなどはしていない。自然発話についてそうしたラベル付けをする事が、従来の技術の説明で述べた様に困難だからである。
【００５２】
この装置はさらに、音響的ターゲット４０の各時間期間について、ＤＰマッチングによって自然発話音声ＤＢ４２の中から比較的近い（ＤＰ距離が小さい、すなわち類似度が高い）音声データを音声合成のための候補として複数個選択し、候補列４６として出力するための候補選択部４４と、候補列４６内の各候補について所定の韻律的属性を求め、その部分について入力ＸＭＬ３０で指定された非言語的情報と合致した韻律的属性を示すもののみを選択するためのフィルタ部４８とを含む。ここで使用される時間期間は、可変長である。
【００５３】
フィルタ部４８が各候補列から求める韻律的属性としては、よく知られている基本周波数ｆ０、音声データのパワー、発話の長さに加えて、上記したＮＡＱを含む。たとえばこれら各要素について、入力ＸＭＬ３０では各発話単位（たとえば文）について予め特徴ベクトル（又は特徴ベクトルを計算するための情報）が非言語情報として付与されている。各候補についてもこれらの情報を計算する事ができ、比較のための特徴ベクトルを作成する事ができる。フィルタ部４８は、各候補について計算された特徴ベクトルと、入力ＸＭＬ３０でその発話単位について付与されていた特徴ベクトルとの間の距離を計算し、最も小さな距離を示した候補であって、かつ連結したときになめらかに連結できる様な候補を選択する。フィルタ部４８は、この様にして最終的に音声合成をするための最終音声データ列５０を出力する。
【００５４】
この装置はさらに、最終音声データ列５０に基づいて波形生成を行なうための波形生成部５２を含む。波形生成部５２が出力する合成音声信号５４は、自然発話音声ＤＢ４２から抽出した音声データに基づいて合成されており、かつその各発話単位は入力ＸＭＬ３０においてその発話単位に付与されていた非言語情報によく合致したものとなる。従って、合成音声信号５４は、自然に聞こえる音声であって、かつ指定された発話モードによく合致したものとなる。
【００５５】
−音声合成装置の動作−
この装置は以下の様に動作する。入力ＸＭＬ３０が前処理部３２に与えられると、前処理部３２は音声合成すべきテキストを各発話単位で作成し、かつ入力ＸＭＬ３０において各発話単位に付与されていた非言語情報を抽出する。波形生成部３６は、バランス文を朗読した音声から作成した朗読音声データベースであるバランス文音声ＤＢ３４から、前処理部３２によって与えられたテキストを合成するための音声データをバランス文音声ＤＢ３４から抽出する。波形生成部３６はこの際、前処理部３２から与えられた非言語情報と一致するラベルが付された音声データを抽出する。波形生成部３６はさらに、抽出した音声データを従来の技術に従ってなめらかに連結し、音声信号合成部３８に与える。
【００５６】
音声信号合成部３８は、この音声データ列に基づいて、従来の技術に従って音声合成を行ない、自然発話音声合成のための音響的ターゲット４０を出力し候補選択部４４に与える。この音響的ターゲット４０の例を図６に示す。図６に示す例では、音響的ターゲット４０は時間期間９２，９４，９６及び９８を含む。この期間は可変長である。またこれらの時間期間は互いに一部重複していてもよい。
図５を参照して、候補選択部４４は、図６に示す各区間９２，９４，９６及び９８について、自然発話音声ＤＢ４２からＤＰマッチングにより音響的ターゲット４０の波形と類似した音声データ候補列１１２，１１４，１１６，１１８をそれぞれ抽出する。音声データ候補列１１２、１１４，１１６，１１８の各々は複数の音声データ候補を含む。本実施の形態では、候補選択部４４は、ＤＰ距離の小さなものから順番に所定の複数個を候補として選択する。候補選択部４４はこれら音声データ候補列１１２、１１４、１１６、１１８を図５に示す候補列４６としてフィルタ部４８に与える。
【００５７】
フィルタ部４８は、たとえば図６に示す時間期間９２について、音声データ候補列１１２に含まれる各候補の特徴ベクトルを算出する。そしてこの特徴ベクトルと、入力ＸＭＬ３０において付与されていた特徴ベクトルとを比較して、その間で計算されるコサイン尺度（すなわち類似度）が小さなものであって、かつ連続する期間の音声データと滑らかに連結できる様な候補１３２を選択する。同様にフィルタ部４８は、時間期間９４，９６，９８等についても複数の候補から候補１３４、１３６、１３８を抽出する。これらが図５に示す最終音声データ列５０となる。
【００５８】
波形生成部５２はこれら最終音声データ列５０を滑らかに連結した合成音声信号５４を出力する。
【００５９】
以上説明した本実施の形態の装置によれば、一旦バランス文音声ＤＢ３４を用いて音響的ターゲット４０を生成し、この音響的ターゲット４０に近く、かつ入力ＸＭＬ３０に付与されていた非言語的特徴と一致した韻律的特徴を示す音声データを自然発話音声ＤＢ４２から抽出する事ができる。この音声データ列から合成した合成音声信号５４を得る事ができる。そのため、合成音声信号５４は、自然に聞こえる音声であってかつ最初に指定された非言語的特徴によく合致したものとなる。また、自然発話音声ＤＢ４２からの抽出のために、自然発話音声ＤＢ４２中の音声データに予めラベル付けをしておく必要はない。バランス文音声ＤＢ３４のラベル付けだけをしておけばよく、これは容易に行なう事ができる。
【００６０】
上記した実施の形態では、候補選択部４４は、ＤＰ距離の小さなものから順番に所定の複数個を選択する。しかし本発明はその様な実施の形態には限定されない。たとえば、候補選択部４４は、ＤＰ距離が所定のしきい値より小さなもののみを候補として選択する様にしてもよい。また、ＤＰ距離の小さなものから順番に、かつ所定のしきい値より小さなもののみを選択する様にしてもよい。
【００６１】
なお、ここに説明した実施の形態の装置は１又は複数のコンピュータ及び当該１又は複数のコンピュータ上で実行されるソフトウェアにより実現する事ができる。そのソフトウェアの制御構造は、図５に示したブロック図とよく対応している。そのため、ここではその詳細は説明しない。当業者であれば、上記した説明からソフトウェアをどの様に構成すればよいかは明らかであろう。
【００６２】
今回開示された実施の形態は単に例示であって、本発明が上記した実施の形態のみに制限されるわけではない。本発明の範囲は、発明の詳細な説明の記載を参酌した上で、特許請求の範囲の各請求項によって示され、そこに記載された文言と均等の意味及び範囲内でのすべての変更を含む。
【図面の簡単な説明】
【図１】 本発明の一実施の形態の装置の原理を説明するための図である。
【図２】 家族に対するＮＡＱと基本周波数ｆ０とを示すための図である。
【図３】 相手の種類によるＮＡＱと基本周波数ｆ０とを示すための図である。
【図４】 発話の目的によるＮＡＱと基本周波数ｆ０とを示すための図である。
【図５】 本発明の一実施の形態の装置のブロック図である。
【図６】 本発明の一実施の形態の装置の動作を説明するための図である。
【符号の説明】
３０ 入力ＸＭＬ、３２ 前処理部、３４ バランス文音声ＤＢ、３６ 波形生成部、３８ 音声信号合成部、４０ 音響的ターゲット、４２ 自然発話音声ＤＢ、４４ 候補選択部、４６ 候補列、４８ フィルタ部、５０ 最終音声データ列、５２ 波形生成部、５４ 合成音声信号[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a speech synthesis technique, and more particularly to a technique for synthesizing a naturally audible speech from a speech database of natural utterances.
[0002]
[Prior art]
Speech synthesis cannot be natural in nature. However, there is a need for technology that synthesizes sounds that sound natural. For example, speech synthesis technology for assisting communication for people who cannot speak for some reason, automatic translation from speech to speech, providing information by speech via phone, or responding to inquiries from customers by phone Is needed.
[0003]
When synthesizing sounds that sound natural, it is necessary to use different tones according to the content of the story. For that purpose, it is necessary to subdivide the speech used for speech synthesis into elements, and attach labels that indicate when the elements are used for speech.
[0004]
Currently, there are several large corpuses of spontaneously spoken speech that could be used to perform such naturally sounding speech synthesis. However, the work of dividing the speech included in the corpus and labeling each of them is enormous. There are also many problems that have not yet been solved in relation to modeling the acoustic features of spontaneous speech.
[0005]
On the other hand, in a speech database (hereinafter referred to as “balance sentence speech DB”) composed of speech obtained by reading out phoneme balance sentences, such labeling is relatively easy. The balance sentence speech DB is a database of all phonemes and all prosody.
[0006]
Conventionally, as a speech synthesis technique using a balanced sentence speech DB, for example, there is a technique called CHATR introduced in Non-Patent Document 1 or Non-Patent Document 2 to select and connect phonemes.
[0007]
The standard method of speech synthesis by connecting phonemes goes through two stages as described in Non-Patent Document 1 or Non-Patent Document 2. In the first stage, an appropriate candidate is selected from several corpora for each segment of speech using an objective cost function that reflects phoneme and prosodic constraints according to the text (target) to be synthesized. To do. In the second stage, one of the candidates of each section is selected and speech synthesis is performed by concatenating them so that the synthesized speech is as smooth as possible and the cost for connection is minimized. To do.
[0008]
The target of this process is usually a previously known symbolic representation (alpha-numeric) that represents the phonetic and prosodic representation of the desired output speech.
[0009]
[Non-Patent Document 1]
Campbell, W. N. , Black, A. W. "CHATR multilingual speech rearrangement synthesis system, IEICE technical report SP96-7, 45-52, 1996 (Campbell, WN." 45-52, 1996)
[Non-Patent Document 2]
Campbell, W. N. , "Processing of speech corpus for CHATR synthesis", Proceedings of International Conference on Speech Processing 183-186, 1997 (Campbell, WN, "Processing a Speech Corps CHATR Synthesis", Proceedings of the Conference) 183-186, 1997)
[Non-Patent Document 3]
P. ALC and E.I. Wilkman, “Amplitude Domain Index for Characterization of Glottal Volume Velocity Waveform Estimated by Inverse Filtering”, Speech Comm. , Vol. 18, No. 2, pp. 131-138, 1996 (P. Alku and E. Vilkman, “Amplitude domain-oriented for charac- terization of the global velocities of p. 38, p.
[Non-Patent Document 4]
P. Ark, T.W. Bextrom, and E.E. Wilkman, “Normalized amplitude index for parameterization of glottic airflow”, J. Am. Acoustic. Soc. Am. , Vol. 112, no. 2, pp. 701-710, 2002 (P. Alku, T. Baeckstrom, and E. Vilkman, “Normalized amplified fortralization of the global flow,” J. Aust.p.o. 701-710, 2002)
[Problems to be solved by the invention]
It is said that most of the research done to date as “corpus-based” speech synthesis was actually about “database” speech synthesis. The difference is how much the utterance style is covered and what kind of utterance style it is.
[0010]
A “corpus” is a collection of text or speech that is somewhat representative of a language that can be used as a starting point for linguistic explanations about a language or as a means of verifying a hypothesis about a language. Say. In this case, for a systematic study of authentic examples of the language actually used, a naturally occurring language (i.e. a set of which has been chosen to characterize a language state or change) It is important to be a collection of text or audio.
[0011]
Text written for the purpose of showing certain linguistic features should not normally be included in a true corpus for linguistic research. This is because they do not meet the criteria of “authentic” and are therefore not “naturally occurring”.
[0012]
But so far, the vast majority of databases used in speech synthesis research have been designed for specific purposes and are usually read carefully by professional announcers. It consists of a studio recording. They are not representative of “speech used”, nor do they include natural utterance styles encountered in everyday life and variations depending on utterance situations.
[0013]
The balance voice DB can be labeled in detail. However, the speech included in the balanced speech DB includes many examples of formal linguistic features of spoken language, but hardly includes features in terms of social and interactive functions by spoken language. When speech synthesis is performed using the balanced speech DB, the synthesized speech obtained as a result is a hard pronunciation and cannot be heard as a natural speech.
[0014]
If speech synthesis is to be developed in a more natural way, it is a corpus that can represent all aspects of spoken language interaction, and the spoken language, such as the state, attitude and intention of the speaker It is necessary to conduct research based on a corpus that also contains non-linguistic information that provides clues to interpret in line with that intent.
[0015]
In order to solve this, it is conceivable to use a natural utterance DB. However, if the natural utterance DB is used for speech synthesis, the labeling work becomes enormous as described above, and it is difficult to model acoustic features for labeling. Therefore, conventionally, there has been a problem that it is difficult to synthesize a sound that can be heard naturally using a naturally uttered speech DB.
[0016]
The present invention has been made to solve such a problem, and an object of the present invention is to provide a speech synthesizer capable of performing speech synthesis that can be naturally heard using a natural speech DB.
[0017]
Another object of the present invention is to provide a speech synthesizer that can synthesize naturally audible speech using a naturally uttered speech DB without labeling the naturally uttered speech DB.
[0018]
Still another object of the present invention is to generate an acoustic target by some means from the first target, and extract speech similar to the acoustic target from the natural utterance speech DB, thereby performing speech synthesis that can be heard naturally. It is to provide a speech synthesizer that can.
[0019]
Another object of the present invention is to provide a speech synthesizer that can synthesize speech that can be heard naturally in an utterance style that matches the non-verbal and paralinguistic features of a target.
[0020]
[Means for Solving the Problems]
The speech synthesizer according to the first aspect of the present invention includes a recitation speech database composed of recitation speech data pre-labeled with linguistic information, a natural utterance speech database composed of spontaneous utterance speech data, and non-language information. To synthesize speech signals corresponding to text information by receiving text information assigned in advance and extracting speech data to which linguistic information that matches non-linguistic information attached to the text information is extracted from the reading speech database Voice selection means for selecting a plurality of spontaneously uttered voice data in order from the smallest defined distance between each part of the speech signal from the spontaneously uttered voice database, For each part of the speech signal, a plurality of spontaneous utterances selected by the candidate selection means from the natural utterance speech database Filter means for calculating a predetermined prosodic feature for each of the data and selecting the one that matches the non-linguistic information given to the text information, and based on the natural utterance data selected by the filter means Means for synthesizing the audio signal.
[0021]
Preferably, the non-linguistic information given in advance to the text information is a feature vector indicating a predetermined prosodic feature, and the filtering unit is configured to select a plurality of spontaneous utterance data selected by the candidate selecting unit. Means for calculating a feature vector indicating a predetermined prosodic feature and selecting the one having the highest degree of similarity with the feature vector previously assigned to the text information is included.
[0022]
More preferably, the predetermined prosodic feature includes at least one of a normalized amplitude index, a power of the speech signal, a duration of the speech signal, and a fundamental frequency.
[0023]
The candidate selecting means selects, for each part of the audio signal, a natural utterance voice database from which a DP distance calculated by DP (Dynamic Programming) matching with each part is smaller than a predetermined threshold value. Means may be included.
[0024]
Candidate selection means is means for selecting, for each part of the audio signal, a plurality of predetermined numbers in order from those having a small DP distance calculated from the natural utterance voice database by DP matching with each part. May be included.
[0025]
A second aspect of the present invention relates to a computer program that, when executed by a computer, causes the computer to operate as any of the speech synthesizers described above.
[0026]
DETAILED DESCRIPTION OF THE INVENTION
  -Voice quality that expresses the characteristics of natural utterances-
  The biggest difference between studio-recorded audio and everyday familiar audio is that the utterance style experienced with everyday audio is very large. This seems to be because the speaker changes the laryngeal settings variously to show the formality of the utterance in that situation when speaking.
[0027]
For one speaker of a speech corpus created by the applicant, speech data obtained by recording for over 100 hours was divided into chunks of speech size. These chunks were further labeled to show the utterance style characteristics in three stages. There are the following three types of labels.
[0028]
(A) Speaker status (emotion and attitude)
(B) Talk style (friendly, polite, soft, hesitant, etc.)
(C) The tone of the speaker during each utterance (Blessy, warm, nervous, etc.)
Here, breathy means breathability and is typically a characteristic of how to speak when speaking politely and gently. The reverse is called pressed.
[0029]
The vector consisting of these three labels was combined with the acoustic features (pitch, power, speaking speed, degree of breathing, etc.) extracted from the speech. Furthermore, principal component analysis (PCA) was performed to reduce the complexity of the resulting multidimensional space. The first dimension of PCA analysis corresponds well to the relationship between the speaker and the other party (good friendship), the second dimension corresponds well to the utterance content (honesty), and the third dimension corresponds to the speaker's attitude (enthusiasm) ) Well supported.
[0030]
This seems to be because the speaker changes the quality of the voice, the width of the pitch, and the expression depending on the relationship with the other party and the purpose of the dialogue. It is common sense to speak to another person in a different way. However, in the voice-related field, almost all data is accumulated on how speech styles and voice characteristics differ when people talk to family members, friends, business acquaintances, other people, and machines. It has never been done.
[0031]
Before the description of the embodiment, the difference between the above-described speech style and voice characteristics as the background will be described. FIG. 1 shows the distribution of normalized amplitude index (NAQ) for two speakers (FIA and FAN). NAQ is obtained by normalizing an amplitude coefficient (AQ) with the fundamental frequency f0.
[0032]
AQ is indicated by Alku in Non-Patent Document 3 and uses an optimized time-varying formant to remove the influence of the vocal tract from the speech signal. Is the estimated derivative of the waveform of the glottal (glottal) airflow obtained by inverse filtering the waveform, and the maximum peak-to-peak amplitude of the waveform is divided by the minimum cycle-to-cycle amplitude of the waveform differentiation. It is a thing. AQ indicates the mode of glottal pronunciation ("voice quality").
[0033]
AQ has a weak correlation with the fundamental period of the speech waveform as it is, but its influence can be reduced by dividing by the fundamental frequency f0. The result is NAQ.
[0034]
The lower half of FIG. 1 shows a histogram of NAQ measured for the speech of two Japanese female speakers (FIA, FAN). In the upper half of FIG. 1, “Presto”, “Early voice (normal)”, and “Blessy ( This is a comparison with the measure for “breathing”. From FIG. 1, although there are some variations depending on individuals, the shape of the entire distribution is similar, and that is described in the above-mentioned document, “Presto”, “Earth (normal)”, and “Blessy ( It can be seen that it falls within the range of “breathing”. The distortion seen in the speaker FAN data can be explained by the predominance of the utterance style (presto), which is more prevalent as explained below.
[0035]
In the following, the fact that this variation is not random, and that this variation can best be explained by correlation with non-linguistic features of speech, such as the relationship with the conversation partner, intention of speech, and speech style, and therefore Indicates that fluctuation should be considered as a prosodic parameter.
[0036]
The applicant collected voice data for about 250 hours and entered text by listening. About 100 hours of them were labeled with respect to the characteristics of the relationship between utterance style and utterance and its purpose. We performed acoustic measurements of speech and analyzed the correlation between perceptual and physical attributes.
[0037]
In the following discussion, we will examine the data obtained from a Japanese female speaker. The woman recorded a daily conversation using a high-performance microphone attached to her head. Analysis was done only on this woman's utterance, but sometimes the other's utterance was also heard by the labeling worker.
[0038]
The data consists of 13,604 utterances that could be properly labeled with acoustic and perceptual labels. “Speech” means a sound part that can be perceived by a person in charge of documentation without a break, and probably corresponds to an “intonation phrase”. Its length ranges from a single syllable to 35 syllables.
[0039]
Data were analyzed using CRAN's public domain statistical software package “R”. Generate a feature set consisting of the other party ("who"), utterance style ("how"), and utterance activity ("for what"), and collate it with the measure of NAQ and the fundamental frequency f0 of speech We examined whether there was any correlation.
[0040]
Dialogue partners were grouped as shown in Table 1 below.
[0041]
[Table 1]

In this embodiment, the utterance style is simplified and divided into groups of “family”, “friend” and “others”, and “speech”, “familiarity”, and “kakuta” for each of the utterances to oneself. It was. There were a total of 24 utterance categories, of which the next five are discussed here. That is, “Provision of information”, “Aizuchi”, “Request for information”, “Mutter”, and “Repetition request”.
[0042]
  -Prosody of speech and NAQ-
  Before normalization, AQIsThere was a correlation between the fundamental frequency f0 and r = −0.406. The NAQ obtained by normalization (NAQ = log (AQ) + log (f0)) had a correlation between the fundamental frequency f0 and r = 0.182.
[0043]
FIG. 2 shows the NAQ and fundamental frequency f0 for utterances to the family. In FIG. 2, m1, m2, m3, m4, m5, m6, and m8 represent a mother, a father, a daughter, a husband, an older sister, an older sister, and an aunt, respectively. FIG. 2 shows some interesting trends. That is, the utterance of the speaker (female) to the daughter (1 year old) shows the highest value in both the fundamental frequency f0 and the breathability. From the breathability, the family order is determined as follows. That is, the order of daughter> sister child> father> mother = sister> aunt> husband. It may be possible for this order to indicate the degree of “attentiveness” in the conversation within the family. The labeler also confirmed that the results were consistent with the impression they were hearing.
[0044]
FIG. 3 shows the NAQ and the fundamental frequency f0 by the conversation partner. In FIG. 3, “f” indicates a friend. “M” represents a family and “t” represents another person. Interestingly, the NAQ value for “a” (attentive utterance) to friends is high, and no difference is seen between “b” (close conversation) and “c” (complex conversation), but between families This is the reverse of this relationship. That is, there is no difference between a careful conversation and a close conversation, whereas the NAQ value is considerably low in a loose conversation. Regarding conversations with others, there were no tedious conversations, but careful and intimate conversations showed NAQ differences as expected.
[0045]
FIG. 4 discusses the difference between speech and its purpose. From what has already been stated, it is predicted that in a careful conversation, the NAQ value will be higher than in a more “fair” conversation. FIG. 4 shows that this prediction is correct. FIG. 4 shows five categories (tweet (“?”), Interjection (“I”), information provision (“e”), information request (“re”), and repetition request (“rz”)). NAQ and fundamental frequency f0.
[0046]
Referring to FIG. 4, the NAQ value for providing information is significantly lower than the value for requesting information. Further, the NAQ value of the repeated request is the highest. “Mutter” is considered to be a different category from the others, but it is also supported by FIG. In other words, for tweet, f0 is extremely low, indicating a breathable (high NAQ value) voice quality.
[0047]
From the above, it can be seen that the voice quality measured by NAQ has a large correlation with the conversation partner, the speech style, and the purpose of the speech. The NAQ changes depending on the degree of “attention” received in conversation, and changes independently from the fundamental frequency. Therefore, this voice quality can be considered as a prosodic feature together with the fundamental frequency f0, the length of speech, and the amplitude, and should be controlled in speech synthesis in order to show semantic non-verbal differences.
[0048]
-Configuration of speech synthesizer-
An embodiment of a speech synthesizer that performs speech synthesis reflecting semantic non-linguistic differences by controlling voice quality measured by NAQ in accordance with the above-described concept will be described below.
[0049]
FIG. 5 shows a block diagram of the speech synthesizer according to this embodiment. Referring to FIG. 5, the speech synthesizer preprocesses an input XML (Extended Mark-Up Language) sentence 30 including an input text synthesis target attribute and attributes representing non-language information, and performs speech synthesis. For the target text generated by the pre-processing unit 32 for creating the target text, the balance sentence speech DB 34 of a specific speaker prepared in advance, and the pre-processing unit 32, an appropriate phoneme is obtained from the balance sentence speech DB. By selecting and connecting columns, a waveform generation unit 36 for generating audio waveform data for the input XML 30 and an audio signal synthesis for synthesizing an audio signal based on the audio waveform data generated by the waveform generation unit 36 Part 38.
[0050]
A conventional speech synthesis technique can be used for both the waveform generation unit 36 and the speech signal synthesis unit 38. Since the voice of the balance sentence voice DB 34 is not a natural voice, the generated voice is hard and cannot be said to be a natural voice. However, since each phoneme included in the balance sentence speech DB 34 is obtained from the reading of the phoneme balance sentence, it can be appropriately labeled. As a result, the audio signal output from the audio signal synthesizer 38 is a raw audio signal that is relatively hard to the non-language information specified by the input XML 30 although it is uncured.
[0051]
The apparatus according to the present embodiment further synthesizes the speech signal obtained as the output of the speech signal synthesizer 38 by using the natural speech speech data as the acoustic target 40 for natural speech synthesis. In other words, the synthesized speech signal 54 close to natural speech is obtained. Therefore, in addition to each component described above, the apparatus according to the present embodiment is prepared in advance by collecting natural utterances of the same speaker (or a person who makes a similar voice) as the speaker of the balanced sentence speech DB 34. The natural utterance voice DB 42 thus used is used. The natural utterance voice DB 42 is obtained by collecting the natural utterances of the above-described speakers, and collects voice data in various situations. However, the voice data in the spontaneously uttered voice DB 42 is not labeled for extracting voice in accordance with the non-language information described above. This is because it is difficult to label such utterances as described in the explanation of the conventional technology.
[0052]
In addition, for each time period of the acoustic target 40, this apparatus uses, as a candidate for speech synthesis, speech data that is relatively close to the natural speech DB 42 by DP matching (DP distance is small, that is, similarity is high). A candidate selection unit 44 for selecting a plurality of candidates and outputting them as candidate strings 46, and obtaining a predetermined prosodic attribute for each candidate in the candidate string 46, and matching that part with the non-linguistic information specified by the input XML 30 And a filter unit 48 for selecting only those showing the prosodic attributes. The time period used here is of variable length.
[0053]
The prosodic attributes that the filter unit 48 obtains from each candidate string includes the NAQ described above in addition to the well-known fundamental frequency f0, power of speech data, and length of speech. For example, for each of these elements, in the input XML 30, a feature vector (or information for calculating the feature vector) is assigned as non-language information in advance for each utterance unit (for example, sentence). Such information can be calculated for each candidate, and a feature vector for comparison can be created. The filter unit 48 calculates the distance between the feature vector calculated for each candidate and the feature vector assigned to the utterance unit in the input XML 30, and is the candidate indicating the smallest distance and connected Select a candidate that can be smoothly connected. In this way, the filter unit 48 outputs a final audio data string 50 for final speech synthesis.
[0054]
The apparatus further includes a waveform generation unit 52 for generating a waveform based on the final audio data string 50. The synthesized speech signal 54 output from the waveform generation unit 52 is synthesized based on the speech data extracted from the natural speech speech DB 42, and each speech unit is non-linguistic information given to the speech unit in the input XML 30. It will match well. Therefore, the synthesized speech signal 54 is a sound that can be heard naturally and closely matches the designated speech mode.
[0055]
-Operation of speech synthesizer-
This device operates as follows. When the input XML 30 is given to the pre-processing unit 32, the pre-processing unit 32 creates a text to be speech synthesized for each utterance unit, and extracts the non-linguistic information given to each utterance unit in the input XML 30. The waveform generation unit 36 extracts voice data for synthesizing the text given by the preprocessing unit 32 from the balance sentence voice DB 34 from the balance sentence voice DB 34 that is a reading voice database created from the voice that has read the balance sentence. . At this time, the waveform generator 36 extracts voice data with a label that matches the non-language information given from the preprocessor 32. Further, the waveform generation unit 36 smoothly connects the extracted audio data according to the conventional technique, and supplies it to the audio signal synthesis unit 38.
[0056]
  The speech signal synthesizer 38 performs speech synthesis according to the conventional technique based on this speech data string, outputs an acoustic target 40 for natural speech synthesis, and provides it to the candidate selector 44. An example of this acoustic target 40 is shown in FIG. In the example shown in FIG. 6, the acoustic target 40 includes time periods 92, 94, 96 and 98. This period is variable length. These time periods may partially overlap each other..
  Referring to FIG. 5, candidate selection unit 44 performs speech data candidate sequence 112 similar to the waveform of acoustic target 40 by DP matching from natural utterance speech DB 42 for each section 92, 94, 96 and 98 shown in FIG. 6. , 114, 116, and 118 are extracted. Each of the audio data candidate columns 112, 114, 116, and 118 includes a plurality of audio data candidates. In the present embodiment, the candidate selection unit 44 selects a predetermined plurality as candidates from the one having the smallest DP distance. The candidate selection unit 44 gives these speech data candidate sequences 112, 114, 116, and 118 to the filter unit 48 as the candidate sequence 46 shown in FIG.
[0057]
For example, for the time period 92 shown in FIG. 6, the filter unit 48 calculates the feature vector of each candidate included in the speech data candidate string 112. Then, the feature vector is compared with the feature vector assigned in the input XML 30, and the cosine scale (that is, the similarity) calculated between them is small. A candidate 132 that can be connected is selected. Similarly, the filter unit 48 extracts candidates 134, 136, and 138 from a plurality of candidates for the time periods 94, 96, 98, and the like. These are the final audio data string 50 shown in FIG.
[0058]
The waveform generator 52 outputs a synthesized voice signal 54 in which these final voice data strings 50 are smoothly connected.
[0059]
According to the apparatus of the present embodiment described above, the acoustic target 40 is generated once using the balance sentence speech DB 34, and the non-linguistic features that are close to the acoustic target 40 and are assigned to the input XML 30 The voice data indicating the matched prosodic features can be extracted from the spontaneous speech DB 42. A synthesized voice signal 54 synthesized from this voice data string can be obtained. Therefore, the synthesized speech signal 54 is a sound that can be heard naturally and well matches the non-linguistic feature that was initially specified. Further, it is not necessary to label the voice data in the natural utterance voice DB 42 in advance for extraction from the natural utterance voice DB 42. It is only necessary to label the balance sentence speech DB 34, and this can be easily performed.
[0060]
In the above-described embodiment, the candidate selection unit 44 selects a predetermined plurality in order from the smallest DP distance. However, the present invention is not limited to such an embodiment. For example, the candidate selection unit 44 may select only candidates whose DP distance is smaller than a predetermined threshold. Alternatively, only those having a smaller DP distance may be selected in order from the smallest DP distance and smaller than a predetermined threshold value.
[0061]
Note that the apparatus according to the embodiment described herein can be realized by one or more computers and software executed on the one or more computers. The control structure of the software corresponds well with the block diagram shown in FIG. Therefore, the details are not described here. It will be clear to those skilled in the art how to configure the software from the above description.
[0062]
The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim in the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are intended. Including.
[Brief description of the drawings]
FIG. 1 is a diagram for explaining the principle of an apparatus according to an embodiment of the present invention.
FIG. 2 is a diagram for showing a NAQ and a fundamental frequency f0 for a family.
FIG. 3 is a diagram for showing a NAQ and a fundamental frequency f0 according to the type of the other party.
FIG. 4 is a diagram illustrating NAQ and fundamental frequency f0 depending on the purpose of speech.
FIG. 5 is a block diagram of an apparatus according to an embodiment of the present invention.
FIG. 6 is a diagram for explaining the operation of the apparatus according to the embodiment of the present invention.
[Explanation of symbols]
30 input XML, 32 preprocessing unit, 34 balance sentence speech DB, 36 waveform generation unit, 38 speech signal synthesis unit, 40 acoustic target, 42 spontaneous speech DB, 44 candidate selection unit, 46 candidate string, 48 filter unit, 50 final audio data string, 52 waveform generator, 54 synthesized audio signal

Claims (6)

A reading speech database consisting of reading speech data that is pre-labeled with language information,
A spontaneous speech database consisting of spontaneous speech data;
Corresponding to the text information by receiving text information pre-assigned non-linguistic information and extracting speech data with linguistic information matching the non-linguistic information attached to the text information from the reading speech database Voice synthesis means for synthesizing a voice signal to be transmitted;
Candidate selection means for selecting a plurality of pieces of spontaneous utterance voice data in order from ones having a small distance defined between each part of the voice signal from the spontaneous utterance voice database;
For each part of the speech signal, a prosodic feature predetermined for each of the plurality of spontaneous utterance data selected by the candidate selection unit is calculated from the spontaneous utterance speech database, and is added to the text information. Filter means for selecting those that match the non-linguistic information;
Means for synthesizing a speech signal based on the natural speech data selected by the filter means. The non-linguistic information given in advance to the text information is a feature vector indicating the predetermined prosodic feature,
The filter means calculates a feature vector indicating the predetermined prosodic feature for each of a plurality of natural utterance data selected by the candidate selection means, and a feature vector pre-assigned to the text information; The speech synthesizer according to claim 1, comprising means for selecting the one having the highest similarity between the two. The speech synthesis apparatus according to claim 2, wherein the predetermined prosodic feature includes at least one of a normalized amplitude index, a power of the speech signal, a duration of the speech signal, and a fundamental frequency. The candidate selecting means, for each part of the speech signal, a DP distance calculated by DP (Dynamic Programming) matching with each part from the spontaneous speech database is smaller than a predetermined threshold value. 4. A speech synthesizer according to any one of claims 1 to 3, comprising means for selecting one. The candidate selecting means is predetermined for each part of the voice signal in order from a smallest DP distance calculated by DP (Dynamic Programming) matching with each part from the spontaneous speech database. The speech synthesizer according to any one of claims 1 to 3, comprising means for selecting only a plurality. A computer program that, when executed by a computer, causes the computer to operate as the speech synthesizer according to any one of claims 1 to 5.

Images