EP0363233A1 - Method and apparatus for speech synthesis by wave form overlapping and adding - Google Patents

Abstract

Speech is synthesised from elements such as diphones. At least the vocalized sounds of the sound elements are subjected to window analysis centred essentially on the start of each pulsed response of the vocal tract to the stimulation of the vocal cords, using a filtering window whose amplitude decreases to zero at the edges of the window and whose width is equal to at least twice the fundamental period of origin or twice the fundamental period of synthesis. The signals corresponding to each sound element which are obtained from the window analysis are replaced, the phase shift of the signals being equal to the fundamental period of synthesis, which is greater or less than the fundamental period of origin, depending on the prosodic data concerning the synthesis frequency. Synthesis is carried out by addition of the phase-shifted signals.

Description

The invention relates to methods and devices for speech synthesis; it relates, more particularly, to the synthesis from a dictionary of sound elements by cutting up the text to be synthesized into microtrames each identified by a serial number of corresponding sound element and by prosodic parameters (information of sound height at the beginning and at the end of the sound element and duration of the sound element), then by adaptation and concatenation of the sound elements by an addition-recovery procedure.

The sound elements stored in the dictionary will frequently be diphones, that is to say transitions between phonemes, which makes it possible, for the French language, to be satisfied with a dictionary of approximately 1300 sound elements; however, different sound elements can be used, for example syllables or even words. The prosodic parameters are determined according to criteria relating to the context: the pitch which corresponds to the intonation, depends on the location of the sound element in a word and in the sentence and the duration given to the element. sound is a function of the rhythm of the sentence.

It should be remembered in passing that the methods of speech synthesis are subdivided into two groups. Those which use a mathematical model of the vocal tract (synthesis by linear prediction, synthesis with formants and synthesis with fast Fourier transform) involve a deconvolution of the source and the transfer function of the vocal tract and generally require around fifty arithmetic operations by digital sample of speech before digital-analog conversion and restitution.

This source-duct voice deconvolution allows on the one hand the modification of the value of the fundamental frequency of the voiced sounds, that is to say sounds which have a harmonic structure and are caused by vibration of the vocal cords, and on the other hand, the compression of the data representing the speech signal.

Those which belong to the second group of processes use synthesis in the time domain by concatenation of waveforms. This solution has the advantage of flexibility of use and the possibility of considerably reducing the number of arithmetic operations per sample. In return, it does not make it possible to reduce the bit rate necessary for transmission as much as the methods based on a mathematical model. However, this drawback disappears when essentially a good quality of reproduction is sought without being hampered by the need to transmit data over a narrow channel.

The speech synthesis according to the present invention belongs to the second group. It finds a particularly important application in the field of the transformation of a spelling chain (constituted for example by the text supplied by a printer) into a speech signal, for example restored directly or emitted on a normal telephone line.

We already know (Diphone synthesis using an overlap-add technique for speech waveforms concatenation, CHARPENTIER et al, ICASSP 1986, IEEE-IECEJ-ASJ International Conference on Acoustics Speech and Signal Processing, pages 2 015-2 018) speech from sound elements using a short-term addition-recovery technique. But these are short-term synthesis signals, with normalization of the overlap of the synthesis windows, obtained by a very complex process:
- analysis of the original signal by synchronous windowing of the voicing;
- Fourier transform of the short-term signal;
- envelope detection;
- homothety of the frequency axis on the spectrum of the source;
weighting of the modified spectrum of the source by the envelope of the original signal;
- inverse Fourier transform.

The present invention aims in particular to provide a relatively simple method and allowing acceptable speech reproduction. It starts from the hypothesis that voiced sounds can be considered as the sum of the impulse responses of a filter, stationary for several milliseconds, (corresponding to the vocal tract) excited by a Dirac sequence, that is to say by a "pulse comb", synchronously with the fundamental frequency of the source, that is to say vocal cords, which is reflected in the spectral domain by a harmonic spectrum, the harmonics being spaced from the fundamental frequency and weighted by an envelope having maxima called formants, depending on the transfer function of the vocal tract.

It has already been proposed (Micro-phonemic method of speech synthesis, Lucaszewic et al, ICASSP 1987, IEEE, pages 1426-1429) to perform speech synthesis where the reduction of the fundamental frequency of voiced sounds, when necessary to respect prosodic data, is carried out by insertion of zeros, the microphonemes stored must then necessarily correspond to the maximum possible height of the sound to be restored, or else (US Pat. No. 4,692,941) to decrease in the same way by insertion of zeros fundamental frequency, and to increase this by decreasing the size of each period. These two methods introduce significant distortions on the speech signal when the fundamental frequency is modified.

The present invention aims to provide a method and a device for concatenating synthesis of waveforms which do not have the above limitation and which make it possible to provide good quality speech, while requiring only a small volume of arithmetic calculations. .

To this end, the invention notably proposes a method characterized in that:
- a windowing centered at the start of each impulse response from the vocal tract to the excitation of the vocal cords (at the start of each voice impulse) can be performed, at least on voiced voices of the sound elements a window having a maximum for said start and an amplitude decreasing to zero at the edge of the window, and
- the windowed signals corresponding to each sound element are replaced with a time offset equal to the fundamental synthesis period to be obtained, less than or greater than the original fundamental period according to the prosodic pitch information of the fundamental frequency and a summation of these signals.

These operations constitute the recovery procedure then addition of the elementary waveforms obtained by windowing the speech signal.

In general, we will use sound elements consisting of diphones.

The width of the window can vary between values less than and greater than twice the original period. In the example of implementation which will be described later, the width of the window is advantageously chosen to be approximately twice the original period in the event of an increase in the fundamental period or approximately twice the final period of synthesis in the event of an increase in the fundamental frequency, in order to partially compensate for the energy modifications due to the change in the fundamental frequency, not compensated for by a possible normalization of the energy, taking into account the contribution of each window to the amplitude of the samples of the digital synthesis signal: in the case of a decrease in the fundamental period, the width of the window will therefore be less than twice the original fundamental period. It is undesirable to go below this value.

Because it is possible to modify the value of the fundamental frequency in both directions, the diphones are memorized with the natural fundamental frequency of the speaker.

With a window of duration equal to two consecutive fundamental periods in the voiced case, elementary waveforms are obtained whose spectrum represents substantially the envelope of the spectrum of the speech signal or short-term broadband spectrum - since this spectrum is obtained by convolution of the harmonic spectrum of the speech signal and the frequency response of the window, which in this case has a bandwidth greater than the distance between harmonics; the temporal redistribution of these elementary waveforms will give a signal having substantially the same envelope as the original signal but a modified distance between harmonics.

With a window of duration greater than two fundamental periods, we obtain elementary waveforms the spectrum of which is still harmonic, or short-term spectrum narrow band - the fact that this time the frequency response of the window is less wide that the distance between harmonics-; the temporal redistribution of these elementary waveforms will give a signal having, like the synthesis signal previous, roughly the same envelope as the original signal except that we will have introduced reverberation terms (signals whose spectrum has a smaller amplitude, a different phase, but the same shape as the amplitude spectrum of the signal original), the effect of which will only be audible beyond window widths of around three periods, this reverberation effect not degrading the quality of the synthesis signal when its amplitude is low.

One can in particular use a Hanning window, although other forms of window are also acceptable.

The processing defined above can also be applied to so-called deaf or unvoiced sounds, which can be represented by a signal whose shape is similar to that of white noise, but without synchronization of the windowed signals: this is intended to '' homogenize the processing on deaf sounds and voiced sounds, which allows on the one hand the smoothing between sound elements (diphones) and between deaf and voiced phonemes, and on the other hand a modification of the rhythm. There is a problem at the junction between diphones. One solution to avoid this difficulty consists in omitting the extraction of elementary waveforms from the two adjacent fundamental transition periods between diphones (in the case of deaf sounds, the voicing marks are replaced by arbitrary marks): we can either define a third elementary wave function by calculating the average of the two elementary wave functions extracted on either side of the diphone, or use the addition-recovery procedure directly on these two elementary wave functions .

• - la Figure 1 est un graphe destiné à illustrer la synthèse de la parole par concaténation de diphones et modification des paramètres prosodiques dans le domaine temporel, conformément à l'invention ;
• - la Figure 2 est un schéma synoptique montrant une constitution possible du dispositif de synthèse, implanté sur un calculateur hôte ;
• - la Figure 3 montre, à titre d'exemple, comment on modifie les paramètres prosodiques d'un signal natu­rel, dans le cas d'un phonème particulier ;
• - les Figures 4A, 4B et 4C sont des graphiques destinés à montrer des modifications spectrales appor­tées à des signaux de synthèse voisés, la Figure 4A montrant le spectre d'origine, la Figure 4B le spectre avec diminution de la fréquence fondamentale et la Figure 4C le spectre avec augmentation de cette fréquence ;
• - la Figure 5 est un graphique montrant un prin­cipe d'atténuation des discontinuités entre diphones ;
• - la Figure 6 est un schéma montrant le fenêtrage sur plus de deux périodes.

The invention will be better understood on reading the following description of a particular embodiment of the invention, given by way of example limiting. The description refers to the accompanying drawings, in which:

• - Figure 1 is a graph intended to illustrate the synthesis of speech by concatenation of diphones and modification of the prosodic parameters in the time domain, in accordance with the invention;
• - Figure 2 is a block diagram showing a possible constitution of the synthesis device, installed on a host computer;
• - Figure 3 shows, by way of example, how the prosodic parameters of a natural signal are modified, in the case of a particular phoneme;
• - Figures 4A, 4B and 4C are graphs intended to show spectral modifications made to voiced synthesis signals, Figure 4A showing the original spectrum, Figure 4B the spectrum with decrease in fundamental frequency and Figure 4C the spectrum with increase in this frequency;
• - Figure 5 is a graph showing a principle of attenuation of discontinuities between diphones;
• - Figure 6 is a diagram showing windowing over more than two periods.

The synthesis of a phoneme is carried out from two diphones stored in a dictionary, each phoneme being composed of two half-diphones. The sound "é" in "period" for example will be obtained from the second half-diphone of "pai" and the first half-diphone of "air".

A phonetic orthographic translation and prosody calculation module (which is not part of the invention) provides, at a given instant, indications identifying:
- the phoneme to be restored, of order P
- the previous phoneme, of order P-1
- the following phoneme, of order P + 1
and giving the duration to be assigned to the phoneme P as well as the periods at the beginning and at the end (Figure 1).

A first analysis operation, which is not modified by the invention, consists in determining, by decoding the names of the phonemes and prosodic indications, the two diphones selected for the phoneme to be used and the voicing.

All the available diphones (1300 for example) are stored in a dictionary 10 provided with a table constituting the descriptor 12 and containing the address of the start of each diphone (in number of blocks of 256 bytes) the length of the diphone and the middle of the diphone (these last two parameters being expressed in number of samples from the start) and voicing marks marking the beginning of the response of the vocal tract to the excitation of the vocal cords in the case of a sound voiced (35 in number for example). Dictionaries of diphones meeting these criteria are available, for example, from the National Center for Telecommunications Studies.

The diphones are then used in a process of analysis and synthesis shown diagrammatically in FIG. 1. This process will be described assuming that it is implemented in a synthesis device having the constitution shown in FIG. 2, intended to be connected to a host computer, such as the central processor of a personal computer. We will also assume that the sampling frequency giving the representation of the diphones is 16 kHz.

The synthesis device (FIG. 2) then comprises a main random access memory 16 which contains a microcomputing program, the dictionary of diphones 10 (that is to say waveforms represented by samples) stored in the order of the addresses of the descriptor, the table 12 constituting the dictionary descriptor, and a Hanning window, sampled for example on 500 points. The random access memory 16 also constitutes micro-frame memory and working memory. It is connected by a data bus 18 and an address bus 20 to an access 22 to the host computer.

Each micro-frame emitted to restore a phoneme (Figure 2) is made up, for each of the two phonemes P and P + 1 which are involved
- the serial number of the phoneme,
- the value of the period at the start of the phoneme, the value of the period at the end of the phoneme, and
- the total duration of the phoneme which can be replaced by the duration of the diphone for the second phoneme.

The device also comprises, connected to the buses 18 and 20, a local computing unit 24 and a switching circuit 26. The latter makes it possible to connect a random access memory 28 serving as an output buffer either to the computer or to a controller 30 of digital / analog converter 32 output. The latter attacks a low-pass filter 34, generally limited to 8 kHz, which supplies a speech amplifier 36.

The operation of the device is as follows.

The host computer (not shown) loads the micro-frames into the table reserved in memory 16, via access 22 and buses 18 and 20, then it commands the start of synthesis to the calculation unit 24 This calculation unit searches for the number of the current phoneme P, of the next phoneme P + 1 and of the previous phoneme P-1 in the micro-frame table, using an index stored in the working memory, initialized to 1. In the case of the first phoneme, the calculating unit searches only for the numbers of the current phoneme and the next phoneme. In the case of the last phoneme, it picks up the number of the previous phoneme and that of the current phoneme.

In the general case, a phoneme consists of two half-diphones; the address of each diphone is sought by matrix addressing in the dictionary descriptor by the following formula:
diphone descriptor number =
number of 1st phoneme + (number of 2nd phoneme-1) * number of diphones

Sounds voiced

The computing unit loads, into working memory 16, the address of the diphone, its length, its middle as well as the thirty-five voicing marks. It then loads, into a descriptor table of the phoneme, the voicing marks corresponding to the second part of the diphone. Then she searches in the waveform dictionary for the second part of the diphone, which she places in a table representing the signal of the analysis phoneme. The marks kept in the phoneme descriptor table are decremented by the value of the middle of the diphone.

This operation is repeated for the second part of the phoneme constituted by the first part of the second diphone. The voicing marks of the first part of the second diphone are added to the voicing marks of the phoneme and incremented by the value of the middle of the phoneme.

In the case of voiced sounds, the calculation unit, from the prosodic parameters (duration, beginning period and end period of the phoneme) then determines the number of periods necessary for the duration of the phoneme, according to the formula:
number of periods = 2 * duration of the phoneme / (start period + end period)

The calculation unit stores in memory the number of marks of the natural phoneme, equal to the number of voicing marks, then determines the number of periods to be eliminated or added by making the difference between the number of synthesis periods and the number of periods of analysis, difference which is fixed by modifying the tone to be introduced from that corresponding to the dictionary.

For each synthesis period selected, the calculation unit then determines the analysis period selected from the phoneme periods based on the following considerations:
the modification of the duration can be considered as the matching, by deformation of the time axis of the synthesis signal, of the n voicing marks of the analysis signal and of the p marks of the synthesis signal, n and p being predetermined integers;
- to each of the p marks of the synthesis signal must be associated the mark closest to the analysis signal.

The duplication or, on the contrary, the elimination of periods equally distributed on all the phoneme modifies the duration of this one.

Note that we will not have to extract an elementary waveform from the two adjacent transition periods between diphones: the operation of addition-recovery of elementary functions extracted from the last two periods of the first diphone and first two periods of the second diphone allows smoothing between these diphones as shown in Figure 5.

For each summary period, the calculation unit determines the number of points to add or delete to the analysis period by making the difference between the latter and the summary period.

As indicated above, it is advantageous to choose the width of the analysis window as follows, illustrated in Figure 3:
- if the synthesis period is less than the analysis period (lines A and B of Figure 3), the size of the window 38 is twice the synthesis period;
- otherwise, the size of the window 40 is obtained by multiplying by two the lowest of the values of the current analysis period and of the previous analysis period (lines C and D).

The calculation unit determines a progress step in reading the values of the window, tabulated for example on 500 points, the step then being equal to 500 divided by the size of the window previously calculated. It reads the samples from the previous period and the current period in the signal phoneme 28 signal buffer memory, weights them by the value of the Hanning window 38 or 40 indexed by the number of the current sample multiplied by the step of advancement in the tabulated window and adds, progressively, the calculated values to the buffer of the output signal indexed by the sum of the counter of the current output sample and the search index of the samples of the analysis phoneme. The current output counter is then incremented by the value of the synthesis period.

Deaf sounds (not seen)

For deaf phonemes, the processing is analogous to the previous one, except that the value of the pseudo-periods (distance between two voicing marks) is never modified: the elimination of pseudo-periods in the center of the phoneme simply decreases the duration of this one.

The duration of deaf phonemes is not increased, except by adding zeros in the middle of the "silent" phonemes.

Windowing is carried out by period to normalize the sum of the values of the windows applied to the signal:
- from the start of the previous period to the end of the previous period, the step forward in reading the tabbed window is (in the case of a tabulation on 500 points) equal to 500 divided by twice the duration of the previous period;
- from the start of the current period to the end of the current period, the advancement step in the tabulated window is equal to 500 divided by twice the duration of the current period plus a constant offset of 250 points.

At the end of the calculation of the signal of a synthesis phoneme, the calculation unit stores the last period of the analysis and synthesis phoneme in the buffer memory 28 which allows the transition between phonemes. The counter of the output current sample is decremented by the value of the last synthesis period.

The signal thus generated is sent, in blocks of 2,048 samples, to one of two memory spaces reserved for communication between the calculation unit and the controller 30 of the digital / analog converter 32. As soon as the first block is loaded into the first buffer zone, the controller 30 is activated by the calculation unit and empties this first buffer zone. During this time, the calculation unit fills a second buffer zone of 2048 samples. The calculation unit then alternately tests these two buffer zones with a flag to load the digital synthesis signal at the end of each synthesis sequence of a phoneme. The controller 30, at the end of reading of each buffer zone, sets the corresponding flag. At the end of the synthesis, the controller empties the last buffer zone and sets a flag for the end of synthesis which the host computer can read via the communication access 22.

The example of analysis and synthesis voiced speech signal spectrum illustrated in FIGS. 4A-4C shows that the temporal transformations of the digital speech signal do not affect the envelope of the synthesis signal, while modifying the distance between harmonics, i.e. the fundamental frequency of the speech signal.

The complexity of the calculation remains low: the number of operations per sample is on average two multiplications and two additions for the weighting and the summation of the elementary functions provided by the analysis.

The invention is susceptible of numerous variant embodiments and, in particular, as indicated above, a window of width greater than two periods, as shown in FIG. 6, possibly of fixed size, can give acceptable results. .

The method of modifying the fundamental frequency on digital speech signals can also be used outside of its application to synthesis by diphones.

Claims (5)

1. Method of speech synthesis from sound elements (words, syllables, diphones ...) characterized in that:
- a windowing analysis is carried out, at least on the voices voiced by the sound elements, substantially centered on the start of each impulse response of the vocal tract to the excitation of the vocal cords using a filtering window having an amplitude decreasing to zero at the edges of the window whose width is at least equal to twice the original fundamental period or twice the fundamental synthesis period,
- the signals resulting from the windowing corresponding to each sound element are replaced, with a time shift of these equal to the fundamental synthesis period, lower or greater than the original fundamental period, according to the prosodic information concerning the fundamental frequency of synthesis,
- Synthesis is carried out by summing the signals thus shifted. 2. speech synthesis method according to claim 1, characterized in that a dictionary of sound elements, for example of diphones is produced, the text to be synthesized is cut up into micro-frames each identified by the number of the element corresponding sound (diphone) and at least one prosodic piece of information, constituted at least by the value of the fundamental frequency at the beginning and at the end of the element and by the duration of the element. 3. Speech synthesis method according to one of claims 1 and 2, characterized in that the width of the window is equal to twice the original period in the event of a decrease in the fundamental frequency or twice the final synthesis period in the event of an increase in the fundamental frequency. 4. Speech synthesis method according to one of claims 1 to 3, characterized in that the window is a Hanning window. 5. A device for synthesizing speech by implementing the method according to claim 1, characterized in that it comprises, connected to buses (18,20): a main random access memory (16) which contains a micro-program of computation, a dictionary of diphones (10) made up of waveforms represented by samples arranged in the order of the addresses of a descriptor (12) of dictionary, and a sampled Hanning window, said random access memory (16) constituting also micro-frame memory and working memory; a local computing unit (24) and a routing circuit (26) making it possible to connect a random access memory (28) serving as an output buffer either to the computing unit or to a digital converter / controller (30) analog (32) output driving a low pass filter (34) which powers a speech amplifier (36).

Images