JPH0372995B2 - - Google Patents

Description

[Detailed description of the invention] The present invention will be explained in the following order.

A. Industrial field of application B. Overview of the disclosure C. Conventional technology D. Problem to be solved by the invention E. Means for solving the problem F. Example F1. Environment of speech recognition system (Figures 2 to 4)
F2 Auditory model and its realization in the acoustic processor of the speech recognition system (Figures 8 to 14) F3 Precision matching (Figures 4, 15, 16)
Figure) F4 Phoneme tree structure (Figure 17) F5 Language model (Figure 2) F6 Approximate shaping (Figure 18) F7 Extension of word path using words selected by acoustic matching (Figures 5 to 7)
Figure 19) Markov model constructed from multiple utterances of the F8 word (Figures 1A, 1B, 1
(Figure C, Figures 20 to 26) G Effects of the Invention A Field of Industrial Application The present invention relates generally to the field of speech recognition, and more particularly to the construction of basic forms in speech recognition systems.

B. Summary of the Disclosure The present invention provides a basic form of Hueneme (the basic form refers to the Ward-Markov model), which takes into account variations in word pronunciation from utterance to utterance.
A set of labels each representing an acoustic type that can be assigned to a minute time interval obtained from the signal processing processor at the front end is taken as a basic phoneme, and the corresponding partial Markov models are concatenated. This approach addresses the problem of constructing a Ward Markov model (called the basic form of the Feneme type). Specifically, the present invention provides a method for (a) uttering multiple utterances of a word to each
(b) form a set of Feneme Markov model phoneme machines, (c) determine the best one phoneme machine P ₁ to generate the Feneme strings, and (d) form a set of Feneme Markov model phoneme machines. (e) determining the best two phoneme basic forms of the form P ₁ P ₂ or P ₂ P ₁ to generate the Hueneme string; (e) aligning the best two phoneme basic forms for each Hueneme string; , (f) Split each Hueneme string into a left part and a right part, with the left part corresponding to the first phoneme machine of the two phoneme basic forms, and the right part corresponding to the first phoneme machine of the two phoneme basic forms. (g) identifying each of the left parts as a left substring and each of the right parts as a right substring; (h) a set of phoneme strings corresponding to the plurality of utterances; Process the left substring set and right substring set in the same way,
Furthermore, if the single phoneme base form produces the substring with a higher probability than the best two phoneme base form does, then the step of prohibiting further splitting of the substring (i) Concatenate unsegmented single phonemes in an order that corresponds to the order of their corresponding Feneme substrings.

It concerns a method of constructing the basic form of a word with a vocabulary of word segments containing steps.

C. Prior Art In some speech recognition systems, an acoustic processor receives speech as input and generates a string of labels for it. These labels are selected by the audio processor from an alphabet, ie, a unique set of labels based on certain characteristics of the input audio.

Generally, an audio processor examines the power spectral characteristics of the input audio over centisecond intervals and assigns a label (called a hue) to each interval. Therefore, according to the input speech, the sound processor generates a corresponding string of sounds.

In statistical methods of recognizing speech, a limited set of models is determined. Each model is a Markov model, ie, a probabilistically limited state phoneme machine, that generates a feneme. This method was published in IEEE Bulletin: Pattern Analysis and Computer Information (PAMI), Volume 5, No. 2 (March 1983 issue).
LRBahl et al.'s paper “Maximum Likelihood Method for Recognizing Continuous Speech” (LRBahl et al, “A
Maximum Likelihood Approach to
“Continuous Speech Recognition”, IEEE
Transactiou on Pattern Analysis and
Machine Intelligence, Vol.PAMI-5, No.2.
March 1983).

According to the probabilistic method, each phoneme machine has the following characteristics: (a) multiple states, (b) transitions between states (each transition has a probability associated with it), (c) at least some Each of the transitions has a plurality of output probabilities representing the probability of producing that particular Feneme. A phoneme machine may include null transitions that do not produce phonemes. Typically, for non-null transitions, there is a probability assigned to each corner of the alphabet.

After the input speech has been converted into a string of Feneme, the phoneme machine can be examined to determine the likelihood of the phoneme machine to generate substrings of Feneme within the string. The test can be performed phoneme by phone to determine the respective likelihood of each phoneme machine producing the substring. Similarly, a sequence of phonemes may be examined to determine the likelihood of phonemes in the sequence to generate a phoneme in the generated string.

Research by IBM has identified various types of phoneme machines. One type is a "phonetic phoneme machine," which stores statistical values that reflect the likelihood of a given phonetic element (when uttered) producing a pheneme in a pheneme string. Another type is a "Feneme-type phoneme machine" which stores statistics that reflect the likelihood of a given Feneme element (when uttered) producing a Feneme in a Feneme string.

Hueneme's phoneme machine has two states S ₁ and S ₂
has. One non-null transition is from S ₁ back to itself. Another non-null and null transition is S ₁
and S ₂ .

Each word in the vocabulary is represented by a predetermined sequence of phonemes (or phoneme machine) called a "word base form." A Hueneme basic form is a sequence of Hueneme phonemes that are concatenated to represent a given word. The basic form of a phonetic alphabet is a sequence of phonemes in a phonetic alphabet that are concatenated to represent a given word.

The likelihood that a word matches the input speech therefore reflects the probability of the basic form producing a hueneme in the string. In other words, Hueneme
The base form with the highest probability of producing a string represents the word that most closely matches the input speech.

How well the basic form corresponds to the word it represents is an important factor influencing the accuracy obtained by probabilistic methods.

One method for determining a basic form for each word in a vocabulary is called the singleton Hueneme basic form method. In this approach, each word is spoken only once. Associated with each cue, the one utterance of that word produces is the phoneme machine that has the highest probability of producing that cue.

In the singleton Hueneme basic form method,
Each phoneme machine is associated with one phoneme. Therefore, for each generated Feneme in the string,
There is one most promising corresponding phoneme machine. The sequence of phoneme machines that corresponds to the utterance of a word represents that word.

D. Problems to be Solved by the Invention The singleton-Feneme basic form approach involves several problems. The pronunciation of particular words can vary considerably. If one utterance that makes up the base form differs significantly from the pronunciation of the word at another time, the quality of speech recognition may deteriorate.

However, constructing a basic form based on multiple utterances of each word is not trivial.
By the way, the phoneme sequence that has the maximum probability of joining multiple utterances, that is, the basic form, is B = P ₁ P ₂ ...P _n (P _i (i = 1, 2, ..., m) is a phoneme) , _N 〓 ⁱ⁼¹ Pr(B|f _i1 f _i2 f _i3 ... f _i1i ). Here, f _{i1 .} . . f _i1i is the feneme string of the i-th utterance. Calculations using this formula are unacceptably expensive even with all known methods.

The aim of the invention is to improve singleton pronunciation by taking into account the changes that can occur between one utterance of a word segment and another.
The purpose is to improve the Hueneme basic form method.
A word segment can be a regular word or a portion thereof.

A further object of the invention is to improve the basic basic form by an iterative process.

Furthermore, it is an object of the invention to allow basic forms constructed from phonemes of phonetic alphabets or other types of phonemes to be used as well.

E. Means for Solving the Problems According to the invention, each basic form can be constructed by combining multiple utterances of the corresponding word segment in a so-called individual solution method, which allows the basic form to be constructed effectively without demanding time or computational requirements. Build on.

Specifically, one embodiment of the invention includes the following steps.

(a) Convert multiple utterances of a word segment into respective hueneme strings.

(b) Form a set of Feneme's Markov model phoneme machines.

(c) Determine the best single-phoneme machine P ₁ to generate multiple feneme strings.

(d) Determine the best two-phoneme basic form of the form P ₁ P ₂ or P ₂ P ₁ to generate multiple Feneme strings.

(e) Align the best diphoneme basic form for each Hueneme string.

(f) Divide each Hueneme string into a left part and a right part, with the left part corresponding to the first phoneme machine in the two-phoneme basic form and the right part corresponding to the second phoneme machine in the two-phoneme basic form. Make it compatible with phoneme machines.

(g) Identify each of the left parts as a left substring and each of the right parts as a right substring.

(h) Treat the set of left substrings in the same way as the set of phoneme strings corresponding to multiple utterances, and furthermore, treat the set of left substrings in the same way as the set of phoneme strings that correspond to multiple utterances, and furthermore that their single-phoneme base form is better than the best two-phoneme base form would produce. If the substring is generated with a high probability, further division of the substring is prohibited.

(i) Treat the set of right substrings in the same way as the set of pheneme strings corresponding to multiple utterances, and furthermore, if their single-phoneme base form is higher than the best diphoneme base form would produce. Prohibits further splitting of a substring when that substring is generated with probability.

(j) Concatenate unsegmented single phonemes in an order that corresponds to the order of their corresponding Hueneme substrings.

F Embodiment F1 Speech Recognition System Environment (FIGS. 2-4) FIG. 2 shows a schematic block diagram of a speech recognition system 1000 that provides an environment for the present invention. This system includes a stack decoder 1002 and an audio processor (AP) 1004 connected to it.
an array processor 1006 for performing fast approximate acoustic matching; an array processor 1008 for performing precise acoustic matching; a language model 1010;
and a workstation 1012.

The acoustic processor 1004 is designed to exchange the audio waveform input into a string of labels, ie, strings of phonemes, each of which approximately identifies the corresponding phonetic symbol. Generally, monophonic codes are defined by a clustering algorithm that can reflect a Gaussian or other distribution of spectral energy or other characteristics. In this system, the acoustic processor 1004 is based on a unique model of human hearing and is
No. 06/665401 (filed on October 26, 1984).

The labels or labels from audio processor 1004 are sent to stack decoder 1002. FIG. 3 shows the logic of stack decoder 1002. That is, stack decoder 1
002 is a search device 1020, a workstation 1012 connected thereto, interfaces 1022, 1024, 1026, and 102.
Contains 8. Each of these interfaces includes an audio processor 1004, an array processor 10
06, 1008 and language model 1010, respectively. In the system shown in FIG.
6 (high-speed matching). Array processor 1006 is designed to examine words in a vocabulary of words and reduce the number of candidate words for a given incoming label string. Fast matching is performed using a stochastically limited state machine (also referred to herein as a Markov model phoneme machine).

Preferably, precision matching tests these words by language model calculations from a list of fast match candidates that have a reasonable likelihood of being spoken words.

Alternatively, precision matching can be used for each word in the vocabulary. In this case, high-speed matching is omitted. Precise matching is performed by a Markov model phoneme machine as shown in FIG. 4 when the phoneme is of the phonetic type.

After fine matching, it is desirable to invoke the language model again and determine the likelihood of the word.

Fast matching, language models, precise matching, and language model procedures must be recognized as one system in which the present invention can be utilized. (of a phonetic alphabet, feneme, or other phoneme type)
Systems that involve only precision matching can utilize the present invention as well.

The purpose of the stack decoder 1002 is to
The task is to determine the word string that gives the highest probability for the string y ₁ y ₂ y ₃ .

This can be expressed mathematically as follows.

Max(Pr(W|Y) (1) This is the maximum probability of W giving Y over all word strings W. As is well known, Pr
(W|Y) can be written as follows.

Pr(W|Y)=Pr(W)・Pr(W|Y)/Pr(Y)
(2) However, Pr(Y) is unrelated to W.

One way to determine the most likely path (i.e. sequence) of consecutive words W☆ is to examine each possible path and determine the probability of each path resulting in the label string that we are trying to decode. . Then select the path with the highest associated probability. For a vocabulary of 5000 words, this method becomes unwieldy and impractical, especially when the strings of words are long.

Two other known methods of finding the maximum likelihood word sequence W* are Viterbi decoding and stack decoding. Each of these techniques is described in IEEE Proceedings on Pattern Analysis and Machine Information,
PAMI Vol. 5 No. 2, March 1983 issue
L R Bahl et al, “Maximum Likelihood Approach to Continuous Speech Recognition” (L R Bahl et al, “A
Maximum Likelihood Approach to
Continuous Speech Recognition，”IEEE
Transactions on Pattern Analysis and
Machine Intelligence, Vol. PAMI-5, No. 2,
(March 1983), respectively.

The stack decoding method in this paper is related to a single stack decoding. That is, paths of different lengths are listed in a single stack by likelihood, and decoding is performed based on this single stack. Single stack decoding is due to the fact that the likelihood depends somewhat on the path length and therefore normalization is generally performed. However, normalization requires that if the normalization factor is not estimated correctly,
Improper searching can result in excessive searching and searching errors.

The Viterbi method does not require normalization, but is generally only practical for small tasks.

With large vocabularies, the essentially time-synchronous Viterbi algorithm may have to interface with asynchronous acoustic matching components. In this case, the result is that the interface is not suitable.

An alternative novel apparatus and method (discussed below), invented by LRBahl et al., is capable of decoding the most likely word sequence W☆ with lower computational demands and higher accuracy than other techniques. Related to method. In particular, a technique is provided which is characterized by multiple stack decoding and a unique decision method to determine which word sequence to expand at a given time. According to this determination method, paths of relatively short length are not penalized because of their shortness, but are instead judged by their relative likelihood. The novel apparatus and method shown in FIGS. 5, 6 and 7 will be described in detail below.

Stack decoder 1002 actually acts to control other elements, but it does not perform much computation. Therefore, the stack decoder 1002 is an IBM VM/370 operating system (model 155, VS2, release 1.7).
It is desirable to include a 4341 processor running under the control of the 4341 processor. Array processors that perform a significant amount of computation use floating point
This is achieved using a commercially available 190L manufactured by System (FPS).

F2 Auditory model and its realization in the acoustic processor of the speech recognition system (Figures 8 to 14)
Figure) Figure 8 shows an audio processor 110 as described above.
A specific example of 0 is shown. Acoustic wave input (e.g.
natural speech) enters an A/D converter 1102 that samples at a predetermined rate. A typical sampling rate is one sample every 50 microseconds. A time window generator 1104 is provided to shape the edges of the digital signal. The output of the time window generator 1104 enters an FFT (Fast Fourier Transform) unit 1106 which provides a frequency spectral output for each time window.

Then, the output of the FFT device 1106 is labeled
L ₁ L ₂ ‥‥L _f is processed. Feature selector 1108, clusterer 1110, prototyper 1112, and encoder 1114 jointly generate labels. When generating labels, prototypes are formed as points (or vectors) in space based on selected features. The acoustic input is characterized by the same selected features to provide corresponding points (or vectors) in space that can be compared to the prototype.

Specifically, when defining a prototype, clustering device 1110 collects a set of points and groups them into clusters. The method of forming clusters is based on probability distributions (such as Gaussian distributions) applied to speech. The prototype of each cluster (with respect to the center locus or other characteristics of the cluster) is stored in the prototype device 11
12. The generated prototype and the acoustic input (both with the same features selected) enter the encoder 1114. Symbolization device 1114
performs a comparison procedure and, as a result, assigns a label to a particular acoustic input.

Selection of appropriate features is an important factor in deriving labels representing acoustic (speech) wave inputs. The acoustic processor is associated with an improved feature selector 1108. A unique auditory model is derived and used according to the acoustic processor. Auditory model, 9th
This will be explained using figures.

FIG. 9 shows a portion of the human inner ear. Specifically, the inner hair cells 1200 and the fluid-containing grooves 1
The distal end 1202 extending to 204 is shown in detail. Further, upstream from the inner hair cell 1200, there is an outer hair cell 1206 and an end portion 1 extending into the groove 1204.
208 is shown. Nerves that transmit information to the brain are connected to the inner hair cells 1200 and the outer hair cells 1206. Electrochemical changes occur in the basement membrane 1210
stimulated by mechanical movement.

It is known in the art that basilar membrane 1210 acts as a frequency analyzer of acoustic wave input, with sections along basilar membrane 1210 responding to respective critical frequency bands. Each portion of basilar membrane 1210 that responds to a corresponding frequency band influences the perceived loudness of the acoustic waveform input. That is,
The loudness of a tone is greater than if two tones of similar power intensity occupied the same frequency band.
It is perceived as louder when the two tones are in separate critical frequency bands. It has been found that there are 22 orders of critical frequency bands defined by the basilar membrane 1210.

In accordance with the frequency response of the basilar membrane 1210, the present invention advantageously defines an input acoustic waveform in some or all of the critical frequency bands and then separately determines the signal components for each defined critical frequency band. inspect. This function is available in FFT device 1
106 (FIG. 8) and feature selection device 1 for each tested critical frequency band.
This is done by providing a separate signal to 108.

The separate inputs are also blocked into time frames (preferably 25.6 milliseconds) by time window generator 1104. Therefore, the feature selector 1108 has 22
It is desirable to include the following signals. Each of these signals represents the sound intensity of a given frequency band for each time frame.

The signal is preferably filtered by a conventional critical band filter 1300 of FIG. The signals are then individually processed by a volume equalization transformer 1302 that perceives changes in volume as a function of frequency. Incidentally, the perceived loudness of a first tone at a given dB level at one frequency may differ from the loudness of a second tone at the same dB level at another frequency. Volume equalization converter 1302
is based on empirical data and transforms the signals in each frequency band so that each is measured on the same loudness scale. For example, volume equalization converter 1
302, with some modifications to the 1933 work of Fletcher and Munson, can map acoustic energy to equivalent loudness. Figure 11 shows the results of a modification to the previous study. According to Figure 11, at 40dB
It can be seen that a 1kHz tone corresponds to the volume level of a 100Hz tone at 60dB.

The volume equalization converter 1302 adjusts the volume according to the curve shown in FIG. 11, producing the same volume regardless of frequency.

In addition to the dependence on frequency, changes in power do not correspond to changes in volume, as can be seen by examining specific frequencies in FIG. That is, the intensity of the sound,
That is, variations in amplitude are not reflected in similar changes in perceived loudness at all points. for example,
At a frequency of 100Hz, a 10dB perceived volume change near 110dB is equivalent to a 10dB change near 20dB.
is much larger than the perceived volume change. This difference is processed by a volume compression device 1304, which compresses the volume in a predetermined manner. Volume compression device 1304
can compress the power P to its cube root P ^1/3 by replacing the volume amplitude measurement value in units of phons with units of sones.

FIG. 12 shows the known empirically determined Hong-to-Thorn relationship. With the use of sone units,
Our model remains nearly accurate even with large audio signal amplitudes. One sone is defined as a 1kHz tone with a volume of 40dB.

FIG. 10 shows a new time-varying response device 13.
06 is shown. The device operates with volume equalization and volume compression signals associated with each critical frequency band. Specifically, for each frequency band examined, the neural firing rate f is determined for each time frame. The firing rate f is defined according to the acoustic processor of the present invention as follows.

f=(So+DL)n (1) where n is the amount of neurotransmitter; So is the spontaneous firing constant involved in neural firing independent of acoustic waveform input; L is the measured sound volume; D is the displacement constant.
So·n corresponds to the spontaneous neural firing rate that occurs regardless of the presence or absence of sound wave input, and DLn corresponds to the firing rate due to acoustic wave input.

An important point is that the present invention has the characteristic that the value of n changes with time according to the following equation.

dn/dt=Ao−(So+Sh+DL)n (2) where Ao is the recruitment constant; Sh is the spontaneous neurotransmitter decay constant. The new function shown in equation (2) is that while neurotransmitters are generated at a constant rate Ao, (a) decay (Sh・n), (b) spontaneous firing (So・n), and (c) This takes into account the loss due to neural firing (DL・n) caused by acoustic wave input. It is assumed that these modeled phenomena occur at the locations shown in FIG.

As is clear from equation (2), the next amount of neurotransmitter and the next firing rate are proportional to at least the square of the current amount of neurotransmitter, indicating the fact that the acoustic processor of the present invention is nonlinear. There is. In other words, the amount of neurotransmitter in state (t+△t) is
It is equal to the amount of neurotransmitter in the state (t+dn/dt・Δt). Therefore, n(t+Δt)=n(t)+(dn/dt)·Δt (3) holds true.

Equations (1), (2) and (3) represent the operation of the time-varying signal analyzer. Time-varying signal analyzers point to the fact that the auditory system is time-adaptive, and the signals of the auditory nerve are non-linearly related to the acoustic wave input. Incidentally, the acoustic processor of the present invention provides the first model for implementing nonlinear signal processing in a speech recognition system to better track the apparent temporal changes in the neural system.

In order to reduce the number of unknown terms in equations (1) and (2), the present invention uses the following equation that is applied to a constant volume L.

So+Sh+DL=1/T (4) where T is the measurement of the time after the audio wave input is generated until the auditory response drops to 37% of its maximum value. T is a function of loudness and is taken from a known graph that displays the attenuation of the response for various loudness levels by the sound processor of the present invention. That is, when a tone of constant volume is generated, a high level response initially occurs, after which the response decays with a time constant T toward a steady state level. When there is no acoustic wave input, T=T ₀ . This is 50
It is about milliseconds. If the volume is L _nax , T=
T _nax . This is about 30 milliseconds. Ao=
By setting it to 1, 1/(So+Sh) becomes
If L=0, it is determined to be 5 centiseconds. L is
When L _nax and L _nax = 20 sones, the following equation holds true.

So+Sh+D(20)=1/30 (5) Based on the above data and formula, So and Sh are determined by formulas (6) and (7) shown below.

So=DL _nax / [R+(DL _nax T ₀ R)-1] (6) Sh=1/T ₀ -S _p (7) However, R=f stable state |/f stable state | L _nax /L= 0 (8) f steady state represents the firing rate at a given volume when dn/dt is 0.

R is the only variable left in the audio processor. Therefore, the performance of this processor is changed simply by changing R. That is, R is one parameter that can be adjusted to change performance, usually meant to minimize steady-state effects versus transient-state effects. Inconsistencies in output patterns for similar speech inputs are generally caused by differences in frequency response, speaker differences, background noise, and other factors (affecting the steady-state portion of the speech signal but not the transient portion). ) It is desirable to minimize steady-state effects since they are caused by distortion. The value of R is preferably set to optimize the error rate of the complete speech recognition system. The optimal value thus found is R=1.5. In that case, the values of So and Sh are 0.0888 and 0.11111, respectively, and the value of D is 0.00666.

FIG. 13 is a flow diagram of the operation of the audio processor according to the present invention. Preferably, the digitized audio during a 25.6 millisecond time frame, sampled at 20 kHz, is passed through a Hanning window 1320 and its output is double Fourier transformed in a DFT 1322 at 10 millisecond intervals. The conversion output is filtered in block 1324 to provide a power density output in each of at least one frequency band (preferably all critical frequency bands, or at least 20 bands). The power density is then converted from the recorded loudness to a volume level at block 1326. This operation is the first
By changing the graph in Figure 1 or later
It is carried out on the basis of the limit values derived by the process outlined in the figure.

In FIG. 14, the spacing limit T _f and audibility limit T _h of each of the first filtered frequency bands m are set to be 120 dB and 0 dB, respectively (block 1340). The audio counter, total frame register, and histogram register are then reset (block 134).
2).

Each of the histograms includes bins, with each bin representing the number of samples or counts during which the power or similar measurement (for a given frequency band) is within a respective range. In the present invention, the schtogram preferably represents the number of centiseconds during which the volume is within each of a plurality of volume ranges (for each given frequency band). for example,
In the third frequency band, there may be 20 centiseconds between 10 dB and 20 dB power. Similarly, in the 20th frequency band, between 50dB and 60dB, the total
There may be 150 centiseconds out of 1000 centiseconds. Percentiles are taken from the total number of samples (ie, centiseconds) and the binned counts.

At block 1344, the frame of filter output for each frequency band is examined, and at block 1346, the bins in the appropriate histogram (one per filter) are incremented. block 1348
Then, the total number of bins with amplitudes greater than 55 dB is tallied for each filter (i.e., frequency band) to determine the number of filters that indicate the presence of speech. At block 1350, a minimum number (e.g. 20
If there is no filter in 6), block 1
At 344, the next frame is inspected. If there are enough filters to indicate the presence of audio, block 135
2, increment the voice counter. The audio counter is incremented at block 1354 until audio appears for 10 seconds and at block 1356 new T _f and T _h values are determined for each filter.

The new T _f and T _h values for a given filter are determined as follows. For T _f , the dB of the retained bin for the 35th sample from the top of 1000 bins
The value (i.e. the 96.5th percentile of volume) is
BIN _H is defined, and T _f is set to T _f =BIN _H +40dB. For T _h , from the lowest bin (0.01)
The dB value of the bin holding the (total number of bins - voice count)-th value is defined as BIN _L. That is,
BIN _L is a 1% bin of the number of samples in the histogram excluding those classified as audio. T _h is defined as T _h =BIN _L -30dB.

Blocks 1330 and 1332 in FIG.
Then, the amplitude of the sound is updated with the limit value as described above, and based on the updated limit value, the amplitude of the sound is converted into units of sones and compressed. An alternative method of introducing and compressing Sohn units is to take the filter amplitude "a" (after the bins have been incremented) and convert it to dB by:

a ^dB = 20 log ₁₀ (a) - 10 (9) Each of the filter amplitudes is then compressed to a range between 0 dB and 120 dB to give equivalent loudness by:

a ^eq1 = 120 (a ^dB − T _h ) / (T _f − T _h ) (10) Next, a ^eq1 is converted from the volume level (in units of phons) to an approximate value of the volume in units of sones (at 40 dB) using the following formula. It is desirable to map a 1kHz signal to 1).

L ^dB = (a ^eq1 −30)/4 (11) Next, the approximate value L _s of the sound volume per son is given by the following equation.

L _s =10(L ^dB )/20 (12) At step 1334, L _s is used as input to equations (1) and (2) to determine the output firing rate f for each frequency band. For 22 frequency bands, a 22-dimensional vector characterizes the acoustic wave input over consecutive time frames. However, in general, 20
The frequency bands are examined using a conventional filter bank scaled by Mel.

Block 1 before processing the next time frame.
At 337, the "next state" of n is determined according to equation (3).

The acoustic processors described above require improvement for use with DC pedestals where the firing rate f and neurotransmitter content n are large. That is, if the dynamic range of terms in the f and n equations is important, the following equation is derived to lower the height of the pedestal.

In a stable state and when there is no acoustic wave input signal (L=0), equation (2) can be solved for the stable internal state n' as follows.

n'=A/(So+Sh) (13) The internal state of the neurotransmitter quantity n(t) is expressed as a stable state part and a fluctuating part as follows.

n(t)=n′+n″(t) (14) Combining equations (1) and (14), we obtain the firing rate as follows.

f(t) = (So+D・L)(n′+n″(t)) (15) The So・n′ term is a constant, but all other terms are the varying parts of n or (D・Contains an input signal represented by It will be done.

f″(t)=(So+D・L)・[{n″(t)+D・L・A}/(So
+Sh)] (16) Considering equation (3), the “next state” is as follows.

n(t+△t)=n′(t+△t)+n″(t+△t) (17) n(t+△t)=n″(t)+A−(So+Sh+D・L)・(n′+n″) (t)) (18) n(t+△t)=n″(t)-(Sh・n″(t) −(So+Ao・L ^A )・n″(t)-(Ao・L ^A・D) /(So+Sh)+Ao−(So・Ao) +(Sh・Ao))/(So+Sh) (19) Equation (19) becomes as follows if all constant terms are ignored.

n″(t+△t)=n″(t)(1−So・△t)−f″(
t)
(20) Equations (15) and (20) constitute the output and state update equations applied to each filter during each 10 ms time frame. The result of using these formulas is a vector of 20 elements every 10 ms,
Each element of this vector corresponds to the firing rate of a respective frequency band in the mel-scaled filter bank.

Regarding the above embodiment, the flowchart of FIG. 13 uses equations (11) and (16) that define the special case equations for the firing rate f and the "next state" n(t+Δt), respectively.
, except that the expressions f, dn/dt and n(t+Δt) are replaced.

A unique value for each equation term (i.e., t ₀
(5csec, t _L = 3csec, Ao = 1, R = 1.5 and L _nax
= 20) can be set to other values, So, Sh
and D terms have their respective desired values of 0.0888, 0.11111, and
and will be a different value from 0.00666.

The invention can be implemented with a variety of software or hardware.

F3 Precision matching (Fig. 4, Fig. 15, Fig. 16) Fig. 4 shows the phoneme machine 200 of the phonetic symbol type as an example.
Indicates 0. Each phonetic type matching machine is a stochastically limited state machine; (a) multiple states Si; (b) multiple transitions tr (Sj→Si): a transition is between different states, Transitions transition between the same states, and each transition has a corresponding probability. (c) It is characterized by having an actual label probability corresponding to each label that can be generated at a specific transition.

In FIG. 4, seven states S ₁ to _{S 7} and 13 transitions tr1 to tr13 are provided in the precision matching phoneme machine 2000, and three transitions tr11, tr12 and
The path of tr13 is shown with a dashed line. In each of these three transitions, a phoneme may change from one state to another without generating a label. Therefore, such a transition is called a null transition. transition
Labels can be generated along tr1 to tr10. In particular, at least one label along each of transitions tr1-tr10 may have a unique probability of being generated there. For each transition, there is a probability associated with each label that can be generated by the system. That is, if the labels that can be selectively generated from the acoustic channels are
200, each transition (not null) has an associated "actual label probability" of 200, each of which corresponds to the probability that the corresponding label is produced by a phoneme at a particular transition. The actual label probability of transition tr1 is represented by the symbol P followed by a column from 1 to 200 in brackets, as shown.
Each of these numbers represents a given label. In the case of label 1, there is a probability P[1] that the precision matching phoneme machine 2000 generates label 1 at transition tr1. Various actual label probabilities are stored in association with labels and corresponding transitions.

When a string of labels y ₁ y ₂ y ₃ . . . is presented to the precision matching phoneme machine 2000 corresponding to a given phoneme, a matching procedure is performed. The procedure related to the precise matching phoneme machine will be explained with reference to FIG.

FIG. 15 is a trellis diagram of the phoneme machine of FIG. 4. As in the case of the phoneme machine, this trellis diagram also has a null transition from state S ₁ to state S ₇ , state S ₁
shows the transition from to state S ₂ and from state S ₁ to state S ₄ . Transitions between other states are also shown. Additionally, the trellis diagram shows the measured times in the horizontal direction. The starting probabilities q ₀ , q ₁ , and q ₂ are determined by the phoneme at the time t=t ₀ , t=t ₁ or t=t ₂
represents the probability of having a start time in each of the . The respective transitions at each start time are also shown. By the way, consecutive starts (and ends)
It is desirable that the time interval has a length equal to the label time interval.

When using the precision matching phoneme machine 2000 to determine how closely a given phoneme matches the label of an incoming string, the end time distribution of that phoneme is searched to determine the match value for that phoneme. used for. In generating end time distributions to perform precise matching, precision matching phoneme machine 2000 requires accurate and complex calculations.

First, according to the trellis diagram of FIG. 15, time t
Examine the calculations required to obtain the start and end times at = t ₀ . In the case of the example phoneme machine structure shown in FIG. 4, the following probability equation applies.

Pr(S ₇ , t=t ₀ )=q ₀・T(1→7)+ Pr(S ₂ ,t=t ₀ )・T(2→7)+ Pr(S ₃ ,t=t ₀ )・T(3→7) (21) where Pr represents the probability, and T represents the transition probability between the two states in parentheses. This formula is t=t ₀
We show that the probabilities of each of the three states that can result in an end time in S 7 are limited to end time occurrences in state S ₇ in this example.

Next, when we look at the end time t=t ₁ , we have to perform calculations for every state other than state S ₁ . State S ₁ starts at the end time of the previous phoneme.
For convenience of explanation, only the calculations for state S ₄ are shown.

For S ₄ , the calculation becomes:

Pr(S ₄ , t=t ₁ )=Pr(S ₁ ,t=t ₀ )・T(1→4) ・Pr(y ₁ ,1→4)+Pr(S ₄ ,t=t ₀ )・T (4→4) ・Pr(y ₁ , 4→4) (22) Equation (22) shows that at time t=t ₁ , the phoneme machine is in state S _{4
The probability that _} The value obtained by multiplying the probability that a given label y ₁ transitions from state S ₁ to state S ₄ and (b) the probability of being in state S ₄ at time t = t ₀ times the transition from state S ₄ to itself. Multiply the transition probability and further take the given label y ₁ as transitioning from state S ₄ to itself
This shows that it is determined by the sum of the value obtained by multiplying the probability of generating .

Similarly, calculations for other states (except state S ₁ ) are also performed to generate the corresponding probabilities that the phoneme is in the particular state at time t=t ₁ . In general, in determining the probability of being in a target state at a given time, precise matching involves: (a) recognizing the respective probability of each state and each previous state before the transition that leads to the target state; (b) for each said previous state, recognize a value representing the probability of a label that must be generated on a transition between each said previous state and the current state to match that label string; c) Combine the respective values representing the probability of each previous state and the label probability to give the probability of the target state due to the corresponding transition.

The overall probability of being a target state is determined from the target state probabilities due to all transitions leading to it.
The calculation for state S ₇ includes terms for three null transitions, allowing the phoneme to start and end at time t=t ₁ with the phoneme ending in state S ₇ .

As in the case of determining the probabilities for times t=t ₀ and t=t ₁ , the determination of probabilities for other sets of end times is preferably performed to form an end time distribution. The value of the end time distribution for a given phoneme indicates how well the given phoneme is matched to the incoming label.

In determining how well a word matches an incoming label, the phonemes representing the word are processed sequentially. Each phoneme generates an end time distribution of probability values. The phoneme matching value is obtained by summing the end time probabilities and taking the logarithm of the sum.
The start time distribution of the next phoneme is derived by normalizing the end time distribution. This normalization involves, for example, scaling each of the values by dividing them by their sum, such that the scaled values sum to one.

There are at least two ways to determine the number h of phonemes to test for a given word or word string. In the depth-first method, calculations follow a basic format (calculating subtotals for each successive phoneme in succession). If this subtotal is found to be less than or equal to a predetermined limit for a given phoneme position along it,
The calculation ends. Another method, the breadth-first method, involves calculating similar phoneme positions in each word. The calculations are performed sequentially, starting with the first phoneme of each word, followed by the second phoneme of each word, and so on. In the breadth-first method, calculations along the same number of phonemes in each word are compared at the same relative phoneme position. In either method, the word with the largest sum of matching values is the desired target word.

Precise matching is performed using APAL (Array Processor
It is realized in assembly language). This is an assembler 190L manufactured by Floating Point Systems, Inc. By the way, precise matching is based on the actual label probability (i.e. the probability that a given phoneme produces a given label and y with a given transition), the transition probability for each phoneme machine, and the probability that a given phoneme produces a given label and y requires considerable memory to store each of the probabilities of being in a given state at a given time after the start time of . The 190L mentioned above has a match value based on the end time, preferably the log sum of the end time probabilities, a start time based on the previously generated end time probabilities, and a word match score based on the match values of sequential phonemes in the word. is set up to perform each calculation. Furthermore, for precise matching, it is desirable to calculate the tail probabilities of the matching procedure. Tail probability measures the likelihood of consecutive labels independent of words. In a simple example, a given tail probability corresponds to the likelihood of a label following another label. This likelihood is easily determined, for example, from a string of labels generated by some sample speech.

Therefore, precise matching is of the basic form, Markov
Provide sufficient storage to contain model statistics and tail probabilities. For a vocabulary of 5000 words, each word containing about 10 phonemes, the basic format is 5000×10
requires an amount of memory. (Markov for each phoneme)
If there are 70 distinct phonemes (with a model), 200 distinct labels, and 10 transitions with a probability of generating any label, then the statistic is 70 × 10 ×
This would require 200 storage locations.
However, it is preferable that the phoneme machine is divided into three parts (beginning part, middle part and end part) and the statistical table corresponds thereto (preferably one of the three self-loops is included in each part). Therefore, the storage requirement is reduced to 60x2x200. For tail probabilities, 200x200 storage locations are required. 50K integer and 82K floating point storage will work satisfactorily with this array.

The above description relates to a basic phonetic alphabet format including a sequence of phonetic phoneme machines as shown in FIG.

However, the Feneme basic form may also be used in a precision match similar to the precision match outlined above. FIG. 16 shows a grid based on the Hueneme phoneme machine (an example of which is shown in FIG. 19). This figure shows that at any given time, any of three transitions can occur. A null transition (represented by a dashed line) goes from one state to another without generating a label. The second transition allows the generation of labels during a self-loop from a state to itself. The third transition allows for the generation of labels during the transition from one state to another.

As previously suggested, fast matching is optional (although shown in Figure 2). The following discussion relates to an environment involving fast matching, which reduces the number of words examined with precise matching. However, if desired, the fast match can be omitted, in which case each word is processed by a precise match.

F4 Phoneme tree structure (Figure 17) Once the phoneme matching value is determined, the first
As shown in FIG. 7, comparisons are made along the branches of the tree structure 4100 to determine which path of phonemes is most likely to occur. In FIG. 17, the sum of the phoneme match values for phonemes DH and UH1 of the spoken word "the" (from point 4102 to branch 4104) is
It must be a significantly higher value than for each sequence of phonemes branching from MX. Incidentally, the phoneme match value for the first phoneme MX is calculated only once and then used for each base form that extends. (See branches 4104 and 4106.) Additionally, the total score computed along the first sequence of branches is less than the critical value or less than the total score of other sequences of branches. If it is found to be lower and lower, all base forms extending from the initial sequence may be removed from the candidate words at the same time. For example, branch 4108
The basic form associated with ~4118 is discarded as soon as it is determined that MX is not a likely path. The fast matching implementation and tree structure generates an ordered list of candidate words, with significant savings in associated computation.

For storage requests, the phoneme tree structure, phoneme statistics, and tail probabilities are to be stored. For the tree structure, there are 25,000 arcs and four data words characterizing each arc. The first data word represents an index of a subsequent arc or phoneme.
The second data word represents the number of subsequent phonemes along the branch. The third data word represents at which node of the tree structure the arc is located. The data word in Figure 4 represents the current phoneme. Therefore, this tree structure requires a storage space of 25000×4.
In fast matching, there are 100 different phonemes and 200 different phonemes. Since Hueneme has a probability of 1 to be generated somewhere in the phoneme, it is 100×200
A storage space for statistical probabilities is required. For the tail structure, 200x200 storage space is required.
For fast matching, 100K integer and 60K floating point storage space is sufficient.

F5 Language Model (Figure 2) As mentioned above, the probability of correctly selecting a word can be increased by including a language model that remembers information about the word in context (such as triple letters). The language model is described in the above paper.

Preferably, language model 1010 has unique characters. Specifically, a modified trigraph method is used. In accordance with the present invention, sample text is examined to determine the likelihood of each ordered triple word and word pair and single word in the vocabulary. A list of most likely triple words and word pairs is then formed. Additionally, the likelihoods of triple words not in the list of triple words and word pairs not in the list of word pairs are determined, respectively.

According to the language model, if the target word follows two words, a determination is made as to whether the target word and the two preceding words are in the list of triple words. If in a list of triple words, the stored probability assigned to that triple word is specified. If the target word and two preceding words are not in the list of triple words, a determination is made as to whether the target word and its adjacent preceding words are in the list of word pairs. If it is in a list of word pairs, multiply the probability of that word pair by the probability that there is no triple word in the list of triple words, and assign the product to the target word. If said triple word and word pair containing the target word are not in the list of triple words and the list of word pairs, respectively, then the probability of the target word alone is added to the probability that said triple word is not in the list of triple words, and the word pair. Multiply by the probability that is not in the list of word pairs and assign the product to the target word.

F6 Shaping by Approximation (Figure 18) The flowchart in Figure 18 shows the shaping of the phoneme machine used in acoustic matching. At block 5002, a word vocabulary (typically on the order of 5000 words) is defined. At block 5004, each word is displayed as a sequence of phoneme machines. The phoneme machine is shown, for example, as a phonetic phoneme machine, but may alternatively include a sequence of phoneme phonemes. The display of words by the sequence of the phonetic type phoneme machine or the sequence of the Feneme type phoneme machine will be explained below. The phoneme machine sequence of a word is called the word basic form.

Block 5006 arranges the word elementary forms into the tree structure described above. Statistics for each phoneme machine in the basic form of each word can be found in IEEE Bulletin Volume 64 (1976)
F. Jelinek's paper “Continuous speech recognition using statistical methods” (pages 532-556)
“Continuous Speech Recognition by
Statistical Methods “Proceedings of the
IEEE, Vol. 64, 1976, pp. 532-556). (Block 500
8) Block 5009 stores the values used for precise matching. At block 5010, approximations corresponding to the fast matching procedure are used for each model. Approximation may involve replacing actual statistics with estimated statistics and/or limiting the number of labels tested in a match.

Approximate parameter values used in high-speed matching are set in block 5012. At this point, each phoneme machine of each word base form has been shaped by the desired approximation. Additionally, a precision matching phoneme machine is also formed. Acoustic matching can be performed with precision matching alone or in conjunction with fast matching. The phonemes of each word base form are examined along the paths of the tree structure.

F7 Extension of word path by words selected by acoustic matching (Figures 5-7, 19)
(Fig.) Next, a good stack decoding method used in the speech recognition shown in Fig. 2 will be explained.

5 and 6, a plurality of consecutive labels y _{1 .} . . generated at consecutive “label intervals” or “label positions” are shown.

In addition, FIG. 6 shows a plurality of generated words.
The paths are shown: path A, path B and path C. In the context of Figure 5, path A leads to entry "to be or" and path B leads to entry "two b".
, path C would correspond to the entry "too". For a target word path, there is a label (or equivalently, a label interval) that the target word path has the highest probability of being completed. Such a label is called a "boundary label."

For a word path W representing a sequence of words, the most likely ending time (shown in the label string as a "boundary label" between two words) is given by IBM Technical Disclosure Bulletin, Vol. 23, No. 4. September 1980 issue, LRBahlet al. paper “Sonic Acoustic Matching Calculation” (LRBahlet al,
“Faster Acoustic Match Computation”, IBM
Technical Disclosure Bulletin, Vol.23, No.4,
(September 1980). Briefly, this paper focuses on two important questions: (a) how many label strings Y are words (or word sequences); and (b) at what label spacing. It describes how to deal with the termination of partial sentences (corresponding to parts of label strings).

For any given word path, there is a "likelihood value" associated with each label or label interval from the first label of the label string to the boundary word. All of the likelihood values for a given label path collectively represent the "likelihood vector" for a given word path. Therefore, for each word pass there is a corresponding likelihood vector. The likelihood value L _t is shown in FIG.

The "likelihood envelope" Λ _t of a collection of word paths W ¹ , W ² , . . . , W ^s at label interval t is mathematically defined as follows.

Λ _t = max(L _t (W ¹ ), ..., L _t (W ^s )) That is, for each label interval, the likelihood envelope is the highest likelihood associated with any word path in the collection. Contains degree values. A likelihood envelope 8040 is shown in FIG.

A word pass is considered "complete" if it corresponds to a complete sentence. Preferably, the complete path is identified when the typing speaker reaches the end of the sentence, for example by pressing a button.
The input is synchronized with a label interval that marks the end of a sentence. A complete word pass cannot be extended by appending words to it. Partial word paths accommodate incomplete sentences and can be extended.

Partial paths are classified as "alive" or "dead" paths. A Word Pass is “dead” when it has already been extended, but
It is “alive” when it has not yet been extended. By this classification, a path that has already been extended to form at least one longer extended word path will not be considered for extension again at the next time.

Each word path can be characterized as "good" or "bad" with respect to the likelihood envelope. A word path is a good word path if it has a likelihood value that is within the maximum likelihood envelope with the label corresponding to its boundary label. Otherwise, word
Pass is a bad word pass. Although it is desirable, it is not necessary to reduce each value of the maximum likelihood envelope by a fixed value to act as a good (bad) limit level.

There is a stack element for each label interval. Each live word path is assigned to a stack element corresponding to the label interval that corresponds to the boundary label of such live path. A stack element may have 0.1 or more word path entries (listed in order of likelihood value).

Next, the steps performed by stack decoder 1002 of FIG. 2 will be described.

Forming the likelihood envelope and determining which word passes are good are interrelated as shown in the stack decoding technique flow diagram of FIG.

In the flowchart of FIG. 7, block 8050
First, the null pulse enters the first stack (0). At block 8052, the (complete) stack element containing the previously determined complete path, if any, is provided. Each complete path in a (complete) stack element has a likelihood vector associated with it. The likelihood vector of the complete path with the highest likelihood at its boundary label first determines the maximum likelihood envelope. If there is no complete path to a (complete) stack element, the maximum likelihood envelope is initialized to -∞ at each label interval. Furthermore, the maximum likelihood envelope may be initialized to - even if a complete path is not specified. Envelope equation setting occurs in blocks 8054 and 8056.

After the maximum likelihood envelope is initialized, it is reduced by a predetermined amount to form a △-good region in the information of the reduced likelihood, and a △-bad region below the reduced likelihood. . The larger Δ, the larger the number of word passes that are considered to be extendable. When using log ₁₀ to determine L _t , a value of 2 gives satisfactory results. It is desirable, but not necessary, that the value of Δ be uniform along the length of the label interval.

A word path is marked "good" if it has a likelihood of a boundary label falling within the Δ-good region. Otherwise, the word pass is marked as "bad".

As shown in Figure 7, the loop that updates the likelihood envelope and marks a word path as a "good" (extendable) or "bad" path is the longest unmarked word path. The search begins at block 8058. If more than one unmarked word path is in the stack corresponding to the longest word path length, then the word path with the highest likelihood for its boundary label is selected. If the password is found, block 8060
Then, check whether the likelihood at that boundary label is within the △-good region. If it is not in the good region, block 8062 marks the path as .DELTA.--in the bad region, and block 8058 searches for the next unmarked live path. If it is in the good region, block 8064 marks the path as △- in the good region, and block 8066 updates the likelihood envelope to include the likelihood value of the path marked "good". do. That is, for each label interval, the updated likelihood value is (a) the current likelihood value within its likelihood envelope, and (b) the likelihood associated with the word path marked “good”. determined as the larger likelihood value between the values. This operation occurs in blocks 8064 and 8066. After the envelope has been updated, we return to block 8058 and again look for the longest, best unmarked living word path.

This loop is repeated until there are no more unmarked word paths. When there are no more unmarked word passes, block 8070
, the shortest word path marked "good" is selected. If there are two or more "good" word paths with the shortest length, block 80
At 72, the word path with the highest likelihood for its boundary label is selected and the selected shortest path is extended. That is, at least one likely successor word is determined by successfully performing a fast match, language model, fine match, and language model procedure, as described above. For each potential successor word, an extended word path is formed. Specifically, an extended word path is formed by appending a likely successor word to the end of the selected shortest word path.

After the selected shortest word path forms an extended word path, the selected word path
The path is removed from the stack in which it was an entry, and each extended word path is inserted into the appropriate stack in its place. especially,
The extended word path becomes an entry on the stack corresponding to its boundary label (block 80).
72).

The operation of extending selected pulses in block 8072 will be described in conjunction with the flowchart of FIG.

At block 6000 of FIG. 19, (of FIG. 2)
Audio processor 1004 generates a string of labels. Label string is block 600
2, and at block 6002,
One of the basic or improved approximate matching procedures is performed to obtain an ordered list of candidate words. Thereafter, block 6004 uses the language model as described above. After using the language model, the remaining target words along with the generated labels are sent to the fine match processor at block 6006. At block 6008, the fine match yields a list of remaining candidate words that are better presented to the language model. Probable words (as determined by rough matches, fine matches, and the language model) are used to extend the path found in block 8070 of FIG. Each of the potential words determined in block 6008 (Figure 19) can be appended separately to the found word path to form multiple extended word paths (block 6010). .

In FIG. 7, after the extension path is formed and the stack is re-formed, the process returns to block 8052 to repeat the process.

Therefore, at each iteration, the shortest, best "good" word path is selected and extended. A word path marked as a "bad" path in one iteration may become a "good" path in a later iteration. Then,
The characteristics of whether a living word path is a "good" or "bad" path are assigned independently at each iteration. In practice, the likelihood envelope does not change significantly from one iteration to the next, so the word
The calculations that determine whether a pass is good or bad are done efficiently. Furthermore, normalization becomes unnecessary.

When identifying complete sentences, it is desirable to include block 8074. That is, if there are no living word paths left unmarked and no "good" word paths to extend, decoding terminates. The complete word path with the highest likelihood for each of its boundary labels is identified as the most likely word sequence of the input label string.

In the case of continuous speech where sentence ends are not identified, path extension is performed continuously or up to a predetermined number of words as desired by the user of the system.

Markov model constructed from multiple utterances of F8 words (Figures 1A, 1B, 1C,
Figures 20 to 26) A flowchart consisting of Figures 1A and 1B (Figures 1A and 26)
The arrangement relationship in Figure 1B is shown in Figure 1C).
Outlines the steps to construct a basic basic form. A "base form" is a sequence of phoneme machines representing word segments (preferably words) found in the speech recognition system's vocabulary. A word segment is preferably a dictionary word, but may also represent a predetermined portion of a dictionary word, such as a syllable of the dictionary word.

The first step (block 9000) in the embodiment of the invention of FIGS. 1A and 1B is to convert the utterance of a word segment into a string of phrases (or labels). As previously mentioned, an audio processor typically generates a string of words in response to the utterance of a word segment. For each utterance there is a corresponding Hueneme string.

Figure 20 shows the Feneme string FS ₁ of N.
Indicates FS _N. Each of these feneme strings is generated in response to the utterance of a corresponding given word segment. Each block represents a flame in the string. These coins are identified in each string as coins 1-1i.

In accordance with the present invention, a set of phoneme machines (ie, Markov models) is created. Each phoneme machine has at least two states; each transition going from one state to another; a probability associated with each transition; and multiple output probabilities for at least some transitions (each output probability is a given corresponding to the likelihood of generating a Feneme at a particular transition). FIG. 21 shows a simple sample Feneme phoneme machine 9002.

Phoneme machine 9002 has states S ₁ and S ₂ . One transition t ₁ leaves state S ₁ and returns to itself, with probability P _t1 (S ₁ |S ₁ ). For transition t ₁ ,
There is a respective probability associated with producing each of the fenemes f ₁ to f _n at transition t ₁ . Similarly, a transition t ₂ between states S ₁ and S ₂ is defined by (a) its associated probability P _t2 (S ₂ | S ₁ ), (b) its value of producing each of the fenemes f ₁ to f _n Has probability. The null transition t ₃ represents a transition that does not produce an output, i.e. a fueneme, and its associated P _t3
(S ₂ | S ₁ ). The phoneme machine 9002 thereby allows the generation of any number of sound effects (such as when transition t ₁ repeats), but no sound sound is generated when transition t ₃ continues.

Each phoneme machine has different probabilities or statistics associated with it. Although it is desirable, it is not necessary for the same set of phoneme machines to have the same configuration and differ only in statistics. Statistics are generally determined during the shaping session.

When a set of phoneme machines is formed and applied to all of the phoneme strings produced by the utterance of a given word segment, a decision is made as to which phoneme machine gives the best base form of length 1. (block 9004).
The best basic form of phoneme length 1 (P ₁ ) can be determined by examining each phoneme machine in the set and by
It is found by determining the probability of generating each of the Feneme strings FS ₁ to FS _N. The N probabilities extracted for each particular phoneme machine yield a joint probability assigned to that particular phoneme machine by taking their product. The phoneme machine that yields the highest joint probability is selected as the best basic form P ₁ of length 1.

Find the best basic form of length 2 having the form P ₁ P ₂ or P ₂ P ₁ while preserving the phoneme P ₁ . That is, append each phoneme to the trailing edge of P ₁ to form a respective ordered phoneme pair, and append each phoneme to the trailing edge of P ₁
to the leading edge of each to form each ordered phoneme pair. The joint probability of each ordered phoneme pair is then obtained. The ordered phoneme pair that yields the highest joint probability of producing a phoneme string is considered the best base form of length 2 (block 9006).

The best elementary forms of length 2, i.e., the ordered pairs with highest joint probabilities, are then aligned as in the well-known Viterbi alignment (block 900
8). Briefly, the alignment represents which phoneme in each string corresponds to each phoneme of the ordered phoneme pair. (At this point, the phoneme has been represented by the phoneme machine. Therefore, the phoneme and phoneme machine terms are corresponding entities.) Following alignment, the feneme string FS ₁ ~ _{FS N}
Find a match in each. For each of the Hueneme strings FS ₁ to FS _N , the matching points are (length 2
is defined as the point where the phonemes P ₁ and P ₂ (of the best basic form of ) have the greatest probability of touching. Another way to look at it is that the coincidence point is the Hueneme string FS ₁ ~
Each FS _N can also be considered as a point that divides into a left part and a right part. In this case, the left part of all the Feneme strings represents a common set of tones, and the right part of all the Feneme strings likewise represents a common set of tones (block 901).
0). Each left portion is considered a left substring and each right portion is considered a right substring (block 9012).

The left substring and right substring are then treated similarly, but separately, by a separate processing scheme.

For the left substring, find instead the best single phoneme base form P _L with the highest joint probability. (Bronck 9014). While preserving the phoneme P _L , pairs are formed in the order in which each phoneme in the set is added before it, and pairs are formed in the order in which each phoneme in the set is added after it. Then, find the fixed-order pair P _L P _A or P _A P _L that has the highest joint probability of producing a wager in the left substring (block 9016). As mentioned above, this is considered the best basic form of length 2 for the left substring.

The best joint probability of length 2 for the left substring is compared to the joint probability P _L alone (block 9018 in Figure 1B). If the joint probability P _L is larger, the phoneme P _L is placed in the connected basic form (block 9020). If the joint probability P _L is smaller, then P _L
P _A or P _A P _L is aligned against the left substring. (Block 9022). A match is found in each of the left substrings, and each left substring is then split into a (new) left part and a (new) right part (block 9024).

The same procedure is applied to each right substring of the initially split Feneme strings FS ₁ to FS _N. The best single basic form P _R (from block 9030) and the best basic form P _R P _B or P _B P _R of length 2 found in block 9034 are
A comparison is made in block 9032. If the joint probability of P _R is larger, then the phoneme is added to the connected basic form.
Place _PR (block 9020). If P _R is smaller, the basic forms of the two phonemes are aligned and each right substring is divided into a left part and a right part at the matching point (block 9036).

The splitting cycle repeats for each substring for which the length-2 best base form has a higher joint probability than the best one-phoneme base form. That is, the operation of splitting a substring into two parts and splitting one or both of them (after alignment) into a new substring can be performed one after the other until only one basic phoneme form remains.

The basic form of one phoneme is concatenated in the same order as the substrings it represents. The concatenated basic form is the Hueneme string FS ₁ ~
FS represents one consecutive phoneme corresponding to consecutive substrings of _N. As will be explained later, a substring may contain zero, one, or more than one sound, which causes the pronunciation to change from utterance to utterance.

The concatenated base forms described above represent the basic base forms of word segments, eg lexical words. A modification of the connected basic format is incorporated into the flowchart of FIG. Figure 22 is Figure 1B,
Continuing from block 9020, one phoneme (P _L and P _R ) is placed to form a concatenated base form. This refinement aligns the concatenated primitives to the eneme string (block 9050). For each Feneme string FS ₁ to FS _N , this alignment represents that the Feneme in that string corresponds to the respective phoneme machine (if any), and is useful for splitting the string based on phoneme correspondence ( Block 9052).

Analyze each section to be divided and determine the best single phoneme for that division (block 9)
054). Due to the alignment, the best single phoneme of the feneme in the divided section may be different from the single phoneme in the previously aligned and concatenated base form.

If they are different (block 9058), replace each best single phoneme with the corresponding single phoneme in the previously aligned and concatenated base form to generate a new concatenated base form (block 9056). .
The new base form is then aligned (block 9050), segmented (9052), search for new best phonemes (block 9054), and
Properly replace phonemes in connected basic forms. As shown in the flowchart of FIG. 22, this cycle can be repeated to obtain a sequentially processed base format.

If the old best phoneme in the concatenated base form is the same as the new best phoneme of the given division (block 9058), then this phoneme is fixed in place in the concatenated base form (block 9060). Once all phonemes are fixed in their ordinal positions, an improved basic form results (block 90
62).

The basic format of Feneme will be explained with reference to FIGS. 23 to 26. P ₁ is Hueneme String FS ₁
~FS is initially found to be the best basic form of length 1 of _N.

Using P ₁ as one phoneme, the second phoneme is determined to form the best ordered phoneme pair of the feneme strings FS ₁ to FS _N . This is shown in FIG. In FIG. 24, each of the feneme strings FS ₁ to _{FS N} is divided at the point where the probability that the phoneme P ₁ touches the phoneme P ₂ is greatest. In FIG. 25, the left and right portions are determined. These sections are then inspected separately, such as the multiple strings of fines in FIG. With individual processing, the phonemes in each string are represented by successively more phonemes. If the probability of a given phoneme is greater than the probability of the two retrieved phonemes, segmentation is stopped and the given phoneme is placed at its respective position along the sequence of such unsplit phonemes.

Figure 26 is a sample of one phoneme P ₁ arranged, and the corresponding Feneme string FS ₁
~FS represents a substring of _N. In FS ₁ , the phoneme
P ₁ is associated with one Feneme, in FS ₂ P ₁ is associated with a null, and in FS ₃ P ₁ is associated with the generation of two Fenemes. The same applies below.

To refine the base form, we perform a Viterbi alignment of the Hueneme string of each utterance to the concatenated base form. In the connected basic format, for each phoneme, a phoneme aligned with it is determined in sequence. If no phoneme is aligned to a phoneme, that phoneme is deleted. In other cases, find the phoneme that maximizes the probability of producing a Feneme aligned with it (i.e. in the sections that have been divided for it), and replace the pre-existing phoneme with the maximum probability If the probability that it is a phoneme is smaller, replace it. This step may or may not be repeated, as desired. If repeated,
The repetition ends when the phoneme is replaced.

Although the present invention is implemented in PL/I on the IBM System 3084 MVS, it can also be implemented in any of a variety of languages on any of a variety of computing systems.

The best basic form is characterized by the previous example as the basic form with the highest joint probability, if the joint probability is the product of the probabilities associated with each Hueneme string. The best basic form and highest joint probability may be determined differently according to the invention. Incidentally, the highest average probability, or some predetermined distribution, can be used in determining the highest joint probability.

Furthermore, the invention may be practiced by dividing into three or more parts at the same time. For example, the payneme string for each utterance may be initially divided into three parts (left, middle and right sections). Each divided section is then examined separately in an individual processing manner. However, a section divided into two is preferable to a section divided into three or more.

Furthermore, there are no fixed restrictions on the order of division and arrangement. In one embodiment, the splitting and alignment is performed until the splitting stops, determining successively smaller left portions. The leftmost phoneme in the concatenated base form is thereby determined first. Thereafter, the second phoneme from the left is determined using the concatenated basic form. As an alternative, the present invention also contemplates other segmentation and alignment routines to select the desired phonemes in concatenated elementary form.

G. Effects of the Invention According to the present invention, the basic format constructed in a speech recognition system can be improved.

[Brief explanation of drawings]

FIG. 1 is a diagram showing the arrangement relationship between FIGS. 1A and 1B, and FIGS. 1A and 1B show a method for constructing a basic form of a word segment based on multiple utterances according to the present invention. 2 is a schematic block diagram of a system environment in which the present invention may be implemented; FIG. 3 is a block diagram detailing the stack decoder within the system environment of FIG. 2; and FIG. FIG. 5 is a diagram showing the steps of continuous stack decoding. FIG. 6 is a diagram showing the stack decoding method. Figure 7 is a flowchart of the stack decoding method, Figure 8 is a diagram showing the elements of the acoustic processor, Figure 9 is a diagram showing the parts of a typical human ear representing the locations forming the components of the acoustic model, and Figure 10. The figure is a block diagram showing the part of the sound processor, Figure 11 is a diagram showing the relationship between sound intensity and frequency used in the design of the sound processor, Figure 12 is a diagram showing the relationship between the horn and the horn, and Figure 13 is a diagram showing the relationship between sound intensity and frequency. a flowchart showing how acoustic features are represented by the acoustic processor of FIG. 8;
Figure 14 is a flowchart showing how the limits are updated in Figure 13, Figure 15 is a diagram showing the trellis or lattice of the precision matching procedure, and Figure 16 shows the phoneme machine used to perform the matching. Figure 17 is a diagram showing the phoneme tree structure that allows processing of multiple words at the same time, Figure 18 is a flowchart showing the shaping of the Markov model phoneme machine, and Figure 19 is a diagram showing the extension of the word path. The flowchart shown in Figure 20 is 1
Figure 21 is a diagram showing a sample Hueneme phoneme machine, and Figure 22 is a diagram showing a Sueneme sequence obtained from N's utterances of two word segments. Flowchart appended to the flowchart of FIG. 1B; FIG. 23 is a diagram illustrating the best basic form of phoneme length 2 to use for each feneme string produced in response to one of a plurality of utterances; FIG. Figure 25 shows each hueneme string split at points determined where phoneme P ₁ consistently touches phoneme P ₂ ; Figure 25 shows the splits identified as left and right parts. FIG. 26 is a diagram showing each phoneme and corresponding portion of the Feneme strings FS ₁ to FS _N. 1000...Voice recognition system, 1002...
Stack decoder, 1004...Acoustic processor, 1006, 1008...Array processor, 1010...Language model, 1012...Workstation, 1020...Search device, 102
2,1024,1026,1028...interface.

Claims (1)

[Claims] 1. A word Markov model formed by concatenating word partial Markov models corresponding to a set of labels each representing an acoustic type that can be assigned to a minute time interval, and a label of unknown input speech. The method for generating the word Markov model used in speech recognition to find the likelihood that the unknown input speech is generated in relation to the word Markov model by comparing Characterized Ward Markov model generation method. (a) Generating multiple label strings from multiple utterances of one word to be generated. (b) dividing the plurality of label strings into repeating label string parts; This step (b) consists of: (i) determining a pair of word partial Markov models that generate the plurality of label strings or label string parts in the target range to be divided with maximum probability; a substep of dividing the target range into left and right target ranges based on matching with the models; Substep (i) above until generating the plurality of label string portions in the previous target range.
It has a substep of repeating. (c) After the final division in step (b), the word part Markov model that generates the label included in the divided part with the highest probability is assigned to each divided part of the plurality of label strings. Generate a chain of Markov models,
Step of making this the word Markov model of the word to be generated.