JP5982297B2 - Speech recognition device, acoustic model learning device, method and program thereof - Google Patents

Description

  The present invention relates to a speech recognition technique using an acoustic model based on a neural network and a technique for learning the acoustic model.

  In the following explanation, the symbols “^”, “~”, etc. used in the text should be written immediately above the character immediately after it, but are written immediately before the character due to restrictions on the text notation. . In the formula, these symbols are written in their original positions. Further, the processing performed for each element of a vector or matrix is applied to all elements of the vector or matrix unless otherwise specified.

<Voice recognition device 90>
A processing flow of the speech recognition apparatus 90 is shown in FIG. 1, and a functional block diagram is shown in FIG. The speech recognition apparatus 90 mainly includes a feature amount extraction unit 91 and a word string search unit 92. The feature quantity extraction unit 91 converts voice signal data (recognition voice data) into a time-series feature quantity vector x _t for each frame (waveform obtained by cutting out a predetermined time length from the voice waveform) t (s91). The word string search unit 92 uses the acoustic model stored in the acoustic model storage unit 93 and the language model stored in the language model storage unit 94 to output a time-series feature amount vector output from the feature amount extraction unit 91. calculating a score for the score line and a language model for the acoustic model (voice feature vector) x _t. Further, the word string search unit 92 searches for a word string matching the feature vector x _t time series with reference to these scores (s92). The speech recognition apparatus 90 outputs a word string that is a search result finally obtained by the word string search unit 92 as a recognition result. Here, the acoustic model and the language model are created in advance using learning data or the like. Here, a method for creating an acoustic model will be described.

[Acoustic model]
The acoustic model is a model of acoustic features of speech, and the speech data is converted into symbols such as phonemes and words by referring to the recognition speech data and the acoustic model. For this reason, the creation of the acoustic model greatly affects the performance of the speech recognition apparatus. The speech recognition apparatus 90 uses the feature quantity extraction unit 91 to convert the speech data into {x ₁ , x ₂ ,..., X _t , ...} (x _t ∈R ^D , R is a set of real numbers, and t is a frame number or its converted into D feature quantity vector x _t series of dimensions, such as time) corresponding to the frame number. Usually, in the acoustic model for speech recognition, the relationship between each phoneme and the sequence of the feature vector xt is _expressed by a Left-to-right type hidden Markov model (hereinafter also referred to as “HMM”).

The feature vector x _t of the time series of these models, a sequence of state variables _{{s 1, s 2, ...} , s t, ...} is shifted in accordance with an order Markov chain, depending on its state variables s _t probability Modeled as sampled (output) from the distribution. Therefore, the information actually recorded in the memory as an acoustic model can be divided into two types: a state transition probability matrix P and an output distribution function parameter Λ. Here, it is assumed that the state transition probability matrix P is a known matrix, and a configuration in the case of learning the output distribution function parameter Λ will be described.

  The output distribution is generally expressed by a mixed Gaussian distribution or a neural network (hereinafter also referred to as “NN”), and Λ is a parameter thereof. An acoustic model expressing Λ with a mixed Gaussian distribution is called a mixed Gaussian distribution acoustic model, and an acoustic model expressing Λ with an NN is called an NN acoustic model.

[NN acoustic model]
NN acoustic model, when the state variable is s _t, the probability that a feature vector x _t is output using the NN parameter lambda, defined as follows.

Here, the denominator of p (s _t) denotes the probability of occurrence of state variables s _t, and the advance calculated by counting the frequency of occurrence of the state variable s in the training data. For example, the appearance probability p (s) is the ratio of the appearance frequency of each state variable s to the sum of the appearance frequencies of all the state variables in the learning data. In addition, when the amount of learning data is not sufficient, it may be assumed to be a constant value.

The p (s _t | x _t , Λ) of a molecule is defined as follows using a kind of NN called a multilayer perceptron (Non-patent Document 1).

i = 1,2, ..., L, L is the number of layers, H ⁽ⁱ⁾ is the number of units in the i-th layer, h ⁽ⁱ⁾ _j (x _t ; Λ) is the feature vector x _{t at the} input A real number indicating the state of the jth unit in the ith layer at a given time. For convenience, H ⁽⁰⁾ is D, and h ⁽⁰⁾ _j (x _t ; Λ) is h ⁽⁰⁾ _j (x _t ; Λ) = x _{t, j} , that is _{, j of} the feature vector x _t Let the element. In this NN acoustic model, hyperparameters determined in advance before learning are the number of layers L and the number of units H ⁽ⁱ⁾ in each layer. The remaining free variables, ie the coupling matrix

And bias vector

Is represented by the NN parameter Λ such that Λ = {W ⁽ⁱ⁾ , b ⁽ⁱ⁾ | ^∀ i}.

[About creating acoustic models]
The acoustic model is created by a probabilistic statistical method using a group of a series X ⁽ⁿ⁾ of a plurality of feature vectors x _t obtained from given learning data (hereinafter also referred to as “learning feature sequence group”) X = {X ⁽¹⁾ , X ⁽²⁾ ,…, X ⁽ⁿ⁾ ,…} and a group of sequences s ⁽ⁿ⁾ of a plurality of state variables s _t of learning data (hereinafter referred to as “learning state variable sequence group”) (Also called) S = {s ⁽¹⁾ , s ⁽²⁾ ,..., S ⁽ⁿ⁾ ,. Where n is an utterance index, and X ⁽ⁿ⁾ is a time series describing the acoustic features of one utterance (for example, a sentence),

In this way, it is expressed as a time series of a plurality of speech feature amount vectors. Similarly, s ⁽ⁿ⁾ is a label sequence having the same sequence length as X ⁽ⁿ⁾ ,

In this way, it is expressed as a time series of a plurality of state variables. The label sequence may be handled probabilistically, but here the label sequence is treated as known. However, the present invention itself can be applied as it is even if it is given probabilistically.

  Given these learning data, the optimal acoustic model parameter ^ Λ is defined as the acoustic model parameter Λ that maximizes the matching rate F (Λ, X, S) to the learning data, for example, as follows. The

  This optimization can be performed by the back propagation method (see Non-Patent Document 1).

[Minimum Error Linear Transformation (hereinafter also referred to as “MELT”)
In order to perform speech recognition with high accuracy, it is necessary to use an acoustic model learned using learning data recorded in the same environment (the surrounding environment such as noise and reverberation) when the same speaker as the recognition target is recognized. desirable. However, since it is difficult to create an acoustic model for each speaker and environment, it is common to perform speech recognition using an acoustic model learned from a speaker that is different from the recognition target or from learning data recorded in a different environment. Is. As a technology to improve speech recognition accuracy when using a speech recognition model learned from learning data recorded in a different speaker / different environment from the recognition target, the trained acoustic model and the recognition target speaker There are known adaptation techniques for correcting for the environment.

MELT (see Non-Patent Document 2) is known as an adaptation technique of a neural network acoustic model. In environment adaptation and / or speaker adaptation of the NN acoustic model using MELT, a connection weight matrix W ⁽ⁱ⁾ corresponding to a certain layer i is converted into a transformation matrix Γ and a pre-adaptation weight matrix ~ W ⁽ⁱ⁾ as follows: Use to update.

  This extension coincides with the conventional NN when the transformation matrix Γ is a unit matrix (Γ = I). That is, it can be considered that the parameters of the acoustic model targeted by MELT take into account Γ in addition to the conventional NN parameter Λ.

  In the learning step of MELT, the NN parameter Λ is estimated as follows using learning data X and S similar to those of the conventional NN.

At the time of recognition, the optimal transformation matrix ^ Γ is estimated by executing the following optimization using the adaptation data ~ X and ~ S collected in advance from the same recognition environment or the same speaker.

  This optimization can be performed using the steepest gradient method as in the learning of NN.

D. E. Ramelhart, G. E. Hinton, R. J. Williams, “Learning Representations by Back-Propagating Errors”, Nature, 1986, Vol. 323, pp. 533-536. J. Trmal, J. Zelinka, L. Muller, "ON Speaker Adaptive Training of Artificial Neural Networks", Proc. Interspeech, 2010.

  In order to realize speaker adaptation / environment adaptation by MELT, it is necessary to collect a sufficient amount of adaptation data ~ X, ~ S. As an attempt to reduce the amount of data for adaptation ~ X, ~ S, restrictions have been put on the possible values of Γ, but even in those attempts, a certain amount of data for adaptation ~ X, ~ S is accumulated. There is a need to.

  In an environment where speech recognition is actually used, there are many cases where the adaptation data ~ X and ~ S cannot be stored in advance, and only one utterance data that is going to be recognized is used for fast adaptation. The demand for methods is high. However, in a speech recognition apparatus based on the NN acoustic model, a technique for realizing such real-time adaptive processing is not known.

  An object of the present invention is to provide a speech recognition technique that uses a plurality of NN acoustic models having different properties to quickly adapt to a speaker's utterance style and a difference in acoustic environment (noise / reverberation) of the usage environment.

  In order to solve the above problems, according to a first aspect of the present invention, a speech recognition device is provided with a storage unit that stores a language model and a plurality of neural network acoustic models that differ for each latent class, and is input. The weight of each latent class is estimated from the speech data, and speech recognition is performed on the speech data based on the estimated weight for each latent class and a plurality of neural network acoustic models different for each language model and latent class.

  In order to solve the above-described problem, according to another aspect of the present invention, an acoustic model learning device includes a plurality of latent class prior distribution parameters indicating the likelihood of occurrence of a latent class, and feature quantities that are different for each latent class. Acoustic model storage that stores feature quantity generation distribution parameters that are generation distribution parameters, neural network parameters of neural network acoustic models, and latent class distributions that indicate the likelihood of occurrence of latent classes after observation of learning data Part, latent class distribution, learning state variable sequence group that is a group of state variable sequences of learning speech data, and learning feature amount sequence group that is a group of feature amount sequences of learning speech data The neural network learning unit that updates the neural network parameters, the latent class distribution, and the input speech features for learning. A feature quantity generation distribution learning unit for updating the feature quantity generation distribution parameter, a latent class advance distribution learning unit for updating the latent class prior distribution parameter using the latent class distribution, a neural network parameter, and a feature quantity generation distribution parameter. A latent class distribution parameter, a latent class distribution learning unit for updating the latent class distribution using the input learning state sequence and the learning speech feature quantity, and a neural network parameter, a feature quantity generation distribution parameter, a latent Until the update of the class prior distribution parameter and the latent class distribution converges, the processes in the neural network learning unit, the feature amount generation distribution learning unit, the latent class prior distribution learning unit, and the latent class distribution learning unit are repeated.

  In order to solve the above-described problem, according to another aspect of the present invention, the speech recognition method includes a language model and a plurality of neural network acoustic models different for each latent class stored in a storage unit, Estimates the weight of each latent class from the input speech data, and performs speech recognition on the speech data based on the estimated weight for each latent class and a plurality of neural network acoustic models different for each language model and latent class. .

  In order to solve the above-described problem, according to another aspect of the present invention, an acoustic model learning method includes a latent class indicating the likelihood of occurrence of a plurality of latent classes in the acoustic model storage unit, which are different for each latent class. A prior distribution parameter, a feature generation distribution parameter that is a parameter of a feature generation distribution, a neural network parameter of a neural network acoustic model, and a latent class distribution indicating the likelihood of occurrence of a latent class upon observation of learning data A learning feature quantity sequence that is a group of a latent class distribution, a state variable series group for learning that is a group of state variables of learning speech data, and a series of feature quantities of the speech data for learning Neural network learning step to update neural network parameters using group, latent class distribution, and input learning A feature quantity generation distribution learning step for updating feature quantity generation distribution parameters using speech feature quantities, a latent class advance distribution learning step for updating latent class prior distribution parameters using latent class distribution, and a neural network parameter A latent class distribution learning step for updating the latent class distribution using the feature quantity generation distribution parameter, the latent class prior distribution parameter, the input learning state sequence and the learning speech feature quantity, and the neural network parameter, the feature Until the update of the quantity generation distribution parameter, the latent class prior distribution parameter, and the latent class distribution converges, the processes in the neural network learning step, the feature amount generation distribution learning step, the latent class prior distribution learning step, and the latent class distribution learning step are repeated.

  High speed to the speaker / environment, which was impossible in the past in a speech recognition device using an acoustic model consisting of NN, which is generally said to have higher performance than a speech recognition device using an acoustic model consisting of a mixed Gaussian distribution There is an effect that adaptation (adaptation processing is performed each time an utterance is processed without accumulating adaptation data) becomes possible.

The figure which shows the processing flow of the speech recognition apparatus of a prior art. The functional block diagram of the speech recognition apparatus of a prior art. The figure which shows the processing flow of the acoustic model learning apparatus which concerns on 1st embodiment. The figure which shows the structural example of the acoustic model learning apparatus which concerns on 1st embodiment. The figure which shows the processing flow of the speech recognition apparatus which concerns on 1st embodiment. The figure which shows the structural example of the speech recognition apparatus which concerns on 1st embodiment. The figure which shows the simulation result of the speech recognition apparatus which concerns on 1st embodiment.

  Hereinafter, embodiments of the present invention will be described.

<First embodiment>
<Points of first embodiment>
[NN acoustic model used in this embodiment]
In order to actively incorporate changes in the environment and speakers into the model, this embodiment attempts to model the joint probability distribution p (x, s) of the state variable series s and the feature quantity series x. Where x is a random variable corresponding to the acoustic feature sequence {x ₁ , x ₂ , ..., x _t , ...}, and s is a state variable sequence {s ₁ , s ₂ , ..., s _t , Is a label series corresponding to. In the conventional NN acoustic model, the joint probability distribution p (x, s) introduces a single NN parameter Λ, and p (x, s | Λ) = p (x | s, Λ) p (s) Was modeled as: In the present embodiment, in consideration of the potential environment / speaker factors inherent in the utterance, it is defined as a mixed model (hereinafter referred to as a latent class model) using the latent class k as follows.

Here, K is the number of latent classes assumed in advance. Here, a latent class refers to a group having a homogeneous propensity to be classified by potential (not directly observable) environmental / speaker factors. When such a latent class model is used, the connection probability is the product of the feature generation probability p (x | k), the state variable probability p (s | x, k), and the latent class prior probability p (k). You can think of it. Here, assuming the independence of each element in the time series and introducing different parameters to each distribution function, the following expression is obtained.

Here, Λ _k is an NN parameter in the latent class k, and Θ _k is a parameter of the feature quantity generation distribution in the latent class k (hereinafter also referred to as “feature quantity generation distribution parameter”). Equation (8) expresses the joint probability distribution p (x, s) between the state variable series s and the feature quantity series x, and the probability p (x, s | k, Λ calculated from the NN acoustic model for each latent class k. _k , Θ _k ) and the probability p (k) indicating the probability of occurrence of the latent class k is defined as the sum of all latent classes.

In this latent class model, NN is used to _define p (s _t | x _t , Λ _k ) as in equation (2 ') (NN parameter Λ _k refers to a different variable depending on latent class k) .

If we define the probability p (x _t | Θ _k ) using a conventional multivariate continuous distribution function and parametrize p (k) to directly estimate the multinomial distribution, that is, the probability p (k) = p _k The adjustable parameters of the model are the NN parameter Λ _k and the feature value generation distribution parameters Θ _k and p _k . Hereinafter, p _k is referred to as a latent class prior distribution or a latent class prior distribution parameter from the viewpoint of indicating the probability of a latent class (indicating the likelihood of each latent class for all latent classes). The “latent class distribution q _{n, k} ” described later indicates the likelihood of the latent class k after observing the corresponding nth data, and the “latent class prior distribution p _k ” Refers to a different distribution.

The multivariate continuous distribution used as the probability p (x _t | Θ _k ) includes a normal distribution and a mixed normal distribution represented by the following expressions.

here

Μ _k is an average vector of feature generation probabilities p (x | k) belonging to latent class k, and V _k is a covariance matrix of feature generation probabilities p (x | k) belonging to latent class k. Hereinafter, a parameter estimation method considering the latent class k will be described.

<Parameter estimation method considering latent class k>
Since the defined latent class model is a mixture distribution, all parameters (Θ _k , Λ _k , p _k ) can be estimated using an existing EM algorithm (see Reference 1).
[Reference 1] AP Dempster, NM Laird, DB Rubin, “Maximum Likelihood from Incomplete Data via the EM Algorithm”, Journal of Royal Statistical Society, 1977, Series B, Vol. 39, No. 1, pp.1-38 .

(Mixed model and EM algorithm)
Hereinafter, an outline of a method for estimating a parameter using the EM algorithm will be described.

Consider defining a probability distribution p (A | Θ) for variable A using parameter Θ. By defining the probability distribution of variable A in several latent classes k using the parameter θ _k defined for each latent class k (probability distribution p (A | k, θ _k )) and by marginalizing k It is possible to define a probability distribution p (A | Θ) (see equation (10)).

In this case, the parameter Θ is

It becomes.

  A technique for improving expressive power by modeling in this way a plurality of small classes that can model a complex probability distribution is called mixed modeling.

As learning of the mixed model, a method of maximizing the log likelihood in the learning data is generally taken. Using learning data A ⁽¹⁾ , A ⁽²⁾ ,..., A ⁽ⁱ⁾ ,..., Optimal parameters can be set as follows.

In general, a method called an EM algorithm is used for learning a mixed model (see Reference 1). In the EM algorithm, the lower bound is derived as follows using Jensen's inequality and occupancy parameters q _{i, k} (Σ _k q _{i, k} = 1).

Since this equality holds when q _{i, k} = p (k | A ⁽ⁱ⁾ , Θ), the parameter that maximizes the lower bound while sequentially estimating q _{i, k} from the current estimation parameter Θ ' By estimating Θ, an EM algorithm can be constructed. That is, the EM algorithm is executed by repeating the following two steps.
E-step: Estimate q _{i, k} = p (k | A ⁽ⁱ⁾ , Θ ′) from the given parameter Θ ′.
M-step: The obtained q _{i, k} is substituted into the above equation (12), Θ that maximizes Q (Θ; q _{i, k} ) is obtained, and is substituted into Θ ′.

Note that the M-step of the EM algorithm does not necessarily require an exact solution that maximizes Q (Θ; q _{i, k} ), and may be substituted with a numerical solution as necessary.

<Estimation of parameters in this embodiment>
In the EM algorithm, it is necessary to calculate the estimated value of the latent class distribution (q _{i, k} in Equation (12)), but in the model of this embodiment, there is a latent class k for each utterance n, and the latent class k is Given the probability distribution of the observed values given (p (x, s | k, Λ _k , Θ _k ) in Eq. (8), the latent class distribution estimate q _{n, k} Using parameter estimates Λ _k ′ _{, Θ k} ′ _, and p _k ′, they are expressed as follows:

In the case of the model of this embodiment, the optimization of the lower bound (Q (Θ; q _{i, k} ) in Equation (12)) after the estimated value q _{n, k of} the latent class distribution is determined is written as follows: be able to.

  Since this optimization is an optimization of the sum of a plurality of terms that do not share a variable, an optimum value is obtained by decomposing each variable and solving the optimization. That is, the optimization of equation (14) is broken down into the following 3 × K optimization problems.

Here, p (s _t ⁽ⁿ⁾ | x _t ⁽ⁿ⁾ , Λ _k ) in Expression (15) is the feature vector x _t ^{(n of the} _tth frame of the acoustic model Λ _k and the nth learning data. ⁾ and an output probability of the state variables s _t ⁽ⁿ⁾ to the condition can be regarded as correct answer probabilities of acoustic models lambda _k and the feature amount vector x _t ⁽ⁿ⁾ for the state variables s _t ^(n). Further, the estimated value q _{n, k} of the latent class distribution in equation (15) can be regarded as the weight of the latent class k in the nth learning data. Equation (15) shows that the sum of the products of the correct answer probability of the state variable sequence for each learning data and the weight q _{n, k} of the latent class k in the learning data is the maximum for all learning data and all latent classes. It means that the NN parameter Λ _k is updated so that

Equation (16) is the sequence {x ₁ ⁽ⁿ⁾ , x ₂ ^{( )} of the feature vector x _t ⁽ⁿ⁾ from the probability distribution indicated by the feature generation distribution parameter Θ _k for all learning data and all latent classes. ⁿ⁾ , ..., x _t ⁽ⁿ⁾ , ...} are generated model parameters so that the sum of products of the probability of sampling (output) and the weight q _{n, k} of latent class k in the learning data is maximized. It is meant to update the Θ _k.

Equation (17) means that the latent class prior distribution p _k that is closest to the estimated latent class distribution q _{n, k} is optimized.

Optimization for the latent class prior distribution p _k (equation (17)) is

It is known that there exists an optimal solution. That is, the latent class prior distribution p _k can be obtained as a ratio of the latent class distribution of each latent class to the sum of the latent class distributions of all the latent classes.

Also, regarding the optimization of the feature quantity generation distribution parameter Θ _k (equation (16)), it is known that an optimal solution can be derived analytically if it is a simple distribution such as a normal distribution.

However, with regard to the optimization related to the NN parameter Λ _k (equation (15)), the optimal solution cannot be derived analytically due to the characteristics of the NN except in a few cases. Therefore, it is necessary to numerically solve the optimization of the NN parameter Λ _k by using the back propagation method (see Non-Patent Document 1).

In order to construct the EM algorithm specifically based on these, the learning feature sequence group X = {X ⁽¹⁾ , X ⁽²⁾ ,…, X ⁽ⁿ⁾ ,…} and the learning state variable sequence group S = {s ⁽¹⁾ , s ⁽²⁾ ,…, s ⁽ⁿ⁾ ,…}, parameter estimates

, The following steps are repeated to optimize.
E-step (1): p (x _t ⁽ⁿ⁾ | Θ _k ′) is calculated for all k, n, and t according to equation (9 ′).

E-step (2): For all k, n and t, p (s _t ⁽ⁿ⁾ | x _t ⁽ⁿ⁾ , Λ _k ′) is calculated according to the equation (2 ′).

E-step (3): The latent class distribution q _{n, k} is calculated according to the equation (13) for all k, n.

M-step (1): Optimization (equation (15)) is executed by the back propagation method (see Non-Patent Document 1) using the obtained latent class distribution q _{n, k} .

M-step (2): Optimization (equation (16)) is executed using the obtained latent class distribution q _{n, k} by a method corresponding to the feature value generation distribution.

M-step (3): Using the obtained latent class distribution q _{n, k} , the latent class prior distribution p _k

Update like this.

A series of processing can be accelerated by introducing the following Viterbi approximation to the latent class distribution q _{n, k} .

In the Viterbi approximation, the following approximate expression is used instead of the expression (13) for calculating the latent class distribution q _{n, k} .

Here, δ _{i, j} is a Kronecker delta, and is a variable that is 1 only when i = j and 0 in other cases. By using this approximation, many elements of the latent class distribution q _{n, k} become zero, so that the substantial calculation time can be greatly reduced.

<Acoustic model learning apparatus 100>
FIG. 3 shows a processing flow of the acoustic model learning apparatus 100 according to the first embodiment, and FIG. 4 shows a configuration example thereof.

  The acoustic model learning device 100 includes an acoustic model storage unit 110, an acoustic model learning unit 120, a latent class distribution learning unit 130, and an iterative control unit 140. The acoustic model learning device 100 receives the learning feature quantity sequence group X and the learning state variable sequence group S, and learns and outputs K NN acoustic models having different latent classes k using these data.

<Acoustic model storage unit 110>
In the acoustic model storage unit 110, as the NN acoustic model, the NN parameter Λ _k , the feature amount generation distribution parameter Θ _k , the latent class prior distribution parameter p _k, and the latent class distribution q _{n, k} for each latent class k are stored. Is stored.

Prior to learning each parameter, the acoustic model learning device 100 initializes the NN parameter Λ _k and the latent class distribution q _{n, k} (s101), and stores the initial values in the acoustic model storage unit 110.

A random number is substituted for the initial value of the latent class distribution q _{n, k} . Any random number satisfying Σ _k q _{n, k} = 1, q _{n, k} > 0 may be used. For example, a positive integer {1, ..., K} q n with r _n chosen uniformly _{randomly, k} = [delta] _{n, rn} (where subscript rn represents r _n) is initialized as Also good. Note that the initialization process of the latent class distribution is omitted, an appropriate value is set in advance in the latent class distribution q _{n, k} and stored in the acoustic model storage unit 110, and the value is used as an initial value. Also good. In this way, by setting different values as initial values of the latent class distribution q _{n, k} , it is possible to learn NN acoustic models having different properties for each latent class k with respect to the same learning data.

Any value can be given as the initial value of the NN parameter Λ _k . For example, the NN parameter Λ trained by the above-described conventional NN acoustic model creation method using all the learning data is used as the NN parameter for all k. Let Λ _k . This learning can be executed by performing the equation (15) with the setting of q _{n, k} = 1 in the NN learning unit 121 described later. The result is stored in the acoustic model storage unit 110.

<Acoustic model learning unit 120>
The acoustic model learning unit 120 includes an NN learning unit 121, a feature amount generation distribution learning unit 122, and a latent class prior distribution learning unit 123. The acoustic model learning unit 120 includes a learning state variable sequence group S = {s ⁽¹⁾ , s ⁽²⁾ ,..., S ⁽ⁿ⁾ , ...} and a learning feature quantity sequence group X = {X ⁽¹⁾ , NN parameter Λ _k , feature quantity generation distribution parameter Θ _k and latent class prior distribution parameter p _k are learned from X ⁽²⁾ ,..., X ⁽ⁿ⁾ ,. The processes of the NN learning unit 121, the feature quantity generation distribution learning unit 122, and the latent class prior distribution learning unit 123 may be performed in any order.

(NN learning unit 121)
The NN learning unit 121 uses the latent class distribution q _{n, k} read from the acoustic model storage unit 110 and the input learning state variable sequence group S and learning feature amount sequence group X to obtain an equation (15 ) To learn the NN parameter Λ _k (s102) and update the NN parameter Λ _k stored in the acoustic model storage unit 110.

For example, this corresponds to executing the above-described M-step (1). Note that the iterative process by the back-propagation method of M-Step (1) ends the update process when it is repeated a predetermined number of times.

(Feature generation generation learning unit 122)
The feature value generation distribution learning unit 122 uses the latent class distribution q _{n, k} read from the acoustic model storage unit 110 and the input feature value series X for learning to satisfy the feature (16). The generation distribution parameter Θ ′ _k is learned (s 103), and the feature amount generation distribution parameter Θ _k stored in the acoustic model storage unit 110 is updated.

(Latent class prior distribution learning unit 123)
The latent class prior distribution learning unit 123 uses the latent class distribution q _{n, k} read from the acoustic model storage unit 110 to learn the latent class prior distribution parameter p _{k according} to the equation (17 ′) (s104), The latent class prior distribution parameter p _k stored in the model storage unit 110 is updated.

<Latent class distribution learning unit 130>
The latent class distribution learning unit 130 includes each parameter (NN parameter Λ _k , feature value generation distribution parameter Θ _k , latent class prior distribution parameter p _k ) updated by the acoustic model learning unit 120 and the input learning state variable. Using the sequence group S and the learning feature sequence group X, the latent class distribution q _{n, k} is learned by the equation (13) (s105), and the latent class distribution q _n, Update _k .

The probability p (s ⁽ⁿ⁾ _t | x ⁽ⁿ⁾ _t , Λ _t ′) and the probability p (x ⁽ⁿ⁾ _t | Θ _k ′) appearing in equation (13) are What was calculated | required in the feature-value production | generation distribution learning part 122 may be used.

<Repetition control unit 140>
The iterative control unit 140 determines whether or not the update process has converged (s106), and if it has converged, ends the update process for the learning state variable series group S and the learning feature quantity series group X. For example, the execution time may be measured and determined to have converged when the predetermined time is reached, or the number of updates in the acoustic model learning unit 120 or the latent class distribution learning unit 130 is counted and the predetermined number of times is reached. Then, it may be determined that it has converged. If it is not determined that it has converged, a control signal is output to the acoustic model learning unit 120 and the latent class distribution learning unit 130 so as to repeat the processing.

<Effect>
With such a configuration, a plurality of NN acoustic models having different properties can be learned from the same learning data.

<Principle of voice recognition>
The speech recognition apparatus 200 is different from the conventional speech recognition apparatus in that it includes a plurality of different NN acoustic models for each latent class k. A plurality of NN acoustic models different for each latent class k can be constructed by the acoustic model learning device 100. The speech recognition apparatus 200 performs speech recognition by using a plurality of different NN acoustic models for each latent class k.

Up to now, when there are multiple acoustic models with mixed Gaussian distribution, the methods to improve the speech recognition accuracy by using them are as follows: (1) System combination method (see Reference 2), (2) Latent class advance A method based on re-estimation of the distribution p _k and (3) a method based on model selection have been proposed.
[Reference 2] G. Evermann, P. Woodland, "Posterior probability decoding, confidence estimation and system combination", Proc. NIST Speech Transcription Workshop, 2000

  However, there are multiple NN acoustic models, and the technology to properly use these models has not been known.

  In the following, an embodiment in which the “method by model selection” is applied to a plurality of NN acoustic models will be described. The feature of this embodiment is a configuration for performing speech recognition using a plurality of acoustic models having different latent classes. Thus, the specific processing part of the voice recognition may be realized in any way. For example, an embodiment in which a method based on system combination or latent class distribution estimation is applied may be used.

  That is, the speech recognition apparatus 200 estimates the weight of each latent class from the input speech data, and based on the estimated weight for each latent class and a plurality of NN acoustic models that differ for each latent class, Performs voice recognition on voice data. There are various methods for estimating the weight and processing the NN acoustic model using the weight.

In the model selection method, a latent class k that is considered to correspond to input speech data is first estimated, and speech recognition is performed using only an acoustic model corresponding to the latent class k. Therefore, it is considered that the weight for the selected NN acoustic model is estimated as 1 and the weight for the other NN acoustic models is estimated as 0. It should be noted that the posterior distribution of the latent class for the feature quantity series x not included in the learning data needs to be expressed as a sum (Σ _{s ′} ) for all possible state variable series s ′ as follows.

In general, since this sum cannot be easily calculated, 1-best approximation is used in this embodiment. 1-best in approximation, one by one the optimal state variable sequence ^ s _k firstly the maximum likelihood for each latent class k, is calculated as follows.

In this way using the optimal state variable sequence ^ s _k obtained is approximated as follows posterior distribution of the latent class.

Here, α and β are adjustment items (scale factors) for model synthesis, and values are set in advance.

  Using this approximation, the latent class ^ k corresponding to the input speech X is obtained as follows.

Then, the conventional speech recognition process is executed using the acoustic model parameter ^ Λ _k corresponding to the latent class ^ k. By estimating the latent class ^ k for each utterance, it is possible to recognize in consideration of the latent class according to the environment of the utterance and the characteristics of the speaker.

<Voice recognition apparatus 200>
An embodiment of the speech recognition apparatus 200 configured based on the above theory will be described. FIG. 5 shows a processing flow of the speech recognition apparatus 200 according to the present embodiment, and FIG. 6 shows a configuration example thereof.

  The speech recognition apparatus 200 includes a feature amount extraction unit 210, an optimal state variable sequence estimation unit 220, an optimal latent class estimation unit 230, a word string search unit 240, an acoustic model storage unit 250, and a language model storage unit 260.

  The speech recognition apparatus 200 receives speech data for recognition, performs speech recognition on the speech data, and outputs a recognition result word string.

<Acoustic model storage unit 250 and language model storage unit 260>
The acoustic model storage unit 250 stores K NN acoustic models (Λ _t , Θ _k , p _k , q _{n, k} ) having different latent classes k learned by the acoustic model learning device 100. The language model storage unit 260 stores language models. A language model based on existing technology may be used.

<Feature Extraction Unit 210>
Feature amount extraction unit 210 receives the input audio data, extracts a feature vector x _t from the voice data (s201), and outputs. An existing technique can be used for feature amount extraction.

<Optimum State Variable Sequence Estimator 220>
The optimum state variable sequence estimation unit 220 is a part of a group X ⁽ⁿ⁾ of a series X ⁽ⁿ⁾ of a plurality of feature quantity vectors x _t extracted by the feature quantity extraction unit 210 (hereinafter also referred to as “recognition feature quantity series group”). (For example, a sequence X ^{(n) of} a plurality of feature vectors x _t ⁽ⁿ⁾ for one utterance), and a latent class prior distribution parameter p _k and an NN parameter Λ _{k for} each latent class stored in the acoustic model storage unit 250 Then, using the feature quantity generation distribution parameter Θ _k , the equation (20a) is calculated, K optimum state variable sequences (^ s _k ) are estimated (s 202), and passed to the optimum latent class estimation unit 230.

<Optimum latent class estimation unit 230>
Optimum latent classes estimator 230, a part of the recognition feature amount sequence group X (e.g. X ^(n)), and the optimal state sequence ^ s _k estimated by the optimum state variable sequence estimating unit 220, the acoustic model storage unit 250 Using the latent class prior distribution parameter p _k , the NN parameter Λ _k , and the feature quantity generation distribution parameter Θ _k stored in, for a part of the recognition feature quantity sequence group X by Equation (21) and Equation (22) The optimal latent class ^ k is selected (s203) and output to the word string search unit 240.

<Word string search unit 240>
The word string search unit 240 uses the language model stored in the language model storage unit 260, the acoustic model stored in the acoustic model storage unit 250, and the recognition feature quantity sequence group X in the same manner as the conventional speech recognizer. Then, a word string that matches the recognition speech data (more specifically, the recognition feature quantity sequence group X) is searched (s204), and the word string that is the search result is output. However, unlike the word string search unit 92 in the conventional speech recognition apparatus 90, the optimum latent selected by the optimum latent class estimation unit 230 among the plurality of NN acoustic models stored in the acoustic model storage unit 250 as an acoustic model. Use NN parameter Λ _{^ k} corresponding to class ^ k. At that time, by estimating the optimal latent class ^ k for each utterance n and changing the NN parameter Λ _{^ k} to be used, high-speed adaptation to the speaker / environment becomes possible.

<Simulation results>
FIG. 7 shows a simulation result of speech recognition of the speech recognition apparatus 200 according to the first embodiment. The corpus used for the simulation was TIMIT. The number of utterances in the learning set and the evaluation set is 3,696 utterances and 392 utterances, respectively. The number of latent classes K is 2, and the number of hidden units H ⁽ⁱ⁾ of NN is 1024. Multivariate Gaussian distribution was used for the feature generation distribution. In learning, the initial value of q _{n, k} was set to q _{n, k} = δ _{n, rn} using r _n uniformly selected from positive integers {1, ..., K}. Some values ((1) α = 1.0, β = 1.0, (2) α = 1.0, β = 0.0, (3) α = 0.0 for scale factors α, β (see equation (21)) in model synthesis , β = 1.0).

  As shown in FIG. 7, it was confirmed that the speech recognition apparatus 200 according to the first embodiment was able to improve accuracy by using the NN acoustic model. In addition, as to how to select the latent class to be used, it was confirmed that sufficient performance improvement can be seen even when the setting α = 1.0, β = 1.0, which is considered to be the closest approximation.

<Effect>
High speed to the speaker / environment, which was impossible in the past in a speech recognition device using an acoustic model consisting of NN, which is generally said to have higher performance than a speech recognition device using an acoustic model consisting of a mixed Gaussian distribution There is an effect that adaptation (adaptation processing in real time without accumulating adaptation data) becomes possible.

<Other variations>
The present invention is not limited to the above-described embodiments and modifications. For example, the various processes described above are not only executed in time series according to the description, but may also be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. In addition, it can change suitably in the range which does not deviate from the meaning of this invention.

<Program and recording medium>
In addition, various processing functions in each device described in the above embodiments and modifications may be realized by a computer. In that case, the processing contents of the functions that each device should have are described by a program. Then, by executing this program on a computer, various processing functions in each of the above devices are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Further, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its storage unit. When executing the process, this computer reads the program stored in its own storage unit and executes the process according to the read program. As another embodiment of this program, a computer may read a program directly from a portable recording medium and execute processing according to the program. Further, each time a program is transferred from the server computer to the computer, processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program includes information provided for processing by the electronic computer and equivalent to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In addition, although each device is configured by executing a predetermined program on a computer, at least a part of these processing contents may be realized by hardware.

DESCRIPTION OF SYMBOLS 100 Acoustic model learning apparatus 110 Acoustic model storage part 120 Acoustic model learning part 121 Learning part 122 Feature quantity generation distribution learning part 123 Latent class prior distribution learning part 130 Latent class distribution learning part 140 Repetition control part 200 Speech recognition apparatus 210 Feature quantity extraction Unit 220 optimal state variable sequence estimation unit 230 optimal latent class estimation unit 240 word string search unit 250 acoustic model storage unit 260 language model storage unit

Claims (10)

A storage unit storing a language model and a plurality of neural network acoustic models different for each latent class;
Speech recognition for the speech data based on estimated weights for each latent class and a plurality of neural network acoustic models that differ for each language class and latent class from the input speech data. And
In the storage unit, as the neural network acoustic model, for each latent class, a latent class prior distribution parameter indicating the likelihood of occurrence of a latent class, a feature amount generation distribution parameter that is a parameter of a feature amount generation distribution, and a neural network The neural network parameters of the acoustic model are stored,
Using the latent class prior distribution parameter, the neural network parameter, and the feature value generation distribution parameter, an optimal state variable sequence estimation unit that estimates an optimal state variable sequence for the speech data for each latent class;
Using the latent class prior distribution parameter, the neural network parameter, the feature quantity generation distribution parameter, and the optimal state variable series, an optimal latent class estimation unit that selects an optimal latent class for the speech data;
Using a neural network parameter corresponding to the optimal latent class and the language model, further including a word string search unit for searching a word string for the speech data.
Voice recognition device. A speech recognition apparatus according to claim 1 Symbol placement,
It is assumed that the latent class distribution indicates the probability of occurrence of the latent class after observing the learning data,
The neural network parameter uses the latent class distribution and the input learning state variable series group and learning feature quantity series group to maximize the sum of products of the correct probability of the state variable series and the latent class distribution. It was obtained by asking so that
The feature amount generation distribution parameter uses the latent class distribution and the input speech feature amount for learning so that the sum of the product of the probability that the feature amount of the learning data is output and the latent class distribution is maximized. Obtained by asking for
The latent class prior distribution parameter is obtained by calculating the ratio of the latent class distribution of each latent class to the sum of the latent class distributions of all the latent classes using the latent class distribution.
Voice recognition device. A plurality of latent class prior distribution parameters indicating the likelihood of occurrence of latent classes, feature quantity generation distribution parameters that are parameters of feature quantity generation distribution, neural network parameters of a neural network acoustic model, and learning data An acoustic model storage unit that stores a latent class distribution indicating the likelihood of occurrence of a latent class after observing
Using the latent class distribution, a learning state variable sequence group that is a group of state variables of learning speech data, and a learning feature amount sequence group that is a group of feature amounts of learning speech data, A neural network learning unit for updating the neural network parameters;
A feature quantity generation distribution learning unit that updates the feature quantity generation distribution parameter using the latent class distribution and the input speech feature quantity for learning;
A latent class prior distribution learning unit that updates the latent class prior distribution parameters using the latent class distribution;
A latent class distribution learning unit that updates the latent class distribution using the neural network parameter, the feature quantity generation distribution parameter, the latent class prior distribution parameter, the input learning state series, and the learning speech feature quantity. Including
Until the update of the neural network parameter, the feature amount generation distribution parameter, the latent class prior distribution parameter and the latent class distribution converges, the neural network learning unit, the feature amount generation distribution learning unit, the latent class prior distribution learning unit, and Repeat the process in the latent class distribution learning unit,
Acoustic model learning device. The acoustic model learning device according to claim 3 ,
k is a latent class index, n is an utterance index, t is a frame index, q _{n, k} is the latent class distribution, s _t ⁽ⁿ⁾ is a state variable of learning speech data, and x _t ⁽ⁿ⁾ is learned And the neural network learning unit updates the neural network parameter Λ _{k according} to the following equation:

The feature quantity generation distribution learning unit updates the feature quantity generation distribution parameter Θ _{k according} to the following equation:

The latent class prior distribution learning unit updates the latent class prior distribution parameter p _{k according} to the following equation:

The latent class distribution learning unit updates the latent class distribution q _{n, k according} to the following equation:

Acoustic model learning device. Assume that the language model and multiple neural network acoustic models that differ for each latent class are stored in the storage unit,
Speech recognition for the speech data based on estimated weights for each latent class and a plurality of neural network acoustic models that differ for each language class and latent class from the input speech data. And
In the storage unit, as the neural network acoustic model, for each latent class, a latent class prior distribution parameter indicating the likelihood of occurrence of a latent class, a feature amount generation distribution parameter that is a parameter of a feature amount generation distribution, and a neural network Suppose that the neural network parameters of the acoustic model are stored,
Using the latent class prior distribution parameter, the neural network parameter, and the feature value generation distribution parameter, an optimal state variable sequence estimation step for estimating an optimal state variable sequence for the speech data for each latent class;
Using the latent class prior distribution parameter, the neural network parameter, the feature quantity generation distribution parameter, and the optimal state variable series, an optimal latent class estimating step of selecting an optimal latent class for the speech data;
A word string search step of searching for a word string for the speech data using the neural network parameter corresponding to the optimal latent class and the language model;
Speech recognition method. The speech recognition method according to claim 5 ,
It is assumed that the latent class distribution indicates the probability of occurrence of the latent class after observing the learning data,
The neural network parameter uses the latent class distribution and the input learning state variable series group and learning feature quantity series group to maximize the sum of products of the correct probability of the state variable series and the latent class distribution. It was obtained by asking so that
The feature amount generation distribution parameter uses the latent class distribution and the input speech feature amount for learning so that the sum of the product of the probability that the feature amount of the learning data is output and the latent class distribution is maximized. Obtained by asking for
The latent class prior distribution parameter is obtained by calculating the ratio of the latent class distribution of each latent class to the sum of the latent class distributions of all the latent classes using the latent class distribution.
Speech recognition method. The acoustic model storage unit includes a plurality of latent class prior distribution parameters indicating the likelihood of occurrence of latent classes, feature quantity generation distribution parameters that are parameters of the feature quantity generation distribution, and neural network acoustic models. A neural network parameter and a latent class distribution indicating the likelihood of occurrence of a latent class upon observation of learning data are stored.
Using the latent class distribution, a learning state variable sequence group that is a group of state variables of learning speech data, and a learning feature amount sequence group that is a group of feature amounts of learning speech data, A neural network learning step of updating the neural network parameters;
A feature quantity generation distribution learning step for updating the feature quantity generation distribution parameter using the latent class distribution and the input speech feature quantity for learning,
A latent class prior distribution learning step of updating the latent class prior distribution parameter using the latent class distribution;
A latent class distribution learning step of updating the latent class distribution using the neural network parameter, the feature quantity generation distribution parameter, the latent class prior distribution parameter, the input learning state series and the learning speech feature quantity. Including
Until the update of the neural network parameter, the feature quantity generation distribution parameter, the latent class prior distribution parameter and the latent class distribution converges, the neural network learning step, the feature quantity generation distribution learning step, the latent class prior distribution learning step, and Repeat the process in the latent class distribution learning step,
Acoustic model learning method. The acoustic model learning method according to claim 7 ,
k is a latent class index, n is an utterance index, t is a frame index, q _{n, k} is the latent class distribution, s _t ⁽ⁿ⁾ is a state variable of learning speech data, and x _t ⁽ⁿ⁾ is learned The neural network parameter Λ _k is updated according to the following equation in the neural network learning step as the feature amount of the voice data for use:

In the feature quantity generation distribution learning step, the feature quantity generation distribution parameter Θ _k is updated by the following equation:

The latent class prior distribution learning step updates the latent class prior distribution parameter p _{k according} to the following equation:

In the latent class distribution learning step, the latent class distribution q _{n, k} is updated according to the following equation:

Acoustic model learning method. Program for causing a computer to function as a speech recognition apparatus according to claim 1 or claim 2, wherein. A program for causing a computer to function as the acoustic model learning device according to claim 3 or 4 .

Images