CN103117060B - For modeling method, the modeling of the acoustic model of speech recognition - Google Patents

Abstract

The invention relates to a modeling method of an acoustic model for speech recognition and a speech recognition system. The method includes: training an initial model, the modeling unit is a triphone state clustered by a phoneme decision tree, and the model also provides a state transition probability; Perform mandatory alignment to obtain its frame-level state information; pre-train the deep neural network to obtain the initial weights of each hidden layer; use the error back-propagation algorithm to train the initialized network based on the obtained frame-level state information, Update weights. The present invention adopts the context-dependent triphone state as the modeling unit, based on the deep neural network modeling, uses the restricted Boltzmann algorithm to initialize the weights of each hidden layer of the network, and the weights can also be used in the follow-up. The error propagation algorithm is updated, which can effectively alleviate the risk of falling into local extremum during the network pre-training, and further improve the modeling accuracy of the acoustic model.

Description

Modeling method and modeling system of acoustic model for speech recognition

technical field

The invention relates to the field of speech recognition, in particular to a modeling method and modeling system of an acoustic model for speech recognition.

Background technique

The current mainstream framework for speech recognition is based on statistical pattern recognition. A typical speech recognition system framework is shown in Figure 1: it includes speech collection and front-end processing modules, feature extraction modules, acoustic model modules, language model modules, and decoder modules. The basic process of speech recognition is as follows: the speech collection device collects the human speech and performs feature extraction after front-end processing. The extracted feature sequence, such as MFCC or PLP, obtains its observation probability through the acoustic model, and sends it to the decoder in combination with the language model probability to obtain the most effective possible text sequences. The modeling of the acoustic model is based on the Hidden Markov Framework, and the probability distribution of speech features is modeled by using a mixed Gaussian model. The mixed Gaussian model will make some inappropriate assumptions about speech features and their distribution, such as the assumption of linear independence of adjacent speech features, and its observation probability obeys the mixed Gaussian distribution. In addition, when the Gaussian mixture model is used for parameter training, the objective function is to maximize the likelihood probability of the observed features, but the maximum a posteriori criterion is used for decoding, and the probability model is inconsistent. It can be seen that the traditional acoustic model has low modeling accuracy, resulting in poor speech recognition effect.

Contents of the invention

In view of the above problems, an embodiment of the present invention proposes a modeling method and a modeling system for an acoustic model for speech recognition.

In a first aspect, an embodiment of the present invention proposes a method for modeling an acoustic model for speech recognition, the method comprising: using training data to train a Hidden Markov-Mixed Gaussian HMM-GMM model, the HMM-GMM The modeling unit of the model is the triphone state after the phoneme decision tree clustering of the speech features of the training data, and the HMM-GMM model obtains the state transition probability of the triphone state through the expected maximum EM algorithm; based on The HMM-GMM model is forcibly aligning the triphone sub-states of the speech features of the training data to obtain the frame-level state information of the speech features; pre-training the deep neural network as the acoustic model to obtain Initialize the parameters of the weights of each hidden layer of the deep network; based on the triphone state of the training data voice feature, the deep neural network is trained using the error back propagation algorithm, and the weights of each hidden layer are updated .

Preferably, based on the HMM-GMM model, the triphone sub-states of the speech features of the training data are forcibly aligned to obtain frame-level state information of the speech features, specifically: based on the HMM-GMM model, Corresponding the speech features of the training data with their most probable triphone states to obtain frame-level state information of the speech features.

Preferably, the pre-training of the deep neural network as the acoustic model to obtain parameters for initializing the weights of the hidden layers of the deep network is specifically: using a restricted Boltzmann machine based on the The training data is trained layer by layer until convergence, and the weights of each hidden layer of the deep network are initialized with the obtained parameters.

In a second aspect, an embodiment of the present invention proposes a modeling system for an acoustic model of speech recognition, which includes: a first module for training a Hidden Markov-Mixed Gaussian HMM-GMM model with training data, the The modeling unit of the HMM-GMM model is the triphone state after the phoneme decision tree clustering of the speech features of the training data, and the HMM-GMM model obtains the state transition of the triphone state through the expected maximum EM algorithm Probability; the second module is used to forcibly align the triphone sub-states of the speech features of the training data based on the HMM-GMM model, and obtain the frame-level state information of the speech features; the third module is used to perform as The deep neural network of the acoustic model is pre-trained to obtain parameters for initializing the weights of each hidden layer of the deep network; the fourth module is used to adopt an error based on the triphone state of the training data voice feature The backpropagation algorithm trains the deep neural network and updates the weights of its hidden layers.

Preferably, based on the HMM-GMM model, the second module performs forced alignment on the triphone states of the speech features of the training data to obtain frame-level state information of the speech features, specifically: the second module Based on the HMM-GMM model, the speech features of the training data are associated with their most probable triphone states, and frame-level state information of the speech features is obtained.

Preferably, the third module pre-trains the deep neural network as the acoustic model to obtain parameters for initializing the weights of the hidden layers of the deep network, specifically: the third module uses limited The Boltzmann machine performs layer-by-layer training based on the training data until convergence, and uses the obtained parameters to initialize the weights of each hidden layer of the deep network.

The embodiment of the present invention adopts the triphonic state, based on the deep neural network modeling, and uses the restricted Boltzmann algorithm to initialize the weights of the hidden layers of the network, and the weights can also be obtained later by means of the reverse error propagation algorithm. The update can effectively alleviate the risk of easily falling into local extremum during the network pre-training, and further improve the modeling accuracy of the acoustic model.

Description of drawings

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Fig. 1 is a schematic diagram of an existing speech recognition system;

Fig. 2 is a block diagram of a speech recognition system based on a context-dependent deep neural network according to an embodiment of the present invention;

3 is a schematic diagram of a modeling method for an acoustic model for speech recognition according to an embodiment of the present invention;

Fig. 4 is a schematic diagram of a modeling system of an acoustic model for speech recognition according to an embodiment of the present invention.

Detailed ways

The technical solutions of the embodiments of the present invention will be described in further detail below with reference to the drawings and embodiments.

Considering that the mixed Gaussian model needs to make inappropriate assumptions about speech features and their probability distributions, the embodiment of the present invention uses a context-dependent deep neural network instead of the mixed Gaussian model for acoustic model modeling. The deep neural network includes a plurality of hidden layers, and its modeling unit is a context-dependent triphone state clustered by a phoneme decision tree. The basic block diagram of the whole system is shown in Figure 2.

The minimum cross-entropy criterion is used as the objective function during deep neural network training. Because it has multiple hidden layers, its error function has many local extremums, which makes it easy for the deep neural network to fall into local extremums during the training process and cause premature failure. convergence. To solve this problem, the neural computing field proposed to initialize the weight parameters through neural network pre-training, and then use the traditional error back propagation algorithm to train the network parameters. The pre-training algorithm uses a restricted Boltzmann machine, which is a two-way graph model, including a visible layer and a hidden layer, in which there is no interconnection between the units of the same layer and the units of different layers are dense Link. The model defines the joint distribution of visible layer and hidden layer variables through an energy function, and the specific formula is as follows:

Among them, v is the visible layer variable, h is the hidden layer variable, E(v,h) is the energy function, p(v,h) is its joint distribution probability, and the maximum observation feature likelihood probability p(v) is passed during training, Its weight parameter update formula is as follows:

Δw _ij ＝＜v _i h _j ＞ _data -＜v _i h _j ＞ _model

w _ij (t+1)=w _ij (t)+Δw _ij

Among them, w _ij is the connection weight, t is the number of iterations, and <> means to take the mean value of the variables in the brackets.

By training the restricted Boltzmann machine layer by layer, its parameters are used to initialize the deep neural network, so that its initial weight falls into a better starting point in the weight space, which alleviates the network from falling into local extremes during training to a certain extent. value risk. At the same time, the triphone state clustered by the phoneme decision tree is used as the teacher signal of the neural network, which includes the context relationship of the phoneme, making the modeling of the acoustic model more refined and accurate.

Fig. 3 is a schematic diagram of a modeling method of an acoustic model for speech recognition according to an embodiment of the present invention. The method includes: Step 1, establishing an initial model. Specifically, a Hidden Markov-Mixed Gaussian HMM-GMM model is trained with the training data, and the modeling unit of the HMM-GMM model is the triphone state after the phoneme decision tree clustering of the speech features of the training data, The HMM-GMM model obtains the state transition probability of the triphone state through the expected maximum EM algorithm;

Step 2, obtaining speech feature frame-level state information of the speech features of the training data. Specifically, based on the HMM-GMM model, the three-phone sub-states of the speech features of the training data are forcibly aligned to obtain frame-level state information of the speech features;

Preferably, based on the HMM-GMM model, the triphone sub-states of the speech features of the training data are forcibly aligned to obtain frame-level state information of the speech features, specifically: based on the HMM-GMM model, Corresponding the speech features of the training data with their most probable triphone states to obtain frame-level state information of the speech features.

Step 3, initialize the weights of each hidden layer of the deep neural network. Specifically, pre-training the deep neural network as the acoustic model to obtain parameters for initializing the weights of each hidden layer of the deep network;

Step 4, update the weights of each hidden layer of the deep neural network. Specifically, based on the triphone state of the speech feature of the training data, an error back propagation algorithm is used to train the deep neural network, and update the weights of each hidden layer.

Preferably, the pre-training of the deep neural network as the acoustic model to obtain parameters for initializing the weights of the hidden layers of the deep network is specifically: using a restricted Boltzmann machine based on the The training data is trained layer by layer until convergence, and the weights of each hidden layer of the deep network are initialized with the obtained parameters.

It should be noted that the hidden Markov-mixed Gaussian HMM-GMM model can also be written as a hidden Markov/mixed Gaussian HMM/GMM model.

The pre-training in step 3 can be regarded as a kind of unsupervised training. The training in step 3 can be regarded as a supervised training.

In addition, the pre-training in step 3 and step 2 can be performed at the same time.

When the HMM-GMM model is used as an acoustic model for speech recognition, based on the Bayesian formula, the posterior probability generated by the speech feature through the deep neural network is converted into a likelihood probability and sent to the decoder for decoding. After decoding, the The text sequence of is taken as the recognized speech content. The effect of speech recognition can be evaluated based on the difference between the recognized speech content and the real original speech. According to this effect, the performance of the deep neural network as an acoustic model in the speech recognition system can be evaluated, and retraining can be considered if necessary, and even redesign of the state transition probability in the HMM-GMM model can be considered.

Fig. 4 is a schematic diagram of a modeling system of an acoustic model for speech recognition according to an embodiment of the present invention. The modeling system includes: a first module for training a Hidden Markov-Mixed Gaussian HMM-GMM model with training data, the modeling unit of the HMM-GMM model undergoes phoneme decision-making for the phonetic features of the training data The triphone state after the tree clustering, the HMM-GMM model obtains the state transition probability of the triphone state through the expected maximum EM algorithm; the second module is used for based on the HMM-GMM model, the The triphonic sub-states of the speech features of the training data are forcibly aligned to obtain the frame-level state information of the speech features; the third module is used to pre-train the deep neural network as the acoustic model to obtain the deep neural network for initializing the deep layer The parameter of the weight of each hidden layer of network; The 4th module, for adopting error backpropagation algorithm to train described deep neural network based on the triphone state of described training data speech feature, update its each hidden layer the weight of.

Preferably, based on the HMM-GMM model, the second module performs forced alignment on the triphone states of the speech features of the training data to obtain frame-level state information of the speech features, specifically: the second module Based on the HMM-GMM model, the speech features of the training data are associated with their most probable triphone states, and frame-level state information of the speech features is obtained.

Preferably, the third module pre-trains the deep neural network as the acoustic model to obtain parameters for initializing the weights of the hidden layers of the deep network, specifically: the third module uses limited The Boltzmann machine performs layer-by-layer training based on the training data until convergence, and uses the obtained parameters to initialize the weights of each hidden layer of the deep network.

In the embodiment of the present invention, a deep neural network is used instead of a mixed Gaussian model to model an acoustic model, and a triphone state with context-dependent characteristics is used in modeling, and different from the mixed Gaussian model, some voice features and their distribution need to be done. For certain assumptions, the posterior probability of speech features is directly given. The triphonic state fully considers the context correlation of language, making the modeling unit more detailed, and the multiple hidden layers are more similar to the principle of the human speech perception system, which is conducive to the extraction of high-order feature information. In the embodiment of the present invention, the restricted Boltzmann algorithm is used to initialize the weights of the hidden layers of the network, and the weights can be updated later with the help of the reverse error propagation algorithm, which can effectively alleviate the difficulty of network pre-training. Risk of getting stuck in local extrema and further improving the modeling accuracy of acoustic models.

Those skilled in the art should further appreciate that the example modules and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the possibilities of hardware and software For interchangeability, in the above description, the composition and steps of each example have been generally described according to their functions. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may implement the described functionality using different methods for each particular application, but such implementation should not be considered as exceeding the scope of the present application.

The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be implemented by hardware, software modules executed by a processor, or a combination of both. Software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other Any other known storage medium.

It should be pointed out that the above are only preferred embodiments of the present invention, and are not intended to limit the implementation scope of the present invention. Those skilled in the art with professional knowledge can realize the present invention from the above implementation examples. Any changes, modifications and improvements made within are covered by the patent scope of the present invention. That is, the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or equivalent replacement without departing from the spirit and scope of the technical solution of the present invention.

Claims (6)

1. for a modeling method for the acoustic model of speech recognition, it is characterized in that, described method comprises:

A hidden Markov-mixed Gaussian HMM-GMM model is trained with training data, the modeling unit of this HMM-GMM model is the three-tone state of phonetic feature after phoneme decision tree-based clustering of described training data, described HMM-GMM model, by expecting that maximum EM Algorithm for Training obtains, obtains the state transition probability of described three-tone state simultaneously;

Based on described HMM-GMM model, pressure alignment is carried out to described training data phonetic feature, obtains the three-tone status information of described phonetic feature frame rank;

Pre-training is carried out to obtain the parameter of the weight of each hidden layer for initialization deep layer network to the deep-neural-network as described acoustic model;

Phonetic feature frame level status information based on described training data phonetic feature adopts error backpropagation algorithm to train described deep-neural-network, upgrades the weight of its each hidden layer.

2. modeling method as claimed in claim 1, it is characterized in that, described based on described HMM-GMM model, pressure alignment is carried out to the three-tone state of described training data phonetic feature, obtain described phonetic feature frame level status information, be specially: based on described HMM-GMM model, most probable to described training data phonetic feature and its three-tone state is carried out corresponding, obtain described phonetic feature frame level status information.

3. modeling method as claimed in claim 1, it is characterized in that, describedly pre-training is carried out to the deep-neural-network as described acoustic model be specially with the parameter of the weight obtaining each hidden layer for deep layer network described in initialization: utilize limited Boltzmann machine successively to train to convergence based on described training data, by the weight of each hidden layer of deep layer network described in the parameter initialization obtained.

4. for a modeling for voice recognition acoustic model, it is characterized in that, described modeling comprises:

First module, for training a hidden Markov-mixed Gaussian HMM-GMM model with training data, the modeling unit of this HMM-GMM model is the three-tone state of phonetic feature after phoneme decision tree-based clustering of described training data, and described HMM-GMM model is by expecting that maximum EM algorithm obtains the state transition probability of described three-tone state;

Second module, for based on described HMM-GMM model, carries out pressure alignment to described training data phonetic feature, obtains the three-tone status information of described phonetic feature frame level;

3rd module, for carrying out pre-training to obtain the parameter of the weight of each hidden layer for initialization deep layer network to the deep-neural-network as described acoustic model;

Four module, for adopting error backpropagation algorithm to train described deep-neural-network based on the phonetic feature frame level status information of described training data phonetic feature, upgrades the weight of its each hidden layer.

5. modeling as claimed in claim 4, it is characterized in that, described second module is based on described HMM-GMM model, pressure alignment is carried out to the three-tone state of described training data phonetic feature, obtain described phonetic feature frame level status information, be specially: most probable to described training data phonetic feature and its three-tone state, based on described HMM-GMM model, is carried out corresponding by described second module, obtain described phonetic feature frame level status information.

6. modeling as claimed in claim 4, it is characterized in that, described 3rd module is carried out pre-training to the deep-neural-network as described acoustic model and is specially with the parameter of the weight obtaining each hidden layer for deep layer network described in initialization: described 3rd module utilizes limited Boltzmann machine successively to train to convergence based on described training data, by the weight of each hidden layer of deep layer network described in the parameter initialization obtained.