CN104700828B - The construction method of depth shot and long term memory Recognition with Recurrent Neural Network acoustic model based on selective attention principle - Google Patents

Abstract

A method for constructing an acoustic model of deep long-term short-term memory recurrent neural network based on the principle of selective attention. By adding attention gate units to the acoustic model of deep long-term short-term memory recurrent neural network, it can represent the instantaneous functional changes of neurons in the auditory cortex. The difference between the gate unit and other gate units is that other gate units have one-to-one correspondence with the time series, while the attention gate unit reflects the short-term plasticity effect, so there are gaps in the time series; The above-mentioned neural network acoustic model obtained by data training can realize robust feature extraction and construction of a robust acoustic model for Cross-talk noise, and improve the robustness of the acoustic model by suppressing the impact of non-target flow on feature extraction The purpose of this method; this method can be widely used in various machine learning fields such as speaker recognition, keyword recognition, and human-computer interaction involving speech recognition.

Description

Acoustic model of deep long short-term memory recurrent neural network based on the principle of selective attention type build method

technical field

The invention belongs to the field of audio technology, in particular to a method for constructing an acoustic model of a deep long-term short-term memory cyclic neural network based on the principle of selective attention.

Background technique

With the rapid development of information technology, speech recognition technology has the conditions for large-scale commercialization. At present, speech recognition mainly uses continuous speech recognition technology based on statistical models, and its main goal is to find the word sequence with the highest probability represented by a given speech sequence. The task of the continuous speech recognition system based on the statistical model is to find the most probable word sequence represented by the given speech sequence, usually including the construction of acoustic models and language models and their corresponding search and decoding methods. With the rapid development of acoustic models and language models, the performance of speech recognition systems has been greatly improved in an ideal acoustic environment. The existing Deep Neural Network-Hidden Markov Model (DNN-HMM ) is initially mature, and effective features can be automatically extracted through machine learning methods, and context information corresponding to multi-frame speech can be modeled, but each layer of this type of model has millions of parameters, and the input of the next layer It is the last output, so it is necessary to use GPU equipment to train the DNN acoustic model, and the training time is long; the highly nonlinear and parameter sharing characteristics also make it difficult for DNN to perform parameter adaptation.

Recurrent Neural Network (RNN) is a neural network in which there are directed cycles between units to express the dynamic time characteristics of the network. It is widely used in handwriting recognition and language models. Speech signals are complex time-varying signals with complex correlations on different time scales. Therefore, compared with deep neural networks, the recurrent connection function of recurrent neural networks is more suitable for processing such complex time series data.

As a type of recurrent neural network, the Long Short-Term Memory (LSTM) model is more suitable than the recurrent neural network for processing and predicting long-term sequences with event lags and uncertain times. The deep LSTM-RNN acoustic model proposed by the University of Toronto, which adds a memory block, combines the multi-level representation capabilities of deep neural networks with the ability of recurrent neural networks to flexibly use long-span contexts, making phoneme recognition errors based on the TIMIT library The rate fell to 17.1%.

However, the gradient descent method used in the cyclic neural network has the problem of gradient dissipation (vanishing gradient), that is, in the process of adjusting the weight of the network, as the number of network layers increases, the gradient dissipates layer by layer, resulting in its effect on weight adjustment. The effect is getting smaller and smaller. The two-layer deep LSTM-RNN acoustic model proposed by Google adds a linear recurrent projection layer (Recurrent Projection Layer) to the previous deep LSTM-RNN model to solve the gradient dissipation problem. Comparative experiments show that the Frame Accuracy and its convergence speed of RNN are obviously inferior to those of LSTM-RNN and DNN; in terms of word error rate and its convergence speed, the word error rate of the best DNN after several weeks of training is 11.3%; while the word error rate of the two-layer deep LSTM-RNN model was reduced to 10.9% after training for 48 hours, and after 100/200 hours of training, the word error rate was reduced to 10.7/10.5(%).

The Deep Bidirectional Long Short-Term Memory Recurrent Neural Networks (DBLSTM-RNN) acoustic model proposed by the University of Munich defines mutually independent forward and backward layers in each cyclic layer of the neural network. Layer, and use multiple hidden layers to perform higher-level representation of the input acoustic features, and perform supervised learning on noise and reverberation to achieve feature projection and enhancement. This method achieves a word error rate reduction from 55% of the baseline to 22% in the SNR [-6dB, 9dB] range on the 2013 PASCAL CHiME dataset.

However, the complexity of the actual acoustic environment still seriously affects and interferes with the performance of the continuous speech recognition system. Even if the current mainstream DNN acoustic model method is used, the continuous speech recognition data set under complex environmental conditions including noise, music, spoken language, repetition, etc. It can only obtain a recognition rate of about 70%, and the noise resistance and robustness of the acoustic model in the continuous speech recognition system still need to be improved.

With the rapid development of acoustic models and language models, the performance of speech recognition systems has been greatly improved in an ideal acoustic environment. The existing DNN-HMM model is initially mature, and effective features can be automatically extracted through machine learning methods, and can be used for Contextual information modeling for multi-frame speech. However, most recognition systems are still very sensitive to changes in the acoustic environment, especially under the interference of cross-talk noise (two or more people talking at the same time), which cannot meet the requirements of practical performance. Compared with the deep neural network acoustic model, there is a directed cycle between the units in the recurrent neural network acoustic model, which can effectively describe the dynamic time characteristics inside the neural network, and is more suitable for processing speech data with complex timing. The long-short-term memory neural network is more suitable than the recurrent neural network for processing and predicting long-term sequences with event lags and uncertain times, so the acoustic model used to build speech recognition can achieve better results.

The human brain has the phenomenon of selective attention when processing speech in complex scenes. The main principle is that the human brain has the ability of auditory selective attention. In the auditory cortex area, the top-down control mechanism is used to suppress non-target flow. And the purpose of enhancing the target flow. Studies have shown that in the process of selective attention, the short-term plasticity (Short-Term Plasticity) effect in the auditory cortex increases the ability to distinguish sounds. During intense attention, reinforcement processing of sound objects can begin within 50 milliseconds in the primary auditory cortex.

Contents of the invention

In order to overcome the above-mentioned shortcoming of the prior art, the object of the present invention is to provide a kind of construction method of deep long short-term memory recurrent neural network acoustic model based on selective attention principle, set up the deep long short-term memory recurrent neural network acoustic model based on selective attention principle The network acoustic model, by adding the attention gate unit in the deep long short-term memory recurrent neural network acoustic model, to represent the instantaneous functional changes of auditory cortex neurons, the difference between the attention gate unit and other gate units is that other gate units and time series One-to-one correspondence, while the attention gate unit reflects the short-term plasticity effect, so there is an interval in the time series. The above-mentioned neural network acoustic model obtained by training a large amount of speech data containing Cross-talk noise can realize robust feature extraction and construction of a robust acoustic model for Cross-talk noise. Influence can achieve the purpose of improving the robustness of the acoustic model.

In order to achieve the above object, the technical scheme adopted in the present invention is:

A continuous speech recognition method based on the principle of selective attention, comprising the steps of:

The first step is to construct a deep long short-term memory recurrent neural network based on the principle of selective attention

From the input to the hidden layer is defined as a long-term short-term memory recurrent neural network, and the depth refers to the output of each long-term short-term memory recurrent neural network as the input of the next long-term short-term memory recurrent neural network, so repeated, the last long-term short-term memory recurrent neural network The output of the neural network is the output of the whole system; in each long-short-term memory recurrent neural network, the speech signal x _t is the input at time t, x _t-1 is the input at time t-1, and so on, the total time length The input above is x=[x ₁ ,...,x _T ] where t∈[1,T], T is the total time length of the speech signal; the long short-term memory recurrent neural network at time t consists of attention gate, input gate , output gate, forget gate, memory cell, tanh function, hidden layer, multiplier, the long short-term memory cycle neural network at t-1 time is composed of input gate, output gate, forget gate, memory cell, tanh function, hidden layer, Composed of multipliers; the hidden layer output on the total time length is y=[y ₁ ,...,y _T ];

The second step is to construct a deep long short-term memory recurrent neural network acoustic model based on the principle of selective attention

On the basis of the first step, there is an attention gate in the deep long-term short-term memory recurrent neural network corresponding to each interval s, and there is no attention gate in the deep long-term short-term memory recurrent neural network at other times, that is, the depth long-term short-term memory recurrent neural network based on the principle of selective attention has no attention gate. The short-term memory recurrent neural network acoustic model consists of a deep long-short-term memory recurrent neural network with interval-existing attention gates.

How to recognize in a complex environment, especially under the interference of cross-talk noise, has always been one of the difficulties in speech recognition, which hinders the large-scale application of speech recognition. Compared with the prior art, the present invention learns from the phenomenon of selective attention in the human brain when processing speech in complex scenes to suppress non-target flow and enhance target flow. The gate unit is used to represent the instantaneous functional changes of neurons in the auditory cortex. The difference between the attention gate unit and other gate units is that the other gate units correspond to the time series one by one, while the attention gate unit reflects the short-term plasticity effect, so in time There are gaps in the sequence. Using this method on some continuous speech recognition datasets containing Cross-talk noise can achieve better performance than the deep neural network method.

Description of drawings

Fig. 1 is the flowchart of deep long short-term memory recurrent neural network based on selective attention principle of the present invention.

Fig. 2 is a flow chart of the deep long short-term memory neural network acoustic model based on the selective attention principle of the present invention.

detailed description

The implementation of the present invention will be described in detail below in conjunction with the drawings and examples.

The invention realizes continuous speech recognition by utilizing a deep long-short-term memory cycle neural network acoustic model based on the principle of selective attention. However, the models and methods provided by the present invention are not limited to continuous speech recognition, and can also be any methods and devices related to speech recognition.

The present invention mainly comprises the steps:

The first step is to construct a deep long short-term memory recurrent neural network based on the principle of selective attention

As shown in Figure 1, input 101 and input 102 are voice signal input x _t and x _t-1 (t ∈ [1, T], T is the total time length of voice signal) at t moment and t-1 moment; The long short-term memory recurrent neural network is made up of attention gate 103, input gate 104, forget gate 105, memory cell 106, output gate 107, tanh function 108, tanh function 109, hidden layer 110, multiplier 122 and multiplier 123; The long-short-term memory recurrent neural network at time -1 consists of an input gate 112 , a forgetting gate 113 , a memory cell 114 , an output gate 115 , a tanh function 116 , a tanh function 117 , a hidden layer 118 , a multiplier 120 and a multiplier 121 . The output of the hidden layer at time t and time t-1 is output 111 and output 119 respectively.

Among them, the input 102 is simultaneously used as the input of the input gate 112, the forget gate 113, the output gate 115 and the tanh function 116, the output of the input gate 112 and the output of the tanh function 116 are sent to the multiplier 120, and the output after the operation is used as the output of the memory cell 114. Input, the output of the memory cell 114 is used as the input of the tanh function 117, the output of the tanh function 117 and the output of the output gate 115 are sent to the multiplier 121, and the output after the operation is used as the input of the hidden layer 118, and the output of the hidden layer 118 is the output 119.

The output of the input 101, the memory cell 114 and the output of the multiplier 121 are jointly used as the input of the attention gate 103, the output of the attention gate 103 and the output of the multiplier 121 are jointly used as the input of the tanh function 108, the output of the attention gate 103, the memory cell 114 The output of the output of the multiplier 121 and the output of the multiplier 121 are jointly used as the input of the input gate 104, the forget gate 105 and the output gate 107 respectively, the output of the forget gate 105 and the output of the memory cell 114 are sent to the multiplier 124, the output of the input gate 104 and the tanh function The output of 108 is sent to multiplier 122, and the output of multiplier 124 and the output of multiplier 122 are as the input of memory cell 106, and the output of memory cell 106 is as the input of tanh function 109, the output of tanh function 109 and the output of output gate 107 The output of the multiplier 123 is used as the input of the hidden layer 110, and the output of the hidden layer 110 is the output 111.

That is: the parameters at time t∈[1,T] are calculated according to the following formula:

G _{atten_t} ＝sigmoid(W _ax x _t +W _am m _t-1 +W _ac Cell _t-1 +b _a )

G _{input_t} ＝sigmoid(W _ia G _{atten_t} +W _im m _t-1 +W _ic Cell _t-1 +b _i )

G _{forget_t} ＝sigmoid(W _fa G _{atten_t} +W _fm m _t-1 +W _fc Cell _t-1 +b _f )

Cell _t ＝G _{forget_t} ⊙Cell _t-1 +G _{input_t} ⊙tanh(W _ca G _{atten_t} +W _cm m _t-1 +b _c )

G _{output_t} ＝sigmoid(W _oa G _{atten_t} +W _om m _t-1 +W _oc Cell _t-1 +b _o )

m _t ＝G _{output_t} ⊙tanh(Cell _t )

y _t ＝softmax _k (W _ym m _t +b _y )

Wherein G _{atten_t} is the output of the attention gate 103 at the time t, G _{input_t} is the output of the input gate 104 at the time t, G _{forget_t} is the output of the forgetting gate 105 at the time t, Cell _t is the output of the memory cell 106 at the time t, and G _{output_t} is the time at t The output of output gate 107, m _t is the input of hidden layer 110 at time t, y _t is the output 111 at time t; x _t is the input 101 at time t, m _t-1 is the input of hidden layer 118 at time t-1, Cell _t-1 is the output of the memory cell 114 at time t-1; W _ax is the weight between the attention gate a at time t and the input x at time t; W _am is the attention gate a at time t and the hidden layer input m at time t-1 W _ac is the weight between attention gate a at time t and memory cell c at time t-1, W _ia is the weight between input gate i at time t and attention gate a at time t, W _im is time t The weight between input gate i and hidden layer input m at time t-1, W _ic is the weight between input gate i at time t and memory cell c at time t-1, W _fa is the forgetting gate f at time t and attention at time t The weight between gate a, W _fm is the weight between the forgotten gate f at time t and the hidden layer input m at time t-1, W _fc is the weight between the forgotten gate f at time t and memory cell c at time t-1, W _ca is the weight between memory cell c at time t and attention gate a at time t, W _cm is the weight between memory cell c at time t and hidden layer input m at time t-1, W _oa is the output gate o and gate a at time t Pay attention to the weight between the gate a at time t, W _om is the weight between the output gate o at time t and the hidden layer input m at time t-1, W _oc is the weight between the output gate o at time t and memory cell c at time t-1 b _a is the deviation of attention gate a, b _i is the deviation of input gate i, b _f is the deviation of forgetting gate f, b _c is the deviation of memory cell c, b _o is the deviation of output gate o Deviation, b _y is the deviation of the output y, and different b represents different deviations; and there are where x _k represents the input of the k∈[1,K]th softmax function, l∈[1,K], express to all Sum; ⊙ stands for matrix element multiplication.

The second step is to construct a deep long short-term memory recurrent neural network acoustic model based on the principle of selective attention

On the basis of the first step, there is an attention gate in the deep long-short-term memory recurrent neural network corresponding to each interval s (s=5), and there is no attention gate in the deep long-term short-term memory recurrent neural network at other moments, that is, based on selectivity Attention-based deep long-term short-term memory recurrent neural network The acoustic model consists of a deep long-term short-term memory recurrent neural network with attention gates at intervals. As shown in Figure 2, the established acoustic model of the deep long-term short-term memory recurrent neural network based on the principle of selective attention, the deep long-term short-term memory recurrent neural network at time t has an attention gate 201, and the deep long-term short-term memory recurrent neural network at time t-s There is an attention gate 202, and so on.

Claims (2)

1. A method for building a deep long-short-term memory recurrent neural network acoustic model based on the principle of selective attention, comprising the steps of: The first step is to construct a deep long short-term memory recurrent neural network based on the principle of selective attention From the input to the hidden layer is defined as a long-term short-term memory recurrent neural network, and the depth refers to the output of each long-term short-term memory recurrent neural network as the input of the next long-term short-term memory recurrent neural network, so repeated, the last long-term short-term memory recurrent neural network The output of the neural network is the output of the whole system; in each long-short-term memory recurrent neural network, the speech signal x _t is the input at time t, x _t-1 is the input at time t-1, and so on, the total time length The input above is x=[x ₁ ,...,x _T ] where t∈[1,T], T is the total time length of the speech signal; the long short-term memory recurrent neural network at time t consists of attention gate, input gate , output gate, forget gate, memory cell, tanh function, hidden layer, multiplier, the long short-term memory cycle neural network at t-1 time is composed of input gate, output gate, forget gate, memory cell, tanh function, hidden layer, Composed of multipliers; the hidden layer output on the total time length is y=[y ₁ ,...,y _T ]; The parameters at time t∈[1,T] are calculated according to the following formula: G _{atten_t} ＝sigmoid(W _ax x _t +W _am m _t-1 +W _ac Cell _t-1 +b _a ) G _{input_t} ＝sigmoid(W _ia G _{atten_t} +W _im m _t-1 +W _ic Cell _t-1 +b _i ) G _{forget_t} ＝sigmoid(W _fa G _{atten_t} +W _fm m _t-1 +W _fc Cell _t-1 +b _f ) Cell _t ＝G _{forget_t} ⊙Cell _t-1 +G _{input_t} ⊙tanh(W _ca G _{atten_t} +W _cm m _t-1 +b _c ) G _{output_t} ＝sigmoid(W _oa G _{atten_t} +W _om m _t-1 +W _oc Cell _t-1 +b _o ) m _t ＝G _{output_t} ⊙tanh(Cell _t ) y _t ＝softmax _k (W _ym m _t +b _y ) Where G _{atten_t} is the output of the attention gate at time t, G _{input_t} is the output of the input gate at time t, G _{forget_t} is the output of the forget gate at time t, Cell _t is the output of the memory cell at time t, and G _{output_t} is the output of the output gate at time t , m _t is the input of the hidden layer at time t, y _t is the output at time t; x _t is the input at time t, m _t-1 is the input of the hidden layer at time t-1, Cell _t-1 is the time of t-1 The output of the memory cell; W _ax is the weight between the attention gate a at time t and the input x at time t, W _am is the weight between the attention gate a at time t and the input m of the hidden layer at time t-1, W _ac is the weight at time t Note the weight between gate a and memory cell c at time t-1, W _ia is the weight between input gate i at time t and attention gate a at time t, W _im is the hidden layer between input gate i and time t-1 at time t The weight between input m, W _ic is the weight between input gate i at time t and memory cell c at time t-1, W _fa is the weight between forgetting gate f at time t and attention gate a at time t, W _fm is The weight between the forgotten gate f at time t and the hidden layer input m at time t-1, W _fc is the weight between the forgotten gate f at time t and the memory cell c at time t-1, W _ca is the memory cell c and t at time t Always pay attention to the weight between the gate a, W _cm is the weight between the memory cell c at the time t and the input m of the hidden layer at the time t-1, W _oa is the weight between the output gate o at the time t and the attention gate a at the time t, W _om is the weight between the output gate o at time t and the hidden layer input m at time t-1, W _oc is the weight between the output gate o at time t and the memory cell c at time t-1; b _a is the weight of the attention gate a Bias, b _i is the deviation of input gate i, b _f is the deviation of forgetting gate f, b _c is the deviation of memory cell c, b _o is the deviation of output gate o, b _y is the deviation of output y amount, different b represents different deviations; and there are where x _k represents the input of the k∈[1,K]th softmax function, l∈[1,K], express to all Sum; ⊙ represents matrix element multiplication; The second step is to construct a deep long short-term memory recurrent neural network acoustic model based on the principle of selective attention On the basis of the first step, there is an attention gate in the deep long-term short-term memory recurrent neural network corresponding to each interval s, and there is no attention gate in the deep long-term short-term memory recurrent neural network at other times, that is, the depth long-term short-term memory recurrent neural network based on the principle of selective attention has no attention gate. The short-term memory recurrent neural network acoustic model consists of a deep long-short-term memory recurrent neural network with interval-existing attention gates. 2. according to the construction method of the deep long-short-term memory recurrent neural network acoustic model based on the selective attention principle of claim 1, it is characterized in that, said s=5.