CN111063335B - End-to-end tone recognition method based on neural network - Google Patents

Abstract

The invention discloses an end-to-end tone recognition method based on a neural network. The speech recognition acoustic model is trained on the set, and the start and end times of each syllable are obtained by forced alignment; the selected training speech data and the tone label of each syllable are sent to the end-to-end tone recognition model for training optimization, and the optimized Neural network model parameters; continuously adjust the neural network model parameters to select the optimal network model parameters; obtain test speech, under the condition of given sentence content text, use forced alignment to obtain the start and end time of each syllable; if not given, Use automatic speech recognition to obtain the start and end time of each syllable; send the selected training speech data and syllable time stamps to the end-to-end tone recognition model for recognition, and finally obtain the tone type of each syllable in each test data .

Description

An End-to-End Tone Recognition Method Based on Neural Networks

technical field

The invention relates to the field of Mandarin Chinese tone recognition, in particular to an end-to-end tone recognition method based on a neural network.

Background technique

With the rapid development of artificial intelligence technology, the research on speech technology is also deepening, including speech recognition, speech synthesis, speech separation, speech conversion and speaker recognition and other technical fields. In these fields, the pitch of speech is found through experiments It has a great influence on the experimental results of tonal languages. In Mandarin Chinese, there are five tones, namely Yinping, Yangping, Shangsheng, Qusheng and no tones. Two-tone up (/), three-tone bend (∨), four-tone down (\) and no tone. Tone is a very important part in Mandarin. If the tone is wrong, ambiguity will occur, which will lead to errors in speech comprehension. It is necessary to study the tone of Mandarin. Tone recognition is an important research direction in the field of speech. Its main purpose is to obtain the tones of tonal speech more accurately, and to improve the accuracy of tasks such as speech recognition and speech synthesis. The traditional tone recognition adopts the classical classification algorithm, that is, the front-end feature extraction and the back-end classifier. The traditional tone classification includes the acquisition of fundamental frequency features as features, and two separate stages for tone classification.

For the extraction method of fundamental frequency features, time domain analysis method, frequency domain analysis method or hybrid method can be adopted. Time domain analysis method includes autocorrelation method, average amplitude difference method, etc., and frequency domain analysis method includes cepstrum method. These methods are all artificially designed heuristic fundamental frequency extraction algorithms, and the settings of the algorithms are artificially set based on experimental phonetics. For the back-end tone classification model, when classifying tone, the classifier models in traditional pattern recognition are mainly used, such as support vector machine model, Gaussian mixture model, decision tree model, Gaussian mixture model-Hidden Markov model, conditional Random field model, or neural network model, etc.

The above tone classification models are classified into two categories: tone classification models based on frame features and tone classification models based on segment features. Tone classification models based on frame features include Gaussian mixture model-hidden Markov model, conditional random field model; tone classification models based on segment features include support vector machine model, Gaussian mixture model, decision tree model and so on. The tone classification model based on frame features can directly process the extracted fundamental frequency related features, and use the form of variable-length sequence as input to calculate the posterior probability of the tone model under the condition of the given input sequence for tone classification. The segment feature-based method can only deal with the input feature vector of fixed dimension, so it is necessary to first propose the feature sequence related to the fundamental frequency, and then use the artificial method to convert the fundamental frequency feature sequence into a fixed-dimensional observation vector containing the tone information, and then send it to the base frequency feature sequence. The classifier of segment features trains the tone model, and finally classifies the tone data to be tested according to the tone model to obtain correct recognition results, thereby completing the entire tone recognition process.

There are two main problems in the current tone classification technology:

On the one hand, the traditional fundamental frequency extraction method is not perfect enough, and the extracted fundamental frequency value is not accurate enough. The inaccuracy of this fundamental frequency extraction leads to inaccurate classification results in subsequent tone classification;

On the other hand, only using the fundamental frequency value obtained by the traditional fundamental frequency extraction method when classifying the tone, only using the artificially designed fundamental frequency feature cannot completely contain the information that is helpful for tone classification, and will also make the final tone classification. The results may not be optimal. The traditional tone recognition method is divided into two stages: feature extraction and classifier training. Each stage requires a lot of tuning parameters, which makes the two-stage tone recognition not necessarily able to achieve the overall optimal results.

SUMMARY OF THE INVENTION

The invention provides an end-to-end tone recognition method based on a neural network, which jointly learns the traditional fundamental frequency feature structure extracted first and the tone classification architecture later, performs end-to-end tone recognition, and realizes accurate classification of the tone. See the description below for details:

An end-to-end tone recognition method based on a neural network, the method comprising:

1. Training the tone recognition system model:

Build an end-to-end tone recognition model, and determine various hyperparameters required for the number of layers of the neural network and the number of hidden layer nodes;

Train a speech recognition acoustic model on the training set, using forced alignment to obtain the start and end times of each syllable;

The selected training speech data and the tone label of each syllable are sent to the end-to-end tone recognition model for training optimization, and the optimized neural network model parameters are obtained;

Constantly adjust the parameters of the neural network model and select the optimal network model parameters;

2. Tone recognition:

Obtain the test speech, and use forced alignment to obtain the start and end time of each syllable under the condition of given sentence content text; if not given, use automatic speech recognition to obtain the start and end time of each syllable;

The selected training speech data is sent to the end-to-end tone recognition model for identification, and finally the tone type of each syllable in each test data is obtained.

The method constructs a trainable deep neural network model, and then combines the fundamental frequency extraction neural network with the tone decoding neural network to form an end-to-end neural network tone classification model, and the two parts of the network parameters are trained and tuned simultaneously in the training phase. .

Wherein, the fundamental frequency extraction neural network is an encoder-decoder based on a recurrent neural network, and the network is divided into two parts: a fundamental frequency encoder network and a fundamental frequency decoder network.

Further, the base frequency encoder network uses a recurrent neural network to encode the speech, and the base frequency decoder network starts to predict the base frequency label from the last frame of the speech, and converts it into a trainable according to the predicted base frequency label. The fundamental frequency embedding vector of , predicts the fundamental frequency label of the previous moment through the fundamental frequency coding of the previous moment, until the fundamental frequency label of the first frame is predicted;

After predicting the tone label of each frame, it is converted into the fundamental frequency value sequence of the whole speech by using the pre-defined label and frequency correspondence.

Wherein, the tone decoding neural network is divided into two parts: a tone representation network and a label-related tone classification network;

The tone representation network maps the predicted fundamental frequency sequence into a vector of fixed dimension according to each syllable;

The tone classification network predicts the tone type of the current syllable according to the predicted label of the previous syllable and the fixed dimension vector of the current syllable.

Further, the tone classification network predicts the tone type of the current syllable according to the label predicted by the previous syllable and the fixed dimension vector of the current syllable is specifically:

First, according to the fixed dimension representation of the first syllable and the fixed dimension corresponding to the beginning of a sentence, it is spliced and sent to the tone classification network to predict the tone type of the first syllable;

Convert the predicted tone type of the first syllable to the corresponding tone label and then send it to the tone classification network with the fixed dimension representation of the second syllable as a joint input to obtain the tone type of the second syllable;

This cycle is repeated until the tone of the last syllable is predicted.

The beneficial effects of the technical scheme provided by the present invention are:

1. Compared with the traditional independent two-stage (basic frequency feature extraction and tone classification) tone recognition method, the present invention can reduce the shortage of artificially designed algorithms through the end-to-end joint method, so that the tone classification results can obtain better results. classification result;

2. The present invention improves the accuracy of Mandarin tone recognition, breaks the traditional frame of tone recognition divided into two stages (basic frequency feature extraction and tone classification), and constructs an end-to-end tone model. The end-to-end model can jointly train the two networks used in the two stages of traditional tone recognition: the feature extraction network and the tone classification network as a whole network. This method can bypass the manual design process, and the network of the entire model can be The parameters can be jointly optimized, so that the accuracy of Mandarin Chinese tone recognition can be obtained, which is suitable for tonal problems in tonal languages;

3. For the feature extraction network, the present invention uses an encoder-decoder fundamental frequency extraction network, which uses a recurrent neural network encoder to map the entire input sequence into a feature of a fixed dimension vector, and infer the fundamental frequency label corresponding to each frame in reverse time order according to this feature vector;

4. A label-related tone decoding network is adopted for the tone recognition network. When predicting the current tone label, the decoding network not only uses the fundamental frequency of the current syllable to extract the network features, but also uses the tone label value predicted by the previous tone. This makes it possible to take into account the influence of contextual tone types in tone recognition, and obtain better tone recognition results.

Description of drawings

Figure 1 is the overall framework diagram of the end-to-end tone recognition network;

Fig. 2 is a network diagram of fundamental frequency feature representation;

Figure 3 is a network diagram of tone embedding representation;

Figure 4 is a diagram of the tone prediction network.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the embodiments of the present invention are further described in detail below.

The invention identifies the tone category represented by the pronunciation of each syllable from the isolated or continuous speech stream of Mandarin, and proposes a fundamental frequency feature extraction network and tone label based on an encoder-decoder (Encoder-Decoder) based on a cyclic neural network. The end-to-end tone recognition method combined with the correlation recognition network forms a unified network model from the traditional fundamental frequency (Pitch or F0) feature acquisition architecture and tone classification architecture later, which realizes the need for no explicit extraction of fundamental frequency features. end-to-end tone recognition.

The function of tone recognition is to be able to identify the tones in the speech, and to obtain the tone information contained in the speech, so that the acquired tone information can meet the requirements of tasks such as speech recognition, speech synthesis, and speech conversion, so that the tasks can be realized more accurately. At the same time, for some non-native speakers who are learning a second tonal language, such as foreigners learning Chinese, tone recognition will help correct mistakes and improve learning efficiency.

The encoder-decoder network structure is proposed in the fundamental frequency feature extraction network to consider the accuracy of fundamental frequency extraction, the tone label-related decoding network is used in the tone recognition network, and finally the two networks are combined as a whole network , and jointly optimize the parameters. The method can make the result of tone recognition more accurate and more efficient, and can have a good effect in scientific research or daily application.

Example 1

In order to solve the above problem, the present invention adopts an end-to-end tone recognition method based on neural network, which converts the traditional two-stage classification problem of first extracting fundamental frequency related features and then carrying out tone classification into a single-stage network model for Joint tuning of parameters to achieve end-to-end tone recognition.

At present, end-to-end based methods have become the research hotspot of current artificial intelligence technology, such as end-to-end speech recognition, end-to-end speech synthesis, end-to-end speech conversion, etc. End-to-end technology can reduce the manual setting of experimental hyperparameters and reduce get better performance.

The invention overcomes the shortcomings of the separate design of the fundamental frequency extraction stage and the tone classification stage of the traditional two-stage tone recognition method. Replace the artificial heuristic algorithm of the traditional fundamental frequency extraction method with a data-driven trainable deep neural network model, and then combine the fundamental frequency extraction neural network with the tone decoding neural network, and finally form an end-to-end neural network tone classification Model.

The invention divides the end-to-end tone recognition deep neural network model into two sub-networks: a fundamental frequency extraction network part and a tone decoding network part. For the fundamental frequency extraction network, an encoder-decoder fundamental frequency extraction model network based on recurrent neural network is proposed. The network is divided into two parts: base frequency encoder network and base frequency decoder network. The baseband encoder network is a recurrent neural network that, given an input sequence, maps the input sequence to a sequence of hidden states of the same length. The baseband decoder network is a feedforward neural network. In order to predict the baseband label at the current moment, the baseband label at the next moment and the hidden state of the encoder at the current moment need to be used as the common input of the baseband decoder to determine the current moment. fundamental frequency label.

After obtaining the fundamental frequency label of the entire speech, it is converted into a sequence of continuous fundamental frequency values through the mapping relationship between the label and the fundamental frequency value. This mapping relationship can adopt a nonlinear mapping relationship between fixed fundamental frequency labels and fundamental frequency values, or a method called fundamental frequency embedding can be used to convert the predicted fundamental frequency labels into real numbers using a trainable fundamental frequency pool. Then the fundamental frequency value output by the fundamental frequency prediction network is sent to the tone decoder for decoding.

The tone decoder can use traditional deep feedforward neural network, recurrent neural network or convolutional neural network. The pitch decoder predicts the pitch of each syllable in a sentence based on the output of the fundamental frequency extraction network. The present invention proposes a tone decoding network based on tone label correlation. The network is divided into two parts: the tone representation network and the label-dependent tone classification network. The tone representation network maps the variable-length fundamental frequency sequence corresponding to each syllable of the fundamental frequency extraction network into a fixed-dimensional vector. The label-dependent tone classification network predicts the type of the current tone based on the fixed-dimensional representation of the current syllable and the label type of the previous syllable.

The specific implementation process is that each tone classification network predicts the label of each syllable in turn according to the order of each syllable: first, according to the fixed dimension of the first syllable, the fixed dimension corresponding to a BOS (sentence start) is spliced and sent into the tone The classification network predicts the tone type of the first syllable; according to the predicted tone type of the first syllable, it is converted into the corresponding tone label and then sent to the tone classification network with the fixed dimension representation of the second syllable as joint input to obtain the second The tone type of the syllable, and so on, until the tone of the last syllable is predicted.

Aiming at the tone recognition of Mandarin Chinese, the invention improves the ability of automatically classifying the tone of each syllable in the speech from the original audio.

The description of the present invention is used to obtain the pitch information contained in the speech, so that the obtained pitch information can meet the requirements of tasks such as speech recognition, speech synthesis, and speech conversion, so that the tasks can be realized more accurately. Tone recognition For some non-native speakers of a second tonal language, such as foreigners learning Chinese, tone recognition will help correct pronunciation errors and improve the learning effect of the second language. The method can make the result of tone recognition more accurate than the traditional method, and can have a good effect in scientific research or daily application.

The technology of the invention overcomes the shortcomings of the traditional two-stage tone classification method, adopts a data-driven neural network to replace the traditional fundamental frequency extraction algorithm, and then combines the fundamental frequency extraction network and the back-end tone recognition network to form a unified On the training data, the parameters of the entire network are jointly tuned to obtain better tone recognition results. An encoder-decoder based extraction network is designed for the fundamental frequency extraction network, which can better extract fundamental frequency features. A context-dependent tone prediction network is designed for the tone decoding network, which can better perform tone recognition.

Example 2

The scheme in embodiment 1 is further introduced below in conjunction with specific examples, and is described in detail below:

1. Training the tone recognition system model:

Step 1: Select a certain amount of Mandarin speech data as the training data (also referred to as samples) of the tone model;

Step 2: Build an acoustic model for training automatic speech recognition, and use the forced alignment method to obtain the start and end time of each syllable under the condition of given sentence content and text;

Step 3: Build an end-to-end tone recognition model, and determine various hyperparameters required for the number of layers of the neural network and the number of hidden layer nodes;

Step 4: Perform necessary preprocessing on the data according to the selected input of the end-to-end tone recognition model;

Step 5: Send the selected training speech data into the constructed end-to-end tone recognition model for training and optimization, and obtain the optimized neural network model parameters, wherein the training speed depends on the configuration of the machine and the scale of the training data;

Step 6: Continuously adjust the neural network parameters of the tone classification model, and continuously observe the results of the training model, select the optimal network model parameters, and save the trained tone classification model neural network parameters.

2. Tone recognition:

Step 1: Obtain the test speech. Under the condition of given sentence content text, use the forced alignment method to obtain the start and end time of each syllable; under the condition of no sentence content text, use automatic speech recognition method to obtain each syllable. the start and end times of the syllables;

Step 2: Perform necessary preprocessing on the data according to the selected input of the end-to-end tone recognition model;

Step 3: The selected training speech data is sent to the end-to-end tone recognition model with trained parameters for identification, and finally the tone type of each syllable in each test data is obtained.

Example 3

The schemes in Embodiments 1 and 2 are further introduced below in conjunction with specific examples and calculation formulas, and are described in detail below:

(1) Training an acoustic model for speech recognition

Before performing the pitch model task, speech data is required to train a speech recognition acoustic model based on Gaussian Mixture Model-Hidden Markov Model (GMM-HMM) or Deep Neural Network-Hidden Markov Model (DNN-HMM). On the training data, given the pronunciation label of each speech, the speech recognition acoustic model is used to obtain the start and end time of each syllable corresponding to the speech input using the forced alignment method in speech recognition technology. On the test data, when the annotation content text of the sentence is not given, the method of speech recognition and decoding is used to obtain the syllable text corresponding to the speech input and the start and end time of each syllable. The phoneme boundary information of the phoneme segment can be obtained from the aligned phoneme segments, which is used as the boundary basis for tone recognition and classification.

(2) Preprocessing network

The input is the original waveform of the speech, and the fundamental frequency feature extraction network can use the original waveform sampling. At this time, the input adopts 1024 speech samples. The input can also be normalized cross-correlation function coefficients calculated by the normalized cross-correlation function.

The calculation method is: in the fth frame, intercept a window sequence w _f from the speech sequence and normalize the window, and then intercept a subsequence v _f,l of length n from w _f , where l is the time delay. Index, representing the offset of v _f,l in w _f . Different normalized cross-correlation function coefficients are calculated according to different time lag indices l.

The normalized cross-correlation function coefficients are calculated using the following formula:

Among them, v _f,0 is v _f,l with time lag index l=0, and A is a penalty factor set by human experience.

(3) Training the fundamental frequency feature extraction network

The pitch frequency is extracted by artificial labeling or traditional pitch frequency extraction algorithm, and the N fundamental frequency states corresponding to the fundamental frequency value are used as the training labels of the network for fundamental frequency feature extraction. Given the input value of the base frequency network from the 1st frame to the Fth frame, the encoder RNN of the base frequency network recurrent neural network performs forward calculation, and the output of the hidden layer (h ₁ , h ₂ obtained when the calculation reaches the F frame ,...,h _F ):

h _f =σ(Wx _f +Vh _f-1 +b) (2)

Among them, σ is the Sigmoid function; W is the transformation matrix of x _f ; x _f is the input of the current f-th frame of the recurrent neural network; V is the transformation matrix of h f- ₁ ; h _f-1 is the f-1th frame The output of the hidden layer; b is the bias vector.

映射为基频表示向量

基频提取网络根据f+1时刻预测出的基频标签

经过基频嵌入表示池转换为对应的基频嵌入

再与当前时刻循环网络隐含层输出h_f进行拼接，再通过softmax层获得f帧的基频输出标签：In the decoding stage, there is a module in the base frequency extraction network called base frequency embedding pool. The fundamental frequency label that the embedding pool will predict

Mapping to fundamental frequency representation vector

The fundamental frequency label predicted by the fundamental frequency extraction network according to the time f+1

After the base frequency embedding representation pool is converted to the corresponding base frequency embedding

It is then spliced with the output h _f of the hidden layer of the recurrent network at the current moment, and then the base frequency output label of the f frame is obtained through the softmax layer:

where Z(·) represents an affine transformation. In the F-th frame, in order to calculate the fundamental frequency embedded representation at time F+1, the fundamental frequency label at F+1 is required according to formula (3), which exceeds the maximum range of the fundamental frequency label of the sentence. The +1 frame adopts an EOS (end of sentence) label, which corresponds to the maximum number of base frequency labels, and takes the entire base frequency to represent the last one in the embedding pool. After the base frequency label of the F-th frame is predicted according to the above steps, the embedded representation of the F-th frame is found according to the look-up table of the base-frequency embedded representation pool, and then the embedded representation of the predicted label of the F-th frame is compared with the forward direction at time F-1. The hidden layer of the network outputs h _F-1 for splicing, and then sends it to the softmax layer according to formula (3) to obtain the fundamental frequency label at the time of F-1, and iterates in turn until it goes back to time F-2, time F-3, ..., time f. , until time 1.

转换为

以及f帧的隐含状态h_f预测f时刻的基频标签

When training the above fundamental frequency extraction network, a teacher forcing method is used to train the convergence speed and training effect. The teacher-forced method refers to using Equation (4) to replace Equation (3) during training with the pitch frequency labels of the f+1 frames that are actually annotated

convert to

and the hidden state h _f of the f frame predicts the fundamental frequency label at time f

In the implementation process, direct training and teacher-forced methods are used for random selection, and the random coefficient is set based on experience, in this case, it is set to 0.5. In the training phase, given the waveform input of the entire sentence, the posterior probability of the fundamental frequency label at each moment f is output through the network:

According to the fundamental frequency of each frame, the mutual entropy is used as the objective function, and the network parameters are optimized according to the stochastic gradient descent method using the back-propagation algorithm.

When extracting the fundamental frequency feature based on the trained fundamental frequency extraction neural network, the fixed network parameters remain unchanged, and the original or normalized cross-correlation function coefficients of the audio waveform are input to predict the fundamental frequency label value of each time frame. The corresponding relationship obtains the actual fundamental frequency prediction value.

(4) Tone Decoding and Recognition Network

The tone decoding and recognition network is shown in Fig. 4, which is a forward decoding network related to tone labels. The network consists of two parts: a representation network for syllable tone embedding and a context-dependent tone decoding network. The feature of the syllable tone embedding representation network is that the fundamental frequency output by the fundamental frequency extraction network is converted into a fixed-dimensional vector of the variable-length fundamental frequency sequence of each syllable according to the boundary information of each syllable. The output of the fundamental frequency extraction network is extracted according to the boundary information of each syllable, and then spliced according to the 9 frames before and after, and sent to the syllable tone embedding representation network to obtain the embedded representation of each syllable with a fixed dimension.

The feature of the context-dependent tone decoding network is to use the traditional fundamental frequency extraction feature or the fundamental frequency value predicted by the fundamental frequency extraction neural network to predict the tone label of each syllable in the speech. The tone embedding representation pool includes vector representations corresponding to 6 tone labels, of which 5 embedding vectors represent 5 Chinese tones, and the other embedding vector represents the tones at the beginning of sentences. Among them, the tone embedding pool is also optimized and trained in the back-propagation algorithm.

与当前音节的嵌入表示

拼接送入声调分类网络进行分类预测得到第1个音节的声调类型

将第1个声调类型对应的声调嵌入向量

再与第2个音节的嵌入

表示相连接送入声调预测网络得到第2个音节的声调标签

依次预测第3个，直至最后一个音节为止。When predicting the tone type of the beginning (1st) syllable, embed the beginning tone into the vector

with the embedded representation of the current syllable

The splicing is sent to the tone classification network for classification and prediction to obtain the tone type of the first syllable

Embed the tone corresponding to the first tone type into the vector

Again with the embedding of the 2nd syllable

Indicates the tone label of the second syllable obtained by connecting the input to the tone prediction network

Predict the 3rd one in sequence until the last syllable.

(5) End-to-end tone recognition neural network model

The end-to-end tone recognition neural network model connects the fundamental frequency extraction network and the tone classification network to form a total tone recognition network, and the network parameters are optimized at the same time. The fundamental frequency labels of 9 frames before and after the current frame output by the fundamental frequency extraction network are converted into the fundamental frequency embeddings of 9 frames by searching N fundamental frequency embedding pools as the input of the tone model. Or use the weighted fundamental frequency value to perform 9-frame splicing as the back-end method, and obtain better tone recognition results than partial parameter tuning through overall parameter tuning.

To sum up, the advantages of the present invention improve the accuracy of Mandarin tone recognition, reduce the calculation time, break the traditional frame of tone recognition divided into two stages (fundamental frequency feature extraction and tone classification), and construct a An end-to-end tone model. The end-to-end model can jointly optimize the feature extraction stage and the classification stage as a whole network to obtain the accuracy of Mandarin Chinese tone recognition. The network model has good robustness and is suitable for tonal research in tonal languages.

In the embodiment of the present invention, the models of each device are not limited unless otherwise specified, as long as the device can perform the above functions.

Those skilled in the art can understand that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the above-mentioned serial numbers of the embodiments of the present invention are only for description, and do not represent the advantages or disadvantages of the embodiments.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

Claims (3)

1. An end-to-end tone recognition method based on a neural network, the method comprising:

firstly, training a tone recognition system model:

constructing an end-to-end tone recognition model, and determining parameters required by the number of layers of a neural network, the number of nodes of a hidden layer and the like;

training a voice recognition acoustic model on a training set, and acquiring the starting time and the ending time of each syllable by using forced alignment;

sending the selected training voice data and the tone label of each syllable into an end-to-end tone recognition model for training optimization to obtain optimized neural network model parameters;

continuously adjusting parameters of the neural network model, and selecting optimal parameters of the network model;

secondly, tone recognition:

obtaining a test voice, and obtaining the starting time and the ending time of each syllable by using forced alignment under the condition of a given sentence content text; when not given, obtaining the start and end time of each syllable using automatic speech recognition;

sending the selected test voice data into an end-to-end tone recognition model for recognition, and finally obtaining the tone type of each syllable in each test data;

the method comprises the steps of constructing a trainable deep neural network model, combining a fundamental frequency extraction neural network with a tone decoding neural network to form an end-to-end neural network tone classification model, and training and optimizing network parameters of the two parts at the training stage;

the fundamental frequency extraction neural network is an encoder-decoder based on a cyclic neural network, and the network is divided into a fundamental frequency encoder network and a fundamental frequency decoder network;

the base frequency encoder network encodes the voice by utilizing a recurrent neural network, the base frequency decoder network predicts a base frequency label from the last frame of the voice, converts the predicted base frequency label into a trainable base frequency embedding vector according to the predicted base frequency label, and determines the base frequency label at the current time by taking the base frequency label at the next time and the encoder hidden state at the current time as the common input of a base frequency decoder until the base frequency label of the first frame is predicted;

after the base frequency label of each frame is predicted, the mapping relation between the label and the base frequency value defined in advance is used for converting the base frequency label into the base frequency value sequence of the whole voice.

2. The end-to-end tone recognition method based on the neural network as claimed in claim 1, wherein the tone decoding neural network is divided into two parts: tone representation networks and label-dependent tone classification networks;

the tone representation network maps the predicted fundamental frequency value sequence into a vector with fixed dimensionality according to each syllable;

the tone classification network predicts the tone type of the current syllable based on the tone label predicted for the last syllable and the fixed dimensional vector of the current syllable.

3. The end-to-end tone recognition method based on neural network as claimed in claim 2, wherein the tone classification network predicts the tone type of the current syllable according to the tone label predicted by the previous syllable and the fixed dimension vector of the current syllable, specifically:

firstly, according to the fixed dimension representation of the 1 st syllable, splicing with the fixed dimension corresponding to the beginning of a sentence, and sending the fixed dimension representation and the fixed dimension representation into a tone classification network to predict the tone type of the 1 st syllable;

converting the predicted tone type of the 1 st syllable into a corresponding tone label and then sending the corresponding tone label and the fixed dimension representation of the 2 nd syllable as combined input into a tone classification network to obtain the tone type of the 2 nd syllable;

this is repeated until the tone of the last syllable is predicted.

Images