CN115359783A - Phoneme recognition method and device, electronic equipment and storage medium - Google Patents

Abstract

The present invention provides a phoneme recognition method, device, electronic equipment and storage medium. The method includes: determining the speech to be recognized; inputting the speech to be recognized into a phoneme recognition model to obtain the phoneme recognition result output by the phoneme recognition model; the phoneme recognition model The first recognition model is obtained by training the first recognition model based on the sample speeches of multiple languages and the phoneme-level labels of each sample speech. The first recognition model is based on the similarity between the phonemes corresponding to each phoneme node under the second recognition model. The second recognition model is obtained by screening phoneme nodes under the second recognition model, and the second recognition model includes phoneme nodes corresponding to multiple languages. The phoneme recognition method, device, electronic equipment and storage medium provided by the present invention not only reduce the scale of the phoneme recognition model, but also the phoneme recognition model can accurately distinguish phonemes of different languages.

Description

Phoneme recognition method, device, electronic device and storage medium

technical field

The invention relates to the technical field of speech recognition, in particular to a phoneme recognition method, device, electronic equipment and storage medium.

Background technique

In the field of speech recognition, a phoneme is the smallest unit in speech. To improve the accuracy of speech recognition, it is necessary to improve the recognition accuracy of each phoneme in speech.

In practical application scenarios, speech corresponds to different languages. In order to accurately recognize speech in different languages, a sub-model is trained for each language at present, and a phoneme recognition model is constructed based on these sub-models to use the phoneme recognition model. The sub-models in each perform phoneme recognition on the speech of each language, and then obtain the corresponding speech recognition results according to the phoneme recognition results. However, as the types of languages increase, the number of sub-models will also increase, resulting in an increase in the size of the phoneme recognition model, which in turn affects the deployment of the phoneme recognition model on the local chip.

Contents of the invention

The invention provides a phoneme recognition method, device, electronic equipment and storage medium to solve the defect of large-scale phoneme recognition models in the prior art.

The invention provides a phoneme recognition method, comprising:

Determine the voice to be recognized;

Inputting the speech to be recognized into a phoneme recognition model to obtain a phoneme recognition result output by the phoneme recognition model;

The phoneme recognition model is obtained by training the first recognition model based on the sample speeches of multiple languages and the phoneme-level labels of each sample speech, and the first recognition model is based on the phonemes corresponding to each phoneme node under the second recognition model. The similarity between them is obtained by screening the phoneme nodes under the second recognition model, and the second recognition model includes phoneme nodes corresponding to multiple languages.

According to a phoneme recognition method provided by the present invention, the step of determining the first recognition model includes:

Based on the similarity between the phonemes corresponding to each phoneme node, each phoneme node under the second recognition model is clustered to obtain a plurality of clusters;

The current phoneme node is obtained by screening the phoneme nodes in each cluster, and deleting other phoneme nodes in each cluster except the current phoneme node, so as to obtain the first recognition model.

According to a phoneme recognition method provided by the present invention, the second recognition model includes a feature extraction layer and phoneme classification layers corresponding to multiple languages, and each phoneme classification layer is constructed based on phoneme nodes corresponding to each language;

The second recognition model is trained based on the following steps:

Input the sample speech of each language to the feature extraction layer of the second recognition model, and obtain the first phoneme hidden layer feature output by the feature extraction layer of the second recognition model;

The first phoneme hidden layer feature is input to the phoneme classification layer of each language to obtain the first phoneme prediction result output by the phoneme classification layer of each language;

Based on the difference between the phoneme-level label and the first phoneme prediction result, parameter iteration is performed on the feature extraction layer of the second recognition model and the phoneme classification layer of each language to obtain the second recognition model.

According to a kind of phoneme recognition method provided by the present invention, the first phoneme hidden layer feature obtained by the feature extraction layer output of the second recognition model further includes:

Based on the first phoneme hidden layer features, determine word-level hidden layer features and/or sentence-level hidden layer features;

Based on the difference between the word-level label of the sample speech and the word-level prediction result and/or the difference between the language label of the sample speech and the language prediction result, parameterize the feature extraction layer of the second recognition model Iterating to obtain the second recognition model; the word-level prediction result is determined based on the word-level hidden layer features, and the language type prediction result is determined based on the sentence-level hidden layer features.

According to a phoneme recognition method provided by the present invention, the difference between the word-level label based on the sample speech and the word-level prediction result and/or the difference between the language label of the sample speech and the language prediction result, Perform parameter iteration on the feature extraction layer of the second recognition model to obtain the second recognition model, including:

The word-level hidden layer features are input to the word-level classification layer to obtain the word-level prediction results output by the word-level classification layer, and/or, the sentence-level hidden layer features are input to the language classification layer to obtain The language prediction result output by the language classification layer;

performing parameter iteration on the feature extraction layer of the second recognition model based on the difference between the word-level label and the word-level prediction result and/or the difference between the language label and the language prediction result, Obtain the second recognition model.

According to a kind of phoneme recognition method provided by the present invention, the described first phoneme hidden layer feature is used to determine the word-level hidden layer feature and/or sentence-level hidden layer feature, including:

Carry out sliding window to described first phoneme hidden layer feature, obtain described word level hidden layer feature;

The word-level hidden layer features are pooled to obtain the sentence-level hidden layer features.

According to a method for phoneme recognition provided by the present invention, the phoneme recognition model is trained based on the following steps:

fixing the parameters of the feature extraction layer of the first recognition model;

Input the sample speech of each language to the feature extraction layer of the first recognition model, obtain the second phoneme hidden layer feature output by the feature extraction layer of the first recognition model;

The second phoneme hidden layer feature is input to the current phoneme classification layer to obtain the second phoneme prediction result output by the current phoneme classification layer; the current phoneme classification layer is based on the phonemes screened from the second recognition model The node is constructed;

Based on the difference between the phoneme-level label and the second phoneme prediction result, parameter iteration is performed on the current phoneme classification layer to obtain the phoneme recognition model.

The present invention also provides a phoneme recognition device, comprising:

a determining unit, configured to determine the speech to be recognized;

The recognition unit is used to input the speech to be recognized to the phoneme recognition model, and obtain the phoneme recognition result output by the phoneme recognition model;

The phoneme recognition model is obtained by training the first recognition model based on the sample speeches of multiple languages and the phoneme-level labels of each sample speech, and the first recognition model is based on the phonemes corresponding to each phoneme node under the second recognition model. The similarity between them is obtained by screening the phoneme nodes under the second recognition model, and the second recognition model includes phoneme nodes corresponding to multiple languages.

The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and operable on the processor, when the processor executes the program, the phoneme recognition method described in any one of the above is implemented .

The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the phoneme recognition method described above can be implemented.

The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, any phoneme recognition method described above is implemented.

The phoneme recognition method, device, electronic equipment and storage medium provided by the present invention, based on the similarity between phonemes corresponding to each phoneme node under the second recognition model, screen the phoneme nodes under the second recognition model to obtain the first recognition model , not only reduces the scale of the first recognition model, but also retains the phoneme nodes corresponding to different phonemes in the first recognition model, and then based on the sample speech of multiple languages and the phoneme-level labels of each sample speech to the first recognition model After training, not only the scale of the obtained phoneme recognition model is smaller than the second recognition model, but also the phoneme recognition model can accurately distinguish phonemes of different languages.

Description of drawings

In order to more clearly illustrate the present invention or the technical solutions in the prior art, the accompanying drawings that need to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings in the following description are the present invention. For some embodiments of the invention, those skilled in the art can also obtain other drawings based on these drawings without creative effort.

Fig. 1 is a schematic flow chart of the phoneme recognition method provided by the present invention;

Fig. 2 is a schematic flow chart of the method for determining the first identification model provided by the present invention;

Fig. 3 is a schematic flow chart of the second recognition model training method provided by the present invention;

Fig. 4 is a schematic flowchart of another second recognition model training method provided by the present invention;

FIG. 5 is a schematic flowchart of an implementation of Step 420 in yet another second recognition model training method provided by the present invention;

Fig. 6 is a schematic flow chart of the phoneme recognition model training method provided by the present invention;

Fig. 7 is a schematic flowchart of another second recognition model training method provided by the present invention;

Fig. 8 is a schematic structural diagram of a phoneme recognition device provided by the present invention;

FIG. 9 is a schematic structural diagram of an electronic device provided by the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention , but not all examples. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

At present, when performing speech recognition on different languages, a sub-model is usually trained for each language, and a phoneme recognition model is constructed based on these sub-models, so as to use each sub-model in the phoneme recognition model to perform speech recognition in each language. phoneme recognition, and then obtain a corresponding speech recognition result according to the phoneme recognition result. However, as the types of languages increase, the number of sub-models will also increase, resulting in an increase in the size of the phoneme recognition model, which in turn affects the deployment of the phoneme recognition model on the local chip.

In addition, in order to avoid increasing the scale of the phoneme recognition model, it is also possible to introduce a language classification branch, and then insert a gradient inversion layer between the language classification branch and the main branch, so that the phoneme recognition model can learn the language unchanged through gradient confrontation training. features, but this method is suitable for languages with little difference, and the recognition effect is poor for languages with large differences (such as Cantonese, Hokkien, etc., which are quite different from Mandarin).

For this, the present invention provides a phoneme recognition method. Fig. 1 is a schematic flow chart of the phoneme recognition method provided by the present invention, as shown in Fig. 1, the method comprises the following steps:

Step 110, determine the speech to be recognized.

Here, the speech to be recognized is the speech that requires phoneme recognition. The voice to be recognized can be obtained through a sound pickup device. The sound pickup device here can be a smartphone, a tablet computer, or a smart appliance such as a stereo, a TV, and an air conditioner. After the sound pickup device is picked up by a microphone array to obtain the voice to be recognized , the speech to be recognized may also be amplified and denoised, which is not specifically limited in this embodiment of the present invention.

Step 120, input the speech to be recognized into the phoneme recognition model, and obtain the phoneme recognition result output by the phoneme recognition model;

The phoneme recognition model is obtained by training the first recognition model based on sample speech in multiple languages and the phoneme-level labels of each sample speech. The first recognition model is based on the similarity between phonemes corresponding to each phoneme node under the second recognition model , which is obtained by filtering the phoneme nodes under the second recognition model, and the second recognition model includes phoneme nodes respectively corresponding to a plurality of languages.

Here, the phoneme nodes of each language correspond to different phonemes of each language. For example, "a" and "i" in Mandarin are different phonemes, so "a" and "i" in Mandarin correspond to different phoneme nodes. The second recognition model includes phoneme nodes corresponding to multiple languages, so that the second recognition model can distinguish phonemes in different languages through the phoneme nodes corresponding to each language, that is, the second recognition model has the ability to accurately identify phonemes in different languages. ability to distinguish.

However, if there are a large number of different types of languages, there will be too many phoneme nodes in the second recognition model, which will lead to a large amount of calculation parameters of the second model. In addition, considering that there may be phonemes with high similarity in the same language, there may also be phonemes with high similarity in different languages, that is, there may be redundancy in the phoneme nodes corresponding to each language in the second recognition model.

In this regard, the embodiment of the present invention screens the phoneme nodes under the second recognition model based on the similarity between the phonemes corresponding to each phoneme node under the second recognition model, such as based on the similarity between phonemes, for each phoneme node Carry out clustering, cluster the phoneme nodes corresponding to the phonemes with high similarity into one class, then select any phoneme node in the same class as the current phoneme node of the first recognition model, and delete the remaining phoneme nodes in this class, so that The number of phoneme nodes in the first recognition model can be reduced, so as to reduce the scale of the first recognition model, and correspondingly reduce the scale of the phoneme recognition model.

After screening the phoneme nodes under the second recognition model, the redundant phoneme nodes under the second recognition model are filtered out, that is, the phoneme nodes corresponding to similar phonemes are filtered out, that is, the current phoneme nodes included in the first recognition model There are also different types of phonemes, so that after training the first recognition model based on sample speech in multiple languages and the phoneme-level labels of each sample speech, the resulting phoneme recognition model can accurately distinguish phonemes in different languages.

The phoneme recognition method provided by the embodiment of the present invention is based on the similarity between phonemes corresponding to each phoneme node under the second recognition model, and screens the phoneme nodes under the second recognition model to obtain the first recognition model, which not only reduces the The scale of a recognition model, and the phoneme nodes corresponding to different phonemes are reserved in the first recognition model, and then after training the first recognition model based on the sample speeches of multiple languages and the phoneme-level labels of each sample speech, not only makes The scale of the obtained phoneme recognition model is smaller than that of the second recognition model, and the phoneme recognition model can accurately distinguish phonemes of different languages.

Based on the above-mentioned embodiment, Fig. 2 is a schematic flowchart of the method for determining the first recognition model provided by the present invention. As shown in Fig. 2, the steps of determining the first recognition model include:

Step 210, based on the similarity between the phonemes corresponding to each phoneme node, cluster each phoneme node under the second recognition model to obtain a plurality of clusters;

Step 220 : Screen the phoneme nodes in each cluster to obtain the current phoneme node, and delete other phoneme nodes in each cluster except the current phoneme node to obtain the first recognition model.

Specifically, the similarity between phonemes is used to represent the probability that two corresponding phonemes belong to the same category. The higher the similarity between phonemes, the higher the probability that the corresponding two phonemes belong to the same category; The lower the similarity, the lower the probability that the corresponding two phonemes belong to the same category.

Based on the similarity between the phonemes corresponding to each phoneme node, the phoneme nodes corresponding to the same type of phoneme are clustered into one class, and multiple clusters are obtained, that is, the phoneme classes corresponding to the phoneme nodes contained in each cluster class are the same or similar.

In addition, the current phoneme node refers to the phoneme node of the first recognition model, and the phoneme categories corresponding to the current phoneme node in different clusters are different. After obtaining multiple clusters, the current phoneme node is filtered from the phoneme nodes in each cluster. The current phoneme node can be one or more, but the total number of current phoneme nodes in each cluster is less than the second The sum of the number of phoneme nodes in the recognition model.

Optionally, any phoneme node or multiple phoneme nodes in each cluster can be used as the current phoneme node, or the phoneme node whose distance from the center of each cluster is less than the threshold can be used as the current phoneme node, or the distance from each The phoneme node closest to the cluster center is used as the current phoneme node, which is not specifically limited in this embodiment of the present invention.

After the current phoneme node in each cluster is determined, other phoneme nodes in each cluster are of the same or similar category as the current phoneme node, that is, other phoneme nodes can be regarded as redundant factor nodes. In this regard, in the embodiment of the present invention, after obtaining the current phoneme nodes of each cluster, delete other phoneme nodes in each cluster except the current phoneme node to obtain the first recognition model, which can be understood as the first recognition model is derived from the first The model obtained after deleting redundant phoneme nodes in the second recognition model.

It can be seen that, based on the similarity between phonemes corresponding to each phoneme node, the embodiment of the present invention can accurately cluster each phoneme node under the second recognition model, and then obtain multiple clusters. At the same time, the embodiment of the present invention obtains the current phoneme node from the phoneme nodes in each cluster, and deletes other phoneme nodes in each cluster except the current phoneme node, thereby reducing the number of phoneme nodes in the first recognition model and realizing The scale of the first recognition model is reduced, thereby correspondingly reducing the scale of the phoneme recognition model.

As an optional embodiment, when each phoneme node under the second recognition model is clustered, a Gaussian mixture model (Gaussian mixture model, GMM model) and an expectation maximization algorithm (Expectation-Maximum, EM) can be used to The phonemes corresponding to each phoneme node are clustered.

For example, the second recognition model includes phoneme nodes corresponding to N _a languages, and the number of phoneme nodes corresponding to each language is N _c , that is, the total number of phoneme nodes included in the second recognition model is N _a × N _c , if To make the total number of phoneme nodes in the first recognition model be _Nc , each phoneme node can be clustered through the GMM model, and then iteratively adjust the parameters of the GMM model according to the clustering results until the clustering result is the second recognition Each phoneme node in the model is divided into _Nc clusters.

It should be noted that, in the embodiment of the present invention, each phoneme node in the second recognition model can also be divided into other numbers of clusters according to actual requirements, which is not specifically limited in the embodiment of the present invention.

Based on any of the above embodiments, the second recognition model includes a feature extraction layer and phoneme classification layers corresponding to multiple languages, and each phoneme classification layer is constructed based on phoneme nodes corresponding to each language. Fig. 3 is a schematic flow chart of the second recognition model training method provided by the present invention. As shown in Fig. 3, the training steps of the second recognition model include:

Step 310, input the sample speech of each language to the feature extraction layer of the second recognition model, and obtain the first phoneme hidden layer feature output by the feature extraction layer of the second recognition model;

Step 320, input the first phoneme hidden layer feature to the phoneme classification layer of each language, and obtain the first phoneme prediction result output by the phoneme classification layer of each language;

Step 330 : Based on the difference between the phoneme-level label and the first phoneme prediction result, perform parameter iteration on the feature extraction layer of the second recognition model and the phoneme classification layer of each language to obtain the second recognition model.

Specifically, the first phoneme hidden layer feature is used to represent the feature information of each phoneme in the sample speech, which can be understood as a frame-level hidden layer feature. The feature extraction layer of the second recognition model is used to extract the features of the first phoneme hidden layer corresponding to the sample speeches of various languages. Wherein, the feature extraction layer in the second recognition model is shared, that is, the sample speeches of various languages can be feature extracted by the feature extraction layer. In addition, the feature extraction layer of the second recognition model can use neural network models such as DNN (Deep Neural Network, deep neural network), RNN (Recurrent Neural Network, cyclic neural network) or CNN (Convolution Neural Network, convolutional neural network) to extract the first A phoneme hidden layer feature, which is not specifically limited in this embodiment of the present invention.

In addition, the second recognition model also includes phoneme classification layers corresponding to multiple languages, that is, the phoneme classification layers corresponding to each language are independent of each other, so that the phoneme classification layers corresponding to each language can independently learn the phoneme information of the corresponding language, avoiding The impact of pronunciation conflicts between different languages on phoneme recognition, and then accurately recognize the phonemes in the language, and obtain the first phoneme prediction result.

After the first phoneme prediction result is obtained, based on the difference between the phoneme-level label and the first phoneme prediction result, parameter iteration is performed on the feature extraction layer of the second recognition model and the phoneme classification layer of each language, so that the second recognition model is in During the training process, the information of different types of phonemes in each language can be learned as much as possible, so that the second recognition model can accurately identify phonemes in each language.

It can be seen that, based on the difference between the phoneme-level label and the first phoneme prediction result, the embodiment of the present invention performs parameter iteration on the feature extraction layer of the second recognition model and the phoneme classification layer of each language, so that the second The recognition model accurately recognizes phonemes in each language.

As an optional embodiment, the feature extraction layer in the second recognition model may include a first encoding layer and a first attention layer, and the first encoding layer is used to encode sample speeches of various languages to obtain the The first coding feature, the first attention layer is used to perform attention transformation on the first coding feature of each sample speech based on the attention mechanism, so as to obtain the first phoneme hidden layer feature. In addition, the phoneme classification layer of each language in the second recognition model may include a first decoding layer and a first recognition layer, and the first decoding layer is used to decode the first phoneme hidden layer features to obtain the first decoding features of each sample speech , the first recognition layer is used to perform phoneme recognition based on the first decoding feature of each sample speech, to obtain a first phoneme prediction result.

Based on any of the above-mentioned embodiments, FIG. 4 is a schematic flowchart of another second recognition model training method provided by the present invention. As shown in FIG. 4, the training steps of the second recognition model include:

Step 410: After obtaining the first phoneme hidden layer features output by the feature extraction layer of the second recognition model, determine word-level hidden layer features and/or sentence-level hidden layer features based on the first phoneme hidden layer features;

Step 420: Based on the difference between the word-level label of the sample speech and the word-level prediction result and/or the difference between the language label of the sample speech and the language prediction result, perform parameter iteration on the feature extraction layer of the second recognition model to obtain The second recognition model: word-level prediction results are determined based on word-level hidden layer features, and language prediction results are determined based on sentence-level hidden layer features.

Specifically, the word-level hidden layer features are used to represent the feature information of each character in the sample speech. Since each character is constructed based on multiple phonemes, when determining the word-level hidden layer features, it is necessary to base on the multiple first phonemes corresponding to the characters. A phoneme hidden layer feature is determined. The sentence-level hidden layer features are used to represent the feature information of each sentence in the sample speech. Since each sentence is constructed based on multiple characters, when determining the sentence-level hidden layer features, it needs to be based on the multiple characters corresponding to the sentence. Level hidden layer features are determined.

Based on the difference between the word-level tags of the sample speech and the word-level prediction results, when the parameter iteration is performed on the feature extraction layer of the second recognition model, the feature extraction layer of the second recognition model can learn from the word-level level. Different phoneme information, and then can accurately identify different phonemes from the word level.

Based on the difference between the language label of the sample speech and the language prediction result, when the parameter iteration is performed on the feature extraction layer of the second recognition model, the feature extraction layer of the second recognition model can learn different phonemes in each language from the sentence level. information, and thus be able to accurately identify different phonemes from the sentence level.

It can be seen that, in the embodiment of the present invention, based on the difference between the word-level label of the sample speech and the word-level prediction result and/or the difference between the language label of the sample speech and the language prediction result, the feature extraction layer of the second recognition model Performing parameter iterations can enable the second recognition model to accurately recognize different phonemes at a granular character level and/or sentence level, further improving the phoneme recognition effect of the second recognition model.

Based on any of the above-mentioned embodiments, FIG. 5 is a schematic flowchart of an implementation of step 420 in yet another second recognition model training method provided by the present invention. As shown in FIG. 5, step 420 includes:

Step 421: Input the word-level hidden layer features into the word-level classification layer to obtain the word-level prediction results output by the word-level classification layer, and/or input the sentence-level hidden layer features into the language classification layer to obtain the output of the language classification layer Language prediction results;

Step 422 : Based on the difference between the word-level label and the word-level prediction result and/or the difference between the language label and the language prediction result, perform parameter iteration on the feature extraction layer of the second recognition model to obtain the second recognition model.

Specifically, the word-level classification layer is used to determine the word-level prediction result based on the word-level hidden layer features, and the language classification layer is used to determine the language type prediction result based on the sentence-level hidden layer features. Wherein, the word-level prediction result can be understood as the prediction result of each character in the sample speech, and the language prediction result can be understood as the language prediction result of each clause in the sample speech.

Optionally, this embodiment of the present invention may perform parameter iteration on the feature extraction layer of the second recognition model based on the difference between the word-level label of the sample speech and the word-level prediction result, so that the feature extraction layer of the second recognition model can Learn different phoneme information in various languages at the word level, and then be able to accurately identify different phonemes at the word level.

Optionally, in this embodiment of the present invention, parameter iteration may be performed on the feature extraction layer of the second recognition model based on the difference between the language label of the sample speech and the language prediction result, so that the feature extraction layer of the second recognition model can learn from the sentence Learn different phoneme information in each language at the level level, and then be able to accurately identify different phonemes at the sentence level level.

Optionally, in this embodiment of the present invention, the feature extraction layer of the second recognition model may be based on the difference between the word-level label of the sample speech and the word-level prediction result and the difference between the language label of the sample speech and the language prediction result. The parameters are iterated, so that the feature extraction layer of the second recognition model can learn different phoneme information in each language from the word level and sentence level, and then can accurately identify different phonemes from the word level and sentence level.

It should be noted that the word-level classification layer and the sentence-level classification layer can be set in the auxiliary model, that is, the second recognition model does not include the word-level classification layer and the sentence-level classification layer. The word-level and/or sentence-level level assists in training the second recognition model, so that the second recognition model can further accurately identify different phonemes from the word-level and/or sentence-level level, and improve the phoneme recognition effect.

As an optional embodiment, the auxiliary model may include a word-level feature extraction layer, a word-level classification layer, a sentence-level feature extraction layer, and a language classification layer. Among them, the word-level feature extraction layer is used to perform a sliding window on the first phoneme hidden layer feature to obtain the word-level hidden layer feature. The word-level classification layer is used to perform character recognition based on word-level hidden layer features, and obtain word-level prediction results. The sentence-level feature extraction layer is used to pool the word-level hidden layer features output by the word-level feature extraction layer to obtain sentence-level hidden layer features. The language classification layer is used for language identification based on the sentence-level hidden layer features to obtain language prediction results.

Based on any of the above-mentioned embodiments, based on the first phoneme hidden layer features, determine word-level hidden layer features and/or sentence-level hidden layer features, including:

Perform a sliding window on the hidden layer features of the first phoneme to obtain word-level hidden layer features;

The word-level hidden layer features are pooled to obtain sentence-level hidden layer features.

Specifically, since the granularity of character recognition based on word-level hidden layer features is greater than that of phoneme recognition based on the first phoneme hidden layer features, it is necessary to perform a sliding window operation on the first phoneme hidden layer features, for example, the window can be set The length is B, each time the first phoneme hidden layer feature of B frame is sent to the neural network, after the neural network abstracts the word-level hidden layer features, then the word-level hidden layer features are entered into the word-level classification layer, and the word-level prediction results are obtained . The granularity of language recognition based on sentence-level hidden layer features is larger than that of character recognition based on word-level hidden layer features, so word-level hidden layer features can be pooled multiple times through neural networks to generate sentence-level hidden layer features. layer features, and input sentence-level hidden layer features to the language classification layer to obtain language prediction results.

Based on any of the above-mentioned embodiments, FIG. 6 is a schematic flow chart of a phoneme recognition model training method provided by the present invention. As shown in FIG. 6, the training steps of the phoneme recognition model include:

Step 610, fixing the parameters of the feature extraction layer of the first recognition model;

Step 620, input the sample speeches of various languages into the feature extraction layer of the first recognition model, and obtain the second phoneme hidden layer features output by the feature extraction layer of the first recognition model;

Step 630: Input the second phoneme hidden layer feature into the current phoneme classification layer to obtain the second phoneme prediction result output by the current phoneme classification layer; the current phoneme classification layer is constructed based on the phoneme nodes screened from the second recognition model;

Step 640: Perform parameter iteration on the current phoneme classification layer based on the difference between the phoneme-level label and the second phoneme prediction result to obtain a phoneme recognition model.

Specifically, the feature extraction layer of the first recognition model is the feature extraction layer of the trained second recognition model. Since the trained second recognition model has the ability to accurately perform feature extraction, after obtaining the first recognition model, And after fixing the parameters of the feature extraction layer of the first recognition model, the first recognition model can retain the feature extraction capability of the trained second recognition model.

The feature extraction layer of the first recognition model is used to extract the second phoneme hidden layer features corresponding to the sample speeches of various languages. Since the feature extraction layer of the first recognition model is the feature extraction layer of the trained second recognition model, the feature extraction layer of the first recognition model can accurately extract the features of the second phoneme hidden layer.

The current phoneme classification layer is constructed based on the phoneme nodes screened from the second recognition model, that is, the current phoneme classification layer filters redundant phoneme nodes on the basis of the phoneme classification layer of the second recognition model, that is, the current phoneme classification layer The number of phoneme nodes in is less than the number of phoneme nodes in the phoneme classification layer of the second recognition model. Similarly, the current phoneme classification layer is used to perform phoneme recognition based on the second phoneme hidden layer features to obtain a second phoneme prediction result.

After the second phoneme prediction result is obtained, based on the difference between the phoneme-level label and the second phoneme prediction result, the parameters of the current phoneme classification layer are iterated, so that the phoneme recognition model can learn different types of phonemes in each language during the training process. information, so that the phoneme recognition model can accurately identify phonemes in various languages.

It can be seen that the embodiment of the present invention performs parameter iteration on the current phoneme classification layer based on the difference between the phoneme-level label and the second phoneme prediction result, so that the trained phoneme recognition model can accurately recognize phonemes in various languages.

As an optional embodiment, the feature extraction layer in the first recognition model may include a second encoding layer and a second attention layer, and the second encoding layer is used to encode sample speeches of various languages to obtain the The second coding feature, the second attention layer is used to perform attention transformation on the second coding feature of each sample speech based on the attention mechanism, so as to obtain the second phoneme hidden layer feature. In addition, the current phoneme classification layer in the second recognition model may include a second decoding layer and a second recognition layer, and the second decoding layer is used to decode the second phoneme hidden layer features to obtain the second decoding features of each sample speech, The second recognition layer is used to perform phoneme recognition based on the second decoding feature of each sample speech, and obtain a second phoneme prediction result.

Based on any of the above embodiments, the phoneme recognition model is obtained by training the first recognition model based on sample speeches of multiple languages and phoneme-level labels of each sample speech. The first recognition model is obtained by screening the phoneme nodes under the second recognition model based on the similarity between phonemes corresponding to each phoneme node under the second recognition model. The present invention also provides a phoneme recognition model training method, the method comprising:

Fig. 7 is a schematic flow chart of another second recognition model training method provided by the present invention. As shown in Fig. 7, the second recognition model includes a feature extraction layer and phoneme classification layers corresponding to a plurality of languages, and each phoneme classification layer is based on each The phoneme node corresponding to the language is constructed. The second recognition model is obtained based on the sample speech of multiple languages and the phoneme-level label training of each sample speech, specifically: input the sample speech of each language to the feature extraction layer of the second recognition model to obtain the first phoneme hidden layer features, And input the features of the first phoneme hidden layer to the phoneme classification layer of each language to obtain the first phoneme prediction result. Based on the difference between the phoneme-level label and the first phoneme prediction result, the feature extraction layer of the second recognition model and each The language phoneme classification layer performs parameter iteration to obtain the initial second recognition model.

After obtaining the initial second recognition model, based on the first phoneme hidden layer features, determine the word-level hidden layer features and sentence-level hidden layer features, and input the word-level hidden layer features into the word-level classification layer of the auxiliary model to obtain word-level predictions As a result, the sentence-level hidden layer features are input to the language classification layer of the auxiliary model to obtain language prediction results. Based on the difference between the word-level label and the word-level prediction result and the difference between the language label and the language prediction result, the parameter iteration is performed on the feature extraction layer of the initial second recognition model to obtain the second recognition model.

Then, based on the similarity between the phonemes corresponding to each phoneme node, each phoneme node under the second recognition model is clustered to obtain multiple clusters, and any phoneme node in each cluster is retained and each cluster is deleted other phoneme nodes in the class to obtain the first recognition model.

After obtaining the first recognition model, the parameters of the feature extraction layer of the first recognition model are fixed. Input the sample speech of each language to the feature extraction layer of the first recognition model to obtain the second phoneme hidden layer feature, and input the second phoneme hidden layer feature to the current phoneme classification layer to obtain the second phoneme prediction result; wherein, the current phoneme The classification layer is constructed based on the phoneme nodes screened from the second recognition model.

Finally, based on the difference between the phoneme-level label and the second phoneme prediction result, the parameters of the current phoneme classification layer are iterated to obtain a phoneme recognition model. The phoneme recognition model is not only small in scale, but also can accurately distinguish phonemes in different languages. .

The phoneme recognition device provided by the present invention is described below, and the phoneme recognition device described below and the phoneme recognition method described above can be referred to in correspondence.

Based on any of the above-mentioned embodiments, FIG. 8 is a schematic structural diagram of a phoneme recognition device provided by the present invention. As shown in FIG. 8, the device includes:

A determining unit 810, configured to determine the speech to be recognized;

The recognition unit 820 is used to input the speech to be recognized into the phoneme recognition model, and obtain the phoneme recognition result output by the phoneme recognition model;

The phoneme recognition model is obtained by training the first recognition model based on the sample speeches of multiple languages and the phoneme-level labels of each sample speech, and the first recognition model is based on the phonemes corresponding to each phoneme node under the second recognition model. The similarity between them is obtained by screening the phoneme nodes under the second recognition model, and the second recognition model includes phoneme nodes corresponding to multiple languages.

Based on any of the above-mentioned embodiments, the device further includes:

A clustering unit, configured to cluster each phoneme node under the second recognition model based on the similarity between the phonemes corresponding to each phoneme node, to obtain a plurality of clusters;

The pruning unit is configured to filter phoneme nodes in each cluster to obtain the current phoneme node, and delete other phoneme nodes in each cluster except the current phoneme node to obtain the first recognition model.

Based on any of the above embodiments, the second recognition model includes a feature extraction layer and phoneme classification layers corresponding to multiple languages, and each phoneme classification layer is constructed based on phoneme nodes corresponding to each language;

The device also includes:

The first feature extraction unit is used to input the sample speech of each language to the feature extraction layer of the second recognition model, and obtain the first phoneme hidden layer feature output by the feature extraction layer of the second recognition model;

The first phoneme classification unit is configured to input the first phoneme hidden layer feature to the phoneme classification layer of each language, and obtain the first phoneme prediction result output by the phoneme classification layer of each language;

The first parameter iteration unit is configured to perform parameter iteration on the feature extraction layer of the second recognition model and the phoneme classification layer of each language based on the difference between the phoneme-level label and the first phoneme prediction result, to obtain The second recognition model.

Based on any of the above-mentioned embodiments, the device further includes:

The feature determination unit is configured to determine word-level hidden layer features and/or sentence-level hidden layer features based on the first phoneme hidden layer features after obtaining the first phoneme hidden layer features output by the feature extraction layer of the second recognition model feature;

The second parameter iteration unit is configured to, based on the difference between the word-level label of the sample speech and the word-level prediction result and/or the difference between the language label of the sample speech and the language prediction result, The feature extraction layer of the recognition model performs parameter iteration to obtain the second recognition model; the word-level prediction result is determined based on the word-level hidden layer features, and the language type prediction result is determined based on the sentence-level hidden layer features.

Based on any of the above-mentioned embodiments, the second parameter iteration unit includes:

An auxiliary prediction unit, configured to input the word-level hidden layer features to the word-level classification layer, obtain the word-level prediction result output by the word-level classification layer, and/or input the sentence-level hidden layer features To the language classification layer, the language prediction result output by the language classification layer is obtained;

Auxiliary training unit, used to train the features of the second recognition model based on the difference between the word-level label and the word-level prediction result and/or the difference between the language label and the language prediction result The extraction layer performs parameter iteration to obtain the second recognition model.

Based on any of the above-mentioned embodiments, the feature determination unit includes:

A sliding window unit, configured to perform a sliding window on the first phoneme hidden layer feature to obtain the word-level hidden layer feature;

A pooling unit is configured to perform pooling on the word-level hidden layer features to obtain the sentence-level hidden layer features.

Based on any of the above-mentioned embodiments, the device further includes:

A parameter fixing unit, configured to fix parameters of the feature extraction layer of the first recognition model;

The second feature extraction unit is used to input the sample speech of each language to the feature extraction layer of the first recognition model, and obtain the second phoneme hidden layer feature output by the feature extraction layer of the first recognition model;

The second phoneme classification unit is configured to input the second phoneme hidden layer feature to the current phoneme classification layer to obtain the second phoneme prediction result output by the current phoneme classification layer; the current phoneme classification layer is based on the first phoneme classification layer. Two phoneme nodes screened in the recognition model are constructed;

The third parameter iteration unit is configured to perform parameter iteration on the current phoneme classification layer based on the difference between the phoneme-level label and the second phoneme prediction result to obtain the phoneme recognition model.

FIG. 9 is a schematic structural diagram of an electronic device provided by the present invention. As shown in FIG. 9, the electronic device may include: a processor (processor) 910, a memory (memory) 920, a communication interface (Communications Interface) 930 and a communication bus 940, Wherein, the processor 910 , the memory 920 , and the communication interface 930 communicate with each other through the communication bus 940 . The processor 910 can call the logic instructions in the memory 920 to execute the phoneme recognition method, the method comprising: determining the speech to be recognized; inputting the speech to be recognized into the phoneme recognition model, and obtaining the phoneme recognition result output by the phoneme recognition model The phoneme recognition model is obtained by training the first recognition model based on the sample speech of multiple languages and the phoneme-level labels of each sample speech, and the first recognition model is based on the phoneme corresponding to each phoneme node under the second recognition model The similarity between them is obtained by filtering the phoneme nodes under the second recognition model, and the second recognition model includes phoneme nodes corresponding to multiple languages.

In addition, the above-mentioned logic instructions in the memory 920 may be implemented in the form of software function units and may be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes. .

On the other hand, the present invention also provides a computer program product, the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer During execution, the computer can execute the phoneme recognition method provided by each of the above methods, the method comprising: determining the speech to be recognized; inputting the speech to be recognized to the phoneme recognition model to obtain the phoneme recognition result output by the phoneme recognition model; The phoneme recognition model is obtained by training the first recognition model based on the sample speeches of multiple languages and the phoneme-level labels of each sample speech. The similarity is obtained by screening the phoneme nodes under the second recognition model, and the second recognition model includes phoneme nodes corresponding to multiple languages.

In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it is implemented to perform the phoneme recognition methods provided above, and the method includes: determining The speech to be recognized; the speech to be recognized is input to the phoneme recognition model to obtain the phoneme recognition result output by the phoneme recognition model; the phoneme recognition model is based on the sample speech of multiple languages and the phoneme-level label of each sample speech. The first recognition model is obtained by training, and the first recognition model is obtained by screening the phoneme nodes under the second recognition model based on the similarity between phonemes corresponding to each phoneme node under the second recognition model, so The second recognition model includes phoneme nodes respectively corresponding to a plurality of languages.

The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without any creative efforts.

Through the above description of the implementations, those skilled in the art can clearly understand that each implementation can be implemented by means of software plus a necessary general hardware platform, and of course also by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic discs, optical discs, etc., including several instructions to make a computer device (which may be a personal computer, server, or network device, etc.) execute the methods described in various embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims (10)

1. A phoneme recognition method, comprising:

determining a voice to be recognized;

inputting the voice to be recognized into a phoneme recognition model to obtain a phoneme recognition result output by the phoneme recognition model;

the phoneme recognition model is obtained by training a first recognition model based on sample voices of multiple languages and phoneme level labels of the sample voices, the first recognition model is obtained by screening phoneme nodes under a second recognition model based on similarity between phonemes corresponding to the phoneme nodes under the second recognition model, and the second recognition model comprises phoneme nodes corresponding to the multiple languages respectively.

2. The phoneme recognition method of claim 1, wherein the step of determining the first recognition model comprises:

clustering the phoneme nodes under the second recognition model based on the similarity between the phonemes corresponding to the phoneme nodes to obtain a plurality of cluster classes;

and screening the phoneme nodes in each cluster type to obtain a current phoneme node, and deleting other phoneme nodes except the current phoneme node in each cluster type to obtain the first identification model.

3. The phoneme recognition method of claim 1, wherein the second recognition model comprises a feature extraction layer and phoneme classification layers corresponding to a plurality of languages, and each phoneme classification layer is constructed based on phoneme nodes corresponding to each language;

the second recognition model is obtained by training based on the following steps:

inputting sample voices of various languages into the feature extraction layer of the second recognition model to obtain first phoneme hidden layer features output by the feature extraction layer of the second recognition model;

inputting the first phoneme hidden layer characteristics to the phoneme classification layers of all languages to obtain first phoneme prediction results output by the phoneme classification layers of all languages;

and performing parameter iteration on a feature extraction layer of the second recognition model and a phoneme classification layer of each language based on the difference between the phoneme level label and the first phoneme prediction result to obtain the second recognition model.

4. The method for phoneme recognition of claim 3, wherein said obtaining the first phoneme hidden layer features outputted from the feature extraction layer of the second recognition model further comprises:

determining word-level hidden layer features and/or sentence-level hidden layer features based on the first phoneme hidden layer features;

performing parameter iteration on a feature extraction layer of the second recognition model based on the difference between the word-level label and the word-level prediction result of the sample voice and/or the difference between the language label and the language prediction result of the sample voice to obtain the second recognition model; the word-level prediction result is determined based on the word-level hidden layer characteristics, and the language prediction result is determined based on the sentence-level hidden layer characteristics.

5. The phoneme recognition method of claim 4, wherein the performing parameter iteration on the feature extraction layer of the second recognition model based on the difference between the word-level tag of the sample speech and the word-level prediction result and/or the difference between the language tag of the sample speech and the language prediction result to obtain the second recognition model comprises:

inputting the character-level hidden layer characteristics to a character-level classification layer to obtain the character-level prediction result output by the character-level classification layer, and/or inputting the sentence-level hidden layer characteristics to a language classification layer to obtain the language prediction result output by the language classification layer;

and performing parameter iteration on the feature extraction layer of the second recognition model based on the difference between the word-level label and the word-level prediction result and/or the difference between the language label and the language prediction result to obtain the second recognition model.

6. The phoneme recognition method of claim 4, wherein said determining word-level hidden layer features and/or sentence-level hidden layer features based on said first phoneme hidden layer features comprises:

performing sliding window on the first phoneme hidden layer characteristics to obtain the word-level hidden layer characteristics;

pooling the word-level hidden layer characteristics to obtain the sentence-level hidden layer characteristics.

7. The phoneme recognition method of claim 3, wherein the phoneme recognition model is trained based on the following steps:

fixing parameters of a feature extraction layer of the first recognition model;

inputting sample voices of various languages into the feature extraction layer of the first recognition model to obtain second phoneme hidden layer features output by the feature extraction layer of the first recognition model;

inputting the second phoneme hidden layer characteristics to a current phoneme classification layer to obtain a second phoneme prediction result output by the current phoneme classification layer; the current phoneme classification layer is constructed on the basis of phoneme nodes obtained by screening from the second recognition model;

and performing parameter iteration on the current phoneme classification layer based on the difference between the phoneme level label and the second phoneme prediction result to obtain the phoneme recognition model.

8. A phoneme recognition apparatus, comprising:

a determination unit configured to determine a speech to be recognized;

the recognition unit is used for inputting the speech to be recognized into the phoneme recognition model to obtain a phoneme recognition result output by the phoneme recognition model;

the phoneme recognition model is obtained by training a first recognition model based on sample voices of multiple languages and phoneme level labels of the sample voices, the first recognition model is obtained by screening phoneme nodes under a second recognition model based on similarity between phonemes corresponding to the phoneme nodes under the second recognition model, and the second recognition model comprises phoneme nodes corresponding to the multiple languages respectively.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the phoneme recognition method of any one of claims 1 to 7 when executing the program.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the phoneme recognition method of any one of claims 1 to 7.

Images