CN114974228B - Rapid voice recognition method based on hierarchical recognition - Google Patents

Abstract

A rapid speech recognition method based on hierarchical recognition is characterized in that speech with different difficulties is shunted, models are disassembled step by step, and the models with different levels are used for processing speech cases with different difficulties; the invention solves the problem of limited computing resources required by large model modeling by a hierarchical reasoning mode, greatly reduces the complexity of the whole reasoning, saves the computing resources and reduces the service delay.

Description

A Fast Speech Recognition Method Based on Hierarchical Recognition

technical field

The invention relates to the field of voice recognition, in particular to a fast voice recognition method based on hierarchical recognition.

Background technique

With the continuous improvement of computing power and the accumulation of data, the effect of the speech recognition system has made significant progress. The end-to-end modeling method represented by CTC and encoder-decoder makes full use of massive data and has a stronger Modeling ability. In the field of speech recognition, a Conformer model using convolution enhancement was proposed by Google in 2020, which has continuously refreshed the accuracy of speech recognition and has become a conventional method for acoustic modeling of speech recognition. With massive training data, the multi-layer Conformer model has more parameters and has been proven to have stronger modeling capabilities. Usually, the 12-24 layer Conformer model has stronger modeling ability with the increase of the number of model layers and the blessing of massive training data. However, as the number of parameters increases, the greater the amount of calculation in the model reasoning process when performing speech recognition, the more energy consumption, delay, and resources required, which limits the use of large models in actual scenarios. in the application. In order to enable the deep Conformer network to be applied to speech recognition tasks, methods such as reducing the number of neurons in the hidden layer or using matrix decomposition are usually used to reduce the amount of parameters and calculations, but these methods usually also bring certain performance losses. At the same time, the computational complexity of model reasoning in the speech recognition process still increases linearly with the increase in the number of Conformer layers.

Contents of the invention

The purpose of the present invention is to provide a fast speech recognition method based on hierarchical recognition, so as to solve the aforementioned problems in the prior art.

In order to achieve the above object, the technical scheme adopted in the present invention is as follows:

A fast speech recognition method based on hierarchical recognition, comprising the following steps:

S1. Divide the deep network of the Conformer model, divide the deep network with R layers into a shallow network every M layers in the order from the bottom layer to the top layer, and divide the deep network into a shallow network in each of the shallow networks. The layer recognition network leads a tap to be decoded by a shallow Decoder to form F Conformer models with a shallow network; where R and M represent the number of network layers, F represents the number of Conformer models with a shallow network, F=R/ M;

S2. According to the order from the bottom layer to the top layer in the deep network, classify and sort the formed shallow network to form a speech recognition model with F shallow networks, and input speech according to the speech recognition model The level of the shallow network described in is identified step by step, and the degree of difficulty of the input speech is judged;

S3. According to the entropy of the input speech through the shallow network, judge the difficulty of the input speech, and judge whether the input speech needs to be calculated and recognized by the shallow network at the next level; the shallow network The smaller the entropy value of the output, the more certain the speech recognition result output by the shallow network is, and the smaller the ambiguity of the speech recognition result; conversely, the larger the entropy value is, the more uncertain the speech recognition result output by the shallow network is. The greater the ambiguity of the speech recognition results, the need for a network with stronger modeling capabilities for recognition.

A fast speech recognition method based on hierarchical recognition, comprising the following steps

S1. Divide the deep network of the Conformer model, divide the deep network with R layers into a shallow network every M layers in the order from the bottom layer to the top layer, and divide the deep network into a shallow network in each of the shallow networks. The layer recognition network leads a tap to be decoded by a shallow Decoder to form F Conformer models with a shallow network; where R and M represent the number of network layers, F represents the number of Conformer models with a shallow network, F=R /M;

S2. According to the order from the bottom layer to the top layer in the deep network, classify and sort the formed shallow network to form a speech recognition model with F shallow networks, and input speech according to the speech recognition model The level of the shallow network described in is identified step by step, and the degree of difficulty of the input speech is judged;

S3. According to the order of the levels of the shallow networks from small to large, select two adjacent levels of shallow networks, and judge the consistency of the speech recognition results output by the two shallow networks; when adjacent When the speech recognition results of the two shallow networks pass the consistency judgment, the acoustic modeling is considered complete; otherwise, a network with stronger modeling capabilities is required for speech recognition.

Preferably, the formula for calculating the entropy of the input speech through the shallow network is:

Where E represents the entropy value, L represents the number of speech frames, N represents the total number of units in the input speech that need to be recognized, p _li represents the probability that the i-th speech recognition unit in the input speech performs speech recognition in the l frame.

Preferably, the basis for judging the degree of speech difficulty described in step S3 is: setting the threshold value of entropy, when the entropy value of the output of the shallow layer network at the fth level is less than the threshold value, determine the difficulty of the input speech degree of ease, it is determined that the voice recognition result output by the input voice through the shallow network at level f is the final result; otherwise, the input voice continues to pass through the shallow network for voice recognition step by step until the shallow network The entropy value of the layer network is less than the threshold value or the level of the shallow network is F; where f represents the level of the shallow network in the speech recognition model.

Preferably, the basis for judging the degree of speech difficulty described in step S3 is: setting the recognition result difference threshold value threshold, when the speech recognition result difference between the two levels of the shallow network is less than the difference threshold value, that is, diff(result1 ,result2)<threshold, it is considered that the acoustic modeling has been completed; if the difference between the two speech recognition results is greater than the difference threshold, that is, when diff(result1,result2)≥threshold, it is considered that the speech recognition composed of the current shallow network The model’s ability to model speech is insufficient, and continue to perform speech recognition step by step through the shallow model until the speech recognition results of two adjacent shallow networks pass the consistency judgment or the level of the shallow network for class F.

Preferably, in step S3, when the speech recognition results of the two adjacent shallow networks pass the consistency judgment, the speech recognition results of the current two levels of the shallow networks are linearly weighted as the final output result.

Preferably, in step S1, the deep network of the Conformer model is divided in the following manner: for the deep network of the R layer, every M layer leads a tap to be decoded using a shallow Decoder, and a total of F shallow Decoders are set to form F a shallow network.

Preferably, for the speech recognition model with a shallow network formed in step S2, R/M branches are used to perform multi-task joint training on the shallow network with progressive levels.

Preferably, for models with different network depths, all shallow networks share parameters.

The beneficial effects of the present invention are: the present invention discloses a fast speech recognition method based on hierarchical recognition, which splits the speech of different difficulty and disassembles the model step by step, so that the models of different levels can handle cases of different degrees of difficulty ; The present invention solves the problem of limited computing resources required for large model modeling through hierarchical reasoning, greatly reduces the complexity of overall reasoning, and reduces service delays while saving computing resources.

Description of drawings

Fig. 1 is the fast speech recognition flowchart of hierarchical recognition;

Fig. 2 is the fast speech recognition structural diagram of hierarchical recognition;

Fig. 3 is a judgment criterion structural diagram of hierarchical recognition;

Fig. 4 is a structure diagram of the result measurement of hierarchical recognition.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention, and are not intended to limit the present invention.

A fast speech recognition method based on hierarchical recognition, which divides the multi-layer Conformer speech model into multiple levels, judges the speech recognition difficulty of the input from the perspective of the speech model, and judges the speech recognition level according to the speech recognition difficulty; as shown in Figure 1, including The following steps:

S1. Divide the deep network of the Conformer model, divide the deep network with R layers into a shallow network every M layers in the order from the bottom layer to the top layer, and divide the deep network into a shallow network in each of the shallow networks. The layer identification network leads a tap to decode with a shallow Decoder to form R/M Conformer models with a shallow network;

The specific implementation method is as follows: for the deep network of R layer, every M layer draws a tap and uses a shallow Decoder for decoding, then a total of F shallow Decoders are set, where R and M represent the number of network layers, and F represents the The number of Conformer models of the shallow network, F=R/M. As shown in Figure 2, for the 12-layer Conformer model, a shallow Decoder is connected every 3 layers, and 4 shallow Decoders are set up from the bottom layer to the top layer in order to form 4 Conformer models with shallow networks.

S2. According to the order from the bottom layer to the top layer in the deep network, classify and sort the formed shallow network, form a speech recognition model with R/M shallow networks, and use R/M branch voices Perform multi-task joint training on shallow networks with progressive levels, and share parameters in each shallow network;

S3. Recognize the input voice step by step according to the level of the shallow network in the voice recognition model, and judge the difficulty of the input voice;

S41. As shown in FIG. 3 , judge the degree of difficulty of the speech according to the entropy of the speech passing through the shallow network, and judge whether the speech needs to be calculated and recognized by the shallow network at the next level;

The formula for calculating the entropy of the speech passing through the shallow network is:

Wherein E represents the entropy value, L represents the speech frame number, N represents the unit total amount that needs to carry out speech recognition in the input speech, p _li represents the i-th speech recognition unit in the input speech and carries out the probability of speech recognition at the l frame; The smaller the entropy value of the shallow network output, the more certain the speech recognition result output by the shallow network is, and the smaller the ambiguity of the speech recognition result; on the contrary, the larger the entropy value is, the less accurate the speech recognition result output by the shallow network is. It is determined that the greater the ambiguity of the speech recognition results, the need for a network with stronger modeling capabilities to recognize.

Setting the threshold value of entropy, when the entropy value of the shallow layer network output of the fth level is less than the threshold value, determine the degree of difficulty of the speech, and determine that the speech is output through the shallow layer network output of the fth level The speech recognition result is the final result; otherwise, the speech continues to pass through the shallow network for speech recognition step by step, until the entropy value of the shallow network is less than the threshold or the level of the shallow network is the Fth level; where f represents the level of the shallow network in the speech recognition model.

Using the speech difficulty judgment method described in step S41 above, it is still possible that even if the entropy value of the current shallow network output is less than the threshold value, the corresponding output speech recognition result will still be wrong; based on this, the following speech can also be used Difficulty Judgment Method:

S42. As shown in FIG. 4 , according to the order of the levels of the shallow networks from small to large, select adjacent two levels of shallow networks, and the consistency of the speech recognition results output by the two shallow networks Judging; setting the recognition result difference threshold threshold, when the speech recognition result difference of the two levels of the shallow network is less than the difference threshold, that is

When diff(result1, result2)<threshold, it is considered that the acoustic modeling is complete, and the recognition result tends to converge, and the speech recognition results of the current two levels of the shallow network are used as the final result of the output through linear weighting; if the two When the difference between two speech recognition results is greater than the difference threshold, that is, when diff(result1, result2)≥threshold, it is considered that the speech recognition model formed by the current shallow network is not capable of modeling speech, and continues to pass through the shallow model Carry out speech recognition step by step until the speech recognition results of two adjacent shallow networks pass the consistency judgment or the level of the shallow networks is the Fth level.

Compared with step S41, the calculation amount of the speech difficulty judgment method described in the above step S42 is larger, because at least the first two levels of speech recognition must be calculated, but the accuracy of recognition is more guaranteed, and the weighting of the two levels of recognition results Fusion can also play a certain role in model averaging, which can improve the decoding accuracy of the speech recognition model at the same level.

Example

In actual speech recognition tasks, most of the speech recognition tasks are simple cases with clean background and pronunciation. For such speech recognition tasks, shallow networks can be used to complete speech recognition tasks.

Taking the 12-layer deep network of the Conformer model as an example, assuming that 20% of the selected speech detection cases need to use the deep network for speech recognition, even if the judgment method in the above step S42 is used, it can save about 80%*50 % = 40% of the calculation amount, the income is considerable;

Taking the 24-layer deep network of the Conformer model as an example, 20% of the cases in the above detection speech need to use the deep network for speech recognition. Assume that 20% of the case reasoning needs to use the 24-layer network, and 80% of the case reasoning needs to use the 24-layer network. If the network does not exceed 12 layers, the increased complexity will not exceed 100%*20%=20% of the calculation amount, which is much smaller than the 100% increase in the calculation amount of all cases directly through the 24-layer network.

By adopting the above-mentioned technical scheme disclosed by the present invention, the following beneficial effects are obtained:

The invention discloses a fast speech recognition method based on hierarchical recognition, which divides speech with different difficulties and disassembles the model step by step, so that models of different levels can handle cases with different degrees of difficulty; the present invention uses hierarchical reasoning This method solves the problem of limited computing resources required for large model modeling, greatly reduces the complexity of overall reasoning, and reduces service delays while saving computing resources.

The above is only a preferred embodiment of the present invention, it should be pointed out that, for those of ordinary skill in the art, without departing from the principle of the present invention, some improvements and modifications can also be made, and these improvements and modifications can also be made. It should be regarded as the protection scope of the present invention.

Claims (1)

1. A quick speech recognition method based on hierarchical recognition is characterized by comprising the following steps:

s1, dividing deep networks of a Conformer model, dividing the deep networks with R layers into shallow networks every M layers according to the sequence from a bottom layer to a top layer, and leading out a tap from a last layer identification network in each shallow network to decode by using a shallow Decoder to form F Conformer models with shallow networks; wherein R and M represent the number of network levels, F represents the number of Conformer models with shallow networks, F = R/M;

s2, according to the sequence from the bottom layer to the top layer in the deep layer network, carrying out level division and sequencing on the formed shallow layer network to form a voice recognition model with F shallow layer networks, carrying out level-by-level recognition on input voice according to the level of the shallow layer network in the voice recognition model, and judging the difficulty level of the input voice;

s3, judging the difficulty degree of the input voice according to the entropy of the input voice passing through the shallow network, and judging whether the input voice needs to be calculated and identified by the shallow network at the next level; the smaller the entropy value of the shallow network output is, the more certain the speech recognition result of the shallow network output is, the smaller the ambiguity of the speech recognition result is; on the contrary, the larger the entropy value is, the more uncertain the speech recognition result output by the shallow network is, the larger the ambiguity of the speech recognition result is, the network with stronger modeling capability is required to be recognized;

wherein, the calculation formula of the entropy of the input voice passing through the shallow network is as follows:

wherein E represents entropy, L represents number of speech frames, N represents total number of units required for speech recognition in input speech, and p _li Representing the probability of the ith speech recognition unit in the input speech to perform speech recognition in the l frame;

the judgment basis of the speech difficulty degree in the step S3 is as follows: setting an entropy threshold, when the entropy value output by the shallow network of the f-th level is smaller than the threshold, determining the difficulty degree of the input voice, and judging that the voice recognition result of the input voice output by the shallow network of the f-th level is a final result; otherwise, the input voice continues to be subjected to voice recognition step by step through the shallow network until the entropy value of the shallow network is smaller than the threshold value or the level of the shallow network is F level; wherein f represents a level of the shallow network in the speech recognition model;

alternatively, the method comprises the steps of,

s1, dividing deep networks of a Conformer model, dividing the deep networks with R layers into shallow networks every M layers according to the sequence from a bottom layer to a top layer, leading out a tap from the last layer of identification network in each shallow network, and decoding by using a shallow Decoder to form F Conformer models with shallow networks; wherein, R and M represent the number of network layers, F represents the number of Conformer models with shallow networks, and F = R/M;

s2, according to the sequence from the bottom layer to the top layer in the deep layer network, carrying out level division and sequencing on the formed shallow layer network to form a voice recognition model with F shallow layer networks, carrying out level-by-level recognition on input voice according to the level of the shallow layer network in the voice recognition model, and judging the difficulty level of the input voice;

s3, selecting two adjacent shallow networks according to the order of the shallow networks from small to large in level, and judging the consistency of the voice recognition results output by the two shallow networks; when the voice recognition results of two adjacent shallow networks pass consistency judgment, the acoustic modeling is considered to be complete; otherwise, a network with stronger modeling capability is needed for voice recognition;

the judgment basis of the difficulty level of the voice in step S3 is as follows: setting a recognition result difference threshold, namely when the difference of the speech recognition results of the shallow networks of two levels is smaller than the difference threshold, namely diff (result 1, result 2) < threshold, the acoustic modeling is considered to be complete; if the difference between the two speech recognition results is greater than the difference threshold value, namely diff (result 1, result 2) is greater than or equal to threshold, the speech recognition model formed by the current shallow network is considered to have insufficient modeling capacity for speech, and the speech recognition is continuously carried out step by step through the shallow network upwards until the speech recognition results of the two adjacent shallow networks are judged through consistency or the level of the shallow network is the F-th level;

in step S3, when the voice recognition results of two adjacent shallow networks pass consistency judgment, the voice recognition results of the current two levels of shallow networks are used as final output results through linear weighting;

in the step S1, the deep network of the transformer model is divided in the following manner: for the deep network of the R layer, a tap is led out every M layers and decoded by using a shallow Decoder, and F shallow decoders are arranged to form F shallow networks;

aiming at the voice recognition model with the shallow network formed in the step S2, performing multi-task joint training on the shallow network with progressive levels by adopting R/M branches;

for models with different network depths, all parameters are shared in the shallow network.

Images