CN114464201A - A single-channel speech enhancement method based on attention mechanism and convolutional neural network - Google Patents

Abstract

The invention discloses a single-channel speech enhancement method based on an attention mechanism and a convolutional neural network, which includes a training phase and an enhancement phase: the training phase: firstly synthesizing noise and pure speech into noisy speech, and then characterizing the noisy speech Extraction, and then send the pure speech and the noisy speech after feature extraction into the neural network model to learn the mapping relationship between the noisy speech and the pure speech. After the training phase, save the trained model; the enhancement phase : First perform feature extraction on noisy speech, then send it into the trained model for speech enhancement, and finally output the enhanced speech. The single-channel speech enhancement method based on the attention mechanism and the convolutional neural network of the present invention can directly enhance the noisy speech in the time domain, which can not only save the calculation time of the Fourier transform, but also preserve the enhanced speech as much as possible. Phase information can achieve better noise reduction effect.

Description

A single-channel speech enhancement method based on attention mechanism and convolutional neural network

technical field

The invention belongs to the technical field of speech processing, and in particular relates to a single-channel speech enhancement method based on an attention mechanism and a convolutional neural network.

Background technique

Speech enhancement is a technology that restores pure speech from noisy speech. It is divided into single-channel and multi-channel technology according to the number of channels. According to the different processing domains, it is divided into time domain speech enhancement and frequency domain speech enhancement. According to different processing methods, it can also be divided into speech enhancement methods based on signal knowledge and speech enhancement methods based on machine learning. With the rapid development of computer technology, deep learning methods are helpful for solving problems in various fields, so speech enhancement methods based on deep learning are becoming a research hotspot in the field of speech.

In the traditional single-channel speech enhancement method research, it is usually necessary to make certain assumptions about the relationship between the noise signal and the speech signal. It is precisely because of its prerequisites that the performance of the system is limited, and the applicability and enhancement effect are poor. The speech enhancement method based on deep learning does not need to make any assumptions about speech signals and noise signals. Through a large amount of training data, the relationship between pure speech and noise can be directly learned, which overcomes the inherent defects of traditional speech enhancement algorithms and achieves better results. The denoising level and generalization ability.

Speech enhancement methods based on convolutional neural networks can only capture the local information of speech, and cannot efficiently utilize the global information of speech, which is unfavorable for speech enhancement. At the same time, although the neural network based on the attention mechanism can extract the contextual information of speech, it cannot process the local information of speech well. In addition, most popular speech enhancement methods use short-time Fourier transform to convert noisy speech to the frequency domain for processing. Speech enhancement in the frequency domain generally ignores the phase information of the enhanced speech and directly uses the noisy speech. The phase synthesis enhances speech, and the loss of phase information also restricts the effect of enhanced speech.

SUMMARY OF THE INVENTION

In view of the deficiencies in the prior art, the technical problem to be solved by the present invention is to provide a single-channel speech enhancement method based on an attention mechanism and a convolutional neural network, which directly enhances the noisy speech in the time domain, which not only saves The calculation time of the Fourier transform can also preserve the phase information of the enhanced speech as much as possible, and can achieve a better noise reduction effect.

In order to solve the above-mentioned technical problems, the present invention realizes through the following technical solutions:

The present invention provides a single-channel speech enhancement method based on an attention mechanism and a convolutional neural network, including a training phase and an enhancement phase:

Training stage: First, the noise and pure speech are synthesized into noisy speech, then feature extraction is performed on the noisy speech, and then the pure speech and the noisy speech after feature extraction are sent to the neural network model to learn the noisy speech and pure speech After the training phase is over, save the trained model;

Enhancement stage: First, extract the features of the noisy speech, then send it into the trained model for speech enhancement, and finally output the enhanced speech.

Preferably, the neural network model includes an encoding module, a noise reduction module and a decoding module, the noisy speech is first processed by the encoding module, and then sent to the noise reduction module, and then the output of the encoding module is multiplied by the output of the noise reduction module and sent to the Enter the decoding module to obtain the enhanced speech.

Further, the output of the encoding module is first sent to the attention module in the noise reduction module for processing, and then processed by the convolution module and output; in the attention module, the speech features are subjected to layer normalization and multi-head self-attention. After processing and Dropout, the output of the encoding module is added point by point and output; in the convolution module, the speech features are subjected to layer normalization, point-by-point convolution, activation function GLU, depth convolution, batch normalization, activation function Swish, point-by-point convolution and dropout processing and the output of the attention module are added point-by-point and output.

Further, in the decoding module: first, the size of the speech features becomes [B, 128, K, S] after two-dimensional deconvolution, batch normalization and activation function PReLU processing, and then the speech features are overlapped and added. The size becomes [B, 128, L], and finally the speech feature is restored to [B, 1, L] after one-dimensional deconvolution processing.

From the above, the single-channel speech enhancement method based on the attention mechanism and the convolutional neural network of the present invention has at least the following beneficial effects:

1. The method based on the deep learning of the present invention enhances the speech in the time domain, which saves the calculation time of the Fourier transform and at the same time preserves the phase information of the target enhanced speech as much as possible. In addition, the attention module and convolution module proposed in the present invention combine the advantages of attention and convolution, break through the limitation that the convolutional neural network cannot be calculated in parallel, and use the attention mechanism to give it the ability to process long speech sequences The ability to achieve the effect of local and global modeling of speech information.

2. The present invention is based on the deep learning method, and adopts the framework based on the attention mechanism and the convolutional neural network to enhance the noisy speech from the time domain. In this process, the network structure is used to directly learn the mapping relationship from noisy speech to pure speech. The local and global information of noisy speech signals can be fully extracted by using convolutional neural networks and attention mechanisms. At the same time, the present invention directly enhances the noisy speech in the time domain, which not only saves the calculation time of Fourier transform but also preserves the phase information of the enhanced speech as much as possible, and can achieve better noise reduction effect.

The above description is only an overview of the technical solution of the present invention, in order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the description, and in order to make the above and other objects, features and advantages of the present invention more obvious and easy to understand , the following detailed description is given in conjunction with the preferred embodiments and in conjunction with the accompanying drawings.

Description of drawings

In order to describe the technical solutions of the embodiments of the present invention more clearly, the accompanying drawings of the embodiments will be briefly introduced below.

Fig. 1 is the flow chart of the speech enhancement method of the present invention;

Fig. 2 is a neural network model diagram of the present invention;

Fig. 3 is the structural diagram of the coding module of the present invention;

Fig. 4 is the block process diagram of the present invention;

Fig. 5 is the structure diagram of the attention module of the present invention;

Fig. 6 is the structure diagram of the convolution module of the present invention;

FIG. 7 is a structural diagram of a decoding module of the present invention.

Detailed ways

The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. As a part of this specification, the principles of the present invention will be illustrated by examples. Other aspects, features and advantages of the present invention will become apparent from the detailed description. In the figures to which reference is made, the same reference numerals are used for the same or similar parts in different figures.

The invention aims to make full use of the advantage of the deep learning method without making any assumptions about the noise signal and the speech signal, and directly enhance the noisy speech in the time domain. The phase information of the enhanced speech is preserved as much as possible while saving the calculation time of the Fourier transform. And use the convolutional neural network and attention mechanism to better extract the local and global features of the speech signal, and strive to obtain a better noise reduction effect. The specific execution steps of the method of the present invention are as follows:

The speech enhancement of the present invention is carried out in two stages, including a training stage and an enhancement stage, as shown in FIG. 1 .

In the training stage, the noise and pure speech are first synthesized into noisy speech, and then feature extraction is performed on the noisy speech, and then the pure speech and the noisy speech after feature extraction are sent into the neural network model to learn the noisy speech and pure speech. the mapping relationship between them. After the training phase, save the trained model.

In the enhancement stage, firstly extract the features of the noisy speech, then send it into the trained model for speech enhancement, and finally output the enhanced speech.

The neural network model proposed by the present invention is shown in FIG. 2 . The model of the present invention consists of three parts, an encoding module, a decoding module and a noise reduction module. The function of the codec is similar to the Fourier transform, which is to transform the speech signal into a high-dimensional space for the noise reduction module to perform noise reduction processing. The noisy speech is first processed by the encoding module, and then sent to the noise reduction module, then the output of the encoding module is multiplied by the output of the noise reduction module, and finally sent to the decoding module to obtain the enhanced speech.

The internal structure of the encoding module is shown in Figure 3, and the noisy speech of size [B, 1, L] is fed into the network model. Among them, B represents BatchSize, which represents the amount of data that needs to be trained in the neural network each time, and L represents the length of the speech signal. First, after one-dimensional convolution processing, the size becomes [B, 128, L]; then after block processing, the size becomes [B, 128, K, S]. Finally, after two-dimensional convolution, batch normalization and activation function PReLU processing, the size of the output of the encoding module is [B, 256, K, S].

The block division process is shown in Figure 4. First, the one-dimensional speech sequence is divided into frames with a frame length of 2D and a frame shift of D. Then, each frame of speech data is treated as a separate data block and these data blocks are stacked together to form a two-dimensional data matrix. where S represents the number of divisible blocks of the sequence of length L, and K represents the frame length equal to 2D.

The output of the encoding module is first sent to the attention module in the noise reduction module for processing, and then processed by the convolution module for output. During the processing of the noise reduction module, the dimension of the speech feature is still [B, 256, K, S]. The attention module is shown in Figure 5. In the attention module, the speech features are processed by layer normalization, multi-head self-attention and dropout, and the output of the encoding module is added point by point and then output.

The convolution module is shown in Figure 6. In the convolution module, the speech features undergo layer normalization, pointwise convolution, activation function GLU, depthwise convolution, batch normalization, activation function Swish, pointwise convolution and After Dropout processing, the output of the attention module is added point by point and output.

The output of the encoding module and the output of the noise reduction module are multiplied and sent to the decoding module, the size is [B, 256, K, S]. The decoding module is shown in Figure 7. First, the size of the speech features becomes [B, 128, K, S] after being processed by two-dimensional deconvolution, batch normalization and activation function PReLU, and then the speech features are overlapped and added. The size becomes [B, 128, L], and finally the speech feature is restored to [B, 1, L] after one-dimensional deconvolution processing. At this point, the processing of the neural network model ends, and the enhanced speech is output.

Among them, overlap-add is the inverse process of blocking. The overlapping parts of two adjacent data blocks in the two-dimensional data matrix will be merged by means of averaging, and after all blocks are merged, the two-dimensional matrix will be transformed into a one-dimensional speech sequence.

Choose VoiceBank and DEMAND voice data as examples for experiment. The dataset we chose consists of 30 speakers, of which 28 are used for training and 2 are used for testing. The training data consists of 11572 pairs of noisy speech and pure speech, and the test data consists of 824 speech pairs.

The loss function adopted in the present invention combines the two aspects of the time domain and the frequency domain. The loss function in the time-frequency domain can supervise the model to learn more information, resulting in higher speech intelligibility and perceptual quality, which is defined as:

loss=α*loss _F +(1-α)loss _T

表示纯净语音和增强语音的频谱。r和i表示复值的实部和虚部。T表示帧数，F表示频点数，N表示采样数。α是一个可调参数，本发明中设置为0.2。where X and

Represents the spectrum of clean and enhanced speech. r and i represent the real and imaginary parts of the complex value. T represents the number of frames, F represents the number of frequency points, and N represents the number of samples. α is an adjustable parameter, which is set to 0.2 in the present invention.

In order to verify the effectiveness of the method proposed in the present invention, the proposed method is compared with other speech enhancement methods. Comparing methods include SEGAN model based on generative adversarial network and Wave-U-Net model based on U-Net.

Table 1 Comparison of the present invention and different algorithms under different evaluation indexes

Evaluation Metrics\Algorithms Noisy SEGAN Wave-U-Net this invention PESQ 1.97 2.16 2.40 2.42 CSIG 3.35 3.48 3.52 3.82 SSNR 1.68 7.73 9.97 10.2

It can be seen from the table that the method proposed by the present invention is higher than the comparison algorithm in terms of PESQ, CSIG and SSNR, indicating that the method proposed by the present invention has certain advantages.

The method based on the deep learning of the present invention enhances the speech in the time domain, saves the calculation time of the Fourier transform and preserves the phase information of the target enhanced speech as much as possible. In addition, the attention module and convolution module proposed in the present invention combine the advantages of attention and convolution: while breaking the limitation that the convolutional neural network cannot be calculated in parallel, the attention mechanism enables it to process longer speech sequences The ability to achieve the effect of local and global modeling of speech information.

Through experimental verification, the values of the PESQ, CSIG and SSNR indicators between the method proposed by the present invention and the other two methods are compared. See Table 1, the single-channel speech enhancement algorithm based on the present invention achieves better noise reduction performance than the other two algorithms. It shows that the method proposed by the present invention has certain advantages.

The above descriptions are only the preferred embodiments of the present invention, of course, it cannot limit the scope of rights of the present invention. It should be pointed out that for those skilled in the art, without departing from the principles of the present invention, Several improvements and changes are made, and these improvements and changes are also regarded as the protection scope of the present invention.

Claims (4)

1. a single-channel speech enhancement method based on attention mechanism and convolutional neural network, is characterized in that, comprises training phase and enhancement phase: Training stage: First, the noise and pure speech are synthesized into noisy speech, then feature extraction is performed on the noisy speech, and then the pure speech and the noisy speech after feature extraction are sent to the neural network model to learn the noisy speech and pure speech After the training phase is over, save the trained model; Enhancement stage: First, extract the features of the noisy speech, then send it into the trained model for speech enhancement, and finally output the enhanced speech. 2. the single-channel speech enhancement method based on attention mechanism and convolutional neural network as claimed in claim 1, is characterized in that, described neural network model comprises coding module, noise reduction module and decoding module, and the noisy speech first passes through. The encoding module is processed, and then sent to the noise reduction module, then the output of the encoding module is multiplied by the output of the noise reduction module, and finally sent to the decoding module to obtain the enhanced speech. 3. the single-channel speech enhancement method based on attention mechanism and convolutional neural network as claimed in claim 2 is characterized in that, the output of described coding module is first sent to the attention module in the noise reduction module for processing, and then After being processed by the convolution module, it is output; in the attention module, the speech features are processed by layer normalization, multi-head self-attention and dropout, and the output of the encoding module is added point by point; in the convolution module, the speech features are After layer normalization, pointwise convolution, activation function GLU, depthwise convolution, batch normalization, activation function Swish, pointwise convolution and Dropout processing, the output of the attention module is added point by point and output. 4. the single-channel speech enhancement method based on attention mechanism and convolutional neural network as claimed in claim 2, is characterized in that, in decoding module: first, speech feature is through two-dimensional deconvolution, batch normalization and After the activation function PReLU is processed, the size becomes [B, 128, K, S], and then the size of the speech feature becomes [B, 128, L] after overlapping and addition, and finally the size of the speech feature is restored to the one-dimensional deconvolution process. [B,1,L].

Images