CN112735456A - Speech enhancement method based on DNN-CLSTM network - Google Patents

Abstract

The invention is a speech enhancement method based on deep neural network and residual long short-term memory network (DNN-CLSTM). In this method, the speech amplitude feature obtained by spectral subtraction and the speech Mel cepstral coefficient (MFCC) feature obtained by fast Fourier transform are input into the DNN‑CLSTM network model to achieve the purpose of speech enhancement. First, time-frequency masking and windowing are performed on the noisy speech, and the amplitude and phase characteristics of the noisy speech are obtained by using fast Fourier transform, and the noise amplitude of the noisy speech is estimated. The estimated noise signal amplitude is subtracted from the speech amplitude to obtain the spectrally subtracted speech signal amplitude as the first feature input to the neural network. Secondly, perform fast Fourier transform (FFT) on the noisy speech to obtain the spectral line energy of the speech signal and then obtain the MFCC feature of the noisy speech as the second characteristic of the speech signal. The above two features are input into the DNN-CLSTM network for training to obtain a network model, and the minimum mean square error (MMSE) loss function evaluation index is used to evaluate the effectiveness of the model. Finally, input the actual noisy speech set into the trained speech enhancement network model, predict the enhanced estimated amplitude and MFCC, and use the inverse Fourier transform to obtain the final enhanced speech signal. The present invention has high fidelity of speech.

Description

A speech enhancement method based on DNN-CLSTM network

technical field

The invention belongs to the technical field of speech enhancement, in particular to a speech enhancement method based on a DNN-CLSTM network.

Background technique

As one of the main methods of information transmission, voice has been widely used in life. With the development of technology, voice not only plays the role of information transmission between people, but also is widely used in human-computer interaction. voice signal. However, in our communication process, speech signals are often accompanied by a large number of noise signals, such as factory noise, car noise or background noise such as restaurant noise. A speech signal containing a lot of noise will cause a lot of interference when the receiver extracts the useful information contained in the speech signal. In response to this problem, speech signal enhancement technology has received extensive attention.

Speech enhancement refers to the process of separating noise and speech signal when real speech is disturbed by noise. Speech enhancement technology has been widely used, such as mobile communication, speech recognition and many other fields. The main purpose of speech enhancement technology is to improve speech quality and speech intelligibility. At present, speech enhancement methods are mainly divided into three types: spectral subtraction, subspace algorithm and algorithm based on statistical model. With the development of deep learning, neural networks have been applied to the field of speech enhancement.

Spectral subtraction shown in Figure 1 is one of the earliest denoising techniques in speech enhancement techniques. Spectral subtraction denoising is based on the following principle: Assuming that the noise is additive noise, that is, y(m)=x(m)+n(m), where y(m) is the signal containing noise, and x(m) is the pure speech signal, n(m) is additive noise; the pure speech signal can be obtained by subtracting an estimate of the noise spectrum from the noise-containing speech signal. The premise of this assumption is that the noise signal is stationary, so that between speech segments in which the target signal does not exist, the noise signal can be estimated and updated.

Spectral subtraction is a relatively simple speech enhancement algorithm. Its principle is to subtract the estimated noise amplitude spectrum value from the amplitude spectrum value of the input mixed speech signal, and use the insensitivity of the human ear to phase to subtract the phase before the spectrum subtraction. The angle information is directly used in the spectrally subtracted information to synthesize the final spectrally subtracted speech signal. Since spectral subtraction only contains one Fourier transform and inverse Fourier transform, it is computationally less expensive and easy to implement. But in reality, many noises are unstable signals, so after using spectral subtraction to enhance the speech signal, there is often a lot of musical noise in the enhanced speech signal, which leads to the distortion of the speech signal, which makes the intelligibility of the signal and the speech quality. poor.

SUMMARY OF THE INVENTION

The purpose of the present invention is to solve the problems of voice signal distortion, poor signal intelligibility and poor voice quality in the voice enhancement process based on spectral subtraction. In order to achieve the above-mentioned purpose, the present invention provides a kind of speech enhancement method based on DNN-CLSTM network, it is characterized in that, comprises the following steps:

Step 1: Obtain at least two noisy speech signals. The noisy speech signal is formed by adding the pure speech signal and the noise signal:

y(m)=x(m)+n(m)

Among them, y(m) is the noisy speech signal containing noise, x(m) is the pure speech signal, n(m) is the noise signal, and m is the discrete time series;

Step 2: windowing by frame, obtaining the amplitude and phase of the pure speech signal and the noisy speech signal as the first feature: performing windowing and framing processing on the noisy speech signal, and using discrete Fourier transform to obtain the noisy speech The amplitude and phase of the signal; at the same time, in the speech signal segment that does not contain the target signal and only contains noise, the noise is estimated to obtain the amplitude of the noise signal;

Step 3, subtracting the noise signal amplitude from the amplitude of the noisy speech signal, thereby obtaining the spectrum minus the speech signal amplitude as the second feature;

Step 4, seek MFCC as the third feature;

Step 5: Establish a DNN-CLSTM network model for training;

Input the two features of speech signal amplitude and MFCC after noise spectrum reduction into the DNN-CLSTM network for training, and obtain the enhanced estimated amplitude and MFCC; Amplitude and pure MFCC values calculate their respective minimum mean square errors, and input the obtained errors as adjustment signals into the neural network to optimize the network, thereby obtaining a trained network.

The specific process of step four is:

(1) Preprocessing: Preprocessing includes pre-emphasis, framing, and windowing functions;

Pre-emphasis processing: implemented by a first-order high-pass filter, the transfer function of the filter is:

H(z)=1-az ^-1

Among them, a is the pre-emphasis coefficient, which is generally 0.98;

The result of the speech signal x(n) after pre-emphasis processing is:

y(n)=x(n)-ax(n-1)

Frame-by-frame windowing: the overlapping part between two adjacent frames, which is frame shift, is set to 10ms; windowing function: Hamming window windowing is performed on each frame of speech signal: y(n) is divided into After frame windowing, y _i (n) is obtained, which is defined as:

Among them, ω(n) is the Hamming window, and its expression is

Wherein, y _i (n) represents the ith frame of speech signal, n represents the number of samples, and L represents the frame length;

(2) Fast Fourier Transform (FFT)

Perform fast Fourier transform on each frame of speech signal y _i (n) to obtain the frequency spectrum of each frame signal, the expression is as follows:

Y(i,k)=FFT[y _i (n)]

where k represents the kth spectral line in the frequency domain;

(3) Calculate the spectral line energy

The energy E(i,k) of the spectral line of each frame of speech signal in the frequency domain is expressed as:

E(i,k)=[Y(i,k)] ²

(4) Calculate the energy passing through the Mel filter

The energy S(i,m) of the spectral line energy of each frame passing through the Mel filter is defined as:

Among them, N represents the number of FFT points;

The transfer function H _m (k) of each filter is

Among them, f(m) is the center frequency of the mth filter, m is the mth filter, and M is the number of filters;

(5) Calculate MFCC

After taking the logarithm of the energy of the Mel filter, the discrete cosine transform is calculated to obtain the MFCC characteristic parameters, as shown in the following formula:

where j is the spectral line after DCT.

3. The specific process of step 5 is:

(1) DNN network establishment

Input layer: take the spectrally subtracted speech amplitude and MFCC feature as input, and input it into the DNN network. The number of neurons in the input layer is 128;

Fully connected layer: set 32 nodes, set the drop rate to 0.5, and set the activation function to RELU;

Fully connected layer: set 128 nodes, set the drop rate value to 0.5, and set the activation function to RELU;

Fully connected layer: set 512 nodes, set the drop rate value to 0.5, and set the activation function to RELU;

(2) Multi-target feature fusion:

Combine the amplitude and MFCC features enhanced by the DNN network with the amplitude and MFCC features of the original noisy speech;

和

分别代表第k个空间领域中经过DNN预测的MFCC特征和语音幅值；

分别代表第k个空间领域中原始含噪语音的 MFCC特征和语音幅值；in

and

respectively represent the MFCC features and speech amplitude predicted by DNN in the kth spatial domain;

represent the MFCC features and speech amplitudes of the original noisy speech in the kth spatial domain, respectively;

(3) C-LSTM network:

(a) CNN:

Convolution layer: Convolve the results obtained by the DNN network, the number of nodes is set to 64 nodes, the step size is set to 1, the convolution kernel is set to 5*1, and the activation function is set to SELU;

BN layer: normalize the data;

Convolution layer: the number of nodes is set to 64 nodes, the step size is set to 1, the convolution kernel is set to 3*1, and the activation function is set to SELU;

BN layer: normalize the data

Convolutional layer: the number of nodes is set to 128 nodes, the step size is set to 1, the convolution kernel is set to 5*1,

(b) Residual network

Convolve the results obtained by the DNN network, the number of nodes is set to 128 nodes, the step size is set to 1, and the convolution kernel is set to 5*1;

After combining the data obtained by the residual network with the data obtained by the CNN network, the SELU activation function is used;

Max Pooling layer: step size is set to 1, pooling layer size is set to 2

(c) LSTM network:

The bidirectional network nodes of the long-short-term memory network are selected as 128 nodes, and the activation function is the sigmoid function.

(4) Output layer:

Two feedforward neural networks are used as output layers to output the predicted speech signal amplitude and MFCC; the network model uses Adam optimizer to optimize the network parameters; all convolutional layers use edge padding.

(5) Calculate the minimum mean square error objective function

分别代表第k个声学特征空间预测的MFCC特征向量和预测幅值特征

分别代表第k个声学特征空间纯净的MFCC 特征向量和纯净幅值特征。where T=2,

Represent the MFCC feature vector and predicted magnitude feature of the kth acoustic feature space prediction, respectively

represent the pure MFCC feature vector and pure amplitude feature of the kth acoustic feature space, respectively.

The advantages of the present invention are as follows: the enhanced speech signal is stable, has high fidelity and good speech quality.

The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

Description of drawings

Figure 1 is a flow chart of conventional spectral subtraction.

Figure 2 is a flow chart of the DNN-CLSTM network-based speech enhancement method in the training phase.

Figure 3 is a flow chart of the speech enhancement method based on DNN-CLSTM network in the testing phase.

FIG. 4 is a flow chart of MFCC calculation.

Figure 5 is a process architecture diagram of building a DNN-CLSTM neural network.

Figure 6 is the spectrogram of pure speech

Figure 7 is a spectrogram with noise

Figure 8 is the spectrogram processed by the DNN speech enhancement method

Figure 9 is the spectrogram processed by the CNN speech enhancement method

Figure 10 is the spectrogram processed by the LSTM speech enhancement method

Figure 11 is the spectrogram processed by the GRU speech enhancement method

Figure 12 is the spectrogram processed by the DNN-CLSTM speech enhancement method

Detailed ways

In order to overcome the defect of using spectral subtraction to enhance the speech signal, the enhanced speech signal often has a large amount of musical noise, which causes the speech signal to be distorted and makes the signal intelligibility and speech quality poor. A speech enhancement method based on DNN-CLSTM network (as shown in Figures 2 and 3), comprising the following steps:

Acquire two channels of noisy speech signals (you can also obtain more than two channels of noisy speech signals, which can be selected according to actual needs); the noisy speech signals are composed of pure speech signals and noise signals:

y(m)=x(m)+n(m)

Among them, y(m) is the speech signal containing noise, x(m) is the pure speech signal, and n(m) is the noise signal.

Training phase:

1. Framing and windowing

Obtain the amplitude and phase of the pure speech signal and the noisy speech signal, taking the sum processing of the noisy speech as an example:

Windowing and framing the noisy speech signal, and using discrete Fourier transform to obtain the amplitude and phase of the noisy speech signal as the first feature;

Framing a noisy speech signal using a Hamming window:

yw(m)= _w (m)y(m)= _w (m)[x(m)+n(m)]=xw(m)+ _nw (m)

The windowing operation is expressed in the frequency domain as:

Yw(f)= _W (f)*Y(f)= _Xw (f)+ _Nw (f)

Assuming that the signal is processed by windowing, the subscript w of the signal is omitted for simplicity.

Express Y _w (f) in polar form:

where |Y(f)| is the amplitude spectrum, and φ _y (f) is the phase signal Phase[Y(f)].

2. Noise estimation:

计算过程如下：In the speech signal segment that does not contain the target signal and only contains noise, the noise is estimated to obtain the noise signal amplitude; the present invention selects the first five frames of the speech signal as the noise signal segment. Since the magnitude spectrum |N(f)| of the noise is unknown, but |N(f)| can be replaced by an estimate of the average magnitude spectrum in the absence of speech activity, the phase _φn (f) of the noise can be determined by the phase φ _y (f) instead. When the speech signal is absent and there is only noise, the average noise amplitude spectrum is obtained

The calculation process is as follows:

Among them, |N _i (f)| is the frequency spectrum of the ith noise frame, and k is the number of frames in the pure noise signal period.

3. Spectral subtraction

The amplitude of the noise signal is subtracted from the amplitude of the noisy speech signal to obtain the amplitude of the spectrally subtracted speech signal as the second feature (the enhanced speech signal obtained by using spectral subtraction becomes the spectrally subtracted speech signal, and the corresponding amplitude is called the spectrum. The amplitude of the speech signal is subtracted to distinguish it from the enhanced speech signal finally obtained in this implementation);

Using the amplitude of the noisy speech signal to subtract the amplitude of the noise signal, the calculation process is as follows:

代表经过谱减后语音信号的幅值，|Y(f)|^b表示含噪语音信号的幅值，

表示代表噪声段的噪声统计的平均值。α表示谱减噪声系数。b 是幂指数，当指数b＝1时为幅度谱减法,当指数b＝2时，为功率谱减法。in,

represents the amplitude of the speech signal after spectral subtraction, |Y(f)| ^b represents the amplitude of the noisy speech signal,

Represents the mean value of the noise statistics representing the noise segment. α represents the spectral subtraction noise figure. b is the power exponent, when the exponent b=1, it is the amplitude spectrum subtraction, and when the exponent b=2, it is the power spectrum subtraction.

可能会为负值。通常幅度谱的值不应为负，为了避免

为负值，对差分谱进行半波整流，计算过程如下：Since the estimation of the noise signal may produce errors, resulting in the estimated signal amplitude spectrum

May be negative. Usually the value of the magnitude spectrum should not be negative, in order to avoid

is a negative value, half-wave rectification is performed on the difference spectrum, and the calculation process is as follows:

计算过程如下：After spectral subtraction, it is necessary to reduce the power of the spectrally subtracted speech signal, so as to obtain the amplitude of the speech signal after the spectral subtraction stage, that is, the amplitude of the spectrally subtracted speech signal.

The calculation process is as follows:

4. Extract MFCC as the second feature;

(1) Preprocessing

Pre-processing includes pre-emphasis, framing, and windowing functions. Pre-emphasis processing is realized by a first-order high-pass filter. The transfer function of the filter is:

H(z)=1-az ^-1

Among them, a is the pre-emphasis coefficient, which is generally 0.98

The result of the speech signal x(n) after pre-emphasis processing is:

y(n)=x(n)-ax(n-1)

Framing means that the speech signal is a non-stationary time-varying signal from the process of speech generation. The voice signal in a short time can be considered as a stable time-invariant signal, and the short time usually refers to between 10 and 30ms, and this paper takes 20ms. Therefore, short-term analysis technology is usually used to analyze and process the speech signal, and many frames are used to analyze its characteristic parameters. For frame shift, set to 10ms. The purpose of the windowing function is to reduce the spectral leakage in the frequency domain, and window processing is performed on each frame of speech signal. Usually, a Hamming window is used. Compared with the rectangular window, the spectral leakage of the Hamming window is smaller. After y(n) is framed and windowed, y _i (n) is obtained, which is defined as:

Among them, ω(n) is the Hamming window, and its expression is

Among them, y _i (n) represents the ith frame of speech signal, n represents the number of samples, and L represents the frame length.

(2) Fast Fourier Transform (FFT)

Since it is generally not easy to see the characteristics of the voice signal when it is transformed in the time domain, it is usually transformed into the frequency domain for analysis. The frequency spectrum of different frequencies represents different characteristics of the voice signal. Therefore, fast Fourier transform is performed on each frame of speech signal y _i (n) to obtain the frequency spectrum of each frame signal, as shown in the following formula:

Y(i,k)=FFT[y _i (n)]

where k represents the kth spectral line in the frequency domain.

(3) Calculate the spectral line energy

The energy E(i,k) of the spectral line of each frame of speech signal in the frequency domain can be expressed as:

E(i,k)=[Y(i,k)] ²

(4) Calculate the energy passing through the Mel filter

The energy S(i,m) of the spectral line energy of each frame passing through the Mel filter can be defined as:

Among them, N represents the number of FFT points.

The transfer function H _m (k) of each filter is

Among them, f(m) is the center frequency of the mth filter, m is the mth filter, and M is the number of filters, which is usually set to 24.

(5) Calculate the MFCC, see Figure 4, take the logarithm of the energy of the Mel filter and calculate the Discrete Cosine Transform (DCT) to obtain the MFCC feature parameters, as shown in the following formula:

where j is the spectral line after DCT.

8. Training based on DNN-CLSTM network model;

Input the two features of speech signal amplitude and MFCC after noise spectrum reduction into the DNN-CLSTM network for training, and obtain the enhanced estimated amplitude and MFCC; Amplitude and pure MFCC values calculate their respective minimum mean square errors, and input the obtained errors as adjustment signals into the neural network to optimize the network, thereby obtaining a trained network.

Test phase:

1. Windowing and framing the noisy speech signal, and using discrete Fourier transform to obtain the amplitude and phase of the noisy speech signal as the first feature;

作为第二特征；2. After the noisy speech signal is divided into frames and windows, the amplitude of the speech signal after the spectral subtraction stage is obtained, that is, the amplitude of the spectrally subtracted speech signal

as a second feature;

3. After the noisy speech signal is divided into frames and windowed, the MFCC is obtained as the third feature;

4. Perform time-frequency decomposition on the noisy speech signal, and perform feature extraction to obtain E _MRACC (j,m) and ΔE _MRACC (j,m) as the fourth feature of the input of the deep neural network.

5. Input the four features of noise-containing speech amplitude, spectrum minus speech signal amplitude, MFCC and E _MRACC (j,m) and ΔE _MRACC (j,m) signals obtained by feature extraction into the trained DNN-CLSTM network to obtain the enhanced estimated amplitude, MFCC, and masking threshold;

需要进行时域信号恢复从而得到最终增强后的增强语音信号，首先初步增强后的初步增强语音信号的幅值

需要与步骤1中提取出来的含噪语音信号的相位φ_y(f)进行结合，然后使用逆傅立叶变换转化为时域信号

从而得到最终的增强语音信号。此处得到的增强后的MFCC以及掩蔽阈值不参与波形恢复，在网络处理过程中对网络进行优化。6. Combine the amplitude of the enhanced speech signal with the phase of the noisy speech signal obtained in step 1, and perform inverse Fourier transform to obtain the final enhanced speech signal. The initial enhanced speech signal amplitude obtained after neural network training

It is necessary to perform time-domain signal recovery to obtain the final enhanced enhanced speech signal. First, the amplitude of the preliminary enhanced initially enhanced speech signal

It needs to be combined with the phase φ _y (f) of the noisy speech signal extracted in step 1, and then converted into a time domain signal using the inverse Fourier transform

Thus, the final enhanced speech signal is obtained. The enhanced MFCC and masking threshold obtained here do not participate in waveform recovery, and the network is optimized during network processing.

7. Network establishment

The DNN-CLSTM network includes Deep Neural Network (DNN), Convolutional Neural Network (CNN), Residual Network and Bidirectional Long-term Memory Network. The specific process of the DNN-CLSTM neural network is shown in Figure 5:

(1) DNN network establishment

Input layer: The amplitude of the spectrally subtracted speech signal obtained after spectral subtraction is used as input, and the input node is 128 neurons;

Fully connected layer: set 32 nodes, set the drop rate to 0.5, and set the activation function to RELU;

Fully connected layer: set 128 nodes, set the drop rate value to 0.5, and set the activation function to RELU;

Fully connected layer: set 512 nodes, set the drop rate value to 0.5, and set the activation function to RELU;

(2) Multi-target feature fusion:

Combine the amplitude and MFCC features enhanced by the DNN network with the amplitude and MFCC features of the original noisy speech;

和

分别代表第k个空间领域中经过DNN预测的MFCC特征和语音幅值；

分别代表第k个空间领域中原始含噪语音的 MFCC特征和语音幅值；in

and

respectively represent the MFCC features and speech amplitude predicted by DNN in the kth spatial domain;

represent the MFCC features and speech amplitudes of the original noisy speech in the kth spatial domain, respectively;

(a) CNN:

Convolution layer: Convolve the results obtained by the DNN network, the number of nodes is set to 64 nodes, the step size is set to 1, the convolution kernel is set to 5*1, and the activation function is set to SELU;

BN layer: normalize the data,

Convolution layer: the number of nodes is set to 64 nodes, the step size is set to 1, the convolution kernel is set to 3*1, and the activation function is set to SELU;

BN layer: normalize the data

Convolutional layer: the number of nodes is set to 128 nodes, the step size is set to 1, the convolution kernel is set to 5*1,

(b) Residual network

Convolve the results obtained by the DNN network, the number of nodes is set to 128 nodes, the step size is set to 1, and the convolution kernel is set to 5*1;

After combining the data obtained by the residual network with the data obtained by the CNN network, the SELU activation function is used;

Max Pooling layer: step size is set to 1, pooling layer size is set to 2

(3) LSTM network:

The bidirectional network nodes of the long-short-term memory network are selected as 128 nodes, and the activation function is the sigmoid function.

(4) Output layer:

Two feedforward neural networks are used as the output layer to output the enhanced speech signal amplitude and MFCC; the network model uses Adam optimizer to optimize the network parameters; all convolutional layers use the edge filling method.

(5) Calculate the minimum mean square error objective function

分别代表第k个声学特征空间预测的MFCC特征向量和预测幅值特征

分别代表第k个声学特征空间纯净的MFCC 特征向量和纯净幅值特征。where T=2,

Represent the MFCC feature vector and predicted magnitude feature of the kth acoustic feature space prediction, respectively

represent the pure MFCC feature vector and pure amplitude feature of the kth acoustic feature space, respectively.

【Test example】

The speech data used in the experiment comes from the TIMIT dataset, and the noise dataset comes from the Nonspeech noise library and the Noise-15 noise library. In this experiment, the TIMIT dataset contains a total of 6300 utterances. About 80% of the speech is used as the training set, and the other 20% is used as the test speech. All speech is resampled to 16kHz. For the model proposed by the present invention, several typical neural network speech enhancement models are selected for comparison with the method proposed by the present invention, including (a) DNN (b) CNN (c) LSTM (d) GRU. Among them, DNN is a speech enhancement algorithm based on deep neural network, CNN is a speech enhancement algorithm based on convolutional neural network, LSTM is a speech enhancement algorithm based on long short-term memory neural network, and GRU is a speech enhancement algorithm based on gated recurrent neural network. .

All models are trained under SNR conditions of -5dB, 0dB, 5dB, 10dB, 15dB, 20dB, and the performance is evaluated at matching SNR. To test the robustness of the speech enhancement model, the performance is evaluated under mismatched signal-to-noise ratio conditions. PESQ and LSD are two important indicators for evaluating speech. PESQ refers to the subjective speech quality evaluation index. The higher the LESQ score, the better the speech quality. LSD refers to the logarithmic spectral distance index. The lower the LSD score, the better the speech quality. Table 1 shows the test results compared with the other four algorithms (DNN, CNN, LSTM, GRU) under matched noise conditions, and the results of the best performing algorithms are marked in bold. Table 2 shows the test results compared with the other four algorithms (DNN, CNN, LSTM, GRU) under unmatched noise conditions, and the results of the best performing algorithms are marked in bold.

Table 1 Test results under matched noise conditions, the best performance has been marked in bold

Table 2 Test results under mismatched noise conditions, the best performance has been marked in bold

Claims (3)

1. a speech enhancement method based on DNN-CLSTM network, is characterized in that comprising the following steps: Step 1: Acquire the noisy speech signal. The noisy speech signal is formed by adding the pure speech signal and the noise signal: y(m)=x(m)+n(m) Among them, y(m) is the noisy speech signal, x(m) is the pure speech signal, n(m) is the noise signal, and m is the discrete time series; Step 2: frame-by-frame window processing to obtain the amplitude and phase of the pure speech signal and the noisy speech signal; The noisy speech signal is processed by windowing and framing, and the amplitude and phase of the noisy speech signal are obtained by discrete Fourier transform. Using the first five frame signals in the speech segment as noise estimation, get the noise signal amplitude; Step 3: Subtract the amplitude of the noise signal from the amplitude of the noisy speech signal to obtain the amplitude of the spectrum minus the speech signal as the first feature; Step 4: obtain the MFCC of the speech signal as the second feature; Step 5: Establish a DNN-CLSTM network model for training; Input the two features of speech signal amplitude and MFCC after noise spectrum reduction into the DNN-CLSTM network for training, and obtain the predicted amplitude and MFCC; The pure MFCC values calculate their respective minimum mean square errors (MMSE), and input the obtained error as an adjustment signal into the neural network to optimize the network, thereby obtaining a trained network. 2. the speech enhancement method based on DNN-CLSTM network as claimed in claim 1, is characterized in that, the concrete process of described step 4 is: (1) Preprocessing: Preprocessing includes pre-emphasis, framing, and windowing functions; Pre-emphasis processing: implemented by a first-order high-pass filter, the transfer function of the filter is: H(z)=1-az ^-1 Among them, a is the pre-emphasis coefficient, which is generally 0.98; the result of the speech signal x(n) after pre-emphasis processing is: y(n)=x(n)-ax(n-1) Frame-by-frame windowing: the overlapping part between two adjacent frames, which is frame shift, is set to 10ms; windowing function: Hamming window windowing is performed on each frame of speech signal: y(n) is divided into After frame windowing, y _i (n) is obtained, which is defined as:

Among them, ω(n) is the Hamming window, and its expression is

Wherein, y _i (n) represents the ith frame of speech signal, n represents the number of samples, and L represents the frame length; (2) Fast Fourier Transform (FFT) Perform fast Fourier transform on each frame of speech signal y _i (n) to obtain the frequency spectrum of each frame signal, the expression is as follows: Y(i,k)=FFT[y _i (n)] where k represents the kth spectral line in the frequency domain; (3) Calculate the spectral line energy The energy E(i,k) of the spectral line of each frame of speech signal in the frequency domain is expressed as: E(i,k)=[Y(i,k)] ² (4) Calculate the energy passing through the Mel filter The energy S(i,m) of the spectral line energy of each frame passing through the Mel filter is defined as:

Among them, N represents the number of FFT points, and M is the number of filters; The transfer function H _m (k) of each filter is

Among them, f(m) is the center frequency of the mth filter, and m is the mth filter; (5) Calculate MFCC After taking the logarithm of the energy of the Mel filter, the discrete cosine transform is calculated to obtain the MFCC characteristic parameters, as shown in the following formula:

where j is the discrete cosine transform (DCT) spectral line. 3. as claimed in claim 1 based on the speech enhancement method of DNN-CLSTM network, it is characterized in that, the concrete process of described step 5 is: (1) DNN network establishment Input layer: take the spectrally subtracted speech amplitude and MFCC feature as input, and input it into the DNN network. The number of neurons in the input layer is 128; Fully connected layer: set 32 nodes, set the drop rate to 0.5, and set the activation function to RELU; Fully connected layer: set 128 nodes, set the drop rate value to 0.5, and set the activation function to RELU; Fully connected layer: set 512 nodes, set the drop rate value to 0.5, and set the activation function to RELU; (2) Multi-target feature fusion: Combine the amplitude and MFCC features enhanced by the DNN network with the amplitude and MFCC features of the original noisy speech;

in

and

respectively represent the MFCC features and speech amplitude predicted by DNN in the kth spatial domain;

represent the MFCC features and speech amplitudes of the original noisy speech in the kth spatial domain, respectively; (3) C-LSTM network: (a) CNN: Convolution layer: Convolve the results obtained by the DNN network, the number of nodes is set to 64 nodes, the step size is set to 1, the convolution kernel is set to 5*1, and the activation function is set to SELU; BN layer: normalize the data; Convolution layer: the number of nodes is set to 64 nodes, the step size is set to 1, the convolution kernel is set to 3*1, and the activation function is set to SELU; BN layer: normalize the data Convolutional layer: the number of nodes is set to 128 nodes, the step size is set to 1, the convolution kernel is set to 5*1, (b) Residual network Convolve the results obtained by the DNN network, the number of nodes is set to 128 nodes, the step size is set to 1, and the convolution kernel is set to 5*1; After combining the data obtained by the residual network with the data obtained by the CNN network, the SELU activation function is used; Max Pooling layer: step size is set to 1, pooling layer size is set to 2 (c) LSTM network: The bidirectional network nodes of the long-short-term memory network are selected as 128 nodes, and the activation function is the sigmoid function. (4) Output layer: Two feedforward neural networks are used as output layers to output the predicted speech signal amplitude and MFCC; the network model uses Adam optimizer to optimize the network parameters; all convolutional layers use edge padding. (5) Calculate the minimum mean square error objective function

where T=2,

Represent the MFCC feature vector and predicted magnitude feature of the kth acoustic feature space prediction, respectively

represent the pure MFCC feature vector and pure amplitude feature of the kth acoustic feature space, respectively.

Images