CN115101084A - Model training method, audio processing method, device, sound box, equipment and medium - Google Patents

Abstract

The present disclosure relates to a model training method, an audio processing method, an apparatus, a sound box, a device, and a medium, wherein the model training method includes: acquiring a first training sample comprising an amplitude spectrum of a first noisy audio signal and an amplitude spectrum of a first original audio signal; obtaining a first noisy amplitude spectrum for a first frequency band range and a second noisy amplitude spectrum for a second frequency band range based on the amplitude spectrum of the first noisy frequency signal; inputting the first noisy amplitude spectrum into a first processing model in the audio processing model to obtain a first estimated noise reduction amplitude spectrum; inputting the first estimated noise reduction amplitude spectrum and the second noisy amplitude spectrum into a second processing model in the audio processing model to obtain a second estimated noise reduction amplitude spectrum; acquiring a first loss based on the second estimated noise reduction amplitude spectrum and the second original amplitude spectrum; the audio processing model is trained by adjusting model parameters of the second processing model in accordance with the first loss.

Description

Model training method, audio processing method, device, speaker, equipment and medium

technical field

The present disclosure relates to the field of audio processing, and more particularly, to a model training method, an audio processing method, an apparatus, a sound box, a device and a medium.

Background technique

With the promotion of neural networks, more and more neural networks are used in the audio field, such as audio denoising, audio de-reverberation, speech separation, etc. Compared with traditional algorithms, neural networks can often achieve better results. . In the field of audio noise reduction, currently only the 16kHz frequency band is usually considered for noise reduction. Although there are very few systems that consider full-band noise reduction, the noise reduction effect of the frequency band above 16kHz is not ideal, mainly for noise reduction. The sound quality of the rear audio is degraded, or for the high-frequency region of the audio, there is a situation where the noise reduction performance in the high-frequency region is poor due to blurred frequency resolution.

SUMMARY OF THE INVENTION

The present disclosure provides a model training method, an audio processing method, an apparatus, a sound box, a device and a medium, so as to at least solve the above-mentioned problems in the related art.

According to a first aspect of the embodiments of the present disclosure, there is provided a training method for an audio processing model, including: acquiring a first training sample, wherein the first training sample includes an amplitude spectrum of a first frequency signal with noise and a first original The amplitude spectrum of the audio signal, the first frequency signal with noise is obtained by mixing the first original audio signal and the noise signal; based on the amplitude spectrum of the first frequency signal with noise, the first frequency band range is obtained The first noisy amplitude spectrum and the second noisy amplitude spectrum for the second frequency band range; input the first noisy amplitude spectrum into the first processing model in the audio processing model, and obtain the first frequency band for the first processing model. The first estimated noise reduction amplitude spectrum of the range, wherein the first processing model is pre-trained; the first estimated noise reduction amplitude spectrum and the second noisy amplitude spectrum are input into the audio processing model. a second processing model to obtain a second estimated noise reduction magnitude spectrum for the second frequency band; based on the second estimated noise reduction magnitude spectrum and a second original magnitude spectrum, obtain a first loss, the second original magnitude The spectrum is the amplitude spectrum for the second frequency band in the amplitude spectrum of the first original audio signal; by adjusting the model parameters of the second processing model according to the first loss, the audio processing model is performed train.

Optionally, the first frequency band range and the second frequency band range are obtained by dividing the full frequency band range of the amplitude spectrum of the first noisy frequency signal.

Optionally, the first frequency band range is 0-16 kHz, and the second frequency band range is 16-48 kHz.

Optionally, the first processing model is pre-trained by acquiring a second training sample, wherein the second training sample includes the amplitude spectrum of the second noisy frequency signal and the second original audio signal The amplitude spectrum of the second frequency band with noise is obtained by mixing the second original audio signal and the noise signal, and the frequency band range of the second original audio signal is the first frequency band range; The amplitude spectrum of the second frequency signal with noise is input into the first processing model to obtain a third estimated noise reduction amplitude spectrum for the first frequency band; based on the third estimated noise reduction amplitude spectrum and the second original The amplitude spectrum of the audio signal is obtained, and the second loss is obtained; the first processing model is trained by adjusting the model parameters of the first processing model according to the second loss.

Optionally, the amplitude spectrum of the second frequency signal with noise and the amplitude spectrum of the second original audio signal are obtained by performing a short-term analysis on the second frequency signal with noise and the second original audio signal respectively. Fourier transform to obtain the second frequency signal with noise and the second original audio signal in the time-frequency domain; extract the amplitude based on the second frequency signal with noise and the second original audio signal in the time-frequency domain spectrum, to obtain the amplitude spectrum of the second frequency signal with noise and the amplitude spectrum of the second original audio signal.

Optionally, the amplitude spectrum of the first frequency signal with noise and the amplitude spectrum of the first original audio signal are obtained by performing a short-term analysis on the first frequency signal with noise and the first original audio signal respectively. Fourier transform to obtain the first frequency signal with noise and the first original audio signal in the time-frequency domain; extract the amplitude based on the first frequency signal with noise and the first original audio signal in the time-frequency domain spectrum to obtain the amplitude spectrum of the first noise-banded frequency signal and the amplitude spectrum of the first original audio signal.

According to a second aspect of the embodiments of the present disclosure, an audio processing method is provided, including: acquiring an audio signal to be processed; and performing audio processing on the audio signal to be processed by using an audio processing model to obtain a processed audio signal, wherein , the audio processing model is obtained by training based on any one of the audio processing model training methods described in the first aspect.

Optionally, performing audio processing on the to-be-processed audio signal by using an audio processing model to obtain a processed audio signal includes: obtaining, based on an amplitude spectrum of the to-be-processed audio signal, an a first magnitude spectrum and a second magnitude spectrum for a second frequency band range; inputting the first magnitude spectrum into a first processing model in the audio processing model to obtain a first estimated magnitude spectrum for the first frequency band range ; Input the second amplitude spectrum and the first estimated amplitude spectrum into the second processing model in the audio processing model to obtain a second estimated amplitude spectrum for the second frequency band range; The magnitude spectrum is combined with the second estimated magnitude spectrum to obtain an estimated magnitude spectrum; based on the estimated magnitude spectrum, a processed audio signal is obtained.

Optionally, the first frequency band range and the second frequency band range are obtained by dividing the full frequency band range of the amplitude spectrum of the audio signal to be processed.

Optionally, the first frequency band range is 0-16 kHz, and the second frequency band range is 16-48 kHz.

Optionally, the amplitude spectrum of the audio signal to be processed is obtained by: performing short-time Fourier transform on the audio signal to obtain the audio signal in the time-frequency domain; based on the audio signal in the time-frequency domain The amplitude spectrum of the signal is extracted to obtain the amplitude spectrum of the audio signal.

Optionally, obtaining the processed audio signal based on the estimated amplitude spectrum includes: multiplying the estimated amplitude spectrum by a phase corresponding to the estimated amplitude spectrum; performing an inverse short-time-future on the multiplication result. Lie transform to obtain the processed audio signal in the time domain.

According to a third aspect of the embodiments of the present disclosure, there is provided an apparatus for training an audio processing model, comprising: a first training sample obtaining unit configured to: obtain a first training sample, wherein the first training sample includes a first training sample The amplitude spectrum of the noise-banded frequency signal and the amplitude spectrum of the first original audio signal, the first noise-banded frequency signal is obtained by mixing the first original audio signal and the noise signal; the noise-banded amplitude spectrum acquisition unit is It is configured to: based on the amplitude spectrum of the first frequency signal with noise, obtain a first amplitude spectrum with noise for the first frequency band range and a second amplitude spectrum with noise for the second frequency band range; first estimate the noise reduction amplitude spectrum a determining unit, configured to: input the first noisy amplitude spectrum into a first processing model in the audio processing model to obtain a first estimated noise reduction amplitude spectrum for the first frequency band, wherein the The first processing model is pre-trained; the second estimated noise reduction amplitude spectrum determining unit is configured to: input the first estimated noise reduction amplitude spectrum and the second noisy amplitude spectrum into the audio processing model; The second processing model obtains a second estimated noise reduction amplitude spectrum for the second frequency band; the first loss acquisition unit is configured to: based on the second estimated noise reduction amplitude spectrum and the second original amplitude spectrum, obtain a first loss, the second original amplitude spectrum is an amplitude spectrum for the second frequency band in the amplitude spectrum of the first original audio signal; the model parameter adjustment unit is configured to: The loss adjusts the model parameters of the second processing model to train the audio processing model.

Optionally, the first frequency band range and the second frequency band range are obtained by dividing the full frequency band range of the amplitude spectrum of the first noisy frequency signal.

Optionally, the first frequency band range is 0-16 kHz, and the second frequency band range is 16-48 kHz.

Optionally, the first processing model is pre-trained by acquiring a second training sample, wherein the second training sample includes the amplitude spectrum of the second noisy frequency signal and the second original audio signal The amplitude spectrum of the second frequency band with noise is obtained by mixing the second original audio signal and the noise signal, and the frequency band range of the second original audio signal is the first frequency band range; The amplitude spectrum of the second frequency signal with noise is input into the first processing model to obtain a third estimated noise reduction amplitude spectrum for the first frequency band; based on the third estimated noise reduction amplitude spectrum and the second original The amplitude spectrum of the audio signal is obtained, and the second loss is obtained; the first processing model is trained by adjusting the model parameters of the first processing model according to the second loss.

Optionally, the amplitude spectrum of the second frequency signal with noise and the amplitude spectrum of the second original audio signal are obtained by performing a short-term analysis on the second frequency signal with noise and the second original audio signal respectively. Fourier transform to obtain the second frequency signal with noise and the second original audio signal in the time-frequency domain; extract the amplitude based on the second frequency signal with noise and the second original audio signal in the time-frequency domain spectrum, to obtain the amplitude spectrum of the second frequency signal with noise and the amplitude spectrum of the second original audio signal.

Optionally, the amplitude spectrum of the first frequency signal with noise and the amplitude spectrum of the first original audio signal are obtained by performing a short-term analysis on the first frequency signal with noise and the first original audio signal respectively. Fourier transform to obtain the first frequency signal with noise and the first original audio signal in the time-frequency domain; extract the amplitude based on the first frequency signal with noise and the first original audio signal in the time-frequency domain spectrum to obtain the amplitude spectrum of the first noise-banded frequency signal and the amplitude spectrum of the first original audio signal.

According to a fourth aspect of the embodiments of the present disclosure, there is provided an audio processing apparatus, comprising: an audio signal acquisition unit, configured to: acquire an audio signal to be processed; and an audio signal processing unit, configured to: use an audio processing model to The audio signal to be processed is subjected to audio processing to obtain a processed audio signal, wherein the audio processing model is obtained by training based on any one of the training methods described in the first aspect.

Optionally, the audio signal processing unit may be configured to obtain a first amplitude spectrum for the first frequency band range and a second amplitude spectrum for the second frequency band range based on the amplitude spectrum of the audio signal to be processed; The first amplitude spectrum is input into the first processing model in the audio processing model to obtain a first estimated amplitude spectrum for the first frequency band; the second amplitude spectrum and the first estimated amplitude spectrum are input into the a second processing model in the audio processing model to obtain a second estimated amplitude spectrum for the second frequency band range; combining the first estimated amplitude spectrum and the second estimated amplitude spectrum to obtain an estimated amplitude spectrum; based on the The estimated amplitude spectrum is described to obtain the processed audio signal.

Optionally, the first frequency band range and the second frequency band range are obtained by dividing the full frequency band range of the amplitude spectrum of the audio signal to be processed.

Optionally, the first frequency band range is 0-16 kHz, and the second frequency band range is 16-48 kHz.

Optionally, the audio processing device further includes an audio signal amplitude spectrum acquisition unit, and the audio signal amplitude spectrum acquisition unit can be configured to perform short-time Fourier transform on the audio signal to obtain the time-frequency domain. Audio signal; extracting the amplitude spectrum based on the audio signal in the time-frequency domain, to obtain the amplitude spectrum of the audio signal.

Optionally, the audio signal processing unit may be configured to: multiply the estimated amplitude spectrum by a phase corresponding to the estimated amplitude spectrum; perform an inverse short-time Fourier transform on the multiplication result to obtain a time domain the processed audio signal.

According to a fifth aspect of the embodiments of the present disclosure, there is provided a smart speaker, including the audio processing apparatus according to the fourth aspect of the present disclosure.

According to a sixth aspect of embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions are executed by the at least one processor when the at least one processor is caused to perform the training method of the audio processing model according to the first aspect of the present disclosure or the audio processing method according to the second aspect of the present disclosure.

According to a seventh aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to execute the audio according to the first aspect of the present disclosure A training method of a processing model or performing an audio processing method according to the second aspect of the present disclosure.

According to an eighth aspect of the embodiments of the present disclosure, there is provided a computer program product, the instructions in the computer program product can be executed by a processor of a computer device to complete the training method of the audio processing model according to the first aspect of the present disclosure or execute the method according to the present disclosure. The audio processing method of the second aspect is disclosed.

The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

According to the training method, device, device and medium of the audio processing model of the present disclosure, when the training samples are too few, the training process of the second processing model is guided by the output result of the pre-trained first processing model, so that the training process can be improved. The completed second processing model has a good noise reduction effect, thereby enhancing the robustness of the entire audio processing model and improving the noise reduction effect of the audio processing model.

In addition, according to the audio processing method, device, device, speaker and medium of the present disclosure, the audio processing model trained by the audio processing model training method according to the present disclosure is used to perform audio processing, which can avoid processing due to high model complexity. The quality of the post-audio is degraded, and the noise reduction effect of the post-processing audio can be improved.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure and do not unduly limit the present disclosure.

FIG. 1 is an overall system diagram illustrating the training and application of an audio processing model according to an exemplary embodiment of the present disclosure.

FIG. 2 is a flowchart illustrating a training method of an audio processing model according to an exemplary embodiment of the present disclosure.

FIG. 3 is a flowchart illustrating an audio processing method according to an exemplary embodiment of the present disclosure.

4 is a flowchart illustrating audio processing using an audio processing model according to an exemplary embodiment of the present disclosure.

FIG. 5 is a block diagram illustrating a training apparatus 500 of an audio processing model according to an exemplary embodiment of the present disclosure.

FIG. 6 is a block diagram illustrating an audio processing apparatus 600 according to an exemplary embodiment of the present disclosure.

FIG. 7 is a block diagram illustrating the audio signal processing unit 620 according to an exemplary embodiment of the present disclosure.

FIG. 8 is a block diagram of a smart speaker 800 according to an exemplary embodiment of the present disclosure.

FIG. 9 is a block diagram illustrating an electronic device 900 according to an exemplary embodiment of the present disclosure.

Detailed ways

In order to make those skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first", "second" and the like in the description and claims of the present disclosure and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the disclosure described herein can be practiced in sequences other than those illustrated or described herein. The implementations described in the following examples are not intended to represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as recited in the appended claims.

It should be noted here that "at least one of several items" in the present disclosure all means including "any one of the several items", "a combination of any of the several items", The three categories of "the whole of the several items" are juxtaposed. For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "execute at least one of step 1 and step 2", which means the following three parallel situations: (1) execute step 1; (2) execute step 2; (3) execute step 1 and step 2.

Under normal circumstances, when using neural networks to solve the noise reduction problem of audio signals, the effect of processing in the time-frequency domain is often better. The time-frequency domain refers to the short-time Fourier transform of the original time-domain waveform signal (STFT: Short -Time FourierTransform) to the time-frequency domain, after a series of processing, and then the time-frequency domain signal short-time inverse Fourier Transform (ISTFT: Inverse Short-Time Fourier Transform) to the time domain, to obtain the way of the processed waveform. When using neural networks to deal with audio noise reduction in the time-frequency domain, the characteristics of the input network can often be divided into Fourier transform time spectrum, Mel spectrum, subband spectrum, etc. Among them, Mel spectrum and subband spectrum are based on Fourier transform The time spectrum is converted from the transformed time spectrum, so the time spectrum has the most complete time-frequency information. Using the time spectrum as the input feature of the neural network can preserve the sound quality of the audio to the greatest extent and achieve a good noise reduction effect.

For the full-band noise reduction system, when the complete time spectrum is used as a training feature, it will affect the convergence effect of the network, resulting in lower audio quality and higher complexity. However, using the mel spectrum or bfcc (Barker scale-based cepstrum) of psychoacoustic motivation features as training features can effectively reduce the overall complexity, but it will lead to blurred frequency resolution in the high-frequency region, resulting in high-frequency The noise reduction performance of the area is poor. Specifically, for the noise reduction of full-band audio, there are mainly two technical solutions in the related art. One is to perform full-band audio noise reduction by using a 48kHz full noise reduction model trained by mel spectrum or bfcc. The model has poor frequency resolution in the high-frequency region, resulting in poor noise reduction effect of the scheme in the high-frequency region of 16-48kHz; Full-band audio noise reduction, but the sound quality after noise reduction in the frequency region below 16kHz is poor, and the PESQ (Perceptual evaluation of speech quality, subjective speech quality evaluation) score is low.

In order to take into account the sound quality of audio after noise reduction in the frequency region below 16 kHz and improve the noise reduction effect in the high frequency region of 16-48 kHz, the present disclosure proposes a training method, device, equipment and medium for an audio processing model, Audio processing method, apparatus, speaker, device and medium, specifically, when training an audio processing model, in high frequency band audio samples (for example, 48kHz audio signal) relative to low frequency band audio samples (for example, 16kHz audio signal) ) is too small or the data types of high-band audio samples (for example, 48kHz audio signals) and low-band audio samples (for example, 16kHz audio signals) do not match, through a pre-trained audio processing model (for example, using The output result of processing audio with frequency band below 16 kHz guides the training process of another processing model (for example, for processing audio with frequency band of 16 kHz-48 kHz), so that another processing model after training has good noise reduction. effect, thereby enhancing the robustness of the entire audio processing model. In addition, when using the trained audio processing model for audio processing, the amplitude spectrum of the audio signal to be processed is divided into the amplitude spectrum of the low frequency band range (eg, below 16 kHz) and the high frequency band range (eg, 16-48 kHz) The two amplitude spectra are input into two audio processing models for processing, and the audio processing results of the amplitude spectrum in the low frequency band are used to guide the audio processing of the amplitude spectrum in the high and low frequency bands, so as to avoid the complexity of the model. It can reduce the problem of sound quality degradation after noise reduction caused by high degree of noise, and improve the noise reduction effect of audio in high frequency area. Hereinafter, a model training method, an audio processing method, an apparatus, a sound box, an apparatus and a medium according to exemplary embodiments of the present disclosure will be described in detail with reference to FIGS. 1 to 9 .

FIG. 1 is an overall system diagram illustrating the training and application of an audio processing model according to an exemplary embodiment of the present disclosure.

Referring to FIG. 1 , the audio processing model 100 includes a first processing model 110 and a second processing model 120, wherein the first processing model 110 is used to perform noise reduction processing on audio in a low frequency band range (eg, below 16 kHz), and the second processing model is used for noise reduction processing. Model 120 is used to de-noise audio in the high frequency band range (eg, 16-48 kHz). Audio processing models need to be trained before they can be applied. According to an exemplary embodiment of the present disclosure, the first processing model 110 is trained first, and then the second processing model 120 is trained in combination with the first processing model 110 . Specifically, when training the first processing model 110, pre-training processing may be performed on the training samples. For example, time-frequency conversion (for example, short-time Fourier transform) is performed on the original audio signal in the low frequency band range (for example, below 16 kHz) and the noisy audio signal after mixing the original audio signal. In the frequency domain, the amplitude information and phase information of each frame of audio signal (including the original audio signal and the noisy audio signal) are obtained. The amplitude spectrum of the noisy frequency signal can be used as the feature (feature) trained by the first processing model 110, the amplitude spectrum of the original audio signal can be used as the target (label) trained by the first processing model 110, and the feature and label can be sent to the first processing model 110. The processing model 110 is trained to obtain the trained first processing model 110 . When training the second processing model 120, pre-training processing is also performed on the training samples, for example, time-frequency conversion is performed on the original audio signal of the full frequency range (eg, 48 kHz) and the mixed-noise frequency signal after the original audio signal. (For example, short-time Fourier transform), convert from time domain to time-frequency domain, and obtain the amplitude information and phase information of each frame of audio signal (including original audio signal and noisy audio signal). The amplitude spectrum of the high frequency band range (for example, 16-48 kHz) in the noisy audio signal may be used as the training feature of the second processing model 120, and the amplitude spectrum of the high frequency band range (for example, 16-48 kHz) in the original audio signal may be used as the first characteristic. The training target of the second processing model 120 is to input the amplitude spectrum of the low frequency range (for example, below 16 kHz) in the noisy frequency signal into the trained first processing model 120 to obtain the result after noise reduction. The noise result, the amplitude spectrum of the high frequency band in the noisy frequency signal and the amplitude spectrum of the high frequency band in the original audio signal are sent to the second processing model 120 for training, and the trained second processing model 120 is obtained.

After obtaining the trained audio processing model, short-time Fourier transform can be performed on the audio signal to be processed, and the amplitude spectrum of the audio signal to be processed can be extracted based on the result of the short-time Fourier transform. The magnitude spectrum is divided into a low frequency range (eg, below 16 kHz) and a high frequency range (eg, 16-48 kHz), and the two magnitude spectra are input to the first processing model 110 and the second processing, respectively A noise reduction process is performed in the model 120 to obtain estimated amplitude spectra for the low frequency range and the high frequency range, respectively. Afterwards, inverse short-time Fourier transform is performed after combining the two estimated amplitude spectra to obtain a denoised audio signal.

According to the above solution, the amplitude spectrum of the audio signal in the time-frequency domain is divided into two amplitude spectra of the low frequency band range and the high frequency band range according to the frequency band range, and the first processing model and the second processing model of the audio processing model are used. Processing these two amplitude spectra separately can reduce the high overall complexity caused by using the time spectrum of audio as the input feature of the neural network while retaining the time spectrum of the audio as the input feature of the neural network, thereby avoiding The audio quality is degraded after noise reduction, and the noise reduction effect of the audio in the high frequency region is improved.

In addition, in the process of training the second processing model, the audio processing results of the amplitude spectrum in the low frequency band are used to guide the audio processing of the amplitude spectrum in the high and low frequency bands, so that good results can be obtained even when there are few high frequency audio samples. The noise reduction effect, thereby enhancing the robustness of the entire audio processing model.

FIG. 2 is a flowchart illustrating a training method of an audio processing model according to an exemplary embodiment of the present disclosure. Here, the audio processing model may be a neural network (eg, Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), etc.) model, whose input during training may be a noisy frequency signal (training sample), which, when applied, the input may be the amplitude spectrum of the audio signal to be processed.

Referring to FIG. 2, in step 201, a first training sample may be obtained, wherein the first training sample includes an amplitude spectrum of a first frequency signal with noise and an amplitude spectrum of a first original audio signal, the first frequency signal with noise is obtained by converting The first original audio signal is obtained by mixing the noise signal. Here, the first original audio signal may be a pure audio signal or a reverberated audio signal (eg, may be generated by convolving a pure audio signal with a room impulse response), etc., which is not limited. The noise signal may be a noise signal obtained by downloading from the Internet, actual recording, or the like. Specifically, the first original audio signal and the noise signal may be added in the time domain according to a certain signal-to-noise ratio to generate the first frequency signal with noise. In addition, the first original audio signal and the first noisy audio signal may be full-band audio signals (eg, the frequency band is 48 kHz).

According to an exemplary embodiment of the present disclosure, the amplitude spectrum of the first frequency signal with noise and the amplitude spectrum of the first original audio signal may be obtained in the following manner: first, the first frequency signal with noise and the first original audio signal are respectively Short-time Fourier transform is performed to obtain the first frequency signal with noise and the first original audio signal in the time-frequency domain, so that the amplitude information and phase information of the first frequency signal with noise and the first original audio signal in each frame can be obtained. Then, the amplitude spectrum is extracted based on the first frequency signal with noise and the first original audio signal in the time-frequency domain to obtain the amplitude spectrum of the first frequency signal with noise and the amplitude spectrum of the first original audio signal.

For example, for a 48kHz full-band audio signal, if the first original audio signal x and the first noisy audio signal y of length T are respectively x(t) and y(t) in the time domain, where t represents time and 0 <t≤T, then after short-time Fourier transform, x(t) and y(t) can be expressed as the following formulas (1) and (2) in the time-frequency domain:

X _48k (n,k)=STFT(x _48k (t)) (1)

Y _48k (n,k)=STFT(y _48k (t)) (2)

Wherein, X _48k (n, k) represents the time-frequency domain signal of the first original audio signal, Y _48k (n, k) represents the time-frequency domain signal of the first noise-banded frequency signal, and x _48k (t) represents the first original audio signal The time domain signal of the audio signal, y _48k (t) represents the time domain signal of the first frequency signal with noise, n is the frame sequence, 0<n≤N, N is the total number of frames; k is the center frequency sequence 0<k≤ K ₄₈ ; K ₄₈ is the total number of frequency bands.

The amplitude spectrum can be extracted from the time-frequency domain signal X _48k (n,k) of the first original audio signal and the time-frequency domain signal Y _48k (n,k) of the first band noise frequency signal, respectively, as shown in the following formulas (3) and ( 4) shown.

MagX _48k (n,k)=abs(X _48k (n,k)) (3)

MagY _48k (n,k)=abs(Y _48k (n,k)) (4)

Wherein, MagX _48k (n, k) represents the amplitude spectrum of the first original audio signal; MagY _48k (n, k) represents the amplitude spectrum of the first band noise frequency signal.

Referring back to FIG. 2, in step 202, a first noisy amplitude spectrum for the first frequency band range and a second noisy amplitude spectrum for the second frequency band range may be obtained based on the amplitude spectrum of the first noisy frequency signal. Here, the first frequency band range and the second frequency band range are obtained by dividing the full frequency band range of the amplitude spectrum of the first frequency band noise signal. Specifically, the first frequency band range is 0-16kHz, and the second frequency band range is 16- 48kHz. For example, for a 48kHz full-band audio signal, the amplitude spectrum MagY _48k (n,k) of the first noisy audio signal can be divided into a 0-16kHz part (ie, the first noisy amplitude spectrum) and a 16-48kHz part (ie , the second noisy amplitude spectrum), which can be expressed as MagY _0-16k (h,k) and MagY _16-48k (n,k), respectively.

In step 203, the first noisy amplitude spectrum may be input into a first processing model in the audio processing model to obtain a first estimated noise reduction amplitude spectrum for the first frequency band, wherein the first processing model is pre-trained. Here, the first processing model is, for example, a 16kHz audio processing model, and MagY _0-16k (n,k) can be input into the 16kHz audio processing model to obtain MagY _{0-16k_p} (n,k) after noise reduction.

According to an exemplary embodiment of the present disclosure, the first processing model may be pre-trained in the following manner: first, acquiring a second training sample, wherein the second training sample includes the amplitude spectrum of the second noisy frequency signal and the second The amplitude spectrum of the original audio signal, the second frequency signal with noise is obtained by mixing the second original audio signal and the noise signal, and the frequency band range of the second original audio signal is the first frequency band range. Here, the second processing model may be a neural network (eg, deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), etc.) model. The second original audio signal may be a clean audio signal or a reverberated audio signal (eg, may be generated by convolving the clean audio signal with a room impulse response), etc., which is not limited. The noise signal may be a noise signal obtained by downloading from the Internet, actual recording, or the like. Specifically, the second audio signal with noise can be generated by adding the second original audio signal and the noise signal in the time domain according to a certain signal-to-noise ratio. In addition, the frequency band range of the second original audio signal is included in the frequency band range of the first original audio signal (or the frequency band range of the second noisy audio signal is included in the frequency band range of the first noisy audio signal), for example, the first original When the frequency band range of the audio signal is 0-48 kHz, the frequency band range of the second original audio signal may be 0-16 kHz (ie, the first frequency band range).

According to an exemplary embodiment of the present disclosure, the amplitude spectrum of the second noisy frequency signal and the amplitude spectrum of the second original audio signal may be obtained in the following manner: first, the second noisy frequency signal and the second original audio signal are respectively Short-time Fourier transform to obtain the second frequency signal with noise and the second original audio signal in the time-frequency domain, and then extract the amplitude spectrum based on the second frequency signal with noise and the second original audio signal in the time-frequency domain to obtain the first The amplitude spectrum of the two-band noise frequency signal and the amplitude spectrum of the second original audio signal. Here, when the short-time Fourier transform is performed on the second frequency signal with noise and the second original audio signal, the selection can be the same as when the short-time Fourier transform is performed on the first frequency signal with noise and the first original audio signal the STFT length and frame shift length.

For example, for a 16kHz full-band audio signal, if the second original audio signal x and the second noisy audio signal y of length T are respectively x(t) and y(t) in the time domain, where t represents time and 0 <t≤T, then after short-time Fourier transform, x(t) and y(t) in the time-frequency domain can be expressed as the following formulas (5) and (6):

X _16k (n,k)=STFT(x _16k (t)) (5)

Y _16k (n,k)=STFT(y _16k (t)) (6)

Wherein, X _16k (n, k) represents the time-frequency domain signal of the second original audio signal, Y _16k (n, k) represents the time-frequency domain signal of the second noise-banded frequency signal, and x _16k (t) represents the second original audio signal The time domain signal of the audio signal, y _16k (t) represents the time domain signal of the second frequency signal with noise, n is the frame sequence, 0<n≤N, N is the total number of frames; k is the center frequency sequence 0<k≤ K ₁₆ ; 16 is the total number of frequency bands.

The amplitude spectrum can be extracted from the time-frequency domain signal X _16k (n,k) of the second original audio signal and the time-frequency domain signal Y _16k (n,k) of the second noise frequency signal, respectively, as shown in the following formulas (7) and ( 8) shown.

MagX _16k (n,k)=abs(X _16k (n,k)) (7)

MagY _16k (n,k)=abs(Y _16k (n,k)) (8)

Wherein, MagX _16k (n, k) represents the amplitude spectrum of the second original audio signal; MagY _16k (n, k) represents the amplitude spectrum of the second frequency band noise signal. Here, MagY _16k (n, k) can be used as the input of the first processing model, and MagX _16k (n, k) can be used as the training label of the first processing model.

Then, the amplitude spectrum of the second noisy frequency signal may be input into the first processing model to obtain a third estimated noise reduction amplitude spectrum for the first frequency band. Here, for example, MagY _0-16k (n,k) is input into the 16kHz audio processing model, and an estimated denoised MagY _{0-16k_p} (n,k) is obtained.

Then, a second loss is obtained based on the third estimated noise reduction amplitude spectrum and the amplitude spectrum of the second original audio signal, and finally, the first processing model is trained by adjusting the model parameters of the first processing model according to the second loss. The amplitude spectrum of the second original audio signal can be used as a training label, and after obtaining the estimated third estimated noise reduction amplitude spectrum for the first frequency band range through the first processing model, a pre-designed loss function is determined based on these two parameters, And the model parameters of the first processing model are iteratively updated by back-propagation according to the determined loss function. During the training of the first processing model, the batch of audio training samples may be used to adjust (or update) the model parameters of the first processing model, and iteratively adjust (or update) the first processing model with the goal of minimizing the loss function value. Model parameters of a process model until the first process model converges.

Referring back to FIG. 2, in step 204, the first estimated noise reduction amplitude spectrum and the second noisy amplitude spectrum may be input into the second processing model in the audio processing model to obtain a second estimated noise reduction amplitude spectrum for the second frequency band range . Here, the second processing model is, for example, a 16-48kHz audio processing model, and the noise-reduced MagY _16k (n,k) and the second noisy amplitude spectrum of the amplitude spectrum for the first frequency band output from the first processing model can be MagY _16-48k (n,k) is used as the input of the 16-48kHz audio processing model, and the second original amplitude spectrum MagX _16-48k (n,k) is used as the training label to train the 16-48kHz audio processing model.

In step 205, the first loss may be obtained based on the second estimated noise reduction amplitude spectrum and the second original amplitude spectrum, where the second original amplitude spectrum is the amplitude spectrum for the second frequency band in the amplitude spectrum of the first original audio signal ( For example, it can be expressed as MagX _16-48k (n,k)).

At step 206, the audio processing model may be trained by adjusting model parameters of the second processing model according to the first loss. Here, the second original amplitude spectrum can be used as a training label, and after the second estimated noise reduction amplitude spectrum for the second frequency band range is obtained through the second processing model, a pre-designed loss function is determined based on these two parameters, and The model parameters of the second processing model are iteratively updated by back-propagation according to the determined loss function. During the training of the second processing model, the batch of audio training samples can be used to adjust (or update) the model parameters of the second processing model, and iteratively adjust (or update) the first The second process the model parameters of the model until the second process model converges.

According to the above solution, during the training process, the number of the first training samples (for example, 48kHz audio) may be significantly less than that of the second training samples (for example, 16kHz audio). During training, the output of the trained first processing model (with high accuracy) is used as the input of the second processing model, which can guide the training of the second processing model. Therefore, when the number of first training samples is relatively small, The second processing model obtained by training can also obtain a good noise reduction effect when processing audio, which enhances the robustness of the entire audio processing model.

FIG. 3 is a flowchart illustrating an audio processing method according to an exemplary embodiment of the present disclosure.

Referring to FIG. 3, in step 301, an audio signal to be processed may be acquired. Here, the audio signal to be processed is a full-band audio signal (for example, the frequency band is 48 kHz), and the processing process is to perform noise reduction on the audio signal.

In step 302, an audio processing model may be used to perform audio processing on the acquired audio signal to be processed to obtain a processed audio signal, wherein the audio processing model is obtained by training based on the aforementioned training method of the audio processing model.

According to an exemplary embodiment of the present disclosure, for the process of performing audio processing on the acquired audio signal to be processed by using an audio processing model, reference may be made to FIG. Flowchart of audio processing.

Referring to FIG. 4 , in step 401, a first amplitude spectrum for the first frequency band range and a second amplitude spectrum for the second frequency band range may be obtained based on the amplitude spectrum of the audio signal to be processed.

According to an exemplary embodiment of the present disclosure, the first frequency band range and the second frequency band range are obtained by dividing the full frequency band range of the amplitude spectrum of the audio signal to be processed. The two-band range is 16-48kHz. In some embodiments, the amplitude spectrum of the audio signal to be processed can be obtained in the following manner: first, performing a Short-Time Fourier Transform (STFT: Short-Time Fourier Transform) on the audio signal to obtain audio in the time-frequency domain signal, and then extract the amplitude spectrum based on the audio signal in the time-frequency domain to obtain the amplitude spectrum of the audio signal. Specifically, the audio signal to be processed is an audio signal in the time domain. In order to further process the audio signal, the short-time Fourier transform of the audio signal to be processed can be performed to convert it from the time domain to the time-frequency domain. , so as to obtain the time spectrum including the time information, frequency information, amplitude information and phase information of the audio signal to be processed, and the corresponding amplitudes of different frequency signals can be extracted from the time spectrum to form the amplitude spectrum. For the execution process of the short-time Fourier transform, reference may be made to the detailed description in the related art, and details are not described here. In some embodiments, a Fast Fourier Transform (FFT, Fast Fourier Transform) may also be performed on the audio signal to obtain an audio signal in the time-frequency domain, which is not limited.

In step 402, the first amplitude spectrum may be input into a first processing model in the audio processing model to obtain a first estimated amplitude spectrum for a first frequency band range. Here, the audio processing model is a pre-trained neural network (eg, deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), etc.) model, the training process of which is described in detail below, in This is not expanded for now. For a 48kHz full-band audio signal, the amplitude spectrum with a frequency band range of 0-16kHz can be input into the first processing model to obtain a first estimated amplitude spectrum for the 0-16kHz frequency band, wherein the first estimated amplitude spectrum is the estimated pair of The amplitude spectrum of the 0-16kHz frequency band range after noise reduction.

In step 403, the second magnitude spectrum and the first estimated magnitude spectrum may be input into a second processing model in the audio processing model to obtain a second estimated magnitude spectrum for the second frequency band range. For example, for a 48 kHz full-band audio signal, the amplitude spectrum in the frequency band range of 16-48 kHz and the estimated amplitude spectrum in the 0-16 kHz frequency band output from the first processing model can be input into the second processing model. Obtain the estimated magnitude spectrum after noise reduction for the magnitude spectrum in the 16-48 kHz frequency band.

In step 404, the first estimated magnitude spectrum and the second estimated magnitude spectrum may be combined to obtain an estimated magnitude spectrum. Here, for a 48kHz full-band audio signal, the estimated amplitude spectrum of the 48kHz frequency band range after noise reduction is obtained.

At step 405, a processed audio signal may be obtained based on the estimated magnitude spectrum.

According to an exemplary embodiment of the present disclosure, in order to obtain a visual and intuitive full-band audio signal after noise reduction, the estimated amplitude spectrum and the phase corresponding to the estimated amplitude spectrum can be multiplied, and then the multiplication result can be inversely short-term Fourier transform to obtain the processed audio signal in the time domain. Here, for a 48kHz full-band audio signal, the noise-reduced audio signal, for example, but not limited to, can be expressed as:

X0(t)=ISTFT(MagY _{48k_p} (n,k)*PhaY _48k (n,k)) (9)

Among them, X0(t) represents the audio signal after noise reduction; n represents the frame sequence of the audio signal, 0<n≤N, (N is the total number of frames); k represents the center frequency sequence of the audio signal, 0<k≤K ₄₈ , (K ₄₈ is the total number of frequency bands); MagY _{48k_p} (n, k) represents the estimated amplitude spectrum at the time-frequency point (n, k); PhaY _48k (n, k) represents the time-frequency point (n, k) phase at k).

In other embodiments, the processed audio signal may also be obtained directly based on the first estimated amplitude spectrum and the second estimated amplitude spectrum, which is not limited. Specifically, the first estimated amplitude spectrum and the phase corresponding to the first estimated amplitude spectrum can be multiplied to obtain a first product, and an inverse short-time Fourier transform is performed on the first product to obtain the processed first Similarly, similar operations can be performed on the second estimated amplitude spectrum to obtain a processed second audio signal, and the final desired processed audio signal can be obtained by combining the first audio signal and the second audio signal.

FIG. 5 is a block diagram illustrating a training apparatus 500 of an audio processing model according to an exemplary embodiment of the present disclosure.

5 , an apparatus 500 for training an audio processing model according to an exemplary embodiment of the present disclosure may include a first training sample acquisition unit 501 , a noisy amplitude spectrum acquisition unit 502 , a first estimated noise reduction amplitude spectrum determination unit 503 , and a first training sample acquisition unit 501 . Two estimated noise reduction amplitude spectrum determination unit 504 , first loss acquisition unit 505 and model parameter adjustment unit 506 .

The first training sample obtaining unit 501 can obtain a first training sample, wherein the first training sample includes the amplitude spectrum of the first frequency signal with noise and the amplitude spectrum of the first original audio signal. The first original audio signal is obtained by mixing the noise signal. Here, the first original audio signal may be a pure audio signal or a reverberated audio signal (eg, may be generated by convolving a pure audio signal with a room impulse response), etc., which is not limited. The noise signal may be a noise signal obtained by downloading from the Internet, actual recording, or the like. Specifically, the first original audio signal and the noise signal may be added in the time domain according to a certain signal-to-noise ratio to generate the first frequency signal with noise.

According to an exemplary embodiment of the present disclosure, the amplitude spectrum of the first frequency signal with noise and the amplitude spectrum of the first original audio signal may be obtained in the following manner: first, the first frequency signal with noise and the first original audio signal are respectively Short-time Fourier transform is performed to obtain the first frequency signal with noise and the first original audio signal in the time-frequency domain, so that the amplitude information and phase information of the first frequency signal with noise and the first original audio signal in each frame can be obtained. Then, the amplitude spectrum is extracted based on the first frequency signal with noise and the first original audio signal in the time-frequency domain to obtain the amplitude spectrum of the first frequency signal with noise and the amplitude spectrum of the first original audio signal.

For example, for a 48kHz full-band audio signal, if the first original audio signal x and the first noisy audio signal y of length T are respectively x(t) and y(t) in the time domain, where t represents time and 0 <t≤T, then after short-time Fourier transform, x(t) and y(t) can be expressed as formula (1) and formula (2) respectively in the time-frequency domain.

The amplitude spectrum can be extracted from the time-frequency domain signal X _48k (n,k) of the first original audio signal and the time-frequency domain signal Y _48k (n,k) of the first band noise frequency signal, respectively, to obtain the first original audio signal. The magnitude spectrum MagX _48k (n,k) and the magnitude spectrum MagY _48k (n,k) of the first noise-band signal (shown in equations (3) and (4), respectively).

The noisy amplitude spectrum obtaining unit 502 may obtain a first noisy amplitude spectrum for the first frequency band range and a second noisy amplitude spectrum for the second frequency band range based on the amplitude spectrum of the first noisy frequency signal. Here, the first frequency band range and the second frequency band range are obtained from the division of the full frequency band range of the amplitude spectrum of the first frequency signal with noise. Specifically, since the audio noise reduction scheme in the related art exists in the frequency region below 16 kHz The sound quality after noise reduction is poor and the noise reduction effect on the high frequency region of 16-48kHz is poor, so for the 48kHz full-band audio signal, it can be divided into the first frequency band range of 0-16kHz and 16 -48kHz second frequency band range. For example, the amplitude spectrum MagY _48k (n,k) of the first noisy frequency signal can be divided into 0-16 kHz parts (ie, the first noisy amplitude spectrum) and 16-48 kHz (ie, the second noisy amplitude spectrum) part, which can be denoted as MagY _0-16k (n,k) and MagY _16-48k (n,k), respectively.

The first estimated noise reduction amplitude spectrum determining unit 503 may input the first noisy amplitude spectrum into the first processing model in the audio processing model to obtain the first estimated noise reduction amplitude spectrum for the first frequency band, wherein the first processing model pre-trained. Here, the first processing model is, for example, a 16kHz audio processing model, and MagY _0-16k (n,k) can be input into the 16kHz audio processing model to obtain MagY _{0-16k_p} (n,k) after noise reduction. For the training process of the first processing model, reference may be made to the foregoing description about the flowchart of the training method of the audio processing model shown in FIG. 2 , and details are not repeated here.

The second estimated noise reduction magnitude spectrum determining unit 504 may input the first estimated noise reduction magnitude spectrum and the second noisy magnitude spectrum into the second processing model in the audio processing model to obtain the second estimated noise reduction magnitude for the second frequency band range spectrum. Here, the second processing model is, for example, a 16-48kHz audio processing model, and the noise-reduced MagY _16k (n,k) and the second noisy amplitude spectrum of the amplitude spectrum for the first frequency band output from the first processing model can be MagY _16-48k (n,k) is used as the input of the 16-48kHz audio processing model, and the second original amplitude spectrum MagX _16-48k (n,k) is used as the training label to train the 16-48kHz audio processing model.

The first loss obtaining unit 505 may obtain a first loss based on the second estimated noise reduction magnitude spectrum and a second original magnitude spectrum, where the second original magnitude spectrum is the magnitude for the second frequency band in the magnitude spectrum of the first original audio signal Spectrum (for example, can be expressed as MagX _16-48k (n,k)).

The model parameter adjustment unit 506 may train the audio processing model by adjusting the model parameters of the second processing model according to the first loss. Here, the second original amplitude spectrum can be used as a training label, and after the second estimated noise reduction amplitude spectrum for the second frequency band range is obtained through the second processing model, a pre-designed loss function is determined based on these two parameters, and The model parameters of the second processing model are iteratively updated by back-propagation according to the determined loss function. During the training of the second processing model, the batch of audio training samples may be used to adjust (or update) the model parameters of the second processing model, and iteratively adjust (or update) the first The second process the model parameters of the model until the second process model converges.

FIG. 6 is a block diagram illustrating an audio processing apparatus 600 according to an exemplary embodiment of the present disclosure.

6 , an audio processing apparatus 600 according to an exemplary embodiment of the present disclosure may include an audio signal acquisition unit 610 and an audio signal processing unit 620 .

The audio signal acquisition unit 610 may acquire the audio signal to be processed. Here, the audio signal to be processed is a 48kHz full-band audio signal, and the processing process is to perform noise reduction on the audio signal.

The audio signal processing unit 620 may perform audio processing on the acquired audio signal to be processed by using the audio processing model, wherein the audio processing model is obtained by training based on the aforementioned training method of the audio processing model.

According to an exemplary embodiment of the present disclosure, the audio signal processing unit 620 may obtain a first amplitude spectrum for the first frequency band range and a second amplitude spectrum for the second frequency band range based on the amplitude spectrum of the audio signal to be processed, The first amplitude spectrum is input into the first processing model in the audio processing model to obtain a first estimated amplitude spectrum for the first frequency band range, and the second amplitude spectrum and the first estimated amplitude spectrum are input into the second processing model in the audio processing model, A second estimated amplitude spectrum for the second frequency band is obtained, the first estimated amplitude spectrum and the second estimated amplitude spectrum are combined to obtain an estimated amplitude spectrum, and a processed audio signal is obtained based on the estimated amplitude spectrum.

According to an exemplary embodiment of the present disclosure, the audio signal processing unit 620 may further include an amplitude spectrum acquisition unit 621, a first estimated amplitude spectrum determination unit 622, a second estimated amplitude spectrum determination unit 623, an estimated amplitude spectrum determination unit 624, and an audio signal Acquisition unit 625 .

FIG. 7 is a block diagram illustrating the audio signal processing unit 620 according to an exemplary embodiment of the present disclosure.

Referring to FIG. 7 , the amplitude spectrum obtaining unit 621 may obtain a first amplitude spectrum for the first frequency band range and a second amplitude spectrum for the second frequency band range based on the amplitude spectrum of the audio signal to be processed. The first frequency band range and the second frequency band range are obtained from the division of the full frequency band range of the amplitude spectrum of the audio signal to be processed. Specifically, since the audio noise reduction scheme in the related art has noise reduction in the frequency region below 16 kHz In the case of poor sound quality and poor noise reduction in the high frequency region of 16-48kHz, for the 48kHz full-band audio signal, it can be divided into the first frequency band range of 0-16kHz and the first frequency band of 16-48kHz. The second frequency band range. In some embodiments, the audio processing apparatus 600 may further include an audio signal amplitude spectrum acquisition unit 603 (not shown in FIG. 6 ), and the audio signal amplitude spectrum acquisition unit 603 may perform short-time Fourier transform on the audio signal to be processed (STFT: Short-Time Fourier Transform), an audio signal in the time-frequency domain is obtained, and then an amplitude spectrum is extracted based on the audio signal in the time-frequency domain to obtain the amplitude spectrum of the audio signal. Specifically, the audio signal to be processed is an audio signal in the time domain. In order to further process the audio signal, the short-time Fourier transform of the audio signal to be processed can be performed to convert it from the time domain to the time-frequency domain. , so as to obtain the time spectrum including the time information, frequency information, amplitude information and phase information of the audio signal to be processed, and the corresponding amplitudes of different frequency signals can be extracted from the time spectrum to form the amplitude spectrum. For the execution process of the short-time Fourier transform, reference may be made to the detailed description in the related art, and details are not described here. In some embodiments, a Fast Fourier Transform (FFT, Fast Fourier Transform) may also be performed on the audio signal to obtain an audio signal in the time-frequency domain, which is not limited.

The first estimated magnitude spectrum determining unit 622 may input the first magnitude spectrum into the first processing model in the audio processing model to obtain the first estimated magnitude spectrum for the first frequency band range. Here, the audio processing model is a pre-trained neural network model, and for the training process, reference may be made to the foregoing description about the flowchart of the training method for the audio processing model shown in FIG. 3 . For the 48kHz full-band audio signal, the first estimated amplitude spectrum determining unit 403 may input the amplitude spectrum with the frequency band range of 0-16kHz into the first processing model to obtain the first estimated amplitude spectrum for the 0-16kHz frequency band, wherein the first estimated amplitude spectrum is An estimated magnitude spectrum is the estimated magnitude spectrum after noise reduction of the magnitude spectrum in the frequency band range of 0-16 kHz.

The second estimated magnitude spectrum determining unit 623 may input the second magnitude spectrum and the first estimated magnitude spectrum into the second processing model in the audio processing model to obtain a second estimated magnitude spectrum for the second frequency band range. Here, for a 48kHz full-band audio signal, the amplitude spectrum in the frequency band range of 16-48kHz and the estimated amplitude spectrum in the 0-16kHz frequency band range output by the first processing model can be input into the second processing model. Obtain the estimated magnitude spectrum after noise reduction for the magnitude spectrum in the 16-48 kHz frequency band.

The estimated magnitude spectrum determining unit 624 may combine the first estimated magnitude spectrum and the second estimated magnitude spectrum to obtain the estimated magnitude spectrum. Here, for a 48kHz full-band audio signal, the estimated amplitude spectrum of the 48kHz frequency band range after noise reduction is obtained.

The audio signal obtaining unit 625 may obtain the processed audio signal based on the estimated amplitude spectrum.

According to an exemplary embodiment of the present disclosure, in order to obtain a visual and intuitive full-band audio signal after noise reduction, the audio signal acquisition unit 625 may multiply the estimated amplitude spectrum and the phase corresponding to the estimated amplitude spectrum, and then multiply the The result is subjected to an inverse short-time Fourier transform to obtain a processed audio signal in the time domain. Here, for a 48kHz full-band audio signal, the noise-reduced audio signal, for example, but not limited to, can be represented by formula (9).

FIG. 8 is a block diagram of a smart speaker 800 according to an exemplary embodiment of the present disclosure.

Referring to FIG. 8 , a smart speaker 800 according to an exemplary embodiment of the present disclosure includes the audio processing apparatus 600 shown according to the present disclosure. In a specific implementation process, the smart speaker 800 can be applied to scenarios such as video conferences, for example. In a video conference scenario, the smart speaker 800 can include a signal acquisition module, a signal processing module and a signal display module, wherein the signal acquisition module can collect the environment The audio signal in the signal processing module can process the audio signal collected by the signal acquisition module (for example, including using the audio processing method according to the exemplary embodiment of the present disclosure to perform noise reduction processing on the collected audio signal), the signal shows The module can display the audio signal processed by the signal processing module in the environment. Of course, the smart speaker 800 can also be applied to other scenarios, such as a home environment, etc., which is not limited. In different use environments, The composition and structure of the smart speaker 800 may be different. It should be clarified that as long as the smart speaker adopts the audio processing method shown in the present disclosure to perform audio noise reduction, it belongs to the scope of protection of the present disclosure.

FIG. 9 is a block diagram illustrating an electronic device 900 according to an exemplary embodiment of the present disclosure.

9, the electronic device 900 includes at least one memory 901 and at least one processor 902, the at least one memory 901 stores a computer-executable instruction set, when the computer-executable instruction set is executed by the at least one processor 902, executes A training method or an audio processing method of an audio processing model according to an exemplary embodiment of the present disclosure.

As an example, the electronic device 900 may be a PC computer, a tablet device, a personal digital assistant, a smart phone, or any other device capable of executing the above set of instructions. Here, the electronic device 900 is not necessarily a single electronic device, but can also be a collection of any device or circuit capable of individually or jointly executing the above-mentioned instructions (or instruction sets). Electronic device 900 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that interfaces locally or remotely (eg, via wireless transmission).

In electronic device 900, processor 902 may include a central processing unit (CPU), graphics processing unit (GPU), programmable logic device, special purpose processor system, microcontroller, or microprocessor. By way of example and not limitation, processors may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, and the like.

Processor 902 may execute instructions or code stored in memory 901, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transport protocol.

The memory 901 may be integrated with the processor 902, eg, RAM or flash memory arranged within an integrated circuit microprocessor or the like. Furthermore, memory 901 may comprise a separate device, such as an external disk drive, a storage array, or any other storage device that may be used by a database system. The memory 901 and the processor 902 may be operatively coupled, or may communicate with each other, eg, through I/O ports, network connections, etc., to enable the processor 902 to read files stored in the memory.

Additionally, the electronic device 900 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of electronic device 900 may be connected to each other via a bus and/or network.

According to exemplary embodiments of the present disclosure, there may also be provided a computer-readable storage medium storing instructions, wherein the instructions, when executed by at least one processor, cause the at least one processor to perform training of an audio processing model according to the present disclosure method or audio processing method. Examples of the computer-readable storage medium herein include: Read Only Memory (ROM), Random Access Programmable Read Only Memory (PROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Random Access Memory (RAM) , dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM , DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or Optical Disc Storage, Hard Disk Drive (HDD), Solid State Hard disk (SSD), card memory (such as a multimedia card, Secure Digital (SD) card, or Extreme Digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other apparatus configured to store and provide the computer program and any associated data, data files and data structures in a non-transitory manner with the computer program and any associated data, data files and data structures The computer program is given to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, a proxy device, a server, etc. Furthermore, in one example, the computer program and any associated data, data files and data structures are distributed over networked computer systems so that the computer programs and any associated data, data files and data structures are stored, accessed and executed in a distributed fashion by one or more processors or computers.

According to an exemplary embodiment of the present disclosure, there can also be provided a computer program product, the instructions in the computer program product can be executed by a processor of a computer device to complete the method for training an audio processing model according to an exemplary embodiment of the present disclosure or audio processing method.

According to the training method, apparatus, device, and medium of an audio processing model of the present disclosure, there are fewer or fewer high-band audio samples in high-band audio samples (eg, 48 kHz audio signal) relative to low-band audio samples (eg, 16 kHz audio signal) In the case where the data type of the sample (for example, 48kHz audio signal) does not match the data type of the low-band audio sample (for example, 16kHz audio signal), the pre-trained audio processing model (for example, for processing the audio frequency below 16kHz) ) to guide the training process of another processing model (for example, for processing audio with a frequency band of 16kHz-48kHz), so that the other processing model after the training has a good noise reduction effect, thereby enhancing the entire audio processing model. robustness.

In addition, according to the audio processing method, device, device, sound box and medium of the present disclosure, the amplitude spectrum of the audio signal to be processed is divided into an amplitude spectrum in a low frequency band range (eg, below 16 kHz) and a high frequency band range (eg, 16 kHz). -48kHz) amplitude spectrum, and input the two amplitude spectra into two audio processing models respectively for processing, and guide the audio processing of the amplitude spectrum in the high and low frequency bands with the audio processing results of the amplitude spectrum in the low frequency band range, so that it can be Avoid the problem of sound quality degradation after noise reduction due to high model complexity, and improve the noise reduction effect of audio in high frequency regions.

Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or techniques in the technical field not disclosed by the present disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (10)

1. a training method of an audio processing model, is characterized in that, comprises: Obtain a first training sample, wherein the first training sample includes an amplitude spectrum of a first frequency signal with noise and an amplitude spectrum of a first original audio signal, and the first frequency signal with noise is obtained by converting the first original audio It is obtained by mixing the signal and the noise signal; Based on the amplitude spectrum of the first noisy frequency signal, obtain a first noisy amplitude spectrum for the first frequency band range and a second noisy amplitude spectrum for the second frequency band range; inputting the first noisy amplitude spectrum into a first processing model in the audio processing model to obtain a first estimated noise reduction amplitude spectrum for the first frequency band, wherein the first processing model is pre-trained it is good; Inputting the first estimated noise reduction amplitude spectrum and the second noisy amplitude spectrum into a second processing model in the audio processing model to obtain a second estimated noise reduction amplitude spectrum for the second frequency band; Obtain a first loss based on the second estimated noise reduction amplitude spectrum and a second original amplitude spectrum, where the second original amplitude spectrum is a value for the second frequency band in the amplitude spectrum of the first original audio signal Amplitude spectrum; The audio processing model is trained by adjusting model parameters of the second processing model according to the first loss. 2 . The training method of an audio processing model according to claim 1 , wherein the first frequency band range and the second frequency band range are the full frequency range from the amplitude spectrum of the first noisy frequency signal. 3 . divided. 3 . The method for training an audio processing model according to claim 1 , wherein the first frequency band range is 0-16 kHz, and the second frequency band range is 16-48 kHz. 4 . 4. an audio processing method, is characterized in that, comprises: Get the audio signal to be processed; Use an audio processing model to perform audio processing on the to-be-processed audio signal to obtain a processed audio signal, wherein the audio processing model is obtained by training based on the training method of any one of claims 1-3. 5. A training device for an audio processing model, comprising: The first training sample obtaining unit is configured to: obtain a first training sample, wherein the first training sample includes the amplitude spectrum of the first band noise frequency signal and the amplitude spectrum of the first original audio signal, the first band The noise frequency signal is obtained by mixing the first original audio signal and the noise signal; A noisy amplitude spectrum acquiring unit configured to: obtain a first noisy amplitude spectrum for the first frequency band range and a second noisy amplitude spectrum for the second frequency band based on the amplitude spectrum of the first noisy frequency signal ; a first estimated noise reduction amplitude spectrum determining unit, configured to: input the first noisy amplitude spectrum into a first processing model in the audio processing model to obtain a first estimated noise reduction for the first frequency band range Amplitude spectrum, wherein the first processing model is pre-trained; The second estimated noise reduction magnitude spectrum determining unit is configured to: input the first estimated noise reduction magnitude spectrum and the second noisy magnitude spectrum into a second processing model in the audio processing model, and obtain a a second estimated noise reduction magnitude spectrum for the second frequency band; a first loss obtaining unit, configured to: obtain a first loss based on the second estimated noise reduction amplitude spectrum and a second original amplitude spectrum, where the second original amplitude spectrum is an amplitude spectrum of the first original audio signal the magnitude spectrum for the second frequency band range in ; A model parameter adjustment unit configured to: train the audio processing model by adjusting model parameters of the second processing model according to the first loss. 6. An audio processing device, characterized in that, comprising: an audio signal acquisition unit, configured to: acquire an audio signal to be processed; An audio signal processing unit, configured to: use an audio processing model to perform audio processing on the audio signal to be processed to obtain a processed audio signal, wherein the audio processing model is based on any one of claims 1-3 training method is obtained. 7. A smart speaker, comprising the audio processing device according to claim 6. 8. An electronic device, characterized in that, comprising: at least one processor; at least one memory storing computer-executable instructions, Wherein, the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the method for training an audio training model as claimed in any one of claims 1 to 3 or as in The audio processing method of claim 4. 9. A computer-readable storage medium storing instructions, wherein the instructions, when executed by at least one processor, cause the at least one processor to perform as claimed in any one of claims 1 to 3 The training method of the audio training model or the audio processing method according to claim 4. 10. A computer program product comprising computer instructions, characterized in that, when the computer instructions are executed by at least one processor, the training method of the audio training model according to any one of claims 1 to 3 or The audio processing method as claimed in claim 4.

Images