CN116547753A - Machine learning assisted spatial noise estimation and suppression - Google Patents

Abstract

In an embodiment, a method includes: a frequency band of a power spectrum of the input audio signal and microphone covariance are received, and for each frequency band: estimating respective probabilities of speech and noise using a classifier; estimating a set of averages of speech and noise, or a set of averages and covariances of speech and noise, using a directivity model based on the microphone covariances and the probabilities of the frequency bands; estimating means and covariance of noise power based on the probability and the power spectrum using a level model; determining a first noise suppression gain based on the directivity model; determining a second noise suppression gain based on the level model; selecting a first noise suppression gain or a second noise suppression gain or a sum of both based on a signal-to-noise ratio of the input audio signal; and scaling the time-frequency representation of the input signal by the selected noise suppression gain.

Description

Machine Learning-Aided Spatial Noise Estimation and Suppression

Cross References to Related Applications

This application claims priority to U.S. Provisional Application No. 63/110,228, filed November 5, 2020, and U.S. Provisional Application No. 63/210,215, filed June 14, 2021, each of which is filed by References are incorporated herein in their entirety.

technical field

The present disclosure relates generally to audio signal processing, and in particular to noise estimation and suppression in speech communications.

Background technique

Noise suppression algorithms for voice communications have been effectively implemented in edge devices such as phones, laptops, and conferencing systems. A common problem with two-way voice communications is that background noise at each user's location is transmitted along with the user's voice signal. If the signal-to-noise ratio (SNR) of the combined signal received at the edge device is too low, the intelligibility of the reconstructed speech will be reduced, resulting in a poor user experience.

Contents of the invention

Embodiments for machine learning assisted spatial noise estimation and suppression are disclosed. In some embodiments, a method of audio processing includes receiving frequency bands of a power spectrum of an input audio signal and a microphone covariance for each frequency band, wherein the microphone covariance is based on the frequency band of the microphone used to capture the input audio signal configuration; for each frequency band: use a machine learning classifier to estimate the corresponding probabilities of speech and noise; use a directionality model to estimate speech and noise based on the microphone covariance and the probabilities for the frequency band A set of means, or a set of mean and covariance of speech and noise; using a level model (level model) to estimate the mean and covariance of noise power based on said probability and said power spectrum; based on said directional model determining a first noise suppression gain based on a first output; determining a second noise suppression gain based on a second output of the level model; selecting either the first noise suppression gain or the one of the second noise suppression gains, or the sum of the first noise suppression gain and the second noise suppression gain; scaling by the selected first noise suppression gain or second noise suppression gain for the frequency band a time-frequency representation of the input signal; and converting the time-frequency representation into an output audio signal.

In some embodiments, the method further comprises: using said at least one processor, receiving an input audio signal comprising a certain number of blocks/frames; for each block/frame: using said at least one processor to convert said blocks Converting/frames to subbands, each subband having a different frequency spectrum than other subbands; combining the subbands into frequency bands using the at least one processor; and determining subband powers using the at least one processor.

In some embodiments, a machine learning classifier is a neural network comprising an input layer, an output layer, and one or more hidden layers. In an example, the neural network is a deep neural network comprising three or more layers, preferably more than 3 layers.

In some embodiments, the microphone covariance is represented as a normalized vector.

In some embodiments, the method further comprises: determining the first noise suppression gain further comprises: calculating a probability for speech in the frequency band; if the probability for speech in the frequency band is less than a threshold value, calculating the setting the first noise suppression gain equal to a maximum suppression gain; and setting the first noise suppression gain based on a gain ramp if the probability of speech for the frequency band is greater than a threshold.

In some embodiments, the probability of speech is calculated using said set of means and said covariance of speech and noise estimated by said directional model.

In some embodiments, said probability of speech is calculated using said set of means of speech and noise estimated by said directional model and said covariance vector, and a multivariate joint Gaussian density function.

In some embodiments, the method further comprises: determining the second noise suppression gain further comprises: setting the second noise suppression gain equal to a maximum suppression gain if the band power is less than a first threshold; if said frequency band power is between a first threshold and a second threshold (wherein the second threshold is higher than the first threshold), setting said second noise suppression gain equal to zero; and if said frequency band power is higher than the second threshold, the second noise suppression gain is set based on a gain ramp.

In some embodiments, said estimating using said directional model uses time-frequency slices classified as speech and noise, but excludes time-frequency slices classified as reverberation.

In some embodiments, the method further comprises: using a directional model or a level model to estimate a mean of speech based on the microphone covariance of the frequency band and the probability of speech further comprising: using a first order low pass filter calculating a time-averaged estimate of the mean of speech, wherein the mean of speech and the microphone covariance vector are inputs to the filter; and weighting the input of the filter by the probability of speech .

In some embodiments, the method further comprises: using a directional model or a level model based on the microphone covariance of the frequency band and the probability of noise to estimate the mean of noise further comprising: using first order low pass filtering calculating a time-averaged estimate of the mean of noise, wherein the mean of noise and the microphone covariance vector are inputs to the filter; and weighting the input of the filter by the probability of noise .

In some embodiments, the method further comprises: estimating the covariance of speech using a directional model or a level model based on the microphone covariance of the frequency band and the probability of speech further comprising: using a first-order low-pass a filter computing a time-averaged estimate of said covariance of speech, wherein said covariance of speech and said microphone covariance vector are inputs to said filter; Inputs are weighted.

In some embodiments, the method further comprises estimating the covariance of noise using a directional model or a level model based on the microphone covariance of the frequency band and the probability of noise further comprising using a first order low pass a filter computing a time-averaged estimate of said covariance of speech, wherein said covariance of speech and said microphone covariance vector are inputs to said filter; and said probability of passing noise is a function of said filter's Inputs are weighted.

In some embodiments, a system includes: one or more computer processors; and a non-transitory computer-readable medium storing instructions that, when executed by the one or more computer processors, cause The one or more processors perform the operations of any one of the aforementioned methods.

In some embodiments, a non-transitory computer-readable medium stores instructions that, when executed by one or more computer processors, cause the one or more processors to perform any one of the aforementioned methods way.

Other embodiments disclosed herein relate to a system, an apparatus, and a computer-readable medium. The details of the disclosed embodiments are set forth in the accompanying drawings and the description below. Other features, objects and advantages are apparent from the description, drawings and claims.

Certain embodiments disclosed herein provide one or more of the following advantages. The disclosed embodiments use directionality and machine learning (eg, neural networks) to provide low-cost high-quality noise estimation and suppression for voice communication applications. The disclosed embodiments of noise estimation and suppression can be implemented on various edge devices and do not require multiple microphones. The use of neural networks extends to a wide variety of background noises.

Description of drawings

In the drawings, specific arrangements or orderings of schematic elements, such as those representing devices, units, instruction blocks and data elements, are shown for ease of description. Those skilled in the art will understand, however, that the specific ordering or arrangement of schematic elements in the figures is not meant to imply a specific order or sequence of processing or separate processes. Further, the inclusion of a schematic element in a drawing does not imply that such element is required in all embodiments, or that in some implementations, the feature represented by the element may not be included in other elements or be combined with other elements. Combination of elements.

Further, in the drawings, where connecting elements such as solid or dashed lines or arrows are used to illustrate a connection, relationship or association between two or more other schematic elements, there are no such connecting elements It is not meant to imply that a connection, relationship or association cannot exist. In other words, some connections, relationships or associations between elements are not shown in the figures so as not to obscure the disclosure. Additionally, for ease of illustration, a single connection element is used to represent multiple connections, relationships or associations between elements. For example, where a connected element represents the communication of signals, data or instructions, those skilled in the art will understand that such an element represents one or more signal paths that may be required to effect the communication.

Figure 1 is a block diagram of a machine learning assisted spatial noise estimation and suppression system according to some embodiments.

FIG. 2 is a diagram illustrating level model based noise suppression gain calculations according to some embodiments.

FIG. 3 is a graph illustrating noise suppression gain calculation based on a directional model, according to some embodiments.

4A and 4B illustrate flowcharts for processing noise estimation and suppression in voice communications using directionality and machine learning, according to some embodiments.

FIG. 5 is a block diagram of a system for implementing the features and processes described with reference to FIGS. 1-4 , according to some embodiments.

The same reference numbers used in the various figures indicate similar elements.

Detailed ways

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. It will be apparent to those skilled in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments. Several features are described below, each of which may be used independently of the other or in any combination with the other features.

Terminology Explanation

As used herein, the term "comprising" and variations thereof should be understood as open-ended terms meaning "including but not limited to". The term "or" should be read as "and/or" unless the context clearly dictates otherwise. The term "based on" should be understood as "based at least in part on". The terms "one example embodiment" and "example embodiment" should be read as "at least one example embodiment". The term "another embodiment" should be understood as "at least one other embodiment". The term "(to) determine" shall be understood as obtaining, receiving, calculating, estimating, estimating, predicting or obtaining. Also, in the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

overview

Traditional noise suppression solutions capture background noise using two or more microphones, one closer to the user's mouth and one further away. The signals from these two microphones are subtracted to remove background noise common to both signals. However, this technique doesn't work on edge devices with a single microphone, or in the case of mobile phones (where the user shakes or turns the phone while speaking). Other noise suppression algorithms try to continuously find noise patterns in the audio signal and adapt to them by processing the audio frame by frame or block by block, where each block may consist of two or more frames. These existing adaptive algorithms work well for some use cases, but do not scale to a wide variety of background noises.

Recently, deep neural networks have been used to suppress noise in speech communication. However, these solutions require considerable computing power and are difficult to implement on real-time communication systems. The disclosed embodiments use a combination of directionality and machine learning (eg, deep neural networks) to provide low-cost high-quality noise estimation and suppression for voice communication applications.

FIG. 1 is a block diagram of a machine learning assisted spatial noise estimation and suppression system 100 in accordance with some embodiments. The system 100 includes a filter bank 101, a banding unit (banding unit) 102, a machine learning classifier 103 (for example, a neural network such as a deep neural network (DNN)), a direction detection unit 104, a speech/noise direction Sex model 105, noise level model 106, noise suppression gain unit 107, multiplication unit 108 and inverse filter bank 109. In some embodiments, a filter bank 101 (eg, a short-time Fourier transform (STFT)) receives a time domain input audio signal and converts the time domain input audio signal into subbands, each subband The bands have different frequency spectra (eg, time/frequency slices). The sub-bands of each block/frame are input into the band-dividing unit 102, which combines the sub-bands of the block/frame into frequency bands according to a psychoacoustic model and outputs the band-divided power 110 (e.g. with power). The subbands output by the filter bank 101 are also input into a direction detection unit 104 which generates and outputs a microphone covariance vector 112 . In some embodiments, each frequency band has a covariance vector, and a common band-splitting matrix is used in the band-splitting unit 102 and the direction detection unit 104 . In some embodiments, the band splitting unit 102 and the direction detection unit 104 are a single unit that generates the band power (in decibels) and the covariance vector for each band.

The subband power 110 is input into the machine learning classifier 103 . In an embodiment, the machine learning classifier 103 is a pre-trained neural network classifier that estimates and outputs probabilities 111 for a number of classes including, but not limited to, for each block/frame and each Speech class, stationary noise class, non-stationary noise class and reverberation class in each frequency band. In some embodiments, the speech/noise directionality model 105 and the noise level model 106 are driven using a stationary noise class.

In the following disclosure, it is assumed that a machine learning classifier estimates and outputs the probabilities of speech and stationary noise. In embodiments where the machine learning classifier outputs reverberation probabilities, estimated probabilities for speech, stationary noise, non-stationary noise, and reverberation are used to separate reverberation from noise. In some embodiments, any time-frequency tiles classified as reverberation are neither added to the speech/noise directionality model 105 nor to the noise level model 106 .

The probabilities 111 are input into the speech/noise directionality model 105 together with the microphone covariance vector 112 . The probabilities 111 are also input into the speech/noise level model 106 along with the subband powers 110 . For each frequency band and each block/frame, the speech/noise directivity model 105 estimates and outputs the respective mean and/or covariance 114 of speech and noise for each frequency band based on the microphone covariance vector 112 . For each frequency band, for each block/frame, the noise level model 106 estimates and outputs the mean and variance 113 of the noise power. In some embodiments, the noise level model 106 also outputs the mean and variance of the speech.

The noise suppression gain unit 107 receives the output 114 of the speech/noise directivity model 105 and the output 113 of the noise level model 106 , the subband power 110 and the microphone covariance vector 112 , respectively. The noise suppression gain unit 107 calculates and outputs a suppression gain, which is used by the multiplication unit 108 to scale the subbands output by the filter bank 101 to suppress noise in each subband. Then, the inverse filter bank 109 converts the subbands into time domain output audio signals. Each component of system 100 will be described in further detail below. In some embodiments, the output of the machine learning classifier 103 is directly used as a noise gain or used to calculate a noise suppression gain.

machine learning classifier

As mentioned above, the machine learning classifier 103 takes as input the subband power 110 and provides as output the probabilities 111 for speech and noise, which are given by:

{p _s (t, f), p _n (t, f)}, [1]

where t is the block/frame number, and f is the frequency band index, subscript "s" denotes speech, and subscript "n" denotes noise. In some embodiments, the noise is stationary noise based on the assumption that there is one related speaker that is not moving around and that the noise field is therefore relatively stationary. If these assumptions are violated, the modeling performed by 105 and 106 may be inappropriate. In some embodiments, the output of the machine learning classifier 103 may also include the probability of non-stationary noise and the probability of reverberation, and the noise suppression gain which may be directly used by the noise suppression gain unit 107 .

In the illustrated example, the machine learning classifier 103 includes a neural network (NN). The input to the example NN 103 includes 61 subband powers of the current frame, and the input linear dense layer maps the subband powers to 256 features. The 256 features are passed through a set of GRU layers (each GRU layer contains 256 hidden units) and finally a dense layer with some kind of non-linear kernel. The NN 103 outputs relative weights for each class and each frequency band, which are converted into probabilities using a softmax function. The NN 103 is trained using labeled speech mixed with noise and using cross-entropy as a cost function. In an embodiment, the neural network is trained using the Adam optimizer. Other examples of machine learning classifiers include, but are not limited to, k-nearest neighbors, support vector machines, or decision trees.

In an embodiment, the NN 103 may be trained as follows:

1. Obtain a clean voice recorded by a set of close-range microphones. For example, the Voice Cloning Toolkit (Voice Cloning Toolkit, VCTK) of the Voice Technology Center;

2. Obtain a set of noise data. For example, AudioSet data available at http://research.google.com/audioset/; and

3. Before the training process starts, features of speech and noise data are extracted. For example, 61 band energies (in dB) for frequency bands spaced between eg 50 Hz to 8000 Hz calculated at a rate of eg 50 Hz.

4. As part of the feature extraction process:

a. Determine a measure of speech power for each speech vector. For example, passing a voice file through a voice activity detector (VAD), and computing the mean A-weighted power (e.g., in dBFS) for all voices passing through the VAD; and

b. Determine a measure of noise power for each noise vector. For example, compute the mean A-weighted power (eg, in dBFS) over the entire vector.

5. For each speech vector at each training epoch:

a. selecting random noise vectors from the noise data set, associating speech with said random noise vectors;

b. Draw a random SNR with which to mix speech and noise. For example, drawing from a normal distribution with a mean of eg 20dB SNR and a standard deviation of eg 10dB SNR;

c. Determine the gain (in dB) at which to mix noise with speech based on the selected SNR, predetermined speech power (S) and predetermined noise power (N) according to gain=SNR−S+N;

d. Apply reverberation to speech, record in which time-frequency slices significant reverberation has been added;

e. Mix the noise with the reverberated speech using the gain provided. Since the feature is scaled in dB, "mixing" can be approximated by taking the max() of the reverberated speech power and the noise power after applying the gain in each time-frequency slice. When doing so, note in which time-frequency slice the noise ends up dominating the reverberated speech. The final training vector to be presented to the NN during training consists of the 61 frequency band energies per frame and ground-truth class labels based on whether noise, speech or reverberation dominates in each time-frequency slice during the above process; and

f. Show the training vectors to the network and drop the cross-entropy loss against the ground truth computed during the above process.

Microphone covariance matrix

In some embodiments, filterbank 101 is implemented using a short-time Fourier transform (STFT). Let _Xt (k) be the k-th bin data (subband) of the STFT of block/frame t of all microphone inputs. _Xt (k) is a vector of length N, where N is the number of microphones, M is the number of subbands, and α is a weighting factor that weights the contribution of past and estimated covariances. In real-time audio processing, the microphone covariance for frequency band f is calculated by:

in,

and[3]

where B(f) is the set of all STFT bins (ie, subbands) belonging to frequency band f.

Note that equations [2] and [3] hold for "rectangular partitioning", where each subband contributes with a gain of 1 to exactly one output band. In general, there is some weight w _kf in the range [0,1] that describes how much the input subband k contributes to the output frequency band f. For a subband k, all weights w _kf must sum to 1 over all frequency bands f. In this way, arbitrary shaped banding can be performed. For example, instead of rectangular banding, linear, logarithmic or cosine of Mel frequency or triangular shaped banding can also be used.

_{Σk=B(f) wk, f} =1. [5]

In an embodiment, the sub-band powers may be calculated directly from the filter bank 101 . For example, a non-uniform filter bank embedded with a banding scheme. In such an embodiment, no decimation is used, and the block rate defines the block frame t.

To simplify notation, the normalized covariance matrix is rearranged into a real vector 112(vt _,f ), since the covariance matrix is Hermitian and its diagonal elements are real numbers. Normalized covariance is the division of the covariance matrix by its trace such that the covariance matrix only represents direction, while any level components are removed.

If the elements of this covariance matrix are denoted c _m,n , the rearranged vector for a three-microphone system is given by, where m and n are the indices of the covariance matrix elements:

In some embodiments, v _t,f is normalized. A system with more or fewer microphones will have more or fewer elements, and thus the system is scalable to systems with any number of microphones.

Speech/Noise Directivity Model

In some embodiments, the speech/noise directionality model 105 takes as input the covariance vector v _t,f (one for each frequency band) and the probabilities 111 output by the machine learning classifier ₁₀₃ and estimates the mean and /or covariance matrix. At least two embodiments for estimating the mean and/or covariance matrix of v _t,f are described below:

1. Estimate the mean of speech and noise only, and use the estimated mean as the directionality model:

μ _n (t, f) = [1-wp _n (t, f)] · μ _n (t-1, f) + wp _n (t, f) · v _{t, f} , and [7]

μ _s (t, f)=[1-wp _s (t, f)]·μ _s (t−1,f)+wp _s (t,f)·v _t,f . [8]

In equations [7] and [8], μ _n (t, f) and μ _s (t, f) are the directionality models for noise and speech for block/frame t and frequency band f, respectively. They are the means of the normalized microphone covariance matrix. Note that w used in equations [7] and [8] and hereinafter is a weighting factor controlling the length of the temporal averaging window, which may be different for speech and noise or for mean and variance.

2. The mean and covariance vectors of noise and speech define a spatial model that can be calculated as:

μ _n (t, f) = [1-wp _n (t, f)] · μ _n (t-1, f) + wp _n (t, f) · v _{t, f} , [9]

μ _s (t, f) = [1-wp _s (t, f)] · μ _s (t-1, f) + wp _s (t, f) · v _{t, f} , [10]

Noise Level Model

The noise level model 106 takes as input the subband power 110(Lt _,f ) in dB and estimates the mean and variance of the noise for frequency band f and block/frame t:

μ _L (t, f) = [1-wp _n (t, f)] μ _L (t-1, f) + wp _n (t, f) L _{t, f} , [13]

Note that in equations [7] to [14], the mean and covariance of the random variable (level or directionality) are estimated using time averaging under the assumption that the random process is ergodic and stationary over time. The averaging is achieved using a first-order low-pass filter model, where, for each frame, the inputs to the low-pass filter (mean or covariance) are weighted by the probability of speech or noise.

Calculation of Noise Suppression Gain Based on Level Model

FIG. 2 is a graph illustrating the calculation of the noise suppression gain _GL (b,f) based on the noise level model 106 according to some embodiments. The vertical axis is gain (in dB), and the horizontal axis is level (in dB). In Figure 2, _G0 is the maximum suppression gain, and the slope β and k of the gain ramp are tuning parameters:

If the current signal-to-noise level is greater than a predefined signal-to-noise ratio threshold (see equation [19]), the noise suppression gain is _GL , otherwise the speech/noise directivity model 105 is used to calculate _Gs . The system calculates speech probabilities for frequency band f and block/frame t using one of at least two methods as follows:

1. Calculate the probability of speech using the normalized microphone covariance vector v _{t, the mean of f} and the speech/noise directionality model 105:

2. Use the mean and the vector v _{t, f} as a directional model (assuming that the elements of the vector v _{t, f} are joint Gaussian) to calculate the probability of speech:

in, is the multivariate joint Gaussian probability density function.

Calculation of Noise Suppression Gain Based on Directional Model

FIG. 3 is a graph illustrating noise suppression gain calculation based on the speech/noise directivity model 105 . The vertical axis is the gain (in dB), the horizontal axis is the probability of speech, and the noise suppression gain is given by:

where γ and _ks are tuning parameters.

For each frequency band f and each block/frame t, the final suppression gain G(t,f) (e.g. by the noise suppression gain unit 107) is calculated as follows:

where SNR ₀ is the predefined signal-to-noise ratio threshold, and is the estimated signal-to-noise ratio for block/frame t.

In some embodiments, the estimated in [19] This can be obtained by using the VAD (Voice Activity Detector) output to drive an automatic gain control (AGC) component (not shown) to level the speech signal to a predefined power level (in dB units), and subtract the estimated noise level _μL (t,f) from the predefined power level, and estimate the speech level using a method similar to that used for estimating noise, where:

μ _S (t, f) = [1-wp _s (t, f)] · μ _S (t-1, f) + wp _n (t, f) · L _{t, f} , and [20]

is calculated as:

The subbands output by the filter bank 101 are scaled using the final suppression gain G(t,f) in a multiplication unit 108 to suppress noise in each subband. For example, applying the gain G(t,f) to all subbands k belonging to frequency band f:

_Yt (k)= _Xt (k)*G(t,f) for k∈B(f), [22]

where Y _t (k) is the output of the multiplication unit 108 .

The inverse filter bank 109 then converts the output of the multiplication unit 108 into a time-domain output audio signal to obtain a noise-suppressed output signal.

example process

4A and 4B are flowcharts of a process 400 for noise estimation and suppression in speech communications using directional and deep neural networks, according to some embodiments. Process 400 may be implemented using system 500 shown in FIG. 5 .

Process 400 begins by receiving an input audio signal comprising a certain number of blocks/frames (401). For each block/frame, the process 400 continues with the following process: converting the block/frame into subbands (402), where each subband has a different frequency spectrum than the other subbands; combining the subbands into frequency bands and determining each frequency band power in (403); and determine microphone covariance based on the subbands (404).

Process 400 continues with the following process: For each frequency band and each block/frame, use a machine learning classifier (e.g., a neural network) to estimate the corresponding probabilities of speech and noise (405); use a directional model based on microphone covariance and probability to estimate a set of mean values of speech and noise or a set of mean values and covariance (406) of speech and noise; use the level model to estimate the mean value and variance (407) of noise power based on probability and frequency band power; first output to calculate a first noise suppression gain (408); based on a second output of the level model to determine a second noise suppression gain (409); and based on the signal-to-noise ratio of the input audio signal, select the first noise suppression gain or One of the second noise suppression gains, or the sum of the first noise suppression gain and the second noise suppression gain (410).

Process 400 continues by scaling each subband in each frequency band by the selected first noise suppression gain or second noise suppression gain for that band (411), and converting the scaled subbands to An audio signal is output (412).

Example system architecture

FIG. 5 shows a block diagram of an example system for implementing the features and processes described with reference to FIGS. 1-5 , according to an embodiment. System 500 includes any device capable of playing audio, including but not limited to: smartphones, tablet computers, wearable computers, vehicle computers, game consoles, surround systems, kiosks.

As shown, system 500 includes a central processing unit (CPU) 501 capable of being loaded into random access memory ( RAM) 503 programs to execute various processes. In the RAM 503, data required when the CPU 501 executes various processes is also stored as necessary. The CPU 501 , ROM 502 , and RAM 503 are connected to each other via a bus 509 . An input/output (I/O) interface 505 is also connected to the bus 504 .

The following components are connected to the I/O interface 505: an input unit 506, which may include a keyboard, a mouse, etc.; an output unit 507, which may include a display such as a liquid crystal display (LCD) and one or more speakers; a storage unit 508, which includes a hard disk or another suitable storage device; and a communication unit 509 including a network interface card such as a network card (for example, a wired network card or a wireless network card).

In some implementations, the input unit 506 includes one or more microphones located in different locations (depending on the host device) that enable the capture of various formats (e.g., mono, stereo, spatial, immersive and other suitable formats) audio signal.

In some implementations, the output unit 507 includes a system with various numbers of speakers. As illustrated in FIG. 5 , output unit 507 may (depending on the capabilities of the host device) present audio signals in various formats (eg, mono, stereo, immersive, binaural, and other suitable formats).

The communication unit 509 is configured to communicate with other devices (eg, via a network). A drive 510 is also connected to the I/O interface 505 as needed. A removable medium 511 such as a magnetic disk, optical disk, magneto-optical disk, flash drive, or other suitable removable medium is mounted on drive 510 such that a computer program read therefrom is installed into storage unit 508 as desired. Those skilled in the art will appreciate that although the system 500 is described as including the components described above, in practice, some of these components may be added, removed and/or substituted, and all such modifications or changes are fall within the scope of this disclosure.

Aspects of the systems described herein may be implemented in a suitable computer-based sound processing network environment for processing digital or digitized audio files. Portions of an adaptive audio system may include one or more networks comprising any desired number of independent machines including one or more routers (not shown) for buffering and routing data transmitted between computers. out). Such a network can be built on a variety of different network protocols and can be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

According to example embodiments of the present disclosure, the processes described above may be implemented as a computer software program or on a computer-readable storage medium. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising program code for performing a method. In such an embodiment, the computer program may be downloaded and installed from a network via the communication unit 509, and/or installed from a removable medium 511, as shown in FIG.

In general, various example embodiments of the present disclosure may be implemented in hardware or special purpose circuits (eg, control circuits), software, logic or any combination thereof. For example, the units discussed above may be executed by control circuitry (eg, a CPU combined with other components of FIG. 5 ), and thus the control circuitry may perform the actions described in this disclosure. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software (eg, control circuitry) that may be executed by a controller, microprocessor or other computing device. Although various aspects of the example embodiments of the present disclosure have been illustrated and described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that, by way of non-limiting example, the blocks, devices, systems, techniques described herein Or a method may be implemented in hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controllers, or other computing devices, or some combination thereof.

In addition, the various blocks shown in the flowcharts may be regarded as method steps, and/or operations resulting from operation of computer program code, and/or multiple couplings configured to perform the associated function(s) Logic circuit elements. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising program code configured to perform the methods described above.

In the context of the present disclosure, a machine-readable medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may be non-transitory and may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include electrical connections having one or more conductors, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

Computer program code for carrying out the methods of the present disclosure may be written in any combination of one or more programming languages. These computer program codes may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus with a control circuit, so that when the program code is executed by the processor of the computer or other programmable data processing apparatus, Implement the functions/operations specified in the flowchart and/or block diagrams. The program code may execute entirely on the computer, partly on the computer, as a stand-alone software package, partly on the computer and partly on a remote computer or entirely on a remote computer or server, or distributed over one or on multiple remote computers and/or servers.

While this document contains many implementation details, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed as such, in some cases one or more features of the claimed combination may be removed from the combination and the claimed A protected combination may involve a subcombination or a variation of a subcombination. The logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be deleted, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.

Claims (15)

1. An audio processing method, comprising: receiving, using at least one processor, frequency bands of a power spectrum of an input audio signal and a microphone covariance for each frequency band, wherein the microphone covariance is based on a configuration of microphones used to capture the input audio signal; For each frequency band: use machine learning classifiers to estimate the corresponding probabilities of speech and noise; estimating a set of means for speech and noise, or a set of means and covariance for speech and noise, using a directional model based on the microphone covariance and the probability for the frequency band; estimating a mean and covariance of noise power based on the probability and the power spectrum using a level model; determining, using the at least one processor, a first noise suppression gain based on a first output of the directional model; determining a second noise suppression gain based on a second output of the level model using the at least one processor; Using the at least one processor to select one of the first noise suppression gain or the second noise suppression gain, or the combination of the first noise suppression gain and the second noise suppression gain, based on a signal-to-noise ratio of the input audio signal The sum of two noise suppression gains; scaling a time-frequency representation of the input signal using the at least one processor by the selected first noise suppression gain or second noise suppression gain for the frequency band; and The time-frequency representation is converted to an output audio signal using the at least one processor. 2. The method of claim 1, further comprising: receiving an input audio signal comprising a certain number of blocks/frames using the at least one processor; For each block/frame: converting said blocks/frames into subbands, each subband having a different frequency spectrum than other subbands, using said at least one processor; combining the subbands into frequency bands using the at least one processor; and Part-band power is determined using the at least one processor. 3. The method of claim 1, wherein the machine learning classifier is a neural network. 4. The method of claim 1 or 2, wherein the microphone covariance is represented as a normalized vector. 5. The method of any one of the preceding claims 1 to 4, wherein determining the first noise suppression gain comprises: calculating a probability of speech for the frequency band; If the probability for speech in the frequency band is less than a threshold, setting the first noise suppression gain equal to a maximum suppression gain; and The first noise suppression gain is set by gradually changing from the maximum suppression gain towards zero to increase the probability of speech. 6. The method of claim 5, wherein the probability of speech is calculated using the set of means and the covariance of speech and noise estimated by the directional model. 7. The method of claim 5, wherein the set of means and the covariance of speech and noise estimated by the directional model and a multivariate joint Gaussian density function are used to calculate the said probability. 8. The method of any one of the preceding claims 1 to 7, wherein determining the second noise suppression gain comprises: if said band power is less than a first threshold, setting said second noise suppression gain equal to a maximum suppression gain; The noise suppression gain is set by gradually changing from the maximum suppression gain towards zero to increase the probability of speech. 9. A method as claimed in any one of the preceding claims 1 to 8, wherein said estimating using said directional model uses time-frequency slices classified as speech and noise, but excludes time-frequency slices classified as reverberation time-frequency slice. 10. The method according to any one of the preceding claims 1 to 9, wherein, based on the microphone covariance of the frequency band and the probability of speech, a directional model or a level model is used to estimate the mean value of speech further include: A time-averaged estimate of the mean of speech is computed using a first-order low-pass filter, wherein the mean of speech and the microphone covariance are inputs to the filter; The input to the filter is weighted. 11. The method according to any one of the preceding claims 1 to 10, wherein, based on the microphone covariance of the frequency band and the probability of noise, a directional model or a level model is used to estimate the mean value of noise further include: A time-averaged estimate of the mean of noise is computed using a first-order low-pass filter, wherein the mean of noise and the microphone covariance vector are inputs to the filter; and the probability of passing noise contributes to the The input to the filter is weighted. 12. The method according to any one of the preceding claims 1 to 11, wherein the covariance of speech is estimated using a directional model or a level model based on the microphone covariance of the frequency band and the probability of speech Further includes: Computing a time-averaged estimate of said covariance of speech using a first-order low-pass filter, wherein said covariance of speech and said microphone covariance vector are inputs to said filter; and said probabilities of passing speech are The input to the filter is weighted. 13. The method according to any one of the preceding claims 1 to 12, wherein the covariance of noise is estimated using a directional model or a level model based on the microphone covariance of the frequency band and the probability of noise Further includes: Computing a time-averaged estimate of said covariance of noise using a first-order low-pass filter, wherein said covariance of noise and said microphone covariance vector are inputs to said filter; and said probability of passing noise The input to the filter is weighted. 14. A system comprising: one or more computer processors; and A non-transitory computer readable medium storing instructions which, when executed by the one or more computer processors, cause the one or more processors to perform any one of claims 1 to 13 operation. 15. A non-transitory computer readable medium storing instructions which, when executed by one or more computer processors, cause the one or more processors to perform any one of claims 1 to 13 described operation.