CN118974825A - Source separation combining spatial cues and source cues - Google Patents

Abstract

The present disclosure relates to an audio processing method and system for source separation. The method comprises obtaining an input audio signal (A) comprising at least two channels, and processing the input audio signal (A) using a separation module (10) based on spatial cues to obtain an intermediate audio signal (B). The separation module (10) based on spatial cues is configured to determine mixing parameters of at least two channels of the input audio signal (A) and modify the channels based on the mixing parameters to obtain the intermediate audio signal (B). The method further comprises processing the intermediate audio signal (B) using a separation module (20) based on source cues to generate an output audio signal (C), wherein the separation module (20) based on source cues is configured to implement a neural network, which is trained to predict a noise-reduced output audio signal (C) given the intermediate audio signal (B).

Description

Source separation combining spatial cues and source cues

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No. 63/325,108 filed on March 29, 2022, U.S. Provisional Application No. 63/417,273 filed on October 18, 2022, and U.S. Provisional Application No. 63/482,949 filed on February 2, 2023, each of which is incorporated herein by reference in its entirety.

Technical Field

The present invention relates to a method and an audio processing system for source separation based on spatial cues and source cues.

Background Art

Source separation in audio processing relates to systems and methods for isolating a target audio source (e.g., speech or music) present in an original audio signal that includes a mixture of the target audio source and additional audio content, such as stationary or non-stationary noise, background audio, or reverberation effects.

There are two main types of target separation processing, namely spatial cue-based separation that utilizes spatial cues (information describing how the target audio is mixed) and source cue-based separation that utilizes source cues (information describing what the target audio sounds like).

A simple example of spatial cue separation is the extraction of speech from a 5.1 soundtrack of a movie. The spatial cue used for this separation is that speech or dialogue is usually mixed to the center (C) channel, so the spatial separation system only needs to extract the center channel to obtain a spatially separated dialogue channel. Alternatively, separation based on spatial cues involves amplifying the center channel or mixing the center channel with other channels in the 5.1 presentation to obtain a 5.1 presentation with higher dialogue intelligibility.

A simple example of source cue-based separation is to use a bandpass filter whose passband is adapted to match the expected frequency range of the target audio source. If the target audio source is speech, a bandpass filter with a passband of 500 Hz to 8 kHz can be used, because most of the spectral energy of human speech is expected to be in this frequency range. More advanced source cue separation systems operate on audio signals represented in the time-frequency domain and employ neural networks trained to predict the gains of each time-frequency tile of the audio signal, where these gains suppress all audio content that does not belong to the target audio source.

Summary of the invention

The problem with the above solutions is that the source-cue-based separation process using source cues completely ignores the spatial cues, and the spatial-cue-based separation process using spatial cues completely ignores the source cues, which means that not all information is considered when performing target source separation. On the other hand, combining different source separation processes is not easy, and in many cases, combining two or more different target source separation processes results in degraded performance compared to using only one target source separation process.

Therefore, there is a need for an improved target separation method and system that overcomes at least some of the disadvantages mentioned above.

According to a first aspect of the present invention, there is provided an audio processing method for source separation. The method comprises obtaining an input audio signal comprising at least two channels, and processing the input audio signal using a separation module based on spatial cues to obtain an intermediate audio signal. The separation module based on spatial cues is configured to determine mixing parameters of at least two channels of the input audio signal and modify the at least two channels based on the mixing parameters to obtain the intermediate audio signal. The method further comprises processing the intermediate audio signal using a separation module based on source cues to generate an output audio signal, wherein the separation module based on source cues is configured to implement a neural network, which is trained to predict a denoised output audio signal given a sample of the intermediate audio signal.

The source cue-based separation module is configured to remove at least one of stationary noise (such as white noise), non-stationary noise (including time-varying noise, such as traffic noise or wind noise), background audio content (e.g., speech from sources other than the target speaker), and reverberation.

In other words, the spatial cue based separation module is configured to separate the audio content based on how the audio content is mixed, while the source cue based separation module is configured to separate the audio content based on how the audio content sounds.

By first performing spatial cue-based source separation using mixing parameters and then performing neural network-based source cue-based separation, the overall performance of the source separation method is improved. In particular, since the neural network-based source cue-based separation can be trained specifically to operate on spatially separated audio sources, and the previous spatial cue-based separation module achieves such spatial separation, the performance of the source cue-based separation module is improved. In one example, the spatial cue-based separation module modifies the input audio signal to make it close to a center-panned mixture, which is approximately single-channel, and the source cue-based separation module is trained to suppress noise of the center-panned audio signal.

In some embodiments, the spatial cue-based separation module operates at a first time and/or frequency resolution, and the method further comprises providing metadata to the source cue-based separation module by the spatial cue-based separation module, wherein the metadata indicates the time and/or frequency resolution of the spatial cue-based separation module. The method further comprises generating, by the source cue-based separation module, the output audio signal based on the intermediate audio signal and the metadata.

For example, the time and/or frequency resolution of the separation module based on source cues is reduced to match the time and/or frequency of the separation module based on spatial cues. In some examples, the time and/or frequency resolution of the separation module based on source cues is reduced by processing the output of the separation module based on source cues using a smoothing window and/or a smoothing kernel. If the time and/or frequency metadata are not taken into account, the two separation modules will operate independently at different resolutions, which may result in noticeable acoustic artifacts.

In some embodiments, the separation module based on the source cue predicts a source gain mask, which is applied to the intermediate audio signal to suppress noise. Time and/or frequency resolution metadata can be used to smooth the gain mask to form a smoothed gain mask, which is applied to the intermediate audio signal. Wherein the degree of smoothing (i.e., the reduction in resolution) is based on the time and/or frequency metadata.

In some embodiments, the spatial cue-based separation module determines the mixing parameters with a lower (coarser) time and/or frequency resolution than the source cue-based separation module, preferably at least two times lower, more preferably at least four times lower, even more preferably at least six times lower, or most preferably at least eight times lower.

That is, the temporal and/or frequency resolutions of the two modules may differ substantially because the temporal and/or frequency resolutions that are best suited for separation based on spatial cues differ substantially from the corresponding temporal and/or frequency resolutions used to perform separation based on source cues.

According to a second aspect of the present invention, a system for source separation is provided, the system comprising a separation module based on spatial cues, the separation module based on spatial cues being configured to obtain an input audio signal comprising at least two channels and process the input audio signal to obtain an intermediate audio signal, wherein the separation module based on spatial cues is configured to determine mixing parameters of at least two channels of the input audio signal and modify the at least two channels based on the mixing parameters to obtain the intermediate audio signal. The system further comprises a separation module based on source cues, the separation module based on source cues being configured to process the intermediate audio signal to generate an output audio signal by implementing a neural network, the neural network being trained to predict a denoised output audio signal given a sample of the intermediate audio signal.

The system according to the second aspect has the same or equivalent benefits as the method according to the first aspect.Any functionality described with respect to the method may have a corresponding feature in the system or device, and vice versa.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the invention will be described in more detail with reference to the accompanying drawings, which show currently preferred embodiments.

FIG. 1 is a block diagram of an audio processing system for source separation according to some implementations.

2 is a block diagram illustrating an audio processing system for implementing source separation and remixing of input audio signals according to some embodiments.

FIG. 3 is a flow chart describing an audio processing method for source separation according to some embodiments.

4 is a block diagram illustrating an audio processing system for source separation with a source cue based separation module for predicting a source separation gain mask according to some embodiments.

5 is a block diagram illustrating an audio processing system for source separation in cooperation with a classifier unit and a gating unit according to some embodiments.

DETAILED DESCRIPTION

The system and method disclosed in the present application can be implemented as software, firmware, hardware or a combination thereof. In hardware implementation, task division does not necessarily correspond to the division of physical units; on the contrary, a physical component can have multiple functions, and a task can be performed collaboratively by several physical components. Computer hardware can be, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a smart phone, a web device, a network router, a switch or a bridge, or any machine capable of (sequentially or otherwise) executing instructions specifying the action to be taken by the computer hardware. Further, the present disclosure will relate to any set of computer hardware that executes instructions individually or jointly to perform any one or more concepts discussed herein.

Some or all of the components may be implemented by one or more processors that accept a computer-readable (also referred to as machine-readable) code containing a set of instructions that, when executed by one or more processors, perform at least one method described herein. Any processor that is capable of executing a set of instructions (sequential or otherwise) that specifies an action to be taken is included. Thus, an example is a typical processing system (i.e., computer hardware) that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem that includes a hard drive, an SSD, a RAM, and/or a ROM. A bus subsystem may be included for communication between components. The software may reside in the memory subsystem and/or the processor during its execution by the computer system.

The one or more processors may operate as standalone devices or may be connected to other processor(s), such as via a network. Such a network may be constructed on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

The software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, various forms of physical (non-transitory) storage media, such as EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage devices, cassettes, tapes, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Further, it is well known to those skilled in the art that (transitory) communication media typically embodies computer-readable instructions, data structures, program modules or other data in the form of modulated data signals or other transmission mechanisms such as carrier waves, and includes any information transfer medium.

Fig. 1 depicts a source separation audio processing system 1 for performing source separation based on both spatial cues and source cues. The audio processing system 1 obtains an input audio signal A, which is provided to a separation module based on spatial cues 10. The separation module based on spatial cues 10 processes the input audio signal A and outputs an intermediate audio signal B.

The input audio signal A comprises at least two audio channels. For example, the input audio signal A is a stereo or binaural audio signal having a left audio channel and a right audio channel. The separation module 10 based on spatial cues is configured to extract at least one mixing parameter of the input audio signal A and modify the at least two audio channels based on the at least one mixing parameter to obtain the intermediate audio signal B.

Mixing parameters indicate the properties of the mixture of at least two audio channels. One or more mixing parameters can be determined for a single frequency band or for multiple frequency bands and these parameters are regularly updated. For example, the audio signal is divided into a plurality of continuous (optionally overlapping) blocks, and the mixing parameters are determined by aggregating fine-grained mixing parameters on at least one block frequency band. In some embodiments, the mixing parameters indicate at least two channels of translation distribution and at least one of the channel phase difference distribution (e.g., mean or median) of at least two audio channels in the block frequency band. Blocking includes at least two frames, wherein each frame is further divided into a plurality of tiles covering narrow frequency bands, as will be further described below.

The processing performed by the separation module 10 based on spatial cues may require adjusting at least two audio channels to approach a predetermined mix type based on the detected mixing parameters. In some embodiments, at least two different mixing parameters (e.g., both the panning distribution and the inter-channel phase difference distribution) are determined and used when adjusting the mix. An example of this is presented below, in which four mixing parameters θ-center, θ-width, Φ-center, Φ-width are determined and used to adjust the mix. The predetermined mix type is selected based on the capabilities of the subsequent separation module 20 based on source cues. For example, the predetermined mix type can be a mix with approximately central panning and/or a mix with almost no inter-channel phase difference.

For example, a subsequent source cue-based separation module 20 may be configured to process a downmixed version of an intermediate audio signal B having at least two channels. In some embodiments, the source cue-based separation module 20 first extracts a downmixed mid-audio signal from at least two channels of the intermediate audio signal B, analyzes the downmixed mid-audio signal to determine a masking gain to suppress noise in the downmixed mid-audio signal, and applies the masking gain to the channels of the intermediate audio signal B. To this end, the already spatially separated intermediate audio signal B is centrally panned and/or contains almost no inter-channel phase differences, and will be very suitable for processing using this type of source cue-based separation module 20, because, for example, most of the desired content will be included in the downmixed mid-audio signal. By comparison, if the intermediate audio signal B is not spatially separated, there is a risk that the relevant audio content is excluded from the downmix and will not be properly considered by the neural network of the source cue-based separation module 20.

The separation module 10 based on spatial cues can operate in a transform domain (such as in a short-time Fourier transform (STFT) domain or a quadrature mirror filter bank (QMF) domain) or in a time domain (such as in a waveform domain). In either case, the input audio signal A includes at least two audio channels, such as a left channel L and a right channel R of a stereo audio signal. However, the audio channels are not necessarily left channel L and right channel R, and can be, for example, a left channel L and a center channel C presented by 5.1, a center channel C and a right channel R presented by 5.1, or any selection of two audio channels presented arbitrarily. In addition, by "the input audio signal includes at least two audio channels", the input here refers to any audio input with multiple signals, not just such signals conventionally referred to as "channels". For example, the signal of the input audio signal may include surround audio channels, multi-track signals, high-order high-fidelity stereo signals, object audio signals and/or immersive audio signals. The input audio signal A can be divided into a plurality of continuous time domain frames, wherein each frame is further divided into a plurality of tiles, each tile covering a narrow frequency band, thereby giving a fine-grained tile representation. Tiles are sometimes referred to as time-frequency tiles, and as an example, each tile covers a separate STFT frequency bin. Thus, each tile represents a limited duration of the audio signal in a predetermined narrow frequency band. Each fine-grained time-frequency tile represents a very short duration and/or a very narrow frequency band (e.g., approximately one or more orders of magnitude shorter and/or narrower) compared to a partition of all tiles comprising at least two consecutive frames.

The frequency band covered by a tile is usually quite narrow, for example, about 10 Hz, and the duration covered by each tile or frame is also quite short, for example, about 20 ms. The duration covered by a block (including at least two consecutive frames) is longer (for example, 10 consecutive frames), and it is also envisioned that the block can be divided into block bands, wherein the block bands are wider than the bands covered by a single tile. For example, the block can be implemented with block bands such as 400 Hz to 800 Hz, 800 Hz to 1600 Hz, 1600 Hz to 3200 Hz, etc., which are much wider than the narrower bands covered by each tile.

In a first example of the operation of the spatial cue-based separation module 10, the module first detects fine-grained mixing parameters for each tile (e.g., STFT tile) of the input audio signal A. Secondly, the spatial cue-based separation module 10 determines the distribution (multiple) of the fine-grained mixing parameters over multiple tiles, and modifies the channels based on the distribution (multiple) of the fine-grained mixing parameters. When describing this example and other examples, it will be assumed that the audio channels are the left channel L and the right channel R, however, the same processing can be applied to any pair of audio channels as mentioned above, for example, an LC (left and center) pair, an RC (right and center) pair, or an Ls-Rs (left surround and right surround) pair.

For each fine-grained tile, the panning mixing parameters θ of the detected left audio channel L and right audio channel R can be determined as

Therein, Θ ranges from 0 (indicating a fully left L-panned audio signal) to π/2 (indicating a fully right R-panned audio signal), where Θ=π/4 indicates a center-panned audio signal.

Similarly, for each fine-grained tile, the detected inter-channel phase difference mixing parameter Φ of the left audio channel L and the right audio channel R can be determined as

Here, Φ ranges from −π to π, where Φ=0 indicates no inter-channel phase difference between the left audio channel L and the right audio channel R.

Furthermore, the detected signal amplitude mixing parameter (expressed in decibels) can be determined for each fine-grained tile as

U _dB = 10log ₁₀ (|L| ² + |R| ² ). (Equation 3)

The separation module 10 based on spatial cues can detect one or more of the tile-specific mixing parameters in equations 1, 2, and 3 above, and adjust the audio channels to approach, for example, a centrally panned audio signal with no inter-channel phase difference. However, since each tile typically covers a very short duration (e.g., 1 ms to 30 ms, such as 20 ms) and a very narrow frequency range, the tile-specific mixing parameters can change rapidly over time and/or frequency. To this end, the tile-specific translations and inter-channel phase differences of multiple tiles are combined and optionally weighted with tile-specific amplitudes U _dB to form a translation distribution and/or inter-channel phase difference distribution over multiple tiles. These distributions can then be updated and used to adjust the channels at regular intervals (which are much longer than a single tile or frame) to approach a predetermined mixing type.

For example, tiles of multiple frames (such as 5 or 10 frames) are aggregated into an audio signal block, wherein the block includes audio signal content between 200 and 300 ms, and the block can be divided into relatively coarse (e.g., octave or half octave) block frequency bands. In some embodiments, an average translation called Θ-in is determined over all tiles in a block frequency band, and an associated translation distribution parameter called Θ-width is determined over all tiles in a block frequency band, the associated translation distribution parameter indicating a symmetrical deviation from Θ-in, which captures a predetermined proportion of the total signal energy (e.g., 40% of the energy). Similarly, an average inter-channel phase difference called Φ-in is determined over all tiles in a block frequency band, and an associated inter-channel phase distribution parameter called Φ-width is determined for each block frequency band, the associated inter-channel phase distribution parameter indicating a symmetrical deviation from Φ-in, which captures a predetermined proportion of the total signal energy (e.g., 40% of the energy).

Then, the modification of the left audio channel L and the right audio channel R may include adjusting the panning and/or inter-channel phase difference of each blocked frequency band so as to move the θ-center and/or Φ-center to a predetermined position, for example, Φ=0 and θ=0 for a center panned predetermined mix without inter-channel phase difference. Additionally or alternatively, the modification of the left audio channel and the right audio channel may require "squeezing" the corresponding distribution so as to reduce the θ-width and/or Φ-width to a predetermined width or a predetermined factor.

In an exemplary embodiment, the spatial cue-based separation module 10 operates in the STFT domain, with a sampling rate of 48kHz, a frame including 4096 samples, a frame step of 1024 samples (i.e., 75% overlap), and a Hanning window or a square root of a Hanning window is used. The mixing parameters are determined for each block band, where a block includes 10 frames (1 current frame, 4 lookahead frames, and 5 lookback frames) and the block step is 5 frames. That is, when determining the mixing parameters in each block band, approximately 277ms of the total buffered content is considered. In the case of an overlap rate of 75% between tiles (frames) and a block step of 5 frames, the mixing parameters can be updated once every 5×1024 samples (or approximately once every 107ms at a sampling rate of 48kHz), which determines the temporal resolution of the spatial cue-based separation module 10. In addition, it is envisaged that at least one mixing parameter is interpolated between blocks. For example, the mixing parameters are interpolated once per frame, which means that the mixing parameters are updated every 1024 samples (or approximately every 20 ms at a sampling rate of 48 kHz).

The frequency resolution of the spatial cue-based separation module 10 is determined by the number and bandwidth of the block frequency bands into which each block is divided. In an exemplary embodiment, the spatial cue-based separation module 10 operates on quasi-octave block frequency bands with band edges at 0 Hz, 400 Hz, 800 Hz, 1600 Hz, 3200 Hz, 6400 Hz, 13200 Hz, and 24000 Hz, thereby forming seven frequency bands with different bandwidths ranging from a 400 Hz bandwidth covering the 0 to 400 Hz band to a 10800 Hz bandwidth covering the 13200 Hz to 24000 Hz band.

The above-mentioned time and/or frequency resolution of the separation module 10 based on spatial cues is only exemplary, and other alternatives are envisioned. For example, it is envisioned that fewer or more frames are combined when forming blocks, and/or the time step/overlap of tiles (frames) and blocks can vary. However, in general, the separation module 10 based on spatial cues benefits from operating at a relatively low time/frequency resolution compared to the subsequent separation module 20 based on source cues. For example, the time and/or frequency resolution of the separation module 10 based on spatial cues to determine the mixing parameters is coarser than the time and/or frequency resolution of the source separation module, such as at least twice, at least four times, at least six times, at least eight times, or at least ten times.

For example, the processing of the spatial cue-based separation module 10 can be as described in Master, Aaron S et al., “Dialog Enhancement via Spatio-Level Filtering and Classification”, AES Convention Paper 10427.

As a second example of the operation of the separation module 10 based on spatial cues, the panning and/or inter-channel phase difference mixing parameters determined from the plurality of detected fine-grained mixing parameters can be used as target panning parameters θ and/or target phase difference parameters Φ, as described in U.S. Provisional Application No. 63/318,226, “TARGET MID-SIDE SIGNALSFOR AUDIO APPLICATIONS,” filed on March 9, 2022, which is incorporated herein by reference in its entirety. Here, the mixing parameters can be used to extract a centrally panned target mid audio channel M and a target side audio channel S from an input left audio channel L and an input right audio channel R, as shown below

Among them, the target mid audio signal M will be targeted for any dominant audio source to be included in each frequency band. Then, a central panned middle audio signal with a left audio channel _Lint and a right audio channel _Rint can be extracted from the target mid audio channel M and the target side audio channel S as shown below

_Lint = M + S (Equation 6)

R _int =MS (Equation 7)

The dominant audio source of the input audio signal A has been shifted to the center pan, reducing the inter-channel phase difference. Therefore, extracting the target mid-audio signal M and reconstructing the center-panned left audio channel _Lint and right audio channel Rint _pair is another exemplary method that can achieve spatial source separation based on one or more mixing parameters.

In some embodiments, the target-side audio signal S will be ignored (e.g., set to zero) when determining the center-panned intermediate audio signal channels _Lint , _Rint in Equations 6 and 7. Since many target audio sources will be fully captured by the target mid-audio signal M, the target-side audio signal S will mainly contain unwanted audio signal components, which means it can be ignored.

In the above, different examples of the operation of the separation module 10 based on spatial cues have been presented. It can be understood from these examples that the separation module 10 based on spatial cues performs a detection operation and an extraction operation. The detection operation includes determining at least one detected mixing parameter with a fine-grained time-frequency resolution (e.g., determining at least one detected mixing parameter for each tile), wherein the extraction operation involves smoothing the detected at least one fine-grained mixing parameter over time and/or frequency (e.g., aggregating the fine-grained mixing parameters on the block frequency bands) to obtain a relatively coarser mixing parameter. The time and/or frequency resolution of the separation module 10 based on spatial cues is based on the coarser time and/or frequency resolution of the extraction operation. Then, the coarser at least one mixing parameter is used for the final adjustment of the mixing. That is, the detected fine-grained mixing parameters are not directly used to control the mixing, because this may introduce significant acoustic artifacts due to the rapid adjustment of the mixing (e.g., for each STFT tile).

The spatial cue based separation module 10 outputs a generated intermediate audio signal B comprising spatially mixed audio content that is easier for the source cue based separation module 20 to process (eg, a centrally panned audio signal with almost no inter-channel phase difference).

The source cue-based separation module 20 comprises a neural network that is trained to predict a noise-reduced output audio signal C given a sample of an intermediate audio signal B. The neural network has been trained, for example, to recognize target audio content (e.g., speech or music) and amplify the target audio content and/or has been trained to recognize undesirable audio content (e.g., stationary noise or non-stationary noise) and attenuate the undesirable audio content. To achieve this, the neural network may comprise a plurality of neural network layers and may, for example, be a recurrent neural network.

For example, the neural network in the source cue-based separation module 20 is a U-Net type architecture, where the input to the neural network is the band energy and the output is the real-valued band gain. This type of U-Net architecture is sometimes referred to as U-NetFB. Given a stereo intermediate audio signal B, U-NetFB first downmixes the audio signal before predicting the gain mask based on the downmix audio signal, thereby applying the generated gain mask to the two audio channels of the intermediate audio signal B.

As another example, the neural network in the source cue-based separation module 20 is an aggregated multi-scale convolutional neural network having multiple parallel convolution paths, each convolution path including one or more convolution layers. With this neural network, an aggregate output is formed by aggregating the outputs of the parallel convolution paths, thereby generating an output gain mask based on the aggregate output. This type of neural network is described in more detail, for example, in “METHOD AND APPARATUS FOR SPEECH SOURCE SEPARATION BASEDON A CONVOLUTIONAL NEURAL NETWORK”, which was filed as a PCT application and has publication number WO/2020/232180, which is incorporated herein by reference in its entirety.

As also shown in FIG. 1 , time and/or frequency metadata D is provided to the source-cue-based separation module 20. The time and/or frequency metadata D indicates at least one of the time resolution and frequency resolution of the separation module 10 based on spatial cues when it operates. For example, the time and frequency resolution of the separation module 10 based on spatial cues is the bandwidth of a block step in the time domain and a block band in the frequency domain. As another example, the time and frequency resolution of the separation module 10 based on spatial cues is the frame step in the time domain and the bandwidth of a tile in the frequency domain. That is, the time and/or frequency metadata D indicates at least one of the following: (i) the block step in the time domain and/or the bandwidth of a block band (e.g., a quasi-octave band) in the frequency domain; or (ii) the frame step in the time domain and/or the bandwidth of a tile in the frequency domain. The time and/or frequency metadata D can be obtained from an external source (e.g., specified by a user or accessed from a database), or the time and/or frequency metadata D can be provided by the separation module 10 based on spatial cues to the source-cue-based separation 20.

The source cue-based separation module 20 processes the intermediate audio signal B based on the time and/or frequency metadata D. In some embodiments, the spatial cue-based separation module 10 operates with a much lower (i.e., much coarser) time and/or frequency resolution than the resolution of the source cue-based separation module 20. For example, the spatial cue-based separation module 10 operates with a quasi-octave block band with a bandwidth of at least 400 Hz, and the mixing parameters are updated approximately every 100 ms (blocking) or 20 ms (interpolation). However, the source cue-based separation module 20 can operate on individual tiles (e.g., individual STFT tiles) with a time resolution of a few milliseconds (e.g., 20 ms) and a frequency resolution of approximately 10 Hz.

By providing the time and/or frequency metadata D to the source cue-based separation module 20, the module can then be configured to (i) use its default or typical time and/or frequency resolution, (ii) use the same time and/or frequency resolution of the spatial cue-based separation module 10, or (iii) use a different time and/or frequency resolution than (i) or (ii). As an example of alternative (iii), the source cue-based separation module 20 can be instructed to use a lower/coarser time and/or frequency resolution instead of a finer resolution, even if both the spatial cue-based separation module 10 and the source cue-based separation module 20 normally operate at a finer time and/or frequency resolution.

Using the time and/or frequency metadata D, the source cue based separation module 20 may operate in a mode that is more suitable (in terms of separation performance and mitigation of acoustic artifacts) for combination with the spatial cue based separation module 10, and such mode may differ from its typical operation without the spatial cue based separation module 10. The time and/or frequency metadata D may specify a more appropriate time and/or frequency resolution granularity at which the source cue based separation module 20 should operate.

In some embodiments, the time and/or frequency metadata D indicates that the source cue-based separation module 20 should operate and/or apply smoothing at a time and/or frequency resolution equal to or lower/coarser than the time and/or frequency resolution of the spatial source cue-based separation module 10. For example, the source cue-based separation module 20 operates at a frequency resolution that is the same as the frequency resolution used in the spatial cue-based separation module 10 (e.g., equal to the tile frequency band), and its time resolution is between one and ten times coarser/lower than the time resolution of the spatial cue-based separation module 10 (e.g., between one and ten times the duration of the tile).

Since directly changing the resolution of the source cue-based separation module 20 from its default value may result in an increase in residual noise remaining in the output audio signal C, in some embodiments, the source cue-based separation module 20 can operate with its default time and/or frequency resolution (the default time and/or frequency resolution can be finer than the time and/or frequency resolution of the spatial cue-based separation module 10). In such an embodiment, the time and/or frequency metadata D indicates smoothing in time and/or frequency that will be applied directly to the output audio signal C or to the predicted source gain mask G. For example, smoothing can be configured to establish a frequency resolution that is the same as the frequency resolution used in the spatial cue-based separation module 10 and a time resolution that is between one and ten times coarser/lower than the time resolution of the spatial cue-based separation module 10.

In FIG. 2 , it is shown that the intermediate audio signal B is optionally mixed with the input audio signal A in the intermediate mixing module 30 a to generate a mixed intermediate audio signal B′. The mixed intermediate audio signal B′ is then provided to the source cue-based separation module 20, which processes the mixed intermediate audio signal B′. The intermediate mixing module 30 a generates the mixed intermediate audio signal B′ as a weighted linear combination of the input audio signal A and the intermediate audio signal B output by the spatial cue-based separation module 10. For example, the mixing ratio of the intermediate mixing unit 30 a mixes the intermediate audio signal B with the input audio signal A at a mixing ratio that enhances the intermediate audio signal B by at least 15 dB compared to the input audio signal A. In some embodiments, the input audio signal A is not mixed with the intermediate audio signal B, which can be achieved by completely omitting the intermediate mixing unit 30 a or setting a mixing ratio for mixing the intermediate audio signal B with the input audio signal A that enhances the intermediate audio signal B by ∞ dB compared to the input audio signal A.

By using mixing to reintroduce the input audio signal A into the intermediate audio signal B, some of the unprocessed content of the input audio signal A is reintroduced into the intermediate audio signal B, which may improve the performance of the subsequent source cue-based separation module 20 and/or provide an output audio signal C of higher perceived quality. In general, remixing can mask acoustic artifacts that may have been introduced by the previous separation modules 10, 20. For example, the neural network of the source cue-based separation module 20 can be trained using training data containing a mixture of desired audio content (e.g., speech) and noise, whereby the neural network has learned to suppress noise and/or amplify desired audio content. However, the spatial cue-based separation module 10 may introduce acoustic artifacts that are not present in the training data, which may result in a degradation in the performance of the source cue-based separation module 20. By remixing the input audio signal A, these artifacts are masked, thereby making the mixed intermediate audio signal B' more similar to the training data used to train the neural network of the source cue-based separation module 20. Therefore, by remixing the input audio signal A with the intermediate audio signal B, these and other problems are avoided. On the other hand, the remixing still makes the intermediate audio signal B enhanced compared with the input audio signal A, so the source cue-based separation module 20 still presents the spatially separated (mixed) intermediate audio signal B.

Similarly, an output mixing module 30b is optionally provided for mixing the output audio signal C from the source cue-based separation module 20 with at least one of the input audio signal A and the intermediate audio signal B to generate a mixed output audio signal C'. The mixed output audio signal C' is generated as a weighted linear combination of the output audio signal C and at least one of the intermediate audio signal B and the input audio signal A. For example, the output mixing module 30b mixes the output audio signal C at a mixing ratio that enhances the output audio signal C by 20 dB compared to the intermediate audio signal B and/or the input audio signal A, respectively.

Mixing the input audio signal A and/or the intermediate audio signal B into the output audio signal C may contribute to improving the perceived quality of the final mixed output audio signal C'. In some cases, the processing performed by the separation modules 10, 20 may introduce acoustic artifacts. By remixing the input audio signal A and/or the intermediate audio signal B with the output audio signal C, this problem is overcome, as these artifacts are at least partially masked in the mixed output audio signal C'. In addition, remixing the input audio signal A and/or the intermediate audio signal B with the output audio signal C means that, even if any of the source separation modules 10, 20 suppresses part of the desired audio content, the content will still be present in a limited amount in the mixed output audio signal C'.

With further reference to the flowchart in FIG3 , the audio processing method for source separation will now be described in more detail. At step S1, an input audio signal A is obtained and provided to a separation module 10 based on spatial cues, which processes the input audio signal A to obtain an intermediate audio signal B at step S2. The intermediate audio signal B is optionally provided to an intermediate mixing module 30a, which mixes the intermediate audio signal B with the input audio signal A at step S3 to obtain a mixed intermediate audio signal B'. At step S4, time/frequency metadata D indicating the time and/or frequency resolution used by the separation module 10 based on spatial cues is provided to the separation module 20 based on source cues, and at step S5, the time/frequency metadata D is used by the source separation module 20 for processing the mixed intermediate audio signal B' to form an output audio signal C. The output audio signal C is optionally provided to a mixing module 30b, which mixes the output audio signal C with at least one of the input audio signal A and the intermediate audio signal B at step S6 to obtain a mixed output audio signal C'.

In the above, the audio processing method for separation has been described. It should be understood that the method can be carried out with or without any mixing steps performed by the mixing modules 30a, 30b. In other words, the mixing steps S3, S6 are completely optional and independent of each other, which means that no mixing steps, one of the mixing steps, or both mixing steps can be used while the remaining steps remain unchanged. Similarly, the use of time/frequency metadata D during step S4 is optional, and embodiments of processing based on time/frequency metadata D and embodiments of processing not based on time/frequency metadata are envisioned.

It is further contemplated that processing the input audio signal A using the separation module 10 based on spatial cues may further include transforming the input audio signal A to a domain in which the separation module 10 based on spatial cues operates. For example, the input audio signal A is originally in the time domain, and thus the input audio signal A is transformed to the STFT domain or the QMF domain before being taken into the separation module 10 based on spatial cues. Additionally or alternatively, the intermediate audio signal B is inversely transformed before being provided to the subsequent intermediate mixing unit 30a or the separation module 20 based on source cues. Similarly, processing the intermediate audio signal B using the separation module based on source cues may include transforming and inversely transforming the intermediate audio signal B and the output audio signal C.

In some embodiments, the source cue-based separation module 20 includes a source cue-based gain mask extractor 21 and a gain mask applicator 22, as shown in FIG4 . As described above, the source cue-based separation module 20 includes a neural network trained to generate a noise-reduced output audio signal C. This can be achieved by a neural network trained to predict a source gain mask G, which is implemented as a source cue-based gain mask extractor 21. The source cue-based gain mask extractor 21 outputs the source gain mask G to the gain mask applicator 22, wherein the gain mask applicator 22 applies the source gain mask G to the intermediate audio signal B (mixed intermediate audio signal B') to form the noise-reduced output audio signal C. Applying the source gain mask G may include multiplying the source gain mask G with a corresponding time-frequency domain representation of the intermediate audio signal B (mixed intermediate audio signal B').

The gain mask G is a predicted set of gains, where each tile of the audio signal has a gain. For example, a neural network can be trained to predict a fine-grained gain mask, where each STFT bin of each frame has a gain. The predicted gain suppresses noise present in the audio signal while leaving the target audio content (e.g., speech and/or music). As an example, the intermediate audio signal B (mixed intermediate audio signal B') is divided into a plurality of consecutive frames, where each frame is further divided into N tiles covering the corresponding frequency band, where N≥2. Then, the neural network of the gain mask extractor 21 based on the source cue is trained to predict N gains for each frame (one gain for each tile), thereby suppressing noise in the intermediate audio signal B (mixed intermediate audio signal B'). If the target audio content (e.g., speech and/or music) is concentrated in a first tile (N=i), the first tile i can be associated with a high gain, while a different second tile containing mainly noise is associated with a different lower gain that attenuates the second tile.

The gain mask applicator 22 may further be configured to take into account the time and/or frequency metadata D when applying the source gain mask G. For example, if the source cue based separation module 20 should operate with a time and/or frequency resolution different from a default or typical time and/or frequency resolution, the gain mask applicator 22 may be configured to smooth the source gain mask G before applying it to the intermediate audio signal B (mixed intermediate audio signal B') and/or to smooth the resulting audio signal after applying the source gain mask G. Smoothing will reduce the time and/or frequency resolution (make it coarser) in order to achieve, for example, a resolution that matches (or is lower than) the resolution of the spatial cue based separation module 10. That is, compared to the spatial cue based separation module 10, the source cue based gain mask extractor 21 can operate with fine/high resolution, however, the gain mask applicator 22 smoothes the source cue based gain mask G before application, so that the total time and/or frequency resolution of the source cue based separation module 20 is lower/coarser or equal to the total time and/or frequency resolution of the spatial cue based separation module 10.

In an exemplary embodiment, the spatial cue-based separation module 10 operates in block bands of five to ten octave width bands, while the source cue-based gain mask extractor 21 operates on individual tiles. The source gain mask applicator 22 can then smooth the fine-grained gain mask G predicted by the source cue-based gain mask extractor 21 to match the frequency resolution of the spatial cue-based separation module 10. Different techniques can be used to apply smoothing, such as convolution using a smoothing window (in one dimension) or a kernel (in two dimensions). As described above, the frequency resolution of the spatial cue-based separation module 10 can be equal to the bandwidth of the block bands, which is much lower resolution than the bandwidth of the individual tiles.

In some embodiments, smoothing is performed using convolutional smoothing windows that move only in the time dimension and span the same frequency band as the blocking frequency band of the spatial cue based separation module 10. The duration of the smoothing window is between one and ten times the step length used in the spatial cue based separation module 10. The smoothing window may be a Hamming window.

In some embodiments, the spatial cue based separation module 10 may be implemented in a similar manner as a spatial cue based gain mask extractor that determines or predicts a spatial gain mask, wherein the spatial gain mask is provided to a spatial gain mask applier that applies the spatial gain mask to the input audio signal A to form the intermediate audio signal B. That is, modifying at least two channels of the input audio signal A may include determining and applying the spatial gain mask.

In some embodiments, both the spatial cue-based separation module 10 and the source cue-based separation module 20 utilize a gain mask predicted by each module. The spatial cue-based separation module 10 provides the intermediate audio signal B to the source separation module 20. In some such embodiments, the gain mask predicted by each module 10, 20 is provided to a gain mask combiner and applicator, which combines the two gain masks to form an aggregate gain mask, which is then applied to the input audio signal A to form the output audio signal C. Optionally, the gain mask combiner and applicator also performs smoothing of the resulting combined gain mask.

The gain masks predicted by each module 10, 20 do not have to have the same time and/or frequency resolution, and different techniques such as interpolation, data replication or pooling can be used to match the resolution of different gain masks. There are also many ways to combine gain masks. For example, combining the gain masks of each module 10, 20 may include one of the following combinations: multiplication, selecting the minimum value, selecting the maximum value, selecting the median value, selecting the mean value, or any linear combination of the gain masks.

It is also contemplated that the two modules 10 and 20 may be run in parallel, wherein each module predicts a gain mask based on the input audio signal A and provides the respective gain mask to a gain mask combiner and applicator, which combines the two gain masks to form an aggregate gain mask, which is then applied to the input audio signal A to form an output audio signal C. In such an embodiment, there is no intermediate audio signal B. Optionally, the output audio signal C is mixed with the input audio signal A to form a mixed output audio signal C'.

5 depicts a block diagram of an audio processing system, wherein the source separation audio processing system 1 is used together with a classifier 50 and a gating unit 60 to form a gated output audio signal _CG . The classifier 50 operates on at least one of an input audio signal A, an intermediate audio signal B (mixed intermediate audio signal B'), and an output audio signal C (mixed output audio signal C'), and determines a probability metric indicating the likelihood that the obtained audio signal includes target audio content. The probability metric may be a value, wherein a smaller value indicates a lower likelihood, and a larger value indicates a higher likelihood. For example, the probability metric is a value between 0 and 1, wherein a value closer to 0 indicates a lower likelihood that the audio signal includes the target audio content, and a value closer to 1 indicates a higher likelihood that the audio signal includes the target audio content. The target audio content may be, for example, speech or music. In some embodiments, the classifier 50 includes a neural network trained to predict a probability metric indicating the likelihood that the audio signal includes the target audio content given a sample of the input audio signal, the intermediate audio signal, and/or the output audio signal. For example, the neural network is a residual neural network (ResNet) trained to predict a probability metric given a time-frequency representation of at least one of an input audio signal A, an intermediate audio signal B (mixed intermediate audio signal B'), and an output audio signal C (mixed output audio signal C'). The time-frequency representation includes a plurality of consecutive frames divided into a plurality of tiles. As another example, the classifier 50 includes a feature extractor that extracts one or more features based on the time-frequency representation, thereby providing the at least one feature to a multi-layer perceptron (MLP) neural network or a simplified ResNet neural network trained to predict a probability metric. The feature extraction process can be manually specified, or it is assumed that the feature extraction is performed by a trained feature extraction neural network.

Since the intermediate audio signal B (mixed intermediate audio signal B') and the output audio signal C (mixed output audio signal C') are processed versions of the input audio signal A from which the target audio content has been separated, any of these audio signals may be provided as input to the classifier 50 to improve the accuracy of the likelihood prediction and/or enable the use of a simpler classifier 50. For example, it may be easier for the classifier 50 to accurately determine the probability metric when the audio signals have been separated using spatial cues (and optionally also using source cues) than when the input audio signal A has not been subjected to any separation processing by the separation modules 10, 20.

On the other hand, each separation module 10, 20 may introduce a delay due to processing the audio signal using a predetermined number of look-ahead samples and look-back samples. For this reason, although providing the output audio signal C (mixed output audio signal C') to the classifier 50 can use a simpler classifier 50 (e.g., a less complex neural network with fewer layers and learnable parameters) and/or improve classification accuracy, this will also introduce greater signal processing delays.

The probability metric is provided to the gating unit 60, which controls the gain of the output audio signal C (mixed output audio signal C') based on the likelihood to form a gated output audio signal _CG . For example, if the probability metric determined by the classifier exceeds a predetermined threshold, the gating unit 60 applies a high gain, otherwise the gating unit applies a low gain. In some embodiments, the high gain is unity gain (0 dB), while the low gain is actually muting the audio signal (e.g., -25 dB, -100 dB, or -∞ dB). In this way, the output audio signals C, C' become gated output audio signals _CG that isolate the target audio content. For example, the gated output audio signal _CG includes only speech and is actually muted at time instances when there is no speech.

In some embodiments, the gating unit 60 is configured to smooth the applied gain by implementing a limited transition time from low gain to high gain or from high gain to low gain. With a limited transition time, the switching of the gating unit 60 may become less noticeable and less intrusive. For example, it takes about 180ms to transition from low gain (e.g., -25dB) to high gain (e.g., 0dB), and about 800ms to transition from high gain to low gain, wherein, when there is no target audio content, the output audio signal C is further suppressed by being completely muted (-100dB or -∞dB) after the conversion from high gain to low gain is completed. Alternatively, in order to speed up the startup speed, the transition time from low gain to high gain can be set to be shorter than about 180ms, such as about 1ms.

In this way, the gated output audio signal _CG will enhance the target audio content (e.g., speech) in the following ways: first, the target audio content is separated using spatial cues and source cues so that the target audio content is clearer and easier to understand when it appears in the audio signal; second, the output audio signal C is muted when the target audio content does not appear in the input audio signal A.

Unless otherwise specifically stated, it will be apparent from the following discussion that it should be understood that throughout the disclosed discussion, terms such as "processing," "computing," "calculating," "determining," "analyzing," etc. are utilized to refer to the actions and/or processes of computer hardware or computing systems or similar electronic computing devices to manipulate and/or transform data represented as physical (e.g., electronic) quantities into other data similarly represented as physical quantities.

It should be understood that in the above description of the exemplary embodiments of the present invention, various features are sometimes grouped together in a single embodiment, figure or its description to simplify the present disclosure and help understand one or more of the various creative aspects. However, the method of the present disclosure should not be interpreted as reflecting the intention that the claimed invention requires more features than the features explicitly stated in each claim. On the contrary, as reflected in the following claims, the creative aspects are less than all the features of the single aforementioned disclosed embodiment. Therefore, the claims following the specific embodiments are hereby expressly incorporated into the specific embodiments, wherein each claim is independently a separate embodiment of the present invention. In addition, although some embodiments described herein include some features included in other embodiments without including other features included in other embodiments, as will be understood by those skilled in the art, the combination of features of different embodiments is intended to be within the scope of the present invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

In addition, some embodiments herein are described as a combination of methods or method elements that can be implemented by a processor of a computer system or by other devices that perform functions. Therefore, a processor with an instruction to perform such a method or method element forms a device for performing a method or method element. It should be noted that when the method includes multiple elements (e.g., several steps), unless otherwise stated, any order of these elements is not implied. In addition, the elements of the device embodiments described herein are examples of devices that perform functions performed by the elements in order to perform embodiments of the present invention. In the description provided herein, many specific details are set forth. However, it should be understood that embodiments of the present invention can be practiced without these specific details. In other examples, well-known methods, structures, and techniques are not shown in detail to avoid blurring the understanding of this specification.

Those skilled in the art will recognize that the present invention is by no means limited to the above-described embodiments. On the contrary, various modifications and variations within the scope of the appended claims are possible. For example, the mixing modules 30a, 30b in FIG. 2 may not be used, or only one or two of the mixing modules may be used, regardless of whether each separation module operates using a gain mask and/or whether a gain mask combiner and an applicator are present.

Claims (36)

1. A method of audio processing for source separation, the method comprising:

obtaining an input audio signal (a) comprising at least two channels;

Processing the input audio signal (A) with a spatial cue based separation module (10) to obtain an intermediate audio signal (B),

The spatial cue based separation module (10) is configured to determine mixing parameters of the at least two channels of the input audio signal (a) and to modify the at least two channels based on the mixing parameters to obtain the intermediate audio signal (B); and

Processing the intermediate audio signal (B) with a source prompt-based separation module (20) to generate an output audio signal (C),

The source prompt-based separation module (20) is configured to implement a neural network trained to predict a noise reduction output audio signal (C) given samples of the intermediate audio signal (B).

2. The method of claim 1, wherein the input audio signal (a) is divided into a plurality of consecutive frames, and wherein the mixing parameter is indicative of at least one of:

A panning distribution of the at least two channels over a plurality of frames in at least one frequency band, and

The at least two channels have an inter-channel phase difference distribution in at least one frequency band over a plurality of frames.

3. The method of claim 2, wherein the mixing parameters are determined for a plurality of frequency bands.

4. The method according to any of the preceding claims, wherein the spatial cue based separation module (10) operates at a first temporal and/or frequency resolution, the method further comprising:

Providing, by the spatial cue based separation module (10), metadata (D) to the source cue based separation module (20), the metadata (D) being indicative of a temporal and/or frequency resolution of the spatial cue based separation module (10); and

-Generating, by the source hint based separation module (20), the output audio signal (C) based on the intermediate audio signal (B) and the metadata (D).

5. The method of claim 4, wherein the intermediate audio signal (B) is divided into a plurality of consecutive frames and each frame is divided into a plurality of frequency bands, and wherein generating the output audio signal (C) comprises:

Predicting, by the neural network, a source gain mask, the source gain mask indicating a gain for each frequency band applied to each frame of the intermediate audio signal (B); and

The source gain mask is smoothed based on the metadata (D).

6. The method of claim 5, wherein smoothing the source gain mask comprises:

smoothing is performed over time within a frequency band equal to the frequency band indicated by the frequency resolution of the spatial cue based separation module (10).

7. The method of claim 6, wherein the spatial cue based separation module (10) determines the blending parameter by averaging the detected blending parameter over a set of frames, and

Wherein the smoothing over time is performed using a smoothing window having a receptive field in a time dimension equal to or greater than a total duration of the set of frames.

8. The method of claim 6 or claim 7, wherein the smoothing over time is performed using a hamming window.

9. The method according to any of the preceding claims, wherein the spatial cue based separation module (10) determines that the temporal resolution of the mixing parameters is lower than the temporal resolution of the source cue based separation module (20), preferably at least two times lower, more preferably at least four times lower, most preferably at least six times lower.

10. The method according to any of the preceding claims, wherein the spatial cue based separation module (10) determines that the frequency resolution of the mixing parameter is lower than the frequency resolution of the source cue based separation module (20), preferably at least two times lower, more preferably at least five times lower, most preferably at least ten times lower.

11. The method of any of the preceding claims, further comprising:

mixing the intermediate audio signal (B) with the input audio signal (a) to generate a mixed intermediate audio signal (B'); and

-Providing the mixed intermediate audio signal (B') to the source prompt-based separation module (20).

12. The method of any of the preceding claims, further comprising:

Mixing the output audio signal (C) with the input audio signal (a) to generate a mixed output audio signal (C').

13. The method of any of the preceding claims, further comprising:

-mixing the output audio signal (C) with the intermediate audio signal (B) to generate a mixed output audio signal (C').

14. The method according to any of the preceding claims, wherein the input audio signal (a) is divided into a plurality of consecutive frames, and each frame is divided into a plurality of frequency bands,

Wherein the spatial cue based separation module (10) is further configured to determine a spatial gain mask based on the mixing parameters, the spatial gain mask indicating gains for each frequency band applied to each frame of the input audio signal (a) and modifying the at least two channels by applying the spatial gain mask.

15. The method of claim 14, wherein the intermediate audio signal (B) is divided into a plurality of consecutive frames and each frame is divided into a plurality of frequency bands, and wherein generating the output audio signal (C) comprises:

Predicting, by the neural network, a source gain mask, the source gain mask indicating a gain for each frequency band applied to each frame of the intermediate audio signal (B);

combining the source gain mask with the spatial gain mask to form an aggregate gain mask; and

The aggregate gain mask is applied to the input audio signal (a).

16. The method of any of the preceding claims, further comprising:

providing at least one of the input audio signal (a), the intermediate audio signal (B) and the output audio signal (C) to a classifier (50);

determining, with the classifier (50), a probability metric indicating a likelihood that at least one of the input audio signal (a), the intermediate audio signal (B) and the output audio signal (C) comprises a target audio source; and

-Controlling a gain of the output audio signal (C) based on the probability measure.

17. The method of any of the preceding claims, wherein the source hint based separation module is configured to remove at least one of stationary noise, non-stationary noise, background audio content, and reverberation.

18. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to perform the method according to any one of claims 1 to 17.

19. Computer can the storage medium is read and the data is read, the computer readable storage medium stores a computer program according to claim 18.

20. An audio processing system for source separation, the system comprising:

A spatial cue based separation module (10) configured to obtain an input audio signal (a) comprising at least two channels and to process the input audio signal (a) to obtain an intermediate audio signal (B), the spatial cue based separation module (10) being configured to determine mixing parameters of the at least two channels of the input audio signal (a) and to modify the at least two channels based on the mixing parameters to obtain the intermediate audio signal (B), and

A source-hint based separation module (20) configured to process the intermediate audio signal (B) to generate an output audio signal (C) by implementing a neural network trained to predict a noise reduction output audio signal given samples of the intermediate audio signal (B).

21. The audio processing system of claim 20, wherein the input audio signal (a) is divided into a plurality of consecutive frames, and wherein the mixing parameter is indicative of at least one of:

A panning distribution of the at least two channels over a plurality of frames in at least one frequency band, and

The at least two channels have an inter-channel phase difference distribution in at least one frequency band over a plurality of frames.

22. The audio processing system of claim 21, the mixing parameters are determined for a plurality of frequency bands.

23. The audio processing system according to any of claims 20 to 22, wherein the spatial cue based separation module (10) is configured to operate at a first time and/or frequency resolution and provide metadata (D) to the source cue based separation module (20), the metadata (D) being indicative of the time and/or frequency resolution of the spatial cue based separation module (10), and wherein,

The source hint based separation module (20) is further configured to generate the output audio signal (C) based on the intermediate audio signal (B) and the metadata (D).

24. The audio processing system of claim 23, wherein the intermediate audio signal (B) is divided into a plurality of consecutive frames and each frame is divided into a plurality of frequency bands, and wherein the source hint-based separation module (20) is configured to:

Predicting a source gain mask using the neural network, the source gain mask indicating a gain for each frequency band applied to each frame of the intermediate audio signal (B),

Smoothing the source gain mask based on the metadata (D),

And applying the source gain mask to the intermediate audio signal (B).

25. The audio processing system of claim 24, wherein the source-hint based separation module (20) is configured to smooth the source gain mask over time within a frequency band equal to a frequency band indicated by a frequency resolution of the spatial-hint based separation module (10).

26. The audio processing system of claim 25, wherein the spatial cue based separation module (10) is configured to determine the mixing parameters by averaging the detected mixing parameters over a set of frames and to smooth with a smoothing window having a receptive field in a time dimension equal to or greater than a total duration of the set of frames.

27. The audio processing system of claim 26, wherein the smoothing window is a hamming window.

28. The audio processing system of any of claims 20 to 27, wherein the spatial cue based separation module (10) is configured to determine that the temporal resolution of the mixing parameters is lower than the temporal resolution of the source cue based separation module (20), preferably at least two times lower, more preferably at least four times lower, most preferably at least six times lower.

29. The audio processing system of any of claims 20 to 28, wherein the spatial cue based separation module (10) is configured to determine that the frequency resolution of the mixing parameter is lower than the time resolution of the source cue based separation module (20), preferably at least two times lower, more preferably at least five times lower, most preferably at least ten times lower.

30. The audio processing system of any of claims-0 to 29, further comprising an intermediate mixing module (30 a) configured to mix the intermediate audio signal (a) with the input audio signal to generate a mixed intermediate audio signal (B '), wherein the source prompt-based separation module (20) is configured to process the mixed intermediate audio signal (B').

31. The audio processing system of any of claims-0 to 30, further comprising an output mixing module (30 b) configured to mix the output audio signal (C) with the input audio signal () and/or the intermediate audio signal to generate a mixed output audio signal.

32. The audio processing system of any of claims 20 to 31, further comprising an output mixing module (30B) configured to mix the output audio signal (C) with the input audio signal (a) and/or the intermediate audio signal (B) to generate a mixed output audio signal (C').

33. The audio processing system of any of claims 20 to 32, wherein the input audio signal (a) is divided into a plurality of consecutive frames and each frame is divided into a plurality of frequency bands, and wherein the spatial cue based separation module (10) is further configured to determine a spatial gain mask based on the mixing parameters and apply the spatial gain mask to the input audio signal (a) to form the intermediate audio signal (B), wherein the spatial gain mask indicates a gain for each frequency band applied to each frame of the input audio signal (a) and modifies the at least two channels by applying the spatial gain mask.

34. The audio processing system of claim 33, wherein the intermediate audio signal (B) is divided into a plurality of consecutive frames and each frame is divided into a plurality of frequency bands, and wherein the source hint-based separation module (20) is configured to: predicting a source gain mask with the neural network, the source gain mask indicating a gain for each frequency band applied to each frame of the intermediate audio signal (B); combining the source gain mask with the spatial gain mask to form an aggregate gain mask; and applying the aggregate gain mask to the input audio signal (a).

35. The audio processing system of any of claims 20 to 34, further comprising a classifier (50) configured to: obtaining at least one of the input audio signal (a), the intermediate audio signal (B) and the output audio signal (C); determining a probability measure indicating a likelihood that at least one of the input audio signal (a), the intermediate audio signal (B) and the output audio signal (C) comprises a target audio source; and controlling a gain of the output audio signal (C) based on the probability metric.

36. The audio processing system of any of claims 20 to 35, wherein the source-hint based separation module is configured to suppress at least one of stationary noise, non-stationary noise, background audio content, and reverberation.