CN116997960A - Multi-band ducking in the field of audio signal technology - Google Patents

Abstract

A method for multiband evasion of an audio signal is provided. In some embodiments, the method involves receiving an input audio signal at a decoder, wherein the input audio signal is a downmix audio signal. In some embodiments, the method involves dividing the input audio signal into a first set of frequency bands. In some embodiments, the method involves determining a set of evasion gains, the evasion gains corresponding to frequency bands in the first set of frequency bands. In some embodiments, the method involves generating a wideband decorrelated audio signal, wherein the evasion gains of the set of evasion gains are applied to at least one of: 1) A second set of frequency bands prior to generating the at least one wideband decorrelated audio signal; or 2) a third set of frequency bands separating the at least one wideband decorrelated audio signal.

Description

Multi-band ducking in the field of audio signal technology

Cross-references to related applications

This application claims priority from U.S. Provisional Application 63/268,991, filed on March 08, 2022, and U.S. Provisional Application 63/171,219, filed on April 06, 2021, all of which are incorporated herein by reference in their entirety. .

The present disclosure relates to systems, methods, and media for multi-band ducking of audio signals.

Background technique

For example, ducking of audio signals can be performed to attenuate various types of signals, such as transients. However, traditionally performed ducking of audio signals can lead to various artifacts such as ringing artifacts, undesirable artifacts when rendering spatial scenes, etc.

Symbols and terminology

Throughout this disclosure, including in the claims, the terms "speaker", "loudspeaker" and "audio reproduction transducer" are used synonymously to refer to any sound-generating transducer or group of transducers. energy device. A typical set of headphones includes two speakers. The loudspeaker may be implemented to include multiple transducers (eg woofer and tweeter) which may be driven by a single common loudspeaker feed or multiple loudspeaker feeds. In some examples, the speaker feed(s) may undergo different processing in different circuit branches coupled to different transducers.

Throughout this disclosure, including in the claims, the expression "performing an operation on" a signal or data (such as filtering, scaling, transforming or applying a gain to the signal or data) is used broadly to mean performing an operation directly on the signal or data or Perform operations on a processed version of a signal or data. For example, an operation may be performed on a version of the signal that has been initially filtered or preprocessed before the operation is performed on it.

Throughout this disclosure, including in the claims, the expression "system" is used broadly to refer to a device, system or subsystem. For example, a subsystem implementing a decoder may be referred to as a decoder system, and a system including such subsystems (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M input, while the other X-M inputs are received from external sources) may also be called a decoder system.

Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to mean a processor that is programmable or otherwise configurable (such as with software or firmware) to process data (which may include audio or video or other image data). ) the system or device on which the operation is performed. Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and/or otherwise configured to perform pipelined processing of audio or other sound data, Programming general-purpose processors or computers, and programmable microprocessor chips or chipsets.

Contents of the invention

At least some aspects of the present disclosure can be implemented via methods. Some methods may involve receiving an input audio signal at a decoder, wherein the input audio signal is a downmix audio signal. Some methods may involve dividing the input audio signal into a first set of frequency bands. Some methods may involve determining a set of ducking gains, the ducking gains of the set corresponding to frequency bands of the first set of frequency bands. Some methods may involve generating at least one wideband decorrelated audio signal, wherein the at least one wideband decorrelated audio signal may be used to upmix the downmix audio signal, and wherein a ducking gain of the set of ducking gains is applied at least one of: 1) a second set of frequency bands prior to generating the at least one wideband decorrelated audio signal; or 2) a third set of frequency bands separating the at least one wideband decorrelated audio signal frequency band.

In some examples, the set of ducking gains includes a set of input ducking gains, and the method further includes applying an input ducking gain of the set of input ducking gains to the first before generating the at least one wideband decorrelated audio signal. Two sets of frequency bands. In some examples, ducking signals associated with frequency bands in the second set of frequency bands are aggregated to generate a wideband ducking signal that is provided to a decorrelator configured to generate the at least one wideband decorrelated audio signal.

In some examples, the first set of frequency bands and the second set of frequency bands are two instances of the same set of frequency bands.

In some examples, the set of ducking gains includes a set of output ducking gains, and some methods may further involve: applying an output ducking gain of the set of output ducking gains to the third set of frequency bands to generate at least one set of ducking decorrelation audio signals, each duck-decorrelated audio signal in the at least one set of duck-decorrelated audio signals corresponding to a frequency band in the third set of frequency bands; and causing the duck-decorrelated audio signals in the at least one set of duck-decorrelated audio signals to The correlated audio signals are aggregated to generate at least one wideband ducking decorrelated audio signal that can be used to upmix the downmix audio signal.

In some examples, determining the set of ducking gains includes: determining one or more initial ducking gains; and modifying at least one of the one or more initial ducking gains to generate the set of ducking gains, wherein the one or more initial ducking gains At least one of the initial dodge gains is modified by performing an update and/or releasing control.

In some examples, for a frequency band in the first set of frequency bands, a corresponding ducking gain is determined based on a ratio including outputs of two envelope trackers, the two envelope trackers corresponding to a slow envelope tracker and fast envelope trackers. In some examples, the slow envelope tracker includes an absolute value calculation block and a first low pass filter, and wherein the fast envelope tracker includes the absolute value calculation block and a second low pass filter, The first low pass filter and the second low pass filter have different time constants. In some examples, some methods may further involve applying a high pass filter to at least one of the first set of frequency bands, wherein an output of the high pass filter is provided to one of the two envelope trackers. at least one. In some examples, the high-pass filter is applied to two or more of the first set of frequency bands, and wherein the high-pass filter applied to a first of the two or more frequency bands The filter has a different cutoff frequency than the high pass filter applied to a second of the two or more frequency bands. In some examples, the time constant of the first low pass filter of the slow envelope tracker is longer than the time constant of the second low pass filter of the fast envelope tracker, and wherein the ratio includes the The ratio of the output of the slow envelope tracker to the output of the fast envelope tracker. In some examples, the time constant of the first low pass filter of the slow envelope tracker is longer than the time constant of the second low pass filter of the fast envelope tracker, and wherein the ratio includes the The ratio of the output of the fast envelope tracker to the output of the slow envelope tracker. In some examples, the ratio includes a constant specific to a frequency band in the first set of frequency bands, the constant being selected to control at least one of: 1) application in the second set of frequency bands The amount of ducking gain applied to each frequency band; or 2) the amount of ducking gain applied to each frequency band in the third group of frequency bands.

In some examples, dividing the input audio signal into the first set of frequency bands includes providing the input audio signal to a filter bank. In some examples, the filter bank is implemented as an infinite impulse response (IIR) filter bank or a finite impulse response (FIR) filter bank.

In some examples, the first set of frequency bands, the second set of frequency bands, and/or the third set of frequency bands include three frequency bands.

In some examples, the first set of frequency bands and the third set of frequency bands are the same.

In some examples, the at least one wideband decorrelated signal includes two or more wideband decorrelated signals.

In some examples, some methods further involve upmixing the downmix audio signal using the at least one wideband decorrelated signal and metadata received at the decoder to generate a reconstructed audio signal. In some examples, some methods further involve rendering the reconstructed audio signal to generate a rendered audio signal. In some examples, some methods further involve presenting the rendered audio signal using one or more of: a loudspeaker or headphones.

Some or all operations, functions, and/or methods described herein may be performed by one or more devices according to instructions (eg, software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read only memory (ROM) devices, and the like. Accordingly, some innovative aspects of the subject matter described in this disclosure may be implemented via one or more non-transitory media having software stored thereon.

At least some aspects of the present disclosure can be implemented via devices. For example, one or more devices may be capable of performing, at least in part, the methods disclosed herein. In some embodiments, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates, or Transistor logic, discrete hardware components, or combinations thereof.

The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and description below. Other features, aspects and advantages will become apparent from the description, drawings and claims. Note that the relative dimensions of the following figures may not be to scale.

Description of the drawings

Figure 1 is a block diagram of an example multi-channel codec in accordance with some embodiments.

Figure 2 is a block diagram of a portion of a decoder including an example of a decorrelator and a ducker for implementing multi-band ducking, in accordance with some embodiments.

Figure 3 is a block diagram of an example of a ducker that may be used to implement multi-band ducking in accordance with some embodiments.

4 is a graph of the frequency response of an example filter bank that may be used to implement multi-band ducking in accordance with some embodiments.

Figure 5 is a flowchart of an example process that may be performed by a decoder for performing multi-band ducking in accordance with some embodiments.

Figure 6 illustrates an example use case for an immersive speech and audio services (IVAS) system in accordance with some embodiments.

7 shows a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure.

Similar reference numbers and names indicate similar elements throughout the various drawings.

Detailed ways

Decorrelators are commonly used in decoder devices that utilize multi-channel audio codecs such as Stereo Audio Codec, Parametric Stereo, AC-4, etc. In particular, at the encoder, N channel inputs can be downmixed into M channels, where N>M. The M downmix channels and auxiliary information are encoded into a bit stream and transmitted to the decoder. The decoder can then decode the M channels and the side information and use the side information to upmix or reconstruct the N channels. In particular, the decorrelator of the decoder device may generate N-M decorrelated signals. The decoder can then utilize the M downmix channels, the N-M decorrelated signals and the side information to obtain an approximate reconstruction of the original N channels. In other words, by generating an approximate reconstruction of the original N channels, the decoder can reconstruct the original spatial audio scene.

For example, in the case of stereo audio, where N corresponds to two channels, and where M corresponds to one downmix channel, a decorrelator can generate a decorrelated signal. The decoder can then use the decorrelated signal, the downmix channel and the side information to reconstruct a representation of the original two audio signals. As another example, in the case where N is four channels (such as channels W, a decorrelated signal. The decoder can utilize these three decorrelated signals to reconstruct the original spatial audio scene.

In general, a decorrelator can be used to transform an input audio signal into one or more uncorrelated output signals, which can achieve a controlled sense of width, space, or diffusion while other perceptual properties remain unchanged. Accordingly, decorrelators may be useful for reconstructing audio signals with spatial components. Figure 1 illustrates a specific example of a codec that utilizes a decorrelator in the decoder to reconstruct an encoded audio signal.

Figure 1 is a block diagram of an Immersive Speech and Audio Services (IVAS) codec 150 for encoding and decoding IVAS bitstreams, according to an embodiment. IVAS codec 150 includes an encoder and a remote decoder. The IVAS encoder includes a spatial analysis and downmix unit 152, a quantization and entropy encoding unit 153, a core encoding unit 156 and a mode/bitrate control unit 157. The IVAS decoder includes a quantization and entropy decoding unit 154, a core decoding unit 158, a spatial synthesis/rendering unit 159 and a decorrelator unit 161.

The spatial analysis and downmix unit 152 receives an N channel input audio signal 151 representing an audio scene. Input audio signals 151 include, but are not limited to: mono signals, stereo signals, binaural signals, spatial audio signals (eg, multi-channel spatial audio objects), FOA, higher order high fidelity stereo (HOA), and any other audio data. The spatial analysis and downmix unit 152 downmixes the N channel input audio signals 151 into a specified number of downmix channels (M). In this example, M<=N. The spatial analysis and downmix unit 152 also generates auxiliary information (e.g., spatial metadata) that can be used by the remote IVAS decoder for the M downmix channels, spatial metadata, and decorrelated signals generated from the decoder. N channel input audio signals are synthesized 151. In some embodiments, the spatial analysis and downmix unit 152 implements a Complex Advanced Coupling (CACPL) for analyzing/downmixing stereo/FOA audio signals and/or a spatial reconstructor (CACPL) for analyzing/downmixing FOA audio signals ( SPAR). In other embodiments, spatial analysis and downmix unit 152 implements other formats.

The M channels are encoded by one or more instances of the core codec included in core encoding unit 156. Auxiliary information (eg, spatial metadata (MD)) is quantized and encoded by the quantization and entropy encoding unit 153 . The encoded bits are then packed together into IVAS bitstream(s) and sent to the IVAS decoder. In embodiments, the underlying core codec may be any suitable mono, stereo or multi-channel codec that may be used to generate the encoded bitstream.

In some embodiments, the core codec is the EVS codec. The EVS encoding unit 156 complies with 3GPP TS26.445 and provides a wide range of functions, such as enhanced quality and coding efficiency for narrowband (EVS-NB) and wideband (EVS-WB) speech services, enhancement using super-wideband (EVS-SWB) speech Quality, enhanced quality for mixed content and music in conversational applications, robustness to packet loss and delay jitter, and backward compatibility with the AMR-WB codec.

At the decoder, the M channels are decoded by corresponding one or more instances of the core codec included in core decoding unit 158 , and the side information is decoded by quantization and entropy decoding unit 154 . The primary downmix channels, such as the W channel of the FOA signal format, are fed to the decorrelator unit 161, which generates N-M decorrelated channels. These M downmix channels, N-M decorrelated channels and ancillary information are fed to the spatial synthesis/rendering unit 1 59 which uses these inputs to synthesize or regenerate the original N channel inputs Audio signal, the original N channel input audio signal may be presented by the audio device 160 . In an embodiment, the M channels are decoded by a mono codec other than EVS. In other embodiments, the M channels are decoded by a combination of one or more multi-channel core coding units and one or more mono core coding units.

An example implementation of encoding of an FOA input audio signal with primary channel downmixing is given below. For a 1-channel passive downmix configuration, only the W channel, P (p ₁ , p ₂ , p ₃ ) parameters and P _d (d ₁ , d ₂ , d ₃ ) parameters are encoded and sent to the decoder . P corresponds to a prediction coefficient indicating how many side channels (Y, X, and Z) can be predicted from the W channel. The P _d parameter indicates the remaining energy in the Y, X and Z channels after removing the predicted components.

In the passive downmix coding scheme, the side channels Y, X and Z are predicted at the decoder from the transmitted downmix W channel; three prediction parameters P are used for prediction. The missing energy in the side channels is filled by adding a scaled version of the decorrelated downmix D(W) using the decorrelation parameter _Pd . For passive downmix, the reconstruction of the FOA input can be determined by:

U _pas =pW+P _d D(W),

where, p = [1 p ₁ p ₂ p ₃ ] ^T and P _d = [0 d ₁ d ₂ d ₃ ] ^T , and D(W) describes the channel with W provided as input to the decorrelator block Decorrelator output. _Upas is the reconstructed FOA output at the decoder. Note that this scheme achieves perfect reconstruction in terms of the input covariance matrix, assuming that the decorrelator is perfect and no quantization of prediction and decorrelator parameters is performed.

In an example encoder implementation, the prediction coefficients for the Y channel can be determined by:

In the equation given above, R _YW is the covariance of the W and Y channels, and R _WW is the variance of the W channel.

Similarly, predictions for the other side channels (p ₂ for the X channel and p ₃ for the Z channel) can be determined.

The remaining side channels can be determined by:

Y′＝Yp ₁ *W

X′＝Xp ₂ *W

Z′＝Zp ₃ *W

In an example implementation, the decorrelation parameter d ₁ for the Y channel is determined by:

Here, R _Y'Y' is the variance of the remaining channel Y', and R _WW is the variance of the W channel. Similarly, the decorrelation parameters for the other remaining side channels (d ₂ for the X' channel and d ₃ for the Z' channel) can be determined.

One potential problem with decorrelators is that transients in the input audio signal can become blurred over time in the output channel. For example, transients such as knocks or other types of transients may be blurred over time across multiple channels generated by the decorrelator, which may add undesirable reverberation in frames with transients . Another problem is that the decorrelated signal generated by the decorrelator can still have considerable energy even when the input signal is suddenly shifted. It should be noted that as used herein, the term "offset" is generally used to refer to the end or cessation of a major element or component of an audio signal. In other words, where the input signal to the decorrelator includes sudden stops or shifts, the decorrelated signal may contain considerable energy that obscures the shifts. This in turn may produce artifacts in the reconstructed signal generated based on the decorrelated signals.

Ducking can be used to duck or attenuate transients before providing the input audio signal to the decorrelator. For example, ducking the transient before generating the decorrelated signal(s) can prevent the transient from being blurred over time in the generated decorrelated signal(s). Similarly, in the presence of offsets in the input audio signal, ducking may be performed on the output of the decorrelator to attenuate the decorrelated signal(s). However, ducking has traditionally been performed on a broadband basis. In other words, all frequency bands of the audio signal are ducked with the same gain. This can create artifacts and degrade audio quality. For example, in the presence of transients, applying ducking gain to the input audio signal in a broadband manner can duck high-frequency content, which may be desirable due to transients. However, applying ducking gain in a broadband manner may also duck lower frequency content such as bass, which may reduce overall audio quality and/or create distortion in the overall audio content. To solve the problem of applying ducking equally in all frequency bands, some conventional techniques can apply ducking in the frequency band domain when using multi-band decorrelators. However, due to the computational complexity of implementing a decorrelator, implementing multiple instances of the decorrelator (each operating on a different frequency band) may significantly increase the computational complexity, leading to overuse of computing resources, etc.

This article describes techniques for applying ducking gain on a per-band basis. In particular, ducking gain is determined and applied on a band-by-band basis. For example, this may allow ducking gain to be applied differently to low frequency content compared to high frequency content. In some embodiments, the ducking gain may be an input ducking gain applied to the input audio signal before the input audio signal is provided to the decorrelator. The input ducking gain can be used to duck the transient signal before the transient is provided to the decorrelator, thereby preventing the transient from "entering" the decorrelator. In some embodiments, the ducking gain may additionally or alternatively be an output ducking gain applied to the decorrelated signal generated by the decorrelator. The output ducking gain may be used to duck persistent signals in the generated decorrelated signal(s) that correspond to offsets in the input signal, thereby recovering offsets in the input signal in the decorrelated signal(s). It should be noted that while ducking gain can be determined and applied on a per-band basis, decorrelation can be performed on a broadband basis. Because decorrelators can be computationally expensive to implement, performing decorrelation on a wideband basis while applying ducking on a per-band basis can improve computational efficiency by implementing only one instance of the decorrelator while taking audio into account Ducking gain is applied in a content-frequency-selective manner to improve overall audio quality.

Figure 2 illustrates a block diagram of an example system that may be used by a decoder to implement multi-band ducking in accordance with some embodiments. It should be noted that the various blocks of the system shown in Figure 2 may be implemented using one or more control systems of the device, such as the control system shown in Figure 7 and described below in connection with that figure. As shown in Figure 2, an input audio signal or frame of an input audio signal is provided to a first filter bank 202 (depicted in Figure 2 as "Filter Bank A"). In some implementations, the first filter bank 202 may divide the input audio signal into any suitable number of frequency bands, such as two frequency bands, three frequency bands, eight frequency bands, ten frequency bands, 16 frequency bands, etc. In the example shown in FIG. 2 , the first filter bank 202 divides the input audio signal into three frequency bands, which may respectively correspond to low frequency, intermediate frequency and high frequency. An example of frequency ranges for an embodiment involving three frequency bands is shown in Figure 4 and described below.

Each frequency band can be provided to an instance of the ducker block. For example, because the first filter bank 202 separates the input audio signal into three frequency bands, three ducker blocks are illustrated in FIG. 2 and are described as ducker 204a, ducker 204b and ducker 204b. Device 204c. Each ducker block can generate input ducking gain and/or output ducking gain. In some implementations, the ducking gain may be determined based on the ratio of the outputs of two envelope trackers, each having a different time constant. Envelope trackers can be implemented using absolute value (rectifier) blocks and low-pass filters. For example, the input ducking gain may be determined based on the ratio of the output of a low-pass filter with a long time constant to the output of a low-pass filter with a short time constant. In other words, the input ducking gain can be determined based on the ratio of slow envelope tracking to fast envelope tracking. Instead, the output ducking gain may be determined based on the ratio of the output of the low-pass filter with a short time constant to the output of the low-pass filter with a long time constant. In other words, the output ducking gain can be determined based on the ratio of fast envelope tracking to slow envelope tracking. Examples of long time constants include 60 milliseconds, 70 milliseconds, 80 milliseconds, 90 milliseconds, etc. Examples of short time constants include 3 ms, 4 ms, 5 ms, 10 ms, etc. It should be noted that each ducker block instance may take as input the output of the first filter bank 202 corresponding to a particular frequency band and generate a ducking gain appropriate for that particular frequency band. A more detailed example of a ducker block is shown in Figure 3 and described below in conjunction with this figure.

The input audio signal may be provided to delay block 206. A delayed version of the input audio signal may be provided to a second filter bank 208 (depicted as "Filter Bank B" in Figure 2). Delay block 206 may be used to delay the input audio signal by an amount that separates the input audio signal from the ducking determined by ducker blocks 204a, 204b, and 204c after the input audio signal is divided into the plurality of frequency bands by second filter bank 208. Gain time alignment of the timing of the input audio signal. It should be noted that delay block 206 may be implemented in conjunction with a wideband ducker implementation (eg, where filter banks 202 and 208 are not implemented). Example delays that may be applied by delay block 206 include 1.5 milliseconds, 2 milliseconds, 2.5 milliseconds, etc. In some embodiments, the delay applied by delay block 206 may be a delay to be used in a wideband ducker system, which is then based at least in part on the delay applied by first filter bank 202 and/or The modification is made by the delay applied by the second filter bank 208 .

The input ducking gains determined by ducker blocks 204a, 204b, and 204c may be applied to the frequency band of the delayed version of the input audio signal on a per-band basis. For example, a first input ducking gain corresponding to the first frequency band may be determined based on the first frequency band of the first filter bank 202 . Continuing with the example, the first input ducking gain may then be applied to the corresponding instance of the first frequency band of the second filter bank 208 . As a more specific example, the input ducking gain may be applied by multiplying the input ducking gain by the corresponding band signal gain application blocks 209a, 209b, and 209c. It should be noted that in some implementations, first filter bank 202 and second filter bank 208 may be the same filter bank (e.g., filter banks with the same number of frequency bands, the same frequency response, the same filter type, etc.) of different instances. Rather, in some embodiments, first filter bank 202 and second filter bank 208 may differ in any one or more characteristics, such as number of frequency bands, cutoff frequencies for various frequency bands, type of filters used, etc. . It should be noted that the application of input ducking gain can be used to duck or attenuate transients in the input audio signal. As will be described in more detail below in conjunction with Figures 3 and 5, the input ducking gain applied to higher frequency bands can be higher than the input ducking gain applied to lower frequency bands, such that high frequency signals are ducked more strongly than low frequency signals. or decay.

A wideband ducking signal can be generated after applying the input ducking gain. For example, after the input ducking gain has been applied on a per-band basis in the band set of the second filter bank 208, the frequency bands may be combined, such as by summation, to generate a wideband signal. As a more specific example, the frequency bands may be summed or aggregated via aggregation block 209d. The wideband signal may then be provided to decorrelator 210. Decorrelator 210 may generate one or more decorrelated signals. In some embodiments, the number of decorrelated signals generated by decorrelator 210 may depend on the number of signals to be parametrically reconstructed by the decoder, as described above in connection with FIG. 1 . For example, where the reconstructed audio signal is a stereo signal, the decorrelator 210 may generate a decorrelated signal that may be used to upmix the downmixed signal to generate the original two signals. As another example, in the case where the reconstructed audio signal includes four channels and there is one downmix signal, the decorrelator 210 may generate three decorrelated signals, each of which may be used for reconstruction by the encoder. Parametrically encoded three signals.

The one or more decorrelated signals may be provided to a third filter bank 212 (depicted as "Filter Bank C" in Figure 2). The third filter bank 212 may divide each of the one or more decorrelated signals into multiple frequency bands, eg, two frequency bands, three frequency bands, eight frequency bands, 16 frequency bands, etc. In some embodiments, third filter bank 212 may be another instance of first filter bank 202 and/or second filter bank 208 . Conversely, in some embodiments, the third filter bank 212 may be compatible with the first filter bank 202 and/or the second filter bank in any characteristics (such as cutoff frequencies for various frequency bands, filter types used, etc.) Group 208 is different. It should be noted that in some implementations, third filter bank 212 may be replicated for each decorrelated signal generated by decorrelator 210 .

The output ducking gains may be delayed by corresponding delay blocks 214a, 214b, and 214c, each output ducking gain being determined based on the frequency band of the first filter bank 202 and generated by ducker blocks 204a, 204b, and 204c. Delay blocks 214a, 214b, and 214c may be used to delay the output ducking gain so that the output ducking gain may be time aligned with the frequency band of the third filter bank 212. In some embodiments, the delay applied by each of delay blocks 214a, 214b, and 214c may be based at least in part on the delay generated by third filter bank 212. The delayed output ducking gain may then be applied to each of the one or more decorrelated signals on a per-band basis. For example, the output ducking gain may be applied by multiplying the output ducking gain by the corresponding frequency band signal via gain application blocks 213a, 213b, and 213c. It should be noted that the output ducking gain can be used to duck or attenuate excursions in the input audio signal. An example of a shift is a sudden stop of the input audio signal.

After applying the output ducking gain on a per-band basis, a broadband version of each decorrelated signal can be generated. For example, the ducking frequency bands may be combined (eg, summed) to generate a ducked broadband decorrelated signal. As a more specific example, the ducking bands may be summed or aggregated via aggregation block 213d. The decoder can use the ducked wideband decorrelation signal to upmix the downmix signal and generate a reconstructed audio signal.

It should be noted that the first filter bank 202, the second filter bank 208, and/or the third filter bank 212 may be implemented in any suitable manner. For example, the filter bank may be implemented as an infinite impulse response (IIR) filter bank. As another example, the filter bank may be implemented as a finite impulse response (FIR) filter bank. Various filter bank implementations may have advantages and disadvantages. For example, some filter bank implementations may have longer delays than other filter bank implementations. As mentioned above, various delay blocks can be implemented to account for the delay applied by the filter bank, for example, to ensure that the signal is time aligned before applying ducking gain. It should be noted that the filter bank can achieve and/or approach an "exact reconstruction" where the sum of the unmodified frequency bands is essentially the same as the filter bank's input signal or a delayed version thereof.

As described above, in some embodiments, the input ducking gain and the output ducking gain may be determined by providing a specific frequency band of the input audio signal to two envelope trackers and determining a ratio of the outputs of the two trackers. In some embodiments, each envelope tracker may be associated with a corresponding low-pass filter. In some embodiments, the two low pass filters may have two different time constants, one time constant being much longer than the other. Examples of shorter time constants are 3 ms, 4 ms, 5 ms, 10 ms, etc. Examples of longer time constants are 60 ms, 70 ms, 80 ms, 100 ms, etc. Each low-pass filter can effectively perform envelope tracking for a specific frequency band of the input audio signal provided as the input to the low-pass filter, where one low-pass filter performs slow envelope tracking while the other low-pass filter Perform fast envelope tracking. Each low-pass filter can be characterized by a numerator filter coefficient b and a denominator filter coefficient a, where b=[1-c] and a=[1,-c]. Here, c can be determined based on the time constant of the filter, where c=exp(-1/( _tc *sampling_rate)), where _tc represents the time constant of the filter in seconds. Given a cutoff frequency of -3dB, a low-pass filter with a time constant of 5 milliseconds can have a cutoff frequency of about 32.2Hz, and a filter with a time constant of 80 milliseconds can have a cutoff frequency of about 2.2Hz. In some embodiments, the input ducking gain for a particular frequency band may be determined based on the ratio of the output of a low-pass filter with a longer time constant to the output of a low-pass filter with a shorter time constant. In other words, the input ducking gain may correspond to the ratio of slow envelope tracking to fast envelope tracking. Instead, the output ducking gain for a particular frequency band may be determined based on the ratio of the output of a low-pass filter with a shorter time constant to the output of a low-pass filter with a longer time constant. In other words, the output ducking gain may correspond to the ratio of fast envelope tracking to slow envelope tracking.

In some embodiments, a high-pass filter may be applied before providing specific frequency bands of the input audio signal to the two envelope trackers. A high-pass filter can be used to flatten the frequency spectrum and/or avoid deviations in the presence of low-frequency rumble. In some embodiments, the cutoff frequency of the high-pass filter may depend on the frequency band of the input audio signal to which the high-pass filter is applied. For example, a lower cutoff frequency may be used for lower frequency bands relative to higher frequency bands. In one example, a cutoff frequency of 3kHz may be used for the upper frequency band and a cutoff frequency of 1kHz may be used for the lower frequency band. Examples of cutoff frequencies for high-pass filters include 1kHz, 2kHz, 3kHz, 5kHz, etc. In some implementations, the high-pass filter may be omitted for some frequency bands.

Figure 3 shows a schematic diagram of an example ducker example in accordance with some embodiments. It should be noted that the various blocks of the example ducker instance shown in Figure 3 may be implemented by one or more control systems of the device, such as the control system shown in Figure 7 and described below in connection with that figure. The ducker can take as input a specific frequency band of the input audio signal and can generate as output an input ducking gain and/or an output ducking gain appropriate for that frequency band. As mentioned above, the ducker can take as input the frequency band of the input audio signal. For example, the frequency band may be that of the first filter bank 202, as shown in Figure 2 and described above in connection with that figure. Input ducking gain and/or output ducking gain may be applied to that particular frequency band. It should be noted that the example ducker instance shown in Figure 3 may be essentially repeated for each frequency band of the first filter bank.

As illustrated, the frequency bands of the input audio signal may optionally be high-pass filtered using high-pass filter 302. In some implementations, the cutoff frequency of high-pass filter 302 may depend, at least in part, on the frequency band of the input audio signal processed by the ducker instance. For example, a higher cutoff frequency can be used for a higher frequency band and vice versa. Examples of cutoff frequencies for high-pass filters include 1kHz, 2kHz, 3kHz, 5kHz, etc.

The frequency bands of the input audio signal (or, if used, a high-pass filtered version of the frequency bands of the input audio signal) may be provided to the fast envelope tracker 305 and the slow envelope tracker 307 . Each envelope tracker may include an absolute value calculation block 304 configured to generate the absolute value of the signal. It should be noted that in some embodiments, a relatively small value, depicted as "ε" in Figure 3, may be added to the absolute value of the signal. As discussed below, this prevents divide-by-zero errors when determining input ducking gain and/or output ducking gain. As illustrated in Figure 3, the fast envelope tracker 305 includes a first low pass filter 306 and the slow envelope tracker 307 includes a second low pass filter 308. As illustrated in FIG. 3 , the first low pass filter 306 may have a shorter time constant compared to the second low pass filter 308 . Examples of shorter time constants include 3 ms, 4 ms, 5 ms, 10 ms, etc. Examples of longer time constants are 60 ms, 70 ms, 80 ms, 90 ms, 100 ms, etc.

The output of the first low-pass filter 306 (depicted as "f" in Figure 3, indicating fast envelope tracking) and the output of the second low-pass filter 308 (depicted as "s" in Figure 3, indicating Slow envelope tracking) is provided to the output ducking gain determination block 310. Similarly, the output of first low pass filter 306 and the output of second low pass filter 308 are provided to input ducking gain determination block 312 . The output ducking gain may be determined based at least in part on the ratio of fast envelope tracking to slow envelope tracking. In particular, as illustrated in Figure 3, if the output of the first low-pass filter 306 is denoted as f (i.e., for fast envelope tracking), and the output of the second low-pass filter 308 is denoted as s (i.e., for fast envelope tracking) , for slow envelope tracking), then the initial set of output ducking gains can be determined by the following formula:

The initial set of input ducking gains can be determined by:

It should be noted that the const representing the multiplicative constant may be the same for the output ducking gain and the input ducking gain, or may be different for the output ducking gain compared to the input ducking gain. Example values for const include 1, 1.05, 1.1, 1.15, 1.2, etc. Additionally, it should be noted that the constants c ₁ and c ₂ can be different for each frequency band. In particular, the values of c ₁ and c ₂ may represent the amount of input ducking and output ducking to be applied for the frequency band, respectively. In other words, c ₁ and c ₂ can be used as band-dependent corrections to the ducking gain. For example, it may be advantageous to have no ducking in the lowest frequency band. Correspondingly, c ₁ and c ₂ can be 1 for the lowest frequency band. As another example, a relatively larger amount of ducking may be applied to the highest frequency bands. Accordingly, _c1 and _c2 can be 0 for the highest frequency bands, such that the input ducking gain and output ducking gain are determined as a ratio based on the output of the envelope tracker without band-dependent corrections to the ratios . It should be noted that for a specific frequency band, c ₁ and c ₂ may be the same as each other, or may be different from each other. In some embodiments, c ₁ and c ₂ can be any suitable value in the range of 0 to 1, inclusive.

The initial set of output duck gains may be provided to the output duck gain update block 313 to determine the output duck gains 314 . Similarly, an initial set of input duck gains may be provided to input duck gain update block 315 to determine input duck gains 316 . In some embodiments, the output duck gain update block 313 and the input duck gain update block 315 may be configured to perform smoothing and/or duck release control to avoid undesirable sudden changes in the applied duck gain. For example, where the input audio signal includes a transient, there may be a sudden change in the input ducking gain, such as determined by the input ducking gain determination block 312, in order to duck the transient. Continuing with the example, input duck gain update block 315 may then modify the initial set of input duck gains determined after the transient such that the modified input duck gain transitions smoothly after the sudden change in input duck gain due to the transient.

Example implementations of blocks 313 and 315 are described below. Given an initial value of the input ducking gain denoted in_duck_gains_init and an initial value of the output ducking gain denoted out_duck_gains_init, the actual input ducking gain (denoted in_duck_gains_act) and the actual output ducking gain (denoted out_duck_gains_act) can be determined by the following pseudocode:

For each sample s:

in_duck_state=(in_duck_state-1)*in_duck_c+1

If(in_duck_gains_init(s)＜in_duck_state)

induck_state=in_duck_gains_init(s)

in_duck_gains_act(s)=in_duck_state

In the above, in_duck_state represents the gain state from one time frame to another. The initial value of In_duck_state can be set between 0 and 1. In the pseudocode example given above, in_duck_c represents the release constant, which controls how quickly the ducking gain is released. In other words, in_duck_c can be used to control the transition of ducking gain from low to high values. In the technique described above, the input duck gain is released based on a release constant and then updated in response to a new duck gain sample being less than the release value.

A similar approach can be used to output ducking gain, as shown in the pseudocode sample given below.

For each sample s:

out_duck_state=(out_duck_state-1)*out_duck_c+1

If(out_duck_gains_init(s)＜out_duck_state)

out_duck_state=out_duck_gains_init(s)

out_duck_gains_act(s)=out_duck_state

In the pseudocode example given above, out_duck_state represents the gain state from one time frame to another. The initial value of out_duck_state can be set between 0 and 1. In the example given above, out_duck_c is the release constant that controls how quickly the ducking gain is released. In other words, out_duck_c can be used to control the transition of ducking gain from low to high values. In the example given above, the output duck gain can be released based on a release constant, and can then be updated in response to a new duck gain sample being less than the release value.

As mentioned above, the decoder may implement various filter banks to split the audio signal into a plurality of band-limited signals based on the frequency band of the filter bank. For example, a filter bank may divide an input audio signal into multiple frequency bands to determine input ducking gain and/or output ducking gain on a per-band basis. As another example, a filter bank may divide the input audio signal into multiple frequency bands to apply the input ducking gain on a per-band basis. As yet another example, a filter bank may split a broadband decorrelated signal into multiple bands to which an input ducking gain may have been applied before applying an output ducking gain on a per-band basis. As discussed above, in instances where multiple filter banks are implemented, the filter banks may be multiple instances of the same filter bank, or may differ in one or more characteristics (e.g., number of frequency bands, frequency response, filters used, type, etc.). Filter banks can divide the signal into any suitable number of frequency bands, such as two, three, five, eight, 16, etc. In one example, a filter bank splits the signal into three frequency bands, corresponding to low, mid, and high frequencies. Example types of filters that may be used include infinite impulse response (IIR) filters, finite impulse response (FIR) filters, and the like. Each type of filter may be associated with a different complexity, which may allow implementations to trade off filtering characteristics versus computational complexity.

Figure 4 shows the frequency response of a frequency band of an example filter bank that may be used in accordance with some embodiments. The example shown in Figure 4 utilizes three zero-delay first-order IIR filters. These three filters correspond to low frequency band 402, mid frequency band 404 and high frequency band 406. In the example shown in Figure 4, the cutoff frequency of low frequency band 402 is 200 Hz and the cutoff frequency of high frequency band 406 is 2 kHz. The mid-frequency band 404 is derived from the low-frequency band 402 and the high-frequency band 406, for example, to obtain a perfect reconstruction of the signal passing through the filter bank. Note that in the case where the ducking gain is determined to be 1 or close to 1, perfect reconstruction of the signal allows the signal to effectively remain unmodified. It should be noted that the example shown in Figure 4 is illustrative only, and the filter banks implemented by the decoder may differ from each other in terms of number of bands, cutoff frequency for each band, type of filters used, complexity, latency, etc. The filter bank illustrated in Figure 4 is different.

Figure 5 is a flowchart of an example process 500 for applying ducking gain on a per-band basis, in accordance with some embodiments. In some implementations, the blocks of process 500 may be implemented using a control system of the decoder device. Such a control system is shown in Figure 7 and is described below in connection with this figure. In some embodiments, the blocks of process 500 may be performed in an order other than that shown in FIG. 5 . In some implementations, two or more blocks of process 500 may be performed substantially in parallel. In some implementations, one or more blocks of process 500 may be omitted.

Process 500 may begin at 502 by receiving an input audio signal or input audio signal frame. In some implementations, the input audio signal may be received by a receiver device of the decoder, such as an antenna. In some embodiments, the input audio signal may be received at the decoder from an encoder device that transmitted the input audio signal. It should be noted that in some embodiments, the received input audio signal may be a downmixed audio signal that has been downmixed by the encoder before being transmitted to the decoder. In some such implementations, the decoder may additionally receive metadata or auxiliary information, which may be used to upmix the downmix signal, for example, to generate a reconstructed audio signal, as described above in connection with FIG. 1 .

At 504, process 500 may divide the input audio signal into multiple frequency bands. For example, in some implementations, process 500 may provide the input audio signal to a first filter bank that separates the input audio signal into corresponding frequency bands. Any suitable number of frequency bands may be used, such as two, three, five, eight, 16, etc. In one example, the input audio signal may be divided into three frequency bands corresponding to a low frequency band, a mid frequency band and a high frequency band, similar to the example shown in Figure 4 and described above in connection with this figure.

At 506, process 500 may determine input ducking gains and/or output ducking gains corresponding to the plurality of frequency bands. For example, as shown in Figure 3 and described above in connection with that figure, process 500 may apply two envelope trackers per frequency band, the first envelope tracker corresponding to fast envelope tracking, and the second envelope tracker The network tracker corresponds to slow envelope tracking. As part of envelope tracking, process 500 may apply two low-pass filters to each frequency band after absolute value calculation (e.g., rectification), the first low-pass filter having a relatively short time constant, and the second low-pass filter having a relatively short time constant. Filters have long time constants. The first low-pass filter may generate an output, often referred to herein as f, representing fast envelope tracking, and the second low-pass filter may generate an output, commonly referred to herein as s, representing slow envelope tracking. . As shown in Figure 3 and described above in connection with this figure, the input ducking gain can be determined by:

The output ducking gain can be determined by the following formula:

As shown in the above equation, the input ducking gain and the output ducking gain can be determined based on the ratio of the outputs of the two envelope trackers, where the ratio is based on a constant selected for each frequency band (expressed in the above equation Modified for c ₁ and c ₂ ). For example, the input ducking gain may typically be determined based on the ratio of slow envelope tracking to fast envelope tracking, where the amount each is weighted in the ratio is modified by the constant c ₁ . Similarly, the output ducking gain can generally be determined based on the ratio of fast envelope tracking to slow envelope tracking, where the amount each is weighted in the ratio is modified by the constant _c2 . As mentioned above, the input ducking gain and/or the output ducking gain may then be modified, for example using the input ducking gain update block and/or the output ducking gain update block, as described above in connection with FIG. 3 .

It should be noted that in some embodiments, process 500 may obtain or determine values for c ₁ and c ₂ for a particular frequency band prior to determining the input ducking gain and/or the output ducking gain for the particular frequency band. In some embodiments, the values of c ₁ and c ₂ may be fixed for a specific frequency band. For example, in some embodiments, c ₁ and c ₂ may be fixed to 1 for the lowest frequency band, such that the lowest frequency band is not ducked. Continuing with the example, in some embodiments, c ₁ and c ₂ may be set to 0 for the highest frequency band, such that the input ducking gain is determined based on the ratio of slow envelope tracking to fast envelope tracking without adjustment, and such that Output ducking gain is determined based on the ratio of fast envelope tracking to slow envelope tracking without adjustment.

Additionally, it should be noted that for specific ones of the multiple frequency bands, a high-pass filter may be applied before providing the input signal to the fast envelope tracker and the slow envelope tracker, as shown in Figure 3 and incorporated above described. A high-pass filter can be used to flatten the frequency spectrum and/or avoid deviations in the presence of low-frequency rumble. In some implementations, a high-pass filter may be applied to only a subset of multiple frequency bands. In some embodiments, the cutoff frequency of the high-pass filter may be different for different frequency bands. As described above in conjunction with Figure 3, example cutoff frequencies include 1.5kHz, 2kHz, 2.5kHz, 3kHz, 3.5kHz, 4kHz, etc.

At 508, process 500 may apply input ducking gain to the plurality of frequency bands. As illustrated in FIG. 2 and described above in connection with that figure, in some embodiments, process 500 may be determined by first delaying the input audio signal based at least in part on a delay applied by the first filter bank used in conjunction with block 504 amount and then a second filter bank is applied to the delayed input audio signal to split the delayed input audio signal into multiple frequency bands to apply the input ducking gain. The input ducking gain may then be applied to multiple frequency bands of the delayed input audio signal, for example by multiplying the signal of a particular frequency band by the corresponding input ducking gain or gains for that frequency band. It should be noted that in some embodiments, there may be multiple time-varying input ducking gains for a particular frequency band, such that each sample of a band-limited audio signal in the time domain may be ducked by a corresponding sample of the input ducking gain. In some embodiments, the second filter bank may be a second instance of the first filter bank. In other words, in some embodiments, the filter bank used to determine the ducking gain may have the same characteristics as the filter bank used to generate the multiple frequency bands of the input audio signal to which the input ducking gain is applied. Conversely, in some implementations, the first filter bank may differ from the second filter bank in one or more characteristics (eg, frequency response, number of frequency bands, type of filters used, etc.).

At 510, process 500 may aggregate signals over the plurality of frequency bands to generate a first ducked version of the input audio signal. For example, in some embodiments, process 500 may sum multiple frequency bands. In some implementations, process 500 may generate a time domain version of the aggregated signal to generate a first ducked version of the input audio signal.

At 512, process 500 may generate a decorrelated signal by providing a first ducked version of the input audio signal to a decorrelator. In some implementations, one or more decorrelated signals may be generated. In some embodiments, the number of decorrelated signals generated by the decorrelator may depend on the number of signals to be parametrically reconstructed from metadata or auxiliary information, as illustrated in Figures 1 and 2 and in conjunction with these Figures above. described.

At 514, process 500 may separate the decorrelated signals into multiple frequency bands. In some embodiments, a filter bank may be used to separate each decorrelated signal, as shown in Figures 2 and 4 and described above in connection with these figures. In some embodiments, the filter bank may be the same filter bank used in connection with blocks 504 and/or 508. Conversely, in some embodiments, the filter bank may have one or more different characteristics than the filter bank used in connection with blocks 504 and/or 508 .

At 516 , process 500 may apply output ducking gains, which have been determined at block 506 , to the plurality of frequency bands of the decorrelated signal. In some embodiments, for a specific frequency band, the output ducking gain may be applied to the frequency band by multiplying the corresponding output ducking gain or gains. The output ducking gain may then be applied to multiple frequency bands of the decorrelated signal, for example by multiplying the signal of a particular frequency band by the corresponding output ducking gain or gains for that frequency band. It should be noted that in some embodiments, there may be multiple time-varying output ducking gains for a particular frequency band, such that each sample of a band-limited decorrelated audio signal in the time domain may be ducked by a corresponding sample of the output ducking gain. In some implementations, output ducking gain may be applied to each decorrelated signal separately.

At 518, process 500 may generate a broadband version of the ducked decorrelated signal. For example, for a particular decorrelated signal, process 500 may sum signals across multiple frequency bands after applying output ducking gain. Continuing with the example, process 500 may generate a time domain representation of the summed or aggregated signals to generate a ducked decorrelated signal.

It should be noted that although process 500 describes applying input ducking gain and output ducking gain, in some embodiments, either input ducking gain or output ducking gain may be applied without the other. For example, input ducking gain can be applied to duck transients in a specific frequency band before providing the signal to the decorrelator. Continuing with the example, for example, in the absence of offset, the output ducking gain may not be applied to one or more decorrelated signals. As another example, an output ducking gain may be applied to duck offset portions of one or more decorrelated signals generated by a decorrelator without having to pre-apply an input ducking gain to the signal provided to the decorrelator. As a more specific example, where the input audio signal does not include a specific type of signal, such as a transient signal, the input ducking gain may not be applied.

Additionally, it should be noted that the decoder can utilize the decorrelated signal of each duck to upmix the downmixed input audio signal. For example, as shown in Figure 1 and described above in connection with this figure, the ducked decorrelated signal may be provided to a spatial reconstruction codec that obtains the encoder(s) provided by the encoder. Decorrelated signals and ancillary information or metadata for dodging and upmixing of downmixed input audio signals. In some implementations, the upmixed audio signal may then be rendered, for example, to create a spatial perception when the rendered audio signal is presented. In some implementations, the decoder device may cause the rendered audio signal to be presented, for example, through one or more loudspeakers, headphones, etc.

Figure 6 illustrates an example use case of the IVAS system 600 according to an embodiment. In some embodiments, various devices communicate through a call server 602 configured to call devices from the Public Switched Telephone Network (PSTN) or Public Land Mobile Network (PLMN), such as illustrated by PSTN/Other PLMN 604 Receive audio signals. Use cases support legacy devices 606 that render and capture audio in mono only, including but not limited to: supporting Enhanced Voice Service (EVS), Multi-Rate Wideband (AMR-WB), and Adaptive Multi-Rate Narrowband (AMR-NB) equipment. The use case also supports user equipment (UE) 608 and/or 614 that captures and renders stereo audio signals, or UE 610 that captures a mono signal and binaurally renders it into a multichannel signal. The use case also supports immersive and stereo signals captured and rendered by video conferencing room systems 616 and/or 618, respectively. The use case also supports stereo capture and immersive rendering of stereo audio signals for a home theater system 620 and mono capture and immersive rendering of audio signals for a virtual reality (VR) rig 622 and immersive content ingestion 624 by a computer 612 .

7 is a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure. As with the other figures provided herein, the types and numbers of elements shown in Figure 7 are provided as examples only. Other embodiments may include more, fewer, and/or different types and numbers of elements. According to some examples, apparatus 700 may be configured to perform at least some of the methods disclosed herein. In some implementations, apparatus 700 may be or may include a television, one or more components of an audio system, a mobile device (such as a cell phone), a laptop computer, a tablet device, a smart speaker, or another type of device.

According to some alternative embodiments, apparatus 700 may be or may include a server. In some such examples, apparatus 700 may be or include an encoder. Thus, in some cases, the apparatus 700 may be a device configured for use within an audio environment, such as a home audio environment, whereas in other situations, the apparatus 700 may be a device configured for use in the "cloud" , for example, server.

In this example, device 700 includes interface system 705 and control system 710 . In some implementations, interface system 705 may be configured to communicate with one or more other devices of the audio environment. In some examples, the audio environment may be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. In some implementations, the interface system 705 may be configured to exchange control information and associated data with audio devices of the audio environment. In some examples, control information and associated data may relate to one or more software applications being executed by device 700.

In some implementations, interface system 705 may be configured to receive a content stream or to provide a content stream. The content stream may include audio data. Audio data may include, but may not be limited to, audio signals. In some cases, the audio data may include spatial data such as channel data and/or spatial metadata. In some examples, the content stream may include video data and audio data corresponding to the video data.

Interface system 705 may include one or more network interfaces and/or one or more external device interfaces (eg, one or more Universal Serial Bus (USB) interfaces). According to some implementations, interface system 705 may include one or more wireless interfaces. Interface system 705 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and/or a gesture sensor system. In some examples, interface system 705 may include one or more interfaces between control system 710 and a memory system (such as optional memory system 715 shown in Figure 7). However, in some cases, control system 710 may include a memory system. In some implementations, interface system 705 may be configured to receive input from one or more microphones in the environment.

For example, control system 710 may include a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gates, or transistors Logic, and/or discrete hardware components.

In some implementations, control system 710 may reside in more than one device. For example, in some implementations, a portion of the control system 710 may reside in a device within one of the environments depicted herein, and another portion of the control system 710 may reside in a device outside of the environment, such as a server, mobile device (e.g., smartphone or tablet computer), etc. In other examples, a portion of the control system 710 may reside in a device within an environment, and another portion of the control system 710 may reside in one or more other devices in the environment. For example, a portion of the control system 710 may reside in a device that implements a cloud-based service, such as a server, and another portion of the control system 710 may reside in another device that implements a cloud-based service, such as another server, memory device, etc.). In some examples, interface system 705 may also reside in more than one device.

In some implementations, control system 710 may be configured to perform, at least in part, the methods disclosed herein. According to some examples, control system 710 may be configured to perform splitting the audio signal into multiple frequency bands, determining input ducking gains and/or output ducking gains based on frequency bands, applying input ducking gains per frequency band, applying a decorrelator to the wideband audio signal , methods such as applying output ducking gain on a per-band basis to the decorrelated audio signal.

Some or all of the methods described herein may be performed by one or more devices according to instructions (eg, software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read only memory (ROM) devices, and the like. One or more non-transitory media may be located, for example, in optional memory system 715 and/or control system 710 shown in FIG. 7 . Accordingly, various innovative aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media having software stored thereon. Software may, for example, include functions for splitting the audio signal into multiple frequency bands, determining input ducking gains and/or output ducking gains based on the frequency bands, applying input ducking gains per frequency band, applying a decorrelator to the wideband audio signal, performing the decorrelation of the audio signal on Apply commands for output ducking gain, etc. on a per-band basis. For example, the software may be executed by one or more components of a control system such as control system 710 of FIG. 7 .

In some examples, device 700 may include optional microphone system 720 shown in FIG. 7 . Optional microphone system 720 may include one or more microphones. In some implementations, one or more microphones may be part of or associated with another device (such as a speaker of a speaker system, a smart audio device, etc.). In some examples, device 700 may not include microphone system 720. However, in some such implementations, device 700 may still be configured to receive microphone data for one or more microphones in the audio environment via interface system 710 . In some such implementations, cloud-based implementations of apparatus 700 may be configured to receive microphone data or noise indicators corresponding at least in part to the microphone data from one or more microphones in the audio environment via interface system 710 .

According to some embodiments, the apparatus 700 may include the optional loudspeaker system 725 shown in FIG. 7 . Optional loudspeaker system 725 may include one or more loudspeakers, which may also be referred to herein as "speakers," or more generally as "audio reproduction transducers." In some examples (eg, cloud-based implementations), device 700 may not include loudspeaker system 725 . In some implementations, device 700 may include headphones. Headphones may be connected or coupled to device 700 via a headphone jack or via a wireless connection (eg, Bluetooth).

Some aspects of the present disclosure include a system or device configured (e.g., programmed) to perform one or more examples of the disclosed methods, and a system or device that stores one or more instructions for performing the disclosed methods or steps thereof. A tangible computer-readable medium (for example, a disk) of the code for the examples. For example, some disclosed systems may be or include a programmable general purpose processor, digital signal processor, or microprocessor programmed with software or firmware and/or otherwise Means are configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system including an input device, a memory, and a processing subsystem programmed (and/or otherwise configured) to operate in response to data asserted thereto. One or more examples of performing the disclosed methods (or steps thereof).

Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) configured (e.g., programmed and otherwise configured) to pair The audio signal performs required processing, including performance of one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed system (or elements thereof) may be implemented as a general purpose processor (eg, a personal computer (PC) or other computer system or microprocessor, which may include input devices and memory), A general purpose processor is programmed with software or firmware and/or otherwise configured to perform any of a variety of operations, including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of the inventive system are implemented as general purpose processors or DSPs configured (eg, programmed) to perform one or more examples of the disclosed methods, and the systems further include other elements. . Other elements may include one or more loudspeakers and/or one or more microphones. A general-purpose processor configured to perform one or more examples of the disclosed methods may be coupled to an input device. Examples of input devices include, for example, a mouse and/or a keyboard. A general-purpose processor may be coupled to memory, display devices, and the like.

Another aspect of the present disclosure is a computer-readable medium (such as a disk or other tangible storage medium) storing one or more instructions executable to perform the disclosed method or steps thereof. code that performs one or more examples of the disclosed method or steps thereof.

While specific embodiments of the disclosure and applications of the disclosure have been described herein, it will be apparent to those of ordinary skill in the art that without departing from the scope of the disclosure as described and claimed herein, Many variations may be made to the embodiments and applications described herein. It is to be understood that, while certain forms of the disclosure have been shown and described, the disclosure is not limited to the specific embodiments described and illustrated, or the specific methods described.

Claims (23)

1. A method for processing audio signals, the method comprising: receiving an input audio signal at the decoder, wherein the input audio signal is a downmix audio signal; dividing the input audio signal into a first set of frequency bands; determining a set of ducking gains, the ducking gains of the set of ducking gains corresponding to frequency bands of the first set of frequency bands; and Generating at least one wideband decorrelated audio signal, wherein the at least one wideband decorrelated audio signal is capable of upmixing the downmix audio signal, and wherein a ducking gain in the set of ducking gains is applied to At least one of: 1) a second set of frequency bands prior to generating the at least one wideband decorrelated audio signal; or 2) a third set of frequency bands separating the at least one wideband decorrelated audio signal. 2. The method of claim 1, wherein the set of ducking gains includes a set of input ducking gains, and the method further comprises inputting the set of input ducking gains prior to generating the at least one broadband decorrelated audio signal. Input ducking gain is applied to the second set of frequency bands. 3. The method of claim 2, wherein ducking signals associated with frequency bands in the second set of frequency bands are aggregated to generate a wideband ducking signal, the wideband ducking signal being provided to a decorrelator, said The decorrelator is configured to generate the at least one wideband decorrelated audio signal. 4. The method of any one of claims 1 to 3, wherein the first set of frequency bands and the second set of frequency bands are two instances of the same set of frequency bands. 5. The method of any one of claims 1 to 4, wherein the set of ducking gains includes a set of output ducking gains, and the method further comprises: Applying an output ducking gain of the set of output ducking gains to the third set of frequency bands to generate at least one set of ducking decorrelated audio signals, each of the at least one set of ducking decorrelated audio signals Corresponding to a frequency band in said third set of frequency bands; and Aggregating ducking decorrelated audio signals in the at least one set of ducking decorrelated audio signals to generate at least one wideband ducking decorrelated audio signal, the at least one wideband ducking decorrelated audio signal being able to be used to downmix the audio signal Do upmix. 6. The method of any one of claims 1 to 5, wherein determining the set of ducking gains includes: Determine one or more initial dodge buffs; and At least one of the one or more initial ducking gains is modified to generate the set of ducking gains, wherein the at least one of the one or more initial ducking gains is modified by performing update and/or release control. 7. The method of any one of claims 1 to 6, wherein, for a frequency band in the first set of frequency bands, a corresponding ducking gain is determined based on a ratio comprising the outputs of two envelope trackers, so The two envelope trackers described above correspond to the slow envelope tracker and the fast envelope tracker. 8. The method of claim 7, wherein the slow envelope tracker includes an absolute value calculation block and a first low pass filter, and wherein the fast envelope tracker includes the absolute value calculation block and a second low-pass filter, the first low-pass filter and the second low-pass filter having different time constants. 9. The method of claim 7, further comprising applying a high pass filter to at least one of the first set of frequency bands, wherein an output of the high pass filter is provided to the two envelope trackers at least one of the devices. 10. The method of claim 9, wherein the high-pass filter is applied to two or more of the first set of frequency bands, and wherein the high-pass filter is applied to the two or more The high-pass filter for a first one of the frequency bands has a different cutoff frequency than the high-pass filter applied to a second one of the two or more frequency bands. 11. The method of any one of claims 7 to 10, wherein the first low-pass filter of the slow envelope tracker has a smaller diameter than the second low-pass filter of the fast envelope tracker. The time constant is a longer time constant, and wherein the ratio includes the ratio of the output of the slow envelope tracker to the output of the fast envelope tracker. 12. The method of any one of claims 7 to 11, wherein the first low-pass filter of the slow envelope tracker has a smaller diameter than the second low-pass filter of the fast envelope tracker. The time constant is a longer time constant, and wherein the ratio includes the ratio of the output of the fast envelope tracker to the slow envelope tracker. 13. The method of any one of claims 7 to 12, wherein the ratio includes a constant specific to a frequency band in the first set of frequency bands, the constant being selected to control at least Either: 1) the amount of ducking gain applied to each of the second set of frequency bands; or 2) the amount of ducking gain applied to each of the third set of frequency bands. 14. The method of any one of claims 1 to 13, wherein dividing the input audio signal into the first set of frequency bands includes providing the input audio signal to a filter bank. 15. The method of claim 14, wherein the filter bank is implemented as an infinite impulse response (IIR) filter bank or a finite impulse response (FIR) filter bank. 16. The method of any one of claims 1 to 15, wherein the first set of frequency bands, the second set of frequency bands and/or the third set of frequency bands comprise three frequency bands. 17. The method of any one of claims 1 to 16, wherein the first set of frequency bands and the third set of frequency bands are the same. 18. The method of any one of claims 1 to 17, wherein the at least one broadband decorrelated signal includes two or more broadband decorrelated signals. 19. The method of any one of claims 1 to 18, further comprising upmixing the downmix audio signal using the at least one wideband decorrelation signal and metadata received at the decoder. mix to generate a reconstructed audio signal. 20. The method of claim 19, further comprising rendering the reconstructed audio signal to generate a rendered audio signal. 21. The method of claim 20, further comprising presenting the rendered audio signal using one or more of: a loudspeaker or headphones. 22. An apparatus configured for carrying out the method of any one of claims 1 to 21. 23. One or more non-transitory media having stored thereon software comprising instructions for controlling one or more devices to perform the method of any one of claims 1 to 21.