HK40124189A - Methods, apparatus and systems for performing perceptually motivated gain control - Google Patents

Description

Methods, apparatus, and systems for performing gain control of sensed excitations.

Cross-references to related applications

This application claims priority to U.S. Provisional Application No. 63/378678, filed October 6, 2022, and U.S. Provisional Application No. 63/503533, filed May 22, 2023, the entire contents of each of which are incorporated herein by reference.

Technical Field

This disclosure relates to systems, methods, and media for adaptive gain control in audio environments.

Background Technology

For example, gain control can be used to attenuate a signal to the range desired by the audio codec. To improve the perceptual quality of audio signals with gain control applied at the encoder and inverse gain control applied at the decoder, gain transfer functions that smoothly transition between different gains applied to consecutive frames have been proposed. If there are drastic gain changes between consecutive frames, this method can lead to audible artifacts. Furthermore, in some cases, the gain changes between determined gains of consecutive frames are too large and/or too abrupt to apply a smooth transfer function. In such cases, hard transfer can be used to ensure the signal remains within the expected range. For example, a single bit can be used to convey the information about using hard transfer between gains of consecutive frames. However, such hard transfer can also result in audible artifacts in the decoded and reproduced audio signal that are worse than those introduced by the original overload condition. Therefore, there is a need to use gain transfer functions to improve the perceptual quality of the encoding/decoding system and reduce the number of bits required for encoding.

Notation and naming conventions

Throughout this disclosure, including in the claims, the terms "loudspeaker," "amplifier," and "audio reproduction transducer" are used synonymously to refer to any transducer or group of transducers that emits sound. A typical set of headphones includes two loudspeakers. A loudspeaker can be implemented to include multiple transducers, such as a woofer and a tweeter, which can be driven by a single common loudspeaker feed or multiple loudspeaker feeds. In some examples, the loudspeaker feeds may undergo different processing in different circuit branches coupled to different transducers.

Throughout this disclosure, including in the claims the expression "to perform an operation on" a signal or data, such as filtering, scaling, transforming, or applying gain to the signal or data, is broadly used to mean performing an operation directly on the signal or data or on a processed version of the signal or data. For example, an operation may be performed on a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on the signal.

Throughout this disclosure, including in the claims, the term "system" is used broadly to refer to a device, system, or subsystem. For example, a subsystem implementing a decoder may be referred to as a decoder system, and a system including such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, wherein the subsystem generates M inputs and receives other X-M inputs from an external source) may also be referred to as a decoder system.

Throughout this disclosure, including in the claims, the term "processor" is used broadly to refer to a system or device (e.g., using software or firmware) that is programmable or otherwise configurable to perform operations on data, which may include audio or video or other image data. Examples of processors include field-programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and/or otherwise configured to perform pipelined processing of audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

Summary of the Invention

In view of this, this disclosure provides methods, apparatus and programs for improving automatic gain control, as well as computer-readable storage media, having the features of the respective independent claims.

According to one aspect of this disclosure, a method for performing gain control on an audio signal is provided. The audio signal may be a Higher Order Ambisonics (HOA) audio signal. In this method, a downmixed audio signal of the audio signal to be encoded can be obtained. Obtaining the audio signal may include receiving the downmixed audio signal. Alternatively, it may include determining the downmixed audio signal from the audio signal to be encoded. Furthermore, it may be determined that an overload condition has occurred for a frame of the downmixed audio signal. The overload condition may be a condition where the frame of the downmixed audio signal exceeds a predefined signal range. The predefined signal range may be a signal range desired by the encoder. The encoder may be a core encoder. In response to determining that an overload condition has occurred, a gain conversion function for the frame can be determined. The gain conversion function may be based at least on a gain conversion step size. The gain conversion function may be applied to the frame to generate a gain-adjusted frame of the downmixed audio signal. The gain-adjusted frame may be a attenuated frame or an amplified frame. The gain-adjusted frame and information indicating the gain conversion function may be provided for encoding by the encoder.

By limiting the gain transition function to the gain transition step size, a smooth and less abrupt transition from continuous gain can be achieved. The gain transition step size may not be sufficient to attenuate all samples of a frame to the signal range required by the core encoder. However, the artifacts caused by small overshoots are less noticeable compared to very abrupt increases or decreases in the gain parameter. Therefore, by allowing certain values to exceed the desired signal range, an improved audio experience can be achieved when decoding, rendering, and playing back the signal.

In some embodiments, the gain-adjusted frame may be encoded together with information indicating the gain conversion function.

In some embodiments, the downmixed audio signal may be a spatially encoded downmixed signal.

In some embodiments, the frame of the downmixed audio signal may be the current frame, and the gain conversion function may also be based on a previous gain conversion function applied to frames preceding the current frame.

In some embodiments, the gain transition function may further rely on a smoothing function based on the gain transition step size.

In some embodiments, the gain conversion function may include an instantaneous portion and a steady-state portion. The instantaneous portion may correspond to a conversion from the gain associated with the previous frame to a gain associated with the previous frame adjusted by a gain conversion step size.

In some embodiments, the gain associated with the previous frame that is adjusted by the gain conversion step size, based on the gain adjustment target of the current frame, can be a decrease or an increase of the gain associated with the previous frame by the gain conversion step size.

In some embodiments, the length of the instantaneous portion may be limited by the delay introduced by the codec used by the encoder and decoder.

Therefore, gain control does introduce an additional delay that is essentially zero.

In some embodiments, the length of the instantaneous portion may be equal to or less than the number of samples used by the encoder for encoding operations.

In some embodiments, the gain conversion function can be defined as

Where DBSTEP is the gain conversion step size, l is the sample index, j is the frame index, p() is the smoothing function, l _end indicates that it defines the rightmost index of p(), and L is the number of samples in a frame.

In some embodiments, the gain transition step size may be a predefined value, or it may be determined from a set of predefined values that increase in size. The predefined value or the set of predefined values may be determined based on perceived quality listening tests or objective quality measurement tests. Perceived quality listening tests may be a multi-stimulus test (MUSHRA) with hidden references and anchors. Perceived quality listening tests may be part of the tuning process for automatic gain control at the encoder and decoder.

In some embodiments, the method may further include determining the amount of overload caused by frames of the downmixed audio signal. Furthermore, the gain conversion step size may be determined based on a set of predefined values increasing in magnitude from the overload amount.

Therefore, the gain conversion step size can be adapted to the required rate of change between consecutive frames.

In some embodiments, applying a gain conversion function to a frame to generate a gain-adjusted frame of the downmixed signal may include samples to which the gain conversion function is applied to the downmixed audio signal. The total number of samples may correspond to the frames of the downmixed audio signal.

In some embodiments, encoding the gain-adjusted frame along with information indicating the gain transfer function may include determining a coding scheme based on the gain transfer function. In some cases, the coding scheme may be determined based on the gain transfer step size. In some cases, the coding scheme may be determined based on whether the overload condition has been eliminated. The coding scheme may be one of Modified Discrete Cosine Transform (MDCT) or Algebraically Excited Linear Prediction (ACELP).

Therefore, the encoding scheme can be optimized for a specific audio signal and the required gain conversion step size.

According to another aspect, a method for performing gain control on an audio signal is provided. In this method, an encoded frame of the audio signal can be received by a decoder. The encoded frame of the audio signal can be decoded to obtain a frame of a submixed audio signal and information indicating the gain control applied by the encoder. The inverse gain transfer function to be applied to the frame of the submixed audio signal can be determined at least in part based on the information indicating the gain control applied by the encoder. The information indicating the gain control applied by the encoder may include a gain transfer step size. The inverse gain transfer function can be applied to the frame of the submixed audio signal.

In some embodiments, the method may further include upmixing the downmixed audio signal to generate an upmixed audio signal. The upmixed audio signal may be suitable for rendering.

In some embodiments, the method may further include rendering the mixed signal to produce rendered audio data.

In some embodiments, the method may further include playing back rendered audio data using one or more loudspeakers or headphones.

In some embodiments, the inverse gain conversion function can be determined by inverting the gain conversion function applied to the encoder.

In some embodiments, the inverse gain conversion function may include an instantaneous part and a steady-state part.

Some or all of the operations, functions, and/or methods described herein can be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transient media. Such non-transient media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, some innovative aspects of the subject matter described in this disclosure can be implemented by one or more non-transient media on which software is stored.

At least some aspects of this disclosure can be implemented via apparatus. For example, one or more devices are capable of performing at least partially the methods disclosed herein. In some implementations, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof.

Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the specification, drawings, and claims. Note that the relative dimensions in the following drawings may not be drawn to scale.

Attached Figure Description

Figure 1 is a schematic block diagram of a system for providing gain control of audio signals in the prior art.

Figures 2A and 2B are schematic block diagrams of a system for implementing adaptive gain control according to some embodiments.

Figures 3A and 3B show examples of gain conversion functions that can be implemented by an encoder and inverse gain conversion functions that can be implemented by a decoder, according to some embodiments.

Figure 4 is a flowchart of an example process that can be executed by an encoder to achieve adaptive gain control according to some embodiments.

Figure 5 is a flowchart of an example process that can be executed by a decoder to achieve adaptive gain control according to some embodiments.

Figure 6 illustrates example use cases of an immersive voice and services (IVAS) system according to some embodiments.

Figure 7 shows a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure.

Figures 8A and 8B illustrate example embodiments of an audio codec that utilizes the perceptual excitation of a downmixed signal for gain control, where the gain transition step size is uniform.

Figures 9A and 9B illustrate example embodiments of an audio codec that utilizes the perceptual excitation of a submixed signal for gain control, where the gain transition step size is non-uniform.

The same reference numerals and names in the various figures denote the same elements.

Detailed Implementation

Some coding techniques for scene-based audio, stereo audio, multichannel audio, and/or object audio rely on encoding multi-component signals after a downmixing operation. Downmixing allows for encoding a reduced number of audio components using waveform coding that preserves the waveform, and parametric coding of the remaining components. At the receiver side, parametric metadata indicating the parametric coding can be used to reconstruct the remaining components. Because only a subset of the components is waveform-coded, and the parametric metadata associated with the parametrically coded components can be encoded efficiently in terms of bit rate, such coding techniques can be relatively bit-rate efficient while still allowing for high-quality audio.

One potential problem is that the downmixed channels determined by the spatial encoder may include signals with levels unsuitable for subsequent processing by the core codec that constructs the audio signal bitstream. For example, in some cases, the downmixed signal may have excessively high levels, causing the core codec to overload even though the original input signal is not overloaded in any of its component signals. This can lead to severe distortion, such as clipping in the reconstructed signal after decoding and rendering. This can result in a considerable loss of quality in the final rendered signal. One possible solution is to attenuate the input signal to avoid overloading the core codec. However, this solution may have the drawback of increased grainy noise because the quantizer used to encode the signal may not be operating optimally.

Figure 1 shows a schematic block diagram of a conventional system 100 for performing gain control on encoded high-order ambient stereo (HOA) signals. The schematic block diagram shown in Figure 1 can be used for encoding and decoding MPEG-H signals. MPEG-H is a set of international standards being developed by the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC). MPEG-H consists of several parts, including Part 3, MPEG-H 3D Audio.

At encoder 102, the input HOA signal is processed at 104. This processing may include, for example, decomposition, in which a lower mixed channel is generated. The lower mixed channel may include a set of signals constrained by [-max, max] for a given frame. Because the core encoder 108 can encode signals in the range of [-1, 1), samples of signals associated with lower mixed channels outside the range of the core encoder 108 may cause overload. To avoid overload, gain control 106 adjusts the gain of the frame so that the associated signals are within the range of the core encoder 108 (e.g., within [-1, 1)). The core encoder 108 can be considered as a codec that generates an encoded bitstream. Side information generated by decomposition/processing block 104 (which may include metadata, etc., associated with the parametrically encoded channels) can be combined with the signal generated as the output of the core encoder 108 and encoded in the bitstream.

Decoder 112 receives the encoded bitstream. Decoder 112 can extract side information, and core decoder 116 can extract the undermixed signal. Then, inverse gain control block 120 can reverse the gain applied by the encoder. For example, inverse gain control block 120 can amplify the signal attenuated by gain control 106 of encoder 102. The HOA signal can then be reconstructed by HOA reconstruction block 122. Optionally, the HOA signal can be rendered and/or replayed by rendering/replay block 124. Rendering/replay block 124 can include various algorithms, for example, for rendering the reconstructed HOA output as, for example, rendered audio data. For example, rendering the reconstructed HOA output can involve distributing one or more signals of the HOA output to multiple speakers to achieve a specific perceptual impression. Optionally, rendering/replay block 124 can include one or more amplifiers, headphones, etc., for presenting the rendered audio data.

Gain control 106 can be implemented using the following techniques. Gain control 106 can first determine an upper limit for the signal values in the frame. For example, for an MPEG-H audio signal, this upper limit can be represented as a product where the product is specified in the MPEG-H standard. Given an upper limit, the minimum required attenuation can ensure that the scaled signal samples are limited to the interval [-1, 1). In other words, the scaled samples are within the range of the core encoder 108. This can be determined by the applied gain factor, where e _{_min} can be negative by definition. In some embodiments, amplification can be limited by a maximum amplification factor, where e _{_max} is a non-negative integer. Therefore, to perform both attenuation and amplification, a gain factor of ^2e can be defined, where the gain parameter e is a value in the range [e _{_min} , e _{_max} ]. Therefore, the minimum number of bits required to represent the gain parameter e is determined as _βe = ceil(log _₂ (|e _{_min} | + e_{_max} + 1)).

The gain factor g _{_n} (j) for a specific channel n and frame j can be determined by applying a frame delay corresponding to a HOA block and using the following recursive operation:

In the above formula, g _{_n}(j-2) represents the gain factor applied to frame (j-2), and represents the gain factor adjustment required to calculate the gain factor g _{_n} (j-1) of frame j-1.

This paper discloses a technique for providing adaptive gain control. Specifically, as described herein, a gain parameter that does not introduce additional delay can be determined because the gain parameter can be determined based on lead samples generated for use by the codec. The codec can be used by a perceptual encoder. The determination of the gain transition function is illustrated and described below in conjunction with Figures 2-5.

Figures 2A and 2B show schematic block diagrams of an encoder 202 and a decoder 212 for performing low-latency adaptive gain control according to exemplary embodiments. In the encoder 202, the input HOA signal (or first-order ambient stereo (FOA)) signal is processed by a spatial analysis block 204. For an N-channel HOA input, the spatial analysis block 204 can generate and output a set of M downmixing channels 204A. The number of downmixing channels in this set of M downmixing channels 204A can be in the range of 1 ≤ M ≤ N. Furthermore, the spatial analysis block 204 can generate and output spatial side information 204B for reversing the downmixing operation.

For example, for FOA input, the downmix channel may include a main downmix channel W’, which can be generated by mixing the omnidirectional input signal W with directional input signals X, Y, and Z using various mixing gains, and up to three residual channels X’, Y’, and Z’, each corresponding to a signal component in the X, Y, and Z signals that cannot be predicted from the main downmix signal. In one example, spatial analysis block 204 utilizes the spatial reconstruction (SPAR) technique. SPAR is further described below, which is incorporated herein by reference in its entirety: “Immersive Audio Coding for Virtual Reality Using a Metadata-assisted Extension of the 3GPP EVS Codec”, D. McGrath, S. Bruhn, H. Purnhagen, M. Eckert, J. Torres, S. Brown, and D. Darcy, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 730-734. In other examples, spatial analysis block 204 can utilize any other suitable linear predictive codec of the energy compression transform, such as the Karhunen-Loov transform (KLT). Core encoder 208 can be considered as a codec that generates the encoded audio bitstream 208A. In some implementations, core encoder 208 and core decoder 216 can introduce some lookahead samples, which will be used by adaptive gain control 206 to determine the gain parameters to avoid adding additional latency to the entire encoding process (zero additional latency).

The signals associated with the M lower mixing channels 204A can then be analyzed using adaptive gain control 206. Adaptive gain control 206 can determine whether the signal associated with any of the M lower mixing channels 204A exceeds the expected audio amplitude range of the core encoder 208, and thus would overload the core encoder 208. In some embodiments, if adaptive gain control 206 determines that no gain will be applied, for example, in response to determining that none of the signals in the M lower mixing channels 204A exceed the expected range of the core encoder 208, adaptive gain control 206 can set a flag indicating that gain control will not be applied. The flag indication can be performed by setting the value of the flag, for example, by setting the value of a single bit. If adaptive gain control 206 determines that no gain will be applied, adaptive gain control 206 can leave the flag unset, thus reserving a bit (e.g., the bit associated with the flag). For example, in some implementations, if the spatial metadata bitstream and/or the core encoder bitstream (which may be a perceptual encoder bitstream) are self-terminating, the presence of the gain control flag can be determined by determining whether any unread bits exist in the bitstream. Unread bits can be remaining bits in the bitstream. In the absence of overload conditions, adaptive gain control 206 can output M lower mixing channels 206A. These M lower mixing channels 206A can then be passed to the core encoder 208 for encoding in the bitstream 208A.

Conversely, when adaptive gain control 206 determines to apply gain, it can determine gain parameters and apply gains(multiple) to the M downmix channels based on those parameters. The M downmix channels 206A with applied gain can then be passed to the core encoder 208 for encoding in the bitstream. Furthermore, adaptive gain control 206 can output side information 206B regarding gain control. Information about flags can be included in the side information 206B. Side information encoder 210 can encode spatial side information 204B along with the gain parameters 206B into metadata 210A for transmission in the bitstream. Decoder 212 can then extract and use this metadata to upmix the downmix channels and reverse the gain adjustment. For example, metadata 210A can later be used to reconstruct a representation of the original audio input downmixed by spatial analysis unit 204. Side information encoder 210 can additionally provide side information 208B to the core encoder 208. The core encoder 208 can then use side information 208B to select between encoding techniques. The encoded bitstream 208A and the encoded bitstream with metadata 210A can be multiplexed to form the final bitstream output by encoder 202.

In some implementations, adaptive gain control 206 may determine a gain transfer function that transforms the gain parameter e(j-1) associated with the previous frame (e.g., the (j-1)th frame) and the gain parameter e(j) of the current frame. The gain transfer function may be applied by adaptive gain control 206 on a frame-by-frame basis, where each frame may be one of the M downmixing channels 204A. In some implementations, the gain transfer function may smoothly transform the gain parameter from the gain parameter value of the (j-1)th frame (e.g., e(j-1)) to the gain parameter of the current frame (e.g., e(j)) across samples of the j-th frame. Therefore, the gain transfer function may include two parts: 1) an instantaneous part, where the gain parameter transforms from the gain parameter of the previous frame to the gain parameter of the current frame across samples of the instantaneous part; and 2) a steady-state part, where the gain parameter has the gain parameter value of the current frame for samples of the steady-state part.

In some embodiments, when the gain applied to the current frame is less than the gain applied to the previous frame, the instantaneous portion can be referred to as an instantaneous type with "fading" because the attenuation increases on the samples of the current frame. The case where the gain applied to the current frame is less than the gain applied to the previous frame can be expressed as e(j) > e(j-1). In some embodiments, when the gain applied to the current frame is greater than the gain applied to the previous frame, the instantaneous portion can be referred to as an instantaneous type with "inverse fading" or "non-fading" because the attenuation decreases on the samples of the current frame. The case where the gain applied to the current frame is greater than the gain applied to the previous frame can be expressed as e(j) < e(j-1). In some embodiments, when the gain applied to the current frame is the same as the gain applied to the current frame, the instantaneous portion can be referred to as an instantaneous type with "holding," where the instantaneous portion is not instantaneous but has the same value as the steady-state portion. The case where the gain applied to the current frame is the same as the gain applied to the current frame can be expressed as e(j) = e(j-1).

In some embodiments, the gain transition function depends on the gain transition step size. The gain transition step size can limit the amount of possible transition from the previous frame to the current frame. This is driven by the fact that smaller and smoother gain/attenuation changes that may allow for overload during transitions are perceptually better than larger changes, especially when this is to be further processed by a lossy core encoder that requires the aforementioned predefined range of values as input. By predefining the parameters of the gain transition function in this way, the impact of the parameters on objective or perceived quality can be evaluated. Perceived quality can be measured based on known perceived quality listening tests, such as the Multi-Stimulus Test with Hidden References and Anchors (MUSHRA). Perceived quality listening tests can be part of the tuning process of automatic gain control at the encoder and decoder. Specifically, parameters such as the size of the gain transition step size can be tuned for a specific audio scene and codec until the optimal perceived audio quality is achieved. The encoding/decoding system then uses the tuned parameters.

In the example implementation, the processed output 206A of the automatic gain control 206 is further encoded by a lossy core codec based on algebraically excited linear prediction (ACELP) encoding that is not intended for waveform reconstruction. It has been observed that applying larger gain steps to the ACELP inputs and outputs results in audible glitches in the reconstructed signal and degrades the overall performance of the codec.

In some implementations, when an overload is detected in the current frame, the automatic gain control 206 can also determine the amount of attenuation required for that frame to be within the expected range of the core encoder 208. If there is a large difference in the required attenuation between consecutive frames, applying a conversion function by the core encoder 208 to achieve the desired range [-1, 1) when rendering the audio signal at the decoder may result in audible artifacts. Instead of applying a conversion function to keep each frame within or at the limit of the desired range, the conversion function can be restricted to a specific gain conversion step size. Therefore, regardless of the amount of attenuation required to achieve the expected range of the core encoder 208, the conversion function can only attenuate a single frame by an amount equal to the gain conversion step size, i.e., ±DBSTEPdB. Thus, as an example, if the attenuation of the previous frame is -10dB, the attenuation applied to the first sample of the current frame will be -10dB, and the attenuation applied to the last sample of the current frame will be -10dB±DBSTEP. More precisely, if the overload does not change from the previous frame to the current frame, the gain conversion will be a constant value, for example, -10dB. If the attenuation needs to be changed, the gain conversion function will convert the attenuation from the previous frame to the last sample of the current frame by ±DBSTEPdB.

In some implementations, DBSTEP can be selected such that the attenuation applied by automatic gain control 206 is insufficient to keep the frame within the expected signal range of the core encoder 208. For example, DBSTEP can be a fixed value. When a sharp change in attenuation is required, strong attenuation differences between consecutive frames can be avoided by allowing the frame to be outside the range [-1, 1). Therefore, the attenuation of the frame relative to the previous frame is attenuated by a fixed amount, rather than forcing the frame into the range [-1, 1) through a transfer function or static gain change. Perceived audio quality can be improved by using a transfer function with a specific gain transfer step size because the distortion caused by the frame being outside the range [-1, 1) is less noticeable than the distortion caused by sharp attenuation differences between consecutive frames. Furthermore, an anomalous flag used for switching between smooth transfers and static gain changes can be avoided. Therefore, 1 bit can be saved at the core encoder 208.

In some implementations, DBSTEP can be a single value, such as -1dB. Alternatively, DBSTEP can be selected from a set of increasing fixed values, such as -1dB, -3dB, and -6dB. In this case, the value of DBSTEP can be chosen based on the overload caused by frames without attenuation.

In some implementations, automatic gain control is configured to have the ability to specify a set of target gain values _GT , which can be represented as a table of numbers (e.g., integers) indicating the factor of DBS attenuation provided at each step. This is driven by the fact that smaller variations provide a perceived benefit, however, some _signals may require a higher level of attenuation. Specifying these non-uniform absolute step sizes allows for coverage of a wider attenuation range while providing the benefit of smaller step sizes for many possible scenarios. For example, a set of _GT = {0, 1, 3, 6} with a DBS of -2dB would have an absolute target gain of {DBS* _GT } = {0dB, -2dB, -6dB, -12dB}, taking consecutive steps DBSTEP of {-2dB, -4dB, -6dB}.

One or more such integer tables can be specified, and information regarding the selection of a specific table used on the encoding side can be signaled/sent to the decoder side. In contrast to applications with uniform step sizes that produce a single uniform gain transition shape, applications with non-uniform step sizes result in non-uniform gain transition shapes (level-dependent transition functions).

In some implementations, when DBSTEP is insufficient to decay the current frame to the range [-1, 1), DBSTEP can be applied to frames following the current frame until the range [-1, 1) is reached.

In some implementations, the output level and attenuation information from the automatic gain control 206 system can be used in decision-making processes in other systems such as the core encoder 208. While relaxed requirements can provide perceived benefits, they can affect the core encoder 208 by introducing gain variations or by not meeting stringent requirements and allowing overload conditions to persist. Information such as whether gain control meets requirements, whether gain is applied, and how much gain is applied can be output and passed to the core encoder. This allows for better decision-making, such as choosing an encoding method that better handles gain variations or out-of-range samples. For example, when applying large gain/attenuation steps, the core encoder 208 can use waveform encoding techniques similar to MDCT-based encoding instead of predictive ACELP encoding techniques.

In some embodiments, the instantaneous portion of the gain transition function can be determined using the prototype shape of the instantaneous portion of the gain transition function, wherein the prototype shape is scaled based on the difference between the gain parameters of the current frame and the gain parameters of the previous frame. For example, the prototype shape can be scaled based on e(j) - e(j-1). The gain transition function using such a prototype function p can be expressed as:

Where l_{_end} represents the rightmost index of the definition p, and L represents the number of samples in a frame. For example, the prototype shape of the instantaneous partial gain can be defined as:

Where L is the number of samples in the frame for which p is defined. For example, L could be l _end + 1.

Figure 3A shows examples of gain transition functions, each with an instantaneous portion having an instantaneous type of "fading". In the example shown in Figure 3A, each gain transition function has an instantaneous portion starting from sample 0, which may correspond to the beginning of the current frame, with a gain of 0 dB, where 0 dB is the gain parameter of the previous frame (e.g., frame j-1). In the example shown in Figure 3A, the instantaneous portion of each gain transition function changes to the steady-state portion of the gain transition function over a period of approximately 384 samples. For each of the three gain transition functions shown in Figure 3A, the steady-state portion corresponds to different gain transition step sizes in frame j, with its (negative) gain increasing by 6 dB, 12 dB, and 18 dB relative to the gain of the previous frame, respectively. In other words, as shown in Figure 3A, for the three gain transition functions, exp = -[e(j) - e(j-1)] = -1, -2, and -3, respectively. For each gain transition function shown in Figure 3A, the instantaneous portion has the same length (e.g., approximately 384 samples). Note that the length of the steady-state portion can correspond to an offset related to the latency introduced by the codec, for example, 12 milliseconds in the example shown in Figure 3A. Correspondingly, the length of the instantaneous portion can be related to the reciprocal of the offset. In the example shown in Figure 3A, the length of the instantaneous portion is the frame length (e.g., 20 milliseconds) minus the codec latency (e.g., 12 milliseconds). Note that the codec latency can be the overall encoder algorithm latency, excluding frame size latency.

Additionally, the instantaneous portion of the gain transfer function with "inverse fading" or "non-fading" instantaneous types can be represented as a mirror image of the gain transfer function shown in Figure 3A, flipped across a horizontal line. For example, the horizontal line could be the x-axis.

Referring back to Figure 2B, decoder 212 can receive encoded audio bitstream 208A and metadata bitstream 210A as input, and can reconstruct the HOA signal for, for example, rendering, or directly rendering to the desired output format. In some embodiments, core decoder 216 receives encoded audio bitstream 208A. Additionally, core decoder 216 can receive information 214A extracted from metadata bitstream 210A by side information decoder 214. Core decoder 216 can decode encoded audio bitstream 208A based on information 214A or without any side information knowledge, and output M gain-adjusted lower mixing channels 216A to inverse gain control 220. Side information decoder 214 further extracts gain parameters and spatial side information, and sends this information 214B to inverse gain control 220 and spatial synthesis/rendering/replay block 222. Inverse gain control 220 can then obtain the gain parameters applied by encoder 202 from information 214B. For example, in some implementations, inverse gain control 220 may retrieve from information 214B an indication of the gain conversion step size DBSTEP applied by encoder 202 and/or an arithmetic factor associated with DBSTEP. Furthermore, inverse gain control block 220 may, for example, retrieve the shape of the conversion function, i.e., the shape of the prototype function p, also known as the smoothing function, from memory. Inverse gain control block 220 may then use the obtained gain parameters to reverse the gain applied by encoder 202 and output M lower mixing channels 220A. For example, in some implementations, inverse gain control 220 may construct an inverse gain conversion function that converts the gain parameters from the previous frame to the gain parameters of the current frame. In some implementations, the inverse gain conversion function may be a gain conversion function applied by encoder 202 that is mirrored and vertically adjusted across a center vertical line. For example, the vertical line may be the y-axis.

Turning to Figure 3B, an example of an inverse gain transition function applied by the decoder in response to the gain transition function applied by the encoder as shown in Figure 3A is illustrated according to some implementations. As shown, the inverse gain transition function has a steady-state portion and an instantaneous portion. As shown in Figures 3A and 3B, the durations of the steady-state and instantaneous portions of the inverse gain transition function can correspond to, for example, the same durations of the corresponding steady-state and instantaneous portions of the gain transition function. As shown, each inverse gain transition function shown in Figure 3B starts at 0 dB and transitions to -DBSTEP of the current frame. That is, each inverse gain transition function starts at 0 dB corresponding to the inverse gain applied in the previous frame j-1. When the gain applied by the encoder corresponds to attenuation represented by a gain less than 0 dB as shown in the gain transition function of Figure 3A, the inverse gain applied by the decoder corresponds to amplification with a gain greater than 0 dB, as shown in the gain transition function of Figure 3B. Conversely, when the gain applied by the encoder corresponds to amplification (e.g., a gain greater than 0 dB), the inverse gain applied by the decoder corresponds to attenuation, for example, a gain less than 0 dB.

Referring back to Figure 2B, after applying inverse gain, the M downmixing channels 220A with applied inverse gain are provided to the spatial synthesis/rendering/replay block 222. The spatial synthesis/rendering/replay block 222 can reconstruct the HOA signal using information 214B. For example, if the spatial analysis block 204 uses SPAR technology for spatial encoding, the spatial synthesis/rendering/replay block 222 can utilize SPAR technology to reconstruct one or more channels encoded using metadata 210A. The reconstructed HOA output can then be rendered directly or provided to another entity for rendering. The spatial synthesis/rendering/replay block 222 may include various algorithms, for example, for rendering the reconstructed HOA output as, for example, rendered audio data. For example, rendering the reconstructed HOA output may involve distributing one or more signals of the HOA output to multiple speakers to achieve a specific perceptual impression. Optionally, the spatial synthesis/rendering/replay block 222 may include audio playback devices for presenting the rendered audio data, such as one or more loudspeakers, headphones, etc.

Figure 4 illustrates an example of a process 400, according to some implementations, for determining gain parameters and applying gain to the undermixed signal based on the determined gain parameters. In some implementations, blocks of process 400 may be executed by an encoder device. In some implementations, blocks of process 400 may be executed in a sequence other than that shown in Figure 4. In some implementations, two or more blocks of process 400 may be executed substantially in parallel. In some implementations, one or more blocks of process 400 may be omitted.

At 402, process 400 may obtain (multiple) downmixed audio signals associated with frames of the audio signal to be encoded. These (multiple) downmixed audio signals may be associated with frames of the audio signal to be encoded. For example, in some implementations, process 400 may use any suitable spatial coding technique to determine a set of downmixed channels. Examples of spatial coding techniques include SPAR, linear prediction, etc. This set of downmixed channels may include any number from 1 to N, where N is the number of input channels, for example, 4 in the case of a FOA signal. The downmixed signal may include audio signals of the downmixed channels corresponding to specific frames of the audio signal.

At 404, process 400 may determine whether an overload condition exists for a codec such as the Enhanced Voice Service (EVS) codec and/or for any other suitable codec. For example, in response to determining that the signal corresponding to a frame(s) of the submixed audio signal exceeds a predetermined range (e.g., [-1, 1)) and/or any other suitable range, process 400 may determine that an overload condition exists.

If it is determined at 404 that no overload condition exists (404 is "No"), then process 400 can proceed to 412 and the downmixed signal can be encoded. For example, in some implementations, process 400 can generate a bitstream that encodes the downmixed signal by incorporating side information such as metadata, which the decoder can use to upmix the downmixed signal, for example, to reconstruct the FOA or HOA output.

Conversely, if an overload condition is determined to exist at 404 ("Yes" at 404), process 400 can proceed to 406, and the gain transition function for the frame that leads to avoiding the overload condition can be determined, or the overload can be reduced at least if the change in the overload condition from one frame to the next is greater than the gain transition step size. Furthermore, in 406, the gain transition function can be based on the gain transition step size. Additionally, the gain transition function can be based on the shape of a smoothing function. Furthermore, as described above in conjunction with Figure 2, the gain transition function can have an instantaneous part and a steady-state part, where the steady-state part corresponds to the gain factor of the current frame, and the instantaneous part corresponds to the intermediate gain factor sequence of a sample subset of the current frame, which transitions from the gain factor at the end of the previous frame to the gain factor ±DBSTEP of the previous frame.

When the gain parameter of the previous frame corresponds to less attenuation than the gain parameter of the current frame, the transient portion can be called a transient type with "fading". Conversely, when the gain parameter of the previous frame corresponds to more attenuation than the gain parameter of the current frame, the transient portion can be called a transient type with "inverse fading" or "non-fading". When the gain parameter of the previous frame is the same as the gain parameter of the current frame, the transient portion can be called a transient type with "holding". In the case of a transient portion with "holding", the value of the gain transfer function during the transient portion can be the same as the value of the gain transfer function during the steady-state portion. As described above in conjunction with Figure 2, the duration of the transient portion of the gain transfer function can correspond to the delay duration utilized by the codec.

At 408, process 400 may apply a gain transition function to the downmixed signal associated with the frame. For example, in some implementations, process 400 may scale samples of the downmixed signal using a gain factor indicated by the gain transition function. As a more specific example, in some implementations, the first sample of the current frame may be scaled using a gain factor corresponding to the gain parameter of the previous frame, the last sample of the current frame may be scaled using a gain factor corresponding to the gain parameter ±DBSTEP of the previous frame, and intermediate samples may be scaled using a gain factor corresponding to the gain parameter of the instantaneous or steady-state portion of the gain transition function.

In some implementations, the gain conversion function may be applied only to the downmixed signal of the downmixed channel where an overload condition is detected at block 404. For example, if an overload condition is detected in both the Y' and X' channels, a separate gain conversion function can be determined for each of the Y' and X' channels and applied to the signals of those channels. Continuing this example, the gain conversion function may not be applied to the W' and Z' channels. In this case, for example at block 412, an indication of the channel to which the gain conversion function has been applied, along with the corresponding gain parameter for each channel, can be encoded. Alternatively, in some implementations, if an overload condition exists only for one downmixed channel, the corresponding gain conversion function can be applied to all downmixed channels. In this case, because the gain conversion function is applied to all channels, it is not necessary to send an indication of the channel to which the gain has been applied, which may result in improved bit rate efficiency.

At 410, process 400 can provide the attenuated signal and information indicating the gain transition function to the encoder for encoding. The information indicating the gain transition function can be the gain transition step size and/or an arithmetic factor associated with the gain transition step size. Additionally, the shape of the smoothing function can be provided to the encoder for encoding.

At 412, process 400 may encode the downmixed signal, and if gain is applied, encode information indicating the gain(s) of the frame(s). In the case of gain application, at block 408, the encoded downmixed signal may be the downmixed signal after the application of a gain transformation function. The downmixed signal and any information indicating the gain parameters may be encoded by a codec (e.g., EVS codec, etc.) in relation to any side information (e.g., metadata) that the decoder can use to reconstruct or upmix the downmixed signal to generate an encoded bitstream. The encoded bitstream, along with the metadata, may then be stored and/or transmitted to a receiving device capable of processing steps with a reverse encoder.

It should be noted that in some implementations, process 400 may encode the gain parameter as a set of bits. In some implementations, the gain transition function may indicate a prototype/smoothing function associated with the instantaneous portion of the gain transition function.

When adaptive gain control is enabled for each channel, thus applying a unique gain conversion function to each downmix channel associated with the signal triggering the overload condition, x bits can be used for each channel with gain control enabled, with an additional bit indicating that the gain parameter has been encoded. In this case, the total number of bits used to send gain control information is N _dmx + x * N _agc , where N _dmx represents the number of downmix channels (and where a single bit is used for each of the N _dmx channels to indicate whether gain control is enabled), and where N _agc represents the number of channels with gain control enabled. It should be noted that if gain control is not enabled for a particular frame, N _dmx bits can be used to indicate that gain control is not enabled, for example, 1 bit for each of the N _dmx channels. Note that when the number of downmix channels is 1, for example, when only channel W is waveform-coded, the total number of bits used to send gain control information is x * N _agc . For example, given a single bottom mix channel, if gain control is not enabled for that bottom mix channel (e.g., _Nagc = 0), then 0 bits are used. Continuing with the example, if gain control is enabled (e.g., _Nagc = 1), then x bits are used.

When a single gain conversion function associated with the down mix channel that triggers the overload condition is applied to all down mix channels, fewer bits can be used to send gain control information. For example, x bits can be used to send a single gain parameter for the current frame.

Figure 5 illustrates an example of a process 500, according to some implementations, for obtaining gain parameters used by the encoder and applying an inverse gain transformation function based on the obtained gain parameters. In some implementations, blocks of process 500 may be executed by a decoder device. In some implementations, blocks of process 500 may be executed in a sequence other than that shown in Figure 5. In some implementations, two or more blocks of process 500 may be executed substantially in parallel. In some implementations, one or more blocks of process 500 may be omitted.

Process 500 can begin at 502 by receiving an encoded frame of an audio signal. The received frame (e.g., the current frame) is generally referred to herein as frame j. The received frame may immediately follow a previously received frame, or it may be a frame that does not immediately follow a previously received frame.

In step 504, process 500 can decode the encoded frames of the audio signal to obtain the downmixed signal, and if gain control is applied to the encoder, obtain information indicating the gain control applied to the current frame. This information indicating the gain control applied to the current frame can be the gain transition step size applied by the encoder. Alternatively, this information can be the shape of a smoothing function of the gain transition function applied by the encoder. If the encoder applies gain control on a per-channel basis, process 500 can also identify which downmixed channels have applied gain control.

In 506, procedure 500 can determine the inverse gain transfer function based on the gain transfer step size. In some implementations, procedure 500 can also determine the inverse gain transfer function based on the shape of a smoothing function. The inverse gain transfer function can be computed based on the gain transfer function, or it can be selected from several predefined inverse gain transfer functions.

In some implementations, process 500 may determine the inverse gain transition function as the inverse of the gain transition function applied at the encoder. For example, the inverse gain transition function may correspond to a gain transition function that is mirrored and adjusted across a horizontal line. Mirroring and adjustment may be along the x-axis. An example of such an inverse gain transition function is shown and described above in conjunction with Figure 3B. In some implementations, the inverse gain transition function may have a steady-state portion corresponding to the gain applied to the previous frame. The inverse gain transition function may then have an instantaneous portion that is the inverse of the instantaneous portion of the gain transition function applied at the encoder. For example, if the gain applied to the current frame corresponds to a greater attenuation relative to the previous frame, the inverse gain transition function may have an instantaneous portion transitioning from a smaller amplification to a larger amplification. Conversely, if the gain applied to the current frame corresponds to a smaller attenuation relative to the previous frame, the inverse gain transition function may have an instantaneous portion transitioning from a larger amplification to a smaller amplification. The duration of the instantaneous portion may involve a delay introduced by the codec, wherein the duration of the instantaneous portion is the frame length (e.g., 20 milliseconds) minus the codec delay (e.g., 12 milliseconds). Note that when the delay introduced by the codec is longer than the frame length, the delay of one frame can be used to apply the inverse gain conversion. In some cases, the delay can be obtained from the gain control bits by process 500 (e.g., via the decoder). The inverse gain conversion function can also be used to attenuate signals amplified by the encoder's gain control.

At 508, process 500 can apply an inverse gain conversion function to the downmixed signal to reverse the gain applied by the encoder. For example, applying the inverse gain conversion function can cause the downmixed signal attenuated by the encoder to be amplified to reverse the attenuation. As another example, applying the inverse gain conversion function can cause the downmixed signal amplified by the encoder to be attenuated to reverse the amplification. Then, the output of step 508 can be M downmixed channels with the same gain as the M downmixed channels after step 402 of process 400.

In step 510, process 500 can upmix the undermixed signal. Upmixing can be performed by a spatial encoder. In some examples, the spatial encoder can utilize SPAR technology. The upmixed signal can correspond to a reconstructed FOA or HOA audio signal. In some implementations, process 500 can use side information (e.g., metadata) encoded in the bitstream to upmix the signal, where the side information can be used to reconstruct the parametrically encoded signal. In some implementations, block 510 can be optional, for example, when the undermixed signal can be rendered directly.

In some implementations, at 512, process 500 can render the mixed signal to generate rendered audio data. In some implementations, process 500 can utilize any suitable rendering algorithm to render the FOA or HOA audio signal, such as rendering scene-based audio data. In some implementations, the rendered audio data can be stored in any suitable format, for example, for future presentation or playback. In some implementations, block 512 is optional and can therefore be omitted.

In some implementations, at 514, procedure 500 can allow the rendered audio data to be played back. For example, in some implementations, the rendered audio data can be presented via one or more loudspeakers and/or headphones. In some implementations, multiple loudspeakers can be used, and the multiple loudspeakers can be positioned relative to each other in three dimensions at any suitable location or orientation. In some implementations, procedure 514 is optional and can therefore be omitted.

As described above in conjunction with Figure 4, a set of gain control bits can be used to encode gain control information, such as information indicating gain parameters. In some implementations, a different gain transition function can be determined for each downmixer channel where an overload condition is detected. In such an implementation, gain control bits are needed to indicate whether gain control is applied to each downmixer channel, and the gain transition function parameters are encoded for each downmixer channel where gain control is applied, as described above in conjunction with Figure 4. Alternatively, in some implementations, a single gain transition function determined based on the presence of an overload condition in one downmixer channel can be applied to all downmixer channels. In such an implementation, fewer gain control bits are needed because no separate bit flag is required to indicate whether gain control has been applied to each downmixer channel, resulting in more bit-rate efficient encoding.

Applying the same gain transfer function to all downmix channels, including those without overload conditions, with more efficient bitrate coding can lead to a deterioration in perceived quality, for example, by attenuating signals without codec overload. In contrast, utilizing more targeted gain control (where gain control is applied to each downmix channel in a targeted manner) might require more bits to send the gain control information. However, using additional bits to send targeted (e.g., channel-specific) gain control information might require reallocating bits typically used for waveform encoding of the downmix channels, which could degrade perceived quality in some cases. Therefore, there may be a situational trade-off between applying the same gain transfer function to all downmix channels and applying channel-specific gain control. Whether gain control is applied on all downmix channels or on a per-channel basis, bits associated with gain control information can be allocated from the bits typically used for waveform encoding of downmix channels and/or from the bits typically used for encoding side information (e.g., metadata) for reconstructing FOA or HOA signals from downmix channels, thereby reducing the number of available bits for encoding downmix channels or side information.

Figure 6 illustrates an example use case of an IVAS system 600 according to one embodiment. In some embodiments, various devices communicate via a call server 602, which is configured to receive audio signals from, for example, a Public Switched Telephone Network (PSTN) or Public Land Mobile Network (PLMN) device, as shown by PSTN/other PLMN 604. The use case supports conventional devices 606 that render and capture audio in mono only, including but not limited to: devices supporting Enhanced Voice Service (EVS), Multi-Rate Broadband (AMR-WB), and Adaptive Multi-Rate Narrowband (AMR-NB). The use case also supports user equipment (UE) 608 and/or 614 that capture and render stereo audio signals, or UE 610 that captures mono signals and renders them as multi-channel signals. The use case also supports immersive and stereo signals captured and rendered by video conferencing systems 616 and/or 618, respectively. The use cases also support stereo capture and immersive rendering of stereo audio signals for home theater system 620, and computer 612 for mono capture and immersive rendering of audio signals for virtual reality (VR) device 622 and immersive content ingestion 624.

Figure 7 is a block diagram illustrating examples of components of an apparatus capable of implementing various aspects of the present disclosure. As with the other figures provided herein, the types and quantities of elements shown in Figure 7 are provided by way of example only. Other implementations may include more, fewer, and/or different types and quantities of elements. According to some examples, apparatus 700 may be configured to perform at least some of the methods disclosed herein. In some implementations, apparatus 700 may be or may include one or more components of a television, an audio system, a mobile device (such as a cellular phone), a laptop computer, a tablet device, a smart speaker, or another type of device.

Depending on some alternative implementations, device 700 may be or may include a server. In some such examples, device 700 may be or may include an encoder. Thus, in some instances, device 700 may be a device configured for use in an audio environment such as a home audio environment, while in other instances, device 700 may be a device configured for use in the “cloud,” such as a server.

In this example, device 700 includes an interface system 705 and a control system 710. In some implementations, the interface system 705 may be configured to communicate with one or more other devices in the audio environment. In some examples, the audio environment may be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. In some implementations, the interface system 705 may be configured to exchange control information and associated data with audio devices in the audio environment. In some examples, the control information and associated data may be related to one or more software applications being executed by device 700.

In some implementations, interface system 705 can be configured to receive or provide content streams. The content stream may include audio data. The audio data may include, but is not limited to, audio signals. In some cases, the audio data may include spatial data, such as channel data and/or spatial metadata. In some examples, the content stream may include video data and audio data corresponding to that video data.

Interface system 705 may include one or more network interfaces and/or one or more external device interfaces, such as one or more Universal Serial Bus (USB) interfaces. According to some implementations, interface system 705 may include one or more wireless interfaces. Interface system 705 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and/or a gesture sensor system. In some examples, interface system 705 may include one or more interfaces between control system 710 and a memory system such as optional memory system 715 shown in FIG. 7. However, in some cases, control system 710 may include a memory system. In some implementations, interface system 705 may be configured to receive input from one or more microphones in the environment.

For example, the control system 710 may include a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic, and/or discrete hardware components.

In some implementations, the control system 710 may reside in more than one device. For example, in some implementations, a portion of the control system 710 may reside in a device within one of the environments described herein, and another portion of the control system 710 may reside in a device outside the environment, such as a server, mobile device (e.g., a smartphone or tablet computer), etc. In other examples, a portion of the control system 710 may reside in a device within an environment, and another portion of the control system 710 may reside in one or more other devices within that environment. For example, a portion of the control system 710 may reside in a device implementing a cloud-based service, such as a server, and another portion of the control system 710 may reside in another device implementing a cloud-based service, such as another server, a storage device, etc. In some examples, the interface system 705 may also reside in more than one device.

In some implementations, the control system 710 may be configured to at least partially perform the methods disclosed herein. According to some examples, the control system 710 may be configured to implement methods such as determining gain parameters, applying a gain transition function, determining an inverse gain transition function, applying the inverse gain transition function, and allocating bits for gain control to a bitstream.

Some or all of the methods described herein can be executed by one or more devices according to instructions (e.g., software) stored on one or more non-transient media. Such non-transient media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transient media may reside, for example, in the optional memory system 715 and/or control system 710 shown in FIG. 7. Therefore, various innovative aspects of the subject matter described herein can be implemented in one or more non-transient media on which software is stored. The software may, for example, include instructions for determining gain parameters, applying a gain transition function, determining an inverse gain transition function, applying the inverse gain transition function, allocating bits for gain control to a bitstream, etc. For example, the software can be executed by one or more components of a control system such as the control system 710 of FIG. 7.

In some examples, device 700 may include the optional microphone system 720 shown in FIG. 7. The optional microphone system 720 may include one or more microphones. In some implementations, the one or more microphones may be part of or associated with another device, such as a speaker in a speaker system, a smart audio device, etc. In some examples, device 700 may not include the microphone system 720. However, in some such implementations, device 700 may still be configured to receive microphone data from one or more microphones in an audio environment via interface system 710. In some such implementations, a cloud-based implementation of device 700 may be configured to receive microphone data or at least a noise metric corresponding to the microphone data from one or more microphones in an audio environment via interface system 710.

According to some implementations, device 700 may include the optional speaker system 725 shown in FIG. 7. The optional speaker system 725 may include one or more amplifiers, which may also be referred to herein as “speakers,” or more generally as “audio reproduction transducers.” In some examples (e.g., cloud-based implementations), device 700 may not include speaker system 725. In some implementations, device 700 may include headphones. Headphones may be connected to or coupled to device 700 via a headphone jack or via a wireless connection (e.g., Bluetooth).

Figures 8A and 8B illustrate an example implementation of gain control for perceptual excitation, with sample-uniform gain control of DBSTEP = -1dB on the encoder side. In this particular example, a frame consists of 1024 samples. Sample amplitudes are represented by dashed lines, while the gain applied to each sample is represented by solid lines. As shown in Figure 8A, once the frame generates an overload on the encoder (amplitude greater than 0dB), the gain function changes from no attenuation (0dB) to attenuation of DBSTEP - 0dB = -1dB. Further attenuation is introduced by DBSTEP when the input audio signal exceeds 1dB.

Figure 8B shows the resulting attenuated downmixed audio signal. In this particular example, the value of DBSTEP is large enough that each sample is attenuated below the desired threshold (0dB).

Figures 9A and 9B illustrate an example of “non-uniform” gain control that generates a set of attenuation values {DBS* _GT } = {0,-1,-3,-6} dB or a DBSTEP set {-1,-2,-3} on the encoder side, with DBS = -1 dB and _GT = {0,1,3,6}. As shown in Figures 8A and 8B, the sample amplitudes are represented by dashed lines, while the gain function is represented by solid lines. Using the DBSTEP set, automatic gain control can react to overload caused by the encoder by attenuating the signal with incremental values in each frame. As shown in Figure 9B, the gain transition step size is not large enough that all samples are below the desired threshold (0 dB). This may cause distortion when the audio signal is rendered at the decoder, but the distortion caused by overload at the encoder is not as noticeable as the distortion caused by very sudden gain changes.

Some aspects of this disclosure include systems or devices configured (e.g., programmed) to perform one or more examples of the disclosed methods, and tangible computer-readable media, such as a disk, storing code for implementing one or more examples of the disclosed methods or steps. For example, some of the disclosed systems may be or include a programmable general-purpose processor, digital signal processor, or microprocessor, programmed with software or firmware and/or otherwise configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system including input devices, memory, and a processing subsystem programmed (and/or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to declared data.

Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor configured (e.g., programmed and otherwise configured) to perform desired processing on (multiple) audio signals, including performing one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed system (or elements thereof) may be implemented as a general-purpose processor, such as a personal computer (PC) or other computer system or microprocessor, which may include input devices and memory programmed with software or firmware and/or otherwise configured to perform any of the various operations including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of the system of the present invention are implemented as general-purpose processors or DSPs configured (e.g., programmed) to perform one or more examples of the disclosed methods, and the system also includes other elements. Other elements may include one or more speakers and/or one or more microphones. A general-purpose processor configured to perform one or more examples of the disclosed methods may be coupled to an input device. Examples of input devices include, for example, a mouse and/or a keyboard. The general-purpose processor may be coupled to memory, a display device, etc.

Another aspect of this disclosure is a computer-readable medium, such as a disk or other tangible storage medium, that stores code for performing one or more examples of the disclosed methods or steps, for example, by an executable encoder.

While specific embodiments and applications of this disclosure have been described herein, it will be apparent to those skilled in the art that many variations of the described embodiments and applications are possible without departing from the scope of the disclosure described and claimed herein. It should be understood that while certain forms of this disclosure have been shown and described, this disclosure is not limited to the specific embodiments described and shown or the specific methods described.

Various aspects and implementations of this disclosure can also be understood from the following exemplary embodiments (EEE), which are not claims.

EEE1. A method for performing gain control on an audio signal, the method comprising:

Obtain the downmixed audio signal of the audio signal to be encoded;

It was determined that an overload condition had occurred in the frame of the downmixed audio signal;

In response to determining that the overload condition has occurred, a gain conversion function for the frame is determined, wherein the gain conversion function is based at least on a gain conversion step size;

The gain conversion function is applied to the frame to generate a gain-adjusted frame of the downmixed audio signal; and

The gain-adjusted frame and information indicating the gain conversion function are provided for encoding by the encoder.

EEE2. The method described in claim EEE1, wherein the method further comprises:

The gain-adjusted frame is encoded together with information indicating the gain conversion function.

EEE3. The method described in the foregoing statement, wherein obtaining the downmixed audio signal of the audio signal to be encoded comprises:

Receive the downmixed audio signal; or

The downmixed audio signal is determined from the audio signal to be encoded.

EEE4. The audio signal described in any of the foregoing statements is a high-order ambient stereo (HOA) audio signal.

EEE5. According to any of the methods described in the foregoing statement, the downmixed audio signal is a spatially encoded downmixed signal.

EEE6. According to any of the methods described in the foregoing statement, the overload condition is a condition in which the frame of the downmixed audio signal exceeds a predefined signal range.

EEE7. According to the method described in EEE6, the predefined signal range is the signal range expected by the encoder.

EEE8. According to any of the methods described in the foregoing statements, the frame of the downmixed audio signal is the current frame, and the gain conversion function is also based on a previous gain conversion function applied to the previous frame of the current frame.

EEE9. According to any of the methods described in the foregoing statements, the gain conversion function further depends on a smoothing function based on the gain conversion step size.

EEE10. The method according to EEE8, wherein the gain conversion function includes an instantaneous portion and a steady-state portion, and wherein the instantaneous portion corresponds to a conversion from a gain associated with the previous frame to a gain associated with the previous frame adjusted by the gain conversion step size.

EEE11. The method according to EEE10, wherein, depending on the gain adjustment target of the current frame, the gain associated with the previous frame adjusted by the gain conversion step size is either attenuating the gain associated with the previous frame by the gain conversion step size or amplifying the gain conversion step size.

EEE12. The method according to EEE10 or 11, wherein the length of the instantaneous portion is limited by the delay introduced by the codec used by the encoder.

EEE13. According to the method of EEE12, the length of the instantaneous portion is equal to or less than the number of samples used by the encoder for encoding operations.

EEE14. The method according to any one of EEE10 to 13, wherein the length of the instantaneous portion is greater than one sample.

EEE15. According to any of the methods described in the foregoing statements, wherein the gain conversion function is defined as

Where DBSTEP is the gain conversion step size, l is the sample index, j is the frame index, p() is the smoothing function, l _end indicates that it defines the rightmost index of p(), and L is the number of samples in a frame.

EEE16. According to any of the methods described in the foregoing statement, the gain conversion step size is a predefined value.

EEE17. According to any of the methods described in the foregoing statement, the gain conversion step size is determined from a set of predefined values that increase in size.

EEE18. The method according to EEE17, wherein the method further comprises:

Determine the amount of overload caused by the frame of the downmixed audio signal;

The gain conversion step size is determined based on a set of predefined values that increase from the magnitude of the overload.

EEE19. The method according to any of the foregoing statements, wherein the gain conversion step size is determined based on perceived quality listening tests or objective quality measurements.

EEE20. According to the method described in EEE19, the perceived quality listening test is the MUSHRA multistimulus test with hidden references and anchors.

EEE21. The method according to any of the foregoing statements, wherein applying the gain conversion function to the frame to generate a gain-adjusted frame of the undermixed signal comprises:

The gain conversion function is applied to samples of the downmixed audio signal, wherein the total number of samples corresponds to the frame of the downmixed audio signal.

EEE22. The method according to any one of EEE2 or EEE3 to 21, wherein encoding the gain-adjusted frame together with information indicating the gain conversion function comprises:

The encoding scheme is determined based on the gain conversion function.

EEE23. The method according to EEE22, wherein determining the coding scheme based on the gain conversion function includes:

The encoding scheme is determined based on the gain conversion step size.

EEE24. The method according to EEE22, wherein determining the coding scheme based on the gain conversion function includes:

The encoding scheme is determined based on whether the gain conversion function can eliminate the overload condition.

EEE25. The method according to any one of EEE22 to 24, wherein the encoding scheme is one of Modified Discrete Cosine Transform (MDCT) or Algebraic Coded Excited Linear Prediction (ACELP).

EEE26. According to any of the methods described in the foregoing statement, the gain-adjusted frame is an attenuated frame or an amplified frame.

EEE27. A method for performing gain control on an audio signal, the method comprising:

The decoder receives the encoded frames of the audio signal;

The encoded frames of the audio signal are decoded to obtain frames of the submixed audio signal and information indicating the gain control applied by the encoder;

The inverse gain transition function to be applied to the frame of the downmixed audio signal is determined at least in part based on information indicating the gain control applied by the encoder, wherein the information indicating the gain control applied by the encoder includes a gain transition step size; and

The inverse gain conversion function is applied to the frame of the downmixed audio signal.

EEE28. The method according to EEE27, wherein the method further comprises:

The downmixed audio signal is upmixed to generate an upmixed audio signal, wherein the upmixed audio signal is suitable for rendering.

EEE29. The method according to EEE28 further includes rendering the overmixed signal to generate rendered audio data.

EEE30. The method according to EEE29 further includes playing back the rendered audio data using one or more of a loudspeaker or headphones.

EEE31. The method according to any one of EEE27 to 30, wherein the information indicating the gain control applied by the encoder further includes information indicating a smoothing function.

EEE32. The method according to any one of EEE27 to 31, wherein the inverse gain conversion function is determined by inverting the gain conversion function applied by the encoder.

EEE33. The method according to any one of EEE27 to 32, wherein the inverse gain conversion function comprises an instantaneous portion and a steady-state portion.

EEE34. According to the method of EEE33, the length of the instantaneous portion is limited by the delay introduced by the codec used by the decoder.

EEE35. An apparatus configured to implement any one of EEE1-34.

EEE36. A program comprising instructions that, when executed by a processing device, cause the processing device to perform the method according to any one of EEE1-34.

EEE37. A storage medium for storing a program according to EEE36.

EEA38. A method for performing gain control on an audio signal, the method comprising:

The spatially encoded submixed audio signal is received by the automatic gain control system;

Determine if an overload condition occurs in one or more frames of the received signal;

In response to the overload condition, an attenuated signal is generated by applying a gain function to the received signal to attenuate the overload, the gain function depending on (1) an attenuation level parameter, (2) a gain function shape specifying a corresponding attenuation level for each of the one or more frames, or (3) a combination of the attenuation level parameter and the gain function shape; and

The attenuation signal and attenuation level parameters are provided to the core encoder for encoding.

EEE39. The method according to EEE38, wherein the attenuation level parameter comprises a table of numbers, each number corresponding to a corresponding attenuation level to be applied continuously to the one or more frames.

EEE40. The method according to EEE39, wherein each number has the same value, indicating that each step of attenuation causes the signal to attenuate by the same amount.

EEE41. According to the method of EEE39, the value of the number increases, indicating that each step of attenuation causes the signal to attenuate by a greater amount than the previous step.

EEA42. The method according to any one of EEE38-41, comprising guiding the core encoder to encode the audio signal using different encoding schemes based on the attenuation level parameter.

EEA43. The method according to any one of EEE38-42, including changing the shape of the gain function based on different values of the attenuation level parameter.

EEA44. An apparatus configured to implement the method described in any of EEE38-43.

EEA45. One or more non-transient media having software stored thereon, the software including instructions for controlling one or more devices to perform the methods described in any one of EEE38-43.

Claims (37)

1. A method for performing gain control on an audio signal, the method comprising: Obtain the downmixed audio signal of the audio signal to be encoded; It was determined that an overload condition had occurred in the frame of the downmixed audio signal; In response to determining that the overload condition has occurred, a gain conversion function for the frame is determined, wherein the gain conversion function is based at least on a gain conversion step size; The gain conversion function is applied to the frame to generate a gain-adjusted frame of the downmixed audio signal; and The gain-adjusted frame and information indicating the gain conversion function are provided for the encoder to encode. 2. The method according to claim 1, wherein the method further comprises: The gain-adjusted frame is encoded together with information indicating the gain conversion function. 3. The method according to the preceding claim, wherein obtaining the downmixed audio signal of the audio signal to be encoded comprises: Receive the downmixed audio signal; or The downmixed audio signal is determined from the audio signal to be encoded. 4. The method according to any of the preceding claims, wherein the audio signal is a high-order ambient stereo (HOA) audio signal. 5. The method according to any of the preceding claims, wherein the downmixed audio signal is a spatially encoded downmixed signal. 6. The method according to any of the preceding claims, wherein the overload condition is a condition in which the frame of the downmixed audio signal exceeds a predefined signal range. 7. The method of claim 6, wherein the predefined signal range is the signal range desired by the encoder. 8. The method according to any of the preceding claims, wherein the frame of the downmixed audio signal is the current frame, and the gain conversion function is also based on a previous gain conversion function applied to the previous frame of the current frame. 9. The method according to any of the preceding claims, wherein the gain conversion function further depends on a smoothing function based on the gain conversion step size. 10. The method of claim 8, wherein the gain conversion function comprises an instantaneous portion and a steady-state portion, and wherein the instantaneous portion corresponds to a conversion from a gain associated with the previous frame to a gain associated with the previous frame adjusted by the gain conversion step size. 11. The method of claim 10, wherein, depending on the gain adjustment target of the current frame, the gain associated with the previous frame adjusted by the gain conversion step size is either a decrease or an increase of the gain associated with the previous frame by the gain conversion step size. 12. The method of claim 10 or 11, wherein the length of the instantaneous portion is limited by a delay introduced by the codec used by the encoder. 13. The method of claim 12, wherein the length of the instantaneous portion is equal to or less than the number of samples used by the encoder for encoding operations. 14. The method according to any one of claims 10 to 13, wherein the length of the instantaneous portion is greater than one sample. 15. The method according to any of the preceding claims, wherein the gain conversion function is defined as Where DBSTEP is the gain conversion step size, l is the sample index, j is the frame index, p() is the smoothing function, l _end indicates that it defines the rightmost index of p(), and L is the number of samples in a frame. 16. The method according to any of the preceding claims, wherein the gain conversion step size is a predefined value. 17. The method according to any of the preceding claims, wherein the gain conversion step size is determined from a set of predefined values that increase in size. 18. The method of claim 17, wherein the method further comprises: Determine the amount of overload caused by the frame of the downmixed audio signal; The gain conversion step size is determined based on a set of predefined values that increase the overload from the stated size. 19. The method according to any of the preceding claims, wherein the gain conversion step size is determined based on a perceived quality listening test or an objective quality measurement. 20. The method of claim 19, wherein the perceived quality listening test is the MUSHRA multistimulus test with hidden references and anchors. 21. The method according to any of the preceding claims, wherein applying the gain conversion function to the frame to generate a gain-adjusted frame of the downmixed signal comprises: The gain conversion function is applied to samples of the downmixed audio signal, wherein the total number of samples corresponds to the frame of the downmixed audio signal. 22. The method according to claim 2 or any one of claims 3 to 21, wherein encoding the gain-adjusted frame together with information indicating the gain conversion function comprises: The encoding scheme is determined based on the gain conversion function. 23. The method of claim 22, wherein determining the encoding scheme based on the gain conversion function comprises: The encoding scheme is determined based on the gain conversion step size. 24. The method of claim 22, wherein determining the encoding scheme based on the gain conversion function comprises: The encoding scheme is determined based on whether the gain conversion function can eliminate the overload condition. 25. The method according to any one of claims 22 to 24, wherein the encoding scheme is one of Modified Discrete Cosine Transform (MDCT) or Algebraic Coded Excited Linear Prediction (ACELP). 26. The method according to any of the preceding claims, wherein the gain-adjusted frame is an attenuated frame or an amplified frame. 27. A method for performing gain control on an audio signal, the method comprising: The decoder receives the encoded frames of the audio signal; The encoded frames of the audio signal are decoded to obtain frames of the submixed audio signal and information indicating the gain control applied by the encoder; The inverse gain transition function to be applied to the frame of the downmixed audio signal is determined at least in part based on information indicating the gain control applied by the encoder, wherein the information indicating the gain control applied by the encoder includes the gain transition step size; and The inverse gain conversion function is applied to the frame of the downmixed audio signal. 28. The method of claim 27, wherein the method further comprises: The downmixed audio signal is upmixed to generate an upmixed audio signal, wherein the upmixed audio signal is suitable for rendering. 29. The method of claim 28, further comprising rendering the overmixed signal to generate rendered audio data. 30. The method of claim 29, further comprising playing back the rendered audio data using one or more of a loudspeaker or headphones. 31. The method according to any one of claims 27 to 30, wherein the information indicating the gain control applied by the encoder further includes information indicating a smoothing function. 32. The method according to any one of claims 27 to 31, wherein the inverse gain conversion function is determined by inverting the gain conversion function applied by the encoder. 33. The method according to any one of claims 27 to 32, wherein the inverse gain conversion function comprises an instantaneous portion and a steady-state portion. 34. The method of claim 33, wherein the length of the instantaneous portion is limited by the delay introduced by the codec used by the decoder. 35. An apparatus configured to implement the method of any one of claims 1-34. 36. A program comprising instructions that, when executed by a processing device, cause the processing device to perform the method according to any one of claims 1-34. 37. A storage medium for storing the program according to claim 36.