RU2783150C1 - Dynamic processing in devices with different playback functionalities - Google Patents

Abstract

FIELD: acoustics.

SUBSTANCE: invention relates to acoustics. The audio data processing method consists in obtaining by the control system and through the system of interfaces the configuration data of dynamic processing of individual speakers for each of the plurality of speakers of the listening environment. The configuration data contains a set of playback limit threshold values ​​for each speaker of the plurality of speakers. The control system determines the listening environment dynamics processing configuration data for a plurality of speakers, wherein the determination of the listening environment dynamics processing configuration data includes averaging the playback limitation thresholds over the plurality of speakers. Audio data is received by the control system through the system of interfaces, containing one or more audio signals and associated spatial data, the spatial data containing at least one of the channel data or spatial metadata. Performed by the dynamics processing control system on the audio data based on the configuration data of the dynamics processing of the listening environment to generate the processed audio data. The processed audio data is rendered by the control system for playback through a set of loudspeakers containing at least some of the plurality of loudspeakers. The rendered audio signals are transmitted through the interface system to a set of loudspeakers.

EFFECT: reducing unwanted shift in the spatial balance of the mix.

16 cl, 33 dwg

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority over Spanish Patent Application No. P201930702 filed July 30, 2019, US Provisional Application No. 62/971421 filed February 7, 2020, US Provisional Application No. 62/705410 filed June 25, 2020 U.S. Provisional Application No. 62/880115, filed July 30, 2019, and U.S. Provisional Application No. 62/705143, filed June 12, 2020, each of which is hereby incorporated by reference in its entirety.

FIELD OF TECHNOLOGY

The present invention relates to systems and methods for reproducing, and rendering for playback, audio data using some or all of the speakers in a set of speakers.

BACKGROUND OF THE INVENTION

Audio devices, including but not limited to smart audio devices, are widespread and are becoming common features in many homes. While existing systems and methods for controlling audio devices provide advantages, improved systems and methods would be desirable.

SYMBOLS AND TERMINOLOGY

Throughout this specification, including the claims, the terms "speaker" and "loudspeaker" are used interchangeably to refer to any sound-producing transducer (or set of transducers) driven by a single signal applied to a speaker. A typical set of headphones contains two speakers.

Throughout this specification, including the claims, the expression performing an operation "on" a signal or data (e.g., filtering, scaling, transforming, or applying a gain to a signal or data) is used broadly to mean performing an operation directly on the signal or data, or on processed version of the signal or data (for example, on the version of the signal that was subjected to pre-filtering or pre-processing before performing a specified operation on it).

Throughout this specification, including the claims, the term "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be called a decoder system, and a system containing such a subsystem (e.g., a system that generates X outputs in response to a number of inputs, in which the subsystem generates M inputs and the remaining X−M inputs signals are received from an external source), may also be referred to as a decoder system.

Throughout this specification, including the claims, the term "processor" is used broadly to refer to a system or device programmed or otherwise made (for example, using software or firmware) capable of performing operations on data (for example, , audio or video or other image data). Example processors include a field programmable gate array (or other configurable integrated circuit or chipset), a digital signal processor programmed and/or otherwise configured to perform pipelining on audio or other audio data, a general purpose programmable processor, or a computer, and programmable microprocessor integrated circuit or chipset.

Throughout this specification, including the claims, the term "connects" or "connected" is used to mean either direct or indirect connection. Thus, if a first device is connected to a second device, the connection may be through a direct connection or through an indirect connection through other devices or connections.

In the context of this document, the term "smart audio device" is used to refer to an intelligent device that is either a dedicated audio device or a virtual digital assistant (eg, a connected virtual digital assistant). A special purpose audio device is a device (for example, a television or mobile phone) containing or connected to at least one microphone (and optionally containing or connected to at least one speaker and/or at least one camera), and/or at least one speaker (also optionally containing or connected to at least one microphone), and performed largely or primarily to achieve one purpose. While a TV can usually play (or be considered capable of playing) audio from program material, in most cases a modern TV will run some kind of operating system that runs applications locally, including an application for watching TV. Likewise, audio input and output on a mobile phone can do a lot, but it is serviced by applications running on the phone. In this sense, a special purpose audio device comprising speaker(s) and microphone(s) is often configured to run a local application and/or service to directly use the speaker(s) and microphone(s). Some special purpose audio devices may be configured to group with each other in order to perform playback of audio data in a certain zone or user configurable area.

A virtual digital assistant (for example, a connected virtual digital assistant) is a device (for example, a smart speaker or an embedded voice assistant device) containing or connected to at least one microphone (and optionally containing or connected to at least one speaker and/or at least one camera) and which may enable the use of multiple devices (other than the virtual digital assistant) for applications that are in a sense available in the cloud or otherwise not implemented in or on the virtual digital assistant itself. Virtual digital assistants can sometimes work together, for example, in a very separate and conditional manner. For example, two or more virtual digital assistants can work together in the sense that one of them, i.e. the one who is most confident that he heard the wake-up word, responds to this word. The connected devices may form some kind of group that may be controlled by a single master application, which may be (or implement) a virtual digital assistant.

In the context of this document, the term "wake-up word" is used in a broad sense to refer to any sound (for example, a word spoken by a person, or some other sound), while the intelligent audio device is configured to wake up in response to detection ("audibility" ) sound (using at least one microphone contained in or connected to an intelligent audio device, or at least one other microphone). In this context, "waking up" means entering a state in which the device is waiting for (i.e., trying to hear) a sound command. In some cases, what may be referred to in this document as a "wake up word" may contain more than one word, such as a phrase.

As used herein, the term "wake-up word detector" refers to a device capable of (or software containing instructions for configuring the device) continuously searching for a match between real-time audio (eg, speech) features and a trained model. Typically, a wakeup word event is triggered whenever the wakeup word detector determines that the probability of detecting a wakeup word exceeds a predetermined threshold. For example, this threshold may be a predetermined threshold tuned to provide a satisfactory trade-off between false access and false reject rates. After a wake-up word event occurs, the device may enter a state (which may be referred to as a "wake" or "attention" state) in which it listens for a command and, upon receiving the command, moves to a larger, more computationally intensive recognizer.

SUMMARY OF THE INVENTION

Some embodiments include methods for rendering (or rendering and playing) a spatial audio mix (e.g., rendering an audio data stream or multiple audio data streams) for playback by at least one (e.g., all or some) of the smart audio devices of a set of smart audio devices and/or at least one (eg, all or some) of the speakers of the other set of speakers. Some embodiments are methods (or systems) for such rendering (eg, including generating signals to speakers) as well as playback of rendered audio data (eg, playback of generated signals to speakers).

One class of embodiments includes methods for rendering (or rendering and playing) audio data by at least one (eg, all or some) of a plurality of coordinated (organized) smart audio devices. For example, a set of smart audio devices present (on the system) in a user's home can be arranged to handle multiple use cases simultaneously, including flexible rendering of audio data for playback by all or some (i.e., speaker(s) enclosed in or connected to some or all) from smart audio devices.

Some embodiments of the present invention are systems and methods for processing audio data, which includes rendering audio data (e.g., rendering a spatial audio mix, e.g., by rendering an audio data stream or multiple audio data streams) for playback by at least two speakers (e.g., all or some from the speakers of the speaker set), and which includes:

(a) combining individual speaker dynamics processing configuration data (such as clipping thresholds (playback limitation thresholds) of individual speakers, whereby listening environment dynamics processing configuration data for a plurality of speakers (such as combined thresholds) is determined);

(b) performing dynamics processing on audio data (eg, audio data stream(s) reflecting a spatial audio mix) using listening environment dynamics configuration data for a plurality of speakers (eg, combined thresholds) to generate processed audio data; and

(c) rendering the processed audio data into signals fed to speakers.

In some embodiments, the processing of audio data includes:

(d) performing dynamics processing on rendered audio signals according to individual speaker dynamics configuration data for each speaker (e.g., clipping signals applied to speakers according to playback clipping thresholds associated with the respective speakers, whereby clipped signals to the speakers).

The speakers may be the speakers of at least one (eg, all or some) of the smart sound devices in the smart sound device set (or may be connected to at least one of the smart sound devices in the smart sound device set). In some implementations, to generate limited speaker signals in step (d), the speaker signals generated in step (c) may be processed by a second dynamic processing step (e.g., a dynamics processing system associated with each speaker) , for example, to generate clipped (i.e., dynamically clipped) signals to the speakers before they are finally played back through the speakers. For example, signals applied to the speakers (or a subset or portion thereof) may be delivered to the dynamics processing system of each individual speaker (e.g., to the dynamics processing subsystem of an intelligent audio device, wherein the intelligent audio device comprises a corresponding one of the speakers or is connected with it), and the processed audio output from each of said dynamics processing systems can be used to generate a clipped speaker signal (eg, a dynamic clipped speaker signal) for a corresponding one of the speakers. After speaker-specific dynamics processing (in other words, dynamics processing performed independently for each of the speakers), the processed (eg, dynamically clipped) signals applied to the speakers can be used to drive the speakers for sound reproduction.

The first stage of dynamics processing (in step (b)) can be configured to reduce the distracting shift in spatial balance that would otherwise occur if steps (a) and (b) were skipped and dynamically processed (e.g., clipped) signals generated to the loudspeaker that arise from step (d) in response to the original audio data (rather than in response to the processed audio data generated in step (b)). This can prevent unwanted shifts in the spatial balance of the mix. The second dynamics processing step in step (d) operating on the rendered speaker signals from step (c) may be configured to ensure that there is no speaker distortion since the dynamics processing in step (b) may optionally reduce signal levels below the thresholds for all loudspeakers. Combining the individual speaker dynamics configuration data (e.g., combining the thresholds in the first step (step (a)) in some examples may involve (e.g., include) an averaging step of the individual speaker dynamics processing configuration data (e.g., clipping thresholds) in the speakers (e.g., in smart audio devices) or taking the minimum value of individual speaker dynamics processing configuration data (eg clipping thresholds) in speakers (eg in smart audio devices).

In some implementations, when the first stage of dynamics processing (in step (b)) operates on audio data indicative of spatial mixing (e.g., audio data of an object-based audio program containing at least one object channel in the world, and optionally at least at least one speaker channel), this first step can be implemented in accordance with the technical solution for processing audio objects by using spatial zones. In this case, the combined individual speaker dynamics configuration data (e.g., combined clipping thresholds) associated with each of the zones can be obtained using (or as) a weighted average of the individual speaker dynamics configuration data (e.g., individual speaker clipping thresholds), and this weighting may be provided or determined at least in part by each speaker's spatial proximity to the zone and/or position within the zone.

In one class of embodiments, an audio rendering system may render at least one audio stream (e.g., a plurality of audio streams for simultaneous playback) and/or play the rendered stream(s) through a plurality of randomly placed speakers, wherein at least one (e.g., two or more) of the specified program streams represent (or define) a spatial mix.

Aspects of the present invention may include a system configured (eg, programmed) to perform one or more of the disclosed methods or steps thereof, and a tangible, non-volatile, computer-readable storage medium on which the data is permanently stored (eg, a disk or other tangible storage medium), and which stores code to execute (eg, executable code to execute) one or more of the disclosed methods or steps thereof. For example, some embodiments may be or comprise a general purpose programmable processor, digital signal processor, or microprocessor programmed using software or firmware and/or otherwise configured to perform any of a variety of operations on data, including one or more of the disclosed methods or steps thereof. Such a general purpose processor may be or comprise a computer system including an input device, a storage device, and a processing subsystem programmed to (and/or otherwise capable of) executing one or more of the disclosed methods (or steps thereof) in response to transmitted her data.

At least some aspects of the present invention may be implemented using techniques such as audio data processing techniques. In some cases, these methods may be at least partially implemented by the control system, such as those described herein. Some such methods include obtaining by the control system and through the interface system individual speaker dynamics processing configuration data for each of the plurality of speakers in the listening environment. In some cases, individual speaker dynamics processing configuration data for one or more speakers from a plurality of speakers may correspond to one or more functionality of one or more speakers. In some examples, the individual speaker dynamics processing configuration data comprises a set of individual speaker dynamics processing configuration data for each speaker of the plurality of speakers. Some such methods include determining by the control system the configuration data of the dynamic processing of the listening environment for a plurality of speakers. In some examples, the determination of the listening environment dynamics processing configuration data is based on a set of individual speaker dynamics processing configuration data for each speaker of a plurality of speakers.

Some such methods include receiving by the control system, via the system of interfaces, audio data containing one or more audio signals and associated spatial data. In some examples, the spatial data contains channel data and/or spatial metadata. Some such methods include performing dynamics processing on the audio data based on the configuration data of the listening environment dynamics processing to generate processed audio data by the control system. Some such methods include rendering, by the control system, the processed audio data for playback through a loudspeaker set comprising at least some of the plurality of loudspeakers in order to obtain rendered audio signals. Some such methods involve delivering rendered audio signals via an interface system to a set of speakers.

In some examples, the individual speaker dynamics configuration data may comprise a set of playback limit threshold data for each speaker of the plurality of speakers. The play limit threshold dataset may, for example, comprise play limit thresholds for each of the plurality of frequencies.

According to some examples, determining the listening environment dynamic processing configuration data may include determining minimum playback limit thresholds across a plurality of speakers. In some cases, the determination of the listening environment dynamic processing configuration data may include averaging playback limit thresholds over multiple speakers. In some examples, determining the listening environment dynamics processing configuration data may include averaging the play limit thresholds to obtain average play limit thresholds across a plurality of speakers, determining minimum play limit thresholds across a plurality of speakers, and interpolating between the minimum play limit thresholds and the average play limit thresholds. In some such examples, averaging the play restriction thresholds may include determining a weighted average of the play restriction thresholds. According to some implementations, the weighted average may be based at least in part on the characteristics of the rendering process implemented by the control system.

In some examples, performing dynamics processing on audio data may be based on spatial zones, with each of the spatial zones corresponding to a subset of the listening environment. According to some such examples, the weighted average of playback clipping thresholds may be based at least in part on speaker activation values by the rendering process depending on the proximity of the audio signal to the spatial regions. In some examples, the weighted average may be based at least in part on the speaker contribution value for each speaker in each of the spatial zones. According to some such examples, each speaker contribution value may be based at least in part on one or more nominal spatial positions in each of the spatial zones. In some such examples, the nominal spatial positions correspond to standard channel locations, such as standard channel locations in Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, or Dolby 9.1 surround mixing. In some cases, each speaker contribution value may be based at least in part on the activation value of each speaker corresponding to the rendering of audio data at each of the one or more nominal spatial positions in each of the spatial zones.

In some implementations, the method may also include performing dynamics on the rendered audio signals in accordance with individual speaker dynamics configuration data for each speaker in the set of speakers to which the rendered audio signals are delivered.

In some examples, rendering the processed audio data may include determining a relative speaker set activation value in accordance with one or more dynamically configurable functions. One or more dynamically configurable functions may be based on, for example, one or more audio signal properties, one or more speaker set properties, and/or one or more external input signals.

According to some implementations, performing dynamic processing on audio data may be based on spatial zones. Each of the spatial zones may correspond to a subset of the listening environment. In some such implementations, dynamic processing may be performed separately for each of the spatial regions. In some cases, the determination of the configuration data of the dynamic processing of the listening environment may be performed separately for each of the spatial zones.

In some examples, the individual speaker dynamics processing configuration data may comprise a set of dynamic range compression data for each speaker of the plurality of speakers. According to some such examples, a dynamic range compression data set may contain threshold data, input-to-output ratio data, rise data, attenuation data, and/or knee data.

According to some implementations, the determination of the listening environment dynamics processing configuration data may be based at least in part on a combination of dynamics processing configuration data sets across a plurality of speakers. In some examples, the combination of dynamic processing configuration data sets across multiple speakers may be based at least in part on the characteristics of the rendering process implemented by the control system.

In some such examples, performing dynamics processing on audio data may be based on one or more spatial regions. Each of the one or more spatial zones may correspond to the entire listening environment or a subset thereof. In some such examples, the combination of dynamic processing configuration data sets across multiple speakers may be performed separately for each of one or more spatial regions. In some such examples, combining sets of dynamic processing configuration data across multiple speakers separately for each of one or more spatial zones may be at least in part based on speaker activation values by the rendering process depending on the desired location of the audio signal in one or more spatial zones. .

According to some such examples, combining dynamic configuration data sets across multiple speakers separately for each of one or more spatial zones may be based at least in part on a speaker contribution value for each speaker in each of the one or more spatial zones. In some such examples, each speaker contribution value may be based at least in part on one or more nominal spatial positions in each of the one or more spatial zones. In some such examples, the nominal spatial positions may correspond to standard channel locations, such as standard channel locations in Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, or Dolby 9.1 surround mixing. In some cases, each speaker participation value may be at least partially based on the activation value of each speaker corresponding to the rendering of audio data at each of the one or more nominal spatial positions in each of the one or more spatial zones.

Some or all of the operations, functions, and/or methods described herein may be performed by one or more devices in accordance with instructions (eg, software) stored in one or more persistent storage media. Such persistent storage media may include storage devices such as those described herein, including, but not limited to, random access memory (RAM), read-only memory (ROM), etc. e. Accordingly, some of the inventive features of the subject matter described in the present invention may be implemented in a persistent storage medium containing software stored therein.

For example, the software may contain instructions for controlling one or more devices to perform a method including obtaining by the control system and through the interface system individual speaker dynamics processing configuration data for each of the plurality of speakers in the listening environment. In some cases, individual speaker dynamics processing configuration data for one or more speakers from a plurality of speakers may correspond to one or more functionality of one or more speakers. In some examples, the individual speaker dynamics processing configuration data comprises a set of individual speaker dynamics processing configuration data for each speaker of the plurality of speakers. Some such methods include determining by the control system the configuration data of the dynamic processing of the listening environment for a plurality of speakers. In some examples, the determination of the listening environment dynamics processing configuration data is based on a set of individual speaker dynamics processing configuration data for each speaker of a plurality of speakers.

Some such methods include receiving by the control system, via the system of interfaces, audio data containing one or more audio signals and associated spatial data. In some examples, the spatial data contains channel data and/or spatial metadata. Some such methods include performing dynamics processing on the audio data based on the configuration data of the listening environment dynamics processing to generate processed audio data by the control system. Some such methods include rendering, by the control system, the processed audio data for playback through a loudspeaker set comprising at least some of the plurality of loudspeakers in order to obtain rendered audio signals. Some such methods involve delivering rendered audio signals via an interface system to a set of speakers.

In some examples, the individual speaker dynamics configuration data may comprise a set of playback limit threshold data for each speaker of the plurality of speakers. The play limit threshold dataset may, for example, comprise play limit thresholds for each of the plurality of frequencies.

According to some examples, determining the listening environment dynamic processing configuration data may include determining minimum playback limit thresholds across a plurality of speakers. In some cases, the determination of the listening environment dynamic processing configuration data may include averaging playback limit thresholds over multiple speakers. In some examples, determining the listening environment dynamics processing configuration data may include averaging the play limit thresholds to obtain average play limit thresholds across a plurality of speakers, determining minimum play limit thresholds across a plurality of speakers, and interpolating between the minimum play limit thresholds and the average play limit thresholds. In some such examples, averaging the play restriction thresholds may include determining a weighted average of the play restriction thresholds. According to some implementations, the weighted average may be based at least in part on the characteristics of the rendering process implemented by the control system.

In some examples, performing dynamics processing on audio data may be based on spatial zones, with each of the spatial zones corresponding to a subset of the listening environment. According to some such examples, the weighted average of playback clipping thresholds may be based at least in part on speaker activation values by the rendering process depending on the proximity of the audio signal to the spatial regions. In some examples, the weighted average may be based at least in part on the speaker contribution value for each speaker in each of the spatial zones. According to some such examples, each speaker contribution value may be based at least in part on one or more nominal spatial positions in each of the spatial zones. In some such examples, the nominal spatial positions correspond to standard channel locations, such as standard channel locations in Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, or Dolby 9.1 surround mixing. In some cases, each speaker contribution value may be based at least in part on the activation value of each speaker corresponding to the rendering of audio data at each of the one or more nominal spatial positions in each of the spatial zones.

In some implementations, the method may also include performing dynamics on the rendered audio signals in accordance with individual speaker dynamics configuration data for each speaker in the set of speakers to which the rendered audio signals are delivered.

In some examples, rendering the processed audio data may include determining a relative speaker set activation value in accordance with one or more dynamically configurable functions. One or more dynamically configurable functions may be based on, for example, one or more audio signal properties, one or more speaker set properties, and/or one or more external input signals.

According to some implementations, performing dynamics processing on audio data may be based on spatial zones. Each of the spatial zones may correspond to a subset of the listening environment. In some such implementations, dynamic processing may be performed separately for each of the spatial zones. In some cases, the determination of the configuration data of the dynamic processing of the listening environment may be performed separately for each of the spatial zones.

In some examples, the individual speaker dynamics processing configuration data may comprise a set of dynamic range compression data for each speaker of the plurality of speakers. According to some such examples, a dynamic range compression data set may contain threshold data, input-to-output ratio data, rise data, attenuation data, and/or knee data.

According to some implementations, the determination of the listening environment dynamics processing configuration data may be based at least in part on a combination of dynamics processing configuration data sets over a plurality of speakers. In some examples, the combination of dynamic processing configuration data sets across multiple speakers may be based at least in part on the characteristics of the rendering process implemented by the control system.

In some such examples, performing dynamics processing on audio data may be based on one or more spatial regions. Each of the one or more spatial zones may correspond to the entire listening environment or a subset thereof. In some such examples, the combination of dynamic processing configuration data sets across multiple speakers may be performed separately for each of one or more spatial regions. In some such examples, combining sets of dynamic processing configuration data across multiple speakers separately for each of one or more spatial zones may be at least in part based on speaker activation values by the rendering process depending on the desired location of the audio signal in one or more spatial zones. .

According to some such examples, combining dynamic configuration data sets across multiple speakers separately for each of one or more spatial zones may be based at least in part on a speaker contribution value for each speaker in each of the one or more spatial zones. In some such examples, each speaker contribution value may be based at least in part on one or more nominal spatial positions in each of the one or more spatial zones. In some such examples, the nominal spatial positions may correspond to standard channel locations, such as standard channel locations in Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, or Dolby 9.1 surround mixing. In some cases, each speaker participation value may be at least partially based on the activation value of each speaker corresponding to the rendering of audio data at each of the one or more nominal spatial positions in each of the one or more spatial zones.

In some implementations, the apparatus may include an interface system and a control system. The control system may contain one or more single- or multi-chip general purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs). ) or other programmable logic devices, discrete or transistorized logic circuits, discrete hardware components, and/or combinations thereof.

In some implementations, the control system may be configured to perform one or more of the methods disclosed herein. Some such methods may include obtaining, by the control system and through the system interfaces, individual speaker dynamics processing configuration data for each of the plurality of speakers in the listening environment. In some cases, individual speaker dynamics processing configuration data for one or more speakers from a plurality of speakers may correspond to one or more functionality of one or more speakers. In some examples, the individual speaker dynamics processing configuration data comprises a set of individual speaker dynamics processing configuration data for each speaker of the plurality of speakers. Some such methods include determining by the control system the configuration data of the dynamic processing of the listening environment for a plurality of speakers. In some examples, the determination of the listening environment dynamics processing configuration data is based on a set of individual speaker dynamics processing configuration data for each speaker of a plurality of speakers.

Some such methods include receiving by the control system, via the system of interfaces, audio data containing one or more audio signals and associated spatial data. In some examples, the spatial data contains channel data and/or spatial metadata. Some such methods include performing dynamics processing on the audio data based on the configuration data of the listening environment dynamics processing to generate processed audio data by the control system. Some such methods include rendering, by the control system, the processed audio data for playback through a loudspeaker set comprising at least some of the plurality of loudspeakers in order to obtain rendered audio signals. Some such methods involve delivering rendered audio signals via an interface system to a set of speakers.

In some examples, the individual speaker dynamics configuration data may comprise a set of playback limit threshold data for each speaker of the plurality of speakers. The play limit threshold dataset may, for example, comprise play limit thresholds for each of the plurality of frequencies.

According to some examples, determining the listening environment dynamic processing configuration data may include determining minimum playback limit thresholds across a plurality of speakers. In some cases, the determination of the listening environment dynamic processing configuration data may include averaging playback limit thresholds over multiple speakers. In some examples, determining the listening environment dynamics processing configuration data may include averaging the play limit thresholds to obtain average play limit thresholds across a plurality of speakers, determining minimum play limit thresholds across a plurality of speakers, and interpolating between the minimum play limit thresholds and the average play limit thresholds. In some such examples, averaging the play restriction thresholds may include determining a weighted average of the play restriction thresholds. According to some implementations, the weighted average may be based at least in part on the characteristics of the rendering process implemented by the control system.

In some examples, performing dynamics processing on audio data may be based on spatial zones, with each of the spatial zones corresponding to a subset of the listening environment. According to some such examples, the weighted average of playback clipping thresholds may be based at least in part on speaker activation values by the rendering process depending on the proximity of the audio signal to the spatial regions. In some examples, the weighted average may be based at least in part on the speaker contribution value for each speaker in each of the spatial zones. According to some such examples, each speaker contribution value may be based at least in part on one or more nominal spatial positions in each of the spatial zones. In some such examples, the nominal spatial positions correspond to standard channel locations, such as standard channel locations in Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, or Dolby 9.1 surround mixing. In some cases, each speaker contribution value may be based at least in part on the activation value of each speaker corresponding to the rendering of audio data at each of the one or more nominal spatial positions in each of the spatial zones.

In some implementations, the method may also include performing dynamics on the rendered audio signals in accordance with individual speaker dynamics configuration data for each speaker in the set of speakers to which the rendered audio signals are delivered.

In some examples, rendering the processed audio data may include determining a relative speaker set activation value in accordance with one or more dynamically configurable functions. One or more dynamically configurable functions may be based on, for example, one or more audio signal properties, one or more speaker set properties, and/or one or more external input signals.

According to some implementations, performing dynamic processing on audio data may be based on spatial zones. Each of the spatial zones may correspond to a subset of the listening environment. In some such implementations, dynamic processing may be performed separately for each of the spatial regions. In some cases, the determination of the configuration data of the dynamic processing of the listening environment may be performed separately for each of the spatial zones.

In some examples, the individual speaker dynamics processing configuration data may comprise a set of dynamic range compression data for each speaker of the plurality of speakers. According to some such examples, a dynamic range compression data set may contain threshold data, input-to-output ratio data, rise data, attenuation data, and/or knee data.

According to some implementations, the determination of the listening environment dynamics processing configuration data may be based at least in part on a combination of dynamics processing configuration data sets over a plurality of speakers. In some examples, the combination of dynamic processing configuration data sets across multiple speakers may be based at least in part on the characteristics of the rendering process implemented by the control system.

In some such examples, performing dynamics processing on audio data may be based on one or more spatial regions. Each of the one or more spatial zones may correspond to the entire listening environment or a subset thereof. In some such examples, the combination of dynamic processing configuration data sets across multiple speakers may be performed separately for each of one or more spatial regions. In some such examples, combining sets of dynamic processing configuration data across multiple speakers separately for each of one or more spatial zones may be at least in part based on speaker activation values by the rendering process depending on the desired location of the audio signal in one or more spatial zones. .

According to some such examples, combining dynamic configuration data sets across multiple speakers separately for each of one or more spatial zones may be based at least in part on a speaker contribution value for each speaker in each of the one or more spatial zones. In some such examples, each speaker contribution value may be based at least in part on one or more nominal spatial positions in each of the one or more spatial zones. In some such examples, the nominal spatial positions may correspond to standard channel locations, such as standard channel locations in Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, or Dolby 9.1 surround mixing. In some cases, each speaker participation value may be at least partially based on the activation value of each speaker corresponding to the rendering of audio data at each of the one or more nominal spatial positions in each of the one or more spatial zones.

Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and in the description below. Other features, aspects and advantages will be apparent from the description, drawings and claims. It should be noted that the relative dimensions in the following figures may not be drawn to scale.

BRIEF DESCRIPTION OF GRAPHICS

In FIG. 1 is a block diagram showing examples of components of a device capable of implementing various aspects of the present invention.

In FIG. 2 is a plan view of a listening environment, which in this example is a living area.

In FIG. 3 is a block diagram showing examples of system components capable of implementing various aspects of the present invention.

In FIG. 4A, 4B and 4C show examples of playback clipping thresholds and corresponding frequencies.

In FIG. 5A and 5B are graphs showing examples of dynamic range compression data.

In FIG. 6 shows an example of the spatial zones of the listening environment.

In FIG. 7 shows examples of loudspeakers in the spatial zones of FIG. 6.

In FIG. 8 shows an example of nominal spatial positions superimposed on the spatial zones and dynamics of FIG. 7.

In FIG. 9 is a flow diagram that describes one example of a method that can be performed by such a device or system as those disclosed herein.

In FIG. 10 and 11 are diagrams depicting an exemplary set of speaker activation values and object rendering positions.

In FIG. 12A, 12B and 12C show examples of speaker contribution values corresponding to the examples of FIG. 10 and 11.

In FIG. 13 is a graph of speaker activation values in an exemplary embodiment.

In FIG. 14 is a graph of object rendering positions in an exemplary embodiment.

In FIG. 15A, 15B and 15C show examples of speaker participation values corresponding to the examples of FIG. 13 and 14.

In FIG. 16 is a graph of speaker activation values in an exemplary embodiment.

In FIG. 17 is a graph of object rendering positions in an exemplary embodiment.

In FIG. 18A, 18B and 18C show examples of speaker contribution values corresponding to the examples of FIG. 16 and 17.

In FIG. 19 is a graph of speaker activation values in an exemplary embodiment.

In FIG. 20 is a graph of object rendering positions in an exemplary embodiment.

In FIG. 21A, 21B and 21C show examples of speaker contribution values corresponding to the examples of FIG. 19 and 20.

In FIG. 22 is a diagram of an environment, which in this example is a living area.

Like reference numerals and symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION OF EMBODIMENTS

In FIG. 1 is a block diagram showing examples of components of a device capable of implementing various aspects of the present invention. As with other figures presented herein, the types and numbers of elements shown in FIG. 1 are presented by way of example only. Other implementations may contain more, fewer elements, and/or different types and numbers of elements. According to some examples, device 100 may be or include an intelligent audio device configured to perform at least some of the methods disclosed herein. In some implementations, device 100 may be or may include another device capable of performing at least some of the methods disclosed herein, such as a laptop, cell phone, tablet device, smart home hub, etc. In some in such implementations, device 100 may be or include a server.

In this example, device 100 includes an interface system 105 and a control system 110. In some implementations, interface system 105 may be configured to receive audio data. The audio data may comprise audio signals scheduled for playback by at least some of the speakers in the environment. The audio data may comprise one or more audio signals and associated spatial data. The spatial data may include, for example, channel data and/or spatial metadata. The interface system 105 may be configured to deliver rendered audio signals to at least some of the speakers in the environment speaker set. In some implementations, interface system 105 may be configured to receive input from one or more environment microphones.

Interface system 105 may include one or more network interfaces and/or one or more interfaces for external devices (such as one or more universal serial bus (USB) interfaces). In some implementations, interface system 105 may include one or more wireless interfaces. The interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and/or a gesture sensor system. In some examples, interface system 105 may include one or more interfaces between control system 110 and a memory system, such as the optional memory system 115 shown in FIG. 1. However, in some cases, the control system 110 may include a memory system.

For example, control system 110 may include a single or multi-chip general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, a discrete circuit, or a transistor logic circuitry, and/or discrete hardware components.

In some implementations, control system 110 may reside in more than one device. For example, one part of the control system 110 may reside in a device in one of the environments depicted herein, and another portion of the control system 110 may reside in a device outside of that environment, such as a server, a mobile device (e.g., a smartphone or tablet computer), etc. In other examples, one part of the control system 110 may reside in a device in one of the environments depicted herein, and another portion of the control system 110 may reside in one or more other devices in that environment. For example, the functionality of the control system may be distributed across multiple smart audio devices in an environment, or may be split between an organizing device (such as what may be referred to herein as a smart home hub) and one or more devices in that environment. In some such examples, interface system 105 may also reside in more than one device.

In some implementations, control system 110 may be configured to perform at least in part the methods disclosed herein. According to some examples, control system 110 may be configured to implement methods for controlling playback of multiple audio data streams through multiple speakers.

Some or all of the methods described herein may be performed by one or more devices in accordance with instructions (eg, software) stored on one or more permanent storage media. Such read-only storage media may include storage media such as those described herein, including, but not limited to, random access memory (RAM), read-only memory (ROM), etc. One or more persistent storage media may reside, for example, in the optional memory system 115 shown in FIG. 1 and/or in the control system 110. Accordingly, various innovative aspects of the subject matter of the present invention may be implemented in one or more persistent storage media containing software stored thereon. For example, the software may contain instructions for controlling at least one device in order to process audio data. For example, the software may be executable by one or more components of a control system, such as control system 110 of FIG. one.

In some examples, device 100 may include the optional microphone array 120 shown in FIG. 1. The optional microphone system 120 may include one or more microphones. In some implementations, one or more microphones may be part of or associated with another device such as a speaker in a speaker system, smart audio device, etc.

In some implementations, device 100 may include an optional speaker system 125 shown in FIG. 1. Optional speaker system 125 may include one or more speakers. Loudspeakers may sometimes be referred to as "speakers" in this document. In some examples, at least some of the speakers of the optional speaker system 125 may be randomly positioned. For example, at least some of the speakers of the optional speaker system 125 may be placed in locations that do not conform to any standard prescribed speaker layout, such as Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, Dolby 9.1, Hamasaki 22.2, etc. In some such examples, at least some of the speakers of the optional speaker system 125 may be placed in locations that are convenient for a given space (e.g., locations where there is space for speaker placement) rather than in any speaker layout prescribed by the standard.

In some implementations, device 100 may include the optional sensor system 130 shown in FIG. 1. Optional sensor system 130 may include one or more cameras, touch sensors, gesture sensors, motion sensors, etc. In some implementations, optional sensor system 130 may include one or more cameras. In some implementations, the cameras may be stand-alone cameras. In some examples, one or more cameras of the optional sensor system 130 may reside in an intelligent audio device, which may be a special purpose audio device or virtual digital assistant. In some such examples, one or more cameras of the optional sensor system 130 may reside in a television, mobile phone, or smart speaker.

In some implementations, device 100 may include an optional display system 135 shown in FIG. 1. The optional display system 135 may comprise one or more displays, such as one or more light-emitting diode (LED) displays. In some cases, the optional display system 135 may include one or more organic light-emitting diode (OLED) displays. In some examples where device 100 includes display system 135, sensor system 130 may include a touch sensor system and/or gesture sensor system in the vicinity of one or more displays of display system 135. According to some such implementations, control system 110 may be configured to control display system 135 to provide a graphical user interface (GUI), such as one of the GUIs disclosed herein.

According to some examples, device 100 may be or may include an intelligent audio device. In some such implementations, device 100 may be or may include a wake-up word detector. For example, device 100 may be or may include a virtual digital assistant.

In FIG. 2 is a plan view of a listening environment, which in this example is a living area. As with other figures presented herein, the types and numbers of elements shown in FIG. 2 are presented by way of example only. Other implementations may contain more, fewer elements, and/or different types and numbers of elements. In this example, environment 200 includes a living room 210 at the top left, a kitchen 215 at the bottom center, and a bedroom 222 at the bottom right. The rectangles and circles distributed throughout the living area represent a set of speakers 205a-205h, at least some of which, in some implementations, may be smart speakers placed in locations convenient to that space, but not bound to any prescribed circuitry by the standard. placement (arbitrarily placed). In some examples, speakers 205a-205h may be coordinated to implement one or more of the disclosed embodiments.

According to some examples, environment 200 may include a smart home hub for implementing at least some of the disclosed methods. According to some such implementations, a smart home hub may include at least a portion of the control system 110 described above. In some examples, the smart home hub may be implemented by a smart device (such as a smart speaker, mobile phone, smart TV, a device used to implement a virtual digital assistant, etc.).

In this example, environment 200 contains cameras 211a-211e that are distributed throughout the environment. In some implementations, one or more smart audio devices in environment 200 may also include one or more cameras. One or more intelligent audio devices may be special purpose audio devices or virtual digital assistants. In some such examples, one or more cameras of the optional sensor system 130 may reside in or on the television 230, in a mobile phone, or in a smart speaker such as one or more speakers 205b, 205d, 205e, and 205h. Although cameras 211a-211e are not shown in every image of environment 200 presented herein, each of environments 200 may nonetheless contain one or more cameras in some implementations.

With flexible rendering, rendering of spatial audio data can be performed for an arbitrary number of randomly placed speakers. Given the widespread use of smart audio devices (eg, smart speakers) in the home, there is a need to implement a flexible rendering technology that allows consumers to perform flexible rendering of audio data and play the rendered audio data using smart audio devices.

Several technologies have been developed to implement flexible rendering, including: Center of Mass Amplitude Panning (CMAP) and Flexible Virtualization (FV).

In the context of rendering (or rendering and playing) a spatial audio mix (e.g., rendering an audio data stream or multiple audio data streams) for playback by smart sound devices in a smart sound device set (or other set of speakers), speaker types (e.g., in smart sound devices or connected to intelligent audio devices) can vary and therefore the corresponding acoustic functionality of the speakers can vary quite significantly. In the example shown in FIG. 2, the speakers 205d, 205f and 205h are smart speakers with a single 0.6 inch driver. In this example, the speakers 205b, 205c, 205e, and 205f are smart speakers comprising a 2.5" woofer and a 0.8" tweeter. In this example, the speaker 205g is a smart speaker, with a 5.25" woofer, three 2" midrange drivers, and a 1.0" tweeter. Here, loudspeaker 205a is a soundbar containing sixteen 1.1" diameter drivers and two 4" diameter woofers. Accordingly, the low frequency functionality of the smart speakers 205d and 205f is significantly less than other speakers in the environment 200, in particular, speakers containing 4" or 5.25" woofers.

In FIG. 3 is a block diagram showing examples of system components capable of implementing various aspects of the present invention. As with other figures presented herein, the types and numbers of elements shown in FIG. 1 are presented by way of example only. Other implementations may contain more, fewer elements, and/or different types and numbers of elements.

In this example, system 300 includes a smart home hub 305 and speakers 205a-205m. In this example, the smart home hub 305 contains an example of the control system 110 shown in FIG. 1 and described above. According to this implementation, the control system 110 includes a listening environment dynamic configuration data module 310, a listening environment dynamic processing module 315, and a rendering module 320 . Some examples of listening environment dynamic processing module 310, listening environment dynamic processing module 315, and rendering module 320 are described below. In some examples, renderer 320' may be configured to both render and dynamically process the listening environment.

As shown by the arrows between the smart home hub 305 and the speakers 205a-205m, the smart home hub 305 also contains an example of the interface system 105 shown in FIG. 1 and described above. In some examples, smart home hub 305 may be part of the environment 200 shown in FIG. 2. In some cases, the smart home hub 305 may be implemented with a smart speaker, smart TV, cell phone, laptop, etc. In some implementations, the smart home hub 305 may be implemented with software, such as software. from a downloadable software application, or "app". In some cases, smart home hub 305 may be implemented in each of the speakers 205a-m, which operate in parallel to generate the same processed audio signals from module 320. According to some such examples, in each of the speakers, renderer 320 may then generate one or more signals applied to the speakers that correspond to each speaker or group of speakers, and can deliver these signals applied to the speakers to the dynamics processing module of each speaker.

In some cases, speakers 205a-205m may include speakers 205a-205h of FIG. 2, while in other examples the speakers 205a-205m may be or include other speakers. Accordingly, in this example, system 300 contains M speakers, where M is an integer greater than 2.

Smart speakers, as well as many other active speakers, typically use some type of internal dynamics processing to prevent speaker distortion. Associated with such dynamic processing are often signal clipping thresholds (eg, clipping thresholds that are frequency-variable), below which the signal level is maintained dynamically. For example, Dolby's Audio Regulator, one of several Dolby Audio Processing (DAP) audio post-processing software algorithms, provides such processing. In some cases, but not usually in the case of a smart speaker dynamics module, dynamics processing may also involve the use of one or more compressors, valves, expanders, duckers, etc.

Accordingly, in this example, each of the speakers 205a-205m contains respective speaker dynamics processing (DP) modules A-M. These speaker dynamics modules are configured to apply individual speaker dynamics processing configuration data to each individual speaker of the listening environment. The speaker DP module A, for example, is configured to apply the dynamics processing configuration data of the individual speakers corresponding to the speaker 205a. In some examples, individual speaker dynamics processing configuration data may correspond to one or more individual speaker functionality, such as the speaker's ability to reproduce audio data at a particular frequency range and at a particular level without significant distortion.

When rendering spatial audio data across a set of heterogeneous speakers (e.g., speakers of smart audio devices or speakers connected to smart audio devices), each with potentially different playback limitations, care must be taken to perform dynamics processing on the entire mix. A simple solution is to render the spatial mix into speaker signals for each of the participating speakers, and then allow the dynamics module associated with each speaker to operate independently with respect to its corresponding speaker signal, according to that speaker's constraints.

While this approach will prevent distortion of each speaker, it can dynamically shift the spatial balance of the mix in a distracting way. For example, with reference to FIG. 2, suppose a television program is shown on television 230, and that the corresponding audio data is played back by the speakers of environment 200. Assume that the television program is scheduled to render audio data associated with a stationary object (such as a piece of heavy equipment in a factory) at position 244. Additionally Assume that the dynamics module associated with speaker 205d reduces the level of audio data in the low frequency range to a much greater extent than the dynamics module associated with speaker 205b due to the significantly greater functionality of the speaker 205b to reproduce sounds in the low frequency range. If the volume of the signal associated with a stationary object fluctuates, then when the volume becomes higher, the dynamics module associated with the loudspeaker 205d will cause the audio data in the low frequency range to be lowered to a much greater extent than when the same audio data is lowered by the dynamics module. processing associated with speaker 205b. This difference in levels will cause a change in the apparent location of a stationary object. Therefore, an improved solution is required.

Some embodiments of the present invention are systems and methods for rendering (or rendering and playing) a spatial audio mix (e.g., rendering an audio data stream or multiple audio data streams) for playback by at least one (e.g., all or some) of the smart audio devices in a set of smart audio devices. audio devices (eg, a set of coordinated smart audio devices) and/or at least one (eg, all or some) of the speakers of another set of speakers. Some embodiments are methods (or systems) for such rendering (eg, including generating signals to speakers) as well as playback of rendered audio data (eg, playback of generated signals to speakers). Examples of such embodiments include the following.

Systems and methods for processing audio data may include rendering audio data (e.g., rendering a spatial audio mix, e.g., by rendering an audio data stream or multiple audio data streams) for playback by at least two speakers (e.g., all or some of the speakers in a set of speakers), which includes :

(a) combining individual speaker dynamics processing configuration data (such as clipping thresholds (playback limiting thresholds) of individual speakers, whereby listening environment dynamics processing configuration data for a plurality of speakers (such as combined thresholds) is determined);

(b) performing dynamics processing on audio data (eg, audio data stream(s) reflecting a spatial audio mix) using listening environment dynamics configuration data for a plurality of speakers (eg, combined thresholds) to generate processed audio data; and

(c) rendering the processed audio data into signals fed to speakers.

In some implementations, step (a) may be performed by a module such as the listening environment dynamic configuration data processing module 310 shown in FIG. 3. The smart home hub 305 may be configured to receive individual speaker dynamic processing configuration data for each of the M speakers via the interface system. In this implementation, the individual speaker dynamics configuration data comprises a set of individual speaker dynamics configuration data for each speaker of the plurality of speakers. According to some examples, individual speaker dynamics processing configuration data for one or more speakers may correspond to one or more functionality of one or more speakers. In this example, each of the individual speaker dynamics configuration data sets contains at least one type of dynamics configuration data. In some examples, smart home hub 305 may be configured to obtain individual speaker dynamics configuration data sets by querying each of the speakers 205a-205m. In other implementations, smart home hub 305 may be configured to obtain individual speaker dynamics configuration data sets by querying a data structure from previously obtained individual speaker dynamics configuration data sets stored in a storage device.

In some examples, step (b) may be performed by a module such as the listening environment dynamics module 315 of FIG. 3. Some detailed examples of steps (a) and (b) are described below.

In some examples, rendering in accordance with step (c) may be performed by a module such as renderer 320 or renderer 320' of FIG. 3. In some embodiments, the processing of audio data may include:

(d) performing dynamics processing on rendered audio signals according to individual speaker dynamics configuration data for each speaker (e.g., clipping signals applied to speakers according to playback clipping thresholds associated with the respective speakers, whereby clipped signals to the speakers). Step (d) may be performed, for example, by the dynamics modules A-M shown in FIG. 3.

The speakers may include speakers from at least one (eg, all or some) of the smart sound devices in the smart sound device set (or may be connected to at least one of the smart sound devices in the smart sound device set). In some implementations, to generate limited speaker signals in step (d), the speaker signals generated in step (c) may be processed by a second dynamic processing step (e.g., a dynamics processing system associated with each speaker) , for example, to generate signals to the speakers before they are finally played back through the speakers. For example, signals applied to the speakers (or a subset or portion thereof) may be delivered to the dynamics processing system of each individual speaker (e.g., to the dynamics processing subsystem of an intelligent audio device, wherein the intelligent audio device comprises a corresponding one of the speakers or is connected with it), and the processed audio output from each of said dynamics processing systems can be used to generate a speaker signal for a corresponding one of the speakers. After speaker-specific dynamics processing (in other words, dynamics processing performed independently for each of the speakers), the processed (eg, dynamically clipped) signals applied to the speakers can be used to drive the speakers for sound reproduction.

The first stage of dynamics processing (in step (b)) can be configured to reduce the distracting shift in spatial balance that would otherwise occur if steps (a) and (b) were skipped and dynamically processed (e.g., clipped) signals generated to the loudspeaker that arise from step (d) in response to the original audio data (rather than in response to the processed audio data generated in step (b)). This can prevent unwanted shifts in the spatial balance of the mix. The second dynamics processing step, operating on the rendered speaker signals from step (c), may be configured to ensure no speaker distortion, since the dynamics processing in step (b) may optionally reduce signal levels below thresholds for all speakers. Combining the individual speaker dynamics configuration data (e.g., combining the thresholds in the first step (step (a)) in some examples may involve (e.g., include) an averaging step of the individual speaker dynamics processing configuration data (e.g., clipping thresholds) in the speakers (e.g., in smart audio devices) or taking the minimum value of individual speaker dynamics processing configuration data (eg clipping thresholds) in speakers (eg in smart audio devices).

In some implementations, when the first stage of dynamics processing (in step (b)) operates on audio data indicative of spatial mixing (e.g., audio data of an object-based audio program containing at least one object channel in the world, and optionally at least at least one speaker channel), this first step can be implemented in accordance with the technical solution for processing audio objects by using spatial zones. In this case, the combined individual speaker dynamics configuration data (e.g., combined clipping thresholds) associated with each of the zones can be obtained using (or as) a weighted average of the individual speaker dynamics configuration data (e.g., individual speaker clipping thresholds), and this weighting may be provided or determined at least in part by each speaker's spatial proximity to the zone and/or position within the zone.

), в котором каждый динамик индексирован с помощью переменной i. С каждым динамиком i связан набор переменных по частоте порогов ограничения воспроизведения,

, где переменная f отображает индекс в конечное множество частот, при которых определены эти пороги. (Следует отметить, что если размер множества частот равен единице, то соответствующий единственный порог можно считать широкополосным, применимым по всему диапазону частот). Эти пороги используются каждым динамиком в его собственной, независимой функции динамической обработки для ограничения звукового сигнала ниже порогов

для конкретной цели, такой как предотвращение искажения динамика или воспроизведения динамиком за пределами некоторого уровня, который считается недопустимым вблизи него.In an exemplary embodiment, a set of M speakers is assumed (

) in which each speaker is indexed by the variable i . Each speaker i is associated with a set of variables for the frequency of playback limitation thresholds,

, where the variable f maps the index to a finite set of frequencies at which these thresholds are defined. (It should be noted that if the size of the set of frequencies is equal to one, then the corresponding single threshold can be considered wideband, applicable over the entire frequency range). These thresholds are used by each speaker in its own independent dynamics processing function to limit the audio signal below the thresholds.

for a specific purpose, such as preventing speaker distortion or speaker playback beyond some level that is considered unacceptable in its vicinity.

In FIG. 4A, 4B and 4C show examples of playback clipping thresholds and corresponding frequencies. For example, the presented frequency range may cover the range of frequencies audible to the average person (eg, 20 Hz to 20 kHz). In these examples, playback limit thresholds are indicated by the vertical axes of graphs 400a, 400b, and 400c, which are labeled "Level Threshold" in these examples. The playback limit/level thresholds increase in the direction of the arrows on the vertical axes. Reproduction limit/level thresholds may be expressed in decibels, for example. In these examples, the horizontal axes of the plots 400a, 400b, and 400c indicate frequencies that increase in the direction of the arrows on the horizontal axes. The playback limit thresholds indicated by curves 400a, 400b and 400c may be implemented, for example, by individual loudspeaker dynamics modules.

In graph 400a of FIG. 4A shows a first example of a playback limitation threshold depending on frequency. Curve 405a indicates the playback limitation threshold for each respective frequency value. In this example, at a low frequency f _b , the input audio data received at the input level T _i will be output by the dynamics module at the output level T _o . For example, the low frequency f _b may be in the range of 60 to 250 Hz. However, in this example, at a high frequency f _b , the input audio data received at the input level T _i will be output by the dynamics module at the same level, the input level T _i . For example, the high frequency f _b may be in the range above 1280 Hz. Accordingly, in this example, curve 405a corresponds to a dynamics module that applies a significantly lower threshold for low frequencies than for high frequencies. Such a dynamics processing module may be suitable for a speaker that does not include a subwoofer (eg, speaker 205d of FIG. 2).

In graph 400b of FIG. 4B shows a second example of a playback limitation threshold versus frequency. Curve 405b indicates that at the same low frequency f _b shown in FIG. 4A, input audio data received at an input level T _i will be output by the dynamics module at a higher output level T _o . Accordingly, in this example, curve 405b corresponds to a dynamics module that does not apply the same low threshold for low frequencies as curve 405a. Such a dynamics processing module may be suitable for a loudspeaker containing at least a small subwoofer (eg, loudspeaker 205b of FIG. 2).

In graph 400c of FIG. 4C shows a second example of a playback limiting threshold versus frequency. Curve 405c (which is a straight line in this example) indicates that at the same low frequency f _b shown in FIG. 4A, the input audio data received at the input level T _i will be output by the dynamics processing unit at the same level. Accordingly, in this example, curve 405c corresponds to a dynamic processing unit that may be suitable for a loudspeaker capable of reproducing a wide range of frequencies, including low frequencies. It can be observed that, for simplicity, the dynamics module may approximate curve 405c by implementing curve 405d that applies the same threshold for all specified frequencies.

. В отсутствие преимуществ настоящего изобретения, этот описанный сценарий рендеринга может приводить к отвлекающим сдвигам в воспринимаемом пространственном балансе подвергнутого рендерингу пространственного звукового микса. Например, один из M динамиков, предположим на правой стороне области прослушивания, может быть менее функциональным, чем другие (например, при рендеринге аудиоданных в низкочастотном диапазоне), и поэтому пороги

для этого динамика могут быть значительно более низкими, чем те, что для других динамиков, по меньшей мере в конкретном диапазоне частот. В ходе воспроизведения модуль динамической обработки этого динамика будут понижать уровень составляющих пространственного микширования на правой стороне в значительно большей степени, чем составляющих на левой стороне. Слушатели являются чрезвычайно чувствительными к таким динамическим сдвигам между левым/правым балансом пространственного микширования и могут находить результаты весьма отвлекающими.The spatial audio mix can be rendered for multiple speakers using a known rendering system such as Center of Mass Amplitude Panning (CMAP) or Flexible Virtualization (FV). For the component parts of the spatial audio mix, the rendering system generates signals to the speakers, one for each of the multiple speakers. In some of the previous examples, the signals applied to the speakers were then independently processed by the dynamics processing function associated with each speaker using thresholds.

. In the absence of the benefits of the present invention, this described rendering scenario can lead to distracting shifts in the perceived spatial balance of the rendered spatial audio mix. For example, one of the M speakers, let's say on the right side of the listening area, may be less functional than the others (for example, when rendering audio data in the low frequency range), and therefore the thresholds

for this, the speakers can be significantly lower than those for other speakers, at least in a particular frequency range. During playback, the dynamics processing module of this speaker will lower the level of the spatial mixing components on the right side to a much greater extent than the components on the left side. Listeners are extremely sensitive to such dynamic shifts between left/right spatial mixing balance and may find the results quite distracting.

To solve this problem, in some examples, individual speaker dynamics configuration data (eg, play limit thresholds) for individual listening environment speakers are combined to create listening environment dynamics configuration data for all listening environment speakers. The listening environment dynamics processing configuration data can then be used to perform initial dynamics processing in the context of the entire spatial audio mix before it is rendered into signals fed to the speakers. Since this first stage of dynamics processing has access to the entire spatial mix, as opposed to a single independent signal fed to the speaker, the processing can be performed in ways that do not impart distracting shifts to the perceived spatial balance of the mix. Individual speaker dynamics processing configuration data (eg, playback clipping thresholds) may be combined in a manner that eliminates or reduces the amount of dynamics processing performed by any of the independent individual speaker dynamics processing functions.

), которые применяются ко всем составляющим пространственного микширования на первом этапе динамической обработки. Согласно некоторым таким примерам, так как ограничение является одинаковым в отношении всех составляющих, пространственный баланс микширования может сохраняться. Одним способом комбинирования конфигурационных данных динамической обработки отдельных громкоговорителей (например, порогов ограничения воспроизведения) является взятие минимального значения по всем динамикам i:In one example of defining listening environment dynamics configuration data, individual speaker dynamics configuration data (e.g., playback clipping thresholds) for individual speakers can be combined into a single listening environment dynamics configuration data set (e.g., frequency-variable playback clipping thresholds).

) that are applied to all spatial mixing components in the first stage of dynamics processing. According to some such examples, since the constraint is the same for all components, the spatial balance of the mix can be maintained. One way to combine individual speaker dynamics processing configuration data (for example, playback clipping thresholds) is to take the minimum value across all speakers i :

Such a combination essentially eliminates the operation of dynamic processing of each individual speaker, since the spatial mixing is first limited below the threshold of the least functional speaker at each frequency. However, this strategy may be overly aggressive. Many speakers may play at a level below their capability, and the combined playback level of all speakers may be unacceptably low. For example, when applying thresholds in the low frequency range shown in FIG. 4A to a loudspeaker corresponding to the thresholds for FIG. 4C, the reproduction level of the last speaker in the low frequency range will be unnecessarily low. An alternative combination of determining the listening environment dynamic processing configuration data is to take an average (averaging) of the individual speaker dynamics processing configuration data over all the listening environment speakers. For example, in the context of playback restriction thresholds, this average can be defined as follows:

With this combination, the overall playback level can be increased compared to taking the minimum value, since the first stage of dynamics processing clips to a higher level and thus allows louder playback by speakers with more functionality. For speakers whose individual clipping thresholds are below average, their independent dynamics processing functions may still clip their associated signal delivered to the speaker if necessary. However, the first stage of dynamic processing is likely to reduce the need for this constraint, since some initial constraint has been made on the spatial mixing.

. Например, в контексте порогов ограничения воспроизведения эту интерполяцию можно определить следующим образом:According to some examples of determining the dynamics processing configuration data of the listening environment, it is possible to create a tunable combination, with interpolation between the minimum and average values of the dynamics processing configuration data of individual speakers using the tuning parameter

. For example, in the context of playback limit thresholds, this interpolation can be defined as follows:

Other combinations of individual speaker dynamics configuration data are possible, and the present invention is intended to cover all such combinations.

In FIG. 5A and 5B are graphs showing examples of dynamic range compression data. In plots 500a and 500b, input decibel levels are presented on the horizontal axes, and output decibel levels are presented on the vertical axes. As with other disclosed examples, specific thresholds, ratios, and other values are provided by way of example only and are not limiting.

In the example shown in FIG. 5A, the output signal level is equal to the input signal level below the threshold, which in this example is -10 dB. Other examples may include different thresholds, e.g. -20 dB, -18 dB, -16 dB, -14 dB, -12 dB, -8 dB, -6 dB, -4 dB, -2 dB, 0 dB, 2 dB , 4 dB, 6 dB, etc. Above the threshold, various examples of compression ratios are presented. A factor of N :1 means that above the threshold, the output level will rise by 1 dB for every N dB increase in the input signal. For example, a compression ratio of 10:1 (line 505e) means that above the threshold, the output signal level will increase by only 1 dB for every 10 dB increase in the input signal. A compression ratio of 1:1 (line 505a) means that the output signal level is still equal to the input signal level even above the threshold. Lines 505b, 505c, and 505d correspond to aspect ratios of 3:2, 2:1, and 5:1. Other implementations may provide other aspect ratios such as 2.5:1, 3:1, 3.5:1, 4:3, 4:1, etc.

In FIG. 5B shows examples of "knees" that control the change in compression ratio at or near a threshold, which in this example is 0 dB. According to this example, the compression curve containing a "hard" bend consists of two straight segments, a linear segment 510a before the threshold and a linear segment 510b above the threshold. A hard fold may be easier to implement, but it can introduce artifacts.

In FIG. 5B also shows one example of a "soft" fold. In this example, the soft bend spans 10 dB. According to this implementation, above and below 10 dB coverage, the compression ratios of a compression curve containing a soft kink are the same as those of a compression curve containing a hard kink. In other implementations, various other forms of "soft" bends may be provided, which may span more or less decibels, which may indicate a different compression ratio above coverage, etc.

Other types of dynamic range compression data may include rise-up data and fade-out data. "Rise" is the period during which the compressor reduces the gain, for example in response to an increased input level, in order to achieve the gain determined by the compression ratio. The rise time for compressors typically ranges from 25 milliseconds to 500 milliseconds, although other rise times are possible. "Cutoff" is the period during which the compressor increases the gain, for example, in response to a reduced input level, in order to reach the output gain determined by the compression ratio (or the input level if the input level falls below a threshold). For example, the decay time may be in the range of 25 milliseconds to 2 seconds.

Accordingly, in some examples, the individual speaker dynamics processing configuration data may comprise a set of dynamic range compression data for each speaker of the plurality of speakers. The dynamic range compression dataset may contain threshold data, input-to-output ratio data, rise data, attenuation data, and/or knee data. One or more of these types of individual speaker dynamics processing configuration data can be combined to define the listening environment dynamics processing configuration data. As noted above with respect to combining playback clipping thresholds, in some examples, dynamic range compression data may be averaged to determine the listening environment dynamic processing configuration data. In some cases, the minimum or maximum value of the dynamic range compression data (eg, maximum compression ratio) can be used to determine the dynamic processing configuration data of the listening environment. In other implementations, it is possible to create a tunable combination with interpolation between the minimum and average values of the dynamic range compression data for dynamic processing of individual speakers, for example, using a tuning parameter such as the parameter described above with reference to equation (3).

). Такие реализации могут сохранять пространственный баланс микширования, но вызывать другие нежелательные артефакты. Например, когда очень громкая часть пространственного микширования в выделенной области пространства вызывает снижение уровня всего микширования, может возникать «пространственное приглушение». Другие, более мягкие составляющие микширования, отдаленные в пространстве от этой громкой составляющей, могут восприниматься как становящиеся неестественно мягкими. Например, мягкая фоновая музыка может воспроизводиться в окружающем поле пространственного микширования на уровне ниже комбинированных порогов,

, и поэтому на первом этапе динамической обработки не выполняется никакое ограничение пространственного микширования. Затем спереди пространственного микширования (например, на экране в случае звуковой дорожки фильма) мгновенно вводится громкий выстрел, и общий уровень микширования превышает комбинированные пороги. В этот момент первый этап динамической обработки понижает уровень всего микширования ниже порогов

Так как музыка пространственно отделена от выстрела, она может восприниматься в непрерывном потоке музыки как неестественно приглушенная.In some of the above examples, in the first stage of dynamics processing, one set of listening environment dynamics processing configuration data is applied to all spatial mixing components (e.g., one set of combined thresholds,

). Such implementations may maintain the spatial balance of the mix, but introduce other undesirable artifacts. For example, when a very loud portion of a spatial mix in a selected region of space causes the overall mix to be lowered, "spatial muting" may occur. Other softer parts of the mix that are spaced apart from this loud part can be perceived as becoming unnaturally soft. For example, soft background music may be played in an ambient spatial mixing field at a level below the combined thresholds,

, and therefore no spatial mixing constraint is performed in the first stage of dynamics processing. Then, in front of the spatial mix (for example, on the screen in the case of a movie soundtrack), a loud shot is instantly introduced, and the overall mix level exceeds the combined thresholds. At this point, the first stage of dynamics processing lowers the level of the entire mix below the thresholds.

Since the music is spatially separated from the shot, it can be perceived as unnaturally muffled in a continuous stream of music.

To address these issues, some implementations allow independent or partially independent dynamics processing with respect to different "spatial zones" of the spatial mixing. A spatial region can be thought of as a subset of the region of space against which the entire spatial mixing is rendered. While much of the following discussion provides examples of dynamics based on playback clipping thresholds, the concept is equally applicable to individual speaker dynamics configuration data and other types of listening environment dynamics configuration data.

In FIG. 6 shows an example of the spatial zones of the listening environment. In FIG. 6 shows an example of a spatial mixing area (represented by a whole square) subdivided into three spatial zones: Front, Center and Surround.

Although the zones in FIG. 6 are depicted with rigid boundaries, in practice it is preferable to interpret the transition from one spatial zone to another as continuous. For example, a spatial mixing component located in the middle of the left border of a square may contain half of its level related to the front zone and half to the surrounding zone. In this way, the signal level from each component of the spatial mixing in each of the spatial zones can be attributed and accumulated in a continuous manner. The dynamics processing function can then operate independently for each spatial region with respect to the overall signal level assigned to it from the mix. Then, for each spatial mixing component, the results of dynamic processing based on each spatial zone (eg, time-varying frequency gains) can be combined and applied to that component. In some examples, the results of such a combination of spatial zones are different for each component and depend on the assignment of this particular component to each zone. The end result is that spatial mixing components with similar spatial zone assignments receive similar dynamic processing, but independence between spatial zones is allowed. The spatial zones can advantageously be chosen to prevent unwanted spatial shifts, such as left/right imbalance, while at the same time allowing for some spatially independent processing (eg, suppression of other artifacts such as the described spatial muting).

, где индекс j относится к одной из множества пространственных зон. Модуль динамической обработки может действовать независимо в отношении каждой пространственной зоны со связанными с ней порогами

, и результаты можно обратно применить в отношении составных частей пространственного микширования согласно вышеописанному техническому решению.The technical solutions for spatial mixing processing using spatial zones can advantageously be used in the first stage of dynamic processing according to the present invention. For example, for each spatial zone, a different combination of individual speaker dynamics processing configuration data (eg, play limit thresholds) in speakers i can be computed. The set of combined zone thresholds can be represented as

, where the index j refers to one of the many spatial zones. The dynamic processing module can act independently on each spatial zone with its associated thresholds

, and the results can be applied back to the components of the spatial mixing according to the above technical solution.

, каждый из которых связан с требуемым пространственным положением (возможно, переменным во времени). Одним конкретным способом реализации обработки зон является вычисление переменных во времени коэффициентов усиления при панорамировании,

, которые описывают, какая доля каждого звукового сигнала

вносит вклад в зону j в зависимости от требуемого пространственного положения звукового сигнала относительно положения зоны. Эти коэффициенты усиления при панорамировании преимущественно могут быть предназначены для следования закону сохранения энергии при панорамировании, который требует, чтобы сумма квадратов этих коэффициентов усиления была равна единице. На основе этих коэффициентов усиления при панорамировании сигналы зоны,

, можно вычислить как сумму составляющих сигналов, подвергнутых весовой обработке с помощью их коэффициента усиления при панорамировании для данной зоны:Consider the rendered spatial signal as consisting of K separate component signals

, each of which is associated with a desired spatial position (possibly variable in time). One particular way to implement zone processing is to compute time-varying pan gains,

, which describe how much of each audio signal

contributes to the zone j depending on the desired spatial position of the audio signal relative to the position of the zone. These panning gains can advantageously be designed to follow the law of conservation of energy in panning, which requires that the sum of the squares of these panning gains be equal to one. Based on these pan gains, zone signals,

, can be computed as the sum of the signal components weighted by their pan gain for a given zone:

может затем обрабатываться независимо функцией DP динамической обработки, параметризованной порогами зон,

, для получения переменных по частоте и во времени модифицированных коэффициентов усиления зон, G _j :Signal of each zone

can then be processed independently by the dynamic processing DP function parameterized by the zone thresholds,

, to obtain frequency and time variables of the modified zone gains, G _j :

, можно вычислить переменные по частоте и во времени модифицированные коэффициенты усиления для каждого отдельного составляющего сигнала:Then, by combining the modified zone gains in proportion to the panning gains of that signal for the zones,

, you can calculate the frequency and time-varying modified gains for each individual component signal:

, которые могут затем подвергаться рендерингу в сигналы динамиков. These modified signal gains, G _k , can then be applied to each component signal, for example using a filter bank, to obtain dynamically processed component signals,

, which can then be rendered into speaker signals.

можно вычислить как взвешенную сумму порогов ограничения воспроизведения динамиков

с использованием зависящей от пространственных зон и динамиков весовой обработки

: Combining individual speaker dynamics processing configuration data (such as speaker clipping thresholds) for each spatial zone can be done in a variety of ways. As one example, spatial zone reproduction limitation thresholds

can be computed as the weighted sum of speaker playback limit thresholds

using space- and speaker-dependent weighting

:

в зависимости от ответственности каждого динамика за рендеринг составляющих пространственного микширования, связанных с этой зоной, для частоты f.Similar weighting functions can be applied to other types of individual speaker dynamics processing configuration data. Advantageously, the combined individual speaker dynamics configuration data (e.g., playback clipping thresholds) of a spatial area can be shifted to the individual speaker dynamics configuration data (e.g., playback clipping thresholds) for the speakers most responsible for reproducing the spatial mixing components associated with that spatial zone. This can be achieved by setting the weighting factors

depending on the responsibility of each speaker for rendering the spatial mixing components associated with that zone, for frequency f .

на основе этого условного однозначного отображения динамиков в пространственные зоны, но, как и в случае обработки пространственного микширования на основе пространственных зон, предпочтительным может являться более непрерывное отображение. Например, динамик 4 находится весьма близко к передней зоне, и составляющая звукового микса, расположенная между динамиками 4 и 5 (хотя и в условно передней зоне), вероятно, в наибольшей степени будет воспроизводиться комбинацией динамиков 4 и 5. Поэтому имеет смысл внесение вклада конфигурационных данных динамической обработки отдельных громкоговорителей (например, порогов ограничения воспроизведения) динамика 4 в комбинированные конфигурационные данные динамической обработки отдельных громкоговорителей (например, пороги ограничения воспроизведения) передней зоны, а также окружающей зоны. In FIG. 7 shows examples of loudspeakers in the spatial zones of FIG. 6. In FIG. 7 shows the same zones as in Fig. 6, but superimposed with the locations of five exemplary speakers (speakers 1, 2, 3, 4, and 5) responsible for rendering the spatial mix. In this example, speakers 1, 2, 3, 4, and 5 are represented by diamonds. In this particular example, speaker 1 is most responsible for rendering the center zone, speakers 2 and 5 for rendering the front zone, and speakers 3 and 4 for rendering the surrounding zone. You can create weights

based on this conditional one-to-one mapping of speakers to spatial zones, but as in the case of spatial mixing processing based on spatial zones, a more continuous mapping may be preferable. For example, speaker 4 is very close to the front zone, and the component of the sound mix located between speakers 4 and 5 (albeit in the conventionally front zone) is likely to be reproduced to the greatest extent by the combination of speakers 4 and 5. Therefore, it makes sense to contribute configuration individual speaker dynamics data (eg, playback limit thresholds) of the speaker 4 into combined individual speaker dynamics configuration data (eg, playback limit thresholds) of the front zone as well as the surround zone.

значению участия динамика, описывающему относительный вклад каждого динамика i в рендеринг составляющих, связанных с пространственной зоной j. Эти значения можно получить непосредственно из системы рендеринга, ответственной за рендеринг в динамиках (например, из вышеописанного этапа (с)), и набора из одного или более номинальных пространственных положений, связанных с каждой пространственной зоной. Этот набор номинальных пространственных положений может включать набор положений в каждой пространственной зоне.One way to achieve this continuous mapping is to equate the weights

a speaker contribution value describing the relative contribution of each speaker i to the rendering of the components associated with the spatial area j . These values can be obtained directly from the rendering system responsible for rendering in dynamics (eg, from step (c) above) and a set of one or more nominal spatial positions associated with each spatial region. This set of nominal spatial positions may include a set of positions in each spatial zone.

In FIG. 8 shows an example of nominal spatial positions superimposed on the spatial zones and dynamics of FIG. 7. Nominal positions are indicated using numbered circles: the front zone has two positions associated at the top corners of the square, the central zone has a single position associated at the top middle of the square, and the surrounding zone has two positions associated at the bottom corners of the square.

. Это значение представляет общее значение активации динамика i для рендеринга всего набора номинальных положений, связанных с пространственной зоной j. Наконец, значение участия динамика в пространственной зоне можно вычислить как накопленное значение активации,

, нормализованное на сумму всех этих накопленных значений активации для динамиков. Затем весовые коэффициенты можно приблизить к этому значению участия динамика:In order to compute a speaker engagement value for a spatial area, each of the nominal positions associated with that area may be rendered by a renderer to generate speaker activation values associated with that location. These activation values may be, for example, the gain for each speaker in the case of CMAP, or the complex value at a given frequency for each speaker in the case of FV. Then, for each speaker and zone, these activation values can be accumulated for each of the nominal positions associated with the spatial zone to obtain a value

. This value represents the total activation value of speaker i for rendering the entire set of nominal positions associated with spatial region j . Finally, the contribution value of the speaker in the spatial zone can be calculated as the accumulated activation value,

, normalized to the sum of all these accumulated activation values for the speakers. The weights can then be approximated to this speaker participation value:

по всем динамикам i равна единице, что является требуемым свойством для весовых коэффициентов в уравнении 8.The described normalization ensures that the sum

over all speakers, i is equal to one, which is the required property for the weighting factors in Equation 8.

According to some implementations, the above method of calculating speaker participation values and combining thresholds depending on these values can be performed as a static method, in which the resulting combined thresholds are calculated once during the installation procedure, which determines the layout and functionality of the speakers in the environment. It can be assumed that in such a system, once installed, both the configuration data for the dynamics processing of the individual loudspeakers and the way in which the rendering algorithm activates the loudspeakers, depending on the desired location of the audio signal, remain static. In some systems, however, both of these aspects may change over time, for example, in response to changing conditions in the playback environment, and therefore, to account for these changes, it may be necessary to update the combined thresholds in accordance with the above method, either continuously, or in a triggering method on the occurrence of developments.

In response to changes in the listening environment, both CMAP and FV rendering algorithms can be extended to adapt to one or more dynamically configurable features. For example, with reference to FIG. 7, a person near the speaker 3 can speak the wake-up word of the virtual digital assistant associated with the speakers, thus putting the system in a state where it is ready to hear the next command from that person. After saying the wake-up word, the system can determine the person's location using the microphones associated with the speakers. With this information, the system can then choose to redirect the energy of the audio data played from speaker 3 to other speakers so that the microphones on speaker 3 can better hear the person. In such a scenario, speaker 2 in FIG. 7 may for some period of time essentially "take over" the obligations of the speaker 3, and as a result, the values of the participation of the speakers for the surrounding area change significantly; the participation value of speaker 3 decreases and the participation value of speaker 2 increases. The zone thresholds can then be recalculated as they depend on the changed speaker contribution values. Alternatively, or in addition to these changes to the rendering algorithm, the clipping thresholds of speaker 3 can be reduced below their nominal values set to prevent speaker distortion. This can ensure that the residual reproduction of the audio data from the speaker 3 does not go beyond a certain threshold, defined as such that interferes with microphones listening to a person. Since the zone thresholds are also dependent on the individual speaker thresholds, they can also be updated in this case.

In FIG. 9 is a flow diagram that describes one example of a method that can be performed by such a device or system as those disclosed herein. The steps of method 900, like other methods described herein, are not necessarily performed in the order listed. In some implementations, one or more steps of method 900 may be performed concurrently. Moreover, some implementations of the method 900 may include more or fewer steps than are shown and/or described. The steps of method 900 may be performed by one or more devices, which may be (or may include) a control system, such as control system 110 as shown in FIG. 1 and described above, or one of the other examples of disclosed control systems.

According to this example, step 905 includes obtaining by the control system and through the interface system individual speaker dynamics processing configuration data for each of the plurality of speakers in the listening environment. In this implementation, the individual speaker dynamics configuration data comprises a set of individual speaker dynamics configuration data for each speaker of the plurality of speakers. According to some examples, individual speaker dynamics processing configuration data for one or more speakers may correspond to one or more functionality of one or more speakers. In this example, each of the individual speaker dynamics configuration data sets contains at least one type of dynamics configuration data.

In some cases, step 905 may include obtaining sets of individual speaker dynamics processing configuration data from each of the plurality of speakers in the listening environment. In other examples, step 905 may include obtaining individual speaker dynamics configuration data sets from a data structure stored in a storage device. For example, individual speaker dynamics configuration data sets may have been obtained previously, for example, as part of a setup procedure for each of the speakers, and stored in a data structure.

According to some examples, individual loudspeaker dynamics processing configuration data sets may be specialized. In some such examples, sets of individual speaker dynamics configuration data may have been estimated previously based on individual speaker dynamics configuration data for speakers having similar characteristics. For example, step 905 may include a speaker matching process to determine the most similar speaker from a data structure pointing to a plurality of speakers and a corresponding set of individual speaker dynamics processing configuration data for each of the plurality of speakers. The speaker comparison step may be based, for example, on comparing the size of one or more of the woofers, tweeters and/or midrange speakers.

In this example, step 910 includes determining by the control system the listening environment dynamics processing configuration data for the plurality of speakers. According to this implementation, the determination of the listening environment dynamics processing configuration data is based on a set of individual speaker dynamics processing configuration data for each speaker of the plurality of speakers. Determining the dynamics processing configuration data of the listening environment may include combining the dynamics configuration data of individual speakers from a set of dynamics configuration data, for example, by taking an average of the dynamics configuration data of individual speakers of one or more types. In some cases, determining the dynamics processing configuration data of the listening environment may include determining the minimum or maximum values of the dynamics processing configuration data of individual speakers of one or more types. According to some such implementations, determining the dynamics processing configuration data of the listening environment may include interpolation between a minimum or maximum value and an average value of the dynamics processing configuration data of individual speakers of one or more types.

In this implementation, step 915 includes receiving by the control system and through the system of interfaces audio data containing one or more audio signals and associated spatial data. For example, the spatial data may indicate an intended perceived spatial position corresponding to an audio signal. In this example, the spatial data contains channel data and/or spatial metadata.

In this example, step 920 includes a control system performing dynamics processing on the audio data based on the listening environment dynamics configuration data to generate processed audio data. The dynamic processing of step 920 may include any of the dynamic processing methods disclosed herein, which include, but are not limited to, applying one or more playback limit thresholds, compression data, etc.

Here, step 925 includes rendering, by the control system, the processed audio data for playback through a speaker set comprising at least some of the plurality of speakers to obtain rendered audio signals. In some examples, step 925 may include applying a CMAP rendering process, an FV rendering process, or a combination of the two. In this example, step 920 is performed before step 925. However, as noted above, step 920 and/or step 910 may be based at least in part on the rendering process of step 925. Steps 920 and 925 may include performing methods such as those as described above with reference to the listening environment dynamics module and the rendering module 320 of FIG. 3.

According to this example, step 930 includes delivering the rendered audio signals through the interface system to the set of speakers. In one example, step 930 may include delivering rendered audio signals to speakers 205a-205m via smart home hub 305 and its interface system.

In some examples, method 900 may include performing dynamics on the rendered audio signals in accordance with individual speaker dynamics configuration data for each speaker in the set of speakers to which the rendered audio signals are delivered. For example, again with reference to FIG. 3, dynamics processing modules A-M may perform dynamics processing on rendered audio signals in accordance with individual speaker dynamics configuration data for speakers 205a-205m.

In some implementations, the individual speaker dynamics configuration data may comprise a set of playback limit threshold data for each speaker of the plurality of speakers. In some such examples, the play limit threshold dataset may comprise play limit thresholds for each of the plurality of frequencies.

In some cases, determining the configuration data of the dynamic processing of the listening environment may include determining the minimum thresholds for limiting playback across multiple speakers. In some examples, determining the listening environment dynamics processing configuration data may include averaging the playback limit thresholds to obtain average playback limit thresholds over a plurality of speakers. In some such examples, determining the listening environment dynamics processing configuration data may include determining the minimum playback restriction thresholds across a plurality of speakers and interpolating between the minimum playback restriction thresholds and the average playback restriction thresholds.

According to some implementations, averaging the playback restriction thresholds may include determining a weighted average of the playback restriction thresholds. In some such examples, the weighted average may be at least partially based on the characteristics of the rendering process implemented by the control system, for example, the characteristics of the rendering process according to block 925.

In some implementations, performing dynamics processing on audio data may be based on spatial zones. Each of the spatial zones may correspond to a subset of the listening environment.

According to some such implementations, dynamic processing may be performed separately for each of the spatial zones. For example, the determination of the listening environment dynamics processing configuration data may be performed separately for each of the spatial regions. For example, separately for each of the one or more spatial zones, a combination of dynamic processing configuration data sets across a plurality of speakers may be performed. In some examples, combining multiple speaker dynamics configuration data sets separately for each of one or more spatial zones may be at least in part based on speaker activation values by the rendering process depending on the desired location of the audio signal in one or more spatial zones.

In some examples, combining the multiple speaker dynamics configuration data sets separately for each of the one or more spatial zones may be at least in part based on the speaker contribution value for each speaker in each of the one or more spatial zones. Each speaker contribution value may be based at least in part on one or more nominal spatial positions in each of the one or more spatial zones. In some examples, the nominal spatial positions may correspond to standard channel locations in a Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, or Dolby 9.1 surround mix. In some such implementations, each speaker contribution value is at least in part based on the activation value of each speaker corresponding to the rendering of audio data at each of one or more nominal spatial positions in each of the one or more spatial zones.

According to some such examples, the weighted average of playback clipping thresholds may be based at least in part on speaker activation values by the rendering process depending on the proximity of the audio signal to the spatial regions. In some cases, the weighted average may be based at least in part on the loudspeaker participation value for each loudspeaker in each of the spatial zones. In some such examples, each speaker contribution value may be based at least in part on one or more nominal spatial positions in each of the spatial regions. For example, nominal spatial positions may correspond to standard channel locations in Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, or Dolby 9.1 surround mixing. In some implementations, each speaker participation value may be based at least in part on the activation value of each speaker corresponding to the rendering of audio data at each of the one or more nominal spatial positions in each of the spatial zones.

In some implementations, rendering the processed audio data may include determining a relative speaker set activation value in accordance with one or more dynamically configurable functions. Some examples are described below with reference to FIG. 10 and following. One or more dynamically configurable functions may be based on one or more audio signal properties, one or more speaker set properties, or one or more external input signals. For example, one or more dynamically configurable functions may be based on the proximity of the speakers to one or more listeners; the proximity of the loudspeakers to the position of the attractive force, wherein the attractive force is a factor favoring a relatively larger activation value of the loudspeaker located closer to the position of the attractive force; the proximity of the loudspeakers to the position of the repulsive force, with the repulsive force being a factor favoring a relatively smaller activation value of the loudspeaker closer to the position of the repulsive force; the functionality of each loudspeaker relative to other loudspeakers in the environment; synchronization of loudspeakers relative to other loudspeakers; fulfillment of the awakening word; or performing an echo canceller.

The relative speaker activation value in some examples may be based on a cost function of the perceived spatial position of the audio signals when played by the speakers, a criterion for the proximity of the intended perceived spatial position of the audio signals to the speaker positions, and one or more dynamically configurable functions.

In some examples, minimizing the cost function (comprising at least one dynamic speaker activation metric) may result in at least one of the speakers being deactivated (in the sense that each such speaker does not play the corresponding audio content) and at least one of the speakers being activated. (in the sense that each such speaker reproduces at least some part of the rendered audio content). The dynamic speaker activation metric(s) can enable at least one of a variety of behaviors, including warping the spatial representation of audio data away from a particular smart audio device so that its microphone can better hear a speaker, or so that a secondary audio stream can be hear better from the speaker(s) of the smart audio device.

In some implementations, the individual speaker dynamics configuration data may comprise a set of dynamic range compression data for each speaker of the plurality of speakers. In some cases, the dynamic range compression data set may contain one or more of threshold data, input-to-output level ratio data, rise data, attenuation data, or knee data.

As noted above, in some implementations, at least some of the steps of the method 900 that are represented in FIG. 9 may be omitted. For example, in some implementations, steps 905 and 910 are performed during the installation process. In some implementations, once the listening environment dynamic processing configuration data has been determined, steps 905-910 are not re-executed during "runtime" operations until the type and/or location of the listening environment speakers is changed. For example, in some implementations, an initial check may occur to determine if any speakers have been added or removed, if any speaker positions have changed, etc. If yes, then steps 905 and 910 may be implemented. If not, then steps 905 and 910 may not be re-executed before the "duty cycle" operations, which may include steps 915-930.

As noted above, existing flexible rendering solutions include "Center of Mass Amplitude Panning" (CMAP) and "Flexible Virtualization" (FV). From a professional point of view, both of these technical solutions render a set of one or more audio signals, each with an associated desired perceived spatial position, for playback through a set of two or more speakers, with the relative activation value of the speakers from the set depending on the model. the perceived spatial position of the specified audio signals reproduced through these speakers, and the proximity of the desired perceived spatial position of the audio signals to the positions of the speakers. This model ensures that the listener of the audio signal is audible near their intended spatial position, and the Proximity score controls which speakers are used to achieve that spatial impression. In particular, the proximity indicator favors the activation of speakers that are close to the desired perceived spatial position of the audio signal. For both CMAP and FV, this functional relationship is conveniently derived from a cost function written as the sum of two scores, one for spatial aspect and one for proximity:

обозначает положения множества M громкоговорителей,

обозначает требуемое воспринимаемое пространственное положение звукового сигнала, и g обозначает M-мерный вектор значений активации динамиков. Для CMAP каждое значение активации в этом векторе представляет коэффициент усиления для динамика, тогда как для FV каждое значение активации представляет фильтр (в этом, втором случае g можно эквивалентно рассматривать как вектор комплексных значений на определенной частоте, и для образования фильтра на множестве частот вычисляется другое g). Оптимальный вектор значений активации находят путем минимизации функции стоимости по значениям активации:In this case, the set

denotes the positions of the set M of loudspeakers,

denotes the desired perceived spatial position of the audio signal, and g denotes an M -dimensional vector of speaker activation values. For CMAP, each activation value in this vector represents a gain for the speaker, while for FV, each activation value represents a filter (in this second case, g can equivalently be viewed as a vector of complex values at a specific frequency, and another is computed to form a filter over a set of frequencies. g ). The optimal vector of activation values is found by minimizing the cost function over the activation values:

является подходящим. Для решения этой проблемы последующую нормализацию

можно выполнить так, чтобы абсолютный уровень значений активации стал управляемым. Например, может требоваться нормализация вектора на единичную длину, что согласовывается с обычно используемыми правилами панорамирования с постоянной энергией: Known cost function definitions make it difficult to control the absolute level of optimal activation values resulting from the minimization described above, although the relative level between the components

is suitable. To solve this problem, the subsequent normalization

can be done so that the absolute level of activation values becomes manageable. For example, a vector may need to be normalized to one length, which is consistent with the commonly used constant energy panning rules:

и

. Для CMAP

получают из модели, которая размещает воспринимаемое пространственное положение звукового сигнала, воспроизводимого из набора громкоговорителей, в центре массы положений этих громкоговорителей, подвергнутых весовой обработке с помощью связанных с ними активирующих коэффициентов усиления,

(элементов вектора g):The exact behavior of the flexible rendering algorithm is determined by the particular structure of the two cost function measures,

and

. For CMAP

derived from a model that places the perceived spatial position of an audio signal reproduced from a set of loudspeakers at the center of the mass of those loudspeaker positions weighted by their associated activation gains,

(elements of vector g ):

Equation 3 is then converted to a space cost representing the squared error between the desired position of the audio data and the audio data received by the activated speakers:

, в левом и правом ушах слушателя. Теоретически b представляет собой вектор фильтров (по одному фильтру для каждого уха) размера 2x1, однако с ним более удобно обращаться как с вектором комплексных значений размера 2x1 на определенной частоте. Совершая дальнейшие действия с помощью этого представления на определенной частоте, требуемую бинауральную характеристику можно получить из набора индексов HRTF с помощью положения объекта:For FV, the spatial exponent of the cost function is defined differently. The goal is to obtain a binaural characteristic b corresponding to the position of the sound object,

, in the listener's left and right ears. Theoretically, b is a vector of filters (one filter for each ear) of size 2x1, but it is more convenient to treat it as a vector of complex values of size 2x1 at a certain frequency. Working further with this representation at a specific frequency, the required binaural response can be obtained from the set of HRTF indices using the position of the object:

At the same time, a 2x1 binaural response e , obtained in the listener's ears with loudspeakers, is modeled as a 2xM sound transmission matrix, H , multiplied by a vector of complex speaker activation values, g , of size M x1:

, относительно положения слушателя. Наконец, пространственная составляющая функции стоимости задана как квадратичная ошибка между требуемой бинауральной характеристикой (уравнение 14) и бинауральной характеристикой, полученной с помощью громкоговорителей (уравнение 15):The sound transmission matrix, H , is modeled based on a set of speaker positions,

, relative to the position of the listener. Finally, the spatial component of the cost function is given as the squared error between the desired binaural response (Equation 14) and the binaural response obtained from the loudspeakers (Equation 15):

For convenience, the spatial cost function exponent for CMAP and FV given in Equations 13 and 16 can in both cases be converted into a square matrix dependent on speaker activation values, g :

, исключает эту неопределенность и приводит к частному решению со свойствами восприятия, преимущественными по сравнению с другими возможными решениями. Как для CMAP, так и для FV,

построен так, что значения активации динамиков, положение

которых отдалено от требуемого положения звукового сигнала,

, исключаются в большей степени, чем значения активации динамиков, положение которых близко к требуемому положению. Такое построение приводит к оптимальному набору значений активации динамиков, которые являются рассеянными, при этом в значительной степени активируются только динамики в непосредственной близости от требуемого положения звукового сигнала, и на практике это приводит к пространственному воспроизведению звукового сигнала, которое является для восприятия более устойчивым к перемещению слушателя рядом с набором динамиков.where A is a square matrix of size M x M , B is a vector of size 1x M , and C is a scalar. Matrix A has rank 2, so when M > 2, there are an infinite number of speaker activation values, g , for which the spatial deflection exponent is zero. Introduction of the second indicator of the cost function,

, eliminates this uncertainty and leads to a particular solution with perceptual properties that are superior to other possible solutions. For both CMAP and FV,

built so that the activation values of the speakers, the position

which are far from the required position of the sound signal,

, are excluded to a greater extent than the activation values of the speakers, the position of which is close to the desired position. This arrangement results in an optimal set of speaker activation values that are diffuse, with only speakers in close proximity to the desired audio signal position being significantly activated, and in practice this results in a spatial audio signal that is perceptually more resistant to movement. listener next to a set of speakers.

, можно задать как взвешенную по расстояниям сумму квадратов абсолютных значений активаций динамиков. Компактно это представлено в матричной форме в виде: For this, the second indicator of the cost function,

, can be specified as the distance-weighted sum of the squares of the absolute values of the speaker activations. This is compactly represented in matrix form as:

where D is a diagonal degradation matrix of the distance between the desired audio data position and each speaker:

The distance degradation function can take many forms, but a useful parameterization is:

представляет собой евклидово расстояние между требуемым положением аудиоданных и положением динамика, и

и

представляют собой перестраиваемые параметры. Параметр

указывает глобальную силу ухудшения;

соответствует пространственной степени ухудшения расстояния (будут исключены громкоговорители на расстоянии приблизительно

или далее) и

учитывает внезапность наступления ухудшения на расстоянии

.where

is the Euclidean distance between the desired audio data position and the speaker position, and

and

are adjustable parameters. Parameter

indicates the global degradation strength;

corresponds to the spatial degree of distance degradation (speakers at a distance of approx.

or more) and

takes into account the suddenness of the deterioration at a distance

.

Combining the two measures of the cost function given in Equations 17 and 18a results in a total cost function.

Setting the derivative of this cost function with respect to g to zero and solving for g gives the optimal solution for the speaker activation values:

In general, the optimal solution in Equation 20 may result in speaker activation values that are negative in magnitude. For building a flexible renderer with CMAP, these negative activation values may be undesirable, so Equation (20) can be minimized as long as all activation values remain positive.

In FIG. 10 and 11 are diagrams depicting an exemplary set of speaker activation values and object rendering positions. In these examples, the speaker activation values and object rendering positions correspond to speaker positions of 4, 64, 165, -87, and -4 degrees. Other implementations may have more or fewer speakers and/or speakers in different positions. In FIG. 10 are speaker activation values, 1005a, 1010a, 1015a, 1020a, and 1025a, which contain the optimal solution to Equation 20 for these particular speaker positions. In FIG. 11, individual speaker positions are plotted as squares 1105, 1110, 1115, 1120, and 1125, which correspond to speaker activation values, 1005a, 1010a, 1015a, 1020a, and 1025a, respectively, of FIG. 10. In FIG. 11, angle 4 corresponds to speaker position 1120, angle 64 corresponds to speaker position 1125, angle 165 corresponds to speaker position 1110, angle -87 corresponds to speaker position 1105, and angle -4 corresponds to speaker position 1115. In FIG. 11 also shows the ideal object positions (in other words, the positions at which sound objects should be rendered) for a variety of possible object angles as dots 1130a and the corresponding actual render positions for these objects as dots 1135a connected to the ideal object positions by dotted lines 1140a. .

In FIG. 12A, 12B and 12C show examples of speaker participation values corresponding to the examples of FIG. 10 and 11. In FIG. 12A, 12B, and 12C, an angle of -4.1 corresponds to speaker position 1115 of FIG. 11, an angle of 4.1 corresponds to speaker position 1120 of FIG. 11, the -87 angle corresponds to speaker position 1105 of FIG. 11, an angle of 63.6 corresponds to speaker position 1125 of FIG. 11 and an angle of 165.4 corresponds to speaker position 1110 of FIG. 11. These speaker contribution values are examples of "weighting factors" associated with spatial areas disclosed elsewhere in this document. According to these examples, the speaker contribution values shown in FIG. 12A, 12B and 12C correspond to the participation of each loudspeaker in each of the spatial zones shown in FIG. 6: Loudspeaker contribution values shown in FIG. 12A corresponds to the contribution of each speaker in the center zone, the speaker contribution values shown in FIG. 12B corresponds to the contribution of each speaker in the front left and right zones, and the speaker contribution values shown in FIG. 12C corresponds to the participation of each speaker in the rear zone.

The combination of flexible rendering techniques (implemented in accordance with some embodiments) with a set of wireless smart speakers (or other smart audio devices) can result in an easy-to-use spatial audio rendering system with extremely high functionality. When considering interactions with such a system, it becomes clear that dynamic modifications to the spatial rendering may be required to optimize for other problems that may arise during the use of this system. To achieve this goal, one class of embodiments complements existing flexible rendering algorithms (in which the speaker activation value depends on the previously disclosed spatial and proximity metrics) with one or more additional dynamically configurable functions depending on one or more properties of the rendered audio signals, a set speakers and/or other external inputs. According to some embodiments, the existing flexible rendering cost function given in Equation 1 is supplemented with one or more of these additional dependencies according to

представляют дополнительные показатели стоимости, где

представляет набор из одного или более свойств подвергаемых рендерингу звуковых сигналов (например, звуковой программы на основе объектов),

представляет набор из одного или более свойств динамиков, подвергаемых рендерингу звуковых сигналов и

представляет один или более дополнительных внешних входных сигналов. Каждый показатель

возвращает стоимость в зависимости от значений активации, g, в отношении комбинации из одного или более свойств звуковых сигналов, динамиков и/или внешних входных сигналов, обобщенно представленных множеством

. Следует понимать, что набор

содержит по меньшей мере только один элемент из

,

и

.In equation 21, indicators

represent additional indicators of value, where

represents a set of one or more properties of audio signals to be rendered (for example, an object-based audio program),

represents a set of one or more speaker properties that are subject to audio rendering, and

represents one or more additional external input signals. Each indicator

returns the cost depending on activation values, g , with respect to a combination of one or more properties of audio signals, speakers, and/or external input signals, summarized by the set

. It should be understood that the set

contains at least one element from

,

and

.

включают, но без ограничения:Examples

include but not limited to:

• the required perceived spatial position of the audio signal;

• level (possibly variable in time) of the audio signal; and/or

• the spectrum (possibly time-varying) of the audio signal.

включают, но без ограничения:Examples

include but not limited to:

• location of loudspeakers in the listening space;

• frequency response of loudspeakers;

• loudspeaker playback level restrictions;

• parameters of dynamics processing algorithms in speakers, such as limiter gains;

• the result of a measurement or evaluation of sound transmission from one speaker to another;

• criteria for the effectiveness of the echo canceller in relation to speakers; and/or

• the relative timing of the speakers relative to each other.

включают, но без ограничения:Examples

include but not limited to:

• the locations of one or more listeners or speakers in the playback space;

• the result of a measurement or an estimate of the sound transmission from each loudspeaker to the listening location;

• the result of a measurement or evaluation of the sound transmission from the speaker to the set of loudspeakers;

• the location of some other landmark in the playback space; and/or

• the result of a measurement or an estimate of the sound transmission from each speaker to some other landmark in the playback space.

With the new cost function given in Equation 21, the optimal set of activation values can be found by minimizing with respect to g and possibly subsequent normalization as described earlier in Equations 11a and 11b.

в виде взвешенной суммы квадратов абсолютных значений активаций динамиков:Similar to the proximity cost given in Equations 18a and 18b, it is also convenient to express each of the new cost function measures

as a weighted sum of squares of the absolute values of speaker activations:

представляет собой диагональную матрицу весовых коэффициентов

, описывающую стоимость, связанную со значением активации динамика i, для показателя j:where

is a diagonal matrix of weights

, which describes the cost associated with the activation value of the speaker i , for the indicator j :

Combining Equations 22a and 22b with the square matrix version of the CMAP and FV cost functions defined in Equation 19 provides a potentially useful implementation of the commonly used cost function (according to some embodiments) defined in Equation 21:

, можно найти путем дифференцирования уравнения 23 для полученияWith this definition of the new cost function measures, the overall cost function remains a square matrix, and the optimal set of activation values,

, can be found by differentiating Equation 23 to obtain

в зависимости от заданного значения непрерывного ухудшения,

, для каждого из громкоговорителей. В одном иллюстративном варианте осуществления это значение ухудшения представляет собой расстояние от объекта (подлежащего рендерингу) до рассматриваемого громкоговорителя. В другом иллюстративном варианте осуществления это значение ухудшения представляет неспособность данного громкоговорителя воспроизводить некоторые частоты. На основе этого значения ухудшения весовые показатели

можно параметризовать в виде:It is useful to consider each of the weight indicators

depending on the set value of continuous degradation,

, for each speaker. In one exemplary embodiment, this degradation value is the distance from the object (to be rendered) to the speaker in question. In another exemplary embodiment, this degradation value represents the inability of a given loudspeaker to reproduce certain frequencies. Based on this degradation value, the weights

can be parameterized as:

представляет предварительный коэффициент (который учитывает глобальную интенсивность весового показателя), где

представляет порог ухудшения (рядом или за пределами которого весовой показатель становится значительным), и где

представляет монотонно возрастающую функцию. Например, когда

, весовой показатель имеет вид:where

represents a provisional factor (which takes into account the global weighting intensity), where

represents the impairment threshold (near or beyond which the weight becomes significant), and where

represents a monotonically increasing function. For example, when

, the weight indicator looks like:

,

,

представляют собой перестраиваемые параметры, которые указывают, соответственно, на глобальную силу ухудшения, внезапность наступления ухудшения и степень ухудшения. При установке этих перестраиваемых значений следует тщательно следить за тем, чтобы относительное воздействие показателя стоимости

относительно любых других дополнительных показателей стоимости, а также

и

, соответствовало достижению требуемого результата. Например, в качестве эмпирического правила, если требуется, чтобы определенное ухудшение явно доминировало над остальными, то подходящей может являться установка его интенсивности

приблизительно в десять раз больше следующей по величине интенсивности ухудшения.where

,

,

are tunable parameters that indicate, respectively, the global strength of the deterioration, the suddenness of the onset of the deterioration, and the degree of deterioration. When setting these adjustable values, care should be taken to ensure that the relative impact of the cost measure

in relation to any other additional cost indicators, as well as

and

, to achieve the desired result. For example, as a rule of thumb, if you want a certain impairment to clearly dominate the others, setting its intensity to

approximately ten times the next-highest impairment rate.

In the case of excluding all loudspeakers, it is often convenient to subtract the minimum impairment from all post-processing weights so that at least one of the loudspeakers is not excluded:

As noted above, there are many possible use cases that can be implemented using the new cost function measures described herein (and similar new cost function measures used in accordance with other embodiments). In the following, more specific details are described with three examples: moving audio data in the direction of the listener or speaker, moving audio data in the direction away from the listener or speaker, and moving audio data away from the landmark.

принимает форму уравнения 26 со значением непрерывного ухудшения,

, заданным расстоянием i-го динамика от фиксированного местоположения аттрактора,

, и пороговым значением

, заданным максимальным из этих расстояний по всем динамикам: In the first example, what will be referred to in this document as "attractive force" is used to pull the audio data to some position, which in some examples may be the position of the listener or speaker, the position of a landmark, the position of furniture, etc. This position in this document may be called the "position of the force of attraction", or "the location of the attractor". In the context of this document, "attractive force" is a factor that favors a relatively larger activation value of a loudspeaker closer to the position of the attractive force. According to this example, the weighting factor

takes the form of Equation 26 with a continuous degradation value,

, given by the distance of the i -th speaker from the fixed location of the attractor,

, and threshold value

, given by the maximum of these distances over all speakers:

= 20,

= 3, и

вектору, соответствующему положению слушателя/говорящего под углом 180 градусов (нижняя центральная часть графика). Эти значения

,

и

являются лишь примерами. В некоторых реализациях

может находиться в диапазоне от 1 до 100, и

может находиться в диапазоне от 1 до 25.To illustrate a use case with "pulling" audio data to a listener or speaker, in particular, equate

= 20,

= 3, and

a vector corresponding to the position of the listener/speaker at an angle of 180 degrees (bottom center of the graph). These values

,

and

are just examples. In some implementations

can range from 1 to 100, and

can range from 1 to 25.

.In FIG. 13 is a graph of speaker activation values in an exemplary embodiment. In this example, in FIG. 13 are speaker activation values, 1005b, 1010b, 1015b, 1020b, and 1025b, which contain the optimal cost function solution for the same speaker positions as in FIG. 10 and 11, with the addition of an attractive force represented as

.

иллюстрирует влияние весовых коэффициентов аттрактора на оптимальное решение функции стоимости.In FIG. 14 is a graph of object rendering positions in an exemplary embodiment. The positions of the loudspeakers in Fig. 14, 17 and 20 are the same as those shown in FIG. 11. In this example, in FIG. 14 shows the respective ideal object positions 1130b for a plurality of possible object angles, and the corresponding actual render positions 1135b for those objects, which are connected to the ideal object positions 1130b by dashed lines 1140b. Offset Orientation of Actual Render Positions 1135b to a Fixed Position

illustrates the influence of attractor weights on the optimal solution of the cost function.

In FIG. 15A, 15B and 15C show examples of speaker contribution values corresponding to the examples of FIG. 13 and 14. In Figs. 15A, 15B, and 15C, an angle of -4.1 corresponds to speaker position 1115 of FIG. 11, an angle of 4.1 corresponds to speaker position 1120 of FIG. 11, the -87 angle corresponds to speaker position 1105 of FIG. 11, an angle of 63.6 corresponds to speaker position 1125 of FIG. 11 and an angle of 165.4 corresponds to speaker position 1110 of FIG. 11. According to these examples, the speaker contribution values shown in FIG. 15A, 15B and 15C correspond to the participation of each loudspeaker in each of the spatial zones shown in FIG. 6: Loudspeaker contribution values shown in FIG. 15A corresponds to the participation of each speaker in the center zone, the speaker participation values shown in FIG. 15B corresponds to the contribution of each speaker in the front left and right zones, and the speaker contribution values shown in FIG. 15C corresponds to the participation of each speaker in the rear zone.

= 5,

= 2, и

вектору, соответствующему положению слушателя/говорящего под углом 180 градусов (в нижней центральной части графика). Эти значения

,

и

являются лишь примерами. Как отмечено выше, в некоторых примерах

может находиться в диапазоне от 1 до 100, и

может находиться в диапазоне от 1 до 25.To illustrate the use case of pushing audio data away from the listener or speaker, in particular, equate

= 5,

= 2, and

a vector corresponding to the position of the listener/speaker at an angle of 180 degrees (in the lower central part of the graph). These values

,

and

are just examples. As noted above, in some examples

can range from 1 to 100, and

can range from 1 to 25.

.In FIG. 16 is a graph of speaker activation values in an exemplary embodiment. According to this example in FIG. 16 shows the speaker activation values, 1005c, 1010c, 1015c, 1020c and 1025c, which contain the optimal solution of the cost function for the same speaker positions as in the previous figures, with the addition of the repulsive force represented as

.

иллюстрирует влияние весовых коэффициентов репеллера на оптимальное решение функции стоимости.In FIG. 17 is a graph of object rendering positions in an exemplary embodiment. In this example, in FIG. 17 shows ideal object positions 1130c for a variety of possible object angles and corresponding actual render positions 1135c for those objects, which are connected to ideal object positions 1130c by dashed lines 1140c. Offset orientation of actual render positions 1135c away from fixed position

illustrates the effect of repeller weights on the optimal cost function solution.

In FIG. 18A, 18B and 18C show examples of speaker contribution values corresponding to the examples of FIG. 16 and 17. According to these examples, the speaker contribution values shown in FIG. 18A, 18B and 18C correspond to the participation of each loudspeaker in each of the spatial zones shown in FIG. 6: Loudspeaker contribution values shown in FIG. 18A corresponds to the contribution of each speaker in the center zone, the speaker contribution values shown in FIG. 18B corresponds to the contribution of each speaker in the front left and right zones, and the speaker contribution values shown in FIG. 18C corresponds to the participation of each speaker in the rear zone.

вектору, соответствующему положению двери под углом 180 градусов (нижняя центральная часть графика). Для достижения большей силы отталкивания и полного смещения звукового поля в переднюю часть первичного пространства прослушивания приравняем

= 20, и

= 5.Another exemplary use case is to "push" audio data away from a sound-sensitive landmark, such as a door to a baby's room. As in the last example, we equate

vector corresponding to the position of the door at an angle of 180 degrees (bottom center of the graph). To achieve a greater repulsive force and a complete shift of the sound field to the front of the primary listening space, we equate

= 20, and

= 5.

In FIG. 19 is a graph of speaker activation values in an exemplary embodiment. Again, in this example in FIG. 19 shows the speaker activation values, 1005d, 1010d, 1015d, 1020d, and 1025d, which provide an optimal solution for the same set of speaker positions with more repulsion added.

In FIG. 20 is a graph of object rendering positions in an exemplary embodiment. Again, in this example in FIG. 20 shows ideal object positions 1130d for a variety of possible object angles and corresponding actual render positions 1135d for those objects, which are connected to ideal object positions 1130d by dashed lines 1140d. The offset orientation of the actual rendering positions 1135d illustrates the impact of large repeller weights on the optimal cost function solution.

In FIG. 21A, 21B and 21C show examples of speaker participation values corresponding to the examples of FIG. 19 and 20. According to these examples, the speaker contribution values shown in FIGS. 21A, 21B and 21C correspond to the participation of each loudspeaker in each of the spatial zones shown in FIG. 6: Loudspeaker contribution values shown in FIG. 21A corresponds to the participation of each speaker in the center zone, the speaker participation values shown in FIG. 21B corresponds to the contribution of each speaker in the front left and right zones, and the speaker contribution values shown in FIG. 21C corresponds to the participation of each speaker in the rear zone.

In FIG. 22 is a diagram of an environment, which in this example is a living area. The environment shown in Fig. 22 contains a set of intelligent audio devices (devices 1.1) for interacting with audio data, speakers (1.3) for outputting audio data, and controllable lighting fixtures (1.2). In one example, the microphones only contain devices 1.1, and therefore only they detect the presence of a user (1.4) issuing a speech fragment (eg, a command in the form of a wake-up word). Using various methods, information to provide a position estimate (eg, a fine-grained position estimate) of a user issuing (eg, speaking) a wake-up word can be obtained collectively from these devices.

In such a living space, there is a set of natural activity zones in which a person will perform a task or action, or cross a threshold. These areas (zones) of action are where an attempt to estimate the location (eg, to determine an undefined location) or user context to facilitate other aspects of the interface can take place. A rendering system comprising (i.e., implemented with) at least some of the 1.1 devices and 1.3 speakers (and/or optionally at least one other subsystem or device) may operate to render audio data for playback (e.g. , some or all of the speakers 1.3) in the living area or in one or more areas of it. It is contemplated that such a rendering system may be configured to operate in either standard spatial mode or distributed spatial mode, according to any embodiment of the disclosed method. In the example in FIG. The 8 key action areas are:

1. Kitchen sink and cooking area (in the upper left area of the living area);

2. Refrigerator door (to the right of the sink and cooking area);

3. Dining area (in the lower left area of the living area);

4. Open area of the living area (to the right of the sink and cooking area and dining area);

5. Sofa area for watching TV (to the right of the open area);

6. TV itself;

7. Tables; and

8. Door area or hallway (in the upper right area of the living area).

There are often the same number of lighting fixtures with the same placement suitable for the action areas. Some or all of the lighting fixtures may be individually controlled by network facilities.

In accordance with some embodiments, audio data is rendered (e.g., by one of the devices 1.1 or another device of the system of FIG. 22) for playback (according to any disclosed embodiment) by one or more of the speakers 1.3 (and/or speaker(s) one or more devices 1.1).

One class of embodiments includes methods for rendering audio data for playback and/or playback of audio data by at least one (eg, all or some) of a plurality of coordinated (organized) smart audio devices. For example, a set of smart audio devices present (on the system) in a user's home can be arranged to handle multiple use cases simultaneously, including flexible rendering of audio data for playback by all or some of (i.e., speaker(s) of some or all) smart audio devices. There are many interactions with the system that require dynamic rendering and/or playback modifications. Such modifications may be, but are not necessarily, focused on spatial fidelity.

In some embodiments, rendering for playback and/or playback by speaker(s) of a plurality of smart audio devices that are coordinated (organized) is implemented. In other embodiments, rendering is implemented for playback and/or playback by a speaker(s) from another set of speakers.

Some embodiments (e.g., a system or renderer, or a rendering method, or a playback system or method) refer to systems and methods for rendering audio data for playback and/or playback by some or all speakers (i.e., each activated speaker) from a set speakers. In some embodiments, the speakers are speakers of a coordinated (organized) set of smart audio devices. Examples of such embodiments include the following enumerated example embodiments (EEE).

EEE1. A method for rendering audio data for playback by at least two speakers, which includes the following steps:

(a) combining speaker clipping thresholds, thereby defining combined thresholds;

(b) performing dynamics processing on the audio data using the combined thresholds to generate processed audio data; and

(c) rendering the processed audio data into signals fed to speakers.

EEE2. A method according to EEE1, wherein the clipping thresholds are a plurality of one or more playback clipping thresholds that represent clipping at different frequencies.

EEE3. The method according to EEE1 or EEE2, wherein said combination of the clipping threshold comprises taking the minimum of the thresholds of the plurality of speakers.

EEE3. A method according to EEE1 or EEE2, wherein said combination of clipping thresholds includes the step of averaging over the clipping thresholds of a plurality of speakers.

EEE5. A method according to EEE4 wherein said averaging step is a weighted average.

EEE6. The method according to EEE5, in which the specified weight processing is obtained depending on the specified rendering.

EEE7. A method according to any one of EEE1-EEE6, wherein said rendering is spatial.

EEE8. A method according to EEE7, wherein said audio program flow restriction includes a different restriction in different spatial zones.

EEE9. A method according to EEE8, in which the thresholds of each spatial zone are obtained using unique combinations of playback limitation thresholds of a plurality of loudspeakers.

EEE10. A method according to EEE9, in which the unique thresholds of each spatial zone are obtained using a weighted average of the clipping thresholds of a plurality of loudspeakers.

EEE11. A method according to EEE10, wherein the weighting associated with a given loudspeaker for a given zone is obtained based on the participation factor of the speaker associated with that zone.

EEE12. A method according to EEE11, wherein said speaker participation factor is derived based on speaker activation values corresponding to rendering one or more nominal spatial positions related to said limiter spatial zone.

EEE13. The method according to any one of EEE 1-EEE12, which further includes clipping the signals applied to the speakers in accordance with clipping thresholds associated with the respective speaker.

EEE14. A system capable of performing a method according to any one of EEE 1-EEE13.

Many embodiments are technically feasible. It will be apparent to those skilled in the art from the present disclosure how to implement them. This document describes some embodiments.

Some aspects of the present invention include a system or device capable of (eg, programmed) performing any disclosed method, and a tangible computer-readable storage medium (eg, a disk) that stores code for implementing any disclosed method or steps thereof. For example, the system may be or comprise a general purpose programmable processor, a digital signal processor, or a microprocessor programmed using software or firmware and/or otherwise configured to perform any of a variety of operations on data, including the embodiment disclosed method or steps thereof. Such a general purpose processor may be or comprise a computer system including an input device, a storage device, and a processing subsystem programmed to (and/or otherwise capable of) executing the disclosed method (or steps thereof) in response to data being passed thereto.

Some embodiments are implemented as a configurable (e.g., programmable) digital signal processor (DSP) that is configured (e.g., programmed and otherwise configured) to perform the desired processing on the audio signal(s), including performing one or more of the disclosed methods. . Alternatively, some embodiments (or elements thereof) are implemented as a general purpose processor (e.g., a personal computer (PC), other computer system, or microprocessor that may include an input device and a storage device) programmed with software or firmware. providing and/or otherwise configured to perform any of the plurality of operations of one or more of the disclosed methods. Alternatively, elements of some embodiments are implemented as a general purpose processor or DSP configured to (e.g., programmed) perform one or more of the disclosed methods, and the system may also include other elements (e.g., one or more speakers and/or one or more microphones). ). A general purpose processor configured to perform one or more of the disclosed methods may be connected to an input device (eg, a mouse and/or keyboard), a storage device, and, in some examples, a display device.

Another aspect of the present invention is a computer-readable storage medium (eg, a disk or other tangible storage medium) that stores code for executing (eg, executable code for executing) one or more of the disclosed methods or steps thereof.

While specific embodiments and uses of the present invention have been described herein, it will be apparent to those skilled in the art that many changes are possible with respect to the embodiments and uses described herein without departing from the scope of the present invention as described. and stated in this document. It should be understood that while certain forms of the present invention have been shown and described, the scope of the present invention should not be limited to the specific embodiments described and shown or the specific methods described.

Claims (25)

1. A method for processing audio data, including: obtaining by the control system and through the system of interfaces individual speaker dynamics processing configuration data for each of the plurality of speakers of the listening environment, wherein the individual speaker dynamics processing configuration data comprises a set of individual speaker dynamics processing configuration data for each loudspeaker of the plurality of speakers, wherein the individual speaker dynamics processing configuration data the loudspeakers comprise a data set of playback limitation thresholds for each loudspeaker of the plurality of loudspeakers; determination by the control system of the listening environment dynamics processing configuration data for a plurality of speakers, wherein the determination of the listening environment dynamics processing configuration data is based on a set of individual speaker dynamics processing configuration data for each loudspeaker of the plurality of speakers, wherein the determination of the listening environment dynamics processing configuration data includes threshold averaging playback restrictions across multiple speakers; receiving by the control system via the interface system audio data containing one or more audio signals and associated spatial data, the spatial data comprising at least one of channel data or spatial metadata; performing, by the control system, dynamics processing on the audio data based on the configuration data of the listening environment dynamics processing to generate processed audio data; rendering, by the control system, the processed audio data for playback through a speaker set comprising at least some of the plurality of speakers to obtain rendered audio signals; and delivering, via the interface system, the rendered audio signals to the set of speakers. 2. The method for processing audio data according to claim 1, wherein the playback limit threshold data set contains playback limit thresholds for each of the plurality of frequencies. 3. The audio data processing method according to claim 1 or 2, characterized in that determining the configuration data of the dynamic processing of the listening environment includes averaging the playback restriction thresholds in order to obtain average playback restriction thresholds over a plurality of speakers, determining the minimum playback restriction thresholds over a plurality of speakers, and interpolating between minimum playback restriction thresholds; and average reproduction restriction thresholds. 4. The audio data processing method according to claim 3, wherein the averaging of the playback restriction thresholds comprises determining a weighted average playback restriction threshold. 5. The method of processing audio data according to claim 4, wherein the weighted average is at least in part based on the characteristics of the rendering process implemented by the control system. 6. The method of processing audio data according to claim 5, characterized in that performing dynamics processing in relation to audio data is based on spatial zones, and each of the spatial zones corresponds to a subset of the listening environment, while the weighted average of the playback restriction thresholds is at least partially based on the values activation of loudspeakers using the rendering process, depending on the proximity of the audio signal to spatial zones. 7. An audio data processing method according to claim 5 or 6, wherein the weighted average is at least partially based on the loudspeaker participation value for each loudspeaker in each of the spatial zones. 8. The audio data processing method of claim 7, wherein each speaker contribution value is at least in part based on one or more nominal spatial locations within each of the spatial zones. 9. The audio data processing method of claim 8, wherein the nominal spatial positions correspond to the standard channel locations in a Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, or Dolby 9.1 surround mixing. 10. The method of processing audio data according to claim 8 or 9, characterized in that each speaker participation value is at least partially based on the activation value of each speaker corresponding to the rendering of audio data at each of one or more nominal spatial positions within each of the spatial zones . 11. The method of processing audio data according to any one of paragraphs. 1-10, characterized in that it further includes performing dynamics processing on the rendered audio signals in accordance with individual speaker dynamics configuration data for each speaker from the set of speakers to which the rendered audio signals are delivered. 12. The method of processing audio data according to any one of paragraphs. 1-11, characterized in that the individual speaker dynamics processing configuration data contains a set of dynamic range compression data for each speaker of the plurality of speakers. 13. The audio data processing method of claim 12, wherein the dynamic range compression dataset comprises one or more of threshold data, input-to-output ratio data, fade-in data, or fade-out data. 14. The method of processing audio data according to any one of paragraphs. 1-13, characterized in that the individual speaker dynamics processing configuration data for one or more speakers of the plurality of speakers corresponds to one or more functionality of the one or more speakers. 15. System for processing audio data, configured to perform the method according to any one of paragraphs. 1–14, while the system contains: - a smart home hub containing a control system and an interface system; - Lots of loudspeakers wherein the control system comprises a listening environment dynamic processing configuration data module, a listening environment dynamic processing module, and a rendering module. 16. A persistent storage medium containing software stored thereon, the software containing commands for controlling one or more devices in order to perform the method according to any one of paragraphs. 1–14.

Images