CN118972776A - 3D audio system - Google Patents

Abstract

The present description includes technical specifications for defining filters and their distribution networks on a grid in a three-dimensional space that can be presented in a user interface. Multiple audio tracks can be separated from an input mono or stereo or determined from the output of a microphone array, where each audio track is related to a corresponding sound source position in three-dimensional space. The relative position can be set by the listener in the user interface. According to the relative position of the listener and the sound source in three-dimensional space, one or more pairs of filters are selected in the distribution grid of filters. Each pair of filters is applied to the corresponding audio track to generate multiple filtered audio tracks, and then three-dimensional sound is generated from the multiple filtered audio tracks for headphones to listen to. If multiple speakers are used, each of the above separated or determined audio tracks can be directly (i.e., all-pass filtered) driven by an amplifier to drive each corresponding speaker. The sound of multiple speakers can directly constitute a three-dimensional sound field, and the three-dimensional sound can be heard without wearing headphones.

Description

3D audio system

This application is a divisional application of the Chinese patent application with application number 202110585625.9, application date May 27, 2021, and title “Three-dimensional Audio System” (based on and claiming priority of U.S. Provisional Application No. 63/036,797 filed on June 9, 2020 and U.S. Application No. 17/227,067 filed on April 9, 2021), and the entire contents of each application are incorporated herein by reference.

Technical Field

The present description relates to the generation of three-dimensional sound, and in particular to systems and methods for picking up or processing mixed audio tracks into separate sound types and then applying transfer functions to the separate sounds to generate three-dimensional sound that contains spatial information about the sound sources and a three-dimensional (3D) sound field recreated according to user settings.

Background Art

Billions of people around the world listen to music, but most listeners can only listen to music in mono or stereo sound formats, with stereo being a method of sound reproduction. However, stereo sound generally refers to only two audio channels played using two speakers or headphones. More immersive sound technologies such as surround sound require that multiple audio tracks (such as a 5.1 or 7.1 surround sound setup) be recorded and saved, and the sound must be played through an equal number of speakers. Each of the audio channels or tracks contains mixed sounds from multiple sound sources. Therefore, stereo sound is different from "real" sound (e.g., the sound in front of a concert stage) because spatial information about independent sound sources (e.g., the orientation of instruments and vocals) is not reflected in the sound. The method of the present description can use multiple independent audio channels played by multiple speakers or headphones so that the sound from the speakers or headphones comes from various directions as in natural hearing.

People can perceive spatial information and hear "real" three-dimensional (3D) sound as binaural sound (e.g., sound perceived by the left and right ears) with two ears, such as music perceived by binaural in concert halls, theaters, or sports events in stadiums or arenas. However, as mentioned above, today's music technology usually only provides mono or stereo sound without spatial cues or spatial information. Therefore, it is different from music experienced in theaters, arenas and concert halls through headphones or earphones or on speakers or even on multi-track, multi-speaker surround systems, which are usually more pleasant. At present, it is possible to generate 3D sound, for example, by many speakers installed on the walls of a cinema and each speaker is driven by a separate sound track recorded during the film production process. However, this 3D audio system may be very expensive and cannot be implemented in mobile devices (application software), and even cannot be implemented in most home theaters or car audio. Therefore, in today's music and entertainment industry, most music or other audio data is stored and played as mono or stereo sound, where all sound sources (such as vocals and various instruments) are pre-mixed into only one (mono) or two (stereo) tracks.

Most of the audio/sound from video conferencing devices (such as computers, laptops, smart phones, or tablets) is monophonic sound. Although on the display screen, the user (e.g., attendee or participant) can see all the participants of the conference in separate windows, the audio is usually only a monophonic channel with a narrow bandwidth. Using the video of each different attendee, a virtual conference room can be realized, but the audio component cannot match the video component because the 3D sound required for a more accurate (e.g., spatial) virtual reality sound experience cannot be provided. In addition, when two attendees have similar sounding voices, the user may not be able to distinguish the voices when they speak at the same time or even separately. For example, this may happen when the user is looking at a shared document on another screen or video window and the user is not looking at the attendee's face. When more attendees participate in the video conference (e.g., a distance learning classroom), the problem may be more serious. Users need spatial information, such as 3D sound, to help identify which attendee is speaking.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be more fully understood through the detailed description given below and the accompanying drawings of various embodiments of the present invention. However, the accompanying drawings should not be used to limit the present invention to specific implementation examples, but are only for explanation and understanding.

1A-1B illustrate a system for generating three-dimensional sound implemented in accordance with the present teachings.

2A and 2B illustrate a spatial relationship between a sound source and a listener in a three-dimensional space and a data structure and retrieval of an HRTF filter for generating 3D sound reflecting the spatial relationship implemented according to the present description.

FIG3 illustrates a system for training a machine learning model to separate mixed audio tracks implemented in accordance with the present description.

4 illustrates a system for separating and filtering mixed audio tracks using transformed sound signals implemented in accordance with the present teachings.

5A-5E illustrate in waveform and spectrogram an original mixed sound and the separation of the mixed sound into vocals, drums, bass, and other sounds, respectively, according to an implementation of the present teachings.

6 illustrates far-field voice control of a 3D binaural music system with music retrieval via voice and sound separation implemented in accordance with the present teachings.

7A-7D illustrate a GUI for user settings for 3D sound implemented in accordance with the present teachings, including listener positions in the middle of the band (7A-7C) and in front of the band (7D).

FIG. 8 illustrates a system for generating 3D sound using a microphone array according to an implementation of the present disclosure.

9A-9B illustrate beam patterns of a 3D microphone and a 3D microphone array with spatial noise cancellation implemented in accordance with the present teachings.

FIG. 10 shows a conference or virtual concert system for generating three-dimensional sound according to the present description.

11 illustrates a virtual conference room displayed as a GUI for a conference system for generating three-dimensional sound implemented in accordance with the present teachings.

FIG. 12 illustrates a method for generating three-dimensional sound implemented in accordance with the present description.

FIG. 13 illustrates a method for generating three-dimensional sound implemented in accordance with the present description.

14 illustrates a block diagram of hardware of a computer system according to one or more implementations of the present description.

DETAILED DESCRIPTION

This article describes a three-dimensional (3D) configurable soundstage audio system and applications and implementations. A three-dimensional (3D) soundfield refers to sound that includes discrete sound sources located at different spatial locations. A 3D soundstage is a sound that represents a 3D soundfield. For example, soundstage music can allow a listener to have an auditory perception of the independent locations of musical instruments and human voice sources when listening to a given piece of music through headphones, headsets, or speakers. In general, a 3D soundstage can give a listener a perception of spatial information. The soundstage is configurable and can be set by the listener, DJ, software, or audio system. For example, the position of each instrument in the 3D soundfield can be moved, and the listener's position in the 3D soundfield can be statically or dynamically moved close to the preferred instrument position.

In order to listen to or play the 3D sound field, the listener can use binaural sound represented by two tracks, one track for the left ear and one track for the right ear, so that the listener perceives the spatial position information associated with the sound source. Through headphones, headsets or other such devices, binaural sound can have the same experience as 3D sound (for example, feeling from different positions). In addition, 3D sound can also be used directly to play the 3D sound field. In direct 3D sound, sounds are played from a group of speakers located at different 3D positions (for example, corresponding to the expected positions of each sound source in the 3D sound field). Each speaker can play a separate separate track, for example, one speaker for drums and another speaker for bass. The listener can hear the 3D sound field directly from the speakers because they are in the real world 3D sound field. In both binaural and direct 3D sound, the listener's brain can perceive the 3D sound field and can identify and track discrete individual sound sources, just like in the real world, which can be referred to as acoustic virtual reality in this description.

In addition, another method of realizing 3D sound field can be to record binaural sound directly with dedicated binaural/3D microphones. Most existing binaural microphones are just dummy heads with microphones installed in the ears, which may be too large and/or too expensive for many applications. Therefore, this paper describes a 3D microphone that can have a small size by using a very small microphone array and signal processing technology. This small 3D microphone can be used with any handheld recording device (such as a smart phone or tablet). The sound output picked up by the 3D microphone can be binaural, stereo or multi-track recording, where each track is used for a spatial direction associated with the sound source of the 3D sound field.

In this description, the signal-to-noise ratio (SNR) of an audio signal is enhanced using the following three techniques described below. Noise reduction is the process of reducing background noise in an audio channel based on temporal information, such as statistical properties between signal and noise or frequency distribution of different types of signals. A microphone array uses one or more sound beam patterns to enhance the sound from one beam direction while eliminating the sound from outside the beam direction. Acoustic echo cancellation (AEC) uses one or more reference signals to eliminate corresponding signals mixed in the signal collected by the microphone. The reference signal is related to the signal that the AEC is to eliminate.

system

1A-1B illustrate systems 100A and 100B for generating three-dimensional sound implemented in accordance with the present teachings. Systems 100A and 100B may be stand-alone computer systems or networked computing resources implemented in a computing cloud.

Referring to FIG. 1A , the system 100A may include a sound separation unit 102A, a storage unit 104A for storing a plurality of filters (such as head shape related transfer function (HRTF) filters, all-pass filters, or equalization filters), a signal processing unit 106A, and a 3D sound field setting unit 108A having a graphical user interface (GUI) 110A to receive user input. For the sake of simplicity of discussion, the filter hereinafter is referred to as an HRTF filter, although it is understood that the filter can be any type of suitable filter, including an all-pass filter or an equalizer filter. The sound separation unit 102A, the storage unit 104A, and the 3D sound field setting unit 108A can be connected to the signal processing unit 106A. The signal processing unit 106A can be a programmable device that can be programmed according to the settings on the user interface device (not shown) GUI 110A to generate three-dimensional sound.

In the example of Fig. 1A, the input to the sound separation unit 102A is the original mixed audio track composed of mono or stereo signals or audio, and the output of the signal processing unit 106A is the 3D binaural audio for the left ear and the right ear respectively. Each audio track in the input of the mixed audio track or channel can first be separated by the sound separation unit 102A into a group of separated audio tracks (e.g., sound sources associated with one or a group of sound types), wherein each track represents a type (or category) of sound, such as vocals, drums, bass or others (e.g., based on the properties of the corresponding sound source).

Then the signal processing unit 106A can process each separated audio track using the paired HRTF filters from the storage unit 104A to output two audio channels representing the left ear channel and the right ear channel for each separated audio track. In one implementation, the above process can process each input mixed audio track in parallel.

Each pair of HRTF filters (e.g., a pair of left and right HRTF filters 200B of FIG. 2B described below) can be associated with a point on a grid in three-dimensional space (e.g., these HRTF filters can be stored in a database as a distribution network of grid points), and each grid point can be represented by two parameters: an azimuth angle θ and an attitude angle γ (e.g., 202B and 204B in 2B, respectively). The distribution network (mesh) of HRTF filters (e.g., 200B) can be an array of pre-calculated or pre-measured left and right HRTF filter pairs defined on a grid in three-dimensional space (e.g., 200A), wherein each point of the grid has a pair of associated left and right HRTF filters. HRTF filter pairs can be retrieved by applying an activation function, wherein the input of the activation function may include the relative position and distance/range between the sound source and the listener, and the output of the activation function may be a determined HRTF database index for retrieving HRTF filter pairs defined on the grid points. For example, in one activation function implementation, the input of the activation function may be an azimuth angle θ and an attitude angle γ, and the output is a database index for retrieving left and right HRTF filter pairs. The retrieved HRTF filters can then be used to filter the separated tracks. For each separated track, an activation function needs to be called to retrieve the corresponding HRTF filter pair. The values of the azimuth angle θ and the attitude angle γ can be determined according to the user setting specification. For example, as shown in Figure 7A, the azimuth angle θ has the values 0° (voice), 30° (drums), 180° (bass), and 330° (keyboard instrument) and the attitude angle γ is 0, then four pairs of HRTF filters need to be retrieved by the activation function to filter the four separated tracks respectively.

As shown in Figures 2A and 2B, the listener (e.g., 202A) and/or the sound source (e.g., 204A) can move, with angles θ and γ changing over time. Then, a series of new HRTF filter pairs (e.g., 200B) may need to be dynamically retrieved in order to output correct binaural sounds to virtually represent the sounds received by the listener (e.g., 202A) in the 3D sound space (e.g., 200A). Dynamic retrieval of HRTF filters can be facilitated by storing these filters as a distribution network, because the stored HRTF filter pairs may have been associated with any point on a grid in a three-dimensional space, in which the listener and/or the sound source can be located during movement. Range R (210A) can be represented by the volume of the filtered sound. In this way, the closer the distance between the listener and the sound source, the louder the volume.

The left audio tracks of all filter outputs can then be mixed to generate the left channel of binaural sound (e.g., binaural L), while all right channels can be mixed to generate the right channel of binaural sound (e.g., binaural R). When the L and R channels are played simultaneously through headphones or headsets, the listener can experience 3D binaural sound and perceive the spatial location of the sound source in the 3D sound field.

In addition, the listener can set the position and/or volume of each sound source and/or the listener in the 3D sound field through the GUI 110A. Virtually (for example, in acoustic virtual reality), the listener and the sound source can be positioned at any position in the 3D sound field, and the volume of each sound source can be proportional to the distance from the listener's position to the position of the sound source in the 3D sound field. For example, the sound source position and/or volume can be set through the GUI 110A that can be presented via a user interface device. The user interface device can, for example, be in the form of a touch screen on a smart phone (Figures 7A-7D) or a tablet computer. In one implementation, the virtual position of the vocal sound source in the 3D sound field can be in front of the listener, the drum sound source can be in front of the right of the listener, the bass sound source can be behind the other sound sources relative to the listener (e.g., far away), and the "other" instrument (e.g., unidentified sound type or category) can be in front of the left of the listener, and the drum and bass sound sources are set to be louder and the vocals and "other" sound sources are set to be softer by positioning the listener (virtual head) near the drums and bass (Fig. 7C). Then, according to the listener's own settings, the listener can hear the 3D sound field from binaural outputs (e.g., binaural L and binaural R). If the virtual head and the instrument are placed in the same position (e.g., Fig. 7B), the listener will hear the solo sound.

In one implementation, to generate binaural output (e.g., binaural L+R), as shown in FIG. 1A , for each separated audio track associated with the corresponding sound source position, a pair of corresponding HRTF filters can be selected (e.g., from storage unit 104A) to process the separated audio track into two outputs (e.g., by signal processing unit 106A): L and R audio. Finally, a mixer (not shown) can mix all L tracks and all R tracks respectively to output binaural L and R signals. The selection of corresponding HRTF filters will be discussed in further detail below (e.g., see the description of FIG. 2 below). If the mixed audio track is stereo (dual audio track), each audio track needs to be processed as described above to generate mixed binaural sound. When the L and R channels are played simultaneously through headphones or headsets, the listener can experience 3D binaural sound and perceive a 3D sound field.

1B, the system 100B may include a sound separation unit 102B, a 3D signal processing unit 104B, an amplifier 106B, a speaker 108B, and a 3D sound field setting unit 110B having a graphical user interface (GUI) 112B for receiving user input. The sound separation unit 102B and the 3D sound field setting unit 110B may be connected to the signal processing unit 104B. The signal processing unit 104B may be a programmable device that may be programmed to implement three-dimensional sound generation according to the settings received via the GUI 112B presented on a user interface device (not shown).

In the example of FIG. 1B , the input of the sound separation unit 102B is a mixed track composed of the original stereo or mono or stereo signal or audio, and the output from the 3D signal processing unit 104B is a set of tracks to drive multiple speakers 108B through an amplifier 106B. Each of the inputs of the mixed tracks or channels can first be separated by the sound separation unit 102B into a set of separated tracks (e.g., about a corresponding sound source or type), wherein each track represents a type (or category) of sound, such as vocals, drums, bass or other (e.g., based on the properties of the corresponding sound source). Then, each separated track can be processed by the 3D signal processing unit 104B to output a single track to drive a speaker 108B through an amplifier 106B for each processed track. In one implementation, the above process can be performed in parallel for each input mixed track. Then all output tracks can be played through speakers 108B (e.g., at different locations in the real world) to form a real world 3D sound field for the real world position of the listener.

As noted above with respect to FIG. 1A , the listener can set the position and/or volume of each sound source and/or the listener in the 3D sound field through GUI 112B. Virtually (e.g., in acoustic virtual reality), the listener and the sound source can be positioned at any position in the 3D sound field, and the volume of each sound source can be proportional to the distance from the listener's position to the position of the sound source in the 3D sound field. For example, the sound source position and/or volume can be set through GUI 112B, which can be presented via a user interface device. The user interface device can be, for example, in the form of a touch screen on a smart phone or tablet computer. The listener can then hear the 3D sound field according to the listener's own settings output from the speaker 108B.

7A-7D, one can see an implementation of GUI 110A or GUI 112B, which will be described in detail below.

2A-2B illustrate a spatial relationship between a sound source 204A and a listener 202A in a 3D space 200A, and the selection of an HRTF filter 200B for generating 3D sound that reflects the spatial relationship, implemented in accordance with the present teachings.

A head shape dependent transfer function (HRTF) filter (e.g., similar to those stored in memory unit 104A of FIG. 1A ) can characterize how a human listener (with external human ears on his head) receives sound at a first specified location in three-dimensional space from a sound source at a second specified location in the same 3D space. The head, ears, size and shape of the ear canals, density of hair, size and shape of the nasal and oral cavities all change the sound when the sound waves hit the listener and affect the listener's perception by enhancing some frequencies and attenuating others. But the envelope of the response spectrum can be more complex than a simple enhancement or attenuation: it can affect a wide spectrum and/or it can bring significant changes from different sound directions.

With two ears (e.g., binaural hearing), a listener can locate sounds in three dimensions: range (distance); in the up and down directions; and in front and behind, as well as on either side. This is possible because the brain, inner ear, and outer ear (pinna) work together to infer the location. The listener can estimate the location of the sound source by getting cues from each ear (monaural cues) and by comparing the cues received at the two ears (difference cues or binaural cues). What the human brain perceives is the time difference and intensity difference of the sound arriving at each ear. Monophonic cues come from the interaction between the sound source and the human anatomy of the listener, where the original sound source is changed by the inner ear and outer ear (pinna) before entering the ear canal to be processed by the cochlea and the brain. These changes encode the location of the sound source, and the sound can be picked up via the relationship between the location of the sound source and the listener's position. The soundtrack filter based on this relationship is referred to as an HRTF filter in this article. The sound of an audio track is convolved using a pair of HRTF filters to generate binaural signals for the left and right ears respectively, wherein if the source sound is played at the position associated with the pair of HRTF filters, the binaural sound signals (e.g., binaural L+R of FIG. 1A ) correspond to real-world 3D sound field signals that would be heard at the listener's position.

A pair of binaural tracks for the left and right ears of the listener can be used to generate binaural sounds from mono or stereo, which seem to be from a specific position in space. HRTF filters are transfer functions that describe how sounds from a specific position in 3D space reach the listener's position (usually at the outer end of the listener's ear canal). HRTF filters can be implemented as convolution calculations in the time domain or multiplications in the frequency domain to save computing time, as shown in Figure 4 (described more fully below). Multiple pairs of HRTF filters can be applied to multiple tracks from multiple sound sources to generate a 3D sound field represented as a binaural sound signal. The corresponding HRTF filter can be selected based on the listener's settings (i.e., the relative position of the desired sound source to the listener).

2A, a 3D sound space 200A in which a sound source (e.g., 204A) and a listener 202A are located can be represented as a grid having a polar coordinate system. The relative position and distance from the listener 202A to the sound source 204A can be determined based on the following three parameters: azimuth angle θ (202B of FIG. 2B), attitude angle γ (204B of FIG. 2B), and radius R (210A).

2B, the corresponding HRTF filter 200B for each position of the listener in the 3D space 200A can be measured, generated, saved and stored as a function of the polar coordinate system representing the 3D space 200A. Each HRTF filter 200B (e.g., a pair of left and right HRTF filters) can be associated with a point on a grid (e.g., the HRTF filters are stored as a distribution grid) and each grid point can be represented by two parameters: an azimuth angle θ202B and an attitude angle γ204B. According to the user's settings, the system (e.g., 100A of FIG. 1A) will know the spatial relationship between each sound source (e.g., 204A) and the listener 202A, that is, the system will know α206A, β208A and R 210A. Therefore, based on θ=α, and γ=β, the system can retrieve the corresponding HRTF filter pair 200B (e.g., HRTF _Right and HRTF _Left ) for the left and right ears of the listener for the separated audio tracks associated with the sound source 204A. The audio track of the sound source 204A may then be processed (e.g., by the signal processing unit 106A of FIG. 1A ) using the retrieved HRTF filter 200B. The output volume of the generated 3D sound may be a function of the radius R 210A. The shorter the length of R 210A, the greater the output 3D volume.

In one embodiment, for multiple sound sources (such as sound source 204A), the system can repeat the above filter retrieval and filtering operations for each sound source, and then combine (e.g., mix) the filtered audio tracks together for final binaural output or stereo (better than mono) output to two speakers.

As noted above with respect to FIG. 1A , the listener 202A and/or the sound source 204A may move, with the angles θ and γ changing over time. Subsequently, a series of new HRTF filter pairs 200B may need to be dynamically retrieved in order to output correct binaural sounds to virtually represent the sounds received by the listener 202A in the 3D sound space 200A. The dynamic retrieval of the HRTF filters 200B may be facilitated by storing these filters as a distribution grid, since the stored HRTF filter pairs may have been associated with any point on the grid in three-dimensional space, where the listener and/or the sound source may be located during movement.

FIG. 3 illustrates a system 300 for training a machine learning model 308 to separate mixed audio tracks, according to one embodiment of the present description.

Although music can be recorded on multiple tracks using multiple microphones, where each individual track represents each instrument or voice recorded in the studio, the music stream that consumers most often get is mixed into stereo sound. The cost of recording, storing, bandwidth, transmitting and playing multi-track audio may be very high, so most existing music recording and transmission devices (radios or smart phones) are set to use only mono or stereo sound. In order to generate a 3D sound field from a conventional mixed track format (mono and stereo), the system (e.g., the system 100A of Figure 1A or 100B of Figure 1B) needs to separate each mixed track into multiple tracks, where each track represents or separates a type (or category) of sound or instrument. The separation can be performed according to a mathematical model and corresponding software or hardware implementation, where the input is a mixed track and the output is a separated track. In one embodiment, for stereo input, the left track and the right track can be processed jointly or separately (e.g., by the sound separation unit 102A of Figure 1A or the sound separation unit 102B of Figure 1B).

Machine learning in this description refers to a method implemented in software on a hardware processing device that uses statistical techniques and/or artificial neural networks to give computers the ability to "learn" (i.e., gradually improve the performance of a specific task) from data without being explicitly programmed. Machine learning can use a parameterized model (called a "machine learning model"), which can be implemented using supervised learning/semi-supervised learning, unsupervised learning, or reinforcement learning methods. Supervised/semi-supervised learning methods can use labeled training samples to train machine learning models. In order to perform tasks using supervised machine learning models, computers can use samples (usually referred to as "training data") to train machine learning models and adjust the parameters of machine learning models based on performance measurements (e.g., error rates). The process of adjusting the parameters of machine learning models (usually referred to as "training machine learning models") can generate specific models to perform the actual tasks for which they are trained. After training, the computer can receive new data inputs associated with the task and calculate the estimated output of the machine learning model for predicting the results of the task based on the trained machine learning model. Each training sample can include input data and corresponding expected output data, where the data can be in an appropriate form (such as a vector value of alphanumeric symbols or numerical values) as a representation of the track.

The learning process may be an iterative process. The process may include a forward propagation process to calculate the output based on the machine learning model and the input data fed into the machine learning model, and then calculate the difference between the expected output data and the calculated output data. The process may further include a back propagation process to adjust the parameters of the machine learning model based on the calculated difference.

The parameters of the machine learning model 308 for separating mixed tracks can be trained by machine learning, statistics or signal processing techniques. As shown in Figure 3, the machine learning model 308 can have two stages: a training period and a separation period. In the training period of the machine learning model 308, the audio or music recording of the mixed sound can be used as the input of the feature extraction unit 302, and the corresponding separated tracks can be used as the target of the separation model training unit 304, that is, as a sample of the desired separation output. The separation model training unit 304 may include a data processing unit, which includes a data normalization/data perturbation unit 306 and a feature extraction unit 302. Data normalization normalizes the input training data so that they have similar dynamic ranges. Data perturbation generates reasonable data changes to cover more signal situations than the signals available in the training data, so that there are more data for more training. Data normalization and perturbation can be optional, depending on the amount of available data.

The feature extraction unit 302 can extract features from the original input data (e.g., mixed sound) to help with training and separation calculations. The training data can be processed in the time domain (raw data), frequency domain, feature domain, or time-frequency domain by Fast Fourier Transform (FFT), Short Time Fourier Transform (STFT), spectrogram, auditory transform, wavelet or other transforms. FIG. 4 (described more fully below) shows how to perform track separation and HRTF filtering in the transform domain.

The model structure and training algorithm for the machine learning model 308 can be a neural network (NN), a convolutional neural network (CNN), a deep neural network (DNN), a recursive neural network (RNN), a long short-term memory (LSTM), a Gaussian mixture model (GMM), a hidden Markov model (HMM), or any model and/or algorithm that can be used to separate the sound sources in the mixed audio track. After training, in the separation period, the input music data can be separated into multiple tracks by the trained separation model calculation unit 310, and each separated audio track corresponds to a separate sound. In one embodiment, multiple separated audio tracks can be mixed in different ways for different sound effects by user settings (e.g., via the GUI 110A of Figure 1A).

In one embodiment, the machine learning model 300 can be a DNN or CNN, which can include multiple layers, in particular, an input layer for receiving data input (e.g., training period), an output layer for generating output (e.g., separation period), and one or more hidden layers, each hidden layer includes linear or nonlinear computing elements (referred to as neurons) to perform DNN or CNN calculations propagated from the input layer to the output layer, which can transform the data input into output. Two adjacent layers can be connected by a line. Each line can be associated with a parameter value (referred to as a synaptic weight value), which provides a proportional coefficient for the output value of the previous layer of neurons of the input of one or more neurons in the latter layer.

FIG5 shows (described more fully below) waveforms and corresponding spectrograms associated with a mixed track of music (e.g., a mixed sound input) and separated tracks of vocals, drums, bass, and other sounds, where the mixed track is separated by the trained machine learning model 308. The separation calculations can be performed according to the system 400 shown in FIG4.

FIG. 4 illustrates a system 400 for separating and filtering a mixed audio track using a transform domain sound signal, according to one implementation of the present description.

The training data (e.g., time-domain mixed sound signals) can be processed by the separation unit 404 in the time domain (e.g., raw data) (e.g., the sound separation unit 102A of FIG. 1A ), or a forward transform 402 can be used so that the training data can be processed in the frequency domain, feature domain, or time-frequency domain by a fast Fourier transform (FFT), a short-time Fourier transform (STFT), a spectrogram, an auditory transform, a wavelet, or other transform. The HRTF filter 406 (e.g., those stored in the storage unit 104A of FIG. 1A ) can be implemented as a convolution calculation in the time domain, or an inverse transform 408 can be used so that the HRTF filter 406 can be implemented as a multiplication in the frequency domain to save computation time. Therefore, both track separation and HRTF filtering can be performed in the transform domain.

5A-5E illustrate, in waveform and spectrogram, an original mixed sound and the separation of the mixed sound into vocals, drums, bass, and other sounds, respectively, according to an implementation of the present teachings.

Shown in FIG. 5A is a waveform and corresponding spectrogram associated with a mixed track of music (eg, the mixed sound input to system 100A of FIG. 1A ).

Shown in FIG. 5B is a waveform and corresponding spectrogram associated with a separated track of a vocal sound from a mixed track of music.

Shown in FIG. 5C is a waveform and corresponding spectrogram associated with a separated track of a drum sound from a mixed track of music.

Shown in FIG. 5D is a waveform and corresponding spectrogram associated with a separated track of a bass sound from a mixed track of music.

Shown in FIG. 5E is a waveform and corresponding spectrogram associated with a separated track of other sounds (eg, unidentified sound types) from a mixed track of music.

In one embodiment of the present description, the mixed audio tracks are separated using the trained machine learning model 308. The separation calculations may be performed according to the system 400 described above with respect to FIG.

FIG. 6 illustrates far-field voice control of a 3D binaural music system 600 utilizing sound separation implemented in accordance with the present teachings.

First, microphone array 602 can pick up a voice command. Preamplifier/analog-to-digital converter (ADC) 604 can amplify the analog signal and/or convert it to a digital signal. Both the preamplifier and ADC are optional, depending on what kind of microphones are used in microphone array 602. For example, digital microphones may not need them.

The acoustic beamformer 606 forms an acoustic beam(s) to enhance speech or voice commands and suppress any background noise. The acoustic echo canceller (AEC) 608 also uses a reference signal to cancel the speaker sound (e.g., from the speaker 630) picked up by the microphone array 602. The reference signal can be picked up by one or more reference microphones 610 near the speaker 630 or obtained from the audio signal (e.g., from the setup/equalizer unit 624) before sending the audio signal to the amplifier 628 for the speaker 630. The output of the AEC can then be sent to the noise reduction unit 612 to further reduce background noise.

The cleaned speech is then sent to the wake-up phrase recognizer 614, which can recognize the wake-up phrase predefined by the system 600. The system 600 can mute the speaker 630 to further improve the speech quality. Next, the automatic speech recognizer (ASR) 616 can recognize the voice command (e.g., the name of the song music) and then instruct the music retrieval unit 618 to retrieve music from the music library 620. In one embodiment, the wake-up phrase recognizer 614 and the ASR 616 can be combined into one unit. In addition, the retrieved music can then be separated by the sound separation unit 622, which can be similar to the sound separation unit 102A in Figure 1A.

Then, the settings/equalizer unit 624 can adjust the volume of each sound source and/or perform equalization of each track (gain of each frequency band or each instrument or vocal). Finally, as shown in the system 100B of FIG. 1B , the separated music tracks can be played from the speakers 630 as direct 3D sound (via amplifier 628), or as shown in the system 100A of FIG. 1A , the HRTF filter 626 can be used to process the separated tracks to generate binaural sound.

7A-7D illustrate a GUI 700 for user settings of 3D sound implemented in accordance with the present teachings, with selected listener positions being respectively within the band (FIGS. 7A-7C) and in front of the band (FIG. 7D).

In one implementation, the GUI 700 may be arranged so that all sound sources (e.g., from a music band on stage) are represented by band member icons on a virtual stage, and the listener is represented by a listener head icon (wearing headphones to highlight the location of the left and right ears) that can be freely moved around the stage by the user of the GUI 700. In another embodiment, all of the icons in FIGS. 7A-7D can be freely moved around the stage by touch by the user of the GUI 700.

In FIG. 7A , when the listener head icon is placed at the center of the virtual stage, the listener can hear binaural sounds and feel the sound field: vocal sounds are perceived as coming from the front, drum sounds from the right, bass sounds from the back, and other instruments (e.g., keyboards) from the left.

In FIG. 7B , when the listener head icon is placed over the band drummer icon, the listener will be able to hear a separate drum solo track.

In FIG. 7C , when the listener head icon is placed close to the drummer and bassist icons, the sounds of the drums and bass will be enhanced (e.g., increased volume), while the sounds from other instruments (e.g., vocals and others) can be relatively reduced (e.g., reduced volume), so that the listener can feel the enhanced bass and beat effects through the settings via GUI 700.

In FIG. 7D , another virtual 3D sound field setting is shown. In this setting, the listener can virtually feel and hear that the band is in front of him or her, even though this is not the case in a real-world music stage recording. The positions of all band member icons and the listener head icon can be moved anywhere on the display of GUI 700 to set and modify the virtual sound field and listening experience.

GUI 700 can also be applied to a remote control for controlling a TV with a direct 3D sound system, or other similar applications. For example, when a user is watching a movie, she or he can move the listener head icon closer to the voice icon so that the volume of the voice is enhanced and the volume of other background sounds (e.g., music) can be reduced, so that the user can hear the voice more clearly.

FIG. 8 illustrates a system 800 for generating 3D sound using a microphone array 802 , according to one implementation of the present description.

System 800 can be described as a 3D microphone system that can directly pick up and output 3D and binaural sounds. As mentioned herein, a 3D microphone system can include a microphone array system that can pick up sounds from different directions as well as spatial information about the location of the sound source. The system 800 can produce two outputs: (1) multiple tracks, each corresponding to sounds from a direction, where each of the multiple tracks can drive one or a group of speakers to generate a three-dimensional sound field; (2) binaural L and R tracks for earbuds or headphones to represent the 3D sound field in a virtual form.

Each microphone of microphone array 802 can have their signal processed by preamplifier/ADC unit 804. Preamplifier and analog-to-digital converter (ADC) can amplify analog signals and/or convert them into digital signals. Preamplifier and ADC are optional, depending on the microphone components selected for microphone array 802. For example, they may not be necessary for digital microphones.

The acoustic beamformer 806 can simultaneously form acoustic beam patterns pointing to different directions or different sound sources, as shown in FIG9B . Each beam enhances the sound from the “look” direction while suppressing the sound from other directions, thereby improving the signal-to-noise ratio (SNR) and isolating the sound from the “look” direction from the sound from other directions. If necessary, the noise reduction unit 808 can further reduce the background noise output by the beamformer. The output of the beamformer can include multiple audio tracks corresponding to sounds from different directions.

To generate direct 3D sound, multiple tracks can drive multiple amplifiers and speakers to construct a 3D sound field for the listener.

In order to generate binaural output, multiple tracks can be passed through multiple pairs of selected HRTF filters 810 to convert the spatial tracks into binaural sound. The HRTF filter can be selected based on the user's settings (e.g., via the output audio setting unit 814) or based on the actual spatial position of the sound source in the real world. In addition, the mixer 812 can then combine the HRTF outputs into a pair of binaural outputs for the left ear and the right ear respectively. The final binaural output represents the 3D sound field recorded by the microphone array 802.

Since the microphone array 802 has only two acoustic beam patterns pointing to the left and right respectively, the microphone array works as a stereo microphone, which is a special case of a 3D microphone.

9A-9B illustrate beam patterns of a 3D microphone 902 and a 3D microphone array 904 , respectively, with spatial noise cancellation implemented in accordance with the present teachings.

FIG. 9A shows a beam pattern of a 3D microphone 902 , which can pick up sounds from different directions as well as spatial information about the sound source.

9B shows a microphone array 904 (e.g., including a plurality of microphones 902) having beam patterns A and B formed by beamformers A and B, respectively, and the microphone array 904 is arranged to pick up sounds from two different sound sources A and B. Sound picked up from a sound source A in the “look” direction of one acoustic beam (e.g., beam pattern A) is often mixed with sounds picked up from other directions (e.g., the direction of sound source B). In order to cancel sounds from other directions, the 3D microphone array 904 can form another (some) beam patterns, such as beam pattern B, using the same microphone array 904. Sound picked up by beam pattern B can be used to cancel unwanted mixed sounds in the sound picked up by beam pattern A. Sound from the direction of sound source B that has been mixed with the sound from the “look” direction of beam pattern A can then be canceled from the output of beam pattern A. The cancellation algorithm can be provided by an acoustic echo canceller (AEC) unit 906.

FIG. 10 illustrates a conferencing system 1000 for generating three-dimensional sound implemented according to the present description.

The conference system 1000 may include a signal processing and computing unit 1002, a repository 1004 of head shape related transfer function (HRTF) filters, a display unit 1006 with a graphical user interface (GUI), an amplifier 1008, a headset or earphone 1010, and a speaker 1012. For example, the system 1000 may be implemented as software on a user's laptop, tablet, computer, or smart phone connected to a headset. Video and audio conferences (hereinafter referred to as "conferences") may also be referred to as teleconferences, virtual conferences, web conferences, webinars, or video conferences. One such conference may include multiple local and/or multiple remote attendees. In one implementation, the attendees may be connected via the Internet and a telephone network 1014. In another implementation, the conference may be controlled by a cloud server or a remote server via the Internet and a telephone network 1014.

The user of system 1000 may be one of the attendees of a conference or virtual concert. She or he is the owner of a laptop, tablet, computer or smart phone running a conference software with video and audio, and may be wearing a headset 1010. The term "speaker" or "attendee" refers to a person attending a conference. Speaker 1012 may be any device that can convert an audio signal into audible sound. Amplifier 1008 may be an electronic device or circuit for increasing signal power to drive speaker 1012 or headset 1010. Headset 1010 may be headphones, earmuffs, or in-ear audio devices.

The input signal (e.g., from the cloud via 1014) may include video, audio, and a speaker's identification (ID). The speaker's ID may associate the video and audio input to the attendee who is speaking. When there is no speaker's ID, a new speaker ID may be generated by the speaker ID unit 1016, as described below.

The speaker ID unit 1016 can be based on the speaker ID of the video conference session for the speaker, and the speaker ID is obtained from the conference software. In addition, the speaker ID unit 1016 can obtain the speaker ID from the microphone array (e.g., microphone array 802 of Figure 8 or 904 of Figure 9). For example, the microphone array beam pattern (e.g., beam patterns A and B) in Figure 9B can detect the direction of the speaker relative to the microphone array. Based on the detected direction, the system 1000 can detect the speaker ID. Further, the speaker ID unit 1016 can obtain the speaker ID based on the speaker ID algorithm. For example, based on a track composed of the voices of multiple speakers, the speaker ID system can have two periods: training and inference. During training, using available tags, the voice of each speaker is used to train a speaker-dependent model, one model for one speaker. If there is no tag, the speaker ID system can first perform unsupervised training, then annotate the voice from the track with the speaker ID, and then perform supervised training to generate a model for each speaker. During inference, given the conference audio, the speaker identification unit 1016 can use the trained model to process the input sound and identify the corresponding speaker. The model can be a Gaussian mixture model (GMM), a hidden Markov model (HMM), a DNN, a CNN, a LSTM, or a RNN.

For the attendee who is speaking, the video window associated with the attendee can be visually highlighted in the display/GUI 1006, so the user knows which attendee of the meeting is speaking, such as attendee 2 in FIG. 11 as described below. From the position of the speaker, for example, 50 degrees from the user, the system 1000 can retrieve a pair of corresponding HRTF filters from a pre-stored database or memory 1004. The signal processing unit 1002 can perform convolution calculations on the input mono signal using the HRTF filters from the pre-stored database or memory 1004. The output of the signal processing and calculation unit 1002 can have two channels of binaural sound for the left ear and the right ear, respectively. The user or attendee can wear a headset unit 1010 to hear binaural sound and experience 3D sound effects. For example, the user is not looking at the display 1006, but is wearing the headset 1010 and can still perceive which attendee is speaking from the 3D sound, so that the user can feel as if she or he is in a real conference room.

Based on multiple displays/GUIs 1006 and multiple speakers 1012 used in real conference rooms, each speaker 1012 can be dedicated to the sound of one speaker in one display/GUI 1006 in one location. In this case, the user does not need to use the headset 1010, and she or he can experience 3D sound from the speakers 1012. Multiple speakers can be placed in home theaters, movie theaters, sound bars, televisions, smart speakers, smart phones, mobile devices, handheld devices, laptops, PCs, cars, or anywhere there is more than one speaker or sound generator.

FIG. 11 shows a virtual conference room 1100 displayed by a GUI 1006 of a conference system 1000 for generating three-dimensional sound according to an implementation of the present invention.

The virtual conference room 1100 may have multiple windows (1102-1112) including videos of the user and conference attendees. The positions of the windows (1102-1112) may be arranged by the conference software (e.g., running on a laptop) or by the user (e.g., via the display/GUI 1006 of FIG. 10 ). For example, the user may move the windows (1102-1112) around to arrange the virtual conference room 1100. In one embodiment, the center of the conference room 1100 may include a virtual conference table.

As noted above, the virtual conference room 1100 may also be set up by the user so that attendees' video windows (1104-1112) may be virtually placed anywhere in the virtual conference room 1100 using a mouse, keypad, or touch screen, etc. From the position of the speaker (e.g., attendee 2) relative to the user (e.g., the angle from attendee 2's video window 1106 to the user's video window 1102), the relevant HRTF may be selected and automatically applied to attendee 2 as they speak.

method

In order to simplify the description, the method of this description is depicted and described as a series of actions. However, the actions according to this description can occur in various orders and/or simultaneously, and have other actions not presented and described herein. In addition, all the actions shown may not be required to implement the method according to the described subject matter. In addition, those skilled in the art will understand and appreciate that the method may be represented as a series of interrelated states via state diagrams or events alternatively. In addition, it should be understood that the method described in this specification can be stored on an article of manufacture, so that such a method is transmitted and transferred to a computing device. The term "article of manufacture" used herein is intended to cover a computer program accessible from any computer-readable device or storage medium.

The method may be performed by a processing device that may include hardware (e.g., circuits, dedicated logic), computer-readable instructions (e.g., running on a general-purpose computer system or a dedicated machine), or a combination of the two. The method and each of its individual functions, routines, subroutines, or operations may be performed by one or more processors of a computer device that performs the method. In some embodiments, the method may be performed by a single processing thread. Alternatively, the method may be performed by two or more processing threads, each thread performing one or more individual functions, routines, subroutines, or operations.

FIG. 12 shows a method 1200 for generating three-dimensional sound according to one implementation of the present invention.

In one implementation, the method 1200 may be performed by a signal processing unit of the system 100A of FIG. 1A or the subsystem 100B of FIG. 1B .

At 1202, the method includes receiving a three-dimensional space (e.g., 200A of FIG. 2A ) and a specification of a distribution grid of head-related transfer function (HRTF) filters (e.g., 200B of FIG. 2B ) defined on a grid in the three-dimensional space, wherein the three-dimensional space is presented in a user interface of a user interface device (e.g., GUI 110A of FIG. 1A ).

At 1204 , the method includes determining (eg, via the sound separation unit 102A of FIG. 1A ) a plurality of audio tracks (eg, separated audio tracks), wherein each of the plurality of audio tracks corresponds to a respective sound source (eg, a human voice).

At 1206 , the method includes representing a listener (eg, listener 202A of FIG. 2A ) and sound sources (eg, sound sources 204A of FIG. 2A ) of the plurality of audio tracks in three-dimensional space.

At 1208, the method includes generating multiple HRTF filters (e.g., 200B of FIG. 2B ) corresponding to a user setting of at least one of a listener position or a sound source position in three-dimensional space (e.g., via GUI 110A of FIG. 1A ), based on a distribution network of HRTF filters (e.g., stored in storage unit 104A of FIG. 1A ) and the positions of the sound source and the listener in the three-dimensional space.

At 1210, the method includes applying each of a plurality of HRTF filters (e.g., 200B of FIG. 2B) to a corresponding one of a plurality of separate audio tracks to generate a plurality of filtered audio tracks; and

At 1212, the method includes generating three-dimensional sound based on the filtered audio track.

FIG. 13 shows a method 1300 for generating three-dimensional sound according to one embodiment of the present description.

At 1302 , the method includes picking up sound from a plurality of sound sources using a microphone array (eg, microphone array 802 of FIG. 8 ) including a plurality of microphones (eg, microphone 902 of FIG. 9A ).

At 1304 , the method includes presenting three-dimensional sound using one or more speakers (eg, speakers 108B of FIG. 1B ).

At 1306 , the method includes removing echoes in the plurality of audio tracks using an acoustic echo cancellation unit (eg, AEC 608 of FIG. 6 ).

At 1308 , the method includes reducing noise components in the plurality of audio tracks using a noise reduction unit (eg, noise reduction unit 612 of FIG. 6 ).

At 1310, the method includes processing a plurality of audio tracks with a sound equalizer unit (eg, setup/equalizer unit 624 of FIG. 6).

At 1312, the method includes picking up a reference signal using a reference sound pickup circuit (e.g., reference microphone 610 of FIG. 6 ) located near one or more speakers (e.g., speaker 630 of FIG. 6 ), wherein an acoustic echo cancellation unit (e.g., AEC 608 of FIG. 6 ) will use the picked up reference signal to remove the echo.

At 1314 , the method includes recognizing a voice command using a voice recognition unit (eg, voice recognizer 616 of FIG. 6 ).

hardware

14 depicts a block diagram of a computer system 1400 that operates according to one or more aspects of the present description. In various examples, computer system 1400 may correspond to any signal processing unit/device described in connection with the systems presented herein, such as system 100A of FIG. 1A or system 100B of FIG. 1B .

In some implementations, the computer system 1400 can be connected to other computer systems (e.g., via a network, such as a local area network (LAN), an intranet, an extranet, or the Internet). The computer system 1400 can be executed as a server or client computer in a client-server environment, or as a peer computer in a peer-to-peer or distributed network environment. The computer system 1400 can be provided by the following devices: a personal computer (PC), a tablet computer, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a Web appliance, a server, a network router, a switch or a bridge, a computing device in a vehicle, a home, a room, or an office, or any device capable of executing a set of instructions (sequential instructions or other instructions) that specify the actions to be taken by the device. In addition, the term "computer" shall include any collection of computers, processors, chips, or SoCs that individually or collectively execute a set (or multiple sets) of instructions to perform any one or more of the methods described herein.

In another implementation, the machine operates as a standalone device, or can be connected (e.g., networked) to other machines. In a network deployment, the machine can operate as a server or client machine in a server-client or cloud network environment, or can act as a peer computer in a peer (or distributed) network environment. The machine can be an in-vehicle system, a wearable device, a personal computer (PC), a tablet computer, a hybrid tablet computer, a personal digital assistant (PDA), a mobile phone, or any machine (sequential or other) capable of executing instructions, which specify the actions to be performed by the machine. In addition, although only a single machine is shown, the term "machine" should also be considered to include any collection of machines that individually or collectively execute a set (or multiple sets) of instructions to perform any one or more methods discussed herein. Similarly, the term "processor-based system" should be considered to include any collection of one or more machines that are controlled or operated by a processor (e.g., a chip, a computer, or a cloud server) to individually or collectively execute instructions to perform one or more of the methods discussed herein.

The example computer system 1400 includes at least one processor 1402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both, a processor core, a computing node, a cloud server, etc.), a main memory 1404, and a static memory 1406, which communicate with each other via a link 1408 (e.g., a bus). The computer system 1400 may further include a video display unit 1410, an alphanumeric input device 1412 (e.g., a keyboard), and a user interface (UI) navigation device 1414 (e.g., a mouse). In one implementation, the video display unit 1410, the input device 1412, and the UI navigation device 1414 are incorporated into a touch screen display. The computer system 1400 may additionally include a storage device 1416 (e.g., a drive unit), a sound generating device 1418 (e.g., a speaker), a network interface device 1420, and one or more sensors 1422, such as a global positioning system (GPS) sensor, an accelerometer, a gyroscope, a position sensor, a motion sensor, a magnetometer, or other sensors.

The storage device 1416 includes a machine-readable medium 1424 on which is stored one or more sets of data structures and instructions 1426 (e.g., software) that are embodied or utilized by one or more of the methods or functions described herein. The instructions 1426 may also reside in whole or at least in part in the main memory 1404, the static memory 1406, and/or the processor 1402 when executed by the computer system 1400 having the main memory 1404, the static memory 1406, and the processor 1402 including the machine-readable medium.

Although the machine-readable medium 1424 is shown as a single medium in the example embodiment, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized, cloud, or distributed database and/or associated caches and servers) storing one or more instructions 1426. The term "machine-readable medium" should also be considered to include any tangible medium that is capable of storing, encoding, or carrying instructions for execution by a machine and causes the machine to perform any one or more of the methods described herein, or capable of storing, encoding, or carrying data structures utilized by or associated with such instructions. Thus, the term "machine-readable medium" should be considered to include, but is not limited to, solid-state memories, and optical and magnetic media. Specific examples of machine-readable media include volatile or non-volatile memories, including, but not limited to, for example, semiconductor memory devices (e.g., electrically programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs)) and flash memory devices; magnetic disks such as internal hard disks and removable disks; and CD-ROM and DVD-ROM disks.

Instructions 1426 may also be sent or received over a communication network 1428 using a transmission medium via the network interface device 1420 using any of a number of well-known transmission protocols (e.g., HTTP). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, a mobile phone network, a plain old telephone (POTS) network, and a wireless data network (e.g., Bluetooth, Wi-Fi, 3G, and 4G LTE/LTE-A or WiMAX networks). The term "transmission medium" shall be deemed to include any intangible medium that is capable of storing, encoding, or carrying instructions to be executed by the machine, and includes digital or analog signals or other intangible media to facilitate communication of such software.

The example computer system 1400 may also include an input/output controller 1430 to receive input and output requests from the at least one central processor 1402 and then send device-specific control signals to the devices they control. The input/output controller 1430 may free the at least one central processor 1402 from having to handle the details of controlling each individual type of device.

language

Unless expressly stated otherwise, terms such as "receiving", "associating", "determining", "updating" and the like refer to actions and processes performed or implemented by a computer system that operate on and transform data represented as physical (electronic) quantities within computer system registers and memories into other data similarly represented as physical quantities within computer system memories or registers or other such information storage, transmission or display devices. Moreover, as used herein, the terms "first", "second", "third", "fourth", etc. are intended to be used as labels to distinguish different elements and may not have an ordinal number meaning according to their numerical names.

The examples described herein also relate to a device for performing the methods described herein. The device may be specially constructed to perform the methods described herein, or it may include a general-purpose computer system selectively programmed by a computer program stored in the computer system. Such a computer program may be stored in a computer-readable tangible storage medium.

The methods and illustrative examples described herein are not specific to any particular computer or other device. Various general purpose systems may be used in accordance with the teachings described herein, or it may prove convenient to construct more specialized equipment to perform the methods and/or each of their individual functions, routines, subroutines, or operations. Examples of structures for various of these systems are set forth in the description above.

The above description is illustrative rather than restrictive. Although the present description has been described with reference to specific illustrative examples and implementations, it will be appreciated that the present description is not limited to the described examples and implementations. The scope of the present description should be determined with reference to the appended claims and the full scope of equivalents to which the claims are entitled.

Claims (21)

1.A three-dimensional sound generation system, comprising:

one or more processing devices for:

Receiving a series flow sound consisting of one or more sound sources, wherein each sound source corresponds to one or more sound types;

Receiving a specification, preset, or setting of a three-dimensional space including a location of one or more listeners, one or more sound source locations, one or more sound generating devices, and locations thereof, to generate a desired sound field in the three-dimensional space;

separating the streaming sound into one or more tracks, wherein each track includes the one or more sound types;

combining the separated audio tracks to generate one or more audio channels based on the specifications, preset values, or settings;

Driving the one or more sound generating devices with the one or more audio channels to generate the desired sound field such that the desired sound field produces an equivalent rendering effect that sounds are heard from the one or more sound source locations at the one or more listener locations in the three-dimensional space.

2. The three-dimensional sound generation system of claim 1, wherein the three-dimensional space is an acoustic virtual space of real space comprising a stage space of a band or concert, or non-real space comprising a telephone, video, and/or web conference space, and wherein the specifications, preset values, and/or settings are implemented or modified or updated by a user via one or more user interfaces or one or more computer programs, wherein the user interfaces comprise at least a graphical interface.

3. The three-dimensional sound generation system of claim 1, wherein separating the streaming sound into the one or more audio tracks and combining the separated audio tracks to generate one or more audio channels are implemented in combination or separately by at least one machine learning system, at least one artificial neural network system, at least one digital signal processing system, or a combination thereof.

4. A three-dimensional sound generation system as in claim 1 wherein the sound generation device is a binaural sound generation device comprising earplugs, headphones, or headsets, each track being processed by a retrieved Head Related Transfer Function (HRTF) of a corresponding location and combined into a binaural audio signal to drive the binaural sound generation device to generate the desired sound field; wherein the binaural sound generating device is a stand alone device or part of another device comprising at least an in-vehicle system, a wearable apparatus, a computer, a personal digital assistant, a mobile phone or a display.

5. The three-dimensional sound generation system of claim 1, wherein the sound generation device is an loud sound generation device comprising speakers, the three-dimensional sound generation system being implemented in an open or closed space, the closed space comprising a vehicle and a room, the open space comprising an outdoor venue or a network space; wired or wireless connections between the one or more processing devices and with the one or more sound generating devices, including the internet, wi-Fi, bluetooth, bus, local network, or cloud network.

6. A three-dimensional sound generation system, comprising:

one or more processing devices for:

Receiving a series flow sound consisting of one or more sound sources, wherein each sound source corresponds to one or more sound types;

Receiving one or more desired output sound types and specifications, preset, online setting or user setting of corresponding sound volumes thereof;

separating the streaming sound into one or more sound tracks according to the above specifications or settings;

The user adjusts the volume or frequency response of each soundtrack between 0 and maximum, respectively, or combines into one or more audio channel signals according to specifications and settings to drive a sound generating device comprising a speaker, earplug, earphone or headset for converting the audio channel signals into sound.

7. The three-dimensional sound generation system of claim 6, wherein each output audio signal is solo, vocal only, speech only, instrument only, or a combination thereof, and each soundtrack separation is accomplished by at least one machine learning system, at least one artificial neural network system, at least one signal processing system, or a combination thereof.

8. A three-dimensional sound generation system for video conferencing, virtual concert, or cloud communication, comprising:

A plurality of user terminals are connected through wired or wireless communication;

each user terminal comprises at least one receiving device, at least one processing device and at least one sound generating device;

The user side receives streaming sound formed by sound sources of other user sides or video signals comprising the streaming sound, and separates the streaming sound into one or more sound tracks according to different user sides, wherein each sound track comprises one or more sound source signals;

The user side converts the separated one or more audio tracks into one or more channel signals according to specifications, preset or settings, and drives the at least one sound generating device to generate a sound field, wherein each user side sound source has a specific position in the sound field.

9. A three-dimensional sound generation system according to claim 8, wherein the specification, preset or setting is implemented or on-line adjusted by a graphic display or a computer program in a user interface or user interface device of each user side and/or a designated user side, when there is a video display, the sound source position is set to be at or near the position of the user image display, and when the display position or virtual position of the other user side in the specification, preset or setting is changed, the sound generation device generates a correspondingly changed sound field.

10. The three-dimensional sound generation system of claim 8, wherein separating the streaming sound into the plurality of audio tracks is accomplished by at least one machine learning system, at least one artificial neural network system, at least one digital signal processing system, or a combination thereof, and the streaming sound is separated into the plurality of audio tracks by identifying or detecting different sound source locations from microphone array pickup and/or from known user side sources.

11. The three-dimensional sound generation system of claim 8, wherein the sound generation device comprises at least one speaker, earplug, earphone, headset, or any device capable of converting the channel signal into sound field sound at the particular location.

12. The three-dimensional sound generation system of claim 8, wherein each of the separated sound tracks is processed by a Head Related Transfer Function (HRTF), the function selected according to specifications, preset values, a setting at a user side, or a sound source position displayed at the user side, and the processed sound tracks are combined to generate the channel signal to drive earplugs, headphones, or a headset to generate the sound field.

13. A three-dimensional sound generation system, comprising:

A sound source device and a sound receiving device connected by wired or wireless communication, wherein each device has or is associated with one or more sensors, the sound receiving device comprising one or more sound generating devices;

The sound receiving apparatus receiving an audio signal from a sound source apparatus and receiving relative position information between the sound source apparatus and the sound receiving apparatus, the relative position information being obtained by processing a signal from the sensor,

The sound receiving device or the sound source device or both cooperatively retrieve a related Head Related Transfer Function (HRTF) based on the relative position information, and process the received audio signal using the retrieved HRTF, and drive the sound generating device on the sound receiving device with the processed audio signal to generate a sound field to perceive a direction of sound in the sound field from the sound source device.

14. A three-dimensional sound generation system with a video display, comprising:

One or more clients;

Each user terminal comprises at least one receiving device, at least one processing device, at least one sound generating device and at least one display device;

each user terminal receives a streaming signal composed of a video signal and an audio signal, wherein the video signal is used for displaying an object at a specific position;

identifying a corresponding sound field from the streaming signal, including one or more sound sources associated with the particular location;

Separating the audio signal into one or more audio tracks, wherein each audio track contains one or more sound sources in a corresponding sound field of the audio signal;

combining the separated audio tracks into an audio channel signal according to the identified sound source settings;

the at least one sound generating device is driven using the above-described audio channel signal to reproduce the sound field at the displayed specific position to achieve a rendering effect of sound from the displayed object at the specific position.

15. The three-dimensional sound generation system of claim 14, wherein the object and its particular location are identified from one or a series of video frames of the video signal by at least one machine learning system, at least one artificial neural network system, at least one digital signal processing system, or a combination thereof, and the steps of separating the audio signal into the one or more audio tracks, combining the separated audio tracks into an audio channel signal, and driving the sound generation device are implemented, respectively, by at least one machine learning system, at least one artificial neural network system, at least one digital signal processing system, or a combination thereof, in combination.

16. The three-dimensional sound generation system of claim 14, wherein the sound generation device comprises a speaker, an earplug, an earphone, or a headset; when the sound generating device comprises an earplug, an earphone or a headset, the user side retrieves a Head Related Transfer Function (HRTF) corresponding to the sound source position for processing the separated audio track.

17. A computer method for three-dimensional sound generation, comprising:

Receiving a series flow sound consisting of one or more sound sources, wherein each sound source corresponds to one or more sound types;

Receiving a specification, preset, or setting of a three-dimensional space including a location of one or more listeners, one or more sound source locations, one or more sound generating devices, and locations thereof, to generate a desired sound field in the three-dimensional space;

separating the streaming sound into one or more tracks, wherein each track includes the one or more sound types;

combining the separated audio tracks to generate one or more audio channels based on the specifications, preset values, or settings;

Driving the one or more sound generating devices with the one or more audio channels to generate the desired sound field such that the desired sound field produces an equivalent rendering effect that sounds are heard from the one or more sound source locations at the one or more listener locations in the three-dimensional space.

18. The computer method of claim 17, wherein the three-dimensional space is an acoustic virtual space of real space comprising a stage space of a band or concert, or non-real space comprising a telephone, video, and/or web conference space, and wherein the specifications, preset values, and/or settings are implemented or modified or updated by a user via one or more user interfaces or one or more computer programs, wherein the user interfaces comprise at least a graphical interface.

19. The computer method of claim 17, wherein the steps of separating the streaming sound into the one or more audio tracks and combining the separated audio tracks to generate one or more audio channels are implemented in combination or separately by at least one machine learning system, at least one artificial neural network system, at least one digital signal processing system, or a combination thereof.

20. A computer method according to claim 17, wherein the sound generating device is a binaural sound generating device comprising earplugs, headphones or headsets, each track being processed by a retrieved Head Related Transfer Function (HRTF) of the corresponding location and combined into a binaural audio signal to drive the binaural sound generating device to generate the desired sound field; wherein the binaural sound generating device is a stand alone device or part of another device comprising at least an in-vehicle system, a wearable apparatus, a computer, a personal digital assistant, a mobile phone or a display.

21. A computer method according to claim 17, wherein the sound generating device is a loud sound generating device comprising loudspeakers, the computer method being implemented in an open or closed space, the closed space comprising a vehicle and a room, the open space comprising an outdoor venue or a network space; wired or wireless connections between the one or more processing devices and with the one or more sound generating devices, including the internet, wi-Fi, bluetooth, bus, local network, or cloud network.