JP2020508590A - Apparatus and method for downmixing multi-channel audio signals - Google Patents

Abstract

A method for processing a multi-channel input audio signal is performed on a computing device. The method comprises the steps of selecting a left input channel and a right input channel from a multi-channel input audio signal, wherein the left input channel and the right input channel are a pair of spatially symmetric signal sources. And generating one or more cross-channel features from the left and right input channels, and processing the left and right input channels to produce a left intermediate Generating a channel and a right intermediate channel; and combining each of the left and right intermediate channels with a third input channel of the multi-channel input audio signal to form a two-channel output audio signal. Including. [Selection diagram] FIG. 4B

Description

Technical field
[0001] This application relates generally to audio signal processing, and in particular, to a computer-implemented method, implementation apparatus, and computer-usable program code for downmixing a multi-channel audio signal.

background
[0002] Surround sound is a technique for generating, sending, and playing audio using multiple audio channels surrounding a listener. This is typically achieved by a plurality of individual audio channels. Two popular configurations of multi-channel or surround sound configurations are 5.1 surround sound and 7.1 surround sound. The 5.1 surround sound configuration consists of one pair of front speakers (L and R), one center front channel (C), one pair of side speakers (Ls and Rs), and one bass effect ( LFE), and the order is L, C, R, Ls, Rs, and LFE as in the related art. The 7.1 surround sound configuration consists of one pair of front speakers (L and R), one center front channel (C), one pair of side surround speakers (Lss and Rss), and one pair of rear speakers. Consists of surround speakers (Lrs and Rrs) and one bass effect (LFE), in the order of L, C, R, Lss, Rss, Lrs, Rrs, and LFE as is conventional.

  [0003] Downmixing is a process of converting a program having a multi-channel configuration (for example, a multi-channel audio file) into a program with relatively few channels. For example, using a two-channel stereo playback system to provide a good listening experience to the listener, while using a two-channel stereo playback system to provide a 5.1 surround sound file or 7.1 surround sound file. Files can be downmixed and played.

  [0004] In a conventional downmix process, each of the multi-channel audio inputs is processed individually and separately using a respective processor to produce one or two channel outputs. The process on each channel does not properly account for meaningful information between the speaker pairs. Thus, the audio output obtained using these conventional downmixing processes is relatively inaccurate and may detract from an immersive listening experience.

Overview
[0005] One object of the present application is to develop an audio downmix pipeline that processes input audio channels in pairs. The resulting downmixed output has relatively good accuracy, while maintaining the spatial information of the original multi-channel audio stream. The number of channel inputs and channel outputs can be any number that is meaningful under its definition, but in the following description, as an example, a downmix from 5.1 to stereo and a 7.1 to stereo Use downmix to.

  [0006] According to a first aspect of the present application, a method of processing a multi-channel input audio signal is performed in a computing device having one or more processors, a memory, and a plurality of program modules. A plurality of program modules are stored in memory and executed by one or more processors. The method comprises the steps of selecting a left input channel and a right input channel from a multi-channel input audio signal, wherein the left input channel and the right input channel are a pair of spatially symmetric signal sources. And generating one or more cross-channel features from the left and right input channels, and processing the left and right input channels according to the cross-channel features to produce a left intermediate Generating a channel and a right intermediate channel; and combining each of the left and right intermediate channels with a third input channel of the multi-channel input audio signal to form a two-channel output audio signal. Including.

  [0007] According to another aspect of the present application, a computing device includes one or more processors, a memory, and a plurality of programs stored in the memory and executed by the one or more processors. Module. The plurality of program modules, when executed by one or more processors, enable the computing device to perform the foregoing method of processing a multi-channel input audio signal. According to yet another aspect of the present application, a computer program product is stored on a persistent, computer-readable storage medium in conjunction with a computing device having one or more processors, the computer program product comprising: The product includes a plurality of program modules that, when executed by one or more processors, enable the computing device to perform the aforementioned methods of processing multi-channel audio signals. .

BRIEF DESCRIPTION OF THE FIGURES
[0008] The accompanying drawings are included to provide a further understanding of each embodiment, are incorporated in and constitute a part of this specification, illustrate the embodiments described, and together with the description, It helps to explain the underlying principles. Like reference numbers refer to corresponding parts.

[0009] FIG. 4 is a block diagram illustrating a conventional downmixing process performed on a 5.1 input signal, according to some embodiments. [0010] FIG. 4 is a block diagram illustrating a conventional LoRo downmixing process performed on a 5.1 input signal, according to some embodiments. [0011] FIG. 4 is a block diagram illustrating a conventional downmixing process for virtualizing or spatializing surround sound from a 7.1 input signal, according to some embodiments. [0012] FIG. 1 shows a block diagram of a data processing system configured to perform audio downmixing according to an exemplary embodiment of the present application. [0013] FIG. 3 is a block diagram illustrating an audio downmix pipeline for processing a multi-channel input signal, according to some embodiments. [0013] FIG. 3 is a block diagram illustrating an audio downmix pipeline for processing a multi-channel input signal, according to some embodiments. [0014] FIG. 4 is a block diagram illustrating a signal workflow including a PROC applied to an input pair, according to some embodiments. [0015] FIG. 4 is a block diagram illustrating a signal workflow applied to 7.1 surround sound, according to some embodiments. [0016] According to some embodiments, a user of a software application used to manage the implementation of the signal pipeline, as described with reference to FIG. 4B for a 7.1 surround sound file. Shows an interface or plug-in component of a software application. 5 is a flowchart illustrating a process for downmixing a multi-channel audio signal, according to some embodiments. 5 is a flowchart illustrating a process for downmixing a multi-channel audio signal, according to some embodiments. 5 is a flowchart illustrating a process for downmixing a multi-channel audio signal, according to some embodiments.

Detailed description
Next, embodiments will be described in detail. Examples of embodiments are shown in the accompanying drawings. In the following detailed description, numerous non-limiting, specific details are set forth in order to assist in understanding the subject matter presented herein. However, it will be apparent to one skilled in the art that various alternatives may be used and the subject matter may be practiced without these specific details without departing from the scope of the claims. For example, it will be apparent to one skilled in the art that the subject matter presented herein can be implemented on many types of wireless communication systems, such as smartphones and tablets.

  [0019] Referring now to the figures, an exemplary block diagram of a data processing environment in which the exemplary embodiments may be implemented is presented. It is to be understood that these figures are merely exemplary, and do not represent or imply any limitations on the environment in which various embodiments may be implemented. Many modifications may be made to the depicted environment.

  FIG. 1A is a block diagram illustrating a conventional downmixing process performed on a 5.1 input signal. As shown in FIG. 1A, each input channel (ie, L, C, R, Ls, Rs, and LFE) is processed separately, regardless of its relationship to other channels, and its respective processor module ( PROC). The processor may include one or more sub-modules (not shown), such as gain, time delay, low pass filters, and / or other sound processing modules. The output of each process module on each channel may include one or more channels, depending on the type of process performed on this process module. Finally, the outputs are summed (ie, Σ) into two-channel audio (ie, L and R output channels) in this example.

  FIG. 1B is a block diagram illustrating a conventional left only / right only (LoRo) downmixing process performed on a 5.1 input signal. Each input channel (ie, L, C, R, Ls, Rs, and LFE) will pass individually through the gain module. Adjusting the gain depends on the physical location, as if reproduced by a surround sound system. The surround channel may attenuate beyond the L / R channel, but the relationship between the left and right sides is ignored. The left channel output is created by adding all the channels from the left side plus the attenuated C and LFE signals. In the stereo reproduction setting, the center channel is divided into two since no physical speakers are arranged on the center line. The LFE is also split into two channels. The right channel output is created by summing all channels from the right side plus the attenuated C and LFE signals. In this example, every input channel is processed differently by its individual processors using a simple gain module that receives one channel input and produces one or two channel outputs. Finally, all PROC outputs will be summed based on the intended reproduction position of the input.

  FIG. 1C is a block diagram illustrating a conventional downmixing process for virtualizing or spatializing surround sound from a 7.1 input signal. In this example, each channel of the input multi-channel audio (ie, L, C, R, Lss, Rss, Lrs, Rrs, and LFE), in addition to its individual gain, has its own head related transfer function (HRTF). ), Which represents the intended position of each physical loudspeaker to produce a two-channel output. For example, the left channel input will be processed by its HRTF representing the left channel of the surround sound system speaker. Similar processing will be applied to all other input channels. All of the two sets of channel outputs based on each input channel are summed, resulting in a left channel output and a right channel output, respectively. In this example, every input channel is also processed differently by individual processors (eg, consisting of a gain module and an HRTF filter). This processor takes one channel input and produces a two channel output. All the two channel outputs are summed up to a final two channel output.

  FIG. 2 shows a block diagram of a data processing system 100 configured to perform audio downmixing according to an exemplary embodiment of the present application. In this illustrative example, data processing system 100 includes processor unit 104, memory 106, persistent storage 108, communication unit 110, input / output (I / O) unit 112, display 114, And a communication mechanism 102 that enables communication with one or more speakers 116. Note that the speaker 116 may be internal to the data processing system 100 or external to the data processing system 100. In some embodiments, data processing system 100 includes a laptop computer, a desktop computer, a tablet computer, a mobile phone (such as a smartphone), a multimedia player device, a navigation device, an educational device (such as a child's learning toy). , A game system, an audio / video (AV) receiver, or a controller (eg, a home or industrial controller).

  [0024] Processor unit 104 serves to execute instructions for a software program that can be loaded into memory 106. Processor unit 104 may be a set of one or more processors, or may be a multi-processor core, depending on the particular implementation. Further, processor unit 104 may be implemented using one or more heterogeneous processor systems, where the main processor resides on a single chip with the secondary processor. As another illustrative example, processor unit 104 may be a symmetric multiprocessor system that includes multiple processors of the same type.

  [0025] In these examples, memory 106 may be a random access memory, or any other suitable volatile or non-volatile storage. Persistent storage 108 may take various forms, depending on the particular implementation. For example, persistent storage 108 may include one or more components or devices, such as a hard drive, flash memory, rewritable optical disk, rewritable magnetic tape, or some combination thereof. The media used by persistent storage 108 may be removable. For example, a removable hard drive may be used for persistent storage 108.

  In these examples, the communication unit 110 implements communication with another data processing system or data processing device. In these examples, communication unit 110 is a network interface card. Communication unit 110 may implement communication by using either a physical or wireless communication link, or both.

  [0027] The input / output unit 112 enables data input and output with other devices that may be connected to the data processing system 100. For example, input / output device 112 may provide a connection for user input via a keyboard and mouse. Further, input / output unit 112 may send output to a printer. Display device 114 provides a mechanism for displaying information to a user. The speaker 116 plays a sound toward the user.

  [0028] Instructions for the operating system and applications or programs are located in persistent storage 108. These instructions may be loaded into memory 106 for execution by processor unit 104. The processes of the various embodiments, as described below, may be performed by processor unit 104 using computer-implemented instructions, which may be located in a memory, such as memory 106. Good. These instructions are referred to as program code (or module), computer-usable program code (or module), or computer-readable program code (or module). The processor in processor unit 104 may read and execute. The program code (or module) in various embodiments may be embodied on various physical or tangible computer readable media, such as memory 106 or persistent storage 108.

  [0029] The program code / modules 120 are disposed in a functional form on a computer-readable storage medium 118, which is selectively removable and executed by the processor unit 104. , May be loaded or transferred to the data processing system 100. Program code / module 120 and computer-readable storage medium 118 form computer program product 122 in these examples. In one example, computer readable storage medium 118 may be in a tangible form, such as, for example, an optical or magnetic disk, which is transferred to a storage device, such as a hard drive that is part of persistent storage 108. Inserted or located on a drive or other device that is part of the persistent storage device 108 for storage. In the tangible form, computer readable storage medium 118 may also take the form of a persistent storage, such as a hard drive, thumb drive, or flash memory connected to data processing system 100. The tangible form of the computer-readable storage medium 118 is also called a computer-recordable storage medium. In some cases, computer readable storage medium 118 may not be removable from data processing system 100.

  [0030] Alternatively, the program code / module 120 may process data from the computer readable storage medium 118 via a communication link to the communication unit 110 and / or via a connection to the input / output unit 112. It may be transferred to the system 100. In an illustrative example, the communication links and / or connections may be physical or wireless. Computer-readable media may also take the form of intangible media, such as communication links or wireless transmissions containing the program code / modules.

  [0031] The various components illustrated for data processing system 100 do not place any structural limitations on the manner in which various embodiments may be implemented. Various exemplary embodiments may be implemented in a data processing system that includes components in addition to or in place of those illustrated in data processing system 100. The other components shown in FIG. 1 can be modified from the illustrated illustrative example.

  As an example, the storage device in data processing system 100 is any hardware device that can store data. Memory 106, persistent storage 108, and computer-readable storage medium 118 are examples of storage in tangible form.

  [0033] In another example, the bus system may be used to implement the communication mechanism 102 and may consist of one or more buses, such as a system bus or an input / output bus. The bus system may be implemented using any suitable type of architecture that allows for the transfer of data between various components or devices attached to the bus system. Further, a communication unit may include one or more devices used to send and receive data, such as a modem or a network adapter. Further, the memory may be, for example, the memory 106 or a cache such as found on a controller hub of the interface and memory that may reside within the communication mechanism 102.

  [0034] To overcome the problems with the conventional approaches described in the background of the present application, various embodiments of the present application are described below, which downmix the input audio channels pair by pair. And a system and method for achieving better audio information accuracy and preserving the spatial information of the original multi-channel audio stream. Unlike conventional methods, the relationship between the input channels is considered as a pair. Each pair enters a processor. The pair information is compared and analyzed with each other to produce a denser sound image and better spatial results. In order for this pair to be meaningful, at least one of the modules in the process needs to cross-reference information between the pair. The two channel outputs of each pair will be summed with the single channel to produce a left channel output and a right channel output.

  [0035] FIGS. 3A-3B are block diagrams illustrating an audio downmix pipeline for processing a multi-channel input signal, according to some embodiments. The multi-channel input signal in FIG. 3A includes a left front channel (L), a center front channel (C), a right front channel (R), a left side channel (Ls), a right side channel (Rs), And 5.1 surround sound file 210, including a bass effect (LFE). The multi-channel input signal in FIG. 3B includes a left front channel (L), a center front channel (C), a right front channel (R), a left side surround channel (Lss), and a right side surround channel. (Rss), 7.1 surround sound file 240 including left rear surround channel (Lrs), right rear surround channel (Rrs), and bass effect (LFE).

  [0036] In some embodiments, the system selects one or more input pairs from the multi-channel input signal. In some embodiments, the input pairs correspond to two sets of audio streams intended to be reproduced on symmetrically placed speakers. Thus, this input pair includes a pair of spatially symmetric signal sources. In some embodiments, the input pair includes two audio channels on two sides having the same angle from the centerline (ie, the left and right sides). For example, a front pair includes a left front (L) channel and a right front (R) channel, each at a 30 ° angle to the left and right of the centerline. In another example, the rear pair of the Dolby 7.1 surround sound setting has a left rear surround (Lrs) channel and a right rear surround (Rrs) channel at 135 ° to the left and right of the centerline, respectively. Place at an angle. Each selected input pair is then sent to a respective processor (PROC) that produces an output audio pair.

  [0037] In some embodiments, such as that shown in Figure 3A, out of multiple input channels of a 5.1 surround sound file, the system includes a left front channel (L) and a right front channel (R). Are selected as a pair 222, and the left side channel (Ls) and the right side channel (Rs) are selected as a pair 224. In some embodiments, such as that shown in FIG. 3B, of the multiple input channels of the 7.1 surround sound file, the system may pair 242 the left front channel (L) and the right front channel (R). And the left side surround channel (Lss) and the right side surround channel (Rss) are selected as a pair 244, and the left rear surround channel (Lrs) and the right rear surround channel (Rrs) are selected. Select as pair 246. The channel on the centerline (eg, the center front C channel) and the omni-directional channel (eg, LFE) are single channels and do not pair with any other channels.

  [0038] In some embodiments, each pair will enter a respective processor. For example, in FIG. 3A, pairs 222 and 224 are transmitted to PROCs 232 and 234, respectively, and in FIG. 3B, pairs 242, 244, and 246 are transmitted to PROCs 252, 254, and 256, respectively. In such a process, pairs of information are compared and analyzed with each other to produce a denser sound image and better spatial results. To make this pair meaningful, at least one of the one or more modules in each processor PROC needs to cross-reference information between the two channels in the pair. The output signal of each processor PROC includes two channels, and the two channel outputs of each pair are summed with a single channel (Σ) to form the left channel output (L, as shown in FIGS. 3A and 3B, respectively). ') And an output signal including the right channel output (R').

  [0039] In some embodiments, the pair processor (PROC) can be any feasible "two-in-two-out" audio signal processor. As described above, the processor is comprised of at least one module that incorporates cross-channel features from the input pair. In some embodiments, a pair processor (PROC) includes one or more modules that extract signal information from an input pair. Based on the input pair signal information, the pair processor (PROC) then modifies the output stream for each pair.

  [0040] In some embodiments, a pair processor (PROC) is comprised of a plurality of different components (or modules), including channel-dependent components and / or channel-independent components. In some embodiments, the channel dependent component performs a multi-channel in-multi-channel out process, for example, a two-in-two-out process on a paired processor. In some embodiments, the channel-dependent component generates a plurality of output channels, each based on two or more input channels. In some embodiments, the channel-dependent component uses information from the input signal to adjust the process based on the extracted cross-channel features. In some embodiments, the cross-channel feature may be a comparison of the volume of each input channel, a relationship between frequency spectral characteristics (eg, amplitude and / or phase) of the left and right input channels, and / or a signal start of the left and right input channels. And the difference in amplitude. In some embodiments, a pair processor (PROC) includes one or more channel-dependent components, such as a mid / side (M / S) mixer, and / or a width controller (WC). For example, the M / S mixer can generate a mid signal and a side signal using the sum and difference of the input left and right signals. In another example, the M / S mixer can generate a mid signal and a side signal by comparing overlapping regions of the frequency spectrum of the input signal.

  [0041] In some embodiments, the channel-independent component is a multi-channel in-multi-channel out process (including two-in-two-out) that processes each respective channel of the multi-channel input file separately. In some embodiments, the multiple channels are divided into the same number of mono signals, each mono signal is processed independently, and the channel independent components, as in the case where the processed mono signals of each of the multiple channels are summed together. Handles multi-channel input files. In some embodiments, the pair processor (PROC) includes one or more channel-independent components, such as an equalizer (EQ) and / or a dynamic range compressor (DRC). For example, an equalizer (EQ) module captures each channel individually and creates a corresponding channel without using any information from other channels. The result of the equalizer (EQ) is the same as if each channel were separately equalized with the same input parameters.

  FIG. 4A is a block diagram illustrating a signal workflow including a PROC 420 applied to an input pair 410, according to some embodiments. In some embodiments, PROC 420 includes multiple modules (also referred to as components) configured to process input pair signal 410. In some embodiments, PROC 420 can be any paired processor, such as PROC 232, PROC 234, PROC 252, PROC 254, or PROC 256, as shown in FIGS. 3A-3B. Thus, pair processor 420 can be applied to any input pair, such as pair 222, pair 224, pair 242, pair 244, or pair 246, as shown in FIGS. 3A-3B. In some embodiments, the pair processor 420 includes a mid / side (M / S) mixer 422, an equalizer (EQ) 428, a dynamic range compressor (DRC) 430, and a cross It includes a talk cancellation (XTC) module 432. The output signal of PROC 420 is further processed in width controller (WC) 434 and another dynamic range compressor (DRC) 436 to obtain output pair 440 as shown in FIG. 4A.

  In some embodiments, data processing system 100 first transmits left and right input signals 410 to M / S mixer 422. In some embodiments, M / S mixer 422 generates three components from input pair 410: two side components (S) 424, i.e., left and right sides, and one mid component (M) 426. Is a mixing tool. The left side component represents a sound source that appears only in the left channel, and the right side component corresponds to a sound that appears only in the right channel. The central component is a sound source that appears only at the center of the sound image of the sound stage, for example, main musical elements and dialogs.

  [0044] In doing so, the M / S mixer 422 separates information that is useful for subsequent improvement of the various sound stages and minimizes unnecessary distortion (eg, coloring) in sound quality. Further, this step also helps reduce the correlation between the left and right components. In some embodiments, M / S mixer 422 analyzes the sound image and estimates the sound coming from the center, left, and right. The M / S mixer 422 then splits the two input channels, the left and right signals 410, into one channel mid signal 426 and two channel side signals 424. A more detailed description of the M / S mixer 422 can be found in PCT Application No. PCT / US2015 / 057616, filed October 27, 2015, entitled "Apparatus and Method for Sound Stage Improvements," Is incorporated by reference in its entirety.

  Next, the system transmits each side signal 424 to an equalizer (EQ) 428 to adjust the frequency component of the side signal 424. In some embodiments, the EQ 428 that processes the two side signals 424 includes one or more multi-band equalizers to perform bandpass filtering on the two side signals. In some embodiments, the multi-band equalizer applied to each side signal is the same. In some other embodiments, the multi-band equalizer applied to one side signal is not the same as the multi-band equalizer applied to the other side signal. Nevertheless, their function is to preserve the original timbre of the audio signal and avoid ambiguous spatial cues present in these two signals. In some embodiments, the EQ 428 may be used to select a target sound source based on a spectral analysis of the two side components. In some embodiments, such as shown in FIG. 4A, the EQ 428 produces two output signals 450 and 452. In some embodiments, each of output signals 450 and 452 is a two-channel audio signal. In some embodiments, the EQ 428 applies a respective bandpass filter to each of the two channel side signals 424 to obtain a bandpass filtered signal 450.

  [0046] In some embodiments, EQ 428 also includes a frequency between the input signal of EQ 428 (ie, the two-channel side signal 424) and the output signal of EQ 428 (ie, the two-channel bandpass filtered signal 450). A residual signal 452 is generated based on the band difference. In some embodiments, the data processing system 100 subtracts the left side component and the right side component from the bandpass filtered left side component and the bandpass filtered right side component, respectively, to provide a left side residual component. And a residual component on the right side is generated. In some embodiments, each amplifier is applied to the residual signal and the resulting signal from crosstalk cancellation to adjust the gain of the two signals and then combine the signals together. A more detailed description of EQ428 can be found in PCT Application No. PCT / US2015 / 057616, filed Oct. 27, 2015, entitled "Apparatus and Method for Sound Stage Improvement", which is hereby incorporated by reference. Incorporate as a whole.

  In some embodiments, bandpass filtered signal 450 is transmitted to dynamic range compressor (DRC) 430. In some embodiments, the DRC 430 includes a bandpass filter (different from the EQ428 bandpass filter) for amplifying the two audio signals (ie, the two-channel bandpass filtered signal 450) within a predetermined frequency range. To maximize the sound stage improvement effect provided by the crosstalk cancel block (XTC) 432. In some embodiments, a user (e.g., an acoustician) can adjust the bandpass filter of DRC 430 to exclude certain frequency bands. By doing so, the user can emphasize a particular acoustic event of his choice. For example, after performing equalization on the left and right side components using the first bandpass filter of EQ 428, data processing system 100 may use the second bandpass filter of DRC 430 to perform left side component filtering. A predetermined frequency band is removed from the component and the right side component. Typical bandpass filters used in the EQ block 428 and the DRC block 430 include a fourth-order filter or a Butterworth filter. In some embodiments, after performing equalization to the left and right side components using the first bandpass filter of EQ 428, the data processing system 100 may reduce the first dynamic range compression by the DRC 430 to the left side component. This is performed on the component and the right side component to emphasize a predetermined frequency band for other frequencies. A more detailed description of DRC 430 can be found in PCT Application No. PCT / US2015 / 057616, filed Oct. 27, 2015, entitled "Apparatus and Method for Sound Stage Improvement", which is hereby incorporated by reference. Incorporate as a whole.

[0048] In some embodiments, the output signal from DRC 430 is then sent to a crosstalk cancellation (XTC) module 432 for performing a crosstalk cancellation process. Crosstalk is an inherent problem when playing stereo (ie, two-channel) loudspeakers. When the sound reaches the ears on the opposite side of each speaker, crosstalk occurs, and the original signal is colored with an unnecessary spectrum. The solution to this problem is the crosstalk cancellation (XTC) algorithm. One type of XTC algorithm uses two general-purpose directional binaural transfer functions, such as a head-related transfer function (HRTF) and / or a binaural room impulse response (BRIR), to provide two physical loudspeakers to the location of the listener. Represents the angle of Another type of XTC algorithm system is a recursive crosstalk cancellation method that does not require head related transfer function (HRTF), binaural room impulse response (BRIR), or any other binaural transfer function. The basic algorithm can be formulated as follows.
left [n] = left [n] -A _L * right [n-d _L ]
_{right [n] = right [n} ] -A R * left [n-d R]
Here, A _L and A _R are the attenuation coefficients of the signals, d _L and d _R is a data sample number of the delay from each speaker to the ear opposite. In some embodiments, XTC 432, as shown in FIG. 4A, uses a recursive crosstalk cancellation method or a general-purpose directional binaural transfer function. A more detailed description of XTC 432 may be found in U.S. Patent Application No. 14 / 569,490, filed December 12, 2014, entitled "Apparatus and Method for Sound Stage Improvements," issued December 27, 2016. No. 9,532,156) and U.S. Patent Application No. 15 / 349,822, filed November 11, 2016, entitled "Apparatus and Method for Sound Stage Improvement". These are incorporated by reference in their entirety.

  In some embodiments, as shown in FIG. 4A, the output signal of XTC 432 was provided to amplifier 462, a pair of residual signals 452 was provided to amplifier 464, and intermediate component (M) 426 was also provided to amplifier 466. Later, they are sent to a width controller (WC) 434, where they are processed and combined together.

ここで、−５≦β≦０は、ステージ幅のパラメータである。その結果得られる信号は、β＝０のときサウンドステージ幅が最大になり、β＝−５のときモノ信号に近い。 [0050] In some embodiments, the output signals of amplifiers 462, 464, and 466 are sent to WC 434 to adjust the width of the stage. In some embodiments, WC 434 uses the analyzed information of the input signal pair to control the sound stage width of the output audio signal. The stage width can be as narrow as 0 ° to as wide as 360 ° for a total immersive sound. In some embodiments, the cross-channel information of the pair (eg, output pair 472 or 474) is analyzed and otherwise adjusted. In some embodiments, the user can assign a desired stage width to the width controller, as shown below in FIG. The assigned stage width may affect the WC434 pair addition matrix based on the information analyzed so far. In some examples, the width of the sound stage can be adjusted using the following equation:

Here, −5 ≦ β ≦ 0 is a parameter of the stage width. The resulting signal has a maximum sound stage width when β = 0 and is close to a mono signal when β = −5.

  [0051] In some embodiments, the output signal 476 of the WC 434 is sent to a second dynamic range compressor (DRC) 436 to amplify the overall output level of the audio signal in an audio mastering process. In some embodiments, data processing system 100 performs a second dynamic range compression on the left and right side components by DRC 436 to preserve location cues in the digital audio output signal.

  [0052] As shown in FIG. 4A, the output of the pipeline is a stereo audio signal 440 that includes a left channel (L ') and a right channel (R'). In some embodiments, a signal workflow including PROC 420 can be applied to a 5.1 surround sound file, as shown in FIG. 3A. For example, PROC 232 and / or PROC 234 may be similar to PROC 420, as shown in FIG. 4A.

  FIG. 4B is a block diagram illustrating a signal workflow applied to a 7.1 surround sound file, according to some embodiments. In some embodiments described with reference to FIG. 3B, the input signals of the 7.1 surround sound file are paired, ie, L / R pair 242, Lss / Rss pair 244, and Lrs / Rrs pair 246, respectively. Grouped into Next, L / R pair 242, Lss / Rss pair 244, and Lrs / Rrs pair 246 are transmitted to PROC 252, PROC 254, and PROC 256, respectively. In some embodiments, the respective PROCs of PROC 252, PROC 254, and PROC 256 are similar to PROC 420 described above with reference to FIG. 4A. In some other embodiments, PROC 252, PROC 254, or PROC 256 may include one or more other modules (or components) in various configurations. The output signal of each of PROC 252, PROC 254, and PROC 256 is transmitted to a respective width control, eg, WC 482, WC 484, and WC 486, respectively. WC482, WC484, or WC486 may be similar to WC434 described above with reference to FIG. 4A. The outputs of WC482, WC484, and WC486 are combined with center signal C and bass effect channel LFE to produce output stereo audio signal 488.

  [0054] FIG. 5 illustrates software used to manage the implementation of the signal pipeline, as described with reference to FIG. 4B for 7.1 surround sound files, according to some embodiments. An application user interface (UI) 500 or a plug-in component of a software application is shown. The signal pipeline may include multiple paired processors, such as PROC 420 as shown in FIG. 4A. In this case, there are three pairs: a front pair (L, R), a side pair (Lss, Rss), and a rear pair (Lrs, Rrs). The left panel 510 of the UI 500 controls the gain of each input channel. The control area 520 controls the frequency components of the equalizer (EQ). The control area 530 controls the width of the width control device. As described above with reference to FIG. 4B, each pair passes through a different pair processor PROC having different parameters, and thus uses control region 540 to determine which pair (eg, front pair, Side pair or rear pair) selects whether to input a parameter for executing the PROC process.

  [0055] As mentioned above, sound is an important part of media and entertainment. The sounds appeal to the audience's emotions and draw the audience into the story. To get a more immersive listening experience, multi-channel surround sound is introduced. The multi-channel audio format utilizes multiple audio tracks to reconstruct the sound on a corresponding multi-channel sound reproduction system. Downmixing can occur on both the production side and the reproduction side. On the production side, the sound mixer usually starts mixing with the highest number of channels and downmixes to a lower number of channels. On the reproduction side, the multi-channel audio track can be downmixed to a relatively small number of channels to match the channel number of the reproduction system. In both cases, the goal is to maintain the use and placement of sounds that match the original intent.

  As shown in FIGS. 1A to 1C, the conventional downmixing method applies a process to each individual track. The relationship between the tracks is not taken into account. The methods and systems presented herein receive audio input on a pair-by-pair basis. Multi-channel audio tracks are first grouped into pairs according to the physical arrangement of the reproduction system. Second, the relationship between the two channels of the pair will be analyzed. Finally, the pair input is processed based on the analysis result. By incorporating the relationship between pairs, the spatial information of the original multi-channel audio input can be better preserved. As a result, the multi-channel audio output downmixed with the systems and methods introduced herein is more accurate and closer to the original multi-channel audio input than the same audio input downmixed in a conventional manner. Create a nice soundscape.

  [0057] FIGS. 6A-6C are flowcharts illustrating a process of downmixing a multi-channel audio signal using the data processing system 100 shown in FIG. 2, according to some embodiments. The data processing system 100 selects a left input channel and a right input channel from the multi-channel input audio signal (602). In some embodiments, the left input channel and the right input channel correspond to a pair of spatially symmetric signal sources. In some embodiments, the multi-channel audio input signal is the 5.1 surround sound file 210 of FIG. 3A or the 7.1 surround sound file 240 of FIG. 3B. In some embodiments, 5.1 surround input signal 210 includes left front channel L and right front channel R as pair 222 and left side channel Ls and right side channel Rs as pair 224. In some embodiments, the 7.1 surround input signal 240 includes a left front channel L and a right front channel R as a pair 242, a left side surround channel Lss and a right side surround channel Rss as a pair 244, And a left rear surround channel Lrs and a right rear surround channel Rrs as a pair 246.

  [0058] Next, data processing system 100 generates one or more cross-channel features from the selected pair of left and right input channels (604). In some embodiments, the one or more cross-channel features include a comparison of the volumes of the left and right input channels of the pair, a relationship between frequency spectral characteristics (eg, amplitude and / or phase) of the left and right input channels, and / or It includes the difference between the signal start time and amplitude of the left and right input channels.

  [0059] The data processing system 100 then processes (606) according to the selected pair of cross-channel features, the left input channel, and the right input channel to generate a left intermediate channel and a right intermediate channel. In some embodiments, the left and right input channels of the pair are processed using a processor (PROC) as shown in FIGS. 3A-3B and 4A-4B. In some embodiments, the PROC includes one or more modules as shown in FIG. 4A.

  Next, the data processing system 100 combines (608) each of the left intermediate channel and the right intermediate channel with the third input channel of the multi-channel input audio signal to form a two-channel output audio signal. I do. For example, as shown in FIG. 3A, a left intermediate channel and a right intermediate channel (eg, an L / R pair or an Ls / Rs pair) processed by respective PROCs may include a center channel C and / or bass effect (LFE). To generate a two-channel output audio signal L '/ R'. Similarly, as shown in FIG. 3B, the left intermediate channel and the right intermediate channel (eg, L / R pair, Lss / Rss pair, or Lrs / Rrs pair) processed by respective PROCs are center channels C and And / or combined with a bass effect (LFE) to produce a two-channel output audio signal L ′ / R ′.

  [0061] In acoustics, a sound stage is usually defined as the area between the leftmost and rightmost perceived position in audio reproduction. That is, the sound stage is the limit of how far a sound object can be perceived. Thus, the soundstage width is defined as the distance between the left and right boundaries. In the normal case, the sound stage width of a stereo reproduction is the separation distance between two loudspeakers. In this application, the concept of the soundstage is adapted to be applied independently to each symmetric pair of channels. For example, in some embodiments, data processing system 100 uses a width controller (eg, WC 434) to further adjust (610) the sound stage width associated with the left intermediate channel and the right intermediate channel before the left processing. The intermediate channel and the right intermediate channel are combined with the third input channel. In some embodiments, data processing system 100 receives user input specifying a sound stage width of the two-channel output audio signal (612). User input can be received at the UI 500, as shown in FIG.

  [0062] In some embodiments, processing 606 the left and right input channels further comprises extracting an intermediate component, a left side component, and a right side component from the left and right input channels (614). including. For example, as shown in FIG. 4A, input pair 410 is processed by M / S mixer 422 to generate intermediate component 426 and left and right side components S424. In some embodiments, after processing 616 the left and right input channels, and the left and right side components, the data processing system 100 combines these with the intermediate components to form the left intermediate channel and the right Create an intermediate channel.

  [0063] In some embodiments, processing 606 the left and right input channels includes equalizing (eg, by EQ block 428 of FIG. 4A) to the left and right side components using a bandpass filter. (618) to obtain the left band-pass filtered component and the right band-pass filtered component (eg, the output signal of the EQ 450). In some embodiments, such as the left side residual component and the right side residual component 452 as shown in FIG. 4A, this equalization process further reduces the difference between the left side component and the left bandpass filtered component. A left side residual component is generated based on the difference (620), and a right side residual component is generated based on a difference between the right side component and the right band-pass filtered component (620).

  [0064] In some embodiments, after performing equalization to the left and right side components, data processing system 100 may perform first dynamic range compression (eg, according to DRC 430 of FIG. 4A) on the left band. Performed 622 on the pass-filtered component and the right band-pass filtered component (eg, generated by EQ 428 of FIG. 4A), respectively, and accordingly the left compressed component and the right compressed component. Get the ingredients.

  [0065] In some embodiments, after performing the first dynamic range compression, data processing system 100 generates crosstalk cancellation (eg, by XTC 432 of FIG. 4A) (eg, by DRC 430 of FIG. 4A). ) Perform 624 on the left and right compressed components, respectively, to obtain the left side component with crosstalk cancellation and the right side component with crosstalk cancellation.

  In some embodiments, the data processing system 100 combines the crosstalk-eliminated left side component and the crosstalk-eliminated right side component, the left and right side residual components, and the intermediate component ( 626), generate a left intermediate channel and a right intermediate channel. In some embodiments, this combining step further comprises adjusting (628) the sound stage width associated with the left intermediate channel and the right intermediate channel (e.g., by WC 434 of FIG. 4A), and then combining these with the third input channel. And combining. For example, as shown in FIG. 4B, the left and right intermediate channels generated by each PROC are sent to each WC to adjust the sound stage width. The conditioned signal is then combined with a third input channel, eg, a C or LFE channel, to generate an output stereo signal 488.

  [0067] In some embodiments, after adjusting the sound stage width, the data processing system 100 performs a second dynamic range compression (eg, according to the DRC 436 of FIG. 4A) (630) to obtain a left intermediate channel and a right intermediate channel. Create an intermediate channel.

  [0068] Finally, it is noted that the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both hardware and software elements. In a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, and the like.

  [0069] Further, the present invention provides a computer program accessible from a computer usable or computer readable medium that provides program code for use with or for use by a computer or any instruction execution system. It can take the form of a product. For purposes of this description, computer-usable or computer-readable media includes, stores, communicates, propagates, or transports programs for use by or with a system, apparatus, or device of instruction execution. It can be any tangible device that can do this.

  [0070] This medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or device or device), or a propagation medium. Examples of computer readable media include semiconductor memory or solid state storage, magnetic tape, removable computer diskettes, random access memory (RAM), read only memory (ROM), hard magnetic disks, and optical disks. included. Current examples of optical disks include compact disk read only memory (CD-ROM), compact disk read / write (CD-R / W), and DVD.

  [0071] A data processing system suitable for storing and / or executing program code will include at least one processor coupled directly or indirectly to a storage element via a system bus. These storage elements include a local memory used for actually executing the program code, a mass storage device, and at least temporarily store at least some program code so as to temporarily store the mass code during execution. A cache memory may be included to reduce the number of times code must be retrieved from the device. In some embodiments, the data processing system is implemented in the form of a semiconductor chip (eg, a system-on-chip) that integrates all components of a computer or other electronic system on a single chip substrate.

  [0072] Input / output or I / O devices (including, but not limited to, keyboards, displays, pointing devices, etc.) couple to the system either directly or through intervening I / O controls. be able to.

  [0073] A network adapter is also coupled to the system such that the data processing system couples to another data processing system or a remote printer or storage device via an intervening private or public network. It may be. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.

  The description of the present application has been presented for purposes of illustration and description, and is not exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those skilled in the art. To best explain the principles of the invention, its practical application, and various modifications suitable for the particular intended use, the invention will be understood by those skilled in the art for various embodiments. An embodiment has been chosen and described in order to

  [0075] Terminology used in the description of the embodiments herein is for the purpose of describing particular embodiments only, and is not intended to limit the scope of the claims. As used in the description of the embodiments and the appended claims, the singular forms "a," "an," and "the" include plural reference unless the context clearly dictates otherwise. . It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The terms "comprises" and / or "comprising" as used herein identify the presence of specified features, entities, steps, acts, elements and / or components, but one It will be further understood that this does not exclude the presence of or other elements, completeness, steps, acts, elements, components and / or groups thereof.

  [0076] Terms such as "first" and "second" may be used herein to describe various elements, but these elements should not be limited by these terms. Will be understood. These terms are used exclusively to distinguish one element from another. For example, a first port may be referred to as a second port, and similarly, a second port may be referred to as a first port without departing from the scope of the embodiments. The first port and the second port are both ports, but not the same port.

  [0077] Numerous modifications and alternatives to the embodiments described herein will occur to those skilled in the art having the benefit of the foregoing description and the teachings presented in the associated figures. Therefore, the scope of the appended claims should not be limited to the specific examples of disclosed embodiments, but modifications and other embodiments are included within the scope of the appended claims. Please understand that it is something to be done. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

[0078] The embodiments have been selected and described in order to best explain the underlying principles and their practical applications, thereby, along with various modifications suitable for the particular use contemplated. The underlying principles and various embodiments will be best utilized by those skilled in the art.

Claims (42)

A computer-implemented method for processing a multi-channel input audio signal, comprising:
A computing device having one or more processors, a memory, and a plurality of program modules stored in the memory and executed by the one or more processors;
Selecting a left input channel and a right input channel from the multi-channel input audio signal, wherein the left input channel and the right input channel correspond to a pair of spatially symmetric signal sources;
Generating one or more cross-channel features from the left input channel and the right input channel;
Processing the left input channel and the right input channel to generate a left intermediate channel and a right intermediate channel according to the characteristics of the cross channel;
A computer-implemented method comprising combining each of the left intermediate channel and the right intermediate channel with a third input channel of the multi-channel input audio signal to form a two-channel output audio signal.   2. The computer-implemented method of claim 1, further comprising: after adjusting a sound stage width associated with the left intermediate channel and the right intermediate channel, combining them with the third input channel.   3. The computer-implemented method of claim 2, further comprising receiving a user input specifying the sound stage width of the two-channel output audio signal. The step of processing the left input channel and the right input channel further comprises:
Extracting an intermediate component, a left side component, and a right side component from the left input channel and the right input channel;
The computer-implemented method of claim 1, further comprising: after processing the left side component and the right side component, combining them with the intermediate component to generate the left intermediate channel and the right intermediate channel. Method. Processing the left side component and the right side component,
Performing equalization on the left side component and the right side component using a band pass filter to obtain a left band pass filtered component and a right band pass filtered component;
A left side residual component is generated based on a difference between the left side component and the left band pass filtered component, and a difference between the right side component and the right band pass filtered component is generated. Generating the right-side residual component based on the computer-implemented method.   After performing equalization on the left side component and the right side component, the left band-pass filtered component and the right band-pass filtered component are each subjected to a first dynamic range compression, The computer-implemented method of claim 5, further comprising obtaining a left compressed component and a right compressed component accordingly.   After performing the first dynamic range compression, crosstalk cancellation is performed on the left compressed component and the right compressed component, respectively, so that the crosstalk-eliminated left side component and the crosstalk-eliminated right component are removed. 7. The computer-implemented method of claim 6, further comprising obtaining a side component. Combining the left side component after crosstalk cancellation and the right side component after crosstalk cancellation, the left side residual component and the right side residual component, and the intermediate component to form the left intermediate channel and the right intermediate channel Generating, wherein the step of combining further comprises:
The computer-implemented method of claim 7, comprising adjusting a sound stage width associated with the left intermediate channel and the right intermediate channel, and then combining them with the third input channel.   9. The computer-implemented method of claim 8, further comprising performing a second dynamic range compression after adjusting the sound stage width to generate the left intermediate channel and the right intermediate channel.   The computer-implemented method of claim 1, wherein the left input channel is a left front channel and the right input channel is a right front channel.   The computer-implemented method of claim 1, wherein the left input channel is a left surround channel and the right input channel is a right surround channel.   The computer-implemented method of claim 1, wherein the left input channel is a left rear surround channel and the right input channel is a right rear surround channel.   The computer-implemented method of claim 1, wherein the third input channel is a center channel.   The computer-implemented method of claim 1, wherein the third input channel is a bass effect channel. A computing device for processing a multi-channel input audio signal, comprising:
One or more processors;
Memory and
A plurality of program modules stored in the memory and executed by the one or more processors;
The plurality of program modules, when executed by the one or more processors,
Selecting a left input channel and a right input channel from the multi-channel input audio signal, wherein the left input channel and the right input channel correspond to a pair of spatially symmetric signal sources;
Generating one or more cross-channel features from the left input channel and the right input channel;
Processing the left input channel and the right input channel to generate a left intermediate channel and a right intermediate channel according to the characteristics of the cross channel;
Combining the left intermediate channel and the right intermediate channel each with a third input channel of the multi-channel input audio signal to form a two-channel output audio signal. A computing device that allows the device to execute.   16. The method of claim 15, further comprising: after adjusting a sound stage width associated with the left intermediate channel and the right intermediate channel, combining them with the third input channel. Computing device.   The computing device of claim 16, further adapted to: receive user input specifying the sound stage width of the two-channel output audio signal. The step of processing the left input channel and the right input channel further comprises:
Extracting an intermediate component, a left side component, and a right side component from the left input channel and the right input channel;
The computing of claim 15, comprising processing the left side component and the right side component and then combining them with the intermediate component to generate the left intermediate channel and the right intermediate channel. apparatus. Processing the left side component and the right side component,
Performing equalization on the left side component and the right side component using a band pass filter to obtain a left band pass filtered component and a right band pass filtered component;
A left side residual component is generated based on a difference between the left side component and the left band pass filtered component, and a difference between the right side component and the right band pass filtered component is generated. Generating the right-side residual component based on the computing device.   After performing equalization on the left side component and the right side component, the left band-pass filtered component and the right band-pass filtered component are each subjected to a first dynamic range compression, 20. The computing device of claim 19, further adapted to perform obtaining a left compressed component and a right compressed component accordingly.   After performing the first dynamic range compression, crosstalk cancellation is performed on the left compressed component and the right compressed component, respectively, so that the crosstalk-eliminated left side component and the crosstalk-eliminated right component are removed. 21. The computing device of claim 20, further adapted to perform obtaining a side component. Combining the left side component after crosstalk cancellation and the right side component after crosstalk cancellation, the left side residual component and the right side residual component, and the intermediate component to form the left intermediate channel and the right intermediate channel Further comprising performing the combining step, wherein the combining step further comprises:
22. The computing device of claim 21, comprising adjusting a sound stage width associated with the left middle channel and the right middle channel before combining them with the third input channel.   23. The computer of claim 22, further comprising, after adjusting the sound stage width, performing a second dynamic range compression to generate the left intermediate channel and the right intermediate channel. Device.   The computing device of claim 15, wherein the left input channel is a left front channel and the right input channel is a right front channel.   The computing device of claim 15, wherein the left input channel is a left surround channel and the right input channel is a right surround channel.   The computing device of claim 15, wherein the left input channel is a left rear surround channel and the right input channel is a right rear surround channel.   The computing device according to claim 15, wherein the third input channel is a central channel.   The computing device of claim 15, wherein the third input channel is a bass effect channel. A computer program product stored on a persistent computer readable storage medium, with a computing device having one or more processors for processing audio signals, the computer program product being executed by the one or more processors. When,
Selecting a left input channel and a right input channel from the multi-channel input audio signal, wherein the left input channel and the right input channel correspond to a pair of spatially symmetric signal sources;
Generating one or more cross-channel features from the left input channel and the right input channel;
Processing the left input channel and the right input channel to generate a left intermediate channel and a right intermediate channel according to the characteristics of the cross channel;
Combining the left intermediate channel and the right intermediate channel each with a third input channel of the multi-channel input audio signal to form a two-channel output audio signal. A computer program product that includes a plurality of program modules that allow a device to execute.   The computing device is further adapted to adjust a sound stage width associated with the left middle channel and the right middle channel before combining them with the third input channel. A computer program product according to claim 29.   31. The computer program product of claim 30, wherein the computing device is further adapted to receive user input specifying the sound stage width of the two-channel output audio signal. The step of processing the left input channel and the right input channel further comprises:
Extracting an intermediate component, a left side component, and a right side component from the left input channel and the right input channel;
30. The computer of claim 29, further comprising: after processing the left side component and the right side component, combining them with the intermediate component to generate the left intermediate channel and the right intermediate channel. Program products. Processing the left side component and the right side component,
Performing equalization on the left side component and the right side component using a band pass filter to obtain a left band pass filtered component and a right band pass filtered component;
A left side residual component is generated based on a difference between the left side component and the left band pass filtered component, and a difference between the right side component and the right band pass filtered component is generated. Generating a right side residual based on the computer program product. The computing device comprises:
After performing equalization on the left side component and the right side component, the left band-pass filtered component and the right band-pass filtered component are each subjected to a first dynamic range compression, 34. The computer program product of claim 33, further adapted to obtain a left compressed component and a right compressed component accordingly. The computing device comprises:
After performing the first dynamic range compression, crosstalk cancellation is performed on the left compressed component and the right compressed component, respectively, so that the crosstalk-eliminated left side component and the crosstalk-eliminated right component are removed. 35. The computer program product of claim 34, further adapted to obtain a side component. The computing device comprises:
Combining the left side component after crosstalk cancellation and the right side component after crosstalk cancellation, the left side residual component and the right side residual component, and the intermediate component to form the left intermediate channel and the right intermediate channel Further comprising performing the combining step, wherein the combining step further comprises:
36. The computer program product of claim 35, comprising adjusting a sound stage width associated with the left intermediate channel and the right intermediate channel before combining them with the third input channel. The computing device comprises:
37. The computer program product of claim 36, further comprising performing a second dynamic range compression after adjusting the sound stage width to generate the left intermediate channel and the right intermediate channel.   30. The computer program product of claim 29, wherein the left input channel is a left front channel and the right input channel is a right front channel.   30. The computer program product of claim 29, wherein the left input channel is a left surround channel and the right input channel is a right surround channel.   30. The computer program product of claim 29, wherein the left input channel is a left rear surround channel and the right input channel is a right rear surround channel.   30. The computer program product according to claim 29, wherein the third input channel is a central channel. 30. The computer program product according to claim 29, wherein the third input channel is a bass effect channel.

Images