KR101615776B1 - Apparatus and method for coding and decoding multi-object audio signal using different analysis stages - Google Patents

Abstract

An apparatus and method for encoding and decoding multi-object audio signals of various channels are provided.
The encoding apparatus generates second audio signals by downmixing a plurality of multi-object audio signals composed of different channels, and generates additional information including header information and space queue information for each of the second audio signals Wherein the downmixing unit includes a downmixing unit for generating a plurality of second audio signals, a first encoding unit for encoding the second audio signals, and a second encoding unit for encoding the additional information to generate a bitstream, Sub-bands, and extracts additional information for each of the divided sub-bands.

Description

[0001] APPARATUS AND METHOD FOR CODING AND DECODING MULTI-OBJECT AUDIO SIGNAL USING DIFFERENT ANALYSIS STAGES [0002]

The present invention relates to an apparatus and method for encoding and decoding multi-object audio signals, and more particularly, to an apparatus and method for encoding and decoding multi-object audio signals composed of various channels using different analysis steps.

Here, a multi-object audio signal having various channels means an audio signal composed of channels (for example, mono, stereo, and 5.1 channels) in which audio objects are different from each other as multi-object audio signals.

The present invention has been derived from research conducted by the Ministry of Culture, Sports and Tourism and the Korea Creative Content Agency as part of the technology development of user-created music contents service [Task:

According to the conventional audio decoding technique, the user can only listen to the audio content manually.

A device for encoding and decoding a plurality of audio objects, each of which is composed of various channels, capable of consuming various audio objects by combining audio contents in various ways by controlling audio objects composed of channels different from each other according to user's needs And methods are required.

In this regard, the conventional SAC (Spatial Audio Coding) is a technique for expressing, transmitting and restoring a multi-channel audio signal as a downmixed mono or stereo signal and a spatial cue, and can transmit a high-quality multi-channel audio signal even at a low bit rate .

However, the SAC is capable of encoding and decoding only audio signals which are multi-channel but one object, and multi-object audio signals composed of multi-channel and multi-object audio signals such as mono, There is a problem that it can not be done.

In addition, the conventional binaural cue coding (BCC) can decode a multi-object audio signal. However, since the audio object is limited to a mono channel, the multi-object audio signal is composed of various channels other than a mono channel The object audio signal can not be decoded.

Therefore, by encoding and decoding a plurality of audio objects, which are composed of various channels, which can consume various audio objects by controlling each of a plurality of audio objects composed of channels different from each other according to the user's needs and combining one audio content in various methods Apparatus and method are required.

According to an aspect of the present invention, there is provided an encoding apparatus for generating second audio signals by downmixing a plurality of multi-object audio signals composed of mutually different channels, and generating header information for each of the second audio signals, A first coding unit for coding the second audio signals; and a second coding unit for coding the additional information to generate a bitstream, wherein the downmixing unit comprises: There is provided an encoding apparatus for dividing second audio signals into a plurality of sub-bands and extracting additional information for each of the divided sub-bands.

The second audio signal may be divided into a plurality of sub-bands according to a frequency band.

The additional information may be extracted for each of the sub-bands of the high frequency band and the sub-band of the low frequency band.

The subbands of the low frequency band may be frequency converted before extracting the additional information.

The frequency transform may be a discrete Fourier transform.

According to one aspect of the present invention, in a decoding apparatus, a downmix audio signal is recovered from an input audio signal, and additional information including header information and spatial cue information is extracted from an additional information bitstream included in the input audio signal An input signal analyzer, an audio object extractor for recovering the reconstructed downmix audio signal into an object-specific audio signal using the additional information extracted from the input signal analyzer, Wherein the audio object extractor divides the reconstructed downmix audio signal into a plurality of subbands and outputs the reconstructed downmix audio signal to a plurality of subbands for each of the plurality of subbands, A decoding device for decoding the object-specific audio signal using the information Is provided.

The restored downmix audio signal may be divided into a plurality of sub-bands according to a frequency band.

The plurality of sub-bands may be used to recover the per-object audio signal using sub-information for each sub-band of the high frequency band and the sub-band of the low frequency band.

The subband of the low frequency band may be transformed to the frequency of the restoration of the object-specific audio signal.

The inverse frequency transform may be an inverse discrete Fourier transform.

According to an aspect of the present invention, there is provided an apparatus for encoding a multi-object audio signal composed of channels different from each other, the apparatus comprising: a downmixing unit for downmixing a multi-object audio signal composed of different channels to a downmixed audio signal; Downmixing means for extracting additional information including header information and spatial cue information for each of the configured multi-object audio signals; An encoding means for encoding the downmixed audio signal; And additional information encoding means for generating the additional information as a bitstream, wherein the header information includes identifier information for each of the multi-object audio signals having different channels; And channel information for a multi-object audio signal composed of the different channels.

According to an aspect of the present invention, there is provided a method of encoding a multi-object audio signal composed of channels different from each other, the method comprising: downmixing a multi-object audio signal composed of different channels to one downmixed audio signal; A downmixing step of extracting additional information including header information and spatial cue information for each of the constructed multi-object audio signals; An encoding step of encoding the downmixed audio signal; And a side information encoding step of generating the side information as a bit stream, wherein the header information includes identifier information for each of multi-object audio signals having different channels; And channel information for a multi-object audio signal composed of the different channels.

According to an aspect of the present invention, there is provided an apparatus for decoding a multi-object audio signal having channels different from each other, the apparatus comprising: a decoder for decoding a downmix audio signal from an input audio signal, Input signal analyzing means for extracting additional information including queue information; An audio object extracting means for extracting the reconstructed downmix audio signal into an object-specific audio signal using the additional information extracted from the input signal analyzing means, And output means for outputting an audio signal as a multi-object audio signal, wherein the header information includes identifier information for each of the multi-object audio signals having different channels; And channel information for a multi-object audio signal composed of the different channels.

According to an aspect of the present invention, there is provided a method of decoding a multi-object audio signal composed of channels different from each other, the method comprising: restoring a downmix audio signal from an input audio signal; An input signal analyzing step of extracting additional information including the queue information; An audio object extracting step of restoring the restored downmix audio signal into an object-specific audio signal using the additional information extracted from the input signal analyzing step; And outputting the restored object-specific audio signal as a multi-object audio signal using control information for the input audio signal, wherein the header information includes at least one of the multi- Identifier information; And channel information for a multi-object audio signal including the different channels.

According to an aspect of the present invention, there is provided an apparatus for decoding a multi-object audio signal having channels different from each other, the apparatus comprising: a decoder for decoding a downmix audio signal from an input audio signal, Input signal analyzing means for extracting additional information including queue information; Additional information control means for controlling the additional information extracted from the input signal analysis means by using control information on the input audio signal; And output means for outputting the restored downmix audio signal as a multi-object audio signal using the controlled additional information, wherein the header information includes identifier information for each of the multi-object audio signals having different channels; And channel information for a multi-object audio signal composed of the different channels.

According to another aspect of the present invention, there is provided a method of decoding a multi-object audio signal comprising channels different from each other, the method comprising: restoring a downmix audio signal from an input audio signal; And additional information including spatial cue information; An additional information control step of controlling the additional information extracted from the input signal analysis step by using control information on the input audio signal; And outputting the restored downmix audio signal as a multi-object audio signal using the controlled additional information, wherein the header information includes identifier information for each of the multi-object audio signals having different channels; And channel information for a multi-object audio signal including the different channels.

An apparatus and method for encoding and decoding multi-object audio signals composed of various channels using different analysis steps are provided. The user can actively consume audio content as needed.

By varying the analysis method of the low-band subband and the high-band subband, the coding efficiency is maximized within a given amount of information, so that the sound quality of the reconstructed object signal can be improved.

1 is a block diagram illustrating a multi-object audio encoding apparatus according to an embodiment of the present invention;
FIG. 2 is a detailed block diagram of the mono channel down mixer of FIG. 1 according to an embodiment of the present invention; FIG.
3 is a detailed block diagram of the stereo channel downmixer of FIG. 1 according to an embodiment of the present invention.
FIG. 4 is a detailed block diagram illustrating a multi-channel down mixer of FIG. 1 according to an embodiment of the present invention. FIG.
5 is a detailed block diagram illustrating a second down mixer of FIG. 1 according to an embodiment of the present invention;
FIG. 6 is a diagram illustrating a structure of a side information bitstream generated from the side information encoder of FIG. 1 according to an embodiment of the present invention; FIG.
FIG. 7 is a detailed structural diagram of a bitstream of additional information shown in FIG. 6 according to an embodiment of the present invention; FIG.
FIG. 8 is a detailed structure diagram of the additional information bit stream shown in FIG. 6 according to another embodiment of the present invention; FIG.
FIG. 9 is a block diagram of an apparatus for decoding multi-object audio signals of various channels according to an embodiment of the present invention; FIG.
10 is a block diagram of an apparatus for decoding multi-object audio signals of various channels according to an embodiment of the present invention.
11 is a flowchart illustrating a multi-object audio encoding method according to an embodiment of the present invention.
12 is a flowchart illustrating a multi-object audio decoding method according to an embodiment of the present invention.
13 is a flowchart illustrating a multi-object audio decoding method according to another embodiment of the present invention.
FIG. 14 is a detailed structure diagram of a basic down mixer according to an embodiment of the present invention; FIG.
15 illustrates sub-band divided signals in accordance with an embodiment of the present invention.
16 is a diagram illustrating an example of windowing for each signal according to an embodiment of the present invention;
17 is a flowchart of a downmixing method according to an embodiment of the present invention;
18 is a detailed configuration diagram of a SAC decoder according to an embodiment of the present invention;
19 is a diagram illustrating an example of windowing for each signal according to an embodiment of the present invention;
Figure 20 illustrates an example of superposition-addition for each windowed signal in accordance with an embodiment of the present invention;

Hereinafter, some embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. Like reference symbols in the drawings denote like elements.

FIG. 1 illustrates a multi-object audio encoding apparatus according to an embodiment of the present invention. In this embodiment, the channels of a plurality of input audio objects are mono, stereo, and 5.1 channels, respectively.

1, a multi-object audio encoding apparatus according to an embodiment of the present invention includes a first down mixer 101, a second down mixer 103, an audio encoder 105, a side information encoder 107, And a multiplexer (multiplexer) 109.

The first down mixer 101 includes a mono channel down mixer 111, a stereo channel down mixer 113, and a multi-channel down mixer 115.

The first down mixer 101 identifies the multi-object audio signals of various channels inputted using the header information of the input audio object as mono, stereo, and multi-channel, and groups them by channel. Accordingly, the multi-object audio signals of various channels are grouped by channels and down-mixed by down mixers 111, 113, and 115 for respective channels.

The first downmixer 101 extracts the downmixed audio signal and the additional information including the spatial cue from the input audio object.

That is, sound sources are grouped by the same channel and input to the first down mixer 101.

The mono channel downmixer 111 extracts a downmix signal and additional information including a spatial cue from the input mono audio object, and the stereo channel downmixer 113 extracts a downmix signal and a spatial cue from the input stereo audio object And the multi-channel down-mixer 115 extracts additional information including a down-mix signal and a spatial cue from the inputted multi-channel (for example, 5.1 channel) audio object.

The audio encoder 105 encodes the second downmix signal output from the second downmixer 103.

The side information encoder 107 generates a side information bit stream using the side information output from the first down mixer 101 and the side information output from the second down mixer 103. Here, the information included in the additional information bitstream will be described in detail in FIG.

The multiplexer 109 multiplexes the encoded signal received from the audio encoder 105 and the additional information bit stream received from the additional information encoder 107 to generate a bit stream to be transmitted to the decoding apparatus.

The first downmix signal output from the first downmixer 101 is a stereo signal or a mono signal. That is, the downmix signal output from the mono channel downmixer 111 is a mono signal, and the downmix signals output from the remaining downmixers 113 and 115 are mono or stereo signals.

The second downmixer 103 downmixes the first downmix signal output from the first downmixer 101 and outputs the additional information including the spatial cue analyzed in the second downmixing process . Here, the second downmix signal is a mono or stereo signal depending on the mode.

Here, the additional information includes header information for restoring and controlling the spatial cue and the audio signal. The additional information will be described in detail in Fig.

FIG. 2 is a detailed block diagram of the mono channel down mixer 111 of FIG. 1 according to an embodiment of the present invention. In this embodiment, the input mono audio objects are N (m1, ..., mN).

As shown in FIG. 2, the mono channel down mixer 111 includes a basic down mixer 1 in a cascade structure.

The number of basic down mixers 201 included in the mono channel down mixer 111 is determined according to the number N of mono audio objects. That is, when the number of mono audio objects is N, the number of basic down mixers 201 is N-1. When there is one mono audio object, the input signal is bypassed without a basic down mixer.

On the other hand, according to the embodiment, one basic down mixer 1 can be used N-1 times in a cascade manner.

Basically, the basic downmixer 1 downmixes two input signals, generates one downmixed mono signal, and extracts additional information including a spatial cue for the input signal.

The first basic downmixer 201a generates one downmixed mono signal using two mono audio objects input to the mono channel downmixer 111 and extracts additional information including a spatial cue.

Next, the second basic downmixer 1 (201b) uses the downmixed mono signal output from the first basic downmixer 201a and the mono audio object input to the mono channel downmixer 111, Generates a mixed mono signal, and extracts additional information including a spatial cue.

The (N-1) -th basic down-mixer 201d uses the down-mix mono signal output from the (N-2) -th basic down mixer 1 (not shown) and the mono audio object input to the mono channel down- Generates a downmixed mono signal, and extracts additional information including a spatial cue.

Here, the spatial cue is information used in the process of encoding and decoding an audio signal. The spatial cue is extracted in the frequency domain. The spatial cue is a spatial cue, Information.

For example, as a spatial cue that can be utilized as an embodiment of the present invention, a spatial cue that can be utilized as an audio signal level difference (CLD) between audio signals representing power gain information of an audio signal, Channel level difference (ICLD), inter-channel time difference (ICTD), inter-channel correlation (ICC) and virtual source location information indicating correlation information between audio signals, But is not limited thereto.

Here, the additional information includes header information for restoring and controlling the spatial cue and the audio signal. The additional information will be described in detail in Fig.

3 is a detailed block diagram of the stereo channel down mixer 113 of FIG. 1 according to an embodiment of the present invention. The stereo audio objects input in this embodiment are M (SL1, ..., SLM and SR1, ..., SRM) respectively of a left signal and a right signal.

Stereo audio objects input to the stereo channel down mixer 113 are separated into a left signal and a right signal of stereo and are grouped.

As shown in FIG. 3, the stereo channel downmixer 113 includes a basic downmixer 1 (201). The basic down mixer 201 included in the stereo channel down mixer 113 needs 2 * (M-1) to down mix the M left signal and M right signal. Here, as described in FIG. 2, one basic down mixer 1 may be used 2 * (M-1) times depending on the embodiment.

As shown in FIG. 2, the M-1 basic down mixers 201a to 201le for analyzing the M left signals analyze the input signals to generate one downmixed left signal, And extracts the additional information included therein.

As shown in FIG. 2, the M-1 basic downmixers 201a to 201re for analyzing the M right signals generate a downmixed right signal by analyzing the inputted signals, And extracts the additional information including the queue.

As described with reference to FIG. 2, when there is one stereo audio object, the left signal and the right signal to be input may be bypassed.

Therefore, the stereo channel downmixer 113 generates a downmixed left signal and a downmixed right signal, thereby outputting a stereo downmix signal and extracting additional information including a spatial cue.

Here, the additional information includes header information for restoring and controlling the spatial cue and the audio signal. The additional information will be described in detail in Fig.

4 is a detailed configuration diagram illustrating a multi-channel down-mixer 115 of FIG. 1 according to an embodiment of the present invention. In this embodiment, there are P number of 5.1-channel audio objects input.

As shown in FIG. 4, the multi-channel down-mixer 115 is a down mixer according to the conventional MPEG Surround or Spatial Audio Coding (SAC) technology, Extracts additional information, and downmixes the audio signal to a mono or stereo downmix signal.

That is, the multi-channel down-mixer 115 extracts spatial cues from P multi-channel audio objects, which are input signals, and transmits the spatial cues, and down-mixes the audio signals into mono or stereo signals. Generally, most of the multi-channel audio objects are one.

5 is a detailed configuration diagram showing a second down mixer 103 of FIG. 1 according to an embodiment of the present invention.

The second down-mixer 103 down-mixes (second down-mixes) a signal output from the first down-mixer 101 to output a stereo down-mix signal and extracts additional information including a spatial cue .

As shown in FIG. 5, the second down mixer 103 includes basic down mixers 201 f and 201 g and a basic down mixer 501.

If the downmixed signals from the stereo channel downmixer 113 and the multi-channel downmixer 115 are stereo, the downmixed stereo signals are grouped into left and right signals, respectively. The basic downmixers 201f and 201g downmix the grouped left and right signals. The downmix mono signals output from the respective basic downmixers 201f and 201g are representative downmix signals of the left and right signals.

That is, the basic down mixer 201f down-mixes the downmixed left signal output from the stereo channel downmixer 113 and the downmixed left signal output from the multi-channel downmixer 115, One representative down mix left signal is output, and additional information is extracted.

The basic down mixer 201g down-mixes the downmixed right signal output from the stereo channel downmixer 113 and the downmixed right signal output from the multi-channel downmixer 115, One representative downmix right signal is output, and additional information is extracted.

Here, as described in FIG. 2, one basic down mixer 1 may be used twice depending on the embodiment.

Next, the basic down-mixer 2 501 receives a down-mix mono signal output from the mono channel down-mixer 111, a representative down-mix left signal output from the basic down-mixers 201 f and 201 g, and a representative down- Mixes the signal, outputs the entire downmix left signal and the entire downmix right signal, and extracts additional information including the spatial cue.

Here, the additional information includes header information for restoring and controlling the spatial cue and the audio signal. The additional information is described in detail in Fig.

The basic down mixer 201 and the basic down mixer 501 downmix an input audio signal based on Equations (1) and (2), respectively.

Where w _b ^ij adjusts the downmixing level of the input audio signal as a weighting factor. S _b ^j (f) is a mono signal or stereo left signal or right signal as the input audio signals of the basic down mixer 201 and the basic down mixer 501. The subscript b is an index representing a subband, and each weighting factor w _b ^ij can be defined for each subband.

The weighting factors can be defined differently depending on the purpose of presentation of the input audio object. For example, in order for an arbitrary mono signal S _b ^j (f) to be encoded as a main signal, a weighting factor corresponding to S _b ^j (f) may be defined as a relatively large value. If the ^{_{w b 11 = 0.7, w b}} 12 = 0.3 In Equation (1), the down-mix signal ^{_{S b k (f) = 0.7}} S b 1 (f) + 0.3 S b S b 1 (f a ² (f) ) Is downmixed to the main signal.

The weighting factor may be determined according to a constraint condition of the presentation purpose for the downmix signal. Here, the constraint is a constraint on a sound scene. For example, in a downmixed audio signal, an audio signal for a violin and the like is multiplied by a weight Elements can be set to 0.7 and 0.3, respectively. Constraint information is determined by input from outside such as system or user.

Meanwhile, the weighting factor should be reflected in the spatial cue level information. For example, when CLD is used as a spatial cue, spatial cue level information can be predicted as shown in Equation (3) for Equation (1).

을 이용하여 각 신호의 파워 합이 계산될 수 있다. A _b 및 A _{b + 1} 는 서브밴드의 경계(boundary)를 나타낸다.Here, P () is a power operator,

The power sum of each signal can be calculated. A _b and A _{b + 1} denote the boundaries of the subbands.

The basic downmixer 2 (501) extracts a spatial cue in the same way as a three-to-two (TTT) box of MPEG Surround.

6 is a configuration diagram illustrating a structure of a side information bit stream generated from the side information encoder 107 of FIG. 1 according to an embodiment of the present invention.

As shown in FIG. 6, the additional information bitstream includes header information and a spatial cue.

The header information includes information for restoration and reproduction of a multi-object audio signal composed of various channels, and defines channel information for the audio object and an ID of the corresponding audio object, thereby decoding the audio object of a mono, stereo, Information can be provided.

That is, for example, the identification ID and the object-specific information can be defined such that the specific audio object coded can be distinguished as a mono audio signal or a stereo audio signal.

The header information may include Spatial Audio Coding (SAC) header information, audio object information, and preset information.

In one embodiment, the SAC header information is information generated in a spatial cue-based audio encoding process and may include time-slot information. The SAC header information is extracted in the process of extracting additional information by the first downmixer 101 and the second downmixer 103.

In one embodiment, the audio object information includes information for identifying whether the downmix audio objects are mono, stereo, or multichannel, and object ID information. For example, the number of audio objects per channel (the number of mono audio objects, the number of stereo audio objects, the number of multi-channel audio objects), the index information of the audio objects of each channel (ID, whether the audio object is mono, Information).

In one embodiment, the preset information is additional information of header information, and control information of each object is defined in advance.

For example, the preset information includes preset mode information and preset mode support information. The preset mode information may include, for example, a karaoke mode, a solo object extraction mode (guitar performance extraction, piano performance extraction, etc.), preferred rendering information, and basic playback mode setting information.

The preset mode support information is information for supporting karaoke mode, for example, as vocal index information in karaoke mode, information for supporting a solo object extraction mode, corresponding object index information, rotation, elevation, speed, and the like), and basic stereo and multi-channel playback mode settings, and includes optimal rendering information for each audio object.

In addition, the spatial cue included in the additional information includes spatial cue information for each object of the input multi-object audio signal.

The format of the additional information can be variously configured according to the designer's selection.

FIG. 7 is a detailed structure diagram of the additional information bitstream shown in FIG. 6 according to an embodiment of the present invention, and shows additional information on a multi-object audio signal composed of mono and stereo channels.

7, the header information includes the number of audio objects per channel (the number of mono audio objects, the number of stereo audio objects) and the index information of audio objects of each channel (ID, whether the audio object is a mono, stereo, , And the additional information bitstream includes a spatial cue. In the embodiment of FIG. 7, CLD and ICC are used as an example of the space queue.

As shown in Fig. 7, the spatial information (CLD, ICC) corresponding to each mono and stereo object is included in the additional information. That is, the spatial cue information corresponding to each input audio object must be included in the additional information.

FIG. 8 is a detailed structure diagram of the additional information bitstream shown in FIG. 6 according to another embodiment of the present invention, and shows additional information on a multi-object audio signal composed of mono, stereo, and multi-channels.

As shown in FIG. 8, the header information includes the number of audio objects per channel (the number of mono audio objects, the number of stereo audio objects, the number of multi-channel audio objects) , Multi-channel identification information), and the additional information bitstream includes a spatial cue. In one embodiment of FIG. 8, CLD and ICC are used as an example of a space queue.

Here, the spatial cue for a multi-channel object can be represented by a side information bit stream by cascaded multiplexing with a spatial cue for mono and stereo objects. The spatial cues extracted by the mono channel downmixer 111, the stereo channel downmixer 113 and the second downmixer 103 are spatial cues for the mono and stereo audio objects of FIG. 8, The spatial cue extracted by the spatial cue 115 is a spatial cue for the multi-channel audio object of Fig.

9 is a block diagram of an apparatus for decoding multi-object audio signals of various channels according to an embodiment of the present invention.

The apparatus for decoding multi-object audio signals of various channels according to the present embodiment may extract spatial cue information from an audio bitstream generated from the encoding apparatus of FIG. 1, (Audio signal including mono, stereo, and multi-channel audio objects) composed of various channels.

9, an apparatus for decoding multi-object audio signals of various channels according to the present embodiment includes a demultiplexer (DEMUX) 901, an audio decoder 903, a side information analyzer 905, An audio object extracting unit 907, and a rendering processing unit 909.

The demultiplexer 901 separates an audio information bit stream and a side information bit stream from an audio bit stream generated by the encoding apparatus shown in Fig.

The audio decoder 903 restores the downmix audio signal from the audio information bit stream separated by the demultiplexer 901.

The additional information analysis unit 905 extracts additional information including spatial cue information of each audio object from the additional information bit stream separated by the demultiplexer 901. [

The audio object extracting unit 907 recovers an object-specific audio signal from the downmix audio signal using the header information of the additional information extracted from the additional information analyzing unit 905.

The header information includes the number of audio objects per channel (the number of mono audio objects, the number of stereo audio objects, the number of multi-channel audio objects) and the index information of audio objects of each channel (ID, whether the audio object is mono, Information). Accordingly, the audio object extracting unit 907 extracts the object from the downmix audio signal output by the audio decoder 903 based on the header information and the spatial cue information of the additional information extracted from the additional information analyzing unit 905, It is possible to restore a separate audio signal.

The rendering processing unit 909 generates rendering control information (e.g., position and size of a spatial audio object) and output channel control information (e.g., a position and a size of a spatial audio object) for each audio object restored by the audio object extracting unit 907 (5.1 or 7.1 channel, stereo) from the outside, and outputs the audio signal by positioning the audio signal of each object restored from the audio object extracting unit 907.

10 is a block diagram of an apparatus for decoding multi-object audio signals of various channels according to an embodiment of the present invention. In an embodiment of FIG. 10, unlike the embodiment of FIG. 9 which renders the reconstructed audio signal for each object, the additional information is controlled and the audio object is rendered by rendering the audio object according to the controlled additional information.

10, an apparatus for decoding multi-object audio signals of various channels according to another embodiment of the present invention includes a demultiplexer 901, an audio decoder 903, a side information analyzing unit 905, A control unit 1001 and a SAC decoder 1003. [

The demultiplexer 901, the audio decoder 903 and the additional information analyzing unit 905 may have the same configuration as the demultiplexer 901, the audio decoder 903 and the additional information analyzing unit 905 shown in FIG.

The additional information control unit 1001 may control rendering control information (e.g., position and size of a spatial audio object) and output channel control information (e.g., a position and a size of a spatial audio object) of the downmix audio signal restored by the audio decoder 903 (For example, signal size information and correlation information of each audio object) extracted from the additional information analysis unit 905 is input to the external input signal Respectively.

The SAC decoder 1003 restores the downmix audio signal restored from the audio decoder 903 into multi-object audio signals of various channels by using the side information controlled by the side information control unit 1001. [

The SAC decoder 1003 restores the object-specific audio signal from the downmix audio signal using the header information of the sub information controlled by the sub information controller 1001. [

In the header information, the number of audio objects per channel (the number of mono audio objects, the number of stereo audio objects and the number of multichannel audio objects), and the index information of the audio objects of each channel (ID and whether the audio object is mono, The SAC decoder 1003 generates the downmix audio signal output from the audio decoder 903 based on the header information and the spatial cue information of the sub information controlled by the sub information controller 1001, It is possible to restore the object-specific audio signal from the signal.

11 is a flowchart illustrating a multi-object audio encoding method according to an embodiment of the present invention.

As shown in FIG. 1, input multi-object audio signals of various channels are identified as mono, stereo, or multi-channel by header information of the input audio object and are grouped by channel (S1101).

Next, in step S1101, the sound sources grouped by the same channel are downmixed, and additional information including a space queue is extracted (S1103).

That is, the downmix signal and the additional information including the spatial cue are extracted from the inputted mono audio object, the additional information including the downmix signal and the spatial cue is extracted from the input stereo audio object, And the additional information including the downmix signal and the spatial cue are extracted from the audio object.

The first downmix signal output from the step S1103 is a stereo signal or a mono signal. That is, the downmix signal output from the input mono audio object is a mono signal, and the downmix signal output from the stereo audio object and the multi-channel audio object is a mono or stereo signal.

Next, the first downmix signal output from step S1103 is second downmixed, and additional information including the spatial cue analyzed in the second downmixing process is extracted (S1105). Here, the second downmix signal is a mono or stereo signal depending on the mode.

Next, the second downmix signal output from step S1105 is encoded (S1107).

Next, a side information bitstream is generated using the side information output from step S1103 and the side information output from step S1105 (S1109).

Next, the encoded signal from step S1107 and the additional information bit stream generated from step S1109 are multiplexed and a bitstream to be transmitted to the decoding apparatus is generated (S1111).

12 is a flowchart illustrating a multi-object audio decoding method according to an embodiment of the present invention.

As shown in FIG. 12, the audio information bit stream and the additional information bit stream are separated from the audio bit stream generated in step S1111 (S1201).

Next, the downmix audio signal is recovered from the audio information bit stream separated by the step S1201 (S1203).

Next, additional information including spatial cue information of each audio object is extracted from the bitstream separated by the step S1201 (S1205).

Next, the object-specific audio signal is restored from the downmix audio signal using the header information of the additional information extracted from step S1205 (S1207).

In the header information, the number of audio objects per channel (the number of mono audio objects, the number of stereo audio objects and the number of multichannel audio objects), and the index information of the audio objects of each channel (ID and whether the audio object is mono, Information), the object-specific audio signal can be recovered from the downmix audio signal output in step S1203 based on the header information and the spatial cue information of the additional information extracted from the step S1205 .

Next, the rendering control information (e.g., the position and size of the spatial audio object) and the output channel control information (e.g., 5.1 or 7.1 channel, stereo) for each audio object restored by step S1207 The audio signal of each object restored from the step S1207 is inputted, and a multi-object audio signal is outputted (S1209).

13 is a flowchart illustrating a multi-object audio decoding method according to another embodiment of the present invention.

As shown in FIG. 13, the audio information bit stream and the additional information bit stream are separated from the audio bit stream generated in step S1111 (S1301).

Next, the downmix audio signal is recovered from the audio information bit stream separated by the step S1301 (S1303).

Next, additional information including spatial cue information of each audio object is extracted from the bitstream separated by the step S1301 (S1305).

Next, the rendering control information (for example, the position and size of the spatial audio object) and the output channel control information (e.g., 5.1 or 7.1 channel, stereo) for each audio object restored by the step S1303 The additional information (for example, the signal size information and the correlation information of each audio object) extracted from the step S1305 is controlled in accordance with the external input signal (S1307).

Next, the downmix audio signal recovered from the step S1303 is restored to a multi-object audio signal of various channels using the additional information controlled by the step S1307 (S1309).

The object-specific audio signal is restored from the downmix audio signal using the header information of the additional information controlled in step S1307.

In the header information, the number of audio objects per channel (the number of mono audio objects, the number of stereo audio objects and the number of multichannel audio objects), and the index information of the audio objects of each channel (ID and whether the audio object is mono, The audio signal per object is restored from the downmix audio signal output in step S1303 based on the header information and the spatial cue information of the sub information controlled by the step S1307 .

14 is a detailed structural diagram of basic downmixers 201a to 201d according to an embodiment of the present invention.

The basic down-mixer includes a first sub-band splitter 1410, a second sub-band splitter 1420, a transformer 1440, a high-band spatial cue extractor 1440, cue extraction 1430 and a low-band spatial cue extraction 1450 and a sub-band concatenator 1470. The low-

The first sub-band divider 1410 divides the input signal X _R1 [n] into sub-bands according to the frequency band to generate M filtered signals X _R11 [k] to X _R1M [k].

In the input signal X _R1 [n], R represents the right channel, 1 represents the first stereo object, and n represents the time axis. The channel and stereo object are exemplary.

In the sub-band divided signals X _R11 [k] to X _R1M [k], R represents the right channel, the previous number 1 represents the first stereo object, and the following numbers represent the sub- , And k represents a time axis.

15 illustrates sub-band divided signals in accordance with an embodiment of the present invention.

In order, X _R1M [k] is the signal on the time axis of the highest frequency band, and X _R11 [k] is the signal on the time axis of the lowest frequency band.

The j low frequency signals of some of the sub-band divided signals, for example X _R11 [k] to X _R1j [k], are input to the converter 1440.

The converter 1440 windowing each of the input signals.

16 shows an example of windowing for each signal according to an embodiment of the present invention.

Each of the input signals X _R11 [k] to X _R1j [k] consists of N subframes (solid lines). Each of the input signals is bounded by N (dotted lines) of the previous sub-frame of the input signal.

The previous N subframes may be stored in the history buffer of the transcoder.

Since the time axis of the input signal is k, the time axis representing N of the previous sub-frames is k-N.

The transformer 1440 transforms the windowed signal into a frequency domain (time to frequency transform (T / F transform)).

The transform may be, for example, a discrete fourier transform (DFT) or a modified discrete cosine transform.

The transform may be a long-block time-to-frequency conversion scheme such as 1024-DFT or 2048-DFT.

Such a long-block transform can reduce the amount of computation required for the transform.

Since both the subframe of the input signal and the previous subframe of the input signal are both 2N in total, 2N windowing and 2N frequency conversion are performed.

Thus, the sub-band divided signals X _R11 [k] to X _R1j [k] are converted into signals S ¹ _{b, 1} (f) through S ¹ _{b, j} (f) through the transducer 1440, .

The superscript of the signal S is an object index representing a stereo object, the subscript b of the subscript is a frequency bin index, and the number of subscripts is a sub-band index representing a sub-band.

The transducer 1440 may be multiple for each of the sub-band divided signals.

In addition, the transformer 1440 may be a component included in the first sub-band divider 1410 and the second sub-band divider 1420, and the low band- May be included.

The second sub-band splitter 1420 divides the right channel input signal X _R2 [n] of the second stereo object into sub-bands according to the frequency band to generate M filtered signals X _R21 [k] to X _R2M [k].

The transducer 1440 may be operable to receive j low frequency signals of some of the signals partitioned by the second sub-band divider 1420, e.g., X _R21 [k] through X _R2j [k] And a frequency domain transform operation to output signals S ² _{b, 1} (f) to S ² _{b, j} (f).

The number of bands to which the frequency conversion is applied (for example, j) may be increased or decreased in consideration of the sampling rate of the input signal and the like.

The low band band space cue extractor 1450 extracts the frequency converted signals S ¹ _{b, 1} (f) to S ¹ _{b, j} (f) and S2 _{b, 1} (f) to S ² _{b, j} (f)).

The low-band-band space cue extractor 1450 generates and outputs additional information such as a space cue based on the input signal.

The low-band-band space cue extractor 1450 can extract a spatial cue according to Equation (3).

For example, Equation (3) may be applied, assuming that the T / F converted signal of the first sub-band of the second stereo object is S ² _{b, 1} (f). Also, the T / F converted signal of the first sub-band of the first stereo object may be represented by S ¹ _{b, 1} (f).

For example, the low-band-band space queue extractor 1450 extracts S ² _b (f) as S ¹ _{b, 1} (f), S ¹ _b (f) Can be replaced with S ² _{b, 1} (f), and the above equation can be used.

The highband band space cue extractor 1430 extracts Mj high frequency signals X _{R1 (j + 1)} [k] from the first sub-band divider 1410 and the second sub-band divider 1420, X _R1M [k], and X _{R2 (j + 1)} [k] to X _R2M [k].

The highband band space cue extractor 1430 generates and outputs additional information such as a space cue based on the input signal.

The highband band space cue extractor 1430 can extract the space cue according to Equation (4) below.

이다.here,

to be.

The input signal of the low-band band space cue extractor 1450 is analyzed in the frequency domain.

The input signal of the highband band space cue extractor 1430 is applied to a band-filtered signal, and a power operator is applied on the time axis.

Each of the plurality of signal combiners 1460 receives a corresponding sub-band filtered signal (e.g., X _R11 [k] and X _R21 [k]) from the first sub-band divider 1410 and the second sub- [k]), and combines and outputs the two signals.

The signal combiner 1460 may be as many as the number of sub-bands. Also, the signal combiner 1460 can process a plurality of signal pairs.

The signal combiner may be included in the sub-band sequencer 1470.

The signal coupled by the signal combiner 1460 is input to the sub-band chain coupler 1470.

The sub-band sequencer 1470 cascades the combined signals and outputs a downmix signal.

That is, downmixing is performed by the signal combiner 1460 and the sub-band sequencer 1470.

Since the above embodiment receives a right channel signal as an input, the downmix signal is a right downmix signal.

17 is a flowchart illustrating a downmixing method according to an embodiment of the present invention.

First, the two input signals, such as X _R1 in accordance with the time-base-n [n] and X _R2 [n] is the M sub in accordance with the band-band signals (X _R1 [n] is X _R11 [k] to X _R1M [ k] and X _R2 [n] is divided into X _R21 [k] to X _R2M [k] (S1710).

Some of the divided sub-band signals, e.g., k-j of the high frequency band are used to generate additional information of the high band band (S1720).

Details of the generation of the additional information of the high band are described above with reference to FIG.

The other part of the divided sub-band signals, e.g., j in the low frequency band, is windowed (S1730).

The windowed signal is converted into a frequency domain (S1740).

The signal converted into the frequency domain is used to generate additional information of a low-band (S1750).

Details of the windowing, the frequency domain transformation, and the generation of the additional information of the low-band band have been described above with reference to FIG.

The divided sub-band signals are downmixed to generate a downmix signal (S1760).

Details of the downmix signal generation have been described above with reference to FIG.

18 is a detailed configuration diagram of a SAC decoder 1003 according to an embodiment of the present invention.

The SAC decoder 1003 includes a sub-band splitter 1810, a high band band up mixer 1820, a converter 1830, a low band band up mixer 1840, a delay 1850, an inverse transformer 1800 And a sub-band sequencer 1870.

The sub-band splitter 1810 receives the decoded right downmix signal as an input. The sub-band divider 1810 divides the input signal into sub-bands according to a frequency band as shown in FIG. 15 to generate M filtered signals X _R1 [k] to X _RM [k] And outputs it.

Of the signals divided into sub-bands, j low frequency band signals of, for example, X _R1 [k] to X _Rj [k] are input to the converter 1830.

The converter 1830 window-converts each of the input low frequency signal bands and converts the frequency bands into frequency bands.

Therefore, the sub-band divided signals X _R1 [k] to X _Rj [k] are frequency-domain transformed into signals S _{b, 1} (f) to S _{b, j} (f) through the transducer 1830 .

The details of the windowing and the frequency conversion are the same as those described for the converter 1440 in FIG.

The low-band band upmixer 1840 receives the frequency-converted signals S _{b, 1} (f) to S _{b, j} (f), and receives the additional information bit stream. The low-band band upmixer 1840 performs upmixing on the input signal based on the additional information bitstream.

The upmixing corresponds to Equation (3) and Equation (4), and is a process of extracting a gain value from a general CLD value.

The relationship between the power P ₁ of the first stereo object and the power P ₂ of the second stereo object is expressed by Equation (5).

Therefore, the upmixing may be performed according to Equation (6) below.

The input signal S _{b, 1} (f) is divided into S ¹ _{b, 1} (f) for the first stereo object and S ² _{b, 1} (f) for the second stereo object according to the additional information. And the input signals for the other sub-bands are also output in accordance with the additional information.

Then, the signal for the first stereo objects (S ¹ _{b, 1} (f) to S ¹ _{b, j} (f)), and wherein the S (signal for the second stereo object ¹ _{b, 1} (f) to S ¹ _{b, j} (f) are processed separately. In this embodiment, further processing of the signal for the second stereo object is not shown.

Next, the signals S ¹ _{b, 1} (f) to S ¹ _{b, j} (f) for the first stereo object are transformed into X _R11 [k] to X _R1J [k] by the inverse transformer 1860, do. The conversion process through the inverse transformer 1860 will be described in detail below.

The inverse transformer 1860 may perform an inverse frequency transform, such as an inverse discrete fourier transform (IDFT) or an inverse transform, for each of the input signals S ¹ _{b , 1} (f) through S ¹ _{b ,} And performs discrete cosine transform.

Next, the inverse transformer 1860 performs windowing on each of the inverse-frequency-converted signals to generate a windowed signal.

19 shows an example of windowing for each signal according to an embodiment of the present invention.

Each of the input signals S ¹ _{b, 1} (f) to S ¹ _{b, j} (f) consists of N subframes (solid lines). Each of the input signals is bounded by N (dotted lines) of the previous sub-frame of the input signal.

The inverse transformer 1860 performs an overlap-add operation on the windowed signal to output X _R11 [k] to X _R1J [k], which are signals before frequency conversion.

The overlapping portion is for eliminating the window effect.

FIG. 20 shows an example of superposition-addition for each windowed signal in accordance with an embodiment of the present invention.

The windowed signal is represented as a current low-band sub-band with 2N subframes as shown in FIG.

The superpositioner 2020 receives the current low-band sub-band output 2040, that is, the next N sub-frames of the sub-frame of the windowed signal and the previous sub-frame 2030 of the previous low- form N subframes to generate low-band sub-band outputs X _R11 [k] to X _R1J [k] (2050).

The previous N subframes may be stored in the history buffer 2010 of the transcoder.

That is, the input signal S ¹ _{b, 1} (f) of the inverse transformer 1860 is converted into X _R11 [k] through the above-described IDFT, windowing,

The high band band up mixer 1820 receives Mj high frequency band signals of, for example, X _{R (j + 1)} [k] to X _RM [k] among the sub- And receives an information bit stream. The high band band up mixer 1820 performs up-mixing on the input signal based on the additional information bit stream.

The upmixing corresponds to Equation (3) and Equation (4), and is a process of extracting a gain value from a general CLD value.

The relationship between the power P ₁ of the first stereo object and the power P ₂ of the second stereo object is as shown in Equation (5).

Therefore, the upmixing may be performed according to the following Equation (7).

The input signal X _RM [k] is separated and output as X _R1M [k] for the first stereo object and X _R2M [k] for the second stereo object according to the additional information, And are output separately according to the additional information.

Thereafter, the signals X _{R1 (j + 1)} [k] to X _R1M [k] for the first stereo object and the signals X _{R2 (j + R2M} [k]) are separately processed. In this embodiment, further processing of the signal for the second stereo object is not shown.

The signals X _{R1 (j + 1)} [k] to X _R1M [k] for the first stereo object separated by the highband band upmixer 1820 are input to the delay unit and output to a delay unit ). The delay is to synchronize with the lowband band signals (X _R11 [k] to X _R1J [k]) that have been processed by the converter 1830 and the inverse transformer 1860.

The sub-band sequencer 1870 receives the highband band signals X _{R1 (j + 1)} [k] to X _R1M [k] through the delay 1850 and the low- And _{receives the} band-band signals X _R11 [k] to X _R1J [k] and generates a concatenated signal X _R1 [nD] through the sub-band chain. The concatenated signal is delayed by D.

The chain combines up-sampling signals of each sub-band through a conventional up-sampling process.

The high band band up mixer 1820, the delay 1850, the converter 1830, the low band band up mixer 1840 and the inverse transformer 1860 are respectively provided with a plurality There can be dogs. The high band band up mixer 1820, the delayer 1850, the transformer 1830, the low band band up mixer 1840 and the inverse transformer 1860 are connected to the sub- Lt; / RTI >

21 is a flowchart illustrating a decoding method according to an embodiment of the present invention.

First, the decoded downmix signal is divided into M sub-band signals X _R1 [k] to X _RM [k] according to the frequency band (S2110).

Some of the divided sub-band signals, for example, k-j signals in a high frequency band, are subjected to high-band-up mixing and separated into a signal for the first stereo object and a signal for the second stereo object (S2120).

Details of the high-band band-up mixing have been described above with reference to FIG.

The high-band band-up mixed signal is delayed for synchronization with other signals (S2130).

Details of the delay of the highband band signal have been described above with reference to FIG.

On the other hand, a part of remaining sub-band signals, for example, j signals in a low frequency band are windowed (S2140), and the windowed signal is frequency-domain transformed (S2150).

The details of the windowing and the frequency domain transformation have been described above with reference to FIG.

The frequency-domain-converted low-band-band signal is low-band-up-mixed and separated into a signal for the first stereo object and a signal for the second stereo object (S2160).

Details of the low-band band upmixing have been described above with reference to FIG.

The low-band band upmixed signal is reverse-frequency-converted (S2170). The inverse-frequency-converted signal is windowed (S2175), and the windowed inverse-frequency-converted signal is superimposed-added to remove the window effect (S2180).

Details of the inverse frequency conversion, windowing and superposition-addition have been described above with reference to FIG.

The high-band-delay and low-band-superimposed-added signals are sub-band cascaded (S2190).

Details of the sub-band chain have been described above with reference to FIG.

An output signal is generated by the sub-band chain and the procedure ends.

The method according to an embodiment of the present invention can be implemented in the form of a program command which can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

101: 1st down mixer
103: Down mixer
105: Audio Encoder
107: additional information encoder
109: multiplexer
111: Mono channel downmix
112: Stereo channel downmix
115: Multichannel Downmix

Claims (15)

delete delete delete delete In the decoding method,
Decoding a downmix audio signal from an audio information bitstream generated by an encoding apparatus;
Extracting additional information from the additional information bitstream generated by the encoding apparatus;
Controlling the additional information according to rendering control information and output channel control information for the downmix audio signal;
And decoding the object-specific audio signal from the downmix audio signal using the controlled additional information
Lt; / RTI >
Wherein the decoding of the object-specific audio signal comprises:
Dividing the downmix audio signal into M subband signals according to a frequency band;
Windowing the first through jth subband signals of the lower frequency band among the M subband signals;
Transforming the windowed subband signals into a frequency domain;
Upmixing the frequency-domain-converted subband signals according to the sub information bit stream and separating them into low band signal signals for each of the stereo objects;
Frequency-band-shifting the low-band signals;
Banding the previous subframe of the inverse frequency-converted low-band signal with the inverse frequency-converted low-band signals;
Superposing subsequent subframes and previous subframes of the windowed lowband-band signals to output a low-band-band signal from which the window effect has been removed;
Upmixing and delaying Mj subband signals having a higher frequency band than the j-th subband signal having a lower frequency among the M subband signals to output a highband band signal; And
And decoding the audio signal for each object by subband-banding the low-band signal and the high-band signal from which the window effect has been removed
/ RTI > 6. The method of claim 5,
Wherein the step of controlling the additional information comprises:
And controlling the signal size information and the correlation information of the audio object among the additional information according to the rendering control information and the output channel control information. delete 6. The method of claim 5,
Wherein the step of outputting the high-
Upmixing the Mj subband signals into additional information and separating them into highband band signals for each of the objects;
And outputting the low-band signal by delaying and outputting the high-band signal according to a time required for converting the j sub-band signals into a window and transforming the j sub-
/ RTI > delete In the encoding method,
Grouping a plurality of multi-object audio signals composed of different channels by channels;
Mixing down the multi-object audio signals grouped by the channel to generate a downmix audio signal, and extracting additional information;
Mixing the downmix audio signals to generate an audio information bitstream; And
Generating a side information bitstream using the side information
/ RTI >
Wherein the extracting comprises:
Dividing the multi-object audio signals into M sub-band signals;
Windowing the previous subframe of the first through jth subband signals of the lower frequency band among the M subband signals by combining the first subframe and the jth subband signals;
Transforming the windowed subband signals into a frequency domain;
Extracting additional information of a low-band signal using a frequency-domain-converted subband signal;
Extracting additional information of a high-band band using Mj subband signals having a higher frequency band than the j-th subband signal having a lower frequency band among the M subband signals; And
And downmixing the M sub-band signals divided in the multi-object audio signals to generate a downmix audio signal
/ RTI > 11. The method of claim 10,
Wherein the extracting comprises:
Mixing the multi-object audio signals grouped by the channel with a first downmixer to generate a first downmix signal and extracting additional information; And
Mixing the first downmix signal with a second downmixer to generate the downmix audio signal and extracting additional information
/ RTI > 12. The method of claim 11,
Wherein the first down mixer comprises:
Wherein the grouped multi-object audio signals include N-1 basic downmixers in a cascade structure when the number of audio objects included in each of the channels of the grouped multi-object audio signals is N. 12. The method of claim 11,
The step of generating the first downmix signal and extracting additional information includes:
Identifying channels of the grouped multi-object audio signals as one of a mono channel, a stereo channel, and a multi-channel; And
Downmixing the channels of the grouped multi-object audio signals with a mono channel downmixer, a stereo channel downmixer, or a multi-channel downmixer according to the identification result
/ RTI > 14. The method of claim 13,
Wherein the step of generating the downmix audio signal and extracting additional information comprises:
Mixing the first downmix signal output from the stereo channel downmixer and the first downmix signal output from the multi-channel downmixer to output a representative downmix signal, and extracting additional information; And
Mixing the first downmix signal and the representative downmix signal output from the mono channel downmixer to generate a second downmix signal and extracting additional information
/ RTI >

delete

Images