JP5696828B2 - Signal processing device - Google Patents

Description

  The present invention relates to a signal processing apparatus that determines the contents of an input audio signal.

  In recent years, multi-channel audio devices have become widespread. A multi-channel audio device reproduces audio signals of channels (multi-channel), such as 5.1 channels, which are more than two stereo channels, and outputs these signals from a plurality of speakers installed in various places in the room. This is an apparatus for reproducing audio having a three-dimensional spread (Patent Document 1).

  In the conventional multi-channel audio signal, the content of the audio signal assigned to each channel (channel assignment) is almost unified. That is, speech such as speech is assigned to the center channel, musical sounds such as BGM are assigned to the front left and right channels, and other sounds such as environmental sounds and sound effects are assigned to the surround left and right channels.

  The multi-channel audio device has a function of performing sound field control that creates a reverberation of a virtual space such as a hall by adding a reflected sound or a reverberation sound to a reproduced audio signal. However, if the effect of reflected sound or reverberation sound is strongly added to speech such as speech, the clarity will be reduced and it will be difficult to hear what the performer is talking about, so the channel where the speech is played back Is set to be smaller than the other channels.

  In the case of the conventional content described above, speech such as speech is generally allocated to the center channel. Therefore, in the conventional multi-channel audio apparatus, the sound field control amount of the center channel is reduced and the sound field of other channels is reduced. The control amount was previously set to be large or medium.

  However, with the start of terrestrial digital broadcasting and the like, multi-channel audio content that can be played back at home is diversifying, and the number of channels that are not assigned to channels as in conventional movies is increasing. In other words, content in which speech is assigned to front channels and surround channels that are not center channels is also increasing.

  When such multi-channel audio content is reproduced with the conventional setting of the sound field control amount, a strong reflected sound or reverberation effect is applied to speech such as speech, and the clarity is lowered. Further, when a musical sound such as BGM is played on the center channel, there is a problem that the sound field effect is not applied to the BGM and the atmosphere cannot be raised.

  Therefore, it is conceivable to detect what kind of sound is reproduced in which channel and adjust the sound field control amount of each channel. In particular, it is conceivable to detect in which channel the voice is being reproduced and to reduce the control amount of the sound field of the channel in which the voice is being reproduced. As methods for detecting speech from an audio signal, methods such as Patent Documents 2 and 3 have been proposed.

  Japanese Patent Application Laid-Open No. H10-228667 describes detecting speech using an autocorrelation function of a time waveform of voiced sound. Japanese Patent Application Laid-Open No. H10-228867 describes that an occupancy degree of a harmonic structure component in an acoustic signal is obtained by using an instantaneous frequency analysis, and a voice section is detected based on this.

JP-A-8-275300

Japanese Examined Patent Publication No. 4-55320

Japanese Patent No. 3892379

However, in the method of Patent Document 2, since the speech is only detected based on the autocorrelation function of the time waveform, a signal having a periodicity such as a sine wave but not having a harmonic structure is erroneously detected as the speech. There was a problem. Moreover, in the method of Patent Document 3, it is necessary to perform instantaneous frequency analysis for every frame and all frequency bands, and the amount of calculation is enormous.
In any of the above systems, it is impossible to discriminate between spoken voice, single tone, ensemble tone and other sounds.

  An object of the present invention is to provide a signal processing apparatus that can accurately detect the contents of an audio signal by a process as simple as possible.

The present invention relates to a musical tone determination unit that determines whether or not the audio signal is a musical tone by comparing the energy of the scale frequency component of the audio signal with the energy of the entire band component, and the musical tone determination unit determines whether the audio signal is a musical tone when it is not determined that the by determining the presence or absence of harmonic of the audio signal, said audio signal determines harmonic determination section whether the other sound or a tone sound of waves, the harmonicity When the determination unit determines that the audio signal is a harmonic sound, the audio signal is based on whether the pitch frequency of the audio signal matches the scale frequency or the presence or absence of fluctuation of the pitch frequency. A speech / musical sound determination unit for determining whether the signal is a voice or a music is provided.
In the above invention, the harmonicity determination unit determines the presence or absence of harmonics based on an autocorrelation function of a frequency spectrum by short-time Fourier transform, and the speech / musical sound determination unit is based on the autocorrelation function. Means for obtaining an accurate pitch frequency by performing an instantaneous frequency analysis only in the vicinity of the obtained approximate pitch frequency may be included.

  According to the present invention, it is possible to discriminate the content (speech / musical tone, etc.) of an audio signal with a relatively simple process.

Block diagram of an audio apparatus including a signal processing unit according to an embodiment of the present invention The figure which shows the example of the channel allocation of a multichannel audio signal Block diagram of the signal processor The flowchart which shows the process of the content discrimination | determination part of the signal processing part The figure explaining the musical tone determination process of a content determination part Flowchart showing harmonic determination processing of content determination unit Diagram showing frequency spectrum and autocorrelation function of various audio signals Flow chart showing speech / musical sound determination processing of content determination unit The figure explaining the correlation between the frequency bin of STFT and the instantaneous frequency

<Configuration of audio device>
FIG. 1 is a block diagram of an audio apparatus including a signal processing unit according to an embodiment of the present invention. The audio device has a content reproduction device 2, an audio amplifier 1, and a plurality of speakers 3. The audio amplifier 1 has a signal processing unit 4 and an amplifier circuit 5.

  The content playback apparatus 2 is composed of, for example, a DVD player that plays back a DVD such as a movie, a satellite, and a TV broadcast tuner that receives a terrestrial TV broadcast. The content reproduction device 2 inputs a multi-channel (for example, 5.1 channel) audio signal to the audio amplifier 1. The signal processing unit 4 of the audio amplifier 1 performs processing such as equalizing and sound field control on the multi-channel audio signal input from the content reproduction device 2 and then inputs the processed signal to the amplifier circuit 5. The amplifying circuit 5 individually amplifies the input multi-channel audio signals and outputs them to the speakers 3 corresponding to the respective channels.

  The plurality of speakers 3 are installed at various locations in the listening room, and sound of each channel is emitted from each speaker 3, thereby forming a sound field that spreads in the listening room.

《Example of content channel assignment》
Here, with reference to FIG. 2, channel assignment of a multi-channel audio signal input from the content reproduction apparatus 2 to the audio amplifier 1 will be described.

  FIG. 2A is a diagram showing channel assignment of multi-channel audio signals of general movie content. In this embodiment, a 5.1 channel audio signal will be described as an example. The 5.1-channel audio signal includes a center channel C, a front left channel FL, a front right channel FR, a surround (rear) left channel SL, a surround (rear) right channel SR, and a subwoofer channel SW. Among these, since the subwoofer channel SW is configured by collecting the deep bass signals of other channels, the number of channels input from the content reproduction apparatus 2 is five. Therefore, in the following, channel assignment of five channels, center channel C, front left channel FL, front right channel FR, surround left channel SL, and surround right channel SR will be described.

  In the case of general content, speech such as speech is assigned to the center channel C, musical sounds such as BGM are assigned to the front left and right channels FL and FR, and other sounds (such as sound effects and environmental sounds) are assigned to the surround left and right channels SL and SR.

  In general, the amount of sound field effect (sound field control amount) is reduced in order to prevent the content spoken from being obscured from being spoken. For musical sounds such as BGM, the sound field control amount is increased so that the sound is rich. For other sounds such as environmental sounds and sound effects, the sound field control amount is set to a medium level. Accordingly, the sound field control amount of the center channel C is set to “small”, the sound field control amounts of the front left and right channels FL and FR are set to “large”, and the sound field control amounts of the surround left and right channels SL and SR are set to “medium”.

  On the other hand, FIG. 5B is a diagram showing an example of channel assignment of content other than general movie content, for example, a multi-channel audio signal for digital television broadcasting. In this example, the center channel C is silent, and speech and BGM such as speech are assigned to the front left channel FL, musical sounds such as BGM are assigned to the front right channel FR, and other sounds are assigned to the surround left and right channels SL and SR.

  In such a case, the sound field control amount of the center channel C is arbitrary (the sound field effect is substantially zero because there is no input signal), the sound field control amounts of the front left and right channels FL and FR are “small”, and the surround left and right channels. The sound field control amount of SL and SR is set to “medium”.

  That is, the voice and musical sound are synthesized and output in the front left channel FL. In this case, the voice is given priority, and the sound field control amount is set to “small”. In addition, the front right channel FR is only a musical sound, but if the balance of the sound field control of the left and right channels is lost, it may give the listener an unstable impression. “Small”. In this case, the sound field control amount of the front right channel FR may be set to “large” in accordance with the musical sound, or may be set to “medium” in the middle of them.

<Configuration of signal processing unit>
FIG. 3 is a diagram illustrating a configuration example of the signal processing unit 4. The signal processing unit 4 is a functional unit that performs various processes such as equalizing and applying a sound field effect. FIG. 3 shows only a component that provides the sound field effect. The input unit 10 includes five input units, a center channel input unit 10C, a front left channel input unit, a front right channel input unit, a surround left channel input unit, and a surround right channel input unit. Audio signals of each channel (C, FL, FR, SL, SR) are input.
Hereinafter, as with the input unit 10 described above, the description of each individual channel is omitted for the components provided in parallel for five channels.

  The audio signal input from the input unit 10 is input to the content determination unit 14 and the delay unit 11. The content determination unit 14 is provided in parallel for five channels, and determines the content of the audio signal of each channel. The content is information indicating whether the audio signal is a voice / musical sound / other sound.

  The content discriminating unit 14 discriminates speech / music / other sounds by measuring the presence / absence of a harmonic structure, modulation spectrum, harmonic structure, frequency change rate, and the like. Details of the determination processing of the content determination unit 14 will be described later.

  The delay unit 11 delays the audio signal by a time necessary for the content determination unit 14 to determine the content of the audio signal. Thereby, the control delay of the sound field control based on the determination result of the content determination unit 14 is eliminated.

  The determination result of the content determination unit 14 is input to the coefficient control unit 15. The coefficient control unit 15 determines a sound field control amount for the audio signal of each channel according to the contents of the audio signal of each channel. The sound field control amount is determined by the rules as shown in FIG. The content determination unit 14 determines a sound field control amount for the audio signal of each channel, and outputs a coefficient for controlling the audio signal to an input level corresponding to the sound field control amount. The coefficient is input to the coefficient multiplier 16.

  The coefficient multiplier 16 multiplies the audio signal delayed by the delay unit 11 by the coefficient input from the coefficient controller 15 and inputs the result to the adder 17. The coefficient multiplication unit 16 is provided in parallel for five channels. The adder 17 adds and synthesizes 5-channel audio signals each multiplied by a coefficient. The level of the added and synthesized audio signal is controlled by the level control unit 18, and then a sound field effect including an initial reflection sound and a reverberation sound is applied by the sound field effect generation unit 19.

  As the level of the audio signal input to the sound field effect generation unit 19 increases, the sound field effect sound (reflected sound, reverberation sound) generated by the sound field effect generation unit 19 increases. Therefore, the degree of the sound field effect given to the audio signal of each channel is controlled by the coefficient generated by the coefficient control unit 15.

  Based on the sound field data 20, the sound field effect generator 19 reproduces the sound of the sound in a hall or a room. That is, the initial reflection sound and reverberation sound generated in the hall and the room are generated. This processing includes filter processing for simulating changes in frequency characteristics due to spatial propagation and reflection, initial reflected sound generation processing by delay and coefficient multiplication, rear reverberation sound generation processing, and the like.

  The sound field effect sound generated by the sound field effect generation unit 19 is added to the dry audio signal via the coefficient multiplication unit 21 and the addition unit 12. A coefficient multiplier 21 and an adder 12 are also provided in parallel for five channels. In general, since the clarity of speech is higher when a sound field effect sound is not added to a channel such as a speech output channel, the coefficient multiplier 21 causes the sound field effect sound to be transmitted to the speech channel. Set the addition gain to 0.

  The coefficient input to the coefficient multiplier 21 may be set by the coefficient controller 15. The coefficient of the channel where the voice is output can be set to “0” and the coefficient of the other channels can be set to “1”, but the coefficient value is changed to an intermediate value between “0” and “1” for each channel. You may let them.

  With such control, each channel plays a wide and rich sound field effect during the period other than the line is played, and when the line is played, it reduces the amount of sound field effect on the line. It is possible to suppress excessive noise and achieve both a rich sound field effect and clear lines.

<< Description of Processing of Content Determination Unit 14 >>
The content determination process of the content determination unit 14 will be described with reference to FIGS. This process is executed every frame (40 ms). In the harmonic determination process (S4), in addition to the self frame, the data of the three frames before and after are used together, so the determination process is delayed by three frames. The delay unit 11 delays the audio signal by the delay time of this discrimination process.

  FIG. 4 is a flowchart showing the entire content determination process. First, a musical tone determination process is performed (S1). The musical tone determination process is a process of measuring a ratio occupied by a scale frequency component among the frequency components of the audio signal. Details of the tone determination will be described later with reference to FIG. If it is determined by the tone determination process that the tone is a tone (YES in S2), “musical tone” is output as the content determination result (S3), and the process ends.

If the musical sound is not determined by the musical sound determination process (NO in S2), the harmonic determination process is performed (S4). The harmonic determination process is a process for determining whether the audio signal has harmonic characteristics, that is, whether it has a spectral structure including a fundamental tone and an overtone component that is an integral multiple of the fundamental tone. Details of the harmonic determination processing will be described later with reference to FIG. If it is determined by the harmonic determination process that there is no harmonic (NO in S5), “other sound” is output as the content determination result (S6). On the other hand, that there is by Ri harmonicity the harmonic determination process when it is determined (YES in S5), since the audio signal is considered to be speech or tone, speech / tone determination processing (S7) Do.
That is, voices and musical sounds have harmonics, but sounds such as environmental sounds and sound effects do not have harmonics.

  In the speech / musical sound determination process, an accurate fundamental frequency (pitch) is calculated, and whether the audio signal is a musical sound based on whether the pitch matches the scale frequency or there is no large fluctuation in the pitch. Determine if it is a voice. Details of the voice / musical tone determination processing will be described later with reference to FIG. If the determination result is speech, “speech” is output as the content determination result (S9). If the determination result is a musical sound, “musical sound” is output as the content determination result (S10).

  FIG. 5A is a flowchart showing a musical tone determination process. In this process, it is determined whether or not the audio signal is a musical tone (particularly a ensemble musical tone) by measuring the energy of the scale frequency component in the energy of the entire frequency band of the audio signal.

  First, the energy of the scale frequency component in the audio signal and the energy of the entire frequency band are measured (S20). FIG. 5B shows a block diagram of a functional unit that measures the energy of the audio signal. The energy measurement of the scale frequency component is obtained by adding the energy of 12 sounds of a specific octave. As the specific octave, an octave in which a melody is played, for example, a C3 to B3 octave may be used. For this reason, a BPF filter having 12 to 12 scales of C to B is provided. The frequency components that have passed through each filter are integrated to obtain the energy of each frequency component, and these are added. This sum is the energy of the scale frequency component. On the other hand, the audio signal is directly integrated to obtain energy in the entire frequency band.

  The energy of the scale frequency component obtained in S20 and the energy of all frequency band components are compared (S21). If the ratio is equal to or greater than a predetermined ratio, that is, the ratio of the scale frequency component energy is a predetermined value. If so ("YES" in S22), "musical sound" is output as the determination result (S23). On the other hand, when the ratio of the energy of the scale frequency component does not reach the predetermined value (NO in S22), the process ends without outputting the determination result.

  Thus, since it is possible to determine whether or not the audio signal is a musical tone only by a plurality of BPF filter processing and integration processing, if the audio signal is determined to be a musical tone by this processing, S4 and subsequent steps in FIG. This processing can be omitted, and the processing load can be greatly reduced. Further, in this musical tone determination process, even a musical tone of an ensemble composed of a plurality of musical instruments that does not exhibit a clear harmonic characteristic can be easily detected because many components appear in the scale frequency.

FIG. 6 is a flowchart showing harmonic determination processing. In this process, the audio signal is subjected to short-time Fourier transform (STFT), and the autocorrelation of its frequency spectrum is obtained to obtain the presence / absence of harmonics and the peak frequency (approximate pitch frequency).

  Here, the STFT is performed using data for five frames including the data of the current frame and the two frames before and after that. Further, an average value obtained by adding the STFT result of the previous frame and the STFT result of the next frame to the STFT result of the current frame is used as the frequency spectrum P (T) of the current frame. Therefore, the frequency spectrum of the current frame is obtained three frames after the current frame.

  By averaging the frequency spectra of a plurality of frames in this way, the frequency components that are continuously present are emphasized. That is, since noise components such as background sounds do not exist continuously, they are not emphasized on the spectrum. However, components that exist continuously such as speech and musical sounds have their harmonic components emphasized on the spectrum. As a result, even if a voice or musical sound with a low level buried in the background sound is present in the audio signal, it is possible to detect this and measure the peak frequency.

  In FIG. 6, first, short-time Fourier transform is performed by the above-described method to obtain a frequency spectrum P (T) of the current frame (time: T) (S31). FIG. 7A shows an example of the FFT result. This example is a signal spectrum of speech only.

  Next, the autocorrelation of this frequency spectrum is detected (S32). 7B, 7C and 7D show examples of autocorrelation functions. FIG. 7B shows the autocorrelation function of the frequency spectrum of only the speech shown in FIG. 7A, and the autocorrelation clearly appears. FIG. 7C is a diagram illustrating an example of an autocorrelation function of the frequency spectrum of an audio signal including speech and components other than speech. Since the frequency band occupied by the voice is narrow, autocorrelation appears in the range where the frequency difference is small, but the autocorrelation is disturbed in the range where the frequency difference is large. FIG. 7D is a diagram showing an example of the autocorrelation function of the frequency spectrum of other sounds. As described above, since the other sounds do not have harmonics, there is no autocorrelation of the frequency spectrum.

  The first peak of the autocorrelation function is detected, and the frequency difference between the peaks is set as the peak frequency Fa (S33). As illustrated in FIG. 7D, when the peak frequency Fa cannot be detected (NO in S34), a determination result of “no harmonics” is output (S39), and this process ends.

  When the peak frequency Fa is detected, the peak frequency Fa is compared with the pitch frequency F (T-3) of the immediately preceding frame (T-3) (S35). If the difference is less than or equal to the predetermined value (substantially coincides) (YES in S36), the peak frequency Fa is set to the pitch frequency F (T-2) in the current frame (F-2) of the audio signal (S37). ). Then, the determination result “having harmonics” and the pitch frequency “F (T-2)” are output (S38), and the harmonics determination process is terminated.

  On the other hand, if the difference between Fa and F (T-3) is greater than the predetermined value (NO in S36), the determination result of “no harmonics” is output (S39), and the processing operation is terminated. .

  The harmonic nature of speech and musical sound does not appear and disappear instantaneously, but continues for multiple frames, so the harmonic nature is only when the peak frequency of this time is almost the same as the pitch frequency of the previous frame. It is judged that there is, and false detection is prevented.

  FIG. 8 is a flowchart showing the voice / musical tone determination process. In this process, the exact pitch frequency of the current discrimination target frame is calculated, and whether this audio signal is a musical sound based on whether this pitch frequency matches the scale frequency or if there is no large fluctuation in the pitch frequency. It is determined whether the voice is spoken.

  First, the instantaneous frequency was analyzed using the STFT frequency spectrum obtained by the processing of FIG. 6 and the pitch frequency F (T-2) of the accuracy of the STFT resolution obtained by the frequency resolution, and obtained based on this. An accurate pitch frequency is set to Fe (T-2) (S50). That is, the instantaneous frequency analysis is not performed for the entire frequency band, but the instantaneous frequency analysis is performed only in the vicinity of the approximate pitch frequency F (T-2) obtained by the STFT. Thereby, the processing amount in the instantaneous frequency analysis can be greatly reduced.

  The instantaneous frequency is obtained as a time derivative φ ′ of the phase φ of the signal component waveform of each frequency bin of the STFT. Usually, the instantaneous frequency φ ′ substantially coincides with the frequency of each frequency bin and shows a linear function correlation as shown in FIG. 9A. It is known that when there is a component Fe, the instantaneous frequency φ ′ of the frequency bin in the vicinity of the frequency component Fe in the STFT becomes a substantially constant value. In this case, it can be estimated that the vertical value of the intersection of the approximate pitch frequency F obtained by the STFT and the substantially horizontal portion of the correlation curve is the accurate pitch frequency Fe. In this way, it is possible to obtain an accurate pitch frequency Fe (T-2) with an accuracy of 0.2 Hz.

The accurate pitch frequency Fe (T-2) is compared with the scale frequency (S51). In this process, the frequencies of all 12 semitones in the octave range where a musical sound can exist are compared with Fe (T-2). If they substantially match (YES in S52), it is determined that the audio signal is a musical tone, and a musical tone determination result is output (S56), and the process is terminated. On the other hand, when Fe (T-2) does not match the scale frequency, the accurate pitch frequency Fe (T-3) obtained in the previous frame is compared with the current accurate pitch frequency Fe (T-2). (S53). If the previous accurate pitch frequency Fe (T-3) and the current accurate pitch frequency Fe (T-2) are substantially the same (YES in S54), there is almost no fluctuation in the pitch frequency, so that the determination of the musical tone is made. The result is output (S56). On the other hand, if the previous accurate pitch frequency Fe (T-3) and the current accurate pitch frequency Fe (T-2) do not match (NO in S54), the voice is judged because the pitch fluctuates. The result is output (S55).

  That is, the musical sound is a sound having a stable frequency, but the voice has a frequency inflection, and there is a relatively large pitch fluctuation (fluctuation). If there is no accurate pitch Fe of the previous frame (this process was not performed in the previous frame), it is determined in S54 that they do not match. Note that it is possible to determine not only a human voice but also an animal cry as a voice by this voice / musical sound determination process.

  The audio signal that is not determined as a musical sound by the musical sound determination process (S1 of FIG. 4 or the process of FIG. 5) and is determined as a musical sound by the speech / musical sound determination process is, for example, a single tone performance such as one flute. Therefore, it is a musical sound with a small energy occupied by a musical scale, or a musical instrument with a pitch that does not match the western 12-tone musical scale such as a folk instrument.

  In this embodiment, the contents discriminating unit 14 is provided for all the channels and the contents of all the channels are discriminated. However, it is not always necessary to discriminate the contents of all the channels. Only the channel) may be discriminated. Further, it is not necessary to discriminate all the contents of the voice / musical sound / other sounds, and only a part of the contents (for example, the voice) may be discriminated.

《Still Write》
In the above embodiment, the sound field effect of adding the initial reflected sound or the reverberation sound to the audio signal has been described, but the signal processing in the present invention is not limited to the sound field effect.

  In the above embodiment, a 5.1 channel multi-audio signal has been described as an example. However, the number of channels of the multi-channel audio signal is not limited to 5.1 channels.

DESCRIPTION OF SYMBOLS 1 Audio amplifier 4 Signal processing part 14 Content determination part 15 Coefficient control part 16 Coefficient multiplication part 19 Sound field effect production | generation part

Claims (2)

A musical sound determination unit that determines whether the audio signal is a musical sound by comparing the energy of the scale frequency component of the audio signal with the energy of the entire band component;
A harmonic that determines whether the audio signal is harmonic or other sound by determining whether or not the audio signal is harmonic when the audio signal is not determined by the musical sound determination unit. A sex determination unit;
When the audio signal is determined to be a harmonic sound by the harmonic determination unit, whether or not the pitch frequency of the audio signal matches the scale frequency, or based on the presence or absence of fluctuation of the pitch frequency A voice / musical sound determination unit for determining whether the audio signal is a voice or a music;
A signal processing apparatus comprising: The harmonic determination unit determines the presence or absence of harmonics based on the autocorrelation function of the frequency spectrum by short-time Fourier transform,
The signal according to claim 1, wherein the speech / musical sound determination unit includes means for obtaining an accurate pitch frequency by performing an instantaneous frequency analysis only in the vicinity of an approximate pitch frequency obtained based on the autocorrelation function. Processing equipment.

Images