JPH10228296A - Acoustic signal separation method - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To enable separation of these sound source signals with high accuracy even when there are many sound sources and these fluctuate variously. SOLUTION: A rising edge of a sound is detected by paying attention to power fluctuation of an input audio signal, and the input audio signal is classified at the rising edge (13).
A storage waveform having a fundamental frequency falling within a certain range is selected as a candidate from the storage means 14 (15), and the sum of the results of performing FIR filter processing on each candidate and the mean square error of the corresponding z are minimized. The impulse response h _n of the FIR filter is obtained, and the corresponding candidate is subjected to FIR filter processing using this h _n. If the resulting power is large, it is determined that the candidate sound source is included in z. I do.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a sound signal separation method for separating and extracting sounds of individual sound sources included in a sound signal based on the sound signal in which sounds from a plurality of sound sources are mixed. .

[0002]

2. Description of the Related Art Conventionally, as for an acoustic signal separation method, there is known a method of separating sound sources by a filter device such as a comb filter that passes only a specific frequency band. However, this method has a drawback that it is generally difficult to separate when a large number of sound sources exist, since a proper separation process cannot be performed when a plurality of sound sources share a certain frequency band.

Further, a method is known in which after performing frequency analysis on an input audio signal, the audio signal is separated by a clustering technique by focusing on the characteristics of the power spectrum. However, since this method performs processing from the bottom up, there is a disadvantage that the processing cannot be properly performed when noise is mixed in or when a large number of sound sources are included.

There is also known a method in which a model of a sound source is stored in a device in the form of a power spectrum or the like, and a model suitable for an input sound signal is selected and collated to separate sound signals. However, this method has a drawback that it is not possible to cope with the variety and fluctuation of sound sources because the model is fixed. Therefore, in each of the above methods, when a large number of sound sources are present and the sound sources are various and have fluctuations, it is difficult to expect sufficient sound signal separation processing.

[0005]

SUMMARY OF THE INVENTION The present invention can sufficiently separate even a large number of sound sources, even if the sound sources are diverse and have fluctuations, that is, compared with known methods. It is an object of the present invention to provide an audio signal separation method capable of separating an audio signal with high accuracy.

[0006]

According to the present invention, an input audio signal is temporally divided, and all the waveforms possibly contained in the divided input audio signal are stored in the waveform storage means. Candidate waveforms are selected from among the stored waveforms to obtain candidate waveforms, and the coefficients of the filter processing are determined so as to minimize the mean square error between the sum of the results of performing filter processing on each of these candidate waveforms and the divided input acoustic signal waveform. The filter processing of the obtained filter coefficient is performed on each candidate waveform.

[0007]

Next, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows a functional configuration of an acoustic signal separating apparatus to which the method of the present invention is applied. In the following description, as an application example of this apparatus, a case where music performance is separated into performances for each musical instrument will be described as an example.

The acoustic signal separating apparatus 10 has an input terminal 1
The audio signal waveform of the mixed sound from 1 is input and output terminal 12
Output an acoustic signal waveform for each sound source. The input acoustic signal (waveform) is sampled at, for example, 48 kHz or 96 kHz, and is input as a time series of digital values of each sample. The rising component of the sound contained in the sound signal from the input terminal 11 is detected by the waveform classifying means 13,
The sound signal is temporally divided. This division may be made at regular intervals.

[0009] A waveform storage means 14 stores in advance a template of a sound source waveform targeted by the apparatus 10. The candidate waveform selection means 15 analyzes the basic sound features such as the fundamental frequency and the power envelope for the input sound signal waveform for each section, and refers to the result, and stores it in the waveform storage means 14. From the existing waveforms, a waveform that may be included in the input acoustic signal waveform is selected.

The coefficient determining means 16 determines the sum of the respective waveforms obtained by applying the filter operation to each of the selected waveforms and the coefficient of the filter operation that minimizes the root mean square error with the input acoustic signal waveform. Is done. The determined coefficients of the filter operation are set in the filter operation means 17, and the filter operation is performed on each of the waveforms selected by the candidate waveform selection means 15. The result of each filter operation is output to an output terminal 12 as an output separated for each sound source.
Is output to

Next, the above means 13, 15, 16, 1
Each process in 7 will be specifically described below. As shown in FIG. 2, the waveform classifying means 13 first reads an input audio signal (step 101), and detects a rising edge of a sound included in the input audio signal by paying attention to a power fluctuation or the like of the input audio signal (step 101). Step 102). Next, an input audio signal is output as a segmented audio signal from the last detected rise to the currently detected rise time (step 103).
Subsequently, it is checked whether or not the input audio signal is continuously input (step 104). If the input audio signal is continuously input, the processing after step 102 is repeated, and if the input is completed, the processing is ended.

As shown in FIG. 3, the candidate waveform selecting means 15 first reads the segmented input sound signals divided by the waveform dividing means 13 (step 201). Next, a frequency component is extracted from each of the divided input audio signals (step 20).
2) Extract sound features such as fundamental frequency and power envelope (step 203). This feature amount is used to select a stored waveform of a sound that may be included in the divided input sound signal. Since the stored waveform of the sound is stored in the waveform storage means 14 in advance, it is inspected sequentially (steps 204 to 208). First, it is checked whether there is an untested stored waveform (step 204), and if so, one untested stored waveform is selected (step 2).
05). Next, the fundamental frequency of the stored waveform is compared with the frequency of the frequency component extracted in step 202 to check whether the frequency falls within a certain range (step 2).
06). If it does not fall within a certain range, it is unlikely that the stored waveform is included in the divided input audio signal, and the process returns to step 204. The certain range is determined, for example, as follows. That is, if the fundamental frequencies of the stored waveforms are arranged in the order of their magnitudes, looking at a certain fundamental frequency, the frequency is lower by a half between the fundamental frequencies immediately below it, and is higher by a half between the fundamental frequency immediately above it. Those that fall within the frequency range are considered as candidates. For example, when a stored waveform is provided for each semitone, since the frequency of the semitone is increased by about 6%, those having a fundamental frequency within ± 3% are selected as candidates. If it falls within a certain range in step 206, it is further checked whether or not there is a contradiction in the characteristic amount (for example, a sound range that cannot be pronounced) (step 2).
07). If there is a contradiction, it is unlikely that the stored waveform is included in the segmented input sound signal, and the process returns to step 204. If there is no inconsistency, the stored waveform is likely to be included in the segmented input sound signal,
Add to the candidate waveform (step 208) and step 204
Return to In step 204, if there is no untested stored waveform, the candidate waveform found up to that point is output (step 209), and the process ends.

The coefficient determining means 16 first reads the segmented input sound signals divided by the waveform dividing means 13 (step 301). Next, the candidate waveform selected by the candidate waveform selecting means 15 is read (step 302). Then, in order to obtain a filter coefficient that minimizes the mean square error between the waveform obtained by applying the filter operation to each candidate waveform and the divided input sound signal, a simultaneous equation is formed. It is created (step 303). If you decide to use a FIR type as a filter, the waveform of the result of applying a filter operation to the candidate waveform _{y n (k) = Σ m} = 0 M-1 h n (m) r n (k-m) (1 ). Here, k is a sampled time, n is a suffix for counting candidate waveforms, y _n (k) is a value of a time k resulting from applying a filter operation, h is an impulse response of an FIR filter, r is a candidate waveform, M is the order of the filter. A waveform obtained by adding the waveform of the result of applying a filter operation to each candidate waveform, the mean square error between the divided input sound signal J = E _{[{z (k) -Σ n =} 0 N-1 y n (K)｝ ² ] (2) Here, z (k) is the value of the time k of the segmented input sound signal waveform, N is the number of candidate waveforms, and E is the time average. A necessary condition for minimizing this is that the partial differential ∂J / ∂h _n (m) becomes zero for all n and m. With this condition, N × M pieces of simultaneous linear equations ^{_{Σ n = 0 N-1 Σ}} m = 0 M-1 E _{[r i (k-l) r} j (k-m) ] h _n (m) = E _{[r i (k-m) z} (k) ] (3) can be derived. Equation (3) is created in step 303. Subsequently, equation (3) is solved (step 304). Since the number of unknowns is equal to the number of equations, equation (3) can be solved by finding the inverse of the coefficient matrix. The obtained coefficient is used in step 30.
Output at 5.

The filter calculating means 17 first reads the filter coefficients obtained by the coefficient determining means 10 (step 401) as shown in FIG.
The candidate waveform selected in step 5 is read (step 40).
2). Subsequently, the filter operation of Expression (1) is performed (in the case of an FIR type filter) (step 403), and the waveform of the operation result is output (step 404). This waveform is a signal waveform separated for each sound source.

As described above, in the present invention, candidates are selected from the stored waveforms, and the filter coefficients are determined so that the square error between the sum of the results obtained by filtering the candidate stored waveforms and the divided input sound signal is minimized. That is, a filter coefficient close to the characteristic of the segmented input sound signal is determined, and a filter characteristic that is not included in the segmented input sound signal in the selected candidate storage waveform does not pass even through the filter. Become. Further, when the candidate storage waveform is close to the sound source waveform present in the segmented input sound signal, the filter coefficient according to the deformation of the waveform between the candidate storage waveform and the sound source waveform is determined. Then, when the candidate storage waveform is filtered, the waveform deformation of the corresponding input sound waveform in the divided input sound signal is absorbed, and a large output is obtained.

For example, waveforms of the same time portion (100 ms to 130 ms from the rise) when the height F4 is played with almost the same strength on the piano manufactured by Company A and the piano manufactured by Company B are shown in FIGS. 6A and 6B.
As shown in the figure, the waveforms are similar as a whole, but different from each other. By processing the waveform of FIG. 6B with a 40th-order FIR filter, the waveform of FIG. 6C can be made to be considerably close to the waveform of FIG. 6A.

Therefore, when each candidate storage waveform is subjected to the filtering process by the filter operation means 17, the output average power of the same sound source waveform included in the divided input sound signal becomes large, and the average power becomes the average power of the sound source waveform. A value corresponding to the mixing ratio, the output of the candidate storage waveform not included in the divided input audio signal is zero, and the candidate storage waveform is slightly different from the corresponding sound source waveform in the divided input audio signal. However, this is adaptively corrected, and the filtered output is large.

In order for the processing by the coefficient determining means 16 to be effective, the fundamental frequency and phase of the candidate storage waveform r must match the fundamental frequency and phase of the sound source included in the segmented input sound signal z. Is desirable. This is because the filter of the coefficient determination means 16 cannot change the frequency of the signal. For this reason, it is preferable to perform a waveform synchronization process that momentarily matches the phase of the candidate storage waveform r with the phase of the corresponding sound source component in the divided input audio signal z. This waveform synchronization processing is performed, for example, as follows.

As shown in FIG. 7, a divided input acoustic waveform, that is, a reference waveform z is read (step 301), and frequency analysis is performed by a method such as a band filter bank (step 302). Next, since the power analysis on the time-frequency plane is obtained by the frequency analysis, a maximum point (local peak) of the power is found in the frequency direction (step 3).
03). Subsequently, the temporally continuous local peaks are connected to form a continuous local peak (step 30).
4). The connection of the local peaks in time is called a frequency component, and expresses various periodicities existing in the original waveform. This frequency component is output as periodicity information (step 305). The same processing is performed on the candidate storage waveform r to obtain the periodicity information.

Next, as shown in FIG. 8, substantially the same fundamental frequency is selected from among the periodicities existing in the divided input acoustic signal waveform z and the candidate storage waveform r, and the center frequency of the band-pass filter is set to that frequency. Set. An output waveform is obtained by applying the band-pass filter to the sectioned input acoustic waveform z (step 402), and a time series of the phase of the output waveform is stored (step 403). That is, since the output of the bandpass filter is close to a sine wave, the zero-crossing time is obtained from the sample values at times k and k + 1 before and after the sign inversion of the sine wave time series, and the cycle of the sine wave is obtained. A phase value (phase angle) at each time of the sequence is obtained. Subsequently, the same band-pass filter as used in step 402 is applied to the candidate storage waveform to obtain an output waveform (step 404), and the time series of the phase of the output waveform is stored (step 40).
5). Next, the time series difference between the two phases stored in step 403 and step 405 is obtained, and the time series of the phase difference is obtained and output (step 406).

The time series of the phase difference is converted into the time series of the time difference. This conversion is performed by equation (4). δt (k) = (1 / (2πf)) δp (k) (4) where k is the time, δt (k) is the time difference of the time difference k, f is the center frequency of the bandpass filter, δp (k)
Is the phase difference at time k in the phase difference time series.

The corresponding sample value of the candidate stored waveform time series r is delayed or advanced according to each time difference of the time difference time series. As a result, a candidate storage waveform that is instantaneously synchronized in phase with the fundamental frequency of the divided input sound signal z is obtained.

[0023]

Next, an experiment for evaluating recognition accuracy to which the present invention is applied will be described. As shown in FIG. 9, a test pattern is a pattern in which three single sounds sound simultaneously, and the pattern is class 2, that is, a single sound in which at least one set of simultaneously sounding single sounds has a fundamental frequency that is an integer multiple of 1.5. Pattern. In creating a pattern, a single note of a flute, piano, and violin natural musical instrument is recorded in a studio for each semitone in advance (16 bits,
48 kHz), the waveform was stored on a computer, and the waveform was randomly selected and added under the restrictions of class 2 and MIDI note numbers 60 to 74 to create a pattern.

The definition of the recognition rate R is based on the following equation (5). R = 100 · ｛((right-wrong) / total) · (1/2) +1/2｝ (5) The right is included in the output and the pitch of the note in which both the pitch and the tone are correctly recognized. The number, wrong, is the number of notes whose pitch and / or timbre are not correct among the notes included in the output, and total is the total number of notes included in the input (correct answer). From the results of the preliminary experiment, the order of the FIR filter was set to 40 under the condition of template filtering ON. Note that template filtering refers to filtering of equation (1), and template filtering OFF means that the order of the FIR filter is 1
It means that it was done.

In this experiment, using the same waveform as that used for generating the test pattern as the original template or using the same musical instrument as the original template is not appropriate as an evaluation experiment because of the high degree of matching of the waveforms. Therefore, the waveform of the template and the waveform of the test pattern used were recorded from different individuals. This is shown in FIG.

FIG. 11 shows the experimental results. In this table, the conditions (template filtering Off, phase synchronization Off) in the lower right column correspond to sound source identification using a simple matched filter. Therefore, it can be seen that the effectiveness of the processing using the adaptive template of the present invention is clearly shown in comparison with the matched filter. In particular, when phase synchronization is performed, the recognition rate is further increased.

In addition to the benchmark test, a music recognition test was performed on live music. Here, FIG.
For the test song "Firefly Light", which was performed using a violin, flute, and piano of another musical instrument,
The recognition rate R for the sound source identification processing was examined. FIG. 12 shows the result. The values in the figure are the recognition rates for only the sound source identification processing. The qualitative trend of the results is similar to the benchmark test, indicating the effectiveness of the method of the present invention.

As described above, according to the present invention, even if there are a large number of sound sources, and these sound sources are diverse and have fluctuations, the sound can be obtained with higher accuracy than a known method. There is an advantage that signal separation processing can be performed.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a functional configuration example of an acoustic signal separation device to which the method of the present invention is applied.

FIG. 2 is a flowchart showing a processing procedure of a waveform classification unit 13;

FIG. 3 is a flowchart showing a processing procedure of a candidate waveform selection unit 15;

FIG. 4 is a flowchart showing a processing procedure of a coefficient determining unit 16;

FIG. 5 is a flowchart showing a processing procedure of a filter operation means 17;

FIG. 6 is a diagram showing an example of a candidate waveform, a segmented input sound signal waveform, and a segmented input sound signal waveform after adaptive filter processing.

FIG. 7 is a flowchart showing a procedure for acquiring waveform synchronization information.

FIG. 8 is a flowchart showing a procedure for acquiring a phase difference time series.

FIG. 9 is a diagram showing an example of a single sound pattern used in a benchmark test.

FIG. 10 is a diagram showing a musical instrument used in the experiment.

FIG. 11 is a view showing a result of a benchmark test.

FIG. 12 is a diagram showing a result of a sound recognition test.

Claims (5)

[Claims] 1. A process of temporally dividing an input audio signal, and all waveforms possibly contained in each of the divided input audio signals are stored in a waveform storage means. Obtaining candidate waveforms by selecting from the stored waveforms, and obtaining filter coefficients of the filter operations so that an error between the sum of the results of the filter operations performed on the candidate waveforms and the divided input sound signal is minimized. And performing a filter operation with the obtained filter coefficient on each of the candidate waveforms. 2. The step of obtaining the candidate waveform includes the step of extracting a fundamental frequency of the divided input audio signal, and the step of storing a stored waveform having a fundamental frequency falling within a predetermined range with respect to the extracted fundamental frequency. 2. The method according to claim 1, further comprising the step of selecting from the means. 3. The sound according to claim 1, wherein the step of dividing is a step of detecting a rising edge of a sound included in the input audio signal and dividing the interval between adjacent detected rising edges. Signal separation method. 4. The method according to claim 1, wherein a fundamental frequency component of each of the candidate waveforms is phase-synchronized with a fundamental frequency component of the divided input audio signal, and the phase-synchronized candidate waveform is used in a process of obtaining the filter coefficient. Claims 1 to 3
The acoustic signal separation method according to any one of the above. 5. The phase synchronization acquires a time series of a phase difference between a fundamental frequency component of each candidate waveform and a fundamental frequency component of the divided input audio signal, and includes a polarity of each phase difference of the phase difference time series. The time series of the time movement amount converted to the time difference is obtained, and according to each time movement amount of this time movement time series,
The method according to claim 4, wherein the method is performed by moving a sample at a corresponding time of a corresponding candidate waveform.