JP4625933B2 - Sound analyzer and program - Google Patents

Description

  The present invention is directed to a musical sound signal including a singing voice and a plurality of types of instrument sounds recorded on a commercially available CD (compact disc) or the like, and a pitch of a melody sound or a bass sound (in this specification, a fundamental frequency). The present invention relates to a sound analysis apparatus and a program for estimating the

  It is very difficult to estimate the pitch of a specific sound source from a monaural sound signal in which the sounds of many sound sources are mixed. One of the essential reasons why it is difficult to estimate the pitch of a mixed sound is that, in the time-frequency domain, the frequency component of one sound overlaps with the frequency component of another sound that is playing simultaneously. For example, in typical popular music played on singing voices, keyboard instruments (piano, etc.), guitars, bass guitars, drums, etc., part of the harmonic structure of the singing voice that plays the melody (especially the fundamental frequency component) It frequently overlaps with the harmonic component of the guitar, the higher harmonic component of the bass guitar, the noise component included in the sound of the snare drum, and the like. For this reason, a method of locally tracking each frequency component does not function stably for complex mixed sounds. There is a technique for estimating a harmonic structure on the assumption that a fundamental frequency component exists, but such a technique has a major drawback that it cannot handle a missing fundamental phenomenon. Furthermore, if the frequency components of other sounds that are playing at the same time overlap with the fundamental frequency components, they will not function effectively.

  For the above reasons, there has been a technique for estimating the pitch of a single sound or an acoustic signal that contains a single sound with aperiodic noise, but it is recorded on a commercially available CD. There was no technique for estimating the pitch of a mixed sound signal such as an acoustic signal.

  However, in recent years, a technique for appropriately estimating the pitch of each sound included in a mixed sound has been proposed by using a statistical method. This is the technique of Patent Document 1.

  In the technique of this Patent Document 1, a frequency component belonging to a band considered to be a melody sound and a frequency component belonging to a band considered to be a bass sound are separately extracted from an input acoustic signal by a BPF, and each of those bands is extracted. Based on the frequency components, the fundamental frequencies of the melody sound and the bass sound are estimated.

  More specifically, in the technique of Patent Document 1, a sound model having a probability distribution corresponding to the harmonic structure of a sound is prepared, and each frequency component of the band of the melody sound and each frequency component of the band of the base sound are It is considered to be a mixed distribution obtained by weighting and adding each sound model corresponding to various fundamental frequencies. Then, the weight value of each sound model is estimated using an EM (Expectation-Maximization) algorithm.

  This EM algorithm is an iterative algorithm for performing maximum likelihood estimation on a probability model including hidden variables, and a local optimum solution can be obtained. Here, since the probability distribution having the largest weight value can be regarded as the most dominant harmonic structure at that time, the fundamental frequency in the dominant harmonic structure can be obtained as the pitch. . Since this method does not depend on the presence of the fundamental frequency component, the missing fundamental phenomenon can be appropriately handled, and the most dominant harmonic structure can be obtained without depending on the presence of the fundamental frequency component.

  Non-Patent Document 1 discloses a technique in which the following extension is made to the technique of Patent Document 1.

<Extension 1: Sound model multiplexing>
In the technique of Patent Document 1, only one sound model is prepared for the same fundamental frequency, but in reality, a sound having a different harmonic structure may be switched and appear at a certain fundamental frequency. Therefore, a plurality of sound models are prepared for the same fundamental frequency, and the input acoustic signal is modeled as a mixture distribution thereof.

<Extension 2: Estimation of sound model parameters>
In the technique of Patent Document 1, the ratio of the magnitude of each harmonic component is fixed in the sound model (assuming an ideal sound model). This does not necessarily match the harmonic structure in the real world mixed sound, leaving room for refinement to improve accuracy. Therefore, the ratio of harmonic components of the sound model is also estimated by the EM algorithm at each time in addition to the model parameters.

<Extension 3: Introduction of prior distribution for model parameters>
In the technique of Patent Document 1, prior knowledge about the weight of the sound model (probability density function of the fundamental frequency) has not been assumed. However, depending on the application of the fundamental frequency estimation technique, there may be a demand for obtaining a fundamental frequency with fewer false detections even if the fundamental frequency is in the vicinity. For example, for the purpose of performance analysis and vibrato analysis, the approximate basic frequency at each time is prepared as prior knowledge by singing and playing musical instruments while listening to the headphones, and more accurate basics in actual music There is a need to obtain a frequency. Therefore, the framework of the maximum likelihood estimation of the model parameter (weight value for the sound model) in Patent Document 1 is expanded, and maximum posterior probability estimation (MAP estimation; Maximum A Posteriori Probability Estimation) is performed based on the prior distribution regarding the model parameter. I made it. At that time, a prior distribution regarding the ratio of the magnitudes of the harmonic components of the sound model added to the model parameters in <Extension 2> was also introduced.
Japanese Patent No. 3413634 Masataka Goto: "Real-time music scene description system: Overall concept and extension of pitch estimation method", IPSJ SIG 2000-MUS-37-2, Vol.2000, No.94, pp.9 -16, October 16, 2000

  According to the technique disclosed in Non-Patent Document 1, the introduction of the “extension 1” allows each harmonic to be generated when the sound source can generate a plurality of sounds having different harmonic structures, for example. By preparing a plurality of sound models corresponding to the structure, it is expected that the estimation accuracy of the fundamental frequency of each sound is improved. However, if a large number of sound models are used in order to improve the estimation accuracy of the fundamental frequency, it takes a lot of man-hours to create such a large number of sound models, and a large number of sound models are stored. There is a problem that the storage capacity must be secured in the sound analyzer.

  The present invention has been made in view of the circumstances described above, and a sound analysis apparatus and program that can store a relatively small number of sound models and can estimate a fundamental frequency with high estimation accuracy. The purpose is to provide.

  The present invention provides a storage means for storing a plurality of types of sound models each defining a harmonic structure of a plurality of types of sounds generated from a musical instrument, and a plurality of types of sound models stored in the storage means for each fundamental frequency. Interpolation is performed according to the above, and interpolation processing based on the fundamental frequency is applied to the multiple types of sound models that are ordered, and multiple types of sound models corresponding to intermediate frequencies of each of the ordered sound models are generated. And a plurality of types of sound models stored in the storage unit and a plurality of types of sound models generated by the interpolation unit, and a plurality of sound models having various harmonic structures and fundamental frequencies are weighted and added. The weight distribution for each sound model is optimized so that the mixture distribution is composed, and this mixture distribution is the distribution of the frequency components of the input acoustic signal. A probability density function estimating means for estimating the fundamental frequency of the sound of the sound source that is the source of the input acoustic signal, and one or more sound sources in the input acoustic signal based on the probability density function of the fundamental frequency Provided is a sound analysis device comprising a fundamental frequency estimation means for estimating and outputting a fundamental frequency of sound, and a computer program for causing a computer to function as the sound analysis device.

  According to this invention, since the sound model is interpolated based on the fundamental frequency and the fundamental frequency is estimated using the sound model with the sound model supplemented, the number of sound models to be stored in the storage means Even when there is a relatively small amount, the fundamental frequency can be estimated with high accuracy.

  Embodiments of the present invention will be described below with reference to the drawings.

<A. First Embodiment>
<Overall configuration>
FIG. 1 is a diagram showing the processing contents of a sound analysis program according to the first embodiment of the present invention. This sound analysis program includes an acoustic signal such as a sound collection function for acquiring an acoustic signal from the natural world, a playback function for reproducing an acoustic signal of music from a recording medium such as a CD, or a communication function for acquiring an acoustic signal of music via a network. The program is installed and executed on a computer such as a personal computer having an acquisition function. The computer that executes the sound analysis program according to the present embodiment functions as the sound analysis device according to the present embodiment.

The sound analysis program according to the present embodiment estimates the pitch of a certain sound source in a monaural music sound signal acquired through the sound signal acquisition function. As the most important example, the melody line and the bass line are estimated here. The melody is a sequence of single notes that can be heard more prominently than the others, and the bass is the sequence of the lowest single note in the ensemble. The temporal changes are called the melody line Dm (t) and the base line Db (t), respectively. Assuming that the fundamental frequency F0 at time t is Fi (t) (i = m, b) and the amplitude is Ai (t), these are expressed as follows.

  As a means for obtaining the melody line Dm (t) and the base line Db (t) from the input sound signal, the sound analysis program includes an instantaneous frequency calculation 1, frequency component candidate extraction 2, frequency band restriction 3, Each of the melody line estimation 4a, the baseline estimation 4b, and the sound model interpolation process 5 is included. Each process of the melody line estimation 4a and the baseline estimation 4b includes a fundamental frequency probability density function estimation 41 and a fundamental frequency sequential tracking 42 using a multi-agent model. In this embodiment, the processing contents of instantaneous frequency calculation 1, frequency component candidate extraction 2, frequency band restriction 3, melody line estimation 4a, and baseline estimation 4b are described in Patent Document 1 and Non-Patent Document 1 described above. This is basically the same as that disclosed in. The feature of this embodiment is that a sound model interpolation process 5 is added. Hereinafter, the content of each process which comprises the sound analysis program by this embodiment is demonstrated.

<Instantaneous frequency calculation 1>
In this process, the input acoustic signal is applied to a filter bank consisting of a plurality of BPFs, and the instantaneous frequency (Flanagan, JL and Golden, RM: Phase Vocoder, The BellSystem)
Technical J., Vol. 45, pp.1493-1509 (1966)). Here, the above-described Flanagan method is used, the short-time Fourier transform (STFT) output is interpreted as the filter bank output, and the instantaneous frequency is efficiently calculated. When the STFT using the window function h (t) for the input acoustic signal x (t) is given by the equations (3) and (4), the instantaneous frequency λ (ω, t) can be obtained by the equation (5). .

  Here, h (t) is a window function that gives the localization of the time frequency (for example, a time window created by convolving a second-order cardinal B-spline function with a Gaussian function that gives the optimum localization of the time frequency. Such).

A wavelet transform may be used to calculate this instantaneous frequency. Here, the STFT is used to reduce the amount of calculation. However, if only a single STFT is used, the time resolution and frequency resolution in a certain frequency band are deteriorated. Therefore, multi-rate filter banks (Vetterli, M .: A Theory of Multirate Filter Banks, IEEE Trans. On
ASSP, Vol. ASSP-35, No. 3, pp. 356-372 (1987)), and obtain a reasonable time-frequency resolution under the restriction that it can be executed in real time.

<Frequency component candidate extraction 2>
In this process, candidate frequency components are extracted based on the mapping from the center frequency of the filter to its instantaneous frequency (Charpentier, FJ: Pitch detection using the short-term phase spectrum,
Proc. Of ICASSP 86, pp. 113-116 (1986)). Consider a mapping from the center frequency ω of an STFT filter to the instantaneous frequency λ (ω, t) of its output. Then, if there is a frequency component of frequency ψ, ψ is located at the fixed point of this mapping, and the value of the instantaneous frequency around it is almost constant. That is, the instantaneous frequency Ψ _f ^(t) of all frequency components can be extracted by the following equation.

これらの周波数成分のパワーは、Ψ_f ^(t)の各周波数におけるＳＴＦＴパワースペクトルの値として得られるため、周波数成分のパワー分布関数Ψ_p ^(t)(ω)を次のように定義することができる。

Since the power of these frequency components is obtained as the value of the STFT power spectrum at each frequency of ψ _f ^(t) , the power distribution function ψ _p ^(t) (ω) of the frequency component can be defined as follows. it can.

<Frequency band restriction 3>
In this process, the frequency band is limited by weighting the extracted frequency components. Here, two types of BPF are prepared for the melody line and the base line. The melody line BPF can pass most of the main fundamental wave components and harmonic components of a typical melody line, and cuts off a frequency band in which duplication near the fundamental frequency frequently occurs to some extent. On the other hand, the BPF for a bass line can pass many of the main fundamental frequency components and harmonic components of a typical bass line, and the frequency band in which the other performance parts are dominant over the bass line. To some extent.

平均律の半音は１００ｃｅｎｔに、１オクターブは１２００ｃｅｎｔに相当する。 In the present embodiment, the logarithmic scale frequency is expressed in units of cents (originally a scale representing pitch difference (pitch)), and the frequency fHz expressed in Hz is expressed as cents as follows: Convert to fcent.

A semitone of equal temperament corresponds to 100 cent, and one octave corresponds to 1200 cent.

When the frequency response of the BPF at the frequency x cent is BPFi (x) (i = m, b) and the power distribution function of the frequency component is ψ ′ _p ^(t) (x), the frequency component that has passed through the BPF is BPFi. (X) ψ ′ _p ^(t) (x). However, Ψ ′ _p ^(t) (x) is the same function as Ψ _p ^(t) (ω) except that the frequency axis is represented by cent. Here, as a preparation for the next stage, a probability density function p _Ψ ^(t) (x) of a frequency component that has passed through the BPF is defined.

Here, Pow ^(t) is the total power of the frequency components that have passed through the BPF as shown in the following equation.

<Estimation 41 of probability density function of fundamental frequency>
In the fundamental frequency probability density function estimation 41, a fundamental frequency probability density function representing how relatively each harmonic structure prevails with respect to a frequency component candidate that has passed through the BPF is obtained. The processing content of the probability frequency function estimation 41 of the fundamental frequency is the content disclosed in Non-Patent Document 1.

In the estimation 41 of the probability density function of the fundamental frequency, it is assumed that there are Mi types of sound models for the same fundamental frequency (i is for melody (i = M) or bass (i = b)), the fundamental frequency is F, the sound model type is the mth type, and the sound has model parameters μ ^(t) (F, m) The model p (x | F, m, μ ^(t) (F, m)) is defined as follows.

This sound model is obtained by modeling how many harmonic components appear at which frequency when the fundamental frequency is F. Hi represents the number of harmonic components including the fundamental frequency component, and W _i ² represents the variance of the Gaussian distribution G (x; x0, σ). c ^(t) (h | F, m) represents the magnitude of the h-order harmonic component of the m-th sound model whose fundamental frequency is F, and satisfies the following expression.

When the m-th sound model is used as a sound model of a fundamental frequency F and is used to estimate the probability density function of the fundamental frequency, the weight c ^(t) (h | F, m) in the sound model of the fundamental frequency F As shown in the above equation (16), a weight c ^(t) (h | F, m) that is predefined so that the sum is 1 is used.

ここで、ＦｈｉとＦｌｉは、許容される基本周波数の上限と下限であり、ｗ^（ｔ）（Ｆ，ｍ）は次式を満たすような音モデルの重みである。

In the estimation 41 of the probability density function of the fundamental frequency, the sound model as described above is used, and the probability density function p _Ψ ^(t) (x) of the frequency component is defined as p (x | F , M, μ ^(t) (F, m)) is considered to have been generated from the mixed distribution model p (x | θ ^(t) ).

Here, Fhi and Fli are the upper and lower limits of the allowable fundamental frequency, and w ^(t) (F, m) is the weight of the sound model that satisfies the following equation.

Since it is impossible to assume the number of sound sources in advance for a mixed sound in the real world, it is important to perform modeling in consideration of the possibility of all fundamental frequencies as shown in Equation (17). Finally, if the model parameter θ ^(t) can be estimated from the model p (x | θ ^(t) ) as if the observed probability density function p _Ψ ^(t) (x) was generated, its weight w ^{(t ) Since} (F, m) represents how relatively each harmonic structure is dominant, it can be interpreted as a probability density function p _F0 ^(t) (F) of the fundamental frequency as in the following equation.

Then, the product in order to achieve "expansion 3" described above, theta ^(t) prior distribution _{p 0i} of (theta ^(t)) of the formula (25) and (24) as in equation (23) Give in.

Here, for p _0i (w ^(t) ) and p _0i (μ ^(t) ), the most likely parameters are w _0i ^(t) (F, m) and μ _0i ^(t) (F, m). Sometimes it is a unimodal prior distribution that takes its maximum value. However, Z _w and Z _μ are normalization coefficients, and β _wi ^(t) and β _μi ^(t) (F, m) are parameters that determine how much prior distribution is emphasized, and when 0, No information prior distribution (uniform distribution). ^{_{Further, D w (w 0i (t}} ); w (t)), D μ (μ 0i (t) (F, m); μ (t) (F, m)) , such as: K-L This is the amount of information (Kullback-Leibler's information).

From the above, when the probability density function p _Ψ ^(t) (x) is observed, the parameter θ ^(t) of the model p (x | θ ^(t) ) is changed to the prior distribution p _0i (θ ^(t) ). It can be seen that the problem to be estimated based on the problem should be solved. The maximum posterior probability estimator (MAP estimator ⁾ of θ ^(t) based on this prior distribution can be obtained by maximizing the following equation.

Since this maximization problem is difficult to solve analytically, θ ^(t) is estimated using the aforementioned EM (Expectation-Maximization) algorithm. The EM algorithm performs maximum likelihood estimation from incomplete observation data (in this case, p _Ψ ^(t) (x)) by repeatedly applying an E step (expectation step) and an M step (maximization step) alternately. Iterative algorithm for In the present embodiment, by repeating the EM algorithm, the probability density function p _Ψ ^(t) (x) of the frequency component that has passed through the BPF is converted into a plurality of sound models p (x | F, In the case of a mixed distribution obtained by weighting and adding m, μ ^(t) (F, m)), the most likely weight parameter θ ^(t) (= {w ^(t) (F, m), μ ^(t) (F, m)}). Here, in each iteration of the EM algorithm, with respect to the parameter θ ^(t) (= {w ^(t) (F, m), μ ^(t) (F, m)), the old parameter estimation value θ _old ^(t) ( = {W _old ^(t) (F, m), μ _old ^(t) (F, m)}) and update the new (more likely) parameter estimate θ _new ^(t) (= {w _new ^{(t )} (F, m), μ _new ^(t) (F, m)}). As the initial value of θ _old ^(t) , the final estimated value at the previous time t−1 is used. A recurrence formula for obtaining a new parameter estimated value θ _new ^(t) from the old parameter estimated value θ _old ^(t) is as follows. Note that the process of deriving the recurrence formula is described in detail in Non-Patent Document 1, so please refer to that.

^{_{W ML (t) (F,}} m) in the formula (29) and (30) and ^{_{c ML (t) (h |}} F, m) ^{_{is, β wi (t) = 0}} , β μi (t) (F , M) = 0, which is an estimated value in the case of no information prior distribution, that is, maximum likelihood estimation, and is given by the following equation.

Through these iterative calculations, the probability density function p _F0 ^(t) (F) of the fundamental frequency in consideration of the prior distribution is obtained from w ^(t) (F, m) by the equation (22). Furthermore, the ratio c ^(t) (h | F, m) of the magnitude of each harmonic component of all sound models p (x | F, m, μ ^(t) (F, m)) is also obtained. 1 ”to“ Extended 3 ”are realized.

こうして得られた周波数を音高とする。これが、本実施形態において、基本周波数の確率密度関数に基づいて入力音響信号における１または複数の音源の音の基本周波数を推定して出力する基本周波数推定手段としての処理である。 In order to determine the most dominant fundamental frequency Fi (t), as shown in the following equation, the probability density function p _F0 ^(t) (F) of the fundamental frequency (from the equation (22), the equations (29) to (32) What is necessary is just to obtain | require the frequency which maximizes) obtained as the final estimated value which repeated calculation).

Let the frequency obtained in this way be the pitch. This is processing as basic frequency estimation means for estimating and outputting the fundamental frequency of the sound of one or more sound sources in the input acoustic signal based on the probability density function of the fundamental frequency in the present embodiment.

<Frequency tracking 42 of basic frequency by multi-agent model>
In the probability density function of the fundamental frequency, if multiple peaks corresponding to the fundamental frequency of the sound that is playing at the same time are antagonized, these peaks may be selected one after another as the maximum value of the probability density function. The result obtained simply may not be stable. Therefore, in the processing as the fundamental frequency estimation means in the present embodiment, in order to estimate the fundamental frequency from a global viewpoint, the trajectories of a plurality of peaks are tracked continuously in the time change of the probability density function of the fundamental frequency. Select the most dominant and stable fundamental frequency trajectory among them. In order to control such tracking process dynamically and flexibly, a multi-agent model is introduced.

  The multi-agent model is composed of one feature detector and a plurality of agents (see FIG. 2). The feature detector picks up the prominent peaks in the probability density function of the fundamental frequency. The agent basically follows the trajectory driven by those peaks. In other words, the multi-agent model is a general-purpose framework that temporally tracks features that stand out in the input. Specifically, the following processing is performed at each time.

(1) After the probability density function of the fundamental frequency is obtained, the feature detector detects a plurality of conspicuous peaks (peaks exceeding a threshold that dynamically changes according to the maximum peak). Then, for each conspicuous peak, the promising peak is evaluated in consideration of the total power Pow ^(t) of frequency components. This is realized by regarding the current time as a time several frames ahead and prefetching and tracking the peak trajectory up to that time.

(2) When there is an agent already generated, the prominent peak is exclusively assigned to an agent having a locus close to it while interacting with each other. If multiple agents are candidates for assignment, assign them to the agent with the highest reliability.

(3) If the most promising and conspicuous peak has not yet been assigned, a new agent that tracks that peak is generated.

(4) Each agent has a cumulative penalty and disappears when it exceeds a certain threshold.

(5) An agent that has not been assigned a conspicuous peak receives a certain penalty, and tries to find the next peak to be tracked directly from the probability density function of the fundamental frequency. If the peak is not found, a penalty is applied. Otherwise, the penalty is reset.

(6) Each agent self-evaluates the reliability based on the weighted sum of the degree of how promising and conspicuous the peak assigned at present is and the reliability at the previous time.

(7) The fundamental frequency Fi (t) at time t is determined based on an agent having high reliability and a large total power along the track of the peak being tracked. The amplitude Ai (t) is determined by extracting a harmonic component or the like of the fundamental frequency Fi (t) from Ψ _p ^(t) (ω).

<Improvement of this embodiment (sound model interpolation processing 5)>
In general, the spectrum shape of sound produced by an instrument varies depending on the pitch (fundamental frequency). Therefore, in order to improve the estimation accuracy of the fundamental frequency, sounds with various fundamental frequencies are collected from the instrument, and the probability density function of the fundamental frequency is calculated using many sound models created from these sounds. It is preferable to perform estimation 41. However, when such a large number of sound models are used for the estimation 41 of the probability density function of the fundamental frequency, it takes a great amount of man-hours to create the large number of sound models. There is a problem that a storage capacity for storage must be secured in the sound analyzer. Therefore, in the present embodiment, the following improvements are made. That is, the storage device of the sound analyzer stores only a relatively small number of representative sound models corresponding to various fundamental frequencies in association with the respective fundamental frequencies, and this storage device is used when the sound analysis program is executed. A large number of sound models are generated from a relatively small number of representative sound models stored in, and passed to the probability frequency function estimation 41 of the fundamental frequency.

  In the present embodiment, as means for generating a large number of sound models from a relatively small number of representative sound models stored in the storage device and transferring them to the estimation 41 of the probability density function of the fundamental frequency, FIG. The sound model interpolation process 5 shown is added to the sound analysis program. In this sound model interpolation process, a plurality of types of sound models stored in the storage device are ordered according to each fundamental frequency, and the plurality of types of ordered sound models are subjected to interpolation processing based on the fundamental frequency, This is a process of generating a plurality of types of sound models corresponding to the intermediate fundamental frequency of each sound model. The sound analysis program according to the present embodiment executes the sound model interpolation process 5 at the beginning of the execution thereof, a representative sound model stored in the storage device, and the sound model obtained by the sound model interpolation process 5 Are delivered to the probability frequency function estimation 41 of the fundamental frequency.

FIG. 3 shows a specific example of representative sound model selection and sound model interpolation processing 5. In this example, a representative fret is selected for every five frets of all guitar frets, a sound model of the guitar sound when each representative fret is pressed with a finger is created, and the sound analyzer stores the representative sound model. Store in the device. Then, the sound model interpolation process 5 generates a sound model of the guitar sound corresponding to the intermediate frets sandwiched between the representative frets. In the sound model interpolation process 5, the h-order overtone component (h = 1 to Hi) of the sound model corresponding to the intermediate fret is converted into the h-order overtone component (h = H =) of the sound model corresponding to the low-frequency side representative fret of the intermediate fret. 1 to Hi) and the h-order harmonic component (h = 1 to Hi) of the sound model corresponding to the high-side representative fret of the intermediate fret. Various types of sound model interpolation processing 5 can be considered. In a preferred embodiment, the fundamental frequency of the sound model corresponding to the low-frequency side representative fret is Fa, the h-order overtone component is c (h | Fa, ma), and the basic frequency of the sound model corresponding to the high-frequency side representative fret is Fb. , H (th) overtone component is c (h | Fb, mb), the fundamental frequency of the sound model corresponding to the intermediate frets is Fc, and the h th overtone component is c (h | Fc, mc), The h-order overtone component c (h | Fc, mc) of the sound model corresponding to the intermediate fret is obtained by primary interpolation.

  In the estimation 41 of the probability density function of the fundamental frequency, both the sound model corresponding to the intermediate fret obtained in this way and the representative sound model corresponding to the representative fret originally stored in the storage device are used.

It is preferable to select a fundamental frequency for creating a representative sound model by paying attention to the characteristics of the overtone characteristics determined by the structure of the musical instrument. Specifically, in the fundamental frequency region where the overtone structure changes rapidly, it is effective to create a representative sound model closely and store it in the storage device.
For example, in a guitar, the harmonic structure changes continuously up to the highest fret of a certain string, but to produce a sound that is higher by a semitone than that, you must play another string that can produce a higher tone, The structure becomes discontinuous. Furthermore, the harmonic structure changes gradually near the open strings and low frets of the strings, and the harmonic structure changes greatly even with a difference of one fret near the high frets. It is effective to select a fundamental frequency that creates a representative sound model more densely as the frequency becomes higher. On the piano, the lowest range is one string, the lower range is two strings, the higher range is three strings, the highest range is three strings, but unlike the others, there is no mute mechanism, etc. The string structure differs depending on the frequency band, and in response to this, the tone color, that is, the overtone structure, also changes discontinuously at a specific frequency. If representative sound models are densely arranged at such discontinuous points, it is possible to improve the estimation accuracy of the fundamental frequency even with a small number of sound models.

  According to the present embodiment described above, while reducing the amount of data of the sound model to be stored, the sound source characteristics that differ for each sound range are expressed in more detail as the sound model, and further, the actual input sound is obtained by adjusting a small number of parameters. The shape of the sound model can be adjusted optimally.

<B. Second Embodiment>
In the first embodiment, considering that the spectrum shape of the sound differs depending on the fundamental frequency, a sound model corresponding to more types of fundamental frequencies is generated by interpolation processing from relatively few representative sound models. In the present embodiment, in the estimation 41 of the probability density function of the fundamental frequency, the sound model originally has for each type of each sound model (representative sound model and sound model obtained by the sound model interpolation process 5). The range of the fundamental frequency is set according to the fundamental frequency that has been set, the weight value for the sound model at a frequency outside the set range of the fundamental frequency is limited, and the weight value for each sound model is optimized. Further details are as follows.

First, regarding the estimation of the probability density function of the fundamental frequency by the EM algorithm, the application range of the fundamental frequency is determined for each type of sound model. Various methods are conceivable for determining the lower limit Flm and the upper limit Fhm of the application range of the fundamental frequency. For example, if the m-th type sound model p (x | F, m, μ ^(t) (F, m)) is a sound model of a guitar sound at a fret position, it is adjacent to the fundamental frequency at that fret position. The intermediate frequency between the fundamental frequency at the lower fret position and the fundamental frequency at the adjacent fret position may be Fhm. Alternatively, a wider range may be set between Flm and Fhm, and the applicable range of the fundamental frequency may be overlapped between adjacent sound models of the fundamental frequency.

In this embodiment, in the course of the EM algorithm, the m-th type sound model p (x | F, m, μ ^(t) (F, m)) That is, it is not used for estimating the probability density of the fundamental frequency such that F <Flm or F> Fhm. Specifically, the m-th type of sound model p (x | F, m, μ ^(t) (F, m)) has many types of fundamental frequencies F in the EM algorithm if no measures are taken. However, when repeating the recurrence formulas (29) to (32), the sound models p (x | F, m, μ ^(t) ( For the sound model corresponding to the fundamental frequency F such that F <Flm or F> Fhm among F, m)), the initial value of the weight value w (F, m) for them is set to zero.

In this way, the m-th type sound model p (x | F, m, μ ^(t) (F, m)) has a probability of the fundamental frequency F such that F <Flm or F> Fhm. It will no longer be used for density estimation. Such processing is performed for all types of sound models.

  According to this aspect, in accordance with the range of the fundamental frequency of each sound that can be generated from the sound source, the applicable fundamental frequency range (Flm, Fhm) of the sound model corresponding to the sound is defined individually. Thus, an appropriate fundamental frequency can be estimated in consideration of the fundamental frequency range of each sound.

<Other embodiments>
Although one embodiment of the present invention has been described above, the present invention may have other embodiments. For example:

(1) The sound model interpolation processing 5 is not limited to linear interpolation, and widely known interpolation methods such as zero-order interpolation and spline interpolation can be widely used. Furthermore, since a typical sound model may include an error, a sound model between the representative sound models may be obtained by using autoregression.

(2) The use of many types of sound models with different shapes contributes to the improvement of the estimation accuracy of the fundamental frequency, but the improvement of the estimation accuracy of the fundamental frequency is not improved even if many sound models having the same shape are used. I can't expect it. Therefore, in the sound model interpolation process 5, interpolation may be performed not only for all of the representative sound models but only for a partial section in which the shape of the sound model changes to some extent.

(3) In the sound model interpolation processing 5, the sound model may be interpolated by an interpolation calculation method that differs depending on the frequency domain. For example, a sound model is obtained by primary interpolation in a frequency region where the change in the shape of the sound model with respect to the change in the fundamental frequency is obtained. A mode of obtaining a sound model by interpolation is conceivable. Alternatively, instead of changing the interpolation calculation method itself, parameters for interpolation calculation may be changed according to the frequency domain.

(4) In each of the above embodiments, the final fundamental frequency is determined by causing the multi-agent to track the fundamental frequency obtained by the fundamental frequency probability density function estimation 41. However, the fundamental frequency probability density function is estimated. If the probability of erroneous estimation is low at 41 and a highly reliable estimation result is obtained, tracking by a multi-agent may be omitted.

(5) In each of the above embodiments, in addition to “extension 1” (sound model multiplexing), “extension 2” (estimation of sound model parameters) is introduced into the sound analyzer. In the estimation of the probability density function of the fundamental frequency without introducing, for example, only the recurrence formula (29) out of the recurrence formulas (29) and (30) is sequentially calculated, and the weight w (F, m) of the sound model is calculated. Only updating may be performed.

(6) In each of the above embodiments, “Extended 3” (introduction of a prior distribution relating to model parameters) is introduced into the sound analyzer, but the sound analyzer may be configured not to introduce this.

It is a figure which shows the processing content of the sound analysis program which is 1st Embodiment of this invention. It is a figure which shows time-dependent tracking of the fundamental frequency by the multi agent model comprised by one feature detector and a some agent. It is a figure which shows the selection example of the representative sound model in the same embodiment, and the content of the sound model interpolation process.

Explanation of symbols

1 ...... Calculation of instantaneous frequency, 2 ... Calculation of frequency component candidates, 3 ... Limitation of frequency band, 4a ... Estimation of melody line, 4b ... Estimation of baseline, 41 ... Probability density of fundamental frequency Function estimation, 42 …… Tracking of fundamental frequency by multi-agent model, 5 …… Sound model interpolation processing.

Claims (5)

A storage means for storing a plurality of types of sound models each defining a harmonic structure of a plurality of types of sounds generated from a musical instrument , each storing a plurality of types of sound models corresponding to a fundamental frequency unique to each sound ;
A plurality of types of sound models stored in the storage means are ordered according to each fundamental frequency, and the plurality of types of ordered sound models are subjected to interpolation processing based on the fundamental frequency, and each of the ordered sound models Interpolation means for generating a plurality of types of sound models corresponding to intermediate frequencies of
Using a plurality of types of sound models stored in the storage unit and a plurality of types of sound models generated by the interpolation unit, a mixed distribution obtained by weight-adding a plurality of sound models having various harmonic structures and fundamental frequencies is obtained. And a weight value for each sound model is optimized so that the mixed distribution is a distribution of frequency components of the input acoustic signal, and the optimized weight value of each sound model is a sound source that is the source of the input acoustic signal. A probability density function estimating means for estimating the fundamental frequency of the sound as a probability density function;
And a fundamental frequency estimating means for estimating and outputting a fundamental frequency of sound of one or a plurality of sound sources in the input acoustic signal based on a probability density function of the fundamental frequency. The probability density estimation unit for each type of tone models, in accordance with the fundamental frequency of the tone models, set the range of the fundamental frequency, weight values for the sound model in the range of frequencies of the fundamental frequency to the set The sound analysis apparatus according to claim 1, wherein the weight value for each of the sound models is optimized by limiting the sound value.   2. The sound according to claim 1, wherein the fundamental frequency estimation means detects a plurality of peaks in the probability density function, and outputs a fundamental frequency having high reliability and high power based on reliability of each peak. Analysis equipment. The sound analysis apparatus according to any one of claims 1 to 3, wherein the interpolation means executes the interpolation processing by an interpolation calculation method that differs depending on a frequency domain. Computer
A storage means for storing a plurality of types of sound models each defining a harmonic structure of a plurality of types of sounds generated from a musical instrument, each storing a plurality of types of sound models corresponding to a fundamental frequency unique to each sound;
A plurality of types of sound models stored in the storage means are ordered according to each fundamental frequency, and the plurality of types of ordered sound models are subjected to interpolation processing based on the fundamental frequency, and each of the ordered sound models Interpolation means for generating a plurality of types of sound models corresponding to intermediate frequencies of
Using a plurality of types of sound models stored in the storage unit and a plurality of types of sound models generated by the interpolation unit, a mixed distribution obtained by weight-adding a plurality of sound models having various harmonic structures and fundamental frequencies is obtained. And a weight value for each sound model is optimized so that the mixed distribution is a distribution of frequency components of the input acoustic signal, and the optimized weight value of each sound model is a sound source that is the source of the input acoustic signal. A probability density function estimating means for estimating the fundamental frequency of the sound as a probability density function;
Fundamental frequency estimation means for estimating and outputting a fundamental frequency of sound of one or more sound sources in the input acoustic signal based on a probability density function of the fundamental frequency;
A computer program characterized by functioning as a computer program.

Images