JP5405130B2 - Sound reproducing apparatus and sound reproducing method - Google Patents

Description

  The present invention relates to a sound reproduction device and a sound reproduction method for performing sound reproduction control of a plurality of speakers under a noisy environment such as a passenger compartment.

  Hitherto, techniques for reproducing sound with a high sense of presence using a plurality of speakers have been widely studied. For example, 5.1ch surround playback is a typical technology. In addition, auto volume control technology that monitors the amount of noise and controls the playback volume accordingly, so that sound can be played at a sufficient volume even in noisy environments such as the interior of a car. Have been studied (see, for example, Patent Document 1).

  Furthermore, a technique for highly accurately suppressing sounds other than a desired target sound by using a plurality of channels of digital filter processing using a plurality of microphones has been studied (for example, see Non-Patent Document 1). This technique is intended to extract only sound in a desired direction without causing distortion by digital filter processing of a plurality of channels. By applying this sound source separation technique, it is possible to monitor the amount of noise with high accuracy.

JP-A-4-235600

Masato Togami, Akio Amano “Speech Recognition Technology Under Noise with Human Symbiotic Robot EMIEW”, Measurement and Control, Vol.46, No.6, June 2007

  However, in the conventional auto volume control technology (described in Patent Document 1), when the direction of the noise source viewed from the user listening position and the sound image localization direction when the reproduced sound is reproduced by the speaker array overlap, it is difficult to hear the sound. There was a problem of becoming. In other words, it is considered that the human auditory process incorporates a function for listening to sounds for each direction of arrival based on information on arrival time differences between both ears and amplitude differences. However, if the sound directions overlap, it is considered that such a function makes it impossible to distinguish the speaker playback sound.

  The present invention has been made in view of such problems, and it is an object of the present invention to provide a sound reproducing device and a sound reproducing method capable of listening to a desired sound with an easy-to-hear sound even in an environment where noise exists. To do.

In order to solve the above-described problem, the sound reproducing device according to the present invention estimates the relative sound source direction viewed from the microphone array or the like by, for example, sound source separation processing using a plurality of microphone arrays, and the estimated sound source direction. Has a sound source direction conversion processing unit for converting the sound source direction from the user position. The sound reproducing device includes a sound source direction conversion processing unit that converts the sound source direction at the estimated position of the microphone array into the sound source direction at the user listening position. Furthermore, the sound reproduction device calculates the sound source direction at the user listening position of a noise source other than the reproduced sound source based on the sound source direction converted by the sound source direction conversion processing unit and the like, and at the user listening position of the noise source. It has an output coefficient setting unit for controlling the sound image localization direction so that the sound source direction and the sound image localization direction of the speaker array as a reproduction sound source are different.

  According to the present invention, it is possible to listen to a desired sound with a sound that is easy to hear even in an environment where noise exists.

It is explanatory drawing which shows the example of application of the sound reproduction apparatus of 1st Embodiment by this invention. It is a hardware block diagram which shows the sound reproduction apparatus of 1st Embodiment by this invention. It is a block diagram which shows the program structure of 1st Embodiment by this invention. It is explanatory drawing which shows the geometric image of a sound source position conversion process. It is the block diagram which showed the structure which superimposes the output coefficient set in this embodiment on an output source, and outputs it from a speaker. It is a block diagram which shows the detailed structure of a sound source position conversion part. It is a block diagram which shows the 1st example of the output coefficient determination part of FIG. 3 in detail. It is an example of the histogram produced | generated by 1st Embodiment of this invention. It is explanatory drawing which shows an example of the localization direction seen from the sound source position and user position of the estimated noise, and the user position of the speaker synthetic | combination wavefront. It is a block diagram which shows the 2nd example of the output coefficient determination part of FIG. 3 in detail. It is a block diagram which shows the direction matrix calculation part of a modification. It is a block diagram which shows the sound source separation part of FIG. 3 in detail. It is a flowchart which shows the adaptive process of a sound source separation filter. It is a block diagram which shows the sound source position estimation part of FIG. 3 in detail. It is a block diagram which shows the acoustic echo canceller of FIG. 3 in detail. It is explanatory drawing which shows the relationship between the software block of 1st Embodiment by this invention, and hardware. It is a block diagram which shows the structure which controls the output method of audio output sounds, such as music, in this embodiment. It is a flowchart which shows the process which determines an output coefficient determination timing. 5 is a timing chart showing an example of output coefficient setting timing and audio source reproduction timing. It is a hardware block diagram which shows the sound reproduction apparatus of 2nd Embodiment by this invention. It is a block diagram which shows the software structure of the sound field reproduction system which reproduces the sound field in a virtual sound source position using the sound source position conversion process in the user listening position by this invention.

  Hereinafter, the best mode for carrying out the present invention (hereinafter referred to as “embodiment”) will be described in detail with reference to the accompanying drawings.

FIG. 1 is an explanatory diagram showing an application example of the sound reproducing device 1 according to the first embodiment of the present invention.
The outline of the sound reproducing device 1 will be described with reference to FIG. A microphone array 101 having a plurality of microphones 102 is installed in a vehicle compartment 11 such as an automobile 10. The direction of noise arrival is estimated from the sound recorded by the microphone array 101. Then, a speaker output coefficient is set for each speaker 112 so that the sound reproduced by the speaker 112 is localized in a direction different from the noise arrival direction at the user listening position. With such a configuration, the user can listen to the speaker playback sound with a sound that is easy to hear.

FIG. 2 is a hardware configuration diagram showing the sound reproducing device 1 according to the first embodiment of the present invention.
The microphone array 101 records sound in the passenger compartment 11 and outputs an analog signal indicating the recorded sound.
The multi-channel A / D converter 202 converts this analog signal into a digital signal for each microphone 102.
The speaker array 111 radiates desired reproduction sound into the passenger compartment 11.

The central processing unit 203 performs digital signal processing on the converted digital signal. Specifically, a noise component included in the digital signal is extracted, and the noise arrival direction is estimated. Then, the speaker output coefficient is controlled from the noise arrival direction. The signal processing program is stored in the non-volatile memory 205, loaded into the volatile memory 204 at the time of execution, and expanded. Also, a memory area necessary for program execution such as work memory is secured in the volatile memory 204. Information such as the arrangement of the microphone 102 is stored in the nonvolatile memory 205.
The central processing unit 203 controls the speaker output coefficient and outputs the generated digital signal (speaker output signal).

The multi-channel D / A converter 206 converts the speaker output signal into an analog signal and outputs it to each of the plurality of speakers 112 of the speaker array 111.
The speaker 112 is sounded by this analog signal and radiates sound into the air.

  Further, the presence / absence of the passenger / passenger is detected by the seat sensor 208, and the speaker output is regarded as the sound source position or the user listening position regardless of whether the passenger / passenger speaks or not. You may make it the structure which controls a coefficient. Specifically, the occupant / passenger may be regarded as a noise source, and control may be performed so that the speaker output sound is localized in a direction different from the occupant direction / passenger direction. Accordingly, the speaker output sound coefficient may be controlled so that the localization direction of the speaker output sound at the passenger position / passenger position differs from the noise arrival direction. By adopting the latter configuration, it is possible to form a sound field in which a desired sound can be easily heard not only by the driver but also by other passengers / passengers.

FIG. 3 is a block diagram showing the program configuration of the first embodiment according to the present invention.
The waveform capturing unit 301 controls the multi-channel A / D converter 202 (see FIG. 2) and acquires a digital signal.
The acoustic echo canceller 307 removes a component (acoustic echo component) caused by the speaker output included in the acquired digital signal. A specific configuration of the acoustic echo canceller 307 will be described later. The acoustic echo canceller 307 operates for each microphone element. The multi-channel signal after acoustic echo cancellation is sent to the sound source separation unit 302.

  Usually, a large number of sound sources exist in the passenger compartment 11. The sound source separation unit 302 separates the large number of sound sources into signals for each sound source. The sound source is separated every time the output signal of the acoustic echo canceller 307 is acquired for a certain time. The separated signals are sent to the sound source position estimation unit 303 for each sound source, and the respective sound source positions are estimated. The estimated sound source position is a relative position between the position of the microphone array 101 and the sound source position. In the present embodiment, since the relative position from the user listening position to the sound source is necessary, the sound source position conversion unit 304 calculates the sound source position viewed from the user listening position (user position) from the prior user listening position information. .

FIG. 4 is an explanatory diagram showing a geometric image of the sound source position conversion process.
Specifically, as shown in FIG. 4, the user position vector V1 that can be obtained from the user position and the position of the microphone array 101 (see FIG. 1) is added to the estimated sound source position vector V2 of the sound source viewed from the microphone array 101. By combining them, it is possible to obtain the converted sound source position vector V3 viewed from the user position. The installation position of the microphone array 101 is a fixed position. In this case, the user position vector V1 is determined if the user position is known. The user position may be preset to “driver's seat 12” or may be determined from information on the passenger position / passenger position detected by the seat sensor 208 (see FIG. 2).

Returning to FIG. 3, the histogram update unit 305 generates a histogram P (θ) of the noise arrival direction from the converted sound source position information. Here, θ is a sound source azimuth angle. The histogram may be a two-dimensional histogram of azimuth angle θ and elevation angle φ in the form of P (θ, φ). Here, the sound source direction of the i-th separated signal is defined as an azimuth angle θ _i and an elevation angle φ _i . Each time there is an incoming noise, the value 1 is added to the histogram P (θ _i , φ _i ) corresponding to the incoming noise θ _i , φ _i . Further, the average power of the i-th separated signal or a function of power may be added to P (θ _i , φ _i ). The histogram may be initialized every time processing is performed by the sound source separation unit 302 (that is, P (θ, φ) = 0 (for all θ and φ)), or sound source separation may be performed. Each time it is done, the past information is slowly forgotten by multiplying it with a forgetting factor α such as P (θ, φ) ← αP (θ, φ) (α is a constant between 0 and 1). Good.

  The output coefficient determination unit 306 determines a speaker output coefficient from the information of the obtained histogram P (θ, φ). The greater the value of the histogram, the greater the noise. The speaker output coefficient is controlled so that the speaker output sound is localized in the direction in which the difference in direction from the direction in which the value of the obtained histogram P (θ, φ) is large. That is, the direction in which the value of the histogram P (θ, φ) is large is a direction in which noise is heard frequently and the noise can be regarded as large. In this embodiment, avoiding the direction in which the noise can be regarded as large, The speaker output sound is localized so that a desired sound can be heard from a direction different from the direction in which noise is heard (typically, the opposite direction).

FIG. 5 is a block diagram showing a configuration in which the output coefficient set in the present embodiment is superimposed on the output source and output from the speaker 112.
The output coefficient storage unit 401 set in the present embodiment is secured on the nonvolatile memory 205 or the volatile memory 204. The output source acquisition unit 403 acquires original signals such as audio and output sound of a hands-free call. And the speaker output part 402 superimposes the output coefficient stored in the output coefficient memory | storage part 401 for every speaker 112 to output, and outputs it. The output coefficient may be a simple volume value, or after being converted into the time-frequency domain by an FIR (Finite Impulse Response) filter or short-time Fourier transform, an output coefficient is set for each frequency, and the output is returned to the time-frequency domain. You may take such a structure.

FIG. 6 is a block diagram showing a detailed configuration of the sound source position conversion unit 304 in FIG.
Relative sound source position P = (x, y, z) ^T as seen from the position of the microphone array 101 of each sound source estimated by the sound source position estimation unit 303 (the superscript T indicates the transposition of a vector / matrix) Is input to the sound source position conversion unit 304.
It is assumed that the microphone position database 504 describes a spatial position p ₂ (x ₂ , y ₂ , z ₂ ) ^T in the passenger compartment 11 of the microphone array 101. The user position extraction unit 502 acquires a spatial position p _u = (x _u , _yu , z _u ) ^T of the user listening position in the passenger compartment 11. The user listening position may be determined from the position of the passenger / passenger detected by the seat sensor 208 or the like, or may be preset by fixing the user listening position to the driver's seat 12 (see FIG. 1) in advance. . The transform vector generator 503 calculates the difference b = p ₂ -p _u spatial position p ₂ of the user listening positions p _u and the microphone array 101. The conversion vector addition unit 505 obtains P ′ = P + b by adding b to the sound source position P at the estimated microphone array position. P ′ is a relative position of the sound source viewed from the user listening position.

  In this way (see FIG. 4), the sound source position at the user listening position can be known by simple vector calculation. The sound source position conversion process may be performed once for each sound source separated and output by the sound source separation unit 302 every time the sound source separation process is executed, or the sound source separation unit 302 may be performed for each sound source or frequency. When separating sounds, the sound source position conversion process may be performed once for each sound source or frequency each time the sound source separation process is performed once.

FIG. 7 is a block diagram showing in detail a first example of the output coefficient determination unit 306 in FIG.
The output coefficient determination unit 306 extracts the relative azimuth angle θ or elevation angle value φ of the sound source from the sound source position for each sound source and for each frequency converted by the sound source position conversion unit 304 (see FIG. 3). This can be estimated by regarding the sound source position (x, y, z) as polar coordinates (r cos θ cos φ, r sin θ cos φ, sin φ). Usually, even if it is assumed that all the sound sources are present on the same horizontal plane in the passenger compartment 11, it may be considered that there is no practical problem, so φ = 0 may be set.

  The direction matrix calculation unit 602 uses a histogram P (θ) or a histogram P (θ indicating the frequency for each sound source direction from the extracted sound source directions (θ) or (θ, φ) for each frequency in the manner described above. , φ).

FIG. 8 is an example of the histogram P (θ) generated in the first embodiment of the present invention.
For each sound source direction θ, the frequency P of the sound source is obtained on the histogram P (θ).

Here, a steering vector used in the following description is defined. A vector a _p (f) whose element is a phase delay amount until the sound of the frequency f existing at the sound source position p reaches each microphone 102 is defined by the following equation (1).

  Here, j represents an imaginary unit. Here, M is the number of microphones 102.

If the human ears are regarded as two microphones 102, the amount of phase delay until the sound reaches the ears from the sound source position p can also be expressed as a _p (f). In this embodiment, since the delay amount common between the microphones 102 of a _p (f) has no particular meaning, a _p (f) does not necessarily need to be defined as the delay amount from the sound source position. It may be defined as a delay amount from the reference microphone 102. In the present embodiment, the delay amount T _{p, m} (f) is defined by the following equation (2) using the first microphone 102 as a reference microphone 102. t _m (p) is the time until the sound at the sound source position p reaches the m-th microphone 102.

Assuming that the microphones 102 are aligned in a straight line like both human ears, and assuming that the sound source position p exists at a distance sufficiently far from the microphone interval, T _{p, m} (f) is given by It can be approximated by (3).

Here, d _m is the distance between the m-th microphone 102 and the first microphone 102. Since c is the speed of sound and is about 340 [m / s] at room temperature, it is usually set to this value. θ is an angle formed by a straight line connecting the microphone array 101 and the sound source position p with respect to a plane orthogonal to the straight line constituting the microphone array 101. This is a relative azimuth angle viewed from the position of the microphone array 101. When the microphone array 101 is other than the linear arrangement, T _{p, m} (f) has a more complicated shape. However, if the geometric arrangement of the microphone array 101 is known anyway, it is obtained by simple geometric calculation. be able to. In the present embodiment, it is assumed that the geometric arrangement of the microphone array 101 is stored in advance in the nonvolatile memory 205 (see FIG. 2), and the steering vector is generated using the information.

Returning to FIG. 7, the direction matrix calculation unit 602 estimates the noise covariance matrix at the position of the microphone array 101 defined by the following equation (4) using the histogram P (θ) or the histogram P (θ, φ). The value R _n (f) is calculated.

Here, J is the number of histogram divisions. J is each divided lattice, θ _j is the azimuth angle of the lattice j, and φ _j is the elevation angle of the lattice j. n _j (f) is a steering vector viewed from the user listening position when it is assumed that a sound source exists at the position of the grid j. That is, R _n (f) is a matrix that is set so as to increase the influence of the histogram having a high frequency.

  Further, H (f) is defined by the following equation (5).

Here, L is the number of speaker elements. h _i (f) is a steering vector viewed from the user listening position when it is assumed that there is a sound source at the i-th speaker position.

Using H (f) and R _n (f), a matrix A (f) is obtained by the following equation (6).

When there are a plurality of user listening positions, a matrix H _i (f) composed of a noise covariance matrix R _{i, n} (f) and a steering vector of the speaker 112 for each user listening position as shown in the following equation (7). A (f) may be configured using

  By configuring A (f) in this way, it is possible to obtain speaker output sound that is easy to hear at a plurality of listening positions.

  The eigenvalue / vector calculation unit 603 obtains an eigenvector S (f) that gives the minimum eigenvalue of A (f). S (f) has as many elements as the number of speaker elements. Here, when each element of S (f) is superimposed on the transfer function of each speaker element and sound is radiated simultaneously from all the speakers 112, the steering vector of the synthesized wavefront of the radiated sound is H (f) S (f ). H (f) S (f) is a steering vector having a maximum difference from the noise steering vector.

The minimum cost coefficient calculation unit 604 obtains the speaker output coefficient S (t) in the time domain by performing inverse Fourier transform on the obtained speaker output coefficient S (f) for each frequency for each speaker (element) 112. Since S (t) can be regarded as an FIR filter, by convolving S (t) with the time-domain sound output from the speaker 112, a combined wavefront in which the steering vector for each frequency becomes H (f) S (f) is obtained. be able to. Further, in order to obtain S (f), after obtaining the sound source position p _min where the difference from the noise steering vector defined by the following equation (8) is maximized, S (f) is obtained by the following equation (9). f) may be obtained.

The resultant wavefront steering vector H (f) S (f) thus obtained completely coincides with the steering vector at the defined sound source position p _min and the output coefficient S (f) is minimized. It becomes.

FIG. 9 is an explanatory diagram illustrating an example of the estimated noise source position, the user position, and the localization direction of the speaker composite wavefront viewed from the user position.
In the present invention, it is possible to set the direction of the combined wavefront in such a way that the difference from the noise source position is large.

FIG. 10 is a block diagram illustrating in detail a second example of the output coefficient determination unit 306.
The output coefficient determination unit 306 has a configuration that selects a speaker (element) 112 having the largest difference in steering vector from noise from a plurality of speakers 112.

The direction matrix calculation unit 702 of the second example calculates R _n (f) in the same manner as the direction matrix calculation unit 602 of the first example shown in FIG.
The speaker inner product calculation unit 703 calculates the inner product of the steering vector of each speaker (element) 112 defined by the following equation (10) and R _n (f).

  The minimum cost coefficient calculation unit 704 selects the speaker (element) 112 having the smallest inner product value according to the following equation (11).

FIG. 11 is a block diagram illustrating a directional matrix calculation unit 801 according to a modification.
The directional matrix calculation unit 801 has a configuration in which a passenger position estimation unit 802 and a known noise position 803 are added to the directional matrix calculation unit 702 shown in FIG. Therefore, the directional matrix calculation unit 801 does not generate the noise covariance matrix R _n (f) only from the sound source information detected by sound source separation, but the occupant / passenger information from the passenger position estimation unit 802. R _n (f) is also generated using a known noise position 803 that represents information of a known sound source such as a wiper sound and an engine sound.

The passenger position estimation unit 802 detects the position where the occupant / passenger is sitting from the information of the seat sensor 208 (see FIG. 2), regards the position as a virtual noise source position, and determines the noise direction histogram P. Add to (θ). The frequency value to be added is a predetermined value. The known noise position 803 reads out preset sound source positions such as wiper sounds and engine sounds, and adds them to the noise direction histogram P (θ). In the direction matrix calculation unit 801, the noise covariance matrix is obtained from the noise source histogram and the noise direction histogram P (θ) generated from the information of the occupant position / passenger position and the known noise position after the conversion to the user listening position. R _n (f) is generated and R _n (f) is output.

FIG. 12 is a block diagram showing in detail the sound source separation unit 302 of FIG.
The digital sound pressure waveform received by each microphone (element) 102 is sent to the buffering unit 901.
The buffering unit 901 stores data for several seconds for each microphone (element) 102, for example, and outputs the data to subsequent processing each time data is stored.
The short-time frequency conversion unit 902 processes the output signal of the buffering unit 901, for example, every several tens [ms]. The unit of processing is called a frame, and the number of points for each microphone (element) 102 processed in one frame is called a frame size Lframe. The processing start position is shifted by one frame shift Lshift every frame. That is, the data to be processed in the τ-th frame is τ * Lshift + Lframe from the τ * Lshift point. Data is converted into the frequency domain by short-time Fourier transform for each frame. For the m-th microphone element, the frequency f component in the frame τ is expressed as x _m (f, τ). Prior to the short-time Fourier transform, processing such as DC component cut and window function superposition may be performed on the waveform (signal). As the window function, a Hamming window, Hanning window, Blackman window, or the like can be applied.

A filter adaptation unit 903 adaptively processes a filter necessary for sound source separation for each frequency f.
The filtering unit 904 uses the sound source separation filter adapted by the filter adaptation unit 903 to separate sound for each sound source for each frame and each frequency. Here, the vector X (f, τ) is defined by [x ₁ (f, τ), x ₂ (f, τ),..., X _M (f, τ)] ^T. That is, X (f, τ) is a vector having elements of signals of all microphones (elements) 102 of frame τ and frequency f. Using the sound source separation filter W from X (f, τ), a separation signal is obtained by the following equation (12).

Here, each element of the vector y (f, τ) corresponds to a time τ and a frequency f component of each separated signal. The separated signal output from the filtering unit 904 is the power normalization unit 905, for each time τ and frequency f.
y _norm (f, τ) ← y (f, τ) / | y (f, τ) |
Normalized by. That is, the power of y _norm (f, τ) takes a value from 0 to 1. Rejection determination section 906 reexamines the sound source / frequency component whose normalized power frame average value is smaller than the threshold as a background noise component, removes it from the sound source separation result, and outputs the sound source / frequency component only for the component equal to or higher than the threshold. . At the time of output, each sound source may be subjected to a short-time inverse Fourier transform to return to a time domain waveform and then output.

FIG. 13 is a flowchart showing an adaptation process of the sound source separation filter W.
It is determined whether or not the value of the sound source separation filter W has sufficiently converged (convergence determination; step S1001). When the filter update count reaches the predetermined number, it may be determined that the filter has converged, or when the power of the non-diagonal term of the nonlinear covariance matrix described later is equal to or less than a predetermined value with respect to the power of the diagonal term It may be determined that it has converged.

If it determines with having converged (Yes of step S1001), a process will be complete | finished and the sound source separation filter W will be output.
If it is not determined that it has converged (No in step S1001), the process proceeds to the next step.

  The processing start position is set at the beginning of the waveform fetched by the buffering unit 901. Further, R (f) described later is cleared to 0 (initialization; step S1002).

  It is determined whether or not the processing start position is equal to or less than the end position of the waveform captured by the buffering unit 901 (i ≦ length? Determination, step S1003).

  When the processing start position does not reach the end position of the waveform (No in step S1003), X (f, τ) for each frame and frequency is filtered to obtain a sound source separated sound y (f, τ) ( Filtering; step S1004).

  Here, since the obtained sound source separation sound is a waveform separated by the sound source separation filter being applied, it is considered that the separation is insufficient. Therefore, R (f) is updated by the following equation (13) (covariance update; step S1005).

  Here, φ (x) is a function corresponding to the differential function of the probability distribution of the sound source and is defined by the following equation (14).

  R (f) is called a nonlinear covariance matrix and means that the separated sound sources become independent as the off-diagonal term approaches zero. The diagonal term corresponds to the size of each sound source. Therefore, the ratio of off-diagonal terms and diagonal terms becomes important. In the convergence determination of the separation filter, this ratio may be checked to determine the convergence.

Next, the processing start position of the waveform is added by the frame shift Lshift (variable update; step S1007).
Then, the processes after step S1003 are repeated.

  When the waveform processing start point has reached the end point of the waveform captured by the buffering unit 901 (No in step S1003), the process proceeds to step S1006.

  The separation filter is updated by the following equation (15) (filter update; step S1006).

η is a variable for controlling the filter update rate. The larger the value, the higher the filter convergence rate, but the greater the possibility that the filter will diverge. The smaller the value is, the slower the filter convergence speed is, but the possibility that the filter diverges becomes lower.
And the process after step S1001 is repeated.

FIG. 14 is a block diagram showing in detail the sound source position estimation unit 303 of FIG.
It is known that the inverse matrix of the separation filter separated by the sound source separation unit 302 (see FIG. 3) is a matrix composed of steering vectors for each sound source.
The inverse matrix calculation unit 1102 extracts the i-th column w (f, τ) ⁻¹ _i of the inverse matrix of the separation filter. Subsequent blocks are executed for each frame and for each frequency. The microphone array 101 is linearly arranged. The sound source position estimation unit 303 in this embodiment divides the microphone (element) 102 constituting the microphone array 101 into two. The divided microphone array 101 is called a subarray. After estimating the sound source direction in each subarray, the direction and distance can be known by taking the intersection of the sound source directions by triangulation.

  Since the sound source direction is estimated using the subarray divided into two, each subarray division unit 1103 includes two direction estimation units 1104, and one intersection estimation unit 1105 includes one direction estimation unit 1105 according to the estimation results of the two direction estimation units 1104. Will be estimated.

  The i-th column of the inverse matrix of the separation filter is divided as shown in the following Expression (16) for each subarray.

  Further, the steering vector when it is assumed that there is a sound source at the sound source position p is also divided into two for each subarray as shown in the following equation (17).

The direction estimation unit 1104 generates the sound source direction ^ θ _{i, 1} (f, τ) and the sound source direction ^ θ _{i, 2} (f, τ) for each subarray based on the following equations (18) and (19). Is estimated.

  The intersection point estimation unit 1105 considers that a sound source exists in the sound source direction estimated from the center position of each subarray, and estimates the sound source direction and distance by triangulation. Since it can be assumed that the distance between the center positions of the subarrays is known in advance, the estimation of the sound source direction and the distance can be easily performed by triangulation.

  The histogram estimation unit 1106 estimates the histogram of the sound source direction and distance obtained for each frequency, determines the sound source direction and distance with the highest histogram frequency as the direction and distance of the sound source, Output the distance.

FIG. 15 is a block diagram showing in detail the acoustic echo canceller 307 of FIG.
The speaker output sound propagates through the space and is received by the microphone array 101. In this embodiment, since the input sound is uniquely determined to be noise, if the acoustic echo canceller 307 is not present, the speaker output sound received by the microphone array 101 is determined to be noise. Therefore, when the acoustic echo canceller 307 is not present, when the speaker output coefficient is set, the speaker output coefficient is set so that the difference from the previous speaker output coefficient is large, and the speaker output coefficient is not stabilized. There is a problem that the localization direction of the sound output changes from time to time in an unstable manner. In order to avoid this problem, it is necessary to remove the speaker output component included in the speaker output sound received by the microphone array 101 in advance.

The reference signal capturing unit 1501 acquires the output sound source signal u (t) from the speaker 112. Each speaker output signal is superimposed with a different output coefficient S _m (t) for each speaker 112. The output coefficient superimposing unit 1503 convolves S _m (t) with u (t) by the following equation (20).

The signal after convolution is defined as u _m (t). Here, u _m (t) is a vector having the same length as that of the subsequent echo amount estimation filter, and is a vector in which signals after convolution are arranged in order from the newest in terms of time. u _m (t) is used as a reference signal of the acoustic echo canceller 307 of the microphone m.

  The input signal buffering unit 1502 buffers the input signal for a predetermined time and outputs it to the subsequent stage.

The filtering unit 1504 convolves the echo amount estimation filter g _m with the reference signal.
Echo canceling unit 1506, by subtracting the estimated echo value from the microphone input signal x _m (t), obtained signal after echo cancellation e _m (t), as in the following equation (21).

The filter update unit 1505 updates the echo amount estimation filter g _m as defined by the following equation (22) so that the signal after echo cancellation approaches 0.

  Here, μ is a filter update coefficient and takes a value from 0 to 1. The signal after echo cancellation output from the echo cancellation unit 1506 is output as an output signal processed by the acoustic echo canceller 307.

FIG. 16 is an explanatory diagram showing the relationship between software blocks and hardware according to the first embodiment of the present invention.
An analog sound pressure value captured by the microphone array 101 including a plurality of microphones 102 is converted into a digital sound pressure value by an A / D conversion processing unit 1602a disposed in the A / D conversion device 1602.
The converted digital sound pressure value is sent to the central processing unit 203 and subjected to various digital signal processing. A waveform capturing unit 1603a (corresponding to the waveform capturing unit 301 in FIG. 3) captures and buffers the digital sound pressure waveform.
The acoustic echo canceller 1603b (corresponding to the acoustic echo canceller 307 in FIG. 3) deletes the speaker output signal component in the captured digital sound pressure waveform.
The signal after echo cancellation is sent to a sound source separation unit 1603d (corresponding to the sound source separation unit 302 in FIG. 3), and is separated for each sound source.
A sound source position estimation unit 1603e (corresponding to the sound source position estimation unit 303 in FIG. 3) estimates a sound source position for each sound source. The estimated sound source position is the sound source position viewed from the microphone array position.
A sound source position converter 1603f (corresponding to the sound source position converter 304 in FIG. 3) converts the sound source position viewed from the microphone array position into the sound source direction viewed from the user listening position.
The output coefficient determination unit 1603g determines the speaker output coefficient so that the difference between the sound source direction viewed from the user listening position and the sound source direction at the user listening position on the synthesized wavefront of the speaker output sound is maximized.
The audio reproduction unit 1603c convolves the determined output coefficient for each speaker 112 with the output sound. Prior information such as work memory and microphone arrangement necessary for the digital signal processing so far is stored in the nonvolatile memory 205 and the volatile memory 204 (see FIG. 2).
A D / A conversion processing unit 1604a disposed in the D / A conversion device 1604 converts the digital signal output from the audio reproduction unit 1603c into an analog signal.
The analog signal is sent to a speaker array 111 including a plurality of speakers 112, and is output as an acoustic signal from each speaker 112 and radiated into the air.

FIG. 17 is a block diagram showing a configuration for controlling an output method of audio output sound such as music in the present embodiment.
As described above, the speaker output coefficient determination unit 1701 determines the speaker output coefficient so that the difference between the sound source direction and the noise direction at the user listening position of the speaker composite wavefront is maximized.

  The audio source acquisition unit 1702 acquires playback sound from a playback device such as a compact disc player. In the audio reproduction 1703, the output coefficient for each speaker 112 is superimposed on the acquired reproduction sound, and then output from each speaker 112 and radiated into the air. Moreover, if the output coefficient is constantly changed every time the noise direction changes, the sound may be difficult to hear. It is desirable that the output coefficient is not changed at least while playing the same source, for example, music of the same music.

FIG. 18 is a flowchart showing a process for determining the output coefficient determination timing.
First, it is determined whether or not the source of the output sound has been changed (source change determination; step S2001). In the case of music, this can be realized by inquiring of the audio device whether or not the reproduced music has been completed.

If the source has been changed (Yes in step S2001), the speaker output coefficient is changed (output coefficient change; step S2002), and the process proceeds to the next step (step S2003).
In the output coefficient change (step S2002), the speaker output coefficient is determined from the updated histogram.

  If the source has not been changed (No in step S2001), and after changing the output coefficient, the waveform of the next time is captured (waveform capture; step S2003).

The acquired waveform is sent to the acoustic echo canceller 307, and the acoustic echo component is eliminated (step S2004).
Next, the sound sources are separated (sound source separation; step S2005).
Then, the sound source position at the microphone position for each sound source is estimated (sound source position conversion estimation; step S2006).
Then, the sound source direction at the user listening position is calculated (sound source position conversion; step S2007).
Then, the histogram of the sound source direction at the user listening position is updated (histogram update; step S2008).
Then, it is determined whether or not the reproduction is finished (step S2009). If the reproduction has ended (Yes in step S2009), the process ends.
If the reproduction is not finished (No in step S2009), the processes in and after step S2001 are repeated.

FIG. 19 is a timing chart showing an example of output coefficient setting timing and audio source playback timing.
It is assumed that the noise direction changes from θ1 to θ2. When the speaker output coefficient is constantly updated, the speaker output coefficient changes at the timing when the noise direction changes. In this example, the output coefficient changes during playback of the source (2), and the user Sounds that are hard to hear. As shown in this example, by adopting a configuration in which the output coefficient is changed at the timing when the sources (2) and (3) are changed, it is possible to reduce discomfort given to the user. If multiple sources are included in a single source such as 5.1ch surround music, the speaker output is output by the number of sound sources in order starting from the difference in the sound source direction of the synthesized wave front of the speaker output sound from the noise source direction. A configuration may be adopted in which a coefficient is selected and superimposed on each sound source.

FIG. 20 is a hardware configuration diagram showing the sound reproducing device 1b according to the second embodiment of the present invention.
A sound reproducing device 1b shown in FIG. 20 shows a hardware configuration when applied to a hands-free call in the passenger compartment 11, and in addition to the configuration shown in FIG. Is added as

Digital sound pressure data in the passenger compartment 11 acquired by the central processing unit 203 is sent to the mobile phone 1801.
The cellular phone 1801 transmits digital sound pressure data to the other party through the telephone network. Further, the sound transmitted from the other party through the telephone network is superimposed on the output coefficient for each speaker calculated in the central processing unit 203 and then sent to the multi-channel D / A converter 206 to be converted into an analog signal.
The analog signal is sent to the speaker array 111, output from each speaker 112, and radiated into the air.
In a hands-free call configuration, sound may be emitted from the user listening position. Therefore, the speaker output coefficient determination unit 1701 of the present embodiment may be configured such that a sound source whose sound source position is in the vicinity of the user listening position among the signals of each sound source after sound source separation is rejected and not regarded as noise. . Further, a configuration may be adopted in which a sound source near the user listening position after sound source separation is transmitted to the mobile phone 1801. By adopting such a configuration, it becomes possible to send a clear sound with little noise to the other party even in the passenger compartment 11 where the noise exists.

FIG. 21 is a block diagram showing a software configuration of a sound field reproduction system that reproduces a sound field at a virtual sound source position using the sound source position conversion process at the user listening position according to the present invention.
The digital waveforms of a plurality of channels captured by the waveform capturing unit 301 are sent to the acoustic echo canceller 307 and the speaker output sound component is removed.
The sound source separation unit 302 separates the removed waveform (signal) for each sound source.
The sound source position estimation unit 303 estimates the sound source position at the position of the microphone array 101 for each separated sound source.
The sound source position conversion unit 304 converts the sound source position to the sound source position at the virtual user listening position.
The speaker volume determination unit 1906 superimposes the steering vector of the sound source viewed from the virtual user listening position on the output signal separated by the sound source separation unit 302.
After repeating the same processing for all sound sources, the waveform recombination unit 1907 integrates and outputs the waveforms for each sound source for each microphone (element) 102.

1 sound reproduction device (first embodiment)
1b Sound reproduction device (second embodiment)
DESCRIPTION OF SYMBOLS 10 Car 11 Car compartment 12 Driver's seat 101 Microphone array 102 Microphone 111 Speaker array 112 Speaker 202 Multi-channel A / D converter 203 Central processing unit 204 Volatile memory 205 Non-volatile memory 206 Multi-channel D / A converter 208 Seat sensor 301 Waveform acquisition unit 302 Sound source separation unit 303 Sound source position estimation unit 304 Sound source position conversion unit 305 Histogram update unit 306 Output coefficient determination unit 307 Acoustic echo canceller 401 Output coefficient storage unit 402 Speaker output unit 403 Output source acquisition unit 502 User position extraction unit 503 Conversion vector generation unit 504 Microphone position database 505 Conversion vector addition unit 602 Direction matrix calculation unit 603 Eigenvalue / vector calculation unit 604 Minimum cost coefficient calculation unit 702 Direction Matrix calculation unit 703 Speaker inner product calculation unit 704 Minimum cost coefficient calculation unit 801 Direction matrix calculation unit 802 Passenger position estimation unit 803 Known noise position 901 Buffering unit 902 Short-time frequency conversion unit 903 Filter adaptation unit 904 Filtering unit 905 Power normalization Unit 906 rejection determination unit 1102 inverse matrix calculation unit 1103 subarray division unit 1104 direction estimation unit 1105 intersection estimation unit 1106 histogram estimation unit 1501 reference signal capturing unit 1503 output coefficient superposition unit 1504 filtering unit 1505 filter update unit 1506 echo cancellation unit 1602 A / D conversion device 1604 D / A conversion device 1701 Speaker output coefficient determination unit 1702 Audio source acquisition unit 1703 Audio playback 1801 Cellular phone 1906 Speaker volume Tough 1907 waveform recombination part

Claims (6)

A sound reproduction device including a plurality of speakers as a reproduction sound source ,
A sound source direction estimation unit for estimating a sound source direction at a position of a microphone array including a plurality of microphones ;
A sound source direction conversion unit that converts the sound source direction at the estimated position of the microphone array into the sound source direction at the user listening position;
Based on the converted sound source direction , a sound source direction at the user listening position of a noise source other than the reproduction sound source is calculated, and sound is emitted from the sound source direction of the noise source at the user listening position and the plurality of speakers. An output coefficient determination unit that determines the output coefficient of each of the plurality of speakers so that the sound image localization direction at the time is different,
A sound reproducing device comprising: Equipped with a seat sensor to detect occupants,
The output coefficient determination unit, the sound reproducing apparatus according to claim 1, characterized in that the operation is regarded as the noise source is present in the passenger position detected by the seat sensor. Equipped with a seat sensor to detect occupants,
The output coefficient determination unit, the sound reproducing apparatus according to claim 1, an occupant position detected by the seat sensor and performing the operation is regarded as the a user listening position. By adding a second position vector that is a position vector of a sound source position other than the reproduction sound source with respect to the microphone array position to a first position vector that is a position vector of a microphone array position with respect to the user listening position, the user listening Calculating a third position vector, which is a position vector of the sound source position with respect to a position, and converting a sound source direction at the position of the microphone array into a sound source direction at a user listening position based on the third position vector; The sound reproducing device according to claim 1. In an environment in which a plurality of microphones and a plurality of speakers are arranged at predetermined positions, a sound reproduction method for reproducing sound from a sound reproduction device using the plurality of speakers as a reproduction sound source ,
The position of the microphone array of a sound source other than the reproduction sound source from the plurality of speakers, using the information on the sound from the microphone array composed of a plurality of microphones and the information on the positional relationship between the microphone array and each of the speakers. A sound source direction estimating step for estimating a sound source direction at
A sound source direction conversion step of converting a sound source direction at the estimated position of the microphone array into a sound source direction at a user listening position;
Based on the converted sound source direction , a sound source direction at the user listening position of a noise source other than the reproduction sound source is calculated, and a sound source direction at the user listening position of the noise source and sound from the plurality of speakers are calculated. An output coefficient determining step for determining the output coefficient of each speaker so that the sound image localization direction when radiated is different;
A sound emission step of emitting sound from the speaker according to the determined output coefficient;
A sound reproduction method comprising: In the output coefficient determination step, the speaker selected from the plurality of speakers is located in a direction in which the sound source direction at the user listening position is different.
The sound reproduction method according to claim 5, wherein:

Images