JPH06105400A - Three-dimensional space reproduction system - Google Patents

Abstract

PURPOSE:To obtain the three-dimensional space reproduction system in which lots of unspecified persons are taken for objects, the configuration of the equipment is simple with high general-purpose performance, natural sound image fixed location and especially a sense of distance is reproduced with fidelity. CONSTITUTION:An attenuator 20 controls a level P of a presentation sound inputted from an acoustic signal source 18 in response to an amplitude of a depth signal to generate a picture with a parallax to a display screen in order to provide a stereoscopic sense to a video image generated by a depth signal generator 12. Moreover, inter-ear mutual correlation coefficient controller 22 controls an energy ratio Er/d of a direct sound versus reverberation sound and an inter-ear mutual correlation coefficient phi of the presentation sound inputted from the attenuator 20 according to the depth signal amplitude to apply the result to right and left ear use receivers 24R, 24L.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional space reproduction system for reproducing a realistic three-dimensional space by fusing an image and a sound image, and more particularly to a HMD (Head Mounted).
3) equipped with a stereoscopic image display device such as a display) and a sound image control device that controls the position of the sound image according to the video signal.
Dimensional space reproduction system.

[0002]

2. Description of the Related Art Improving power and presence by making images three-dimensional has been performed for a relatively long time in the entertainment field of movies and television.

On the other hand, in recent years, attempts have been made to put into practical use a three-dimensional space reproduction system that allows a user to experience another space while staying at home by using a special image display device capable of visually blocking the outside world. ing.

The basic structure of this three-dimensional space reproduction system is to display an image by an HMD (Head Mounted Displa).
y) a small image display device that can be worn on the face,
It consists of a headphone listening device for presenting audio signals and a device for supplying and controlling video signals and audio signals.

The advantage of such a three-dimensional space reproduction system is that it is possible to visually display a three-dimensional image while blocking the outside world, and thus it is easy to psychologically lead to another space. , It means that you can virtually experience other spaces while you are at home.

However, unless special processing is applied to the acoustic signal, the sound appears right above the ears and the head, and at an extreme level, due to a peculiar phenomenon peculiar to listening to the headphones, even though the image appears to be far ahead. In this case, there is a problem that the phenomenon is heard in the head, that is, the spatial positions of the image and the sound image are separated, and the natural spatial reproduction is impaired. Therefore, in order to reproduce a natural sound image localization using a headphone listening system, various methods for processing an acoustic signal suitable for headphone listening have been proposed.

For example, FIG. 5 shows Wenzel, EM et al, "Virt.
ual Acoustics Displays, "NASA Conference Publicati
FIG. 3 is a conceptual diagram of a signal control system for realizing a natural sound image localization using headphone listening, which is quoted from on (1991).

The principle of this signal control system is that the head-related acoustic transfer function (frequency characteristics of sound pressure at the entrance of the ear canal) of a listener, which is said to be closely related to sound image localization, is measured in advance for each sound incident direction. Next, a convolution calculation is performed between the head acoustic transfer function when the sound is incident from the direction to be localized and the acoustic signal to be presented to the listener, and the synthesized signal is received by the headphones, and a natural sound is obtained. It is intended to reproduce sound image localization.

[0009]

It is apparent in the field of acoustic engineering that the head acoustic transfer function is an important clue (Cue) in perceiving the direction of a sound source and the distance to the sound source. In principle, a method for synthesizing a presentation sound according to a change in a head acoustic transfer function to realize natural sound image localization as in the above signal control system makes sense. However, the signal control system using the head acoustic transfer function as the only clue for sound image localization has the following problems (1) to (4) in order to be put to practical use.

(1) Originally, the head acoustic transfer function is unique to each person. In particular, the higher the frequency region that has important information as a clue for sound image localization, the more remarkable the influence of the shape of the auricle, and the larger the individual difference. Therefore, in order to perform accurate sound image control, the head acoustic transfer function must be measured for each listener. Therefore, it is suitable for a special purpose in the laboratory, but not suitable for a general purpose for a large number of unspecified persons.

(2) When measuring the head related transfer function,
Although the tip of the probe tube microphone is placed at the entrance of the external ear, it cannot completely stop the minute movement of the person to be measured, and the tip of the microphone always fluctuates near the entrance of the ear canal. For this reason, in a high frequency region where resonance / anti-resonance peaks / dip caused by dents and folds in the auricle appear remarkably, the sound pressure may change by about 10 dB even if the tip of the microphone is displaced by a few millimeters. It is difficult to measure. Further, even if the head-related transfer function can be accurately measured, the characteristics of headphones for accurately reproducing the head-related transfer function have not been sufficiently studied.

(3) A computer and a special digital signal processing device are required to measure the head acoustic transfer function and perform the convolution calculation of the head acoustic transfer function and the acoustic signal to be presented. It becomes expensive.

(4) A study of headphone listening focusing on the head acoustic transfer function has confirmed that the direction of the sound image can be accurately controlled. However, regarding distance, although it has been confirmed that out-of-head localization by listening to headphones can be achieved, how far away can be controlled, or what acoustic characteristics other than head acoustic transfer functions should be considered as clues. It has not been confirmed.

By the way, when we hear a sound in an actual space, as shown in FIG. 6A, a direct sound that directly reaches from a sound source to a listening point and a sound that is reflected after reflecting on a wall surface or a ceiling are received. Although a group of reflected sounds reaching a point, that is, a reverberant sound is also heard, the head acoustic transfer function is an acoustic characteristic focusing only on the direct sound.

As shown in FIG. 6B, the loudness of the direct sound decreases as the distance to the sound source increases, but the loudness of the reverberant sound does not change. At a distance, the level P of the sound reaching the listening point
Not only becomes smaller, but also the ratio of the loudness of the reverberant sound (energy r of the reverberant sound) to the loudness of the direct sound (energy d of the direct sound), ie, the direct sound to reverberant sound energy ratio E
_{The r / d} becomes large, and the reverberant sound becomes louder at a certain distance or more.

Further, our hearing has the ability to distinguish sounds by the left and right ears, that is, the function of binaural hearing, and plays an important role in perceiving the arrival direction of sound and the distance to the sound source. . For example, when the distance from the sound source increases, not only the ratio of reverberant sound to direct sound increases, but also the degree of correlation between the waveforms of the two systems of sound entering both ears decreases.

As shown in FIG. 6C, when the distance from the sound source increases, the proportion of reverberant sound increases, so that the symmetry of the waveform of the sound incident on the left and right ears distorts. The interaural cross-correlation coefficient Φ shown in (3) decreases, and conversely, the interaural cross-correlation coefficient Φ increases when the sound source is approached.

[0018]

[Equation 1]

As described above, the level P of the reaching sound, the energy ratio E _{r / d of the} direct sound to the reverberant sound, and the interaural cross correlation coefficient Φ are
The acoustic characteristics are closely related to the change in the distance from the sound source to the listening point, and the individual characteristics can be ignored.

The present invention has been made in view of the above points, and the individual difference between listeners as represented by the head acoustic transfer function depends only on the microscopic acoustic characteristics that pose a problem. In addition, by effectively utilizing the macroscopic acoustic characteristics where individual differences do not pose a problem, it is highly versatile that can target an unspecified number of people, and the device configuration is simple and natural sound localization, especially It is an object of the present invention to provide a three-dimensional space reproduction system using a headphone listening system that enables a faithful sense of distance to be reproduced.

[0021]

In order to achieve the above-mentioned object, a three-dimensional space reproduction system according to the present invention comprises a stereoscopic image display device for visually displaying a stereoscopic image and the stereoscopic image display device. A headphone listening system for presenting an audio signal related to a displayed image, and a level of a presentation sound in the audio signal supplied to the headphone listening system according to the size of a depth signal of the image, direct sound vs. reverberation The present invention is characterized by including control means for controlling the energy ratio of sound and the interaural cross-correlation coefficient, and controlling the sense of distance of the sound image according to the sense of depth of the image.

[0022]

That is, according to the three-dimensional space reproduction system of the present invention, the level of the presentation sound is controlled by the control means in accordance with the amplitude of the depth signal that produces an image with parallax on the display screen in order to give a stereoscopic effect to the image. By controlling P, the energy ratio E _{r / d of} the direct sound to the reverberant sound, and the interaural cross-correlation coefficient Φ, and combining the sense of distance between the image and the sound image, a realistic 3
I try to reproduce the dimensional space.

Therefore, for an image with a long sense of distance, the level P of the presented sound is made small, the energy ratio E _{r / d} of the direct sound to the reverberant sound is made large, and the interaural cross correlation coefficient Φ is made small. As a result, the sense of distance to the sound image can be increased, and conversely, for images with a closer sense of distance, the level P of the presented sound is increased and the energy ratio E _r of the direct sound to the reverberant sound is increased. _{/ d} is small, and interaural cross correlation coefficient Φ
By increasing the distance, the sense of distance to the sound image can be made closer, so that it is possible to reproduce a realistic three-dimensional space in which the image and the sound image are fused.

In particular, according to the present invention, not only the energy ratio E _{r / d} of the direct sound to the reverberant sound can be changed but also the interaural cross-correlation coefficient Φ can be changed in accordance with the energy ratio E _{r / d.} The peculiar phenomenon of intracranial localization, which is a major problem of the listening system, can be easily solved, and the sense of distance can be easily controlled. Further, when controlling the sound image localization only with the headphone listening system, there is another big problem that it is difficult to stably localize the sound image forward, but according to the present invention, an acoustic signal corresponding to a video signal is generated. By using the fusion effect of the image and the sound image, it is possible to easily realize the front localization.

[0025]

Embodiments of the present invention will be described below with reference to the drawings. [First Embodiment] FIG. 1 shows the overall configuration of a three-dimensional space reproduction system according to the first embodiment of the present invention.

The three-dimensional space reproduction system in this embodiment comprises a television image signal source 10, a depth signal generator 12,
High-speed phase modulators 14R and 14L, right eye display device 16
R and left-eye display device 16L, acoustic signal source 18, attenuator 20 with variable gain G1, binaural cross-correlation coefficient control device 22, right-ear receiver 24R, and left-ear receiver 2
It is composed of 4L.

That is, the video signal output from the television image signal source 10 is input to the high speed phase modulators 14R and 14L. These high-speed phase modulators 14R and 14L phase-modulate the video signal from the television image signal source 10 within the horizontal scanning period in accordance with the amplitude of the depth signal output from the depth signal generator 12 to change the position of the video. Horizontally offset 2
The video signal of the system is supplied to the display device 16R for the right eye and the display device 16L for the left eye. This creates parallax and gives the viewer a stereoscopic effect of the image.

On the other hand, the depth signal generator 12 is connected to the variable attenuator 20 and the interaural cross-correlation coefficient controller 22 so as to control the perspective of the sound image in accordance with the perspective of the display image. ing.

Here, the interaural cross-correlation coefficient control device 22 is, for example, a two-system reverberation synthesis circuit 30 shown in FIG.
By using R and 30L, not only the cross-correlation coefficient Φ of the acoustic signals of the two systems but also the energy ratio (E _{r / d} ) of the direct sound to the reverberant sound can be controlled at the same time.

That is, in FIG. 2, 30R is a reverberation synthesis circuit for the right channel, 30L is a reverberation synthesis circuit for the left channel, 32 _{1 to} 32 ₄ are attenuators capable of varying the gain G2, and 34 ₁ and 34 ₂ are Delay time τ
1 is a delay element, 36 _{1 to} 36 ₃ are subtractors, and 38 is an adder. Here, the delay element 3
The delay time τ1 of 4 ₁ , 34 ₂ may be fixed or variable, but in this embodiment, it is a fixed value of several msec. The delay time τ1 determines the timing of reverberation generated by the reverberation synthesis circuit.

The acoustic signal supplied from the attenuator 20 and input from the terminal 40 is a reverberation synthesis circuit 3 for the right channel.
0R and the reverberation synthesis circuit 30L for the left channel are input respectively. The acoustic signal input to the right channel reverberation synthesis circuit 30R is input to the attenuator 32 ₁ and the subtractor 36 ₁ , and the signal input to the attenuator 32 ₁ becomes the control signal from the depth signal generator 12. It is attenuated by the amount set based on the above, and output from the output terminal 42 to the right ear receiver 24R.

Further, the attenuated acoustic signal is input to the attenuator 32 _2, and the depth signal generator 1 is also used.
The signal is further attenuated by the amount set based on the control signal from 2 and input to the subtractor 36 ₁ . Then, the subtractor 36 ₁ subtracts from the original input signal, and the output is input to the delay element 34 ₁ . After being delayed by τ1 by the delay element 34 _1, it is added to the output of the attenuator 32 _{1 by} the adder 38 and output from the output terminal 42. Similarly, the operation of repeating this signal from the output terminal 42 through the attenuator 32 ₂ , the subtractor 36 ₁ , the delay element 34 ₁ , and the adder 38 is repeated.

That is, at time t = 0, the inputs of the subtractor 36 ₁ are in opposite phase to each other, so that the subtractor 36 ₁
Signal input to the attenuator 32 ₁ from the input terminal 40,
The difference between the signal input to the subtractor 36 ₁ via the adder 38 and the attenuator 32 ₂ is the subtractor 36 ₁ , the delay element 34 ₁ ,
Only the signal passing through the attenuator 32 ₁ and the adder 38 from the input terminal 40 is output from the output terminal 42 through the loop formed by the adder 38 and the attenuator 32 ₂ , and at time t = τ.
1, t = 2 * τ1, ..., The signal from the loop is output. For example, if an impulse input is input to the input terminal 40 at time t = 0 as shown in FIG. 3A, an output as shown in FIG. 3B is obtained from the output terminal 42.

On the other hand, the reverberation synthesis circuit 30L for the left channel
Has a configuration similar to that of the right channel reverberation synthesis circuit 30R, except that the attenuator 36 is used instead of the adder 38.
_{The difference} is that ₃ is used, that is, the phase behind the delay element 34 ₂ is different.

Therefore, the reverberation synthesis circuit 3 for the left channel
At 0L, when the impulse input shown in FIG. 3A is input at time t = 0, an output as shown in FIG. 3C can be obtained from the output terminal 44.

The correlation coefficient Φ of the left and right signals thus obtained can be changed by changing the gain G2. That is, by using the right and left channel reverberation synthesis circuits 30R and 30L, the correlation coefficient Φ of the two systems of signals output from the binaural cross correlation coefficient control device 22 is ), It can be varied by changing the gain G2 of the attenuators 32 _{1 to} 32 ₄ .

[0037]

[Equation 2]

Therefore, when the gain G2 = 1 of the variable attenuators 32 _{1 to} 32 ₄ is canceled at the points of the subtractors 32 ₁ and 32 _{2 and} the loop does not rotate, the output signal is a direct sound component. Energy ratio E _{r / d of} direct sound to reverberant sound
= 0 and the cross-correlation coefficient Φ = 1 of the two signals. Also the gain

[0039]

[Equation 3] At this time, the energy ratio E _{r / d} of the direct sound to the reverberant sound becomes maximum and the cross-correlation coefficient Φ = 0 of the two signals. Next, the specific operation and effect of the three-dimensional space reproducing apparatus according to the present embodiment will be described.

The video signal output from the television image signal source 10 is input to the high speed phase modulators 14R and 14L.
Then, the video signal is phase-modulated within the horizontal scanning period according to the amplitude of the depth signal output from the depth signal generator 12, and the video signal of two systems in which the video position is horizontally displaced is displayed for the right eye. By supplying to the device 16R and the display device 16L for the left eye, parallax is created to give the viewer a stereoscopic effect of the image.

On the other hand, regarding the sense of distance of the sound image, when the amplitude of the depth signal output from the depth signal generator 12 is large (when the sense of depth of the image is large), the gain G1 of the variable attenuator 20 is reduced. This reduces the level P of the reproduced sound, and the gain G2 of the variable attenuator 32.
It is possible to localize a sound image by increasing the energy ratio E _{r / d} of the direct sound to the reverberant sound and decreasing the interaural cross-correlation coefficient Φ.

On the other hand, when the depth signal output from the depth signal generator 12 is small (when the sense of depth of the image is small), the gain G1 of the variable attenuator 20 is increased to increase the level P of the reproduced sound. By increasing the gain G2 of the variable attenuator 32 to decrease the energy ratio E _{r / d} of the direct sound to the reverberant sound and increasing the interaural cross-correlation coefficient Φ, the sound image is localized in the vicinity. be able to.

As described above, since the perspective of the sound image can be changed according to the depth of the image regardless of individual differences, the image and the sound image are fused to reproduce a natural and realistic three-dimensional space. be able to. [Second Embodiment]

FIG. 4A is a diagram showing the configuration of the second embodiment of the present invention. In this embodiment, the video reproducing system is the same as that of the first embodiment, and therefore only the sound reproducing system is shown in FIG. In the second embodiment, in order to control the sound image localization with higher accuracy, dip filters 50R and 50L are added to the sound reproduction system shown in the first embodiment.

These dip filters 50R and 50L
Is a model that simplifies and models the anti-resonance phenomenon of the auricle occurring at about 8 kHz to 10 kHz, and is also one of the peculiar phenomena of headphone listening by optimizing the position of the center frequency f _O of the dip. It has been confirmed that the rise of the sound image can be suppressed. However, as with head related transfer functions, because of individual differences in aptitude value of the center frequency f _O of the dip, in one fixed filter it could not bring the same effect to an unspecified number of people.

Therefore, the dip filters 50R and 50L in this embodiment have the center frequency f _{O of the} dip as
It is possible to match the suitability value of each viewer.

That is, the dip filters 50R and 50L
As shown in FIG. 4B, a delay device 51 that can vary the delay time τ2, a delay time variable volume 52 for changing the delay time τ2 of the delay device 51, and an output of the delay device 51 are It is configured by an adder 53 that adds the signal input from the input terminal 60 and outputs the added signal to the output terminal 62.

[0048] In dip filter 50 having such a configuration, by varying the delay time τ2 of the delay device 51 by a delay-time variable volume 52, it is possible to change the center frequency f _O of the dip. That is, the transfer function of the dip filter 50 and its absolute value are given by the following equations (3) and (4). Therefore, the filter characteristic can be written as shown in FIG. 4C, and the central frequency f _{O of the} dip can be changed by changing the delay time τ2.

[0049]

[Equation 4]

In the method of using the three-dimensional space reproduction system according to the second embodiment, first, while the viewer listens to the test sound in advance, the delay time variable volume 73 is used so that the sound image is located on the horizontal plane. after suitability of the center frequency f _O of the dip, it may be operated in the same manner as in the first embodiment.
According to the second embodiment, since the rise of the sound image, which is a peculiar phenomenon of listening to headphones, can be surely suppressed, a more natural three-dimensional space can be reproduced.

As described above with reference to the first and second embodiments, by effectively utilizing the macroscopic acoustic characteristics in which individual differences are not a problem, the versatility is high for an unspecified number of people, and It is possible to provide a three-dimensional space reproduction system that uses a headphone listening system that has a simple device configuration and that enables natural sound localization, particularly reproduction of perspective.

Further, by controlling the perspective of the sound image in accordance with the perspective of the image, a phenomenon in which the sound image is attracted to the image, that is, the fusion of the image and the sound image occurs, and it is difficult to control the sound image by conventional headphone listening. Stable forward localization can be easily realized.

Furthermore, since the rise of the sound image, which is one of the peculiar phenomena of listening to headphones, can be suppressed according to the individual difference by a simple method, the fusion of the image and the sound image can be further strengthened.

In the above two embodiments, the simple reverberation synthesis circuits 30R and 30L are used to control the energy ratio E _{r / d} of the direct sound to the reverberation sound and the interaural cross correlation coefficient Φ. While changing the distance from the sound source to the sound receiving point in the actual space, the energy ratio E _{r / d} and the interaural cross correlation coefficient Φ
It is also possible to pick up and use different signal tones.
It goes without saying that it is also possible to measure or simulate the impulse response of the sound receiving point in the actual space, and to control using the synthesized signal obtained by performing the convolution operation of the acoustic signal to be presented and the impulse response.

[0055]

As described in detail above, according to the present invention,
Effective use of macroscopic acoustic characteristics where individual differences do not matter, not depending only on microscopic acoustic characteristics where individual differences such as the head-related acoustic transfer function are problematic. This makes it possible to target an unspecified number of people with high versatility, and the device configuration is simple,
Further, it is possible to provide a three-dimensional space reproduction system using a headphone listening system that enables natural sound image localization, particularly reproduction of a faithful sense of distance.

[Brief description of drawings]

FIG. 1 is a block configuration diagram of a first embodiment of the present invention.

FIG. 2 is a diagram showing a detailed configuration of an interaural cross correlation coefficient control device in FIG.

FIG. 3 is a diagram showing a relationship between input / output signals in the device configuration of FIG.

FIG. 4A is a block diagram of a sound reproduction system according to a second embodiment of the present invention, and FIGS. 4B and 4C are diagrams showing the configuration and filter characteristics of the dip filter in FIG. 4A. Is.

FIG. 5 shows Wenzel, EM et al, "Virtual Acoustics Display" to show the configuration of a conventional 3D space reproduction system.
s, "A drawing quoted from NASA Conference Publication (1991).

FIG. 6A is a diagram schematically showing a direct sound and a reverberant sound that reach from a sound source to a listening point, and FIG. 6B is a relationship between a distance to the sound source and a sound pressure level at the sound receiving point. FIG.
(C) is a diagram showing the relationship between the distance to the sound source, the direct sound-to-reverberant sound energy ratio E _{r / d,} and the binaural cross-correlation coefficient Φ.

[Explanation of symbols]

10 ... Television image signal source, 12 ... Depth signal generator,
14R, 14L ... High-speed phase modulator, 16R ... Right eye display device, 16L ... Left eye display device, 18 ... Acoustic signal source, 2
0 ... Attenuator, 22 ... Interaural cross correlation coefficient control device, 24
R ... right ear receiver, 24L ... left ear receiver, 50R, 5
0L: Dip filter.

Claims (1)

[Claims] 1. A stereoscopic image display device for visually displaying a stereoscopic image, a headphone listening system for presenting an audio signal related to the image displayed on the stereoscopic image display device, and a depth of the image. Control means for controlling the level of the presentation sound, the energy ratio of the direct sound to the reverberation sound, and the interaural cross-correlation coefficient in the acoustic signal supplied to the headphone listening system according to the magnitude of the signal. The three-dimensional space reproduction system is characterized by controlling the sense of distance of the sound image according to the sense of depth of the image.