JP6186627B2 - Multimedia device and program - Google Patents

Description

  The present invention relates to a technique for localizing an image displayed on a display and sound output from a plurality of loudspeakers.

  2. Description of the Related Art In recent years, the capacity of recording devices has been increased, and information devices represented by personal computers, mobile phones, smartphones, and the like can store and reproduce a large number of video contents and audio contents. Furthermore, a large number of video contents and audio contents can be reproduced via the Internet. Here, when video content is played back, the video is generally played back on a display provided in the information device, and the audio is played back from one or more loudspeakers built in the information device.

  However, the position of one or more loudspeakers built in the information device is naturally fixed in advance. On the other hand, in the reproduction of video content, it is generally assumed that the video image is reproduced on a horizontally long display. For this reason, in information equipment having a plurality of loudspeakers, if a plurality of loudspeakers are installed assuming that the display is placed horizontally long, it is suitable for stereo playback of audio in video playback. Is very unnatural because the localization of video and audio does not match.

  For example, in the case of an information device having a housing that is substantially rectangular when shown in a plan view, such as a smartphone, two loudspeakers may be arranged in parallel on one short side of the substantially rectangular shape. Many. When video content is played back on such a smartphone, the user views the video content with the display horizontally long. At this time, since the sound is output from two loudspeakers arranged in parallel on one short side of the substantially rectangular shape, the user can hear the sound from only one of the left and right directions. In such a situation, the localization of video and audio does not match, giving the user a sense of discomfort. In particular, in the case of stereo reproduction with audio, the acoustic effect is significantly impaired.

  In Patent Document 1, three loudspeakers are provided in the upper center, lower left, and lower right of the display, the orientation of the information device is detected, and two loudspeakers suitable for stereo reproduction are selected, so that the above problem is solved. We are trying to solve it. In Patent Document 2, three loudspeakers are provided at the three corners of the display, the orientation of the information device is detected, and two loudspeakers suitable for stereo reproduction are selected to solve the above problem. ing. Furthermore, in Patent Document 3, two loudspeakers are provided on the diagonal line of the display, the orientation of the information device is detected, and the left channel and the right channel are switched to solve the above problem.

JP 2006-174277 A JP 2009-232424 A JP2003-078601A

  However, in the techniques described in Patent Document 1 and Patent Document 2, three loudspeakers are required, and the number of devices increases. As a result, it is not easy to downsize the information device, and the cost may increase. Further, in the method of Patent Document 3, two loudspeakers are arranged on a diagonal line, and video and audio are localized on the diagonal line, which is very unnatural when the information device is large. For this reason, high satisfaction cannot be obtained with a small number of parts. Furthermore, the conventional technology only considers whether the display is set to be vertical or horizontal, and does not consider the position where the user views the video content. For this reason, the conventional technology cannot optimize the audio in accordance with the position of the user.

  The present invention has been made in view of such circumstances, and the video and audio can be changed according to the position at which the user views the video content without changing the number and position of the preinstalled loudspeakers. An object of the present invention is to provide a multimedia device and a program capable of matching the localization of the program.

  (1) In order to achieve the above object, the present invention takes the following measures. That is, the multimedia apparatus of the present invention is a multimedia apparatus having a display and a plurality of loudspeakers, and a position information acquisition unit that acquires a viewing position where a user views multimedia data and an installation position of each loudspeaker. And a virtual sound source generation unit that corrects an input signal so that sound output from each of the loudspeakers is sound adapted to an image displayed on the display at the viewing position. To do.

  In this way, the input signal is corrected so that the sound output from each loudspeaker becomes sound adapted to the image displayed on the display at the user's viewing position, so that the user can view the video content. Accordingly, the installation position of each loudspeaker is virtually moved to generate a virtual sound source, and the localization of video and audio can be matched.

  (2) In the multimedia device of the present invention, the virtual sound source generation unit includes a first head-related transfer function indicating a sound transfer characteristic between the viewing position and an installation position of each loudspeaker, and At least based on a second head-related transfer function indicating a sound transfer characteristic between a viewing position and a virtual loudspeaker installation position that outputs sound adapted to an image displayed on the display to the viewing position. One filter is provided, and an input signal is corrected using the filter.

  As described above, since the input signal is corrected using the filter created based on the first and second head related transfer functions, the virtual loudspeaker is placed at a position in a space different from the installation position of each loudspeaker as a virtual sound source. It is possible to generate a speaker and provide sound adapted to the image displayed on the display from the virtual sound source at the viewing position of the user. This makes it possible to match the localization of video and audio according to the position where the user views the video content.

  (3) In the multimedia device of the present invention, the virtual sound source generation unit includes a plurality of first head-related transfer functions and a plurality of second head-related transfer functions, and is acquired by the position information acquisition unit. The filter is created according to the viewing position of the user and the installation position of each loudspeaker, and the input signal is corrected using the created filter.

  As described above, the filter includes a plurality of first head-related transfer functions and a plurality of second head-related transfer functions, and the filter is selected according to the viewing position of the user acquired by the position information acquisition unit and the installation position of each loudspeaker. Since it is created, it is possible to match the localization of video and audio in real time according to the viewing position of the user and the current installation position of each loudspeaker. For example, when the user tries to view the video content by rotating the information device by 90 degrees, the position of the virtual sound source can be reset following that.

  (4) In the multimedia apparatus of the present invention, the virtual sound source generation unit includes a plurality of filters created in advance according to a plurality of viewing positions or a plurality of installation positions of the respective loudspeakers, and the position information One of the filters is selected according to the viewing position acquired by the acquisition unit and the installation position of each loudspeaker, and the input signal is corrected using the selected filter.

  As described above, a plurality of filters prepared in advance according to a plurality of viewing positions or a plurality of the respective loudspeaker installation positions are provided, and according to the viewing position acquired by the position information acquisition unit and the installation positions of the respective loudspeakers. Since one of the filters is selected, the processing speed is high, and the localization of video and audio can be matched early. For example, the position of the virtual sound source can be set according to the user's selection by selecting one of the filters triggered by the user's selection operation.

  (5) The program of the present invention is a program for controlling the operation of a multimedia device having a display and a plurality of loudspeakers, and a viewing position where a user views multimedia data and an installation position of each loudspeaker. , A first head-related transfer function indicating a sound transfer characteristic between the viewing position and the installation position of each loudspeaker, and sound adapted to the viewing position and an image displayed on the display Is output from each loudspeaker using at least one filter created based on the second head-related transfer function indicating the sound transfer characteristic between the virtual loudspeaker and the installation position of the virtual loudspeaker. Input signal so that the sound is adapted to the image displayed on the display at the viewing position. A process of positive to, that to execute the series of processing on a computer, characterized.

  In this way, the input signal is corrected so that the sound output from each loudspeaker becomes sound adapted to the image displayed on the display at the user's viewing position, so that the user can view the video content. Accordingly, the installation position of each loudspeaker is virtually moved to generate a virtual sound source, and the localization of video and audio can be matched. In addition, since the input signal is corrected using a filter created based on the first and second head related transfer functions, the virtual loudspeaker is placed at a position in a space different from the installation position of each loudspeaker as a virtual sound source. It is possible to provide sound that is generated and adapted to the image displayed on the display at the viewing position of the user from the virtual sound source. This makes it possible to match the localization of video and audio according to the position where the user views the video content.

  According to the present invention, the input signal is corrected so that the sound output from each loudspeaker is adapted to the image displayed on the display at the user's viewing position, so the user views the video content. Depending on the position, the installation position of each loudspeaker is virtually moved to generate a virtual sound source, and the localization of video and audio can be matched.

It is a figure which shows the concept of the multimedia apparatus which concerns on this embodiment. It is a figure which shows the outline | summary of two types of head-related transfer functions. It is a block diagram which shows schematic structure of the multimedia apparatus which concerns on this embodiment. It is a flowchart which shows the operation | movement which virtually moves the installation position of each loudspeaker to a desired position.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings for understanding of the present invention. The following embodiments are examples embodying the present invention, and are not intended to limit the technical scope of the present invention.

  The multimedia device according to the present embodiment has a display and two or more loudspeakers, and in order to match the localization of video and audio, the positional relationship between the display and the two or more loudspeakers, gyro sensor information, and Based on the video content information, virtual sound source reproduction technology is used to match the localization of video and audio.

  FIG. 1 is a diagram showing a concept of a multimedia apparatus according to the present embodiment. FIG. 2 is a diagram showing an outline of two types of head-related transfer functions. That is, in this embodiment, in order to match the localization of video and audio, the position of two or more loudspeakers is virtually moved using a head-related transfer function, and the position is different from the installation position of the loudspeakers. To output sound from the virtual loudspeaker.

First, a case where there are two loudspeakers will be described. In FIG. 1 and FIG. 2, the head-related transfer functions representing the transfer characteristics of the acoustic path from the loudspeaker 1 to the user's left ear and right ear are denoted by H _1L (f) and H _1R (f), respectively. The head-related transfer functions from the loudspeaker 2 to the user's left ear and right ear are _{denoted by} H _2L (f) and H _2R (f), respectively. Here, f is a frequency index, and t is a time index.

On the other hand, in order to virtually move the installation position of two or more loudspeakers to a desired position, the head-related transfer function representing the transfer characteristic of the acoustic path from the position of the movement destination to the user's left ear and right ear _Are V _1L (f) and V _1R (f), respectively. At this time, as shown in Expression (1), if the acoustic signals in the user's left and right ears are equal, a virtual sound source can be generated at a desired position. Here, W _1L (f) and W _2L (f) are virtual sound source reproduction transfer functions that perform convolution with the left channel sound source signal s _L (f, t) to generate a virtual sound source at a desired position. is there. The frequency index f is omitted.

ここで、式（１）のＨに関する２×２行列をＡ、Ｗに関する２次元ベクトルをｘ、Ｖに関する２次元ベクトルをｂとすれば、式（１）は、式（２）のように表すことができる。

Here, if the 2 × 2 matrix relating to H in Equation (1) is A, the two-dimensional vector relating to W is x, and the two-dimensional vector relating to V is b, Equation (1) can be expressed as Equation (2). be able to.

さらに、ｘは、式（３）のように求められる。

Further, x is obtained as shown in Equation (3).

求められたｘ、すなわちＷ_1L（ｆ）およびＷ_2L（ｆ）は、まず、左チャネルの音源信号ｓ_L（ｆ，ｔ）と畳み込み、左チャネルの音源信号ｓ_L（ｆ，ｔ）に対するラウドスピーカ１および２から再生すべき音響信号成分を得る。さらに、右チャネルの音源信号ｓ_R（ｆ，ｔ）は、同様に、Ｗ_1R（ｆ）およびＷ_2R（ｆ）を求め、右チャネルの音源信号ｓ_R（ｆ，ｔ）に対するラウドスピーカ１および２から再生すべき音響信号成分を得る。最後に、ラウドスピーカ１および２のそれぞれの音響信号成分を加算し、ラウドスピーカ１および２から再生することで、２つのラウドスピーカの位置を仮想的に、所望の位置に移動させることが可能となる。

The obtained x, i.e. W _1L (f) and W _2L (f), first, left channel sound source signal s _L (f, t) and the convolution, loud for the left channel sound source signal s _L (f, t) Sound signal components to be reproduced are obtained from the speakers 1 and 2. Further, the right channel sound source signal s _R (f, t) is similarly obtained as W _1R (f) and W _2R (f), and the loudspeaker 1 for the right channel sound source signal s _R (f, t) and 2 obtains an acoustic signal component to be reproduced. Finally, by adding the respective acoustic signal components of the loudspeakers 1 and 2 and reproducing them from the loudspeakers 1 and 2, it is possible to virtually move the positions of the two loudspeakers to desired positions. Become.

  When there are three or more loudspeakers, equation (1) can be expressed as equation (4).

ここで、Ｎはラウドスピーカの数である。ここで、式（４）のＨに関する２×Ｎ行列をＡ、Ｗに関するＮ次元ベクトルをｘ、Ｖに関する２次元ベクトルをｂとすれば、式（２）より、式（３）は、式（５）のように示すことができる。

Here, N is the number of loudspeakers. Here, if the 2 × N matrix related to H in Equation (4) is A, the N-dimensional vector related to W is x, and the two-dimensional vector related to V is b, Equation (3) can be expressed by Equation (3) from Equation (2). It can be shown as 5).

ここで、添字の＋は、Ｍｏｏｒｅ−Ｐｅｎｒｏｓｅの擬似逆行列である。

Here, the subscript + is a Moore-Penrose pseudo inverse matrix.

  The head-related transfer function is measured at a plurality of positions and stored in advance. The position information of the display and the two or more loudspeakers built in the information device is assumed to be known and stored in advance. Information about the rotation direction of the display built in the information device can be acquired from the information device through a predefined application programming interface.

  FIG. 3 is a block diagram showing a schematic configuration of the multimedia apparatus according to the present embodiment. In FIG. 3, the input signal is a stereo or monaural audio signal. The position information acquisition unit 1 provides the virtual sound source generation unit 3 with the relative position between the actual loudspeaker installation position and the virtual loudspeaker user who virtually rings. These pieces of position information may be fixed after a user's viewing posture is defined. The position of the loud speaker that rings virtually may be freely changed by the user. Further, it may be automatically measured based on video acquired from the camera 9 and information acquired using the gyro sensor 11.

  Based on the position information of the loudspeakers provided from the position information acquisition unit 1, the virtual sound source generation unit 3 is based on the position information of these loudspeakers, the head-related transfer function measured in advance, and the characteristics of the existing speakers. A filter for virtually moving the position of the loudspeaker to a desired position is determined. Specifically, one of the filters is selected from the filter set 5 made up of a plurality of filters prepared in advance, is subjected to a convolution calculation process with the input signal, and is a signal (acoustic signal) to be output from an existing loudspeaker. ) Is generated.

  A filter for virtually moving the loudspeaker position to a desired position is based on the loudspeaker installation position, the head-related transfer function measured in advance, and the characteristics of the existing loudspeaker. It may be derived each time according to the change.

  FIG. 4 is a flowchart showing an operation of virtually moving the installation position of each loudspeaker to a desired position. First, position information of the display and two (or more) loudspeakers is read (step S1). Next, the rotation direction information of the display is taken in (step S2). For example, this is the case when the user has rotated a display that has been used in a portrait orientation so that the display is in landscape orientation in order to view video content. Next, the position of two (or more) loudspeakers is virtually moved to a desired position (step S3), and the process proceeds to step S2.

  Thus, according to the present embodiment, the sound output from each loudspeaker is sound adapted to the image displayed on the display at the viewing position of the user. Thereby, according to the position where the user views the video content, the installation position of each loudspeaker is virtually moved to generate a virtual sound source, and the localization of video and audio can be matched.

DESCRIPTION OF SYMBOLS 1 Position information acquisition part 3 Virtual sound source generation part 5 Filter set 9 Camera 11 Gyro sensor

Claims (5)

A multimedia device in which a rotatable display and a plurality of loudspeakers are integrated ,
A position information acquisition unit for acquiring a viewing position where a user views multimedia data and a position of the display and each loudspeaker;
A virtual sound source generation unit that corrects an input signal so that sound output from each of the loudspeakers is sound adapted to an image displayed on the display at the viewing position ;
When the display is rotated, the virtual sound source generator generates a first head-related transfer function indicating a sound transfer characteristic between the viewing position after the display rotation and the position of each loudspeaker, and the viewing Created based on the second head-related transfer function indicating the sound transfer characteristics between the position and the position of the virtual loudspeaker that outputs sound adapted to the image displayed on the rotated display to the viewing position multimedia device, characterized that you correct the input signal using at least one filter. The multimedia apparatus according to claim 1, wherein the position information acquisition unit includes a gyro sensor, and acquires the positions of the display and the loudspeakers based on information acquired using the gyro sensor. . The virtual sound source generation unit includes a plurality of first head-related transfer functions and a plurality of second head-related transfer functions, and the viewing position of the user acquired by the position information acquisition unit, the display, and each of the loudspeakers depending on the position, to create the filter, the multimedia device according to claim 1, wherein the correcting the input signal using the filter created. The virtual sound source generating unit includes a plurality of viewing positions or a plurality of displays and a plurality of filters created in advance according to the positions of the loudspeakers, and the viewing position acquired by the position information acquiring unit and the display wherein according to the position of each loudspeaker, select one of the filters, multi-media device of claim 1, wherein the correcting the input signal using the selected filter. A program for controlling the operation of a multimedia device in which a rotatable display and a plurality of loudspeakers are integrated ,
Processing for obtaining a viewing position where a user views multimedia data and a position of the display and each loudspeaker;
When the display is rotated , a first head-related transfer function indicating a sound transfer characteristic between the viewing position after rotation of the display and the position of each loudspeaker, and the viewing position and the display after rotation Using at least one filter created based on the second head-related transfer function indicating the sound transfer characteristic between the position of the virtual loudspeaker that outputs sound adapted to the image displayed on the viewing position to the viewing position , Causing the computer to execute a series of processes of correcting the input signal so that the sound output from each of the loudspeakers becomes sound adapted to the image displayed on the display at the viewing position. A program characterized by

Images