JP7316974B2 - SOUND COLLECTION DEVICE, SYSTEM, PROGRAM AND METHOD THAT TRANSMITS ENVIRONMENTAL SOUND IN WHICH SPECIAL SOUND SIGNAL IS SUPPRESSED - Google Patents

Description

The present invention relates to technology of a telepresence system via a network.

In the case of a teleconferencing system, the status of the other party cannot be shared unless the network is voluntarily connected. Therefore, even among company employees, for example, employees staying at home or in a shared office may be placed in an isolated work environment and feel alienated (see, for example, Non-Patent Document 1).

In recent years, even within the same company, telepresence systems have come to be used that constantly transmit video and audio between a plurality of bases. It's a videoconferencing system, but it's not only connected during meetings, it's always connected during working hours. According to this system, it is possible to provide an environment in which even employees staying at different remote bases share information about each other's situation and work as if they were working in the same place. A telepresence system can be easily connected via a network not only between domestic and overseas company bases, but also between a company and a home or shared office.
In addition, it can be used not only in companies but also between parents and children who live remotely.

Conventionally, for example, according to Skype (registered trademark), a typical videophone, the status of "presence/absence" is displayed on the display of the other party's terminal in order to share the movement of members at remote sites. can be done.
In addition, there is also a technique for conveying the situation of a member at a remote site by synthesized sound in order to know the situation of a remote site member without looking at the display during work (see, for example, Non-Patent Document 4).
Furthermore, there is also a technique for sharing the situation with each other by constantly transmitting environmental sounds and images around the other party via a telepresence system (for example, see Non-Patent Documents 2 and 3).
Furthermore, there is also a technology of an environmental sound recognition device in which various environmental sounds are mutually recognized between remote bases (see, for example, Patent Document 1).

Japanese Patent No. 6085538

Ministry of Internal Affairs and Communications, "Survey Research Report on Telework Trends and Productivity, Ministry of Internal Affairs and Communications International Strategy Bureau (2010)", [online], [searched March 10, 2020], Internet <URL: https ://www.soumu.go.jp/johotsusintokei/linkdata/h22_06_houkoku.pdf＞ Telepresence: Integrating shared task and person spaces, W Buxton - Proceedings of graphics interface, 1992, [online], [searched on March 10, 2020], Internet < URL: https://www.billbuxton.com/TelepShrdSpce .pdf＞ Proceedings of the Annual Meeting of the Japan Ergonomics Society 406-407, 2009: Always-connected voice conference system for telework HRI 2018: Fribo: A Social Networking Robot for Increasing Social Connectedness through Sharing Daily Home Activities from Living Noise Data., [online], [searched on March 10, 2020], Internet <URL: https://yonsei. pure.elsevier.com/en/publications/fribo-a-social-networking-robot-for-increasing-social-connectedne > Kazuho Ono, "Multichannel Audio", [online], [Searched on March 10, 2020], Internet <URL: https://www.jstage.jst.go.jp/article/itej/68/8 /68_604/_pdf/-char/en＞

However, according to existing telepresence systems and video conference systems, all voices generated at each site are transmitted to the other party. For this reason, the other party may feel noisy because of the harsh noise being transmitted. In addition, there are cases in which voice, which poses a problem of privacy, is transmitted.

Therefore, an object of the present invention is to provide a sound collecting device, a system, a program, and a method for transmitting environmental sound by suppressing a specific sound signal that does not need to be transmitted to the other party among sounds generated in a base. do.

According to the present invention, in a sound collecting device that transmits an environmental sound signal collected by a microphone to a reproducing device that reproduces the sound using a speaker,
a first acoustic database storing acoustic objects associated with acoustic tags;
a removal acoustic tag table for registering acoustic tags to be removed;
an acoustic object detection engine that uses the first acoustic database to detect one or more acoustic objects inherent in the ambient sound signal and identifies acoustic tags for the acoustic objects;
acoustic object suppressing means for suppressing, from an environmental sound signal, an acoustic signal portion of an acoustic object associated with the acoustic tag when each identified acoustic tag is registered in a removal acoustic tag table;
environmental sound transmitting means for transmitting the environmental sound signal from which the acoustic object has been removed to the reproducing device;
and acoustic tag transmitting means for transmitting the acoustic tag associated with the removed acoustic object to the reproducing device.

According to another embodiment of the sound collecting device of the present invention,
connected to environmental sensors,
Environmental sensors are tied to acoustic tags,
It is also preferable that, when receiving a predetermined signal from the environment sensor, the acoustic object suppressing means removes, from the environmental sound signal, an acoustic signal portion of the acoustic object linked to the acoustic tag of the environmental sensor.

According to another embodiment of the sound collecting device of the present invention,
connected to the camera,
an image database for accumulating image objects with associated acoustic tags;
an image object detection engine that uses the image database to detect one or more image objects inherent in the video captured by the camera and identifies acoustic tags for the image objects;
It is also preferable that the acoustic object suppressing means remove, from the environmental sound signal, the acoustic signal portion of the acoustic object linked to the acoustic tag identified by the image object detection engine.

According to the present invention, in a system having the above-described sound collecting device and a reproducing device for reproducing an environmental sound signal received from the sound collecting device,
The playback device
a second acoustic database that accumulates acoustic objects associated with acoustic tags;
an acoustic object mixing means for mixing an acoustic object associated with the acoustic tag with an environmental sound signal using a second acoustic database;
and reproduces an environmental sound signal mixed with an acoustic object by a speaker.

According to another embodiment of the system of the invention,
the acoustic tags and acoustic objects stored in the second acoustic database of the playback device are part or all of the acoustic tags and acoustic objects stored in the first acoustic database of the sound collection device;
Even if the acoustic tags stored in the second acoustic database of the playback device and the acoustic tags stored in the first acoustic database of the sound collection device are the same, they may be acoustic objects based on different acoustic signals. preferable.

According to another embodiment of the system of the invention,
A plurality of sound collecting devices and one playback device are connected via a network,
It is also preferable that each sound collecting device is arranged at a different base, and the reproducing device simultaneously reproduces the environmental sound signals of the different bases.

According to another embodiment of the system of the invention,
The playback device may control acoustic signals output from the plurality of speakers so that environmental sound signals received from each of the plurality of sound pickup devices are played back from different arrival directions for each sound pickup device. preferable.

According to another embodiment of the system of the invention,
The sound collecting device transmits the video captured by the camera to the playback device,
The reproducing device divides each video received by each sound collecting device and reproduces them on a display,
It is also preferable that the reproducing device reproduces the environmental sound signal so that the environmental sound of the sound collecting device comes from the position of the display on which the image of each sound collecting device is projected.

According to the present invention, in a sound collecting device that transmits an environmental sound signal collected by a microphone to a reproducing device that reproduces the sound from a speaker,
a first acoustic database storing acoustic objects associated with acoustic tags;
a removal acoustic tag table for registering acoustic tags to be removed;
a second acoustic database that stores different acoustic objects even if they are the same acoustic tags as those in the first acoustic database;
an acoustic object detection engine that uses the first acoustic database to detect one or more acoustic objects inherent in the ambient sound signal and identifies acoustic tags for the acoustic objects;
acoustic object suppressing means for suppressing, from an environmental sound signal, an acoustic signal portion of an acoustic object associated with the acoustic tag when each identified acoustic tag is registered in a removal acoustic tag table;
an acoustic object mixing means for mixing the acoustic object linked to the acoustic tag with the environmental sound signal using the second acoustic database; and an environmental sound transmitting means for transmitting the environmental sound signal mixed with the acoustic object to the reproducing device;
characterized by having

According to the present invention, in a program that causes a computer installed in a sound pickup device that transmits an environmental sound signal picked up by a microphone to a playback device that plays back with a speaker,
a first acoustic database storing acoustic objects associated with acoustic tags;
a removal acoustic tag table for registering acoustic tags to be removed;
an acoustic object detection engine that uses the first acoustic database to detect one or more acoustic objects inherent in the ambient sound signal and identifies acoustic tags for the acoustic objects;
acoustic object suppressing means for suppressing, from an environmental sound signal, an acoustic signal portion of an acoustic object associated with the acoustic tag when each identified acoustic tag is registered in a removal acoustic tag table;
environmental sound transmitting means for transmitting the environmental sound signal from which the acoustic object has been removed to the reproducing device;
It is characterized by causing the computer to function as acoustic tag transmitting means for transmitting the acoustic tag associated with the removed acoustic object to the playback device.

According to the present invention, in a program for causing a computer installed in a sound collecting device that transmits an environmental sound signal collected by a microphone to a reproducing device that reproduces the sound from a speaker,
a first acoustic database storing acoustic objects associated with acoustic tags;
a removal acoustic tag table for registering acoustic tags to be removed;
a second acoustic database that stores different acoustic objects even if they are the same acoustic tags as those in the first acoustic database;
an acoustic object detection engine that uses the first acoustic database to detect one or more acoustic objects inherent in the ambient sound signal and identifies acoustic tags for the acoustic objects;
acoustic object suppressing means for suppressing, from an environmental sound signal, an acoustic signal portion of an acoustic object associated with the acoustic tag when each identified acoustic tag is registered in a removal acoustic tag table;
an acoustic object mixing means for mixing the acoustic object linked to the acoustic tag with the environmental sound signal using the second acoustic database; and an environmental sound transmitting means for transmitting the environmental sound signal mixed with the acoustic object to the reproducing device;
to make the computer function.

According to the present invention, in a sound collection and reproduction method of a sound collection device for transmitting an environmental sound signal collected by a microphone to a reproduction device for reproduction by a speaker,
The sound collecting device is
a first acoustic database storing acoustic objects associated with acoustic tags;
a removal acoustic tag table for registering acoustic tags to be removed;
a first step of detecting one or more acoustic objects inherent in the ambient sound signal using a first acoustic database and identifying acoustic tags for the acoustic objects;
a second step of removing, from the environmental sound signal, the acoustic signal portion of the acoustic object associated with the acoustic tag, if each identified acoustic tag is registered in the removed acoustic tag table;
and a third step of transmitting the environmental sound signal from which the acoustic object has been removed to the reproducing device, and transmitting the acoustic tag associated with the removed acoustic object to the reproducing device.

According to another embodiment of the sound collection and reproduction method of the present invention,
The playback device
having a second acoustic database that stores acoustic objects associated with acoustic tags;
a fourth step of mixing the acoustic object associated with the acoustic tag with the environmental sound signal using the second acoustic database;
Playing the ambient sound signal mixed with the acoustic object by means of a loudspeaker is also preferably performed.

According to the present invention, in a sound collection and reproduction method of a sound collection device for transmitting an environmental sound signal collected by a microphone to a reproduction device for reproduction from a speaker,
The sound collecting device is
a first acoustic database storing acoustic objects associated with acoustic tags;
a removal acoustic tag table for registering acoustic tags to be removed;
a second acoustic database storing different acoustic objects even if they are the same acoustic tags as the first acoustic database;
a first step of detecting one or more acoustic objects inherent in the ambient sound signal using a first acoustic database and identifying acoustic tags for the acoustic objects;
a second step of removing, from the environmental sound signal, the acoustic signal portion of the acoustic object associated with the acoustic tag, if each identified acoustic tag is registered in the removed acoustic tag table;
a third step of mixing the acoustic object associated with the acoustic tag with the environmental sound signal using the second acoustic database;
and a fourth step of transmitting the environmental sound signal mixed with the sound object to the reproducing device.

According to the sound collecting device, system, program, and method of the present invention, it is possible to transmit environmental sound by suppressing a specific acoustic signal, which does not need to be transmitted to the other party, among voices generated in the base. It is possible to suppress harsh noises and eliminate privacy-related voices, while at the same time telling the other party what kind of environmental sounds have been eliminated. As a result, even members staying at different remote bases can share their situations in comfortable environmental sounds.

1 is a functional configuration diagram of a sound collecting device and a reproducing device according to the present invention; FIG. FIG. 4 is an explanatory diagram of detecting an acoustic object; FIG. 4 is an explanatory diagram of suppressing an acoustic object; FIG. 4 is an explanatory diagram of an environmental sound transmission unit and an acoustic tag transmission unit; FIG. 4 is an explanatory diagram of mixing acoustic objects; 3 is a functional configuration diagram of a sound collection device connected to an environment sensor; FIG. FIG. 2 is a functional configuration diagram of a sound collection device that suppresses acoustic objects from an image captured by a camera; 2 is a functional configuration diagram of a sound collecting device having an acoustic object suppression unit and an acoustic object mixing unit; FIG. 2 is a functional configuration diagram of a playback device that receives environmental sounds from a plurality of sound collecting devices; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a functional configuration diagram of a sound collecting device and a reproducing device according to the present invention.

In the system of the present invention, a sound collecting device 1 and a reproducing device 2 are connected via a network.
According to FIG. 1 , the sound collecting device 1 suppresses a specific acoustic signal from the environmental sound signal picked up by the microphone 101 and transmits the environmental sound signal to the reproducing device 2 . At the same time, the video captured by the camera 102 is also transmitted to the playback device 2 .
The reproducing device 2 reproduces the environmental sound received from the sound collecting device 1 through the speaker 201 . In addition, the playback device 2 can recognize what kind of acoustic signal has been suppressed in the received environmental sound, and can clearly show the suppressed acoustic signal to the user. Furthermore, it is also possible to mix the received environmental sound with another acoustic signal different from the suppressed acoustic signal and reproduce the new environmental sound from the speaker 201 .

<Sound collection device 1>
1, the sound collecting device 1 includes a first acoustic database 11, a removed acoustic tag table 12, an acoustic object detection engine 13, an acoustic object suppressor 14, an environmental sound transmitter 15, an environmental tag It has a transmission unit 16 and a video transmission unit 17 . These functional configuration units can be realized by executing a program that causes a computer installed in the device to function. In addition, the processing flow of these functional configuration units can also be understood as a sound collection and transmission method.

FIG. 2 is an explanatory diagram of detecting an acoustic object.

[First acoustic database 11]
The first acoustic database 11 accumulates acoustic objects (sound objects) to which acoustic tags are linked.
Acoustic tag<->acoustic object An “acoustic tag” is an identifier for specifying an acoustic object.
The "acoustic object" is not limited to the acoustic signal itself, but may be a standard pattern of acoustic features such as a time-series frequency spectrum. As for the audio signal, for example, an audio signal conforming to a standard such as ITU-R Recommendation BS.2051 "Advanced Audio System in Program Production" may be used.

[Removal Acoustic Tag Table 12]
The removal acoustic tag table 12 registers acoustic tags to be removed.
For example, when noise such as the sound of a printer or door opening/closing is to be removed, acoustic tags associated with those acoustic objects are registered. Also, for example, when it is desired to remove the privacy of a person's voice, acoustic tags associated with those acoustic objects are registered.

[Acoustic object detection engine 13]
The acoustic object detection engine 13 uses the first acoustic database 11 to detect one or more acoustic objects inherent in the environmental sound signal, and identifies acoustic tags of the acoustic objects. The specified acoustic tag is output to the acoustic object suppression unit 14 .

The acoustic object detection engine 13 extracts mel-frequency cepstrum coefficients (MFCC) as features and identifies acoustic objects using a neural network based on deep learning (see Non-Patent Documents 3 and 4, for example). It stacks hidden layers pretrained by a constrained Boltzmann machine (RBM)-based autoencoder to build a multi-layered hierarchical network and adds a discriminative network using the output of the final layer to obtain the overall We detect acoustic tags by supervised learning as follows.

According to FIG. 2, an environmental sound signal picked up by the microphone 101 is input to the acoustic object detection engine 13 . For example, various sounds such as those described below are mixed in the environmental sound signal.
"Boonga Shacky"
"Mr. Yamamoto, good morning."
"Gee"
"I met Ito-san yesterday."
"ping pong"
"Kata kata kata kata"
Then, the acoustic object detection engine 13 detects acoustic objects and acoustic tags, for example, as follows.
Acoustic tag 101 (chime sound)
Acoustic tag 167 (printer sound)
Acoustic tag 239 (keyboard tap sound)
Acoustic tag 143 (Person's name "Mr. Yamamoto")
Acoustic tag 52 (Person's name "Mr. Ito")

[Sound object suppression unit 14]
When each identified acoustic tag is registered in the removal acoustic tag table 12, the acoustic object suppression unit 14 suppresses the acoustic object associated with the acoustic tag from the environmental sound signal.

FIG. 3 is an explanatory diagram of suppressing an acoustic object.
According to FIG. 3, it is assumed that the following acoustic tags are registered in the removed acoustic tag table 12, for example.
Acoustic tag 101 (chime sound)
Acoustic tag 167 (printer sound)
Acoustic tag 52 (Person's name "Mr. Ito")
In this case, the acoustic object suppression unit 14 suppresses the acoustic objects associated with these acoustic tags from the environmental sound signal in terms of frequency.

In another embodiment, the acoustic object based on the acoustic tag is not only suppressed, but only when the acoustic level of the acoustic object based on the acoustic tag exceeds a predetermined threshold value. There may be.
For example, according to the above-described embodiment, the acoustic objects based on the acoustic tags assigned to the personal names "Mr. Yamamoto" and "Mr. Ito" are suppressed. can be anything.

FIG. 4 is an explanatory diagram of the environmental sound transmission unit and the acoustic tag transmission unit.

[Environmental sound transmission unit 15]
The environmental sound transmission unit 15 transmits an environmental sound signal in which a specific acoustic object is suppressed to the playback device 2 . As a result, the reproduction device 2 can reproduce the environmental sound from which the specific noise and privacy sound have been removed.

According to FIG. 4, for example, the following environmental sounds are transmitted.
"Mr. Yamamoto, good morning."
"Gee"
"I met with... yesterday."
"Kata kata kata kata"
In this way, for example, "Boongga shucky" and "Ping-pong" like printer noise and "Mr. Ito" like private voice are suppressed.

[Environmental tag transmission unit 16]
The environment tag transmission unit 16 transmits the acoustic tag associated with the suppressed acoustic object to the playback device 2 . As a result, the playback device 2 recognizes that the acoustic object associated with the acoustic tag is suppressed in the received environmental sound signal.

According to FIG. 4, for example, the following acoustic tags are transmitted.
Acoustic tag 101 (chime sound)
Acoustic tag 167 (printer sound)
Acoustic tag 52 (Person's name "Mr. Ito")
The time at which the acoustic object based on the acoustic tag was inserted is also transmitted in synchronization with the environmental sound. This is for specifying the insertion time of the sound object when reproducing the environmental sound.

[Video transmission unit 17]
The video transmission unit 17 transmits the video captured by the camera 102 to the playback device 2 . It is preferable to share and recognize images with the base of the other party.

<Playback Device 2>
According to FIG. 1, the playback device 2 comprises a second acoustic database 21 , an acoustic object mixer 22 and a video playback unit 23 . These functional configuration units can be realized by executing a program that causes a computer installed in the device to function. In addition, the processing flow of these functional components can also be understood as a reception/playback method.

[Second Acoustic Database 21]
The second acoustic database 21 accumulates acoustic objects linked with acoustic tags.

The acoustic tags and acoustic objects stored in the second acoustic database 21 may be part or all of the acoustic tags and acoustic objects stored in the first acoustic database 11 of the sound collection device 1 .
For example, if the acoustic tags and acoustic objects stored in the first acoustic database 11 are not stored in the second acoustic database 21, the environmental sound is reproduced with the acoustic objects suppressed.
On the other hand, when the acoustic tags and acoustic objects stored in the first acoustic database 11 are also stored in the second acoustic database 21, the environmental sounds mixed by the acoustic objects in the second acoustic database 21 are is played. That is, when the acoustic tags stored in the second acoustic database 21 and the acoustic tags stored in the first acoustic database 11 of the sound collection device 1 are acoustic objects based on different acoustic signals, the second Environmental sounds converted by the acoustic objects in the acoustic database 21 are reproduced.

[Sound object mixer 22]
The acoustic object mixing unit 22 uses the second acoustic database 21 to mix the acoustic object associated with the acoustic tag with the environmental sound signal. The mixed environmental sound signal is output to speaker 201 .

FIG. 5 is an explanatory diagram of mixing acoustic objects.

According to FIG. 5, the acoustic object mixer 22 receives the ambient sound signal and the suppressed acoustic tags from the sound pickup device 1 . Then, using the second acoustic database 21, the acoustic object is searched using the received acoustic tag as a key. According to FIG. 5, retrieval is performed as follows.
Acoustic tag 101 <->"lean"
Acoustic tag 52 <->"Pipopa"
The acoustic tag 101, which was "ping-pong" in the first acoustic database 11, is pseudo-voice "lean" in the second acoustic database 21. FIG.
The acoustic tag 52, which was "Mr. Ito" in the first acoustic database 11, is a pseudo-speech "pipopa" in the second acoustic database 21. FIG.
Here, acoustic tag 167 is not searched in second acoustic database 21 . This means that the "boomggashucky" in the first acoustic database remains suppressed. Noise that is particularly annoying can be simply silenced (removed) from the environmental sound.

According to FIG. 5, for example, environmental sounds mixed with the following sound objects are reproduced.
"Mr. Yamamoto, good morning."
"Gee"
"I met Pipopa yesterday~"
“Lean”
"Kata kata kata kata"

[Video playback unit 23]
The video reproducing unit 23 outputs the video received from the sound collecting device 1 to the display 202 . It is preferable to share and recognize images with the base of the other party.

FIG. 6 is a functional configuration diagram of a sound collecting device connected to an environment sensor.

According to FIG. 6, the sound collecting device 1 is connected to an environmental sensor 18 and receives an ON/OFF signal. The environmental sensor may be, for example, a door open/close sensor. An environmental sensor is tied to any one acoustic tag. An ON/OFF signal of the environment sensor is input to the acoustic object suppression unit 14 .

When a predetermined signal is received from the environment sensor 18, the acoustic object suppression unit 14 suppresses the acoustic signal portion of the acoustic object linked to the acoustic tag of the environmental sensor from the environmental sound signal. This makes it possible to suppress noise such as door opening/closing sound from environmental sounds.

FIG. 7 is a functional configuration diagram of a sound pickup device that suppresses an acoustic object from an image captured by a camera.

According to FIG. 7, the sound collecting device 1 receives an image captured by a camera.
Moreover, according to FIG. 7, the sound collecting device 1 further has an image database 190 and an image object detection engine 191 .

[Image database 190]
The image database 190 accumulates image objects linked with acoustic tags.

[Image object detection engine 191]
The image object detection engine 191 uses the image database 190 to detect one or more image objects inherent in the video captured by the camera and identify the acoustic tags of the image objects. The specified acoustic tag is output to the acoustic object suppression unit 14 .

Specifically, the image object detection engine 191 encloses an object (image object) in a frame (bounding box) from the input image or video, and identifies the type (category) of the object. This may be, for example, an SSD (Single Shot Multibox Detector). The SSD divides the image into grids and detects the bounding box at that position from the fit of a plurality of fixed bounding boxes for each grid. One image object fits in that bounding box.
The image object detection engine 191 may be a neural network such as a CNN (Convolutional Neural Network) based on RGB recognition, such as YOLO (You Only Look Once) (registered trademark). .

When the acoustic tag is received from the image object detection engine 191, the acoustic object suppression unit 14 suppresses the acoustic signal portion of the acoustic object linked to the acoustic tag of the environmental sensor from the environmental sound signal. As a result, for example, when the opening and closing of the door is reflected in the video, noise such as the opening and closing sound can be suppressed from the environmental sound.

FIG. 8 is a functional configuration diagram of a sound collecting device having an acoustic object suppressing section and an acoustic object mixing section.

According to FIG. 8, the sound collecting device 1 and the reproducing device 2 in FIG. 1 are combined. In this case, in the sound collection device 1 , the acoustic object mixing unit 22 mixes the environmental sound in which the specific acoustic object is suppressed by the acoustic object suppressing unit 14 with another specific acoustic object. The environmental sound mixed by the reproducing device 2 in the case of FIG. 1 is transmitted from the sound collecting device 1 in the case of FIG.

FIG. 9 is a functional configuration diagram of a reproducing device that receives environmental sounds from a plurality of sound collecting devices.

According to FIG. 9, a plurality of sound collecting devices 1 and one reproducing device 2 are connected via a network. Each sound collecting device 1 is placed at a different base, and a reproducing device 2 simultaneously reproduces environmental sound signals from different bases.
The reproducing device 2 may be arranged with a display, and may display each image received by each sound collecting device 1 in a divided manner. Then, the reproducing device 2 controls the acoustic signals output from the plurality of speakers so that the environmental sounds received from each of the plurality of sound collecting devices are reproduced from different arrival directions for each sound collecting device. Therefore, an environmental sound synthesizing unit 24 is provided. A user who is listening to the environmental sound from the playback device 2 hears the environmental sound signal from the display position of the display showing the video of each sound collecting device as if the environmental sound of the sound collecting device is coming.
For this purpose, multi-channel acoustic technology is used (see, for example, Non-Patent Document 5). According to this technique, a plurality of channels with different directions are installed based on the position of each arranged display, and each channel corresponds to the incoming direction of sound on a one-to-one basis. As a result, even if the display is placed at a position where there are no speakers, the user hears the sound as if it were coming from that direction.

As described in detail above, according to the sound collecting device, system, program, and method of the present invention, among sounds generated within a site, environmental sounds in which specific sound signals that do not need to be transmitted to the other party are suppressed can be sent. It is possible to suppress harsh noises and eliminate privacy-related voices, while at the same time telling the other party what kind of environmental sounds have been eliminated. As a result, even members staying at different remote bases can share their situations in comfortable environmental sounds.

For the various embodiments of the present invention described above, various changes, modifications and omissions within the spirit and scope of the present invention can be easily made by those skilled in the art. The foregoing description is exemplary only and is not intended to be limiting. The invention is to be limited only as limited by the claims and the equivalents thereof.

1 sound collecting device 101 microphone 102 camera 11 first acoustic database 12 removed acoustic tag table 13 acoustic object detection engine 14 acoustic object suppression unit 15 environmental sound transmitter 16 environmental tag transmitter 17 video transmitter 18 environment sensor 190 image database 191 Image object detection engine 2 playback device 201 speaker 202 display 21 second audio database 22 audio object mixing unit 23 video playback unit 24 environmental sound synthesis unit

Claims (14)

In a sound collecting device that transmits an environmental sound signal picked up by a microphone to a reproducing device that reproduces it with a speaker,
a first acoustic database storing acoustic objects associated with acoustic tags;
a removal acoustic tag table for registering acoustic tags to be removed;
an acoustic object detection engine that uses the first acoustic database to detect one or more acoustic objects inherent in the ambient sound signal and identifies acoustic tags for the acoustic objects;
acoustic object suppressing means for suppressing, from an environmental sound signal, an acoustic signal portion of an acoustic object associated with the acoustic tag when each identified acoustic tag is registered in a removal acoustic tag table;
environmental sound transmitting means for transmitting the environmental sound signal from which the acoustic object has been removed to the reproducing device;
and acoustic tag transmitting means for transmitting an acoustic tag associated with the removed acoustic object to a reproducing device. connected to environmental sensors,
Environmental sensors are tied to acoustic tags,
2. The acoustic object suppressing means according to claim 1, wherein, upon receiving a predetermined signal from the environmental sensor, the acoustic signal portion of the acoustic object linked to the acoustic tag of the environmental sensor is removed from the environmental sound signal. sound collection device. connected to the camera,
an image database for accumulating image objects with associated acoustic tags;
an image object detection engine that uses the image database to detect one or more image objects inherent in the video captured by the camera and identifies acoustic tags for the image objects;
3. The sound collecting device according to claim 1, wherein the acoustic object suppression means removes, from the environmental sound signal, an acoustic signal portion of the acoustic object linked to the acoustic tag specified by the image object detection engine. A system comprising the sound collecting device according to any one of claims 1 to 3 and a reproducing device for reproducing an environmental sound signal received from the sound collecting device,
The playback device
a second acoustic database that accumulates acoustic objects associated with acoustic tags;
an acoustic object mixing means for mixing an acoustic object associated with the acoustic tag with an environmental sound signal using a second acoustic database;
and reproducing an ambient sound signal mixed with an acoustic object by a speaker. the acoustic tags and acoustic objects stored in the second acoustic database of the playback device are part or all of the acoustic tags and acoustic objects stored in the first acoustic database of the sound collection device;
Even if the acoustic tags stored in the second acoustic database of the reproducing device and the acoustic tags stored in the first acoustic database of the sound collecting device are the same, they are acoustic objects based on different acoustic signals. 5. A system according to claim 4. A plurality of sound collecting devices and one playback device are connected via a network,
6. The system according to claim 4, wherein each sound collecting device is arranged at a different base, and the reproducing device simultaneously reproduces the environmental sound signals of the different bases. The reproduction device controls acoustic signals output from the plurality of speakers so that environmental sound signals received from each of the plurality of sound collection devices are reproduced from different arrival directions for each sound collection device. 7. A system according to claim 6. The sound collecting device transmits the video captured by the camera to the playback device,
The reproducing device divides each video received by each sound collecting device and reproduces them on a display,
8. The system according to claim 7, wherein the reproducing device reproduces the environmental sound signal so that the environmental sound of the sound collecting device comes from the position of the display on which the image of each sound collecting device is projected. In a sound collecting device that transmits an environmental sound signal picked up by a microphone to a reproducing device that reproduces it from a speaker,
a first acoustic database storing acoustic objects associated with acoustic tags;
a removal acoustic tag table for registering acoustic tags to be removed;
a second acoustic database that stores different acoustic objects even if they are the same acoustic tags as those in the first acoustic database;
an acoustic object detection engine that uses the first acoustic database to detect one or more acoustic objects inherent in the ambient sound signal and identifies acoustic tags for the acoustic objects;
acoustic object suppressing means for suppressing, from an environmental sound signal, an acoustic signal portion of an acoustic object associated with the acoustic tag when each identified acoustic tag is registered in a removal acoustic tag table;
an acoustic object mixing means for mixing the acoustic object linked to the acoustic tag with the environmental sound signal using the second acoustic database; and an environmental sound transmitting means for transmitting the environmental sound signal mixed with the acoustic object to the reproducing device;
A sound collecting device comprising: In a program that causes a computer installed in a sound pickup device that transmits environmental sound signals picked up by a microphone to a playback device that plays back through a speaker,
a first acoustic database storing acoustic objects associated with acoustic tags;
a removal acoustic tag table for registering acoustic tags to be removed;
an acoustic object detection engine that uses the first acoustic database to detect one or more acoustic objects inherent in the ambient sound signal and identifies acoustic tags for the acoustic objects;
acoustic object suppressing means for suppressing, from an environmental sound signal, an acoustic signal portion of an acoustic object associated with the acoustic tag when each identified acoustic tag is registered in a removal acoustic tag table;
environmental sound transmitting means for transmitting the environmental sound signal from which the acoustic object has been removed to the reproducing device;
A program for causing a computer to function as acoustic tag transmission means for transmitting an acoustic tag associated with a removed acoustic object to a playback device. In a program that causes a computer installed in a sound pickup device that transmits environmental sound signals picked up by a microphone to a playback device that reproduces them from a speaker,
a first acoustic database storing acoustic objects associated with acoustic tags;
a removal acoustic tag table for registering acoustic tags to be removed;
a second acoustic database that stores different acoustic objects even if they are the same acoustic tags as those in the first acoustic database;
an acoustic object detection engine that uses the first acoustic database to detect one or more acoustic objects inherent in the ambient sound signal and identifies acoustic tags for the acoustic objects;
acoustic object suppressing means for suppressing, from an environmental sound signal, an acoustic signal portion of an acoustic object associated with the acoustic tag when each identified acoustic tag is registered in a removal acoustic tag table;
an acoustic object mixing means for mixing the acoustic object linked to the acoustic tag with the environmental sound signal using the second acoustic database; and an environmental sound transmitting means for transmitting the environmental sound signal mixed with the acoustic object to the reproducing device;
A program characterized by making a computer function by In a sound pickup and reproduction method for a sound pickup device for transmitting an environmental sound signal picked up by a microphone to a reproduction device for reproduction by a speaker,
The sound collecting device is
a first acoustic database storing acoustic objects associated with acoustic tags;
a removal acoustic tag table for registering acoustic tags to be removed;
a first step of detecting one or more acoustic objects inherent in the ambient sound signal using a first acoustic database and identifying acoustic tags for the acoustic objects;
a second step of removing, from the environmental sound signal, the acoustic signal portion of the acoustic object associated with the acoustic tag, if each identified acoustic tag is registered in the removed acoustic tag table;
A sound pickup and reproduction method, comprising: transmitting an environmental sound signal from which an acoustic object has been removed to a reproducing device; and transmitting an acoustic tag associated with the removed acoustic object to the reproducing device. . The playback device
having a second acoustic database that stores acoustic objects associated with acoustic tags;
a fourth step of mixing the acoustic object associated with the acoustic tag with the environmental sound signal using the second acoustic database;
13. The sound pickup and reproduction method according to claim 12 , further comprising the step of reproducing the environmental sound signal mixed with the sound object by a speaker. In a sound pickup and reproduction method for a sound pickup device for transmitting an environmental sound signal picked up by a microphone from a speaker to a reproduction device for reproduction,
The sound collecting device is
a first acoustic database storing acoustic objects associated with acoustic tags;
a removal acoustic tag table for registering acoustic tags to be removed;
a second acoustic database storing different acoustic objects even if they are the same acoustic tags as the first acoustic database;
a first step of detecting one or more acoustic objects inherent in the ambient sound signal using a first acoustic database and identifying acoustic tags for the acoustic objects;
a second step of removing, from the environmental sound signal, the acoustic signal portion of the acoustic object associated with the acoustic tag, if each identified acoustic tag is registered in the removed acoustic tag table;
a third step of mixing the acoustic object associated with the acoustic tag with the environmental sound signal using the second acoustic database;
and a fourth step of transmitting an environmental sound signal mixed with an acoustic object to a reproducing device.

Images