TWI846005B - Video conference device and method for adjusting camera directions - Google Patents

Abstract

A method for adjusting camera directions which is applicable to a video conference device electrically connecting to a speaker is provided. The video conference device includes au audio processor, a microphone array and a camera. The method includes: receiving, by the audio processor, a far-end audio signal from a far end; the audio processor converting the far-end audio signal into a reference audio signal and sending the reference audio signal to the speaker; playing, by the speaker, a first sound according to the reference audio signal; recording, by the microphone array, a near-end audio signal, wherein the near-end signal includes the first sound and a second sound emitted from a near end; performing, by the audio processor, a filtering operation to generate a filtered audio signal according to the reference audio signal and the near-end signal; computing, by the audio processor, an angular control signal according to the filtered audio signal; and adjusting, by the camera, a camera direction to capture the near-end.

Description

Video conferencing device and method for adjusting the shooting direction of a camera

The present invention relates to video conferencing, and in particular to a video conferencing device capable of tracking a speaker and a method for adjusting the shooting direction of a camera.

As the demand for video conferencing increases, video conferencing devices have been equipped with cameras that have the function of tracking sound sources. When the near-end sound is detected, the camera lens is controlled to face the near-end speaker, so that the conference participants at the far end can receive the near-end speaker's sound and image at the same time.

However, in a meeting, when the speaker at the near end is speaking, the speaker is also playing the voice of the conference participant at the far end. This may cause the camera to shoot towards the speaker instead of the speaker at the near end. However, if the camera tracking is directly paused when the speaker is playing sound, it is still impossible to guarantee that the camera will shoot the subject who is speaking at that time. For example, when there are multiple speakers at the near end, the camera lens may still stay on the previous speaker instead of the person who is currently speaking.

In addition, although algorithms can be used to determine whether there is an echo in the audio input signal received by the near-end microphone, and to control the camera not to track and shoot when an echo is detected, the audio input signal used in this method already contains the sound information played by the speaker, and the signal nature is not conducive to accurately determining the position of the near-end speaker. Therefore, if the echo problem is not solved, the remote user will not be able to see the image of the near-end speaker in real time.

In view of this, the present invention provides a video conference device and a method for adjusting the shooting direction of a camera, so that even if the voice of a remote conference participant in a video conference is played out through a speaker at the near end, the camera can still accurately track the near-end speaker.

A method for adjusting the shooting direction of a camera according to an embodiment of the present invention is applicable to a video conference device, the video conference device includes an audio processor, a microphone array and a camera, the video conference device is electrically connected to a speaker, and the method includes: the audio processor receives a remote audio from a remote end; The audio processor converts the far-end audio into a reference audio and transmits the reference audio to the speaker; the speaker plays a first sound according to the reference audio; the microphone array records a near-end audio, the near-end audio includes the first sound and a second sound from a near-end; the audio processor performs a filtering operation according to the reference audio and the near-end audio to generate a filtered audio; the audio processor calculates an angle control signal according to the filtered audio; and the camera adjusts the shooting direction according to the angle control signal to shoot the near-end.

According to an embodiment of the present invention, a video conference device is electrically connected to a speaker, wherein the speaker is used to play a first sound, and the video conference device includes: an audio processor, which is used to receive a remote audio signal from a remote end, convert the remote audio signal into a reference audio signal, and transmit the reference audio signal to the speaker, wherein the first sound is related to the reference audio signal; the audio processor is based on The reference audio and a near-end audio perform a filtering operation to generate a filtered audio, and an angle control signal is calculated based on the filtered audio; a microphone array is used to record the near-end audio, the near-end audio includes the first sound and a second sound from a near-end; and a camera is electrically connected to the audio processor, and the camera adjusts the shooting direction according to the angle control signal to shoot the near-end.

In summary, the video conference device and the method for adjusting the shooting direction of a camera proposed by the present invention can avoid the camera from mistakenly tracking the speaker that makes the sound when the far end and the near end speak at the same time by removing the speaker signal received by the microphone array, and can also enable the camera to accurately track the speaker at the near end.

The above description of the disclosed content and the following description of the implementation methods are used to demonstrate and explain the spirit and principle of the present invention, and provide a further explanation of the scope of the patent application of the present invention.

The detailed features and characteristics of the present invention are described in detail in the following embodiments, and the content is sufficient for any person skilled in the relevant art to understand the technical content of the present invention and implement it accordingly. According to the content disclosed in this specification, the scope of the patent application and the drawings, any person skilled in the relevant art can easily understand the concept and characteristics of the present invention. The following embodiments are to further illustrate the viewpoints of the present invention, but are not to limit the scope of the present invention by any viewpoint.

FIG1 is a schematic diagram of an application of a video conference device 3 according to an embodiment of the present invention. The video conference device 3 is electrically connected to a processor 1, a speaker 5 and a microphone 7. It should be noted that the arrow directions in FIG1 to FIG3 represent the data transmission directions.

The processor 1 is, for example, a personal computer or a smart phone. The processor 1 can receive remote audio through the network N and transmit the remote audio to the video conference device 3. The remote audio comes from the remote video conference device 3 or a receiver. In one embodiment, the processor 1 is electrically connected to the video conference device 3 via a universal serial bus. In another embodiment, the video conference device 3 has a built-in communication module, so it can directly receive the remote audio from the network N without going through other electronic devices.

The video conference device 3 is electrically connected to the speaker 5 and the microphone 7, and plays the remote audio through the speaker 5, and records the near-end (i.e., local) sound through the microphone 7. In another embodiment, a speakerphone with a built-in speaker 5 and microphone 7 can be used to replace the speaker 5 and microphone 7 shown in FIG. 1, that is, the functions of playing and recording are realized through one device.

FIG. 2 is a block diagram based on FIG. 1 and further illustrates the internal structure of the video conferencing device 3 . The video conferencing device 3 includes an audio processor 32 , a microphone array 34 , a transmission interface 36 , and a camera 38 .

The audio processor 32 is electrically connected to the transmission interface 36, the microphone array 34 and the camera 38. The audio processor 32 is used to receive remote audio from the processor 1 and perform the following operations: convert the remote audio into a reference audio, and transmit the reference audio to the speaker 5 through the transmission interface 36. The transmission interface 36 can use a channel link to transmit a low voltage differential signal (LVDS), but the present invention is not limited to this.

The speaker 5 plays the reference audio to generate the first sound. In other words, the speaker 5 plays the remote audio from the remote end. The microphone array 34 records the near-end audio, which includes the first sound played by the speaker 5 (i.e., the far-end audio mentioned above) and the second sound from the near-end. The "near-end" here is defined as the near-end audio played by the video conference device 3. The second sound is, for example, the voice of a user at the near end. In one embodiment, the microphone array 34 has at least two microphones 341 and 343, each microphone 341 or 343 records a sound component, and the second sound is composed of multiple sound components.

As shown in FIG. 2 , in addition to connecting the speaker 5, the transmission interface is also used to electrically connect the microphone 7 (not the microphones 341 and 343 of the microphone array 34). The microphone 7 is used to record another near-end audio signal, which also includes the first sound played by the speaker 5 and the second sound from the near end. Considering that the near-end speaker may move from a position close to the microphone array 34 to a position close to the speaker 5, or one of the multiple near-end users is originally close to the position where the speaker 5 is located, the microphone array 34 alone may not be able to clearly record the voice of the speaker close to the speaker 5. The other near-end audio signal recorded by the microphone 7 can compensate for the above situation.

When the speaker 5 plays the first sound, the microphone array 34 records the near-end audio, and the microphone 7 records another near-end audio, the audio processor 32 can perform the following operations in real time: perform a filtering operation based on the reference audio and the near-end audio to generate a filtered audio, calculate an angle control signal based on the filtered audio, and transmit the angle control signal to the camera 38.

The camera 38 can adjust the shooting direction according to the angle control signal to shoot the near end. In one embodiment, the camera 38 includes a camera lens and a motor module, and the motor module adjusts the shooting angle of the camera lens according to the angle control signal. In another embodiment, the camera 38 is, for example, a PTZ camera, and its lens can perform operations such as panning, tilting, and zooming in. The angle control signal described in the present invention can correspond to at least one of the above operations.

FIG3 is a block diagram based on FIG2 and further illustrates the internal structure of the audio processor 32. The audio processor 32 includes a conversion circuit 321, an adaptive filter 323, an angle calculation circuit 325, a sound enhancement circuit 327, and a mixer 329.

The conversion circuit 321 is electrically connected to the processor 1, the transmission interface 36, the adaptive filter 323 and the mixer 329. The conversion circuit 321 adopts, for example, the USB Audio Class (UAC) protocol and converts the remote audio into a reference audio accordingly. For example, the remote audio is converted from stereo to mono or/and resampled, such as converting 48kHz stereo to 32kHz mono, and the converted 32kHz mono is used as the reference audio.

The adaptive filter 323 is electrically connected to the conversion circuit 321, the microphone array 34, the sound enhancement circuit 327 and the angle calculation circuit 325. The adaptive filter 323 performs a filtering operation based on the reference audio and the near-end audio to generate a filtered audio. The filtering operation includes: performing a convolution operation based on the reference audio and the coefficient of the adaptive filter 323 to generate a reverse signal, and integrating the near-end audio and the reverse signal to generate the filtered audio. The adaptive filter 323 further updates the coefficient of the adaptive filter 323 based on the filtered audio. In other words, before the speaker 5 is about to play the reference audio, the adaptive filter 323 first obtains the reference audio to be played and the near-end audio recorded by the microphone array 34, thereby estimating the characteristics of the new near-end audio to be recorded by the microphone array 34. The adaptive filter 323 uses a plurality of linear functions, each of which has at least one coefficient. The adaptive filter 323 uses, for example, a Normalized least mean squares filter (NLMS) algorithm to modify the coefficients of these linear functions, thereby reflecting the linear frequency response when the speaker 5 plays the reference audio. The adaptive filter 323 further calculates the reverse signal when the speaker 5 plays the reference audio, and integrates the near-end audio recorded by the microphone array 34 with the reverse signal, thereby filtering out the components belonging to the far-end audio from the near-end audio.

The sound enhancement circuit 327 is electrically connected to the adaptive filter 323 and the mixer 329. The sound enhancement circuit 327 performs at least one of the following operations on the filtered audio: beamforming, noise reduction, residual echo suppression, and automatic gain.

The mixer 329 is electrically connected to the conversion circuit 321, the transmission interface 36 and the sound enhancement circuit 327. After the filtered audio is enhanced by the sound enhancement circuit 327, the mixer 329 performs a mixing operation based on the enhanced filtered audio and another near-end audio to generate a return audio. The return audio will be returned from the conversion circuit 321 to the processor 1, and then returned to the far end by the processor 1 through the network N. The mixer 329 multiplies the output signal of the sound enhancement circuit 327 by a weight, and multiplies the other near-end audio generated by the microphone 7 by another weight, and then adds the two together to achieve the above mixing operation. In one embodiment, the two weights can be adjusted according to the amplitude of the input signal and the other near-end audio. In another embodiment, since human voice belongs to wideband, the input signal and the other near-end audio are multiplied by their respective weights only in their respective designated frequency bands.

The angle calculation circuit 325 is electrically connected to the adaptive filter 323 and the camera 38. The angle calculation circuit 325 uses the Time Difference of Arrival (TDOA) technology to generate an angle control signal based on the filtered audio (the microphone array 34 signal that removes the sound of the speaker 5). Please refer to FIG. 3.

The microphone array 34 includes a plurality of microphones (e.g., microphones 341 and 343), and the filtered audio includes a plurality of filtered audio components, which correspond to the plurality of microphones, respectively. Therefore, the angle calculation circuit 325 calculates a time difference according to two of the filtered audio components to generate the angle control signal. In one embodiment, the microphone array 34 may be configured with at least four microphones, so that the angle calculation circuit 325 generates a three-dimensional angle control signal.

FIG4 is a flow chart of a method for adjusting the tracking direction of a camera 38 according to an embodiment of the present invention. As shown in FIG4, step S1 is that the audio processor 32 receives the remote audio from the processor 1, step S2 is that the conversion circuit 321 in the audio processor 32 converts the remote audio into a reference audio and transmits the reference audio to the speaker 5, step S3 is The speaker 5 plays the first sound according to the reference audio signal; step S4 is for the microphone array 34 to record the near-end audio signal, and the microphone 7 to record another near-end audio signal; step S5 is for the adaptive filter 323 in the audio processor 32 to perform a filtering operation according to the reference audio signal and the near-end audio signal to generate a filtered audio signal; step S6 is for the angle calculation circuit 325 in the audio processor 32 to calculate the angle control signal according to the filtered audio signal; step S7 is for the camera 38 to adjust the shooting direction according to the angle control signal to shoot the near-end.

FIG5 is a detailed flow chart of step S5 of FIG4 : step S51 is the adaptive filter 323 performing a convolution operation based on the reference audio and the coefficient of the adaptive filter 323 to generate a reverse signal; step S52 is the adaptive filter 323 integrating the near-end audio and the reverse signal to generate a filtered audio; step S53 is the adaptive filter 323 updating the coefficient of the adaptive filter 323 based on the filtered audio.

FIG6 is a flow chart of a method for adjusting the tracking direction of a camera 38 according to another embodiment of the present invention, wherein steps S1 to S1 can refer to FIG4. Step S8 is for the sound enhancement circuit 327 in the audio processor 32 to perform enhancement processing according to the filtered audio to generate enhanced audio; step S9 is for the mixer 329 in the audio processor 32 to perform mixing operation according to the enhanced audio and another near-end audio in step S4 to generate a return audio; step S10 is for the conversion circuit 321 in the audio processor 32 to transmit the return audio to the processor 1.

Please note that if the same effect can be achieved, the method does not necessarily have to be performed in the order of the steps in Figures 4 to 6, and other auxiliary steps may also be inserted therein.

In summary, the video conference device and the method for adjusting the shooting direction of a camera proposed by the present invention can not only avoid the camera from mistakenly tracking the speaker that makes the sound when the far-end user and the near-end user speak at the same time by removing the speaker signal received by the microphone array, but also enable the camera to accurately track the near-end speaker. In addition, because the adaptive filter of the present invention has filtered out the components of the far-end audio in the near-end audio, the far-end user can receive zero-echo audio or audio with only a small echo, which greatly improves the communication experience.

Although the present invention is disclosed as above with the aforementioned embodiments, it is not intended to limit the present invention. Any changes and modifications made within the spirit and scope of the present invention are within the scope of patent protection of the present invention. Please refer to the attached patent application for the scope of protection defined by the present invention.

1: Processor 3: Video conferencing device 5: Speaker 7: Microphone 32: Audio processor 34: Microphone array 341, 343: Microphone 36: Transmission interface 321: Conversion circuit 323: Adaptive filter 325: Angle calculation circuit 327: Sound enhancement circuit 329: Mixer N: Network S1~S10, S51~S53: Steps

FIG1 is an application schematic diagram of a video conferencing device according to an embodiment of the present invention; FIG2 is a block diagram of a video conferencing device according to an embodiment of the present invention; FIG3 is a block diagram of an audio processor according to an embodiment of the present invention; FIG4 is a flow chart of a method for adjusting the camera tracking direction according to an embodiment of the present invention; FIG5 is a detailed flow chart of step S5 of FIG4; and FIG6 is a flow chart of a method for adjusting the camera tracking direction according to another embodiment of the present invention.

1: Processor

3: Video conferencing equipment

5: Speaker

7: Microphone

32: Audio processor

34: Microphone array

36: Transmission interface

341, 343: Microphone

N: Network

Claims (10)

A method for adjusting the shooting direction of a camera is applicable to a video conference device, the video conference device includes an audio processor, a microphone array and a camera, the video conference device is electrically connected to a speaker, the method includes: the audio processor receives a remote audio signal from a remote end; the audio processor converts the remote audio signal into a reference audio signal and transmits the reference audio signal to the speaker; the speaker plays a first sound according to the reference audio signal; the microphone array records a near-end audio signal, the near-end audio signal includes the first sound and the near-end audio signal from the remote end; A second sound from a near end; the audio processor performs a filtering operation based on the reference audio and the near end audio to generate a filtered audio, wherein the audio processor includes an adaptive filter that performs an operation based on the reference audio and an adaptive filter coefficient to generate a reverse signal, and integrates the near end audio and the reverse signal to generate the filtered audio; the audio processor further includes an angle calculation circuit that calculates an angle control signal based on the filtered audio; and the camera adjusts the shooting direction based on the angle control signal to shoot the near end. The method of claim 1, wherein the step of the audio processor performing the filtering operation based on the reference audio and the near-end audio to generate the filtered audio includes: the audio processor performs a convolution operation based on the reference audio and the adaptive filter coefficient to generate the reverse signal; the audio processor integrates the near-end audio and the reverse signal to generate the filtered audio; and the audio processor updates the adaptive filter coefficient based on the filtered audio. The method of claim 1, wherein the video conferencing device is further used to electrically connect to another microphone, and the method further includes: the other microphone records another near-end audio, the other near-end audio includes the first sound and the second sound from the near-end; the audio processor performs a mixing operation based on the filtered audio and the other near-end audio to generate a return audio; and the audio processor transmits the return audio to the far-end. As described in claim 3, before the audio processor performs the mixing operation based on the filtered audio and the other near-end audio to generate the return audio, the audio processor further includes performing at least one of the following operations on the filtered audio: beamforming, noise reduction, residual echo suppression, and automatic gain. As described in claim 1, the microphone array includes a plurality of microphones, the filtered audio includes a plurality of filtered audio components respectively corresponding to the microphones, and the audio processor calculates the angle control signal according to the filtered audio, including: the audio processor calculates a time difference according to two of the filtered audio components to generate the angle control signal. A video conference device is electrically connected to a speaker, wherein the speaker is used to play a first sound, and the video conference device includes: an audio processor, which is used to receive a remote audio signal from a remote end, convert the remote audio signal into a reference audio signal, and transmit the reference audio signal to the speaker, wherein the first sound is related to the reference audio signal; the audio processor includes: an adaptive filter, which performs a filtering operation based on the reference audio signal and a near-end audio signal to generate a filtered audio signal, wherein the filtering operation A reverse signal is generated by performing calculations based on the reference audio and an adaptive filter coefficient, and the near-end audio and the reverse signal are integrated to generate the filtered audio; and an angle calculation circuit is used to calculate an angle control signal based on the filtered audio; a microphone array is used to record the near-end audio, and the near-end audio includes the first sound and a second sound from a near-end; and a camera is electrically connected to the audio processor, and the camera adjusts the shooting direction according to the angle control signal to shoot the near-end. The video conferencing device as described in claim 6, wherein the filtering operation is that the audio processor performs a convolution operation based on the reference audio and the adaptive filter coefficient to generate the reverse signal, the audio processor integrates the near-end audio and the reverse signal to generate the filtered audio; and the audio processor updates the adaptive filter coefficient based on the filtered audio. The video conferencing device as described in claim 6 is further used to electrically connect to another microphone, the other microphone is used to record another near-end audio, and the other near-end audio includes the first sound and the second sound from the near-end; the audio processor performs a mixing operation based on the filtered audio and the other near-end audio to generate a return audio, and the audio processor transmits the return audio to the far-end. The video conferencing device as described in claim 6, wherein the audio processor performs at least one of the following operations on the filtered audio before performing the mixing operation based on the filtered audio and the other near-end audio to generate the return audio: beamforming, noise reduction, residual echo suppression, and automatic gain. The video conferencing device as described in claim 6, wherein the microphone array includes a plurality of microphones, the filtered audio includes a plurality of filtered audio components respectively corresponding to the microphones, and the audio processor calculates a time difference based on two of the filtered audio components to generate the angle control signal.