JPH10126757A - Video conference system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To display an image of a participant without a sense of incongruity by displaying an image of a conference participant making an utterance as a dynamic image and displaying others as still images to reduce quantity of transfer data. SOLUTION: A voice level recognition device 15 recognizes change in an utterance party based on voice data and gives the voice data to a data synthesizer 17 and the result of recognition to a data synthesis controller 16. The data synthesis controller 16 selects whose moving image of participants to be transferred to each terminal equipment to make an image of an utterance party as a dynamic image, reduces the image via a data synthesizer 17 and provides an output of synthesis image data. The voice data are synthesized so that each participant receives other voice data of participants than himself and gives the data to a data controller 14, where the data from the data synthesizer 17 are superimposed, the result is transferred to a communication equipment 7 of each terminal equipment via a communication equipment 13 as communication data and the image data are displayed on a display device 5 and the voice data are reproduced by a speaker 4. Then images of other participants than that of the utterance party are displayed as still images.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for displaying moving image data of a conference participant in a system for transferring video / audio data over a network and holding a conference.

[0002]

2. Description of the Related Art A system for connecting a terminal such as a personal computer using a communication line to transfer a moving image, a voice, data, and the like and hold a conference has been conventionally used.

FIG. 7 is a block diagram illustrating a conventional system. The system comprises terminal devices 37, 45, 46, 4
7, a server 48 and a communication line 44.

[0004] A terminal device 37 receives a camera 38 for capturing an image of the conference room provided with the terminal device, a microphone 39 for capturing the audio of the conference room, and receives respective video and audio data, and receives the data as communication data. A data control device 42 for dividing video / audio data transmitted or received from an opposite terminal into respective data, a data reproducing device (speaker 40, display 41) for reproducing each data; Transfer to server,
A communication device 43 for receiving communication data from the server.

[0005] The server 48 receives communication data transmitted from each terminal device, transfers a communication data to each terminal, and divides the data received from the communication device into respective data. The data is transmitted to the data synthesizing device 51, and the data control device 50 converts the data received from the data synthesizing device 51 into communication data, and synthesizes the image data and the audio data among the data received from the data control device. , A data synthesizing device 51 to be passed to the communication device.

[0006] Data of conference participants is input as moving image data from the camera of the terminal device and audio data from the microphone. When the respective data are input, the data control device 50 of each terminal device multiplexes each data as communication data and transfers the multiplexed data to the communication device 43. The communication device 43 transfers the communication data to the server.

[0007] The communication device 49 of the server 48 receives the communication data from the communication device on the terminal device side and passes it to the data control device 50. The data control device divides the received data into moving image data and audio data, and passes each data to the data synthesizing device 51.

The data synthesizing device 51 reduces the received moving image data and converts it into an image in which all the participants are displayed as shown in FIG. The audio data is also converted to a state where all participants can receive the same data,
Each data is passed to the data control device 50. The data control device 50 superimposes the data received from the data synthesizing device 51 again, and as communication data, the communication device 4
Hand over to 9. The communication device 49 transfers the received communication data to the communication device 43 of each terminal device.

Each terminal device passes the communication data received by the communication device 43 to the data control device 42. The data control device 42 divides the communication data into audio data and video data, and reproduces the data suitable for each data. The data is passed to the device and each data is reproduced.

[0010] In the system disclosed in Japanese Patent Application Laid-Open No. 07-298232 and a video conference system,
In this system, a moving image display is switched by a switch without using an MCU, and a communication path on a loop must be used. In the image display system of Japanese Unexamined Patent Publication No. However, the selection method is the switching of the conference chair and does not mention the data format.

[0011]

The first problem is that, in the past, in a video conference or the like, most of the participants in the conference were connected by two-way connections. The use of high-speed lines such as ATMs tends to increase the number of multipoint connection systems in the future.

However, widespread use of high-speed lines such as ISDN is not yet sufficient, and at present, public lines (PO
TS) is often used, and when a multipoint connection system is used, smooth reproduction of a moving image is difficult due to its absolutely low data transfer rate.

When a public line is used, a decrease in the data transfer amount is unavoidable due to its nature. For that reason,
When attempting to transfer data with a large amount of information such as a moving image, the data transfer must be realized by reducing the data amount by extremely compressing or thinning the data.

For this reason, when attempting to display moving image data of all participants, the number of drawn images per unit time becomes extremely small, resulting in a very jerky display or display of a rough image having a low resolution. Will be.

[0015] It is an object of the present invention to provide a video conferencing system that connects multiple points and allows a conference without discomfort without displaying moving images of all participants.

[0016]

According to the present invention, in order to achieve the above object, the video conference system described above displays a moving image of a participant selected based on a comment on a screen. The present invention provides a means for performing a video conference without discomfort by displaying the display images of the other participants as still images on each terminal device.

In practice, the present invention uses the speaker's voice level as a criterion for selecting participants.

For this reason, the system configuration of the present invention, specifically, an audio level recognition device for recognizing the level of the audio data received by the server, and which participant's image is synthesized based on the recognition result. A data synthesis control device for determining whether

The voice level recognition device recognizes the voice level spoken by the participant, and based on the recognition result, the moving image of the participant recognized as the speaker by the data synthesis control device. Are synthesized. That is, in the present invention, the server does not transmit image data other than video data from a terminal other than the current speaker, and transmits only a moving image of the speaker. A video originating from a terminal device to which the server did not transmit image data is displayed as a still image on all terminal devices.

When a participant other than the participant displayed as a moving picture speaks, the speech of the speech is recognized, and the speech is determined as the speaker. When the next speaker is determined, the speaker up to that point is displayed as a still image, and the moving image data of the new speaker is transmitted.

By creating display image data of each participant according to the above procedure, only the participant who is speaking is displayed as a moving image, and the other participants are displayed as still images.
The amount of image data per composite screen transmitted by the server can be reduced, the frame rate can be increased even on a communication line with a small data transfer amount, and a video conference with less discomfort can be held.

In the case where a participant other than the speaker makes a statement, if the statement is very short, for example, the switching of the moving image display frequently occurs, but a certain statement is shorter than a certain time. In such a case, the speaker is not switched to the moving image display. With this method, it is possible to prevent frequent switching of the screen.

[0023]

Next, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a first embodiment of the present invention.

The present system, as shown in FIG.
The system comprises four units 9, 10, and 11, a server 12, and a communication line.

In FIG. 1, a terminal device 1 receives a camera 2 for capturing a video of a participant, a microphone 3 for capturing a voice of a participant, video data and audio data, and transmits them to a server as communication data. A data control device 6 for dividing the image data and the audio data received from the server 12 into respective data, a display 5 for reproducing the image data, and a speaker 4 for reproducing the audio data
And a communication device 7 for transferring communication data to the server and receiving communication data from the server. It is assumed that the display 5 can display up to four people.

The server 12 includes a communication device 13 for receiving communication data transmitted from the terminal device 1 and transferring communication data to the terminal device 1;
And a data controller 14 for dividing the data received from the image data into image data and audio data, sending the data to the data synthesizing device 17, and converting the data received from the data synthesizing device 17 into communication data; And a speech level recognition device 15 that passes the voice data to the data synthesis device 17 and the speech sequence to the data synthesis control device 16 after determining the speech order of the participants; according to the recognition result received from the speech level recognition device 15 A data synthesizing control device 16 for determining which participant's image data is to be transferred to each terminal device, and a data synthesizing device for synthesizing image data and voice data in accordance with an instruction of the data synthesizing control device and passing the data to a communication device 17 are provided.

The data of the conference participants is input as moving image data from the camera 2 of the terminal device 1 and audio data from the microphone 3. When each data is input, the data control device 6 of the terminal device 1 superimposes each data as communication data and passes it to the communication device 7. The communication device 7 transfers the communication data to the server 12.

The communication device 13 of the server 12 receives the communication data from the communication device 7 of the terminal device 1 and passes it to the data control device 14. The data control device 14 divides the received data into a moving image and audio data, and passes the audio data to the audio level recognition device 15 and the image data to the data synthesizing device 17.

When the participant utters a sound of a certain level or higher from the received sound data, the sound level recognition device 15 regards the participant as a speaker, and the sound data is transmitted to the data synthesizing device 17. The recognition result is passed to the data synthesis control device 16.

The data synthesizing control device selects which participant's moving image is to be transferred to each terminal device from the received recognition result, and transmits the selected moving image to the data synthesizing device.

The data synthesizing device 17 selects only the image of the participant selected by the data synthesizing control device 16 from the received image data, reduces the size, and outputs the reduced image data as synthesized image data.

The voice data is synthesized such that each participant receives voice data uttered by a participant other than himself / herself, and each data is passed to the data control device 14.

The data control device 14 is a data synthesizing device 1
7 is superimposed again and passed to the communication device 13 as communication data.

The communication device 13 transfers the received communication data to the communication device 7 of the terminal device 1.

The terminal device 1 passes the communication data received by the communication device 7 to the data control device 6. The data control device 6 divides the communication data into image data and audio data, and reproduces the data suitable for each data. The image data is reproduced on the display 5 and the audio data is reproduced on the speaker 4.

In this case, the method of creating the image data is shown in FIG.
The procedure is as follows.

As shown in FIG. 4, numbers 1 to 4 are assigned to each divided screen, and the participants in front of each terminal are A, B, C, and D.

FIG. 2 is a flowchart for explaining the display switching operation performed by the server. The display switching operation will be described with reference to this flowchart.

In the initial state, no one is displayed on each screen, and, at the same time as the conference is held, the screens are displayed in the order of speeches 1 to 4 (steps 18 to 20 in FIG. 2). A, B,
When speaking in the order of C and D, 1-4 on the display
Are displayed as shown in FIG. In this case, the moving image is displayed at the last utterance D, and the other participants A, B, and C are displayed as still images (steps 21 and 2 in the figure).
2, 23).

Thereafter, when it is detected in step 21 of FIG. 2 that A speaks, A is displayed as a moving image in step 24 instead of D which has been displayed as a moving image.

As a moving image display method, as shown in FIG. 5, the first frame 33 of the image data at the start of the conference is
After synthesizing the image data of all members, send it to each terminal,
After the second frame 34, the server transmits data of only the speaker, and does not transmit data of other participants.
Further, when the speaker is switched, even in the frame 35 immediately after the switching, the image data of all the members is combined and transmitted, and from the next frame, the image data of only the speaker is transmitted.

In the terminal receiving the image data, when receiving the image data in which the data of all the members are combined, the image of all the members is updated, and when the image data of only the speaker is received, the image data of only the speaker is updated. Is updated, and the image data of the other participants is not updated, and a still image is displayed.

FIG. 5 shows an actual screen change.

Next, a second embodiment will be described.

During a conference, when a participant's question is answered very shortly by another participant, the display of the moving image of the speaker is frequently switched, which may make it very difficult to see. Therefore, when the utterance is extremely short and ends within a predetermined time, the utterer does not switch to the moving image display, and the previous utterer continues to display the moving image.

[0046]

As described above, according to the present invention, instead of displaying moving images of all participants on the display device, a moving image of the conference participant who is currently speaking is displayed, and other participants are displayed. Since the amount of data to be transferred can be reduced as a still image display and the participant who is speaking at the point of the conference can be displayed as a moving image, the data transfer amount of the communication line is low and all participants In a video conference system in which it is difficult to display a moving image, it is possible to display participants without discomfort.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of the present invention.

FIG. 2 is a flowchart illustrating a procedure for determining a display order.

FIG. 3 is a configuration diagram of an embodiment of the present invention.

FIG. 4 is an example of a moving image display screen of the present invention.

FIG. 5 is a diagram illustrating a change in transmission image data according to the embodiment of the present invention.

FIG. 6 is a block diagram illustrating a configuration of a conventional video conference system.

[Explanation of symbols]

 1, 9, 10, 11 Terminal device 6, 14 Data control device 7, 13 Communication device 15 Voice level recognition device 16 Data synthesis control device 17 Data synthesis device

Claims (5)

[Claims] An N number of terminals are connected via a communication line and a server, and each of the N number of terminals has video data from the N number of terminals divided into 1 / N screens. A video conferencing system in which each of the N divided screens is reduced and displayed on each of the N divided screens, wherein the server receives: video data and audio data from the N terminals; Based on the level of audio data from your device,
a moving image display terminal selecting means for selecting n (n is an integer of 1 or more and less than N) terminals; and reducing only video data of the selected terminal by 1 / N to all of the N terminals Each of the N terminals updates a split screen for displaying an image from a terminal corresponding to the reduced video data transferred from the server, and A video conferencing system wherein a split screen corresponding to a terminal to which data has not been transferred is not updated. 2. The server, comprising: a selected time storage unit that stores the latest time when the selected terminal is selected; and a time when the moving image display terminal selection unit determines that the selected terminal has been selected. Means for causing the moving image display terminal selecting means to continue selecting the terminal when the difference between the selected time and the time stored in the selected time storing means is equal to or less than a predetermined value. system. 3. The video conference system according to claim 1, wherein said n is 1. 4. The moving image display terminal selecting means,
4. The video conference system according to claim 1, wherein a terminal is selected from terminals that have supplied audio data in which the level of audio data supplied from the plurality of terminals exceeds a certain level. 5. The moving picture display terminal selecting means does not select a terminal whose audio data level has exceeded the predetermined level for a short period of time.
Or the video conference system according to 3.