JP5760783B2 - Communication terminal and program - Google Patents

Description

  The present invention provides a one-person communication such as a conference by transmitting / receiving conference information such as images and sounds of conference attendees between remote locations, a phone call using voice, or a phone call using voice and images. More particularly, the present invention relates to a communication terminal and a program having a sound mute function.

2. Description of the Related Art Conference systems that conduct a conference using a remote communication system and a remote communication terminal between remote locations such as remote locations via a communication network such as the Internet have become widespread.
In this conference system, in a conference room where one of the parties such as attendees who hold the conference is located, the conference system's remote communication terminal is used to shoot and collect voices such as conference room images and speeches. It is converted to electronic data and sent to the remote communication terminal of the other party, and it is displayed on the display of the other party's conference room TV screen, projector, etc. Have a meeting between distant places.

  Usually, in a meeting, there are occasions when a conversation that does not need to be notified to the other party of the meeting or a conversation that does not need to be notified occurs, for example, when it is desired to consult only with the inner ring.

  The same applies to a conference using a conference system. Such a situation occurs in the conference system. In such a case, the audio mute function temporarily reduces or eliminates the voice input of the conference system. There is something that can prevent the other party from hearing your voice.

  When mute is executed, the other party of the conference suddenly cuts off the voice from the other party, which may cause a sense of incongruity, and may cause a misunderstanding that the conference system has failed. is there.

  In order to solve this problem, there is a technique in which when the mute is executed, the other party is informed of the fact and a character, an icon, or the like is displayed on the other party's screen to notify that the mute state is present.

  For example, Patent Document 1 describes a technique in which the operation state of the own conference terminal is transmitted as additional data to the other conference terminal, and the operation state is displayed on the other conference terminal based on the additional data. As one of the additional data, mute ON / OFF of the microphone is described.

In a remote communication terminal of a conference system, there is a microphone for voice input that is incorporated in a remote communication terminal main body as described in Patent Document 2.
In addition, a general microphone that is not dedicated to a remote communication terminal is connected to the remote communication terminal body externally.

  The mute function is executed by operating a mute button, and this mute button is provided in the remote communication terminal main body when a microphone is incorporated in the main body.

In addition, a normal microphone is usually provided with a mute button, and a mute button provided on the microphone is used in a case where such a microphone is connected and used.
As described above, when a general microphone is used externally, the mute function can be executed by operating the mute button provided on the microphone. In this case, the mute function is executed remotely. In order to grasp in the communication terminal main body, it is necessary to know in the remote communication terminal main body information (operation information) that the mute button of the microphone has been pressed.

  In order to know this operation information, conventionally, it has been necessary to have a dedicated device driver (or application) corresponding to each microphone to be used on the remote communication terminal main body side.

  A microphone that is not supported on the remote communication terminal body side, such as when the remote communication terminal body does not have this device driver or is a device driver that does not match the microphone to be used even if it has a device driver Is used, the remote communication terminal main body cannot know that the mute button has been pressed, and as a result, there is a problem that it cannot tell the other party that the mute function has been executed.

Since microphones are used as needed, it is necessary to have all of the device drivers that are compatible with each microphone to be used. Costs, as well as development costs for device drivers for remote communication terminals,
In order to deal with such a problem, Patent Document 3 discloses that the reception side grasps the cause of the silent section by processing the data value of the voice data without modifying the existing packet format. Techniques that enable this are described.

Patent Document 3 describes that, regarding mute, the transmission data conversion unit 120A determines whether or not the microphone 111A is muted.
However, Patent Document 3 does not describe how to determine that the mute is performed, and the problem when the above-described general microphone is used externally is disclosed in Patent Document 3. Even the described technique is not solved.

  The above-mentioned problem exists not only in a conference using a conference system but also in a case where communication is performed between one person or one person or a plurality of persons such as a call using voice or a call using voice and an image. Is.

  As described above, in a communication terminal such as a remote communication terminal having a mute function of a microphone, when a dedicated device driver is not provided, or when a microphone that is not supported on the communication terminal body side is used, the mute function is set. There was a problem that it was impossible to tell the other party that it was done.

  In order to solve this problem, it is determined whether or not the mute state of the microphone being used can be notified, and if it can be notified, the mute state is determined based on the notified mute information, When it is determined that the mute state cannot be notified, the volume input to the voice input unit is further measured, and the mute state is determined based on the measurement result, so that an unsupported microphone can be selected. It may be possible to tell the other party that the sound has been muted even if it is used.

However, even in such a technique, it is necessary that the microphone or the remote communication terminal main body is provided with a mute button, and this cannot be dealt with when the mute button is not provided.
The present invention addresses such problems, and provides a communication terminal and a program that perform a process related to mute based on a change state of a volume level of an audio output without using a mute button.

  The features of the present invention as means for solving the above-described problems are as follows.

The communication terminal according to the present invention is a communication terminal for bidirectionally communicating at least voice in communication with a communication terminal of one or more communication partners via a network, wherein the communication terminal is a voice input for inputting voice. parts and, with the audio output unit that outputs audio transmitted from the other party of communication terminals, and volume change operation means for operating the change in volume level to be more outputted to the audio output unit, an operation of the volume changing operation means A monitoring unit that monitors the volume level output to the sound output unit changed by the above and a transmission / reception unit that communicates via a network, and the monitoring unit is when the volume level is a minimum value, Muting the input of the voice input unit, Create a message added with the destination terminal information and information of the mute state, characterized in that a request for transmission of the message to the transceiver.

  According to the present invention, since the process related to mute is performed based on the change state of the volume level of the audio output, the mute process can be performed even when the mute button is not provided.

FIG. 1 is a schematic diagram of a telecommunications system according to an embodiment of the present invention. FIG. 2 is a hardware configuration diagram of a remote communication terminal according to an embodiment of the present invention. FIG. 3 is an external view of a remote communication terminal according to an embodiment of the present invention. FIG. 4 is a hardware configuration diagram of a remote communication management system, a relay device, or a program providing server according to an embodiment of the present invention. FIG. 5 is a functional block diagram of each terminal, apparatus, and system constituting the telecommunications system according to the embodiment of the present invention. FIG. 6 is a functional block diagram of the voice input unit 15a and the voice output unit 15b shown in FIG. FIG. 7 is a conceptual diagram illustrating the image quality of image data. FIG. 8 is a conceptual diagram showing a modified quality management table. FIG. 9 is a conceptual diagram showing a relay device management table. FIG. 10 is a conceptual diagram showing a terminal authentication management table. FIG. 11 is a conceptual diagram showing a terminal management table. FIG. 12 is a conceptual diagram showing a destination list management table. FIG. 13 is a conceptual diagram showing a session management table. FIG. 14 is a conceptual diagram showing an address priority management table. FIG. 15 is a conceptual diagram showing a transmission rate priority management table. FIG. 16 is a conceptual diagram showing a quality management table. FIG. 17 is a sequence diagram illustrating processing for managing state information indicating the operating state of each relay device. FIG. 18 is a sequence diagram showing processing in a preparation stage for starting remote communication between remote communication terminals. FIG. 19 is a sequence diagram illustrating processing for narrowing down relay devices. FIG. 20 is a process flow diagram illustrating a process of narrowing down relay apparatuses. FIG. 21 is a diagram illustrating a calculation state of priority points when the relay device narrowing-down process is performed. FIG. 22 is a sequence diagram illustrating processing in which the remote communication terminal selects a relay device. FIG. 23 is a process flow diagram showing a process of selecting a relay device at a remote communication terminal. FIG. 24 is a sequence diagram illustrating processing for transmitting and receiving image data and audio data between terminals. FIG. 25 is a process flowchart showing the transmission process in the microphone mute state. FIG. 26 is a diagram showing microphone data. FIG. 27 shows data in the mute state. FIG. 28 shows an example of message data having mute information. FIG. 29 is a diagram showing another data example of a message having mute information. FIG. 30 is a process flow diagram showing a part of the process flow of FIG. 25 (when the microphone mute state can be notified). FIG. 31 is a process flow diagram showing a part of the process flow of FIG. 25 (when the microphone mute state cannot be notified). FIG. 32 is a process flow diagram showing the process in the remote communication terminal of the transmission destination that receives the message having the mute information. FIG. 33 is a diagram showing display on the remote communication terminal of the transmission destination in the mute state. FIG. 34 shows a menu screen for switching between the conference mode and the microphone registration mode. FIG. 35 is a processing flowchart showing processing in the microphone registration mode. FIG. 36 is a diagram showing a dialog screen at the time of microphone registration. FIG. 37 is a diagram showing a dialog screen at the end of microphone registration. FIG. 38 is a process flow diagram showing a part of the process flow of FIG. 35 (when microphone mute information can be acquired). FIG. 39 is a process flow diagram showing a part of the process flow of FIG. 35 (when microphone mute information cannot be acquired). FIG. 40 shows a dialog screen during the microphone registration process. FIG. 41 is a process flowchart showing a transmission process in the mute state by the volume change button. FIG. 42 is a diagram showing an example of data of a message having mute information. FIG. 43 is a diagram showing another data example of a message having mute information. FIG. 44 is a process flow diagram showing another example of the transmission process in the mute state by the volume change button. FIG. 45 is a process flowchart showing the process shown in FIG. 41 or the process for selecting the process shown in FIG. FIG. 46 is a diagram showing display on the remote communication terminal in the mute state. FIG. 47 is a process flow diagram showing the process in the remote communication terminal that receives the message having the mute information. FIG. 48 is a diagram showing display on the remote communication terminal of the transmission destination in the mute state.

<< Overall Configuration of Embodiment >>
Hereinafter, an embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a schematic diagram of a remote communication system 1 according to an embodiment of a communication system of the present invention. First, an outline of the present embodiment will be described with reference to FIG.

  Note that a communication terminal and a communication system are a terminal and a system for performing communication between one person and one person or a plurality of persons by using voice, video, voice and video, or other information and the like. In addition, communication terminals include terminals such as portable mobile phones.

  1 includes a plurality of remote communication terminals 10 (10aa, 10ab,..., 10db) and a display 11 (11aa, 11ab,..., 10db) for each remote communication terminal 10 (10aa, 10ab,..., 10db). 11 db), a plurality of relay devices 30 (30a, 30b, 30c, 30d), and the remote communication management system 50.

  The remote communication terminal 10 is according to an embodiment of a communication terminal, and the remote communication management system 50 is according to an embodiment of a communication management system.

  In the present embodiment, when any remote communication terminal among the remote communication terminals 10 (10aa, 10ab,..., 10db) is indicated, it is referred to as “remote communication terminal 10”, and the display 11 (11aa, 11ab,. , 11db) is referred to as “display 11” when an arbitrary display is indicated, and is referred to as “relay apparatus 30” when any relay device among the relay devices 30 (30a, 30b, 30c, 30d) is indicated.

  The remote communication terminal 10 transmits / receives image data, audio data, and the like to / from other remote communication terminals 10. In the present embodiment, the case where the image of the image data is a moving image will be described. The image of the image data may include both a moving image and a still image. The relay device 30 relays image data and audio data between the plurality of remote communication terminals 10. The remote communication management system 50 manages the remote communication terminal 10 and the relay device 30 in an integrated manner.

  Also, the plurality of routers (70a, 70b,..., 70g) shown in FIG. 1 perform selection of optimal paths for image data and audio data. In the present embodiment, “router 70” is used to indicate an arbitrary router among the routers (70a, 70b,..., 70g). The program providing system 90 includes an HD (Hard Disk) (not shown) in which a remote communication terminal program for causing the remote communication terminal 10 to implement various functions or various means is stored. A program for a remote communication terminal can be transmitted. The HD of the program providing system 90 also stores a relay device program for causing the relay device 30 to realize various functions or various means, and can transmit the relay device program to the relay device 30. . Further, the HD of the program providing system 90 also stores a remote communication management program for causing the remote communication management system 50 to realize various functions or various means, and the remote communication management system 50 stores the remote communication management program. Can be sent.

  Further, the remote communication terminal 10aa, the remote communication terminal 10ab, the relay device 30a, and the router 70a are communicably connected via the LAN 2a. The remote communication terminal 10ba, the remote communication terminal 10bb, the relay device 30b, and the router 70b are communicably connected via the LAN 2b. The LAN 2a and the LAN 2b are communicably connected via a dedicated line 2ab including the router 70c, and are constructed in a predetermined area A. For example, the region A is Japan, the LAN 2a is constructed in a Tokyo office, and the LAN 2b is constructed in an Osaka office.

  On the other hand, the remote communication terminal 10ca, the remote communication terminal 10cb, the relay device 30c, and the router 70c are communicably connected via a LAN 2c. The remote communication terminal 10da, the remote communication terminal 10db, the relay device 30d, and the router 70d are communicably connected via a LAN 2d. The LAN 2c and the LAN 2d are communicably connected by a dedicated line 2cd including the router 70e, and are constructed in a predetermined area B. For example, region B is the United States, LAN 2c is built in a New York office, and LAN 2d is Washington D.C. C. Is built in the office. Area A and area B are connected to each other via routers (70c, 70e) via the Internet 2i.

  The remote communication management system 50 and the program providing system 90 are communicably connected to the remote communication terminal 10 and the relay device 30 via the Internet 2i. The remote communication management system 50 and the program providing system 90 may be installed in the region A or the region B, or may be installed in a region other than these.

  In the present embodiment, the communication network 2 of the present embodiment is constructed by the LAN 2a, the LAN 2b, the dedicated line 2ab, the Internet 2i, the dedicated line 2cd, the LAN 2c, and the LAN 2d.

  The remote communication management system 50, each remote communication terminal 10, and each relay device 30 perform XML-based message exchange, login status, and other status notification by XMPP (eXtensible Messageing and Presence Protocol).

  Specifically, the remote communication management system 50 manages the status of each remote communication terminal 10 such as “online”, “offline”, “meeting”, “mute”, and the like from a certain remote communication terminal 10. When the status is notified to the remote communication management system 50, the remote communication management system 50 performs mediation such as notifying the other remote communication terminals 10 of this status.

  The remote communication terminal 10 transmits / receives image data, audio data, and the like to / from other remote communication terminals 10 via the relay device 30. This transmission / reception is performed using RTP (Real-time Transport Protocol). It is.

  Further, in FIG. 1, four sets of numbers shown under each remote communication terminal 10, each relay device 30, remote communication management system 50, each router 70, and program providing system 90 are in general IPv4. An IP address is simply shown. For example, the IP address of the remote communication terminal 10aa is “1.2.1.3”. In addition, IPv6 may be used instead of IPv4, but IPv4 will be used for the sake of simplicity.

  In addition, although the example of the conference system which performs the conference between the places distant from each other as a telecommunications system is shown, here, the conference is not only in the case of being away from the distance but also in the same building. This includes a conference between conference rooms that are close to each other like another conference room, or between places where the same room cannot hear the voice. In other words, the conference in the present invention means a conference conducted via a remote communication system.

  Therefore, even if it is a meeting held directly face-to-face, when it is held via a remote communication system, it means a meeting in the present invention.

The conference includes not only a plurality of people but also one-on-one. It also includes simple meetings and exchanges such as information exchange.
The conference includes not only business-related matters but also private matters. For example, a conversation for a family living in a distant place to keep in touch is included.

  The meeting information is shown for the image and sound during the meeting, but at least one of them may be used. For example, when only an image is displayed, the attendee's voice may be displayed as subtitles in the image.

  In addition, the conference information includes information presented to the parties (for example, product samples), documents to be distributed, and images that are not distributed but displayed on the screen by a display device such as a projector. As described above, the conference information means information that can be grasped by the attendees at the meeting place in the case where the conference information is held directly and face-to-face.

In addition, the sound in the present embodiment includes not only a sound emitted by a person but also other sounds.
<< Hardware Configuration of Embodiment >>
Next, the hardware configuration of this embodiment will be described. In this embodiment, when there is a delay in the reception of image data at the remote communication terminal 10 as the relay destination, the resolution of the image data of the image data is changed by the relay device 30 and then the remote communication terminal 10 as the relay destination. A case where image data is transmitted to will be described.

  FIG. 2 is a hardware configuration diagram of the remote communication terminal 10 according to an embodiment of the present invention. As shown in FIG. 2, the remote communication terminal 10 of this embodiment includes a CPU (Central Processing Unit) 101 that controls the operation of the entire remote communication terminal 10 and a ROM (Read Only) that stores a program for the remote communication terminal. Memory) 102, RAM (Random Access Memory) 103 used as a work area for the CPU 101, flash memory 104 for storing various data such as image data and audio data, reading various data to the flash memory 104 according to the control of the CPU 101, or An SSD (Solid State Drive) 105 that controls writing, a media drive 107 that controls reading or writing (storage) of data to a recording medium 106 such as a flash memory, and a remote communication terminal for selecting a destination of the remote communication terminal 10 10 to operate 10 or to be described later A plurality of operation buttons 108 used for operations such as a mute button and a volume change button, a power switch 109 for switching on / off the power of the remote communication terminal 10, and data communication using a communication network 2 described later. Network I / F 111, a CMOS (Complementary Metal Oxide Semiconductor) 112 that captures an image of a subject under the control of the CPU 101 and obtains image data, an image sensor I / F 113 that controls driving of the CMOS 112, a microphone 114 that inputs audio, and audio A speaker 115 that outputs a sound, an audio input / output I / F 116 that processes input / output of an audio signal between the microphone 114 and the speaker 115 according to the control of the CPU 101, and a display that transmits image data to the external display 11 according to the control of the CPU 101 I / F117 And a address bus, a data bus, and the like bus line 110 for electrically connecting as shown in Figure 2 the above components.

  The microphone 114 and the speaker 115 are incorporated in the main body of the remote communication terminal 10, but the remote communication terminal 10 is connected to the voice input / output I / F 116 so that separate external devices can be used. It has connection terminals for microphones and speakers. As a result, sound collection and voice output can be performed at a position away from the remote communication terminal 10, and usage according to the situation such as selection and arrangement of microphones and speakers can be performed.

  In FIG. 2, reference numeral 114a indicates an external microphone, and speaker 115a indicates an external speaker.

A part of the operation button 108 is used as a mute button of the microphone 114. This mute button is used when a mute state is established when the microphone 114 incorporated in the main body is used.
When the external microphone 114a is used, the mute button 114b provided on the normal microphone 114a is used to make the mute state. This will be described later.

  When an external microphone is used, the input of the external microphone 114a and the mute button 114b are validated, and the input of the microphone 114 incorporated in the main body and the input of the mute button provided on the main body are performed. It is configured to be invalidated.

A part of the operation buttons 108 described above is also used as a volume change button of the speaker 115. A volume change button 115b is also provided on the external speaker 115a.
The external volume change button 115b can also be used as an operation button for executing mute. This will be described later.

  Similarly to the microphone 114, when an external speaker is used, the output to the external speaker 115 and the input of the volume change button 115b are validated, and the speaker 115 incorporated in the apparatus main body is enabled. And the input of the volume change button are invalidated.

  The connection terminal of the microphone 114a and the speaker 115a uses a USB (Universal Serial Bus) standard terminal.

  In the present embodiment, the mute button and the volume change button of the main body are disabled in correspondence with disabling the microphone 114 and the speaker incorporated in the device main unit and the like, respectively. You can also.

  Note that the recording medium 106 is detachable from the remote communication terminal 10. Further, as long as it is a non-volatile memory that reads or writes data according to the control of the CPU 101, not only the flash memory 104 but also an EEPROM (Electrically Erasable and Programmable ROM) or the like may be used. Further, the COMS 112 is a solid-state imaging device that converts light into electric charges and digitizes a subject image (video), and is not limited to a CMOS as long as it captures a subject, and a CCD (Charge Coupled Device) or the like is used. May be. The display 11 is composed of a liquid crystal or organic EL that displays a subject image, an operation icon, and the like.

  Furthermore, the remote communication terminal program may be distributed in a computer-readable recording medium such as the recording medium 106 as an installable or executable file.

FIG. 3 is an external view of the remote communication terminal 10 according to the present embodiment.
As shown in FIG. 3, the remote communication terminal 10 includes a housing 1100, an arm 1200, and a camera housing 1300.

  An air intake surface (not shown) formed by a plurality of air intake holes is provided on the rear side surface 1110 of the housing 1100, and an air exhaust having a plurality of air exhaust holes formed on the front side surface 1120 of the housing 1100. A surface 1121 is provided.

  Thus, by driving a cooling fan built in the housing 1100, outside air is taken into the housing 1100 from the rear of the terminal 10 through the intake surface and exhausted to the front of the terminal 10 through the exhaust surface 1121.

  A sound collecting hole 1131 is formed on the right side surface 1130 of the housing 1100, and sound can be collected by the microphone 114 built in the housing 1100 through the sound collecting hole 1131.

  The right side surface 1130 of the housing 1100 is provided with a USB (Universal Serial Bus) standard connection terminal 1132 for a microphone and a speaker for connecting the external microphone 114 a and the speaker 115 a to the audio input / output I / F 116. ing.

  A connection terminal (not shown) for connecting the external display 11 to the display I / F 117 is provided on the left side surface 1140 of the housing 1100.

  An operation panel 1150 is formed on the upper surface of the housing 1100 on the right side 1130 side. The operation panel 1150 is provided with an operation button 108 and a power switch 109, and is formed by a plurality of sound output holes for outputting sound from the speaker 115 built in the housing 1100 to the outside. A sound output surface 1151 is formed.

  Further, a housing portion 1160 that is a recess for housing the arm 1200 and the camera housing 1300 is formed on the upper surface of the housing 1100 on the left side surface 1140 side.

The arm 1200 is attached to the housing 1100 via a torque hinge 1210, and the arm 1200 is rotatable with respect to the housing 1100 in a vertical direction within a range where the tilt angle θ1 is 135 degrees upward. And can be held at a desired position after the rotation.
FIG. 3 shows a state where the arm 1200 is held at a position where the tilt angle θ1 is 90 degrees.

  The camera housing 1300 is provided with a CMOS 112, and the camera housing 1300 is attached to an arm 1200 via a torque hinge 1310 so as to be rotatable in the vertical direction and rotatable in the horizontal direction. It is configured to be held at a later desired position.

  The range in which the camera housing 1300 can be rotated and rotated with respect to the arm 1200 is a range of a pan angle θ2 of ± 180 degrees when the camera housing 1300 shown in FIG. In addition, the tilt angle θ3 is within a range of ± 45 degrees, where the camera housing 1300 is in a horizontal state of 0 degrees.

  FIG. 4 is a hardware configuration diagram of the remote communication management system according to the embodiment of the present invention. The remote communication management system 50 includes a CPU 201 that controls the overall operation of the remote communication management system 50, a ROM 202 that stores a remote communication management program, a RAM 203 that is used as a work area for the CPU 201, and an HD (Hard Disk) that stores various data. 204, an HDD (Hard Disk Drive) 205 that controls reading or writing of various data to the HD 204 according to the control of the CPU 201, a media drive 207 that controls reading or writing (storage) of data to a recording medium 206 such as a flash memory, and a cursor , A display 208 for displaying various information such as menus, windows, characters, images, etc., a network I / F 209 for transmitting data using the communication network 2 described later, input of characters, numerical values, various instructions, etc. A keyboard 211 having a plurality of keys, a mouse 212 for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like, and a CD-ROM (Compact Disc Read Only Memory) as an example of a removable recording medium ) A CD-ROM drive 214 for controlling reading or writing of data with respect to 213, and a bus line 210 such as an address bus or a data bus for electrically connecting the above components as shown in FIG. It has.

  The remote communication management program is a file in an installable or executable format, and is recorded on a computer-readable recording medium such as the recording medium 206 or CD-ROM 213 for distribution. Good.

  Further, since the relay device 30 has the same hardware configuration as the remote communication management system 50, the description thereof is omitted. However, the ROM 202 stores a relay device program for controlling the relay device 30. Also in this case, the relay device program is a file in an installable or executable format, and is recorded on a computer-readable recording medium such as the recording medium 206 or CD-ROM 213 for distribution. Good.

  Furthermore, since the program providing system 90 has the same hardware configuration as that of the remote communication management system 50, the description thereof is omitted. However, a program providing program for controlling the program providing system 90 is recorded in the ROM 202. Also in this case, the program providing program is a file in an installable or executable format, and may be recorded and distributed on a computer-readable recording medium such as the recording medium 206 or the CD-ROM 213. Good.

As another example of the detachable recording medium, the recording medium is provided by being recorded on a computer-readable recording medium such as a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), or a Blu-ray Disc. May be.
<< Functional Configuration of Embodiment >>
Next, the functional configuration of this embodiment will be described. FIG. 5 is a functional block diagram of each terminal, device, and system constituting the telecommunications system 1 of the present embodiment. In FIG. 5, the remote communication terminal 10, the relay device 30, and the remote communication management system 50 are connected so that data communication can be performed via the communication network 2. Further, the program providing system 90 shown in FIG. 1 is omitted in FIG. 5 because it is not directly related to the video conference communication.

<Functional configuration of remote communication terminal>
The remote communication terminal 10 includes a transmission / reception unit 11, an operation input reception unit 12, a login request unit 13, an imaging unit 14a, an image display control unit 14b, an audio input unit 15a, an audio output unit 15b, a final narrowing unit 16, and a delay detection unit. 17, a mute processing unit 18 and a storage / reading processing unit 19.

  The mute processing unit 18 performs a mute process, and includes a processing unit 18a, a determination unit 18b, a state management unit 18c, a storage unit 18d, a mode switching unit 18e, a registration unit 18f, and a monitoring unit 18g.

  Each of these units is a function or means realized by any one of the constituent elements shown in FIG. 2 operating according to a command from the CPU 101 according to a program stored in the ROM 102. Further, the remote communication terminal 10 has a storage unit 1000 constructed by the SSD 105 shown in FIG.

(Functions of remote communication terminal)
Next, each part of the remote communication terminal will be described in detail. The transmission / reception unit 11 of the remote communication terminal 10 is realized by the network I / F 111 illustrated in FIG. 2, and transmits / receives various data (information) to / from other terminals, devices, or systems via the communication network 2. The operation input receiving unit 12 is realized by the operation button 108 and the power switch 109 shown in FIG. 2 and receives various inputs by the user. For example, when the user turns on the power switch 109 shown in FIG. 2, the operation input receiving unit 12 shown in FIG. 5 receives the power ON and turns on the power. The login request unit 13 is realized by an instruction from the CPU 101 shown in FIG. 2, and requests the login from the transmission / reception unit 11 to the remote communication management system 50 via the communication network 2 when receiving the power ON. Log-in request information indicating that it is to be transmitted, and the current IP address of the remote communication terminal 10ab are automatically transmitted.

  The imaging unit 14a is realized by the CCD 112 and the imaging element I / F 113 shown in FIG. 2, images a subject, and outputs image data obtained by the imaging. The video display control unit 14 b is realized by the display I / F 117 shown in FIG. 2, and performs control for transmitting image data to the external display 11.

  The voice input unit 15a is realized by the microphone 114 shown in FIG. 2 or the external microphone 114a and the mute button 114b and both, and the voice input / output I / F 116, and inputs the user's voice. Audio is converted into an audio signal, signal processing is performed, and audio data related to the audio signal is output.

The audio output unit 15b is realized by the speaker 115 shown in FIG. 2 or the external speaker 115a and the volume control button 115b and both, and the audio input / output I / F 116, and outputs an audio signal related to audio data. Process, convert to sound and output.
The voice input unit 15a and the voice output unit 15b will be described later in detail.

  The final narrowing unit 16 performs a final narrowing process that finally narrows down from a plurality of relay devices 30 to one relay device 30, so that a measurement unit 16 a and a calculation unit 16 b are executed according to a command from the CPU 101 illustrated in FIG. 2. , And the final selection unit 16c. Among these, the measurement unit 16 a measures the reception date and time when the pre-transmission information is received by the transmission / reception unit 11 for each pre-transmission information described later received by the transmission / reception unit 11. For each pre-transmission information whose reception date and time is measured by the measurement unit 16a, the calculation unit 16b determines the pre-transmission information based on the difference between the measured reception time and the transmission date and time included in the pre-transmission information. The time required from transmission to reception is calculated. The final selection unit 16c finally selects one relay device by selecting the relay device 30 to which the pre-transmission information that required the shortest required time is relayed among the required times calculated by the calculation unit 16b. .

  The delay detection unit 17 is realized by a command from the CPU 101 shown in FIG. 2 and calculates the delay time (ms) of image data or audio data sent from another remote communication terminal 10 via the relay device 30. To detect. The storage / reading processing unit 19 is executed by the SSD 105 shown in FIG. 2, and stores various data in the storage unit 1000 or reads various data stored in the storage unit 1000. The storage unit 1000 stores a terminal ID (Identification) for identifying the remote communication terminal 10, a password, image data, audio data, and the like.

  The terminal ID of this embodiment and the relay device ID described later are identification information such as language, characters, symbols, or various signs used to uniquely identify the remote communication terminal 10 and the relay device 30, respectively. Indicates. Further, the terminal ID and the relay device ID may be identification information in which at least two of the language, characters, symbols, and various signs are combined. In the following description, the remote communication terminal 10 serving as a request source requesting the start of a video conference is referred to as “request source terminal 10A”, and the remote communication terminal 10 serving as a request destination is referred to as “destination terminal 10B”.

  The mute processing unit 18 performs the mute processing of the remote communication terminal 10, and each unit is as follows.

The processing unit 18a manages the overall processing related to mute, acquires information on the mute function of the external microphone 114a, acquires other terminal information to be notified of the status of the own terminal, and a state management unit. An instruction for notification processing is given to 18c. The processing unit 18a performs control when newly registering an external microphone 114a that is not registered.
The determination unit 18b performs processing necessary to determine the type of the voice input unit 15a (for example, the type of the microphone 114a) and to specify the mute state.

The state management unit 18c performs management such as acquisition of the state of the own terminal regarding mute and notification of the state of the own terminal regarding mute to other terminals.
The storage unit 18d stores registration information indicating the type of the microphone 114a, setting information regarding mute, a mute state, and the like.

The mode switching unit 18e switches between a mode for notifying the mute state of the microphone 114a during the video conference and a mode for performing the registration process of the microphone 114a.
When newly registering a microphone 114a that is not registered, the registration unit 18f measures the characteristics of the microphone 114a and performs a registration process.

  The monitoring unit 18g monitors the volume level output from the audio output unit to the minimum state or other volume level, and sends the information to the state management unit 18c when the volume level is changed.

  Next, the voice input unit 15a and the voice output unit 15b will be described with reference to FIG.

  FIG. 6A shows the voice input unit 15a, and shows a case where an external microphone 114a is connected.

  In the audio input unit 15a, each process by the acoustic echo canceller 15a1, the noise reduction 15a2, the echo suppressor 15a3, the automatic gain control 15a4, and the mute switch 15a5 is performed on the signal input from the microphone 114a.

  The acoustic echo canceller 15a1 outputs the received own voice from the other party's speaker, and this voice is collected by the other party's microphone and transmitted to the self side to cancel the acoustic echo generated from the own speaker. To do. The acoustic echo canceller is stipulated in ITU-T (International Telecommunication Union Telecommunication standardization sector) recommendation G.165, and one conforming to this specification is used.

  The noise reduction 15a2 estimates and attenuates a noise signal mixed from a surrounding personal computer, projector, air conditioner or the like. This process uses a well-known spectral subtraction method that estimates and subtracts the spectrum of the noise signal.

  The echo suppressor 15a3 is provided downstream of the acoustic echo canceller 15a1 and attenuates the residual echo signal. The echo remains when the echo canceller cannot learn accurately due to the influence of nonlinearity of the echo signal, noise, double talk (two-way simultaneous call), and the like. The echo suppressor is stipulated in ITU-T recommendation G.165, and one that conforms to this specification is used.

  The automatic gain control 15a4 always keeps the output constant regardless of the input level. Specifically, the input signal is variably controlled so that the output is always constant by increasing the sensitivity when the input signal is weak, and decreasing the sensitivity when the input signal is strong. .

  The mute switch 15a5 is used to pass or cut a signal after processing the echo suppressor 15a3 by accepting on / off of the mute button 114b pressed by the user of the microphone 114a.

  FIG. 6B shows the audio output unit 15b, and shows a case where an external speaker 115a is connected.

  In the audio output unit 15b, the signal after the processing by the line echo canceller 15b1, the noise reduction 15b2, the echo suppressor 15b3, the automatic gain control 15b4, and the volume change control 15b5 is performed on the input signal is the speaker 115a. Is output from.

  The line echo canceller 15b1 ensures good call quality during double talk by erasing line echo generated in the IP or telephone line. The line echo canceller is stipulated in ITU-T recommendation G.168, and one conforming to this specification is used.

  The noise reduction 15b2, the echo suppressor 15b3, and the automatic gain control 15b4 of the audio output unit 15b are the same as the noise reduction 15a2, the echo suppressor 15a3, and the automatic gain control 15a4 of the audio input unit 15a, respectively.

  The volume change control 15b5 changes the volume level of the signal after the automatic gain control 15b4 process by accepting the volume change by the user's operation of the volume change button 115b.

(Functional configuration of relay device)
Next, functions or means of the relay device 30 will be described. The relay device 30 includes a transmission / reception unit 31, a state detection unit 32, a data quality confirmation unit 33, a change quality management unit 34, a data quality change unit 35, and a storage / read processing unit 39. Each of these units is a function or means realized by any one of the constituent elements shown in FIG. 4 operating according to a command from the CPU 201 according to a program stored in the ROM 202. Further, the relay device 30 has a storage unit 3000 constructed by the HD 204 shown in FIG.

(Change quality control table)
In the storage unit 3000, a change quality management DB (Data Base) 3001 configured by a change quality management table as shown in FIG. In the change quality management table, the IP address of the remote communication terminal 10 as a relay destination of image data and the image quality of the image data relayed by the relay device 30 are associated with the relay destination and managed.

  Here, the resolution of the image data handled in the present embodiment will be described. As shown in FIG. 7 (a), the horizontal image is composed of 160 pixels and the vertical image is 120 pixels, and the low resolution image as the base image and the horizontal image as shown in FIG. 7 (b). There are a medium resolution image composed of 320 pixels and 240 pixels vertically, and a high resolution image composed of 640 pixels horizontally and 480 pixels vertically as shown in FIG. 7C. Among these, when passing through a narrow band route, low-quality image data consisting only of low-resolution image data serving as a base image is relayed. When the band is relatively wide, low-resolution image data serving as a base image and medium-quality image data composed of medium-resolution image data are relayed. When the bandwidth is very wide, high-resolution image data composed of low-resolution image data serving as base image quality, intermediate-resolution image data, and high-resolution image data is relayed. For example, in the modified quality management table shown in FIG. 8, when the relay device 30 relays image data to the destination terminal 10db having the IP address “1.3.2.4”, the relay is performed. The image quality (image quality) of the image data is “high quality”.

<Each functional unit of the relay device>
Next, each functional configuration of the relay device 30 will be described in detail. In the following, in describing each part of the relay device 30, a relationship with main components for realizing each part of the relay device 30 among the respective components illustrated in FIG. 2 will also be described.

  The transmission / reception unit 31 of the relay device 30 shown in FIG. 5 is realized by the network I / F 209 shown in FIG. 4, and communicates with other terminals, devices, or systems via the communication network 2 and various data (information ). The state detection unit 32 is realized by a command from the CPU 201 illustrated in FIG. 4, and detects the operating state of the relay device 30 having the state detection unit 32. The operating state includes “ON line”, “OFF line”, and “failing” state.

  The data quality confirmation unit 33 is realized by a command from the CPU 201 shown in FIG. 4, and searches the change quality management DB 3001 (see FIG. 8) using the IP address of the destination terminal 10B as a search key and relays the corresponding data. The image quality of the relayed image data is confirmed by extracting the image quality of the image data to be relayed. The change quality management unit 34 is realized by a command from the CPU 201 shown in FIG. 4 and changes the contents of the change quality management DB 3001 based on quality information to be described later sent from the remote communication management system 50. . For example, during a video conference by transmitting / receiving high-quality image data between the request source terminal 10aa having the terminal ID “01aa” and the destination terminal 10db having the terminal ID “01db”. When a delay in receiving image data occurs at the destination terminal 10db due to the request source terminal bb and the destination terminal ca performing another video conference starting the video conference via the communication network 2, etc., the relay device No. 30 needs to lower the image quality of image data relayed up to now from high image quality to medium image quality. In such a case, the content of the change quality management DB 3001 is changed so as to lower the image quality of the image data relayed by the relay device 30 from the high image quality to the medium image quality based on the quality information indicating the medium image quality.

  The data quality changing unit 35 is realized by an instruction from the CPU 201 shown in FIG. 4, and the image quality of the image data sent from the transmission source terminal 10 is based on the contents of the changed change quality management DB 3001. To change. The storage / reading processing unit 39 is realized by the HDD 205 shown in FIG. 4, and stores various data in the storage unit 3000 and reads various data stored in the storage unit 3000.

<Functional configuration of remote communication management system>
Next, functions or means of the remote communication management system 50 will be described. The remote communication management system 50 includes a transmission / reception unit 51, a terminal authentication unit 52, a state management unit 53, a terminal extraction unit 54, a terminal state acquisition unit 55, a primary narrowing unit 56, a session management unit 57, a quality determination unit 58, a storage / A read processing unit 59, a delay time management unit 60, and a message transmission unit 61 are included.

  Each of these units is a function or means realized by any one of the constituent elements shown in FIG. 4 operating according to a command from the CPU 201 according to a program stored in the ROM 202. Further, the remote communication management system 50 has a storage unit 5000 constructed by the HD 204 shown in FIG.

(Relay device management table)
In the storage unit 5000, a relay device management DB 5001 configured by a relay device management table as shown in FIG. 9 is constructed. In this relay device management table, for each relay device ID of each relay device 30, the operating status of each relay device 30, the reception date and time when the status information indicating the operating status is received by the remote communication management system 50, the relay device 30 The IP address and the maximum data transmission rate (Mbps) in the relay device 30 are associated and managed. For example, in the relay device management table shown in FIG. 9, the relay device 30 a with the relay device ID “111a” has the operating state “ON line” and the date and time when the status information was received by the remote communication management system 50. Is “November 10, 2009, 13:00”, the IP address of the relay device 30a is “1.2.1.2”, and the maximum data transmission rate in the relay device 30a is 100 Mbps. It is shown.

(Terminal authentication management table)
Furthermore, in the storage unit 5000, a terminal authentication management DB 5002 configured by a terminal authentication management table as shown in FIG. In this terminal authentication management table, each password is associated with each terminal ID of all remote communication terminals 10 managed by the remote communication management system 50 and managed. For example, the terminal authentication management table shown in FIG. 10 indicates that the terminal ID of the remote communication terminal 10aa is “01aa” and the password is “aaaa”.

(Terminal management table)
Further, in the storage unit 5000, a terminal management DB 5003 configured by a terminal management table as shown in FIG. 11 is constructed. In this terminal management table, for each terminal ID of each remote communication terminal 10, the operating state of each remote communication terminal 10, the reception date and time when login request information described later is received by the remote communication management system 50, and the remote communication terminal 10 An IP address is associated and managed. For example, in the terminal management table shown in FIG. 11, the remote communication terminal 10aa with the terminal ID “01aa” is in the “ON line” operating state, and the date and time when the login request information is received by the remote communication management system 50 Is “November 10, 2009, 13:40” and the IP address of the remote communication terminal 10aa is “1.2.1.3”.

(Destination list management table)
Further, in the storage unit 5000, a destination list management DB 5004 configured by a destination list management table as shown in FIG. In this destination list management table, all terminal IDs of destination terminals 10B registered as candidates for destination terminal 10B are managed in association with terminal IDs of request source terminal 10A requesting the start of a video conference. For example, in the destination list management table shown in FIG. 12, the candidate of the destination terminal 10B that can request the start of the video conference from the request source terminal 10aa having the terminal ID “01aa” has the terminal ID “01ab”. The remote communication terminal 10ab with the terminal ID "01ba" and the remote communication terminal 10db with the terminal ID "01db" are shown. The destination terminal 10B candidates are updated by being added or deleted in response to a request for addition or deletion from the request source terminal 10A to the remote communication management system 50.

(Session management table)
Further, in this storage unit 5000, a session management DB 5005 configured by a session management table as shown in FIG. 13 is constructed. In this session management table, for each selection session ID used to execute a session for selecting the relay device 30, the relay device ID of the relay device 30 used for relaying image data and audio data, the request source terminal 10A Is sent from the destination terminal 10B, the terminal ID of the destination terminal 10B, the terminal ID of the destination terminal 10B, the reception delay time (ms) when the destination terminal 10B receives image data, and the delay information indicated by this delay time. The reception date and time received and received by the remote communication management system 50 is managed in association with each other. For example, in the session management table shown in FIG. 13, the relay device 30a (relay device ID “111a”) selected in the session executed using the selection session ID “se1” has the terminal ID “01aa”. "And the destination terminal 10db with the terminal ID" 01db "are relayed image data and audio data, and the destination terminal 10db" November 10, 2009 14:00 It is shown that the delay time of the image data at the time is 200 (ms). When performing a video conference between the two remote communication terminals 10, the reception date and time of the delay information may be managed based on the delay information transmitted from the request source terminal 10A instead of the destination terminal 10B. Good. However, when a video conference is performed between three or more remote communication terminals 10, the reception date and time of the delay information is based on the delay information transmitted from the remote communication terminal 10 on the image data and audio data reception side. Manage.

(Address priority management table)
Furthermore, in the storage unit 5000, a priority management DB 5006 configured by an address priority management table as shown in FIG. 14 is constructed. In this address priority management table, in accordance with the difference in the four dot address (Dot Address) portions of the general IPv4 IP addresses, the address priority points are associated and managed so as to increase. The For example, in the address priority management table shown in FIG. 14, when the three values are the same IP address from the top to the bottom of the dot address, the address priority point is “5”. When the two values are the same IP address from the top to the bottom of the dot address, the address priority point is “3”. In this case, whether the value of the lowest dot address is the same or not is not related to the priority. In the case of IP addresses having the same highest value of dot addresses and different second values from the top, the address priority point is “1”. In this case, whether or not the values of the third and lowest dot addresses from the top are the same has no relation to the priority. In the case of IP addresses having different uppermost values of dot addresses, the address priority point is “0”. In this case, whether or not the values of the second, third, and lowest dot addresses from the top are the same has no relation to the priority.

(Transmission speed priority management table)
The priority management DB 5006 built in the storage unit 5000 also includes a transmission speed priority management table as shown in FIG. In this transmission rate priority management table, the transmission rate priority points are associated and managed in accordance with the value of the maximum data transmission rate (Mbps) in the relay device 30 so as to increase. For example, in the transmission rate priority management table shown in FIG. 15, when the maximum data transmission rate in the relay apparatus 30 is 1000 Mbps or more, the transmission rate priority point is “5”. When the maximum data transmission rate in the relay device 30 is 100 Mbps or more and less than 1000 Mbps, the transmission rate priority point is “3”. When the maximum data transmission rate in the relay device 30 is 10 Mbps or more and less than 100 Mbps, the transmission rate priority point is “1”. When the maximum data transmission rate in the relay device 30 is less than 10 Mbps, the transmission rate priority point is “0”.

(Quality control table)
Furthermore, in the storage unit 5000, a quality management DB 5007 configured by a quality management table as shown in FIG. In this quality management table, the image quality (image quality) of the image data relayed by the relay device 30 is associated and managed according to the delay time (ms) of the image data at the request source terminal 10A or the destination terminal 10B.

(Each functional part of the remote communication management system)
Next, each functional unit of the remote communication management system 50 will be described in detail. In the following description, in describing each part of the remote communication management system 50, among the components shown in FIG. 4, the relationship with the main components for realizing each part of the remote communication management system 50 is also described. explain.

  The transmission / reception unit 51 is executed by the network I / F 209 illustrated in FIG. 2, and transmits / receives various data (information) to / from other terminals, devices, or systems via the communication network 2. The terminal authentication unit 52 searches the terminal authentication management DB 5002 in the storage unit 5000 using the terminal ID and password included in the login request information received via the transmission / reception unit 51 as search keys, and is identical to the terminal authentication management DB 5002. Terminal authentication is performed by determining whether or not the terminal ID and password are managed. The state management unit 53 stores the terminal ID of the request source terminal 10A, the operation status of the request source terminal 10A, the remote status in the terminal management DB 5003 (see FIG. 11) in order to manage the operation status of the request source terminal 10A that has requested login. The communication management system 50 stores and manages the reception date and time when the login request information is received and the IP address of the request source terminal 10 in association with each other.

  The terminal extraction unit 54 searches the destination list management DB 5004 (see FIG. 12) using the terminal ID of the request source terminal 10A that requested the login as a key, and is a candidate for the destination terminal 10B that can remotely communicate with the request source terminal 10A. The terminal ID is extracted by reading the terminal ID. Further, the terminal extraction unit 54 searches the destination list management DB 5004 (see FIG. 12) using the terminal ID of the request source terminal 10A that has requested login as a key, and sets the terminal ID of the request source terminal 10A as a candidate for the destination terminal 10B. As well as the terminal IDs of the other request source terminals 10A registered.

  The terminal status acquisition unit 55 searches the terminal management DB 5003 (see FIG. 11) using the candidate terminal ID of the destination terminal 10B extracted by the terminal extraction unit 54 as a search key, and is extracted by the terminal extraction unit 54. Read the operating status for each terminal ID. Thereby, the terminal state acquisition unit 55 can acquire the operation state of the candidate of the destination terminal 10B that can remotely communicate with the request source terminal 10A that has requested the login. Further, the terminal state acquisition unit 55 searches the terminal state management DB 5003 using the terminal ID extracted by the terminal extraction unit 54 as a search key, and also acquires the operating state of the request source terminal 10A that has requested login.

  The primary narrowing-down unit 56 performs a primary narrowing-down process before the final narrowing-down process in order to support a final narrowing-down process that finally narrows down from a plurality of relay apparatuses 30 to one relay apparatus 30. Therefore, the selection session ID generation unit 56a A terminal IP address extraction unit 56b, a primary selection unit 56c, and a priority determination unit 56d. Among these, the selection session ID generation unit 56 a generates a selection session ID used for executing a session for selecting the relay device 30. The terminal IP address extraction unit 56b, based on the terminal ID of the request source terminal 10A and the terminal ID of the destination terminal 10B included in the start request information sent from the request source terminal 10A, the terminal management DB 5003 (FIG. 11). The IP address of each corresponding remote communication terminal 10 is extracted. The primary selection unit 56c selects the relay device ID of the relay device 30 whose operation state is “ON line” from among the relay devices 30 managed by the relay device management DB 5001 (see FIG. 9). The relay device 30 is selected.

  Further, the primary selection unit 56c searches the relay device management DB 5001 (see FIG. 9) based on the IP address of the request source terminal 10A and the IP address of the destination terminal 10B extracted by the terminal IP address extraction unit 56b. By doing so, it is investigated whether each dot address of the IP address of the selected relay device 30 is the same as or different from each dot address in each IP address of the request source terminal 10A and the destination terminal 10B. Further, for each relay device, the primary selection unit 56c has the higher two points with the higher points of the integrated points obtained by integrating the higher points with respect to the remote communication terminal 10 at the address priority points and the transmission speed priority points. By selecting the relay device 30, the relay device 30 is further selected.

  In the present embodiment, the top two relay devices 30 with high points are selected. However, the present invention is not limited to this. If one relay device 30 can be narrowed down as many as possible, the top points with high points are selected. Three or more relay devices 30 may be selected.

  The priority determination unit 56d refers to the priority management DB 5006 (see FIG. 14), and determines an address priority point for each relay device 30 investigated by the primary selection unit 56c. Further, the priority determination unit 56d searches the priority management DB 5006 (see FIG. 15) based on the maximum data transmission rate of each relay device 30 managed by the relay device management DB 5001 (see FIG. 9). The point of the transmission rate priority is determined for each relay device 30 narrowed down by the first narrowing-down process by the primary selection unit 56c.

  The session management unit 57 stores the selection session ID generated by the selection session ID generation unit 56a, the terminal ID of the request source terminal, and the terminal ID of the destination terminal in the session management DB 5005 (see FIG. 13) of the storage unit 5000. Store and manage in association. In addition, the session management unit 57 determines, for each session ID for selection, the session management DB 5005 (see FIG. 13) of the relay device 30 that is finally selected by the final selection unit 16c of the remote communication terminal 10. Store and manage the relay device ID.

  The quality determining unit 58 searches the quality management DB 5007 (see FIG. 16) using the delay time as a search key and extracts the image quality of the corresponding image data, thereby determining the image quality of the image data to be relayed to the relay device 30. To do. The storage / read processing unit 59 is executed by the HDD 205 shown in FIG. 4, and stores various data in the storage unit 5000 or reads various data stored in the storage unit 5000. The delay time management unit 60 extracts the corresponding terminal ID by searching the terminal management DB 5003 (see FIG. 11) using the IP address of the destination terminal 10B as a search key, and further acquires the session management DB 5005 (see FIG. 13). ), The delay time indicated in the delay information is stored and managed in the field portion of the delay time in the record including the extracted terminal ID.

When the remote communication management system 50 receives a message from each remote communication terminal 10, for example, a message regarding mute, at the transmission / reception unit 51, the message transmission unit 61 analyzes the content and sends the message to the destination remote communication terminal 10. Manage to send.
<< Processing and Operation of Embodiment >>
The above is the description of the configuration and function (or means) of the remote communication system 1 according to the present embodiment. Subsequently, the processing method in the remote communication system 1 according to the present embodiment is described with reference to FIGS. 17 to 23. explain. FIG. 17 is a sequence diagram showing a process for managing status information indicating the status of each relay device 30 transmitted from each relay device 30 to the remote communication management system 1. FIG. 18 is a sequence diagram showing processing in a preparation stage for starting remote communication among a plurality of remote communication terminals 10. FIG. 19 is a sequence diagram illustrating processing for narrowing down the relay devices 30. FIG. 20 is a process flow diagram illustrating a process of narrowing down the relay devices 30. FIG. 21 is a diagram illustrating a point calculation state when the relay device 30 performs the narrowing-down process. FIG. 22 is a sequence diagram illustrating a process in which the remote communication terminal 10 selects the relay device 30. FIG. 23 is a process flow diagram showing a process of selecting the relay device 30 at the remote communication terminal. FIG. 24 is a sequence diagram showing processing for transmitting and receiving image data and audio data between remote communication terminals.

  First, a process for managing status information indicating the status of each relay device 30 transmitted from each relay device 30 to the remote communication management system 50 will be described with reference to FIG. First, in each relay device 30, the state detection unit 32 shown in FIG. 5 periodically detects the operating state of the relay device 30 as its own device (S1-1 to S1-4). In order to manage the operation state of each relay device 30 in real time on the remote communication management system 50 side, the transmission / reception unit 31 of each relay device 30 periodically sends each state to the remote communication management system 50 via the communication network 2. Information is transmitted (steps S2-1 to S2-4). Each state information includes the relay device ID for each relay device 30 and the operating state detected by the state detection unit 32 of the relay device 30 related to each relay device ID. In the present embodiment, the relay devices (30a, 30b, 30d) operate normally and are “ON line”, while the relay device 30c is in operation, but the relay operation of the relay device 30c. The case where some trouble has occurred in the program for executing the “OFF line” is shown.

  Next, in the remote communication management system 50, the transmission / reception unit 51 receives each status information sent from each relay device 30, and the relay device status management DB 5001 (in the storage unit 5000 via the storage / read processing unit 59). 9), state information is stored and managed for each relay device ID (steps S3-1 to S3-4). As a result, the operation status of “ON line”, “OFF line”, or “failing” is stored and managed for each relay device ID in the relay device management table as shown in FIG. The At this time, the reception date and time when the status information is received by the remote communication management system 50 is also stored and managed for each relay device ID. If the status information is not sent from the relay device 30, the field portion of the operation state and the field portion of the reception date and time in each record of the relay device management table shown in FIG. The operation status and the reception date and time at the previous reception are shown.

  Next, processing in a preparation stage before starting remote communication between the remote communication terminal 10aa and the remote communication terminal 10db will be described with reference to FIG. First, when the user turns on the power switch 109 shown in FIG. 2, the operation input accepting unit 12 shown in FIG. 5 accepts the power on and turns on the power (step S21). The login request unit 13 automatically transmits login request information indicating a login request from the transmission / reception unit 11 to the remote communication management system 50 via the communication network 2 in response to the reception of the power ON (step S22). ). This login request information includes a terminal ID and a password for identifying the remote communication terminal 10aa as the request source. These terminal ID and password are data read from the storage unit 1000 via the storage / read processing unit 19 and sent to the transmission / reception unit 11. When the login request information is transmitted from the remote communication terminal 10aa to the remote communication management system 50, the remote communication management system 50 that is the reception side can grasp the IP address of the remote communication terminal 10ab that is the transmission side. it can.

  Next, the terminal authentication unit 52 of the remote communication management system 50 uses the terminal ID and password included in the login request information received via the transmission / reception unit 51 as search keys, and the terminal authentication management DB 5002 (see FIG. 10) and terminal authentication is performed by determining whether the same terminal ID and password are managed in the terminal authentication management DB 5002 (step S23). Since the same terminal ID and password are managed by the terminal authentication unit 52, when it is determined that the login request is from the remote communication terminal 10 having a valid use authority, the state management unit 53 In the terminal management DB 5003 (see FIG. 11), the terminal ID of the remote communication terminal 10aa, the operating state, the reception date and time when the login request information is received, and the IP address of the remote communication terminal 10aa are stored in association with each other (step S24). Accordingly, in the terminal management table shown in FIG. 11, the remote communication terminal ID “01aa”, the operation state “ON line”, the reception date and time “2009.11.10.13:40”, and the terminal IP address “ "1.2.1.3" will be managed in association with each other.

  Then, the transmission / reception unit 51 of the remote communication management system 50 sends the authentication result information indicating the authentication result obtained by the terminal authentication unit 52 to the request source terminal 10aa that has made the login request via the communication network 2. Transmit (step S25). In the present embodiment, the case where the terminal authentication unit 52 determines that the terminal has a valid usage right will be described below.

  The terminal extraction unit 54 of the remote communication management system 50 searches the destination list management DB 5004 (see FIG. 12) using the terminal ID “01aa” of the request source terminal 10aa that requested the login as a search key, and performs remote communication with the request source terminal 10aa. The candidate terminal ID of the destination terminal 10B that can be extracted is read out (step S26). Here, the terminal IDs “01ab”, “01ba”, and “01db” of the destination terminals (10ab, 10ba, 10db) corresponding to the terminal ID “01aa” of the request source terminal 10aa are extracted.

  Next, the terminal state acquisition unit 55 uses the terminal IDs (“01ab”, “01ba”, “01db”) of the candidate destination terminal 10B extracted by the terminal extraction unit 54 as a search key, as a terminal management DB 5003 (FIG. 11) and by reading out the operating state (“OFF line”, “ON line”, “ON line”) for each terminal ID extracted by the terminal extraction unit 54, the remote communication terminals (10ab, 10ba) are retrieved. , 10db) is acquired (step 27).

  Next, the transmission / reception unit 51 uses the terminal ID (“01ab”, “01ba”, “01db”) as the search key used in step 27 and the operating status of the corresponding destination terminal (10ab, 10ba, 10db). The destination state information including (“OFF line”, “ON line”, “ON line”) is transmitted to the request source terminal 10aa via the communication network 2 (step S28). As a result, the request source terminal 10aa operates at the current operating state ("OFF line") of the remote communication terminals (10ab, 10ba, 10db) that are candidates for the destination terminal 10B that can remotely communicate with the request source terminal 10aa. , “ON line”, “ON line”).

  Further, the terminal extraction unit 54 of the remote communication management system 50 searches the destination list management DB 5004 (see FIG. 12) using the terminal ID “01aa” of the request source terminal 10aa that requested the login as a search key, and the request source terminal The terminal ID of the other request source terminal 10A in which the terminal ID “01aa” of 10aa is registered as a candidate for the destination terminal 10B is extracted (step S29). In the destination list management table shown in FIG. 12, the terminal IDs of the other request source terminals 10A to be extracted are “01ab”, “01ba”, and “01db”.

  Next, the terminal state acquisition unit 55 of the remote communication management system 50 searches the terminal state management DB 5003 (see FIG. 11) using the terminal ID “01aa” of the request source terminal 10aa that has made the login request as a search key. The operating state of the request source terminal 10aa that has requested login is acquired (step S30).

  Then, the transmission / reception unit 51 includes the terminal management DB 5003 (FIG. 5) among the remote communication terminals (10ab, 10ba, 10db) associated with the terminal IDs (“01ab”, “01ba”, and “01db”) extracted in step S29. 11), the remote communication terminals (10ba, 10db) whose operation state is “ON line” have the terminal ID “01aa” and the operation state “ON line” of the request source terminal 10aa acquired in step S30. The destination state information included is transmitted (steps S31-1, S31-2). When the transmission / reception unit 51 transmits the destination status information to the remote communication terminals (10ba, 10db), the terminal management table shown in FIG. 11 is based on each terminal ID (“01ba”, “01db”). Refer to the IP address of the terminal managed by. As a result, the terminal ID “01aa” of the request source terminal 10aa that requested the login and the operation status are sent to each of the other destination terminals (10db, 10ba) that can remotely communicate with the request source terminal 10aa that requested the login. Can communicate "ON line".

  On the other hand, in the other remote communication terminals 10 as well, when the user turns on the power switch 109 shown in FIG. 4, the operation input accepting unit 12 shown in FIG. Since the process similar to the process of steps S22 to S31-1, 2 is performed, the description thereof is omitted.

  Subsequently, a process of narrowing down the relay devices 30 will be described with reference to FIG. In the present embodiment, the request source terminal 10aa is a remote communication terminal (10ba, 10ba, whose operation state is ON line based on the destination state information received in step S28 among the remote communication terminals 10 as destination candidates. Remote communication with at least one of 10 db) is possible. Thus, hereinafter, a case will be described in which the user of the request source terminal 10aa selects to start remote communication with the destination terminal 10db.

  First, when the user presses the operation button 108 shown in FIG. 2 and selects the remote communication terminal 10db, the operation input receiving unit 12 shown in FIG. 5 performs remote communication with the remote communication terminal 10db. A request to start is accepted (step S41). The transmission / reception unit 11 of the remote communication terminal 10aa includes the terminal ID “01aa” of the request source terminal 10aa and the terminal ID “01db” of the destination terminal 10db, and includes start request information indicating that the remote communication is to be started. It transmits to the remote communication management system 50 (step S42). Thereby, the transmission / reception unit 51 of the remote communication management system 50 receives the start request information and grasps the IP address “1.2.1.3” of the request source terminal 10aa that is the transmission source. Then, based on the terminal ID “01aa” of the request source terminal 10aa and the terminal ID “01db” of the destination terminal 10db included in the start request information, the state management unit 53 uses the terminal management table of the terminal management DB 5003 (see FIG. 11). The field portion of the operating state of the record including the terminal ID “01aa” and the terminal ID “01db” is changed to “busy” (step S43). In this state, the request source terminal 10aa and the destination terminal 10db have not started remote communication (call), but are in a call state, and the other remote communication terminal 10 is remote from the request source terminal 10aa or the destination terminal 10db. When communication is attempted, a voice or display indicating a so-called busy state is output.

  Next, processing for executing a session for selecting the relay device 30 in steps S44 to S48 and steps S61-1 to S66 will be described. First, the selection session ID generation unit 56a generates a selection session ID used to execute a session for selecting the relay device 30 (step S44). Then, the session management unit 57 stores the selection session ID “se1” generated in step S44, the terminal ID “01aa” of the request source terminal 10aa, and the destination in the session management DB 5005 (see FIG. 13) of the storage unit 5000. The terminal ID “01db” of the terminal 10db is associated and stored and managed (step S45).

  Next, the primary screening unit 56 of the remote communication management system 50 relays remote communication between the request source terminal 10aa and the destination terminal 10db based on the relay device management DB 5001, the terminal management DB 5003, and the priority management DB 5006. The primary narrowing down of the relay device 30 is performed (step S46).

  Here, the process in step S46 will be described in more detail with reference to FIG. First, the terminal IP address extraction unit 56b is based on the terminal ID “01aa” of the request source terminal 10aa and the terminal ID “01db” of the destination terminal 10db included in the start communication information transmitted from the request source terminal 10aa. By searching the terminal management DB 5003 (see FIG. 11), the IP addresses (“1.2.1.3”, “1.3.2.4”) of the corresponding remote communication terminals (10aa, 10db) Is extracted (step S46-1). Next, the primary selection unit 56c selects the relay device (30a, 30b, 30d) that is “ON line” among the operating states of the relay device 30 managed by the relay device management DB 5001 (see FIG. 9). Each relay device ID (111a, 111b, 111d) is selected (step S46-2). Further, the primary selection unit 56c extracts the IP address “1.2.1.3” of the request source terminal 10aa and the IP address “1.3.2.4” of the destination terminal 10db extracted in step S46-1. ”To search the relay device management DB 5001 (see FIG. 9), and thereby each IP address (“ 1.2.1. ”) Of the relay devices (30a, 30b, 30d) selected in the above step S46-2. 2. ”,“ 1.2.2.2 ”,“ 1.3.2.2 ”) for each IP address (“ 1.2.1 ”) of the request source terminal 10aa and destination terminal 10db. .3 "," 1.3.2.4 "), it is checked whether it is the same as or different from each dot address (step S46-3).

  Next, the priority determination unit 57c refers to the priority management DB 5006 (see FIG. 14), and determines the point of the address priority for each relay device (30a, 30b, 30d) investigated in step 46-3. Is determined (step S46-4). When the result of this determination process is shown in a table, the state shown in FIG. 21 is obtained. FIG. 21 is a diagram illustrating a calculation state of priority points when the relay device 30 performs the narrowing-down process. In FIG. 21, for each relay apparatus ID, an address priority point, a transmission speed priority point, and an integration point are shown. Further, the address priority points further indicate a point for each requesting terminal 10aa and a point for the destination terminal 10db of each relay device 30. The integration point is the sum of the higher one of the two address priority points and the transmission speed priority point.

  In the present embodiment, the IP address “1.2.1.2” of the relay device 30a is “same as the same” as the IP address “1.2.1.3” of the request source terminal 10aa. Therefore, as shown in FIG. 21, the address priority point is “5”. Further, since the IP address “1.2.1.2” of the relay device 30a is “same.different.different.different” with respect to the IP address “1.3.2.4” of the destination terminal 10db, The address priority point is “1”. Further, the IP address “1.2.2.2” of the relay device 30b is “same, same, different, different” with respect to the IP address “1.2.1.3” of the request source terminal 10aa. The point of the address priority is “3”. Further, since the IP address “1.2.2.2” of the relay device 30b is “same.different.same.different” with respect to the IP address “1.3.2.4” of the destination terminal 10db, The address priority point is “1”. Furthermore, the IP address “1.3.2.2” of the relay device 30d is “same.different.different.different” with respect to the IP address “1.2.1.3” of the request source terminal 10aa. The point of the address priority is “1”. Further, the IP address “1.3.2.2” of the relay device 30d is “same. Same. Same. Different” with respect to the IP address “1.3.2.4” of the destination terminal 10db. The address priority point is “5”.

  Next, returning to FIG. 20, the priority determination unit 57d determines the priority management DB 5006 (FIG. 15) based on the maximum data transmission rate of each relay device 30 managed by the relay device management DB 5001 (see FIG. 9). By searching for (reference), the point of the transmission speed priority is determined for each relay device (30a, 30b, 30d) narrowed down by the first narrowing-down process in step S46-2 (step S46-5). In the present embodiment, as shown in FIG. 9, the maximum data transmission rate of the relay device 30a is 100 (Mbps). Therefore, referring to the transmission rate priority shown in FIG. The degree is 3 points. Similarly, if the maximum data transmission rate of the relay device 30b is calculated to be 1000 (Mbps), the transmission rate priority is 5 points. Similarly, if the maximum data transmission rate of the relay device 30d is calculated to be 10 (Mbps), the transmission rate priority is 1 point.

  Next, for each relay device (30a, 30b, 30d), the primary selection unit 56c selects the higher point of the remote communication terminals (10aa, 10db) and the point of transmission speed priority at the address priority point. Among the integrated points that have been integrated, the top two relay devices 30 with the highest points are selected (step 46-6). In the present embodiment, as shown in FIG. 21, the relay device IDs (111a, 111b, 111d) have the integration points “8”, “8”, and “5”, respectively. The relay device 30a related to “111a” and the relay device 30b related to the relay device ID “111b” are selected.

  When the narrowing-down process in step S46 is completed, the transmission / reception unit 51 illustrated in FIG. 5 relays the number of relay devices 30 thus narrowed down to the destination terminal 10db via the communication network 2. Device narrowing information is transmitted (step S47). The relay device narrowing information includes the number “2” of the relay devices 30 narrowed down in the step 46, the terminal ID “01aa” of the request source terminal 10aa, and the selection session ID “se1”. . As a result, the remote communication terminal 10db determines how many relay devices 30 are in the execution of the session with the selection session ID “se1” and which remote communication terminal 10 has requested to start a video conference. It is possible to grasp the IP address “1.1.1.2” of the remote communication management system 50 that is the transmission source of the relay device narrowing information.

  Then, the remote communication terminal 10db transmits reception completion information indicating that reception of the relay device narrowing information has been completed from the transmission / reception unit 11 to the remote communication management system 50 via the communication network 2 (step S48). This reception completion information includes the session ID “se1”. As a result, the remote communication management system 50 completes the transmission of the number of relay devices executed with the session ID “se1”, and the IP address “1.3.2.4” of the destination terminal 10db as the transmission source. Can be grasped.

  Next, a process in which the destination terminal 10aa selects the relay device 30 will be described with reference to FIG. First, before starting the video conference, the remote communication management system 50 provides pre-relay request information for requesting relay in advance to each of the relay devices (30a, 30b) narrowed down in step S46. Transmit (steps S61-1, 2). This pre-relay request information includes the session ID “se1”, the IP address “01aa” of the request source terminal 10aa, and the destination terminal 10db. As a result, the relay device (30a, 30b) can grasp which selection session it belongs to, what the request source terminal 10A is, and what the destination terminal 10B is, and a prior relay request. It is possible to grasp the IP address “1.1.1.2” of the remote communication management system 50 that is the information transmission source.

  Next, each of the relay devices (30a, 30b) sends its own device from the transmission / reception unit 31 to the request source terminal 10aa grasped in steps S61-1, 2 via the communication network 2 before the start of the video conference. Is transmitted to each relay apparatus (30a, 30b) as the prior transmission request information indicating that the prior transmission information including ping (Packet Internet Groper) described later is transmitted (steps S62-1, 2). This pre-transmission information includes the session ID “se1”. As a result, the request source terminal 10aa grasps that the pre-transmission information is transmitted to each relay device (30a, 30b) in the selection process of the relay device 30 executed with the session ID “se1”, and also transmits the pre-transmission. The IP address (“1.2.1.2”, “1.2.2.2”) of the relay device (30a, 30b) that is the transmission source of the request information can be grasped.

  Note that the IP address of the destination terminal 10db is sent to the relay device 10aa as in step 61-1, without notifying the request source terminal 10ba of the IP address of the destination terminal 10db directly from the remote communication management system 50. The relay device 10aa requests the request source terminal 10ba to transmit the pre-transmission request information to the own device (relay device 10aa) as in step 61-2. This is because the terminal 10 is not notified of the IP address of another remote communication terminal 10 to ensure security.

  Next, the request source terminal 10aa transmits the pre-transmission information from the transmission / reception unit 11 to the relay device (30a, 30b) via the communication network 2 (steps S63-1, 2). Prior to transmission of image data and audio data, this pre-transmission information is transmitted to the destination terminal 10db via each relay device (30a, 30b) instead of these image data and audio data, so that the request source terminal This is information used to measure the time required from transmission of 10aa to reception of the destination terminal db. In addition, the advance transmission information includes a ping for confirming that the request source terminal 10aa, the relay devices (30a, 30b), and the destination terminal 10db are communicably connected, and the advance transmission information from the request source terminal 10aa. And the session ID “se1” is included. Thereby, each relay device (30a, 30b) can grasp that the pre-transmission information has been sent in the execution of the session with the selection session ID “se1”, and is a transmission source of the pre-transmission information. The IP address “1.2.1.3” of the request source terminal 10aa can be grasped.

  Next, each relay device (30a, 30b) responds to the IP address “1.3.2.4” of the destination terminal 10db included in the pre-relay request information received in steps S61-1, 2 above. The prior transmission information is relayed (steps S64-1, 2). As a result, the destination terminal 10db can grasp that the pre-transmission information has been sent in the execution of the session with the session ID “se1”, and can also determine the relay device (transmission source) of the pre-transmission information (transmission source) 30a, 30b) IP addresses ("1.2.1.2", "1.2.2.2") can be grasped.

  Next, the final narrowing unit 16 of the destination terminal 10db finally narrows down to one relay device 30 that relays image data and audio data in the video conference based on the pre-transmission information (step 65).

  Here, the process in step S65 will be described in more detail with reference to FIGS. First, when the measurement unit 16a of the final narrowing unit 16 illustrated in FIG. 5 receives the pre-transmission information relayed by each relay device (30a, 30b) by the transmission / reception unit 11 of the remote communication terminal 10db. Is received (step S65-1). Next, the calculation unit 16b determines, for each pre-transmission information whose reception time is measured, from the transmission of each pre-transmission information based on the difference between the reception date and time and the transmission date and time included in the pre-transmission information. The time required until reception is calculated (step S65-2). Next, the final selection unit 16c determines whether or not all pieces of pre-transmission information corresponding to the number “2” of relay devices 30 scheduled to be relayed have been received in the execution of the session with the session ID “se1” (Step S65). -3). If not all have been received (NO), the final selection unit 16c determines whether or not a predetermined time (in this case, 1 minute) has elapsed since the pre-transmission information was received by the remote communication terminal 10db (step S65). -4). Further, if the predetermined time has not elapsed (NO), the process returns to step S65-1. On the other hand, when all the data is received in step 65-3 (YES) or when a predetermined time has elapsed (YES) in step S65-4, the final selection unit 16c has been calculated by the calculation unit 16c so far. One relay device 30 that relays the pre-transmission information that requires the shortest required time is selected (step S65-5). In the present embodiment, the relay device 30a is selected on the assumption that the pre-transmission information relayed by the relay device 30a has a shorter time from transmission to reception than the pre-transmission information relayed by the relay device 30b. Is shown.

  In the above-described embodiment, the relay device 30a is narrowed down on the destination terminal 10db side. However, the present invention is not limited to this, and the destination terminal 10db transmits the pre-transmission information to the request source terminal 10aa or the remote communication management system 50. By transmitting all the required time information indicating the required time until reception, one relay device 30a may be finally narrowed down on the request source terminal 10aa side or the remote communication management system 50 side.

  Next, the destination terminal 10db transmits selection information indicating that the relay device 30a has been selected from the transmission / reception unit 11 to the remote communication management system 50 via the communication network 2 (step S66). This selection information includes the session ID “se1” and the relay device ID “111a” of the selected relay device 30a. As a result, the remote communication management system 50 can grasp that the relay device 30a has been selected in the execution of the session with the session ID “se1”, and the IP address “1” of the remote communication terminal 10db that is the transmission source of the selection information. .3.2.4 ”.

  Next, the session management unit 56 of the remote communication management system 50 adds the above-mentioned final value to the field part of the relay device ID of the record including the session ID “se1” in the session management table of the session management DB 5005 (see FIG. 13). The relay device ID “111a” of the relay device 30a selected as one is stored and managed (step S67). Then, the transmission / reception unit 51 of the remote communication management system 50 transmits relay start request information indicating a request to start relaying to the relay device 30a via the communication network 2 (step S68). The relay start request information includes the IP addresses (“1.2.1.3” and “1.3.2.4”) of the request source terminal 10aa and the destination terminal 10db to be relayed. Thereby, the relay device 30a establishes a session for remotely communicating the three image data of low resolution, medium resolution, and high resolution and audio data between the remote communication terminals (10aa, 10db) ( Step S69). Accordingly, the remote communication terminals (10aa, 10db) can start a video conference.

  In step S47, the remote communication management system 50 transmits the relay device narrowing information to the destination terminal 10db, so that the relay device selection process is performed on the destination terminal 10db side through steps S48 to S64-1, 2. (Step S65) is performed, but the present invention is not limited to this. In Step S47, the remote communication management system 50 transmits the relay device narrowing information to the request source terminal 10aa, and then Steps S64-1, 2 are performed. Up to this point, the transmission source and reception source of each information may be switched between the request source terminal 10aa and the destination terminal 10db. As a result, the request source terminal 10aa can perform the relay device selection process instead of step S65, and can also transmit the selection information instead of step S66.

  Next, a process of transmitting and receiving image data and audio data to perform a video conference between the request source terminal 10aa and the destination terminal 10db will be described with reference to FIGS. First, the request source terminal 10aa transmits the image data of the subject imaged by the imaging unit 14a and the audio data of the audio input by the audio input unit 15a from the transmission / reception unit 11 to the relay device 30a via the communication network 2. (Step S81). In the present embodiment, the high-quality image data including the low resolution, the medium resolution, and the high resolution shown in FIG. 7 and the audio data are transmitted. Thereby, in the relay device 30a, the transmission / reception unit 31 receives the image data and the audio data of the three resolutions. Then, the data quality confirmation unit 33 searches the change quality management DB 3001 (see FIG. 8) using the IP address “1.3.2.4” of the destination terminal 10db as a search key, and the image quality of the corresponding relayed image data Is extracted to check the quality of the image data to be relayed (step S82). In the present embodiment, the image quality of the confirmed image data is “high quality” and is the same as the image quality of the image data received by the transmission / reception unit 31. The voice data is transferred to the destination terminal 10db (step S83). As a result, the destination terminal 10db receives the image data and the audio data at the transmission / reception unit 11, the image display control unit 14b causes the display 11 to display an image based on the image data, and the audio output unit 15b based on the audio data. Audio can be output.

  Next, the delay detection unit 17 of the remote communication terminal 10db detects the delay time of reception of the image data received by the transmission / reception unit 11 at regular time intervals (for example, every second) (step S84). In the present embodiment, the following description is continued for a case where the delay time is 200 (ms).

  The transmission / reception unit 11 of the destination terminal 10db transmits delay information indicating the delay time “200 (ms)” to the remote communication management system 50 via the communication network 2 (step S85). Thereby, the remote communication management system 50 can grasp the delay time and the IP address “1.3.2.4” of the remote communication terminal 10db that is the transmission source of the delay information.

  Next, the delay time management unit 60 of the remote communication management system 50 searches the terminal management DB 5003 (see FIG. 11) using the IP address “1.3.2.4” of the destination terminal 10db as a search key. The corresponding terminal ID “01db” is extracted, and further, in the session management table of the session management DB 5005 (see FIG. 13), the delay information field part of the record of the terminal ID “01db” is indicated by the delay information. The stored delay time “200 (ms)” is stored and managed (step S86).

  Next, the quality determination unit 58 searches the quality management DB 5007 (see FIG. 16) using the delay time “200 (ms)” as a search key, and extracts the image quality “medium image quality” of the corresponding image data. The image quality is determined as “medium image quality” (step S87).

  Next, the transmission / reception unit 51 uses the relay device ID “111a” associated with the terminal ID “01db” in the session management table of the session management DB (see FIG. 13) as a search key, the relay device management DB 5001 (FIG. 9) and the IP address “1.2.1.2” of the corresponding relay device 30a is extracted (step S88). Then, the transmission / reception unit 51 transmits quality information indicating the image quality “medium image quality” of the image data determined in step 87 to the relay device 30a via the communication network 2 (step S89). This quality information includes the IP address “1.3.2.4” of the destination terminal 10db used as the search key in step S86. Thereby, in the relay device 30a, the change quality management unit 34 stores the IP address “1.3.2” of the remote communication terminal 10 (destination terminal 10db in this case) of the transmission destination in the change quality management DB 3001 (see FIG. 8). .4 ”and the image quality“ medium image quality ”of the image data to be relayed are stored and managed in association with each other (step S90).

  Next, the remote communication terminal 10aa continues to transmit the high-quality image data including the low image quality, the medium image quality, and the high image quality, and the audio data to the relay device 30a in the same manner as in the above step S81 ( Step S91). As a result, in the relay device 30a, as in step S82, the data quality confirmation unit 33 uses the IP address “1.3.2.4” of the destination terminal 10db as a search key to change the quality management DB 3001 (see FIG. 8). ) And the image quality “medium image quality” of the corresponding image data to be relayed is extracted to check the image quality of the image data to be relayed (step S92). In this embodiment, since the image quality of the confirmed image data is “medium image quality” and lower than the image quality “high image quality” of the image data received by the transmission / reception unit 31, the data quality changing unit 35 The image quality of the image data is changed by suppressing the image quality from “high image quality” to “medium image quality” (step S93). Then, the transmission / reception unit 31 transmits the image data in which the image quality of the image data is changed to “medium image quality” and the audio data in which the sound quality of the audio is not changed to the remote communication terminal 10db via the communication network 2. (Step S94). In this way, when reception delay occurs at the destination terminal 10db that receives the image data, the relay device 30a changes the quality of the image so that the person participating in the video conference does not feel uncomfortable. can do.

  Next, processing when the microphone 114a is muted during a conference between the remote communication terminals 10 after the session is established will be described.

  The remote communication terminal 10 performs the mute of the microphone 114a and transmits the mute information via the communication network 2 in FIG. The following processing is described on the assumption that mute is performed in the remote communication terminal 10aa having the terminal ID “01aa” as the identifier and the IP address “1.2.1.3”, and the mute information is notified. To do.

The process shown in FIG. 25 is performed when the remote communication terminal 10aa logs in and the conference is started. In this case, an external microphone 114a is attached to the remote communication terminal 10aa.
1. The processing unit 18a of the mute processing unit 18 acquires the microphone name of the microphone 114a of the audio input unit 15a (step 101-1). In the present embodiment, this processing is performed on Windows (registered trademark), and the microphone name is acquired using the audio mixer API.
2. The processing unit 18a passes the acquired microphone name to the determination unit 18b and requests determination of the microphone mute function of the microphone 114a.
3. The determination unit 18b searches the list-format microphone data stored in the storage unit 18d to determine whether or not the microphone 114a requested by the processing unit 18a can notify mute information (step 101-2). An example of microphone data is shown in FIG. In the figure, microphone names that can be notified of microphone mute are registered.
4). If the acquired microphone name is present in the microphone data of FIG. 26A, the determination unit 18b assumes that the mute information can be notified, and the process <1> is not possible when the microphone name does not exist. The process <2> is performed. Each processing of <1> and <2> will be described later.
5. When the processing of <1> or <2> ends, the processing unit 18a refers to the identifier column of FIG. 27 and acquires other than 01aa that is its own terminal, that is, for example, 01ab to 01bb as a transmission destination (step 101-3). ). FIG. 27 shows the identifiers and mute states of other terminals in the conference. This data is stored in the terminal management DB 5003 of the remote communication management system 50 and transmitted from the remote communication management system 50 by XMPP.
6). The processing unit 18a performs the subsequent processing for the transmission destination acquired in 5.

    First, it is confirmed whether or not there remains a transmission destination to be processed (step 101-4).

    If there is a destination to be processed, the following 7. The subsequent processing is performed.

The process ends when there are no transmission destinations to be processed.
7). When the processing unit 18a requests the state management unit 18c to create a message when there is a destination to be processed, the state management unit 18c performs XMPP IM (Extensible Messaging and Presence Protocol: Instant Messaging and RFC) of RFC3921 as shown in FIG. An XML-based message in accordance with (Presence) is created (step 101-5).
8). Specifically, the state management unit 18c substitutes the identifier of the own terminal and the identifier of the transmission destination terminal for the attributes “from” and “to” of the <presence> tag. In the example of FIG. 28, 01aa is assigned to the identifier of the own terminal, and 01bb is assigned to the identifier of the destination terminal. The mute status is assigned to the <status> tag value. 28A to 28C, “MUTE_ON”, “MUTE_OFF”, and “MUTE_NONE” are assigned to indicate the mute states of “mute ON”, “mute OFF”, and “mute function disabled”, respectively. . In either case, the value of the <show> tag is substituted with “CHAT” indicating that a meeting is in progress.
9. The state management unit 18 c sends the transmission / reception unit 11 to the 8. When requesting the transmission of the message created in step 1, the transmission / reception unit 11 transmits the message to the remote communication terminal 10bb as the destination terminal by XMPP (step 101-6).

  10. When the transmission is completed, it is confirmed whether there are any transmission destinations to be processed (step 101-4).

  In this embodiment, the transmission / reception unit 11 converts the identifier of the own terminal and the identifier of the transmission destination terminal into respective IP addresses and transmits them.

  Also, the above-mentioned conversion can be prevented by directly substituting the IP address of the own terminal and the IP address of the destination terminal for the attributes “from” and “to” of the <presence> tag.

  Next, as another embodiment relating to the transmission of the message, the remote communication management system receives the message transmitted from the remote communication terminal, and the remote communication management system transmits this message to the remote communication terminal of the other party of the conference. A form is demonstrated.

  FIG. 29 shows an example of the message in this case. The IP address of the own terminal 10aa and the IP address of the remote communication management system 50 are substituted for the attributes “from” and “to” of the <presence> tag. In addition, the identifier of the own terminal is substituted for the value of the <id> tag.

  When this message is transmitted in step 101-6 of FIG. 25, the remote communication management system 50 receives the transmission / reception unit 51, and the message transmission unit 61 analyzes the “id” attribute of the <presence> tag according to XMPP IM, With reference to the session management table of FIG. 13, 01db which is the destination terminal of 01aa is specified. Next, referring to the terminal management table in FIG. 11, the IP address of “01db” “1.3.2.4” is specified.

Thereafter, the message transmission unit 61 creates the following message and requests the transmission / reception unit 51 to transmit the message to the specified transmission destination.
<presence from = “1.1.1.2” to = “1.3.2.4”>
<show> CHAT </ show>
<id> 01aa </ id>
<status> MUTE_ON </ status>
</ presence>
The above message shows an example in which the mute state is “MUTE_ON”, but in actuality, the <status> tag is set to the same value as the <status> tag sent from the source terminal. create.

  Thus, as shown in FIG. 29, when the IP address of the own terminal 10aa and the IP address of the remote communication management system 50 are respectively assigned to the attributes “from” and “to” of the <presence> tag, Then, the remote communication management system 50 specifies the IP address of the destination terminal and transmits it to the destination.

4. above. The process <1> will be described with reference to FIG. The process <1> is a process when the mute mute notification can be made when the mute button 114b provided in the microphone 114a of the voice input unit 15a is pressed.
1. The user is notified of mute information by pressing the mute button 114b, and determines whether or not the mute information has been notified (step 105-1).

        2. When the mute information is notified, the processing unit 18a analyzes the notified mute information and determines whether the notified mute state is ON or OFF (step 105-2).

        3. When the notified mute state is ON, the processing unit 18a requests the state management unit 18c to rewrite the mute state corresponding to the identifier 01aa of FIG. 27 to “MUTE_ON” (step 105-3).

            When the mute state is OFF, the state management unit 18c is requested to rewrite to “MUTE_OFF” (step 105-4).

4. After completing the requested processing, the state management unit 18c proceeds to the next processing.
Next, process <2> will be described with reference to FIG. The process <2> is a process in the case where the mute mute notification is impossible when the mute button 114b provided in the microphone 114a of the audio input unit 15a is pressed.
1. The processing unit 18a requests the discrimination unit 18b to search for the microphone name acquired in the processing of FIG. 25 with reference to the microphone name of the microphone data of FIG. 26B stored in the storage unit 18d.
2. If the microphone name is found (step 106-1), the determination unit 18b acquires the threshold value x corresponding to the microphone name from the microphone data in FIG. 26B (step 106-2), and Move to.

If the microphone name is not found (step 106-1), the processing unit 18a rewrites the mute state corresponding to the identifier 01aa in FIG. 27 to “MUTE_NONE” (step 106-9), and proceeds to the next processing in FIG. Move.
3. The determination unit 18b measures the audio signal level after passing through the mute switch 15a5 of FIG. 6 by a predetermined number (for example, 16 points, the time length when the sampling frequency is 16 kHz is 1 ms) (step 106-3).
4. The determination unit 18b determines whether all the levels measured in 3 are within a predetermined value range. In this embodiment, the digital format is 16 bits, with a sign, and the level is -x to x (for example, if the name of the microphone used is “MIC3”) compared to the threshold value x acquired in 2. It is determined whether it is within the range of x = 3), and the result is returned to the processing unit 18a.
5. When the above level is within the range, that is, when the level within the range continues for a predetermined time, the processing unit 18a requests the state management unit 18c to perform the process 6. Otherwise, processing 9 is requested (step 106-4).
6. The state management unit 18c refers to the mute state (FIG. 27) corresponding to 01aa in the storage unit 18d, and determines whether or not “MUTE_OFF”. If “MUTE_OFF”, If not "MUTE_OFF", return to 3. (Step 106-5).
7. The state management unit 18c rewrites the mute state to “MUTE_ON” and responds to the processing unit 18a to branch to the next processing in FIG. 25 (step 106-6).
8. The processing unit 18a branches to the next processing in FIG. 25, and returns to 3 to repeat the processing.
9. 5. When the level is not within the range, that is, when the level within the range does not continue for a predetermined time, the state management unit 18c refers to the mute state (FIG. 27) corresponding to 01aa of the storage unit 18d and determines whether or not “MUTE_ON”. Determine whether. If NO, return to 3. (Step 106-7).
10. When the mute state (FIG. 27) is “MUTE_ON”, the state management unit 18c rewrites the mute state to “MUTE_OFF” and responds to the processing unit 18a to branch to the next process (step 106-8). ).
11. The processing unit 18a branches to the next process and returns to 3 to repeat the process.
In the above process 4, the threshold value x may be measured in advance for the minimum input level of the microphone to be used, and a value smaller than this level may be stored.

  Here, even when the mute switch 15a5 is released and the signal passes, when the volume of the microphone 114a of FIG. 6 can be adjusted and the volume of the microphone 114a is set to the minimum, the audio signal level is Therefore, if this level is the same threshold as the state in which the mute switch 15a5 is turned on, the mute state cannot be determined. In order to cope with this problem, in the present embodiment, the mute switch 15a5 is provided at the last stage of other signal processing. With this configuration, it is possible to reliably distinguish the audio signal level when the mute switch 15a5 is turned on and the audio signal level when the mute switch 15a5 is released.

Next, processing of the remote communication terminal 10 that receives a message having mute information will be described with reference to FIG. In this embodiment, an example will be described in which a message is received by the remote communication terminal 10 having the identifier 01bb.
1. When the transmission / reception unit 11 receives a message having mute information (step 107-1), the processing unit 18a requests the state management unit 18c to perform the next process.
2. The status management unit 18c analyzes the received message and acquires the identifier of the transmission source (in this embodiment, 01aa from the “from” attribute of the <presence> tag. Also, the mute state is determined from the value of the <status> tag. Then, the mute state corresponding to 01aa in Fig. 27 is rewritten to the acquired mute state (step 107-2).
3. If the rewritten mute information is “MUTE_ON”, the processing unit 18a requests the image display control unit 14b to display the corresponding identifier 01aa and mute ON information, and the process ends (step 107-4).
4. If the rewritten mute information is not “MUTE_ON” (step 107-3) but “MUTE_NONE” (step 107-5), the processing unit 18a displays the display of the corresponding identifier and mute function disabled information as an image. The display control unit 14b is requested and the process ends (step 107-6).
5. If the mute information is “MUTE_OFF” (step 107-5), the processing unit 18a requests the image display control unit 14b to display the corresponding identifier and hide the mute information, and the process ends (step 107). -7).

  The screen display / non-display control described in 3 to 5 can be realized by using a well-known screen layer function. That is, a mute information display / non-display layer is created separately from the moving image layer transmitted from the relay device 30, and the mute information is displayed on the conference screen by superimposing this layer on the moving image layer. Can be displayed.

  FIG. 33 is a display example of the display 11 corresponding to the data of FIG. In this case, since the identifier 01aa is “MUTE_ON”, an icon indicating the information and the identifier are displayed.

  Since the identifier 01ab is “MUTE_OFF”, if the mute information icon of 01ab is displayed, it is hidden.

  Since the identifier 01ba is “MUTE_NONE”, an icon (gray display) indicating the information and the identifier are displayed.

  In the above embodiment, the mute information from the microphone 114a can be displayed to the other party even if the mute information function of the microphone 114a used for the remote communication terminal 10 is not supported by a device driver or the like, so that the conference can proceed smoothly. be able to. For example, even if the other party suddenly loses sound, it can be understood that it was muted, not broken.

  In addition, since it is not necessary to develop a device driver specialized for the remote communication terminal 10, the cost required for development can be saved.

  In addition, since a device driver is not required, a microphone preferred by the user can be used for the remote communication terminal 10 and the convenience for the user is improved.

  Next, processing for switching the remote communication terminal 10 between the video conference mode and the external microphone registration mode and newly registering unregistered microphones will be described.

  FIG. 34 is a menu screen displayed on the display 11 by the image display control unit 14b after the remote communication terminal 10 is activated. In the menu of this screen, the cursor is moved to a desired item by the operation button 107 (cursor key) of the operation input receiving unit 12, and the selection information can be input by the enter key (hereinafter, this input is performed). State is called “Select”).

  Here, when the item “start conference” is selected, the mode switching unit 18e of the mute processing unit 18 enters a mode for performing a video conference. During the conference, when the microphone is muted, the remote communication terminal on the other side 10 sends mute information.

On the other hand, when the item “microphone registration” is selected in FIG. 34, the remote communication terminal 10 performs the processing shown in FIG. 35 in the registration unit 18 f of the mute processing unit 18. Note that the processing in FIG. 35 is performed in a use state where a microphone to be registered is already attached to the remote communication terminal 10 and voice is input.
1. The processing unit 18a acquires the name of the microphone attached to the voice input unit 15a (step 202-1).

The registration unit 18f requests the processing unit 18a to display the dialog shown in FIG. 36, and the dialog screen is displayed. In addition, the acquirable flag of the registration unit 18f is set to FALSE. The acquirable flag indicates whether or not the microphone mute information can be acquired. If it can be acquired, it is set to TRUE, and if it cannot be acquired, it is set to FALSE.
2. When it is detected that the user has selected the cancel button on the dialog screen shown in FIG. 36, the screen returns to the menu screen shown in FIG. 34 and ends (step 202-2).

The reason why the user selects the cancel button is when the user wants to register the microphone 114a without the mute button (without the mute function) or when the user wants to cancel the microphone registration process.
3. When the user presses the mute button 114b attached to the microphone 114a attached on the dialog screen shown in FIG. 36, the process continues (step 202-2), and the registration unit 18f displays the microphone mute information. It is determined whether or not acquisition is possible (step 202-3). The user presses the mute button 114b and then presses the OK button.

    In this case, the determination that the microphone mute information can be acquired is made by the fact that the registration unit 18f has actually acquired the microphone mute information indicating that the mute button 114b has been pressed. When the registration unit 18f can acquire microphone mute information, the acquisition enable flag is set to TRUE.

The registration unit 18f requests the processing unit 18a to delete the dialog shown in FIG. 36. If the acquirable flag is TRUE, the registration unit 18f proceeds to processing <1>. If the acquirable flag is FALSE, the process proceeds to process <2>.
4. When the process <1> or the process <2> is completed, the registration unit 18f requests the processing unit 18a to display the dialog shown in FIG.

If it is detected on this screen that the user has selected the OK button, the screen moves to the menu screen of FIG. 34 and ends (step 202-4).
In the present embodiment, the above processing is performed on Windows (registered trademark), and processing 1 and 4 are realized by using an audio mixer API.

FIG. 38 is a diagram for explaining the flow of the process <1>. In process <1>, microphone mute information can be acquired (acquisition flag is TRUE), and when the registration unit 18f detects that the “OK” button in FIG. The request is made to the unit 18a, and the microphone name acquired in step 202-1 of FIG. 35 is added to the microphone data of the storage unit 18d (step 205-1). For example, if the acquired microphone name is “MIC2”, the acquired microphone name “MIC2” is added to the list of microphone names shown in FIG. In this figure, the microphone name “MIC1” has already been registered, and “MIC2” is added to this.
The data is referred to as a microphone capable of notifying microphone mute information in the process of FIG.

Next, the flow of process <2> will be described. FIG. 39 is a diagram for explaining the flow of the process <2>.
1. When the microphone mute information cannot be acquired (when the acquisition flag is FALSE) and the registration unit 18f detects that the “OK” button in FIG. 36 is pressed, the dialog is deleted. A request is made to 18a to proceed to the next processing.
2. The current volume input from the microphone 114a is acquired and stored, and then the volume is maximized (step 206-1).

By maximizing the volume, it is not necessary to store the predetermined value used in process 5 for each type of microphone. That is, by ensuring a sufficient input level when the microphone 114a is not in the mute state, the predetermined value can be handled in common by any microphone. This process is realized by using an audio mixer API of Windows (registered trademark).
3. Measure the digitized audio signal level for a predetermined time (eg 5 seconds). Since the digitization conditions are sampling frequency 16kHz and 16bit encoding, measure 80,000 points.
4. The level x_max that maximizes the absolute value | x | of all the measured levels x is obtained, and when this x_max is smaller than a predetermined value (for example, 100), this value is determined as a threshold value. (Step 206-2).
5. If x_max is larger than a predetermined value (for example, 100), the processing unit is requested to display the dialog shown in FIG. If it is detected that the OK button is pressed, the process returns to process 3 and the process is performed again.
6. The microphone name acquired in 1. of FIG. The threshold value x_max determined in step 1 is added to the microphone data in the storage unit 18d. For example, if the microphone name is “MIC4” and x_max is 4, it is added to the already registered data as shown in FIG. 26B (step 206-3).

In the example of this figure, the microphone name “MIC3” and its threshold value 3 are already registered, and “MIC4” and its threshold value 4 are added to this.
7. Use the audio mixer API to restore the microphone volume stored in 2 above (step 206-4).

  By the registration process described above, the mute information can be notified even when the external microphone 114a used for the remote communication terminal 10 is replaced with another microphone desired by the user.

  That is, the microphone 114a preferred by the user can be added for the remote communication terminal 10, and the convenience for the user is improved.

  Next, mute is executed by operating the volume change button 115b for changing the volume level from the audio output unit 15b, and when the state is changed, that is changed from the transmission / reception unit 11 to the destination remote communication terminal 10. The transmission process will be described. In this case, the volume change button 115b does not have to include the mute button 114b in the external microphone 114a in order to also serve as a mute execution operation button.

FIG. 41 shows processing for transmitting microphone mute information between the remote and remote communication terminals 10 after session establishment. The following processing will be described assuming that mute information is transmitted from the remote communication terminal 10aa to the remote communication terminal 10bb.
1. The monitoring unit 18g monitors an output level change event in the audio output unit 15b, and proceeds to the next processing upon receiving this change event (step 301-1).

      This change event can be realized by an audio mixer API well known in Windows (registered trademark) or the like, and the audio output unit 15b receives button information from the volume change button 115b received by the operation input receiving unit 12. It is generated when analyzing and changing the output level.

That is, this event occurs when the user presses the volume change button 115b of the speaker 115a. In the subsequent processing, processing accompanying access to the voice input / output I / F 116 such as access to the voice input unit 15a and the voice output unit 15b is realized by the audio mixer API.
2. The monitoring unit 18g acquires the changed output level from the audio output unit 18b, and compares whether or not the level is the minimum value (for example, level 0). If the output level is the minimum value, the monitoring unit 18g proceeds to the next process, and otherwise proceeds to the process 7 (step 301-2).
3. The monitoring unit 18g performs mute of the audio input unit 15a using the audio mixer API, and ignores the input from the audio input unit 15a (step 301-3).
4. The monitoring unit 18g sets the mute state MUTE as a variable in the storage unit 18d to MUTE = ON (step 301-4), and the processing unit 18a mutes the screen of its own display 11 to the image display control unit 14b. Request an icon and text to show that is turned on. FIG. 46 shows a display example of the screen.

The image display control unit 14b refers to the icon and text (muted) that are stored in the storage unit 1000 via the storage / readout control unit 19, and indicates that the screen is on the display 11 itself. Are displayed on top of each other.
5. The state management unit 18c creates a message indicating mute on as shown in FIG. 42 (step 301-5). This message is an XML-based message that conforms to XMPP IM (Extensible Messaging and Presence Protocol: Instant Messaging and Presence) of RFC3921, and specifically, the state management unit 18c uses the attributes “from” and “to” of the <presence> tag. Substitute the identifier of the own terminal and the identifier of the counterpart terminal for.
Substitute the mute status for the <status> tag value. 42A and 42B, “MUTE_ON” and “MUTE_OFF” are assigned, respectively, indicating the mute state of “mute on” and “mute off”, respectively. In either case, the value of the <show> tag is substituted with “CHAT” indicating that a meeting is in progress.

  Also in this embodiment, as described above, the remote communication management system receives the message transmitted from the remote communication terminal, and the remote communication management system transmits this message to the remote communication terminal of the other party of the conference. can do.

  FIG. 43 shows an example of the message in this case. As in the above case, the IP address of the own terminal 10aa and the IP of the remote communication management system 50 are set in the attributes “from” and “to” of the <presence> tag. Each address is assigned, and the identifier of the terminal is assigned to the <id> tag value.

  6. When the state management unit 18c requests the transmission / reception unit 11 to transmit the message created in process 5, the transmission / reception unit 11 transmits the message to the remote communication terminal 10bb which is the destination terminal by XMPP, and ends (step 301-6). ).

  7. If the output level is not the minimum value, compare whether the mute status is MUTE = ON. If MUTE = ON, the process proceeds to the next process. If not, the process ends (step 301-7).

  8. When MUTE = ON, the monitoring unit 18g executes the mute release of the audio input unit 15a using the audio mixer API, and can accept the input from the audio input unit 15a (Step 301-). 8).

  9. The monitoring unit 18g sets the mute state MUTE as a variable in the storage unit 18d to MUTE = OFF (step 301-9), and the image display control unit 14b displays the screen of its own display 11 shown in FIG. Request to erase the icon and text indicating that the displayed mute is on.

  10. The state management unit 18c creates a message indicating mute-off as shown in FIG. 43 (b) (step 301-5). When the transmission of the created message is requested, the transmission / reception unit 11 sends the message at the destination terminal. It transmits to a certain remote communication terminal 10bb and ends (step 301-6).

    Through the above processing, even if the microphone 114a of the remote communication terminal 10 is not equipped with a mute button, it is determined that the operator has minimized the output level of the speaker 115a, and mute of the microphone is performed and mute information is transmitted. It can be carried out.

  In other words, the process in the remote communication terminal 10 according to the present embodiment is interrupted when the operator minimizes the output level of the speaker 115a. In such a case, the user is interrupted. It can be said that the user does not intend to hear the other party's voice and wants to interrupt the conversation, and sends mute information of the microphone for indirect transmission of the interruption.

  Further, by transmitting this mute information, it is possible to inform the other party that the system is not malfunctioning, so that there is also an effect of giving a sense of security.

  Process 3 above. This is applicable when the audio input / output I / F 116 supports a mute interface. If the audio input / output I / F 116 does not support a mute interface, the input level may be set to a minimum value (for example, input level 0).

Next, a modification for realizing this processing will be described with reference to the flowchart of FIG.
1. The monitoring unit 18g monitors an output level change event in the audio output unit 15b, and proceeds to the next process when receiving the event (step 303-1).
2. The monitoring unit 18g acquires the changed output level from the audio output unit 15b, and compares whether or not the level is the minimum value (for example, level 0). If the output level is the minimum value, the monitoring unit 18g proceeds to the next step; (Step 303-2).
3. When the output level is the minimum value, the monitoring unit 18g acquires the input level from the voice input unit, and stores the input level in INPUT_LEVEL as a variable in the storage unit 18d (step 303-3).
4. The monitoring unit 18g sets the input level of the audio input unit 15b to the minimum value (for example, level 0) using the audio mixer API (step 303-4).
5. The monitoring unit 18g sets the mute state MUTE as a variable in the storage unit 18d to MUTE = ON (step 303-5), and the image display control unit 114b displays the screen of its own display 11 as shown in FIG. Request to display an icon and text indicating that mute is turned on.
6. The state management unit 18c creates a message indicating mute on as shown in FIG. 43 (a) (step 303-6).
7. When the state management unit 18c requests the transmission / reception unit 11 to transmit the message created by the preprocessing, the transmission / reception unit 11 transmits the message to the remote communication terminal 10bb which is the destination terminal by XMPP, and ends (step 303). -7).
8. If the output level is not the minimum value, it is confirmed whether or not the mute state is MUTE = ON (step 303-2). If MUTE = ON, the process proceeds to the next process, and if not, the process ends (step 303-8).
9. If MUTE = ON, process 3. Referring to the input level stored in step 1, the audio mixer API sets (returns) the input level of the audio input unit 15a to the input level (step 303-9).
10. The monitoring unit 18g sets the mute state MUTE as a variable in the storage unit 18d to MUTE = OFF (step 303-10), and the processing unit 18a displays the self display shown in FIG. 46 on the image display control unit 14b. 11 is requested to delete the icon and text indicating that the mute displayed on the screen 11 is turned on.
5. The state management unit 18c creates a message indicating mute-off as shown in FIG. 43 (b) (step 303-6). When the transmission of the created message is requested, the transmission / reception unit 11 sends the message at the destination terminal. It transmits to a certain remote communication terminal 10bb and ends (step 303-7).

As a result of the above processing, in addition to the mute button 114b not being provided in the microphone 114a of the remote communication terminal 10, the operator can set the speaker output level even if the audio input / output I / F 116 does not support mute control. It is possible to operate as an alternative function of mute by judging the minimization and minimizing the audio input level.
41 and 44 can be appropriately selected by determining whether or not the audio input / output I / F of the audio input unit supports the mute function at the time of activation. . FIG. 45 shows a flowchart for performing this selection.
1. The monitoring unit 18g checks whether or not the audio input / output I / F 116 of the audio input unit 15a supports the mute function. Processing if not supported 3. (Step 304-1). This processing can be realized by acquiring the properties of the audio input / output I / F 116 with the audio mixer API and referring to the attribute of the mute function support.

    2. If the audio input / output I / F 116 supports the mute function, the mute function as a variable in the storage unit 18d is set to MUTE_FUNC = ON and the process ends (step 304-2).

3. If the mute function is not supported, MUTE_FUNC = OFF is set and the process ends (step 304-3).
In order to selectively perform the processing of FIGS. 41 and 44, step 1. In addition, it may be determined by the value of MUTE_FUNC. That is, if MUTE_FUNC = ON, the subsequent processing is performed in the steps shown in FIG. 41, and if MUTE_FUNC = OFF, the subsequent processing may be performed in the steps shown in FIG.

  With the above processing, regardless of the presence or absence of the mute function of the voice input / output I / F 116 of the voice input unit 15a, it is determined that the user has minimized the speaker output level and the mute information is transmitted to the partner terminal. Can do.

Next, processing of a partner terminal (remote communication terminal 10bb) that receives a message having mute information will be described with reference to FIG.
1. The transmission / reception unit 11 receives the message (step 306-1).
2. The state management unit 18c analyzes the received message and acquires the mute state from the value of the <status> tag. (Step 306-2).
3. If the mute state is “MUTE_ON”, the processing unit 18a proceeds to the next process. Otherwise, process 5. (Step 306-3).
4). If “MUTE_ON”, the processing unit 18a requests the image display control unit 14b to display an icon indicating that mute is turned on and a text on the screen indicating the partner terminal on the screen of its own display 11 (step 306). -4).

FIG. 48 shows a display example of the screen. The image display control unit 14b refers to the icon and text (mute) stored in the storage unit 1000 via the storage / readout control unit 19 and displays the superimposed image on the screen indicating the partner terminal and ends. To do.
5. If it is not “MUTE_ON”, the processing unit 18a requests the image display control unit 14b to erase the icon indicating that the mute is turned on on the screen indicating the partner terminal and the text, and the process ends (step 306-5). ).

  With the above processing, the receiving side terminal can receive and display the mute information of the transmitting side terminal, so that it can surely know that the other side has been muted. That is, since it is possible to know that the system is not malfunctioning, it is possible to conduct a conference using the remote communication system with a sense of security.

114a Microphone 114b Mute button 115a Speaker 115b Volume change button 10 Remote communication terminal 11 Transmission / reception unit 15a Audio input unit 15b Audio output unit 16 Final narrowing unit 16a Measurement unit 16b Calculation unit 16c Final selection unit 17 Delay detection unit 18 Mute processing unit 18a Processing unit 18b Determination unit 18c Status management unit 18d Storage unit 18e Mode switching unit 18f Registration unit 18g Monitoring unit
DESCRIPTION OF SYMBOLS 30 Relay apparatus 31 Transmission / reception part 32 State detection part 33 Data quality confirmation part 34 Change quality management part 35 Data quality change part 50 Remote communication management system 51 Transmission / reception part 52 Terminal authentication part 53 State management part 54 Terminal extraction part 55 Terminal state acquisition part 56 Primary narrowing down part 56a Selection session ID generation part 56b Terminal IP address extraction part 56c Primary selection part 56c
56d Priority Determination Unit 57 Session Management Unit 58 Quality Determination Unit 60 Delay Time Management Unit 1000 Storage Unit 3000 Storage Unit 3001 Change Quality Management DB
5000 storage unit 5001 relay device management DB
5002 Terminal authentication management DB
5003 Terminal management DB
5004 Destination list management DB
5005 Session management DB
5006 Priority management DB
5007 Quality Management DB

JP 2008-61060 A JP 2008-28885 A JP 2007-274369 A

Claims (11)

A communication terminal for bidirectionally communicating at least voice in communication with a communication terminal of one or more other parties via a network, the communication terminal including a voice input unit for inputting voice and the communication of the other party A sound output unit that outputs sound sent from the terminal, a volume change operation unit that operates to change a volume level output by the sound output unit, and the sound output unit that is changed by the operation of the volume change operation unit A monitoring unit that monitors the volume level output to the terminal, and a transmission / reception unit that performs communication via a network,
When the volume level is the minimum value, the monitoring unit mutes the input of the audio input unit, creates a message with the destination terminal information of the other party and the information on the mute state, and transmits / receives the message A communication terminal, which requests a part to send the message.   2. The communication terminal according to claim 1, wherein the monitoring unit is configured such that the input of the audio input unit is in a mute state, and the volume level output to the audio output unit is changed to a value other than the minimum value by the operation of the volume changing operation unit. The mute state of the voice input unit is restored, a message with the destination terminal information of the other party and the information of the return state added is created, and the transmission / reception unit is requested to transmit the message. A characteristic communication terminal.   The communication terminal according to claim 1, wherein the monitoring unit stores the input level of the voice input unit when the input of the voice input unit is muted, and then puts the input into the mute state.   4. The communication terminal according to claim 3, wherein the monitoring unit is configured such that the input of the audio input unit is in a mute state, and the volume level output to the audio output unit is changed to a value other than the minimum value by the operation of the volume changing operation unit. The mute state of the voice input unit is restored to the stored input level, and a message is added to the transmission / reception unit by adding the destination terminal information of the other party and the information of the return state. A communication terminal that requests transmission of a message.   3. The communication terminal according to claim 1, wherein the mute state is a state in which voice input to the voice input unit is stopped.   5. The communication terminal according to claim 3, wherein the mute state is a state where the input level of the voice input unit is a zero value or a vicinity thereof.   In the communication terminal according to claim 1 or 2, the monitoring unit determines whether or not the voice input unit has an interface capable of canceling voice input. A communication terminal characterized in that the input level of the voice input unit is set to a zero value or in the vicinity thereof when the voice input unit stops the voice input in the mute state and does not have the interface.   8. The communication terminal according to claim 1, wherein the communication terminal includes an image display control unit that processes and displays the received content of the message, and the monitoring unit sends a message reception notification from the transmission / reception unit. When received, the communication terminal analyzes a voice input state included in the message and requests the image display control unit to display or hide data corresponding to the input state. 9. The communication terminal according to claim 8, wherein the image display control unit displays the data when the voice input state is a mute state . 9. The communication terminal according to claim 8, wherein the image display control unit does not display the data when the voice input state is not the mute state .   A communication terminal program for causing a communication terminal to realize the function according to any one of claims 1 to 10.

Images