JP6006737B2 - Display control method, display control apparatus, display control terminal, and display control program - Google Patents

Description

  The present invention relates to a technique for controlling display of digital content.

  A technique of using one service with a plurality of terminals is known. For example, Non-Patent Document 1 describes an example in which a plurality of users access a single remote display at the same time to perform a collaborative work or a battle game.

  When different users use one service using each terminal, for example, when an operation such as clicking different buttons occurs at the same time, an operation conflict occurs. If an operation conflict occurs, the user's operation may be hindered and an operation unintended by the user may occur.

  To deal with this problem, Non-Patent Document 1 avoids an operation conflict by giving an operation authority called a token to the user. For example, when using an Othello game, a token is alternately given to the user for each turn, or a token is stripped if no operation is performed within a certain time.

Upward, 4 others, "Access control mechanism for simultaneous use of multiple users in remote display environment", Transactions of Information Processing Society of Japan, Information Processing Society of Japan, February 2002, Vol. 43, No. 2, pp. 543-554

  In the Othello game, the timing for moving the token is clear, and the user does not need to explicitly release the token. However, when all users have the authority to operate equally in collaborative work and the like, it is necessary for the users themselves to explicitly acquire and release tokens, and there is a problem that the operation becomes complicated.

  The present invention has been made in view of the above, and an object of the present invention is to appropriately avoid a collision of operations without complicating operations when displaying the same content by linking a plurality of terminals.

The display control method according to the first aspect of the present invention is a display control method for controlling display of a plurality of terminals so that an operation on each of the plurality of terminals is reflected on each of the plurality of terminals. Based on the utterances of each user who operates each, the operation right is moved between the users, and the operation at each of the plurality of terminals is reflected on each of the plurality of terminals regardless of the presence or absence of the operation right. When an operation is performed on each of the plurality of terminals at the same time within a predetermined time, an operation by a user having the operation right is reflected on the plurality of terminals .

In the above display control method, when the speech time of any user is equal to or greater than a predetermined threshold value, characterized in that moving the operation right to the user.

In the display control method, any one of the steps of: receiving operation information from the terminal; receiving user utterance information from the terminal; measuring the user utterance time based on the utterance information ; When the user's utterance time becomes equal to or greater than a predetermined threshold, the step of moving the operation right to the user, and the step of controlling the display of the plurality of terminals based on the operation information, The step of controlling the display controls the display of the plurality of terminals based on the operation information received from the terminal operated by the user having the operation right when the operation information is received from different terminals within a predetermined time. and to Turkey and features.

A display control device according to a second aspect of the present invention is a display control device that controls display of a plurality of terminals so that an operation on each of the plurality of terminals is reflected on each of the plurality of terminals. Operation information receiving means for receiving information, utterance information receiving means for receiving user utterance information from the terminal, and measuring the user's utterance time based on the utterance information, and the utterance time of any user An operation right moving means for moving an operation right to the user when the predetermined threshold value is exceeded, and a display control means for controlling display of the plurality of terminals based on the operation information. Table of the plurality of terminals based on the control means, when receiving the operation information from the different terminals within a predetermined time, the operation information received from the terminal operated by the user with the operation authority To control the features and Turkey.

A display control terminal according to a third aspect of the present invention is a display control terminal that controls the display of the plurality of terminals and the terminal itself so that an operation on each of the plurality of terminals is reflected on each of the plurality of terminals. Operation input means for inputting an operation on the control terminal, voice input means for inputting a voice uttered by a user operating the display control terminal, operation information receiving means for receiving operation information from the operation input means and the terminal, Utterance information receiving means for receiving the user's utterance information from the voice input means and the terminal, and measuring the user's utterance time based on the utterance information, wherein the utterance time of any user is equal to or greater than a predetermined threshold when it becomes, control and operation right moving means for moving the operation right to the user, the display of the plurality of terminals and the display control terminal based on the operation information A display control unit that, the, the display control means, said operation information and another from the different terminals within a predetermined time is received from the operation input unit in the operation information if the received and the predetermined time If the terminal has received the operation information, wherein the benzalkonium control the display of the plurality of terminals and the display control terminal based on the operation information of the user with the operation right.

A display control program according to a fourth aspect of the present invention is characterized in that a computer is operated as each means of the display control device or the display control terminal .

  According to the present invention, when a plurality of terminals are linked to display the same content, it is possible to appropriately avoid an operation conflict without complicating the operation.

It is a whole block diagram including the display control apparatus in this Embodiment. It is a figure which shows the example of the content displayed on the display part of each of several terminal. It is a flowchart which shows the flow of a process when the display control apparatus in this Embodiment receives the event regarding operation. It is a flowchart which shows the flow of a transfer process of the operation right of the display control apparatus in this Embodiment. It is a figure explaining movement of utterance time and an operation right. It is another figure explaining movement of speaking time and operation authority. It is a whole block diagram of the system which distributes and displays a part of Web content which the display control apparatus in this Embodiment acquired.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is an overall configuration diagram including a display control device 1 according to the present embodiment. The display control device 1 shown in the figure is a device that connects a plurality of terminals 2A and 2B and coordinates display between the terminals 2A and 2B. Specifically, the display control apparatus 1 displays the same content on each of the terminals 2A and 2B, and the user operates the terminals 2A and 2B to change the screen of the content, or a change has occurred in the displayed content. In this case, the other terminals 2A and 2B that are not operated also change the display in cooperation.

  The terminals 2A and 2B include an input unit 21, a display unit 22, a communication unit 23, and a microphone 31.

  The input unit 21 inputs an instruction from the user. For example, an input device such as a mouse or a touch panel is provided, and an operation for content displayed on the display unit 22 is input, and the input operation is transmitted to the display control device 1.

  In addition, the input unit 21 connects the microphone 31, inputs the user's voice, and transmits it to the other terminals 2 </ b> A and 2 </ b> B via the display control device 1.

  The display unit 22 displays content in cooperation with the other terminals 2A and 2B via the display control device 1. The content to be displayed may be received from the display control device 1 or may be stored in a storage device provided in the terminals 2A and 2B.

  The communication unit 23 communicates with the display control device 1.

  The display control device 1 includes a cooperation unit 11, an event reception unit 12, a collision determination unit 13, and an operation right movement unit 14.

  The cooperation unit 11 distributes content to be displayed in cooperation between the terminals 2A and 2B or instructs the content to be displayed.

  The event receiver 12 receives an event that has occurred in each of the terminals 2A and 2B. As types of events received from the terminals 2A and 2B, there are events related to operations on content and events related to user utterances.

  When the event receiving unit 12 receives an event related to the operation, the event receiving unit 12 operates the content according to the received event, and the linkage unit 11 notifies all the terminals 2A and 2B of the change that has occurred in the content, and displays it on the terminals 2A and 2B. Link the content that is being used.

  In addition, when receiving an event related to the utterance, the event receiving unit 12 notifies the operation right moving unit 14 to perform operation right transfer processing. The operation right is a right to preferentially operate the content, and when an event related to the operation of the content collides, the event of the user having the operation right is preferentially processed.

  The operation right moving unit 14 moves the operation right according to the user's utterance. In the present embodiment, when the user's utterance time exceeds a predetermined threshold, the operation right is transferred to the user.

  Each unit included in the display control device 1 and the terminals 2A and 2B may be configured by a computer including an arithmetic processing device, a storage device, and the like, and the processing of each unit may be executed by a program. This program is stored in a storage device included in the display control device 1 and the terminals 2A and 2B, and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or provided through a network.

  Next, cooperation between the terminals 2A and 2B will be described.

  FIG. 2 is a diagram illustrating an example of content displayed on the display unit 22 of each of the terminals 2A and 2B.

  In the example of FIG. 2, the display unit 22 of each terminal 2 </ b> A, 2 </ b> B includes an image display area 201 for displaying an image and three buttons 202, 203, 204 for selecting an image to be displayed in the image display area 201. Content is displayed. When the buttons 202, 203, and 204 are operated on either of the terminals 2A and 2B, the images A, B, and C corresponding to the operated buttons 202, 203, and 204 are displayed in the image display areas 201 of the terminals 2A and 2B, respectively. Is displayed. For example, when the button 203 corresponding to the image B is operated on the terminal 2A, the terminals 2A and 2B cooperate with each other via the display control device 1, and the image B is displayed in the image display area 201 of each of the terminals 2A and 2B. .

  Processing of the display control device 1 and the terminals 2A and 2B at this time is as follows. First, when the button 203 is operated on the terminal 2A, the terminal 2A notifies the display control apparatus 1 of an event indicating that the button 203 has been operated. The display control apparatus 1 executes processing corresponding to the operation of the button 203 and notifies the terminals 2A and 2B of an instruction to display the image B in the image display area 201. The terminals 2A and 2B that have received the notification display the image B in the image display area 201.

  As described above, since the operation performed on either one of the terminals 2A and 2B is reflected on the other terminals 2A and 2B, it is possible to have a conversation while operating the same screen between remote points. Become. However, if operations are performed simultaneously on both terminals 2A and 2B, the operations will collide. For example, when the button 203 is operated on the terminal 2A and the button 204 is operated on the terminal 2B at the same time, the display control apparatus 1 receives the event that the button 203 is operated and the event that the button 204 is operated. When the terminals 2A and 2B are operated at the same time, the event processing order may change depending on the order of the events received by the display control apparatus 1, and the operation may not be intended by the user.

  Therefore, in the present embodiment, an operation right is given to one of the users according to the user's utterance, and the terminals 2A, 2A, 2A, 2A, and 2B are separated at the same time within a predetermined time (for example, about several hundred milliseconds to several seconds). When an event related to an operation is received from 2B, it is determined that the operation has collided. Hereinafter, a process when the display control apparatus 1 receives an event related to an operation and an operation right transfer process will be described.

  FIG. 3 is a flowchart showing a processing flow when the display control apparatus 1 according to the present embodiment receives an event related to an operation. In this embodiment, when an event related to an operation is received, a collision determination process is performed, and then a process corresponding to the received event is executed.

  When the display control device 1 receives an event related to an operation from the terminals 2A and 2B (step S11), it is determined whether or not the operation collides (step S12). The event received by the display control device 1 includes information such as a terminal identifier for identifying the terminals 2A and 2B and the type of event that has occurred. The display control device 1 stores the received event together with the event occurrence time (or event reception time) in a storage device provided in the display control device 1, and sends events related to operations from different terminals 2A and 2B within a predetermined time. The operation collision is determined by checking whether or not it has been received. If an event related to an operation is received from different terminals 2A and 2B within a predetermined time, it is determined that the operation has collided.

  If the operation has collided (YES in step S12), it is determined whether or not the user of the terminal 2A or 2B in which the event has occurred has the operation right (step S13). If the user of the terminal 2A or 2B in which the event has occurred does not have the operation right (NO in step S13), the process is terminated without executing the process corresponding to the event.

  If the operation does not collide (NO in step S12), or if the user of the terminal 2A or 2B in which the event has occurred has the operation right (YES in step S13), processing corresponding to the received event is executed (step S14) The result of the process is reflected on all the terminals 2A and 2B (step S15). For example, in the example of FIG. 2, in the event that the button 203 corresponding to the image B is operated in the terminal 2A, the image B is displayed in the image display areas 201 of all the terminals 2A and 2B. When executing the process corresponding to the event, the event from the user terminal 2A, 2B of the user who does not have the operation right is executed after a predetermined time for determining the operation collision. When an event is received from the terminals 2A and 2B of the user having the operation right while waiting for execution of the process, the event of the user having the operation right is executed.

  Next, the operation right transfer process will be described.

  FIG. 4 is a flowchart showing a flow of operation right transfer processing in the present embodiment. In the present embodiment, the operation right of content is moved according to the utterance time of each user who operates terminals 2A and 2B. The user's speech is acquired by the microphone 31 provided in the terminals 2A and 2B. When the user starts utterance, an utterance start event is transmitted from the terminals 2A and 2B to the display control device 1 when the utterance is terminated. Further, during the utterance, an utterance event requesting transfer of the user's utterance data acquired by the terminals 2 </ b> A and 2 </ b> B to the other terminals 2 </ b> A and 2 </ b> B is transmitted to the display control device 1.

  When an event related to utterance is received from the terminals 2A and 2B (step S21), it is determined whether or not the user who is the speaker has the operation right (step S22) and whether or not the event is an utterance end event (step S22). Step S23). If the speaker has an operation right (YES in step S22) or an utterance end event (YES in step S23), the operation right is not transferred, and the process is terminated.

  Subsequently, it is determined whether or not it is an utterance start event (step S24).

  If it is an utterance start event (YES in step S24), the utterance time of the user of the terminal 2A, 2B that transmitted the event is cleared, and then measurement of the utterance time is started (step S28).

  If it is not an utterance start event, that is, if an utterance is in progress (NO in step S24), the utterance time is updated (step S25), and it is determined whether or not the utterance time is equal to or greater than a threshold (step S26).

  If the utterance time is equal to or greater than the threshold (YES in step S26), the operation right is transferred from the user having the current operation right to the speaker (step S27). Regarding the management of the operation right, the display control device 1 may manage the operation right and hold the identifier of the user who has the operation right and the terminal identifiers of the terminals 2A and 2B operated by the user, 2B may manage the operation right, and may describe the presence or absence of the operation right in an event related to the operation.

  If the utterance time is less than the threshold (NO in step S26), the operation right is not transferred and the process is terminated.

  With the above processing, the operation right is given to the user whose utterance time is equal to or greater than the threshold.

  Next, the utterance time and the movement of the operation right will be described using a specific example.

  FIG. 5 is a diagram illustrating an example in which the user A speaks when a user other than the user A has the operation right.

  Initially, the operation right is given to other users than the user A. When the user A starts speaking at time t11, the display control device 1 starts measuring the speaking time of the user A.

  Although user A's utterance ends at time t12, since the time from time t11 to time t12 is less than the threshold, the operation right does not move to user A.

  Thereafter, when the user A starts speaking again at time t13, the display control device 1 starts measuring the speaking time of the user A.

  When the utterance time of the user A reaches the threshold at time t14, the operation right is given to the user A. Note that when the utterance is interrupted as from time t12 to time t13, the measured utterance time is cleared, so that the operation right is given to the user A when the utterance time from time t13 reaches the threshold value. .

  Even after the utterance of the user A ends at the time t15, if the utterance time of another user does not exceed the threshold value, the operation right is still given to the user A.

  Although the user A starts speaking at time t16, since the user A already has the operation right, the process of moving the operation right such as the start of measurement of the utterance time is not performed.

  FIG. 6 is a diagram for explaining the movement of the operation right when the users A and B speak.

  Initially, the operation right is given to the user A. When user B starts speaking at time t22 after user A's speaking ends at time t21, display control device 1 starts measuring user B's speaking time.

  User B's utterance ends at time t23, but since the time from time t22 to time t23 is less than the threshold, the operation right does not move to user B.

  Thereafter, when the user B starts speaking at time t24, the display control apparatus 1 starts measuring the speaking time of the user B.

  When the utterance time of user B reaches the threshold at time t25, the operation right moves from user A to user B.

  At the time point when user B's utterance ends at time t26 and user A starts speaking at time t27, user B has the operating right.

  The operation right may be moved according to the utterance, but when the utterance time is less than the threshold, it is possible to prevent the operation right from moving in a short utterance such as a reply or nodding by not moving the operation right. It becomes.

  In the present embodiment, the terminals 2A and 2B are linked via the display control device 1. However, each of the units of the display control device 1 is provided with either the terminal 2A or 2B, and the display control device in the terminals 2A and 2B. 1 may be performed. When the terminals 2A and 2B are provided with each unit of the display control device 1, it is determined whether or not the operations of the terminals 2A and 2B and the operations of the other terminals 2A and 2B collide. In the present embodiment, two terminals 2A and 2B have been described as an example. However, the present invention can be similarly applied to a case where there are three or more terminals.

  Next, an embodiment in which Web contents are linked by the terminals 2A and 2B will be described.

  FIG. 7 is an overall configuration diagram of a system in which the display control apparatus 1 distributes and displays a part of Web content acquired by the display control apparatus 1 according to the present embodiment to the terminals 2A and 2B.

  The display control apparatus 1 acquires Web content from a server or the like, transmits a part of the acquired Web content to each of the terminals 2A and 2B, and displays a part of the Web content. The linkage unit 11, the event reception unit 12, the collision determination unit 13, and the operation right transfer unit 14 illustrated in FIG. 1 are configured so that the script execution unit 15 uses JavaScript (registered trademark) included in the Web content acquired by the display control device 1. Configured by executing. Communication between the terminals 2A and 2B is performed by configuring the communication unit 16 with WebSocket or the like.

  The terminals 2A and 2B are configured by a browser that operates on a portable terminal equipped with a microphone 31 or the like. The receiving unit 24 receives a part of the Web content from the display control device 1 and stores it in the storage unit 26. The notification unit 25 notifies the display control apparatus 1 of an event that has occurred due to a user operation or the like. The storage unit 26 stores a part of the Web content received from the display control device 1.

  Next, an operation in which the display control apparatus 1 in FIG. 7 displays Web contents on the terminals 2A and 2B will be described.

  The display control apparatus 1 stores HTML data and CSS data included in the acquired Web content in the storage unit 17 and sends JavaScript to the script execution unit 15.

  The storage unit 17 analyzes the HTML data, expands and stores it in a DOM having a tree-like data structure. The communication unit 16 transmits to the terminals 2A and 2B a DOM subtree corresponding to a part of the Web content to be displayed on each of the terminals 2A and 2B from the DOM stored in the storage unit 17. The DOM to be transmitted to the terminals 2A and 2B is determined based on, for example, the attribute specified by the HTML data tag. The terminals 2 </ b> A and 2 </ b> B receive the DOM from the display control device 1 and store it in the storage unit 26 and display it on the display unit 22. The display control apparatus 1 controls the display of the terminals 2A and 2B by transmitting the DOM to the terminals 2A and 2B or deleting the DOM held by the terminals 2A and 2B. Alternatively, the display is controlled by changing the attribute of the DOM element held by the terminals 2A and 2B.

  When the script execution unit 15 accesses the DOM element in the storage unit 17, the terminal 2A, 2B holding the DOM element to be accessed is notified and the DOM element held by the terminal 2A, 2B itself is accessed. . If there is a return value for access to the element, the return value is received from the terminals 2A and 2B.

  Further, when an event occurs in the terminals 2A and 2B, an event occurrence notification is received from the terminals 2A and 2B, the event is generated in the DOM in the storage unit 17, and the registered callback function is executed. The terminals 2A and 2B are provided with a function of notifying the display control apparatus 1 of the type of event that occurred when an event occurred in the DOM element and the identifier of the element in which the event occurred.

  Here, display control by the display control apparatus 1 of FIG. 7 will be described using a specific example. It is assumed that content for displaying the image shown in FIG. 2 is transmitted to each of the terminals 2A and 2B.

  When the button 203 corresponding to the image B is clicked on the terminal 2A of FIG. 2, a click event occurs in the element of the button 203 of the terminal 2A. The terminal 2A notifies the display control apparatus 1 that a click event has occurred in the button 203 element. When the display control device 1 receives the notification of the occurrence of the click event from the terminal 2A, the display control device 1 determines whether or not the operation collides with the other terminal 2B and script the callback function registered in the button 203 element of the storage unit 17 The execution unit 15 executes the process to display the image B in the image display area 201 on both the terminals 2A and 2B. By this processing, the image B is displayed on the display unit 22 of each of the terminals 2A and 2B. If the operations collide, the operation of the user having the operation right is executed, and the displays on the terminals 2A and 2B are updated.

  As described above, according to the present embodiment, when a plurality of terminals 2A and 2B are linked to display the same content, the operation right is moved according to the user's utterance time, and the user's operation collides. If this happens, the operation of the user with the operation right is processed, so that the operation of the user who is speaking is prioritized and the operation collision is appropriately performed without explicitly acquiring and releasing the operation right. It is possible to avoid it. As a result, for example, when the operator performs an operation while explaining to the user, the operator's operation can be prioritized.

  According to the present embodiment, by moving the operation right when the utterance time becomes equal to or greater than a predetermined threshold, the operation right does not move when the user simply responds or nods.

DESCRIPTION OF SYMBOLS 1 ... Display control apparatus 11 ... Cooperation part 12 ... Event receiving part 13 ... Collision determination part 14 ... Operation right transfer part 15 ... Script execution part 16 ... Communication part 17 ... Memory | storage part 2A, 2B ... Terminal 21 ... Input part 22 ... Display 23: Communication unit 24 ... Reception unit 25 ... Notification unit 26 ... Storage unit 31 ... Microphone 201 ... Image display area 202, 203, 204 ... Button

Claims (6)

A display control method for controlling display of a plurality of terminals so that an operation on each of the plurality of terminals is reflected on each of the plurality of terminals.
Based on the utterances of each user who operates each of the plurality of terminals, the operation right is moved between the users,
Regardless of the presence or absence of the operation right, the operation in each of the plurality of terminals is reflected in each of the plurality of terminals.
The display control method according to claim 1, wherein , when an operation is performed on each of the plurality of terminals at the same time within a predetermined time, an operation by a user having the operation right is reflected on the plurality of terminals . The display control method according to claim 1 , wherein when the utterance time of any user becomes equal to or greater than a predetermined threshold , the operation right is transferred to the user. Receiving operation information from the terminal;
Receiving user utterance information from the terminal;
Measuring the user's utterance time based on the utterance information;
When the utterance time of any user is equal to or greater than a predetermined threshold, moving the operation right to the user;
Controlling the display of the plurality of terminals based on the operation information,
In the step of controlling the display, when the operation information is received from different terminals within a predetermined time, the display of the plurality of terminals is performed based on the operation information received from the terminal operated by the user having the operation right. the display control method according to claim 2, wherein the control to Turkey. A display control device that controls display of the plurality of terminals so that an operation in each of the plurality of terminals is reflected in each of the plurality of terminals.
Operation information receiving means for receiving operation information from the terminal;
Utterance information receiving means for receiving user utterance information from the terminal;
Operation right moving means for measuring the user's utterance time based on the utterance information and moving the operation right to the user when the utterance time of any user exceeds a predetermined threshold;
Display control means for controlling display of the plurality of terminals based on the operation information,
The display control unit controls display of the plurality of terminals based on the operation information received from a terminal operated by a user having the operation right when the operation information is received from a different terminal within a predetermined time. display control device comprising a Turkey. A display control terminal that controls the display of the plurality of terminals and its own so that operations in each of the plurality of terminals are reflected in each of the plurality of terminals;
Operation input means for inputting an operation in the display control terminal;
Voice input means for inputting voice spoken by a user operating the display control terminal;
Operation information receiving means for receiving operation information from the operation input means and the terminal;
Utterance information receiving means for receiving user utterance information from the voice input means and the terminal;
An operation right moving means for measuring a user's utterance time based on the utterance information and moving an operation right to the user when the utterance time of any user exceeds a predetermined threshold;
Display control means for controlling the display of the plurality of terminals and the display control terminal based on the operation information,
When the display control means receives the operation information from a different terminal within a predetermined time and when the operation information received from the operation input means within a predetermined time and the operation information is received from another terminal , display control terminal based on the operation information of the user and wherein the benzalkonium control the display of the plurality of terminals and the display control terminal having the operation authority.   A display control program for operating a computer as each means of the display control device according to claim 4 or the display control terminal according to claim 5.

Images