JP2013161247A - Operation device, display device, remote operation system, method of controlling operation device, control program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a remote operation system with excellent responsiveness.SOLUTION: A tablet 100 includes: a position identifying unit 20 that displays, in its own display unit, image data obtained by capturing a television 200, and identifies, from the image data, the position of a first area representing a display unit of the television 200; a coordinate processing unit 21 that converts, on the basis of the position of the first area, an operation position coordinate of an operation for the own display unit into an operation position coordinate in the display unit of the display device; and a signal output unit 22 that outputs the converted operation position coordinate as an input signal of the television 200.

Description

  The present invention relates to an operation device, a display device, a remote operation system, a control method for the operation device, a control program, and a recording medium for the operation device to remotely operate the display device.

  2. Description of the Related Art Conventionally, a technique for remotely operating a remote display device by operating a nearby operation device has been widely used.

  Patent Document 1 discloses a mobile terminal device that functions as a pointing device for a display device (a device on the side that operates the display device, hereinafter referred to as an operation device). Specifically, the controller device transfers a map image created by itself to the display device, and causes the display device to display the map image. The operating device captures the display device on which the map image is displayed with the camera, and compares the captured image with the map image created by the own device. The controller device determines which part of the map image displayed on the display device the image captured by the camera corresponds to. Thus, the controller device can handle the determination result as designation of an input position or coordinates on the display device. This makes it possible to use the operating device (mobile phone) as a pointing device for the display device.

JP 2008-191868 (released on August 21, 2008)

  However, in the above-described conventional configuration, in order for the operating device to function as a pointing device of the display device, an image (map image) held by the operating device must be copied to the display device. Alternatively, it is necessary to copy the video (user interface) displayed on the display device to the operation device. Thus, transmission / reception of data for sharing video has a problem of increasing the communication load between apparatuses. If the video to be shared is large-capacity data such as a moving image, the communication load further increases, and the responsiveness deteriorates, so the above problem is particularly serious.

  Further, in the above-described conventional configuration, in order to designate a predetermined position of the video displayed on the display device, the operation device itself must be moved, and the operation becomes troublesome for the user. This problem is a particularly serious problem unless the operating device is a small portable device.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to realize an operating device that reduces communication load between devices and has excellent responsiveness in an operating device that remotely controls a display device. An operation device, a display device, a remote operation system, a control method for the operation device, a control program, and a recording medium are realized.

  In order to solve the above problems, the operating device of the present invention is an operating device for remotely operating a display device, and display control means for displaying image data obtained by imaging the display device on a display unit of the operating device; In the display unit of the operation device, position specifying means for specifying the position of the first region representing the display unit of the display device from the displayed image data, and an operation signal input to the first region Based on the position of the first area specified by the input receiving means and the position specifying means, the operation position coordinates indicating the position where the operation signal is received on the display unit of the operating device is displayed on the display device. A coordinate processing means for converting the operation position coordinates in the unit and a signal output for outputting the operation position coordinates converted by the coordinate processing means as an input signal to the display device It is characterized in that it comprises a stage.

  According to the above configuration, the display control means of the controller device displays image data obtained by imaging the display device that is the communication partner on the display unit of the controller device. The display portion of the display device should be reflected in the image data. A region in the image data showing the display unit of the display device is referred to as a first region. The position specifying means specifies the position of the first region on the display unit of the controller device (in the tablet coordinate system) based on the displayed image data. As a result, when the input receiving means receives a user operation regarding the content displayed on the display unit, it is determined whether the operated position is within the first area or outside the first area. The device can judge.

  Assume that the input receiving means receives an operation signal input by a user operation on the first area. In this case, the coordinate processing means specifies which position of the first area specified from the image data is specified by the user operation.

  Then, the coordinate processing means converts the operation position coordinates specified on the display unit of the operation device into operation position coordinates on the display unit of the display device. Since the coordinate processing means grasps the positional relationship between the position specified by the user operation and the first area (the area of the display unit of the display device) specified from the image data, the above-described conversion is executed. Can do.

  Finally, the signal output means outputs the operation position coordinates in the display unit of the display device, converted by the coordinate processing means, to the display device as an input signal of the display device.

  By referring to the operation position coordinates on the display unit of the display device, the display device can understand which position on the display unit of the display device has been operated. That is, the display device that has received the input signal from the operation device understands that the position indicated by the operation position coordinates on the display unit of the display device has been operated by the user. And it becomes possible to operate | move according to the content of the understood input signal.

  In this way, the operating device simply transmits the operation position coordinates to the display device, and transmits to the display device that a predetermined position on the display unit of the display device has been operated and responds thereto. Some operation can be performed by the display device. That is, the display device can be remotely operated by outputting an input signal from the operation device. When operating the display device remotely, the operating device does not need to understand what is displayed on the display device. Conversely, the display device does not need to understand what was displayed on the operation device when the operation was accepted. That is, in order to realize remote operation, it is not necessary to share a large amount of data regarding display contents between the display device and the operation device.

  As a result, it is possible to reduce the communication load between the devices and realize an operation device and a remote operation system with excellent responsiveness.

  In the operating device according to the present invention, the display unit of the operating device is formed integrally with an input unit for inputting the operation signal to constitute a touch panel, and the input receiving means is input by a touch operation on the touch panel. Preferably, the coordinate processing means converts the operation position coordinates on the touch panel of the contact operation into operation position coordinates on the display unit of the display device.

  According to the above configuration, when the user performs a touch operation on the display unit of the display device on the image data displayed on the display unit, the coordinate processing unit performs an operation on the display unit of the display device corresponding to the touch position. The operation position coordinates of the position are obtained. Then, the operation position coordinates are output to the display device as an input signal.

  That is, the user can remotely operate the display device as if the user directly touched the screen of the display unit of the display device. In this way, by using an operation device that supports a touch panel, even if the display device is a device that does not support a touch panel, a more natural pseudo-touch operation can be performed on the display device.

  The signal output means of the operating device is an input signal that instructs the display device to select an object displayed on the display unit of the display device that overlaps the operation position coordinates converted by the coordinate processing means. May be output.

  According to the above configuration, when the operation position coordinates are transmitted to the display device as an input signal, the display device can select an object corresponding to the position operated by the user that is being displayed on the display device.

  Further, the signal output means of the operation device moves the selected object displayed on the display unit of the display device to the position indicated by the operation position coordinate converted by the coordinate processing means. An input signal for instructing the apparatus may be output.

  According to the above configuration, it is possible to cause the display device to perform a process of moving the object selected by the display device to a position operated by the user.

  Furthermore, the input receiving means of the operating device receives an operation signal input to a second region other than the first region on the display unit of the operating device, in addition to the first region, The signal output means is an input for instructing the display device to transmit the object displayed on the display unit of the display device, which overlaps the operation position coordinate converted by the coordinate processing means, to the operation device. In the second area of the signal and the display unit of the operation device, the display device is caused to move the object selected by the operation signal to the position indicated by the operation position coordinate converted by the coordinate processing means. Alternatively, at least one of input signals instructed to the user may be output.

  The user operates both the first area showing the display unit of the display device and the other second area. The second area may be an area in the image data that represents the surrounding environment outside the display unit of the display device, or an area in which some information is displayed on the display unit of the operation device. Intuitively for the user, the first area is understood as an area for operating the display device, and the second area is understood as an area outside the display device, that is, an area for operating the operating device. Is natural. Therefore, it is easy for the user to understand naturally that information is transmitted and received between the operation device and the display device by the operation on both the above-described areas.

  According to the above configuration, when the input accepting unit accepts a user operation for both of the above areas, first, the signal output unit corresponds to the position specified by the user operation on the display unit of the display device. Is requested to the display device to be transmitted to the operating device. Thereby, the operating device can receive the object from the display device. That is, transmission / reception of an object from the display device to the operation device is executed by a user operation on both of the above areas. Second, the signal output means displays the object selected by the user operation among the objects displayed on the display unit of the operation device, corresponding to the position specified by the user operation. The display device is instructed to display at the position. Thereby, the display device can display the object displayed on the operation device on the display unit of the display device. That is, transmission / reception of an object from the operation device to the display device is executed by a user operation on both the above-described areas. The signal output means may be capable of performing both the first request and the second instruction described above, or may be configured to be able to perform only one of them.

  As described above, the operation device can receive an operation that is easy for the user to understand intuitively, remotely operate the display device, and can transmit and receive an object suitable for the operation. Here, in order to realize the remote operation, only an input signal (operation position coordinates) is required, and it is not necessary to share a large amount of data regarding display contents between the display device and the operation device. Object transmission / reception may be performed as appropriate regardless of the timing of remote operation.

  As a result, it is possible to reduce the communication load between the devices and realize an operation device and a remote operation system with excellent responsiveness.

  The input receiving means can receive an operation signal input by a drag operation from the first area to the second area as a user operation for both the above areas.

  This operation is an operation reminiscent of transmission / reception of an object from the display device to the operation device. By accepting the above operation, the controller device may perform remote transmission and reception of objects from the display device to the controller device by remotely operating the display device.

  Specifically, for example, the operating device includes an object processing unit that processes an object displayed on the display unit of the operating device, and the input receiving unit receives an operation signal in the first area. The coordinate processing means converts the operation position coordinates of the drag operation on the display unit of the operation device into operation position coordinates on the screen of the display device, and the signal output means is converted by the coordinate processing means. A first input signal for instructing the display device to select the object displayed on the display unit of the display device that overlaps the operation position coordinates and to transmit to the operation device is output, and the display of the display device is displayed. That the selected object displayed on the screen is moved to the position indicated by the operation position coordinate converted by the coordinate processing means. The object processing means outputs the second input signal instructing the apparatus, and the object processing means responds to the first input signal while the input accepting means accepts the operation signal in the second area. It is possible to display and move the object received from the position indicated by the operation position coordinates of the drag operation on the display unit of the operation device.

  Further, when the input receiving means stops receiving an operation signal in the second area, the signal output means sends the entity data associated with the object selected according to the first input signal to the operating device. A third input signal that instructs the display device to transmit to the display device may be output.

  Or the said input reception means can receive the operation signal input by the drag operation from the said 2nd area | region to the said 1st area | region as user operation with respect to both said area | regions.

  This operation is an operation reminiscent of transmission / reception of an object from the operation device to the display device. By accepting the above operation, the controller device may perform remote transmission and reception of objects from the controller device to the display device by remotely operating the display device.

  Specifically, for example, the operation device includes an object processing unit that processes an object displayed on the display unit of the operation device, and a transmission unit that transmits an object stored in the operation device to the display device. The object processing means selects an object that overlaps the operation position coordinates of the drag operation on the display unit of the operation device while the input reception means receives an operation signal in the second area. The selected object is moved to the position indicated by the operation position coordinate of the drag operation on the display unit of the operation device, and the transmission unit transmits the object selected by the object processing unit to the display device. While the input receiving means receives the operation signal in the first area, the coordinate processing is performed. The stage converts the operation position coordinates of the drag operation on the display unit of the operation device into operation position coordinates on the display unit of the display device, and the signal output unit converts the selected object into the coordinate processing unit. The fourth input signal for instructing the display device to display at the position of the display unit of the display device indicated by the operation position coordinates converted by the object is displayed, and the object displayed according to the fourth input signal It is conceivable to output a fifth input signal for instructing the display device to move to the position indicated by the operation position coordinate converted by the coordinate processing means.

  Further, when the input receiving unit stops receiving the operation signal in the first area, the transmitting unit transmits the entity data associated with the object selected by the object processing unit to the display device. May be.

  Alternatively, the input receiving means receives an operation signal input by a first drag operation started on the first area and an operation signal input by a second drag operation started on the second area. Can be accepted.

  The drag operation started simultaneously in both areas is an operation reminiscent of bidirectional object transmission / reception (object exchange) between the operation device and the display device. By accepting the above operation, the operating device may perform the object exchange between the operating device and the display device by remotely operating the display device.

  Specifically, for example, the operation device includes an object processing unit that processes an object displayed on the display unit of the operation device, and a transmission unit that transmits an object stored in the operation device to the display device. The position specifying means specifies the position of the first object that moves from the first area to the second area in accordance with the first drag operation on the display unit of the operating device, and The position of the second object that moves from the second area to the first area in accordance with the second drag operation is specified on the display unit of the operating device, and the position specifying means includes the first object and the first object. Until it is determined that two objects overlap each other, the signal output means outputs the operation position of the first drag operation converted by the coordinate processing means. A sixth input signal for instructing the display device to transmit the first object displayed on the display unit of the display device, which overlaps the coordinates, to the operation device is output, and is displayed on the display unit of the display device. A seventh input signal for instructing the display device to move the selected first object being displayed to the position indicated by the operation position coordinate converted by the coordinate processing means; Transmits the second object selected by the second drag operation to the display device, and outputs the signal when the position specifying means determines that the first object and the second object overlap each other. The means instructs the display device to display the second object displayed on the display unit of the operation device on the display unit of the display device. Outputting eighth input signal, the object processing means, the first objects received from the display device in response to the sixth input signal, it is conceivable to be displayed on the display unit of the operation device.

  Further, when the input receiving means stops receiving an operation signal in the first area and the second area, the signal output means sends the entity data associated with the first object to the operating device. A ninth input signal for instructing the display device to transmit may be output, and the transmission means may transmit the entity data associated with the second object to the display device.

  Furthermore, the operating device includes an object processing unit that processes an object displayed on the display unit of the operating device, and the input receiving unit receives an operation signal input to the second area. The object processing means hides the object selected by the operation signal in the second area of the display unit of the operation device, and the signal output means removes the selected object from the display device. You may output the 10th input signal which instruct | indicates with respect to the said display apparatus to display on a display part.

  Alternatively, the operating device includes object processing means for processing an object displayed on the display unit of the operating device, and the input receiving means receives an operation signal input by an operation on the object on the first area. The signal output means selects the object displayed on the display unit of the display device that overlaps the operation position coordinates converted by the coordinate processing means, and transmits to the operation device to the display device. Outputs a first input signal to instruct the display device to give a visual effect of changing or reducing the shape to the selected object displayed on the display unit of the display device The object processing means outputs the object received from the display device in response to the first input signal. The-object, and displays the operating position coordinates indicating the position of the operation on the display unit of the operation device may be imparted a visual effect of change or expand the shape.

  Since the first area or the second area is not sufficiently secured on the display unit of the operation device, an operation for moving from the first area to the second area or from the second area to the first area is accepted. A situation that cannot be considered.

  According to the configuration described above, even in the situation as described above, the operation device only receives a user operation for one area, and the operation device remotely operates the display device to transmit and receive objects between the operation device and the display device. Can be implemented.

  The display device remotely operated by the operation device of the present invention operates in accordance with the input signal received from the operation device of the present invention. Such a display device also falls within the scope of the present invention.

  A remote operation system constructed by the operation device of the present invention described above and the display device of the present invention described above also falls within the scope of the present invention.

  In order to solve the above-described problem, the control method for the operation device according to the present invention is a control method for the operation device that remotely operates the display device, and displays image data obtained by imaging the display device on the display unit of the operation device. A display control step, a position specifying step for specifying a position of a first region representing the display unit of the display device in the display unit of the operation device from the image data displayed in the display control step, An input receiving step for receiving an operation signal input to the first area, and the operation signal on the display unit of the operating device is received based on the position of the first area specified in the position specifying step. The coordinate processing step for converting the operation position coordinates indicating the position into the operation position coordinates in the display unit of the display device, and the input signal to the display device, It is characterized in that it comprises a signal output step of outputting the converted operation position coordinates at standard processing steps.

  The operating device may be realized by a computer. In this case, a control program for the operating device that causes the operating device to be realized by the computer by operating the computer as each of the above-described means, and recorded the program. Computer-readable recording media are also within the scope of the present invention. In addition, the display device may be realized by a computer. In this case, the display device control program for causing the display device to be realized by the computer by operating the computer as each unit of the display device, and A recorded computer-readable recording medium also falls within the scope of the present invention.

  In order to solve the above problems, the operating device of the present invention is an operating device for remotely operating a display device, and display control means for displaying image data obtained by imaging the display device on a display unit of the operating device; In the display unit of the operation device, position specifying means for specifying the position of the first region representing the display unit of the display device from the displayed image data, and an operation signal input to the first region Based on the position of the first area specified by the input receiving means and the position specifying means, the operation position coordinates indicating the position where the operation signal is received on the display unit of the operating device is displayed on the display device. A coordinate processing means for converting the operation position coordinates in the unit and a signal output for outputting the operation position coordinates converted by the coordinate processing means as an input signal to the display device It is characterized in that it comprises a stage.

  In order to solve the above-described problem, the control method for the operation device according to the present invention is a control method for the operation device that remotely operates the display device, and displays image data obtained by imaging the display device on the display unit of the operation device. A display control step, a position specifying step for specifying a position of a first region representing the display unit of the display device in the display unit of the operation device from the image data displayed in the display control step, An input receiving step for receiving an operation signal input to the first area, and the operation signal on the display unit of the operating device is received based on the position of the first area specified in the position specifying step. The coordinate processing step for converting the operation position coordinates indicating the position into the operation position coordinates in the display unit of the display device, and the input signal to the display device, It is characterized in that it comprises a signal output step of outputting the converted operation position coordinates at standard processing steps.

  As a result, it is possible to reduce the communication load between the devices and realize an operation device and a remote operation system with excellent responsiveness.

It is a block diagram which shows the principal part structure of the tablet in one Embodiment of this invention. It is a figure which shows the outline | summary of the remote control system in one Embodiment of this invention. It is a figure which shows the external appearance of the tablet back surface of this invention. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the information of the coordinate system which the position specific part of a tablet hold | maintains. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the information of the coordinate system which the position specific part of a tablet hold | maintains. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a flowchart which shows the flow of a process of the tablet in one Embodiment of this invention. It is a block diagram which shows the principal part structure of the tablet in one Embodiment of this invention. It is a block diagram which shows the principal part structure of the television in one Embodiment of this invention. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the information of the coordinate system which the position specific part of a tablet hold | maintains. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the information of the coordinate system which the position specific part of a tablet hold | maintains. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the information of the coordinate system which the position specific part of a tablet hold | maintains. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a flowchart which shows the flow of a process of the tablet and television in one Embodiment of this invention. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the information of the coordinate system which the position specific part of a tablet hold | maintains. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the information of the coordinate system which the position specific part of a tablet hold | maintains. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the information of the coordinate system which the position specific part of a tablet hold | maintains. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a flowchart which shows the flow of a process of the tablet and television in one Embodiment of this invention. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the information of the coordinate system which the position specific part of a tablet hold | maintains. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the information of the coordinate system which the position specific part of a tablet hold | maintains. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a flowchart which shows the flow of a process of the tablet and television in one Embodiment of this invention. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the display screen displayed on the display part of a tablet. It is a figure which shows an example of the display screen displayed on the display part of a tablet.

Embodiment 1
An embodiment of the present invention will be described below with reference to FIGS.

  In the embodiments described below, as an example, the operation device of the present invention is a multi-function terminal device (referred to as a tablet) such as a tablet PC, and the display device remotely operated by the operation device of the present invention is a digital television. A case will be described. However, the operation device of the present invention is not limited to a tablet, and can be applied to any relatively small information processing device such as a smartphone, a mobile phone, a PDA (Personal Digital Assistant), a game machine, an electronic dictionary, and a laptop computer. . In addition, the operation device of the present invention is not limited to a digital television, and includes any display unit that displays information and a communication unit that communicates with the operation device, such as a desktop personal computer, a notebook computer, a game machine, and a digital photo frame. The display device can be remotely operated. In the operating device of the present invention, it is not an essential configuration that the input mechanism is a touch panel type.

  In the following drawings, the same or corresponding parts are denoted by the same reference numerals, and description thereof will not be repeated. In addition, dimensional relationships such as length, size, and width, and shapes in the drawings are appropriately changed for clarity and simplification of the drawings, and do not represent actual dimensions and shapes.

[Outline of remote control system]
The remote control system of the present invention is configured by a plurality of devices that can communicate with each other. One of the plurality of devices is a display device having a function of displaying an object, and the other is an operation device for remotely operating the display device. In the remote operation system, one or more display devices exist. The operation device also has a function of displaying an object. When another operation device exists, the operation device may function as a display device for the operation device.

  FIG. 2 is a diagram showing an overview of the remote control system according to the present embodiment. As shown in FIG. 2, the remote operation system of the present invention includes at least one operation device (here, tablet 100), one or more display devices (here, television 200, notebook computer 200a, and digital photo). Frame 200b). The tablet 100 and each display device communicate with each other via an appropriate wireless communication means such as infrared communication such as IrDA and IrSS, Bluetooth (registered trademark) communication, WiFi (registered trademark) communication, and a non-contact type IC card. Is possible. In the present embodiment, each device can perform wireless communication with each other via an access point (not shown) and a home LAN (Local Area Network).

  Hereinafter, the configuration and function of the display device of the present invention will be described using the television 200 as an example, but other display devices are assumed to have the same configuration and function. The television 200 includes a display unit 42 for displaying an object as a display device.

  The tablet 100 has a function of realizing so-called augmented reality (AR). Specifically, as illustrated in FIG. 3, for example, the tablet 100 displays image data (live view image) of the surrounding environment captured by the imaging unit 13 provided on the back surface on the display unit 12 in real time. The tablet 100 associates the communication partner device recognized from the image data with each device identified via the wireless communication means, and device-related information regarding the device as the communication partner in the image data ( Device name etc.) In the example illustrated in FIG. 2, the tablet 100 may display a live view image on the display unit 12 and may superimpose device-related information of each device that appears in the live view image, for example. In addition, the structure which superimposes apparatus related information is not an essential structure in the tablet 100 of this invention. In the example illustrated in FIG. 2, the devices that are captured in the live view image are a television 200, a BD (Blu-ray Disc (registered trademark)) recorder, and a notebook computer 200a. The device-related information is a device name here, but a menu list that displays a list of functions that can be executed by the device may be superimposed as the device-related information. As a technique for realizing the above AR, a known appropriate technique can be adopted.

  In addition, when there are a plurality of display devices shown in the live view image or a plurality of display devices recognized by the tablet 100 as communication partners, the tablet 100 is a display device registered in advance. With reference to the information, the display device shown in the live view image can be associated with the display device recognized by the tablet 100 as the communication partner. For example, the tablet 100 holds a table in which the model image of the display device is associated with the host name (or network address) on the network of the display device, and compares the display device shown in the live view image with the model image. Thus, the communication partner can be recognized by searching for similar model images.

  FIG. 2 shows an example in which the image data obtained by the imaging unit 13 photographing a real space extending in the back of the tablet 100 is displayed on the display unit 12. Therefore, the image data displayed on the display unit 12 and the real space extending in the back are not necessarily seamlessly connected to the user. On the other hand, it is also conceivable that the tablet 100 includes the transparent display unit 12 and sees through the real space spreading in the back of the display unit 12. In this case, the position of the user's eyes is detected by another imaging unit (front camera) provided on the front surface of the tablet 100, and an image viewed by the user through the watermark is displayed on the imaging unit 13 (main camera). It is also possible to realize the display of an object that matches the real space by estimating using the video together.

  Hereinafter, in each figure used for description, a black object represents an object displayed on the display unit 42 of the television 200. A halftone dot object represents an object displayed on the display unit 12 of the tablet 100. Among the black-painted objects, an object with a white mesh pattern is an object displayed on the display unit 42 of the television 200, which is photographed by the imaging unit 13 of the tablet 100, An object in a state displayed on the display unit 12 is represented. That is, the pattern and color applied to the object do not indicate the difference between the objects. Differences between objects are indicated by symbols.

  In the example illustrated in FIG. 2, the object is, for example, an icon image. However, the objects displayed on the tablet 100 and the television 200 are not limited to icon images. The object includes any display object that can be selected by the user using the input unit 11 such as a photograph, a moving image, a graphic, and a GUI (graphical user interface) component.

  When the user selects an object (halftone dot pattern) displayed on the tablet 100, the user only has to touch the area where the target object is displayed on the input unit 11 (such as the touch scene 3a in FIG. 2). ). Further, if the object is dragged while being touched, the object can be moved to another position on the display unit 12. In the tablet 100 of the present invention, in the input unit 11, the area where the screen of the display unit 42 of the television 200 is displayed is touched by the user (such as the touch scene 3 b in FIG. 2). It is characterized in that a predetermined remote operation is performed. The configuration of the tablet 100 for remotely operating the television 200 will be described in detail later.

[Tablet structure]
FIG. 1 is a block diagram showing a main configuration of a tablet according to an embodiment of the present invention. The tablet 100 is configured to include at least the control unit 10, the input unit 11, the display unit 12, the imaging unit 13, the communication unit 14, and the storage unit 15 in order to function as the operation device of the present invention. Although not shown, the tablet 100 has an external interface, an audio output unit, an audio input unit, a call processing unit, a broadcast image receiving unit (a tuner / demodulation unit, etc.) in order to realize functions conventionally provided as a multi-function terminal device. ), GPS, sensors (acceleration sensor, tilt sensor, etc.), and an operation unit such as a switch / key for turning on the power, and various other components that are normally provided in the tablet PC. In addition, regarding the tablet 100, the description is omitted for the general functions of the tablet PC that are not directly related to the present invention.

  The control unit 10 performs overall control of each unit included in the tablet 100. The control unit 10 includes at least a position specifying unit 20, a coordinate processing unit 21, and a signal output unit 22 as functional blocks. The control unit 10 further includes an input receiving unit 23 and a display control unit 24 as necessary.

  Each functional block of the control unit 10 described above is stored in a storage device (storage unit 15) realized by a CPU (central processing unit) or the like using a ROM (read only memory), an NVRAM (non-volatile random access memory), or the like. This can be realized by reading the program stored in a RAM (random access memory) or the like and executing it. Details of each functional block of the control unit 10 will be described later.

  The storage unit 15 is (1) a control program executed by the control unit 10 of the tablet 100, (2) an OS program executed by the control unit 10, and (3) because the control unit 10 executes various functions of the tablet 100. And (4) various data read out when the application program is executed. Alternatively, (5) the controller 10 temporarily stores data and calculation results used for calculations in the course of executing various functions. For example, the above data (1) to (4) are stored in a nonvolatile storage device such as a ROM (read only memory), a flash memory, an EPROM (Erasable Programmable ROM), an EEPROM (Electrically EPROM), an HDD (Hard Disc Drive), etc. Is remembered. For example, the data (5) is stored in a volatile storage device such as a RAM (Random Access Memory). Which data is to be stored in which storage device is appropriately determined based on the purpose of use, convenience, cost, physical restrictions, and the like of the tablet 100. For example, a live view image captured by the imaging unit 13 is temporarily stored in the storage unit 15 realized by a volatile storage device. Alternatively, the object (icon image) displayed on the display unit 12 and the entity data associated with the object are stored temporarily in the storage unit 15 realized by a nonvolatile storage device. For example, if the object is an icon of moving image content, the entity data represents the moving image data itself.

  The input unit 11 is for inputting an instruction signal for the user to operate the tablet 100 via the touch panel. The input unit 11 includes a touch surface and a touch sensor. The touch surface accepts contact of an indicator (indicating the screen position of the display unit 12, here, for example, a finger or a pen). The touch sensor is for detecting contact / non-contact (approach / non-approach) between the indicator and the touch surface and a contact (approach) position. The touch sensor may be realized by any sensor as long as it can detect contact / non-contact between the indicator and the touch surface. For example, it is realized by a pressure sensor, a capacitance sensor, an optical sensor, or the like.

  The display unit 12 displays objects processed by the tablet 100, image data (live view image) of the surrounding environment imaged by the imaging unit 13, a GUI (Graphical User Interface) screen for the user to operate the tablet 100, and the like. To display. The display unit 12 is realized by a display device such as an LCD (Liquid Crystal Display).

  In this embodiment, the input part 11 and the display part 12 are integrally molded, and these constitute a touch panel.

  For example, when the touch panel of the tablet 100 of the present invention is realized by a projected capacitive touch panel, specifically, the touch sensor uses a transparent electrode pattern in the form of a matrix made of ITO (Indium Tin Oxide) or the like. It is formed on a transparent substrate such as plastic. When an indicator (such as a user's finger or pen) touches or approaches the touch sensor, the electrostatic capacity of a plurality of transparent electrode patterns in the vicinity changes. Therefore, the control unit 10 can detect the position where the indicator is in contact or approached by detecting a change in the current or voltage of the transparent electrode pattern.

  In the following, the term “contact” or “touch” refers not only to the state in which the indicator and the touch surface are in complete contact (contact) but also to the extent that the indicator and the touch surface can be detected by the touch sensor. The state of approaching (approaching) is also included.

  The image capturing unit 13 captures a moving image or a still image, and includes various mechanisms and circuits such as a lens, a mirror mechanism, an image sensor, an A / D converter, and an image display circuit. The live view image is acquired as an image received by the image sensor through the lens. In the present embodiment, as illustrated in FIG. 3, the imaging unit 13 is provided on the back surface of the tablet 100 (the surface opposite to the side with the touch panel).

  The communication unit 14 communicates with an external device via a communication network. The communication unit 14 is connected to each device (in particular, the television 200 in this case) via a communication network to realize data transmission / reception. Note that the communication unit 14 may further include a short-range wireless communication unit. In this case, the communication unit 14 as a short-range wireless communication unit wirelessly communicates with an external device at a short distance. The communication unit 14 is not particularly limited, and realizes any one of wireless communication technologies such as infrared communication such as IrDA and IrSS, Bluetooth (registered trademark) communication, WiFi (registered trademark) communication, and a non-contact type IC card. Alternatively, a plurality of these techniques may be realized.

[Functional structure of tablet-details]
The input receiving unit 23 of the control unit 10 receives an operation signal input to the tablet 100 by a user touch operation on the touch panel. The input receiving unit 23 acquires an operation signal output from the touch sensor of the input unit 11, and what kind of touch operation (tap, flick, Recognize whether a pinch, drag, etc.) has been made. The input receiving unit 23 supplies the recognition result, that is, information on the touch position on the input unit 11 (display unit 12) to the coordinate processing unit 21.

  The display control unit 24 arranges information obtained as a result of information processing by each unit of the control unit 10 on the screen of the display unit 12, converts the information into a video signal, and supplies the video signal to the display unit 12. For example, the display control unit 24 arranges the live view image acquired by the imaging unit 13 at a predetermined position on the screen of the display unit 12 and outputs the video signal to the display unit 12. Thereby, the live view image is displayed on the display unit 12.

  FIG. 4 is a diagram illustrating an example (one frame) of a live view image displayed on the display unit 12 of the tablet 100. In the example illustrated in FIG. 4, the display unit 42 of the television 200 displays five objects 2a to 2e. The state is captured by the imaging unit 13 of the tablet 100 and is reflected in the live view image, and the display control unit 24 displays the live view image on the display unit 12 in a full screen arrangement.

  The position specifying unit 20 specifies the position of various displayed information on the display unit 12 on which various information is displayed. Specifically, in the present embodiment, when the live view image captured by the imaging unit 13 is displayed on the display unit 12, the position specifying unit 20 displays the display unit 42 of the television 200 that is reflected in the live view image. The position of the screen area is specified. Further, the position specifying unit 20 may specify the position of each object displayed on the screen area of the display unit 42 in the live view image. The position specifying unit 20 performs an image recognition process to specify a screen area or object area of the display unit 42 in the live view image. In the present embodiment, the position specifying unit 20 can further specify the position of various information displayed on the display unit 12.

  FIG. 5 is a diagram illustrating an example of coordinate system information handled by the position specifying unit 20. In FIG. 5, when the display unit 12 displays the live view image shown in FIG. 4, the position specifying unit 20 specifies the screen area of the display unit 42 and the position of the object displayed in the screen area. An example is shown. Hereinafter, in the live view image, an area representing the screen of the display unit 42 is referred to as a TV area 420.

  The position specifying unit 20 first defines a tablet region 120 that is a screen region of the display unit 12 of the tablet 100. In the present embodiment, the tablet 100 holds in advance in the storage unit 15 a tablet coordinate system in which the upper left end P of the tablet area 120 is the origin (X, Y = 0, 0). The position specifying unit 20 can define the tablet region 120 by the coordinate values in the tablet coordinate system.

  Next, the position specifying unit 20 specifies the screen area of the display unit 42 of the television 200, that is, the position of the television area 420. The position specifying unit 20 performs image recognition processing on one frame of the live view image shown in FIG. 4 and detects a region characteristic of the television screen (the television region 420 in FIG. 5). The position specifying unit 20 can detect the television area 420 by, for example, matching with a model image of the television 200 or extracting a feature amount (rectangular or trapezoidal shape by edge detection).

  The position specifying unit 20 can specify the detected position of the television area 420 by the coordinate value in the tablet coordinate system.

  As illustrated in FIG. 5, the position specifying unit 20 may further specify the positions of the objects 2 a to 2 e displayed on the display unit 42 from the live view image illustrated in FIG. 4. This configuration is not an essential configuration in the present embodiment.

  The position specifying unit 20 specifies the position of various information in the tablet coordinate system, and then supplies the position specifying result to the coordinate processing unit 21.

  The coordinate processing unit 21 displays the coordinate value of the operation position in the input unit 11 (display unit 12) of the touch operation received from the user via the input receiving unit 23 on the screen of another display device that is a remote operation target. This is converted to the coordinate value of the operation position.

  In the present embodiment, specifically, the coordinate processing unit 21 operates the operation position coordinates of the touch operation received on the input unit 11 (hereinafter, the operation position coordinates by the touch operation are particularly referred to as touch position coordinates). Is converted into the operation position coordinates on the screen of the display unit 42 of the television 200 so that the television 200 can understand.

  For example, as illustrated in FIG. 4, it is assumed that the user performs a touch operation on an arbitrary position of the input unit 11. In the example illustrated in FIG. 4, it is assumed that the center of the object 2c displayed on the television 200 in the live view image is touched.

  The input receiving unit 23 notifies the coordinate processing unit 21 of the touch position coordinates of the received touch operation. For example, as illustrated in FIG. 5, the input reception unit 23 supplies the coordinate processing unit 21 with the coordinate values (Xtab_r, Ytab_r) of the point R in the tablet coordinate system.

  The coordinate processing unit 21 determines the touch position coordinates (Xtab_r, Ytab_r) of the operation position (point R) based on the television area 420 specified by the position specifying unit 20, and the operation position coordinates (Xtv_r, Ytv_r) in the TV coordinate system. Convert to The television coordinate system is a coordinate system that can be understood by the television 200 with the upper left corner Q of the screen of the display unit 42 as the origin.

  The coordinate processing unit 21 can grasp the positions of the television area 420, the point Q, and the point R in the tablet coordinate system. Therefore, the coordinate processing unit 21 can grasp the relative positional relationship between the point R and the point Q and the positional relationship between the point R and the television area 420. As a result, the coordinate processing unit 21 can obtain the operation position coordinates (Xtv_r, Ytv_r) of the point R in the television area 420 when the point Q is the origin.

  Note that the coordinate processing unit 21 determines the operation position on the display unit 42 of the television 200 based on the operation position coordinates (Xtv_r, Ytv_r) for the operation position coordinates (Xtv_r, Ytv_r) obtained in the television coordinate system. Any coordinate value may be obtained as long as it can be specified.

  For example, the X coordinate (Xtv_r) of the point R in the television coordinate system is clarified by the ratio of the difference between the X coordinate of the point Q and the X coordinate of the point R to the length of the long side of the television region 420. Further, the Y coordinate (Ytv_r) of the point R in the television coordinate system is clarified by the ratio of the difference between the Y coordinate of the point Q and the Y coordinate of the point R to the length of the short side of the television area 420. That is, the coordinate processing unit 21 can obtain the operation position coordinates (Xtv_r, Ytv_r) based on the ratio while ignoring specific units such as the number of pixels of the television 200.

  The coordinate processing unit 21 supplies the operation position coordinates (Xtv_r, Ytv_r) in the television coordinate system of the received touch operation to the signal output unit 22.

  The signal output unit 22 generates a signal including the operation position coordinates in the television coordinate system converted by the coordinate processing unit 21 as an input signal to the remote operation target display device. The signal output unit 22 transmits the generated input signal to the television 200 via the communication unit 14.

  The television 200 that has received the input signal transmitted from the signal output unit 22 determines which position on the screen of the display unit 42 of the television 200 the position operated by the user based on the operation position coordinates specified in the input signal. Can be identified. Then, the television 200 can understand that the specified position is virtually operated by the user, and can perform the same operation as when the position is actually operated.

  For example, in the example illustrated in FIGS. 4 and 5, the television 200 identifies the operation position point R, regards the object 2c overlapping the point R as having been operated by the user, and has received it from the tablet 100. Processing for the object 2c is executed in accordance with the input signal.

  As described above, the tablet 100 can be remotely operated so as to process a predetermined object with respect to the television 200 only by transmitting the operation position coordinates to the television 200. As a result, since only the operation position coordinates are transmitted, it is possible to reduce the communication load between the devices and realize the tablet 100 with excellent responsiveness.

  Further, when the input and display mechanism of the operation device is realized by a touch panel like the tablet 100, there are the following merits.

  The tablet 100 may reflect the processing result corresponding to the touch operation on the actual screen position of the television 200 corresponding to the position of the screen of the television 200 on the live view image touch-operated on the tablet 100. it can. Therefore, even if the television 200 is a device that does not support touch operation, an operation device that can perform a pseudo touch operation on such a device can be realized. The television 200 and the like are often operated using a remote control device or the like that is usually structurally limited. Therefore, the fact that the touch operation can be realized on the television 200 in a pseudo manner using the tablet 100 can greatly improve the operability and the degree of freedom of the operation on the television 200, which is particularly effective.

[Explanation of operation for realizing pseudo touch operation-specific example]
A method for selecting and moving an object displayed on the television 200 using the tablet 100 will be described with reference to FIGS.

  While viewing the live view image shown in FIG. 4, the user touches an object to be moved among the objects displayed on the television 200 (touch scene 3c). The input receiving unit 23 of the tablet 100 receives this touch operation. As described above, the input receiving unit 23 notifies the coordinate processing unit 21 of the touch position coordinates of the touch position point R (FIG. 5) in the tablet area 120.

  On the other hand, the position specifying unit 20 specifies the position of the television area 420 (coordinate values in the tablet coordinate system) based on one frame of the live view image when the input unit 11 is touched, as shown in FIG. The coordinate processing unit 21 is notified.

  The coordinate processing unit 21 converts the touch position coordinates (Xtab_r, Ytab_r) of the point R in the tablet coordinate system to the operation position coordinates (Xtv_r, Ytv_r) in the television coordinate system based on the specified position of the television area 420. .

  The signal output unit 22 generates an input signal including the operation position coordinates (Xtv_r, Ytv_r) and transmits it to the television 200. Thereby, the television 200 can understand which position on the screen of the display unit 42 is operated. The television 200 selects the object 2c according to this input signal.

  Here, while the user's touch operation (drag) continues as in the touch scene 3d in FIG. 6, the coordinate processing unit 21 receives the drag start point R from the drag start point R as shown in FIG. 7. The relative coordinates of the point R ′, which is the current touch position, are converted into coordinate values in the television coordinate system, which are sequentially transmitted by the signal output unit 22.

  FIG. 7 shows the locus of the touch position that moves from the point R to the point R ′. According to the above configuration, the television 200 can understand that the operation position with respect to the television 200 is moving according to the trajectory based on the sequentially received input signals, and the object 2c is moved according to the trajectory. Let

  Alternatively, the coordinate processing unit 21 obtains a moving direction and a moving distance in the television area 420 for the touch position in order to avoid sequentially converting the touch position constituting the drag locus from the tablet coordinate system to the television coordinate system. It may be a configuration. The television 200 can move the object 2c selected at the starting point according to the designated moving direction and moving distance, starting from the point R specified first.

  When the user can move the object 2c to a desired position, the user ends the touch operation. Accordingly, the coordinate processing unit 21 stops converting the touch position coordinates on the tablet 100 into the operation position coordinates in the television coordinate system. Thereby, the signal output unit 22 stops transmission of the input signal. The television 200 stops the movement of the object 2c according to the stop of the input signal. As shown in FIG. 8, the television 200 can display the object 2c selected by the user at a destination desired by the user.

[Flow of processing to realize pseudo touch operation]
FIG. 9 is a flowchart showing a process flow of the tablet according to the embodiment of the present invention.

  The position specifying unit 20 specifies the television area 420 from the live view image displayed on the display unit 12, and tracks this (S101). That is, the position coordinates of the television area 420 in the tablet coordinate system are temporarily stored in the storage unit 15 sequentially (or every predetermined frame, every predetermined millisecond). Here, the position coordinates of the television area 420 are represented by coordinate values in the tablet coordinate system.

  When the input receiving unit 23 receives a touch operation on the input unit 11 (YES in S102), the position specifying unit 20 is a television area specified at the time when the touch operation is received in S102 (or the time closest to that time). The position coordinates 420 are acquired from the storage unit 15 (S103).

  The coordinate processing unit 21 acquires touch position coordinates indicating the touch position of the touch operation received in S102 from the input receiving unit 23 (S104). The touch position coordinates are represented by coordinate values in the tablet coordinate system.

  Next, the coordinate processing unit 21 converts the touch position coordinates (Xtab_r, Ytab_r) in the tablet coordinate system into the operation position coordinates (Xtv_r) in the television coordinate system based on the position coordinates of the television area 420 specified in S101 and S103. , Ytv_r) (S105). The operation position coordinates in the television coordinate system is position information for understanding which position on the screen of the display unit 42 is virtually operated by the television 200 when the tablet 100 is touch-operated.

  The position indicated by the operation position coordinate corresponds to a position where the user touches the input unit 11 of the tablet 100 and a position where the TV area 420 of the TV 200 displayed on the display unit 12 is touched.

  The signal output unit 22 generates the operation position coordinates (Xtv_r, Ytv_r) in the television coordinate system converted by the coordinate processing unit 21 as an input signal for the television 200, and transmits this to the television 200 (S106). Accordingly, for example, as illustrated in FIG. 4, when the touch operation of the touch scene 3 c occurs in the tablet 100, the operation of selecting the object 2 c displayed on the display unit 42 of the television 200 is performed on the television 200. Done.

  When the input receiving unit 23 continues to receive touch operations (YES in S107), the coordinate processing unit 21 and the signal output unit 22 continue the operation of outputting an input signal to the television 200 (S108). Specifically, the coordinate processing unit 21 starts from the touch position when the touch operation is started (for example, the point R in FIG. 7), and the touch position (for example, the current drag operation is performed at S107). The point R ′) in FIG. 7 is expressed by relative coordinates from the start point. The coordinate processing unit 21 converts the relative coordinates into relative coordinates in the television coordinate system based on the position and size ratio of the television area 420. Then, the signal output unit 22 outputs the relative coordinates in the television coordinate system to the television 200 as an input signal of the television 200. While the drag operation is continued (YES in S107), the coordinate processing unit 21 and the signal output unit 22 sequentially repeat S108. Note that the coordinate processing unit 21 performs coordinate conversion associated with the initial touch operation, performs coordinate conversion at the next drag position, and then performs coordinate conversion of the drag position that continues thereafter, relative to the start point. The current drag position (for example, point R ′ in FIG. 7) may be expressed by the relative coordinates from the previous drag position (for example, point R ″ in FIG. 7) instead of the coordinates.

  When the user ends the drag operation continued in S107 and releases the finger from the input unit 11, the input receiving unit 23 detects the end of the touch operation (NO in S107). Thereby, the coordinate processing unit 21 stops the coordinate conversion in S108, and the signal output unit 22 stops the signal output in S108 (S109).

  As a result, for example, as illustrated in FIG. 8, the television 200 can display the selected object 2c by moving the selected object 2c to a user's desired position in accordance with the user's drag operation.

<< Embodiment 2 >>
Another embodiment relating to the operating device of the present invention will be described below with reference to FIGS.

  In the first embodiment, the operations of the tablet 100 and the television 200 in the case where a series of touch operations on the object are completed within the television area 420 have been described. In the second embodiment, operations of the tablet 100 and the television 200 in a case where the touch operation extends from the inside of the television area 420 to the outside of the television area 420 will be described.

[Tablet structure]
FIG. 10 is a block diagram showing a main configuration of the tablet according to the embodiment of the present invention. The tablet 100 in the present embodiment shown in FIG. 10 is different from the tablet 100 shown in FIG. 1 in that the control unit 10 further includes an object transmission / reception unit 25 and an object processing unit 26 as functional blocks. .

  The object transmission / reception unit 25 transmits / receives an object or entity data corresponding to the object to / from a remote operation target display device. For example, in this embodiment, icon images or entity data (moving images, still images, etc.) associated with the icon images are transmitted to and received from the television 200.

  More specifically, when the input reception unit 23 detects a user's touch operation, the object transmission / reception unit 25 receives an object operated by the touch operation from the television 200 at a predetermined timing. In the present embodiment, the touch operation includes a drag operation extending from the television area 420 to the outside of the television area 420.

  The object transmitting / receiving unit 25 receives the entity data of the selected object from the television 200 at a predetermined timing when the input receiving unit 23 detects the completion of the user's touch operation outside the television area 420. To do. The completion of the touch operation is, for example, a drop operation after the drag operation.

  The object processing unit 26 executes processing for an object. Some specific examples of processing performed by the object processing unit 26 are listed, but the function of the object processing unit 26 is not limited to the following.

  The object processing unit 26 controls display / non-display of objects on the display unit 12. The object processing unit 26 controls selection / deselection of the object displayed on the display unit 12. The object processing unit 26 controls the movement of the displayed object in response to an operation on the object input via the input unit 11. The object processing unit 26 gives a visual effect to the object to be displayed on the display unit 12 in accordance with display / non-display, movement, selection / non-selection, and the like of the object.

  More specifically, the object processing unit 26 visually produces on the display unit 12 that the object displayed on the display unit 42 of the television 200 has been transmitted and received from the television 200 to the tablet 100. For this purpose, the object processing unit 26 displays the object acquired from the television 200 by the object transmitting / receiving unit 25 in accordance with the touch operation of the user on the display unit 12. Specifically, the object processing unit 26 arranges the object on the screen of the display unit 12 with a visual effect as necessary, and supplies the processing result to the display control unit 24. The display control unit 24 generates a video signal according to the processing result of the object processing unit 26 and displays it on the display unit 12.

  In the present embodiment, an object or entity data must be requested from the television 200 so that the object or entity data of the object is supplied from the television 200 at a predetermined timing in accordance with the user's touch operation on the tablet 100. I must. In the present embodiment, the signal output unit 22 generates an input signal for requesting the television 200 for the object or the entity data of the object, and transmits the input signal to the television 200 via the communication unit 14.

[Configuration of TV]
FIG. 11 is a block diagram showing a main configuration of a television according to one embodiment of the present invention. Note that the description of the TV 200 shown in FIG. 11 is omitted for the general functions of the digital TV that are not directly related to the present invention.

  The television 200 is configured to include at least a control unit 40, a storage unit 41, a display unit 42, and a communication unit 43 in order to function as the display device of the present invention.

  The control unit 40 performs overall control of each unit included in the television 200. The control unit 40 includes an object transmission / reception unit 50, a signal reception unit 51, an object processing unit 52, an object position determination unit 53, and a display control unit 54 as functional blocks for functioning as the display device of the present invention.

  Each functional block of the control unit 40 described above is stored in a storage device (storage unit 41) realized by a CPU (central processing unit) or the like using a ROM (read only memory), an NVRAM (non-volatile random access memory), or the like. This can be realized by reading the program stored in a RAM (random access memory) or the like and executing it. Details of each functional block of the control unit 40 will be described later.

  The storage unit 41 stores data that is read or written by the control unit 40 of the television 200 and has substantially the same configuration as the storage unit 15 in the tablet 100 described above.

  The display unit 42 displays various types of information processed by the television 200, and displays an operation screen for a user to operate the television 200 as a GUI (Graphical User Interface) screen. The display unit 42 is realized by a display device such as an LCD (Liquid Crystal Display). In the present embodiment, as shown in FIG. 2, an object (icon image) or the like is displayed on the display unit 42.

  The communication unit 43 communicates with an external device via a communication network. Similar to the communication unit 14 of the tablet 100, the communication unit 43 may further include a short-range wireless communication unit. In the present embodiment, the communication unit 43 is connected to the tablet 100 via a communication network to realize data transmission / reception. For example, the communication unit 43 receives an input signal output from the tablet 100. Further, the communication unit 43 transmits / receives an object or entity data associated with the object to / from the tablet 100.

[Functional configuration of TV-details]
The object transmission / reception unit 50 of the control unit 40 transmits / receives an object or entity data corresponding to the object to / from a remote operation device. For example, in this embodiment, icon images or entity data (moving images, still images, etc.) associated with the icon images are transmitted to and received from the tablet 100. The object transmitting / receiving unit 50 may transmit object or entity data in response to a request from the tablet 100, or transmits the object or entity data at a predetermined timing while the television 200 performs predetermined information processing. May be.

  The signal reception unit 51 receives an input signal transmitted from the tablet 100 and received by the communication unit 43. The input signal includes at least operation position coordinates in the television coordinate system. The signal receiving unit 51 acquires the input signal from the communication unit 43 and recognizes what user operation is virtually performed on which position on the display unit 42 of the television 200. Actual user operations include taps, flicks, pinches, and drags, but these are not operations directly performed on the display unit 42 of the television 200. The signal receiving unit 51 performs the above-described various touch operations on the television. 200 is recognized as being virtually made. The signal receiving unit 51 supplies the recognition result, that is, information on the virtual operation position of the user to each unit of the control unit 40.

  The object processing unit 52 executes processing for an object. The object processing unit 52 can process the object in accordance with the instruction of the input signal received by the signal receiving unit 51. The processing performed by the object processing unit 52 is, for example, display / non-display of an object, selection / deselection of an object, movement of an object, and application of a visual effect to the object, similar to the object processing unit 26 of the tablet 100. .

  The object position determination unit 53 specifies the position in the television coordinate system for the object being processed by the object processing unit 52. Further, the object position determination unit 53 can define the screen of the display unit 42, that is, the television area 420 in the television coordinate system, and specify the positions of the objects arranged inside and outside the television area 420. That is, the object position determination unit 53 can determine whether the object being processed is displayed on the screen of the display unit 42 or protrudes from the screen.

  The display control unit 54 arranges information obtained as a result of information processing by each unit (for example, the object processing unit 52) of the control unit 40 on the screen of the display unit 42 and converts the information into a video signal. To supply.

  In addition, the control part 40 of the television 200 in Embodiment 1 should just be provided with the signal reception part 51, the object process part 52, and the display control part 54 as a functional block at least.

[Explanation of operation for realizing pseudo touch operation-specific example]
A method for moving and displaying an object displayed on the television 200 using the tablet 100 will be described with reference to FIGS. 4 to 6 and FIGS.

  When the tablet 100 receives a drag operation on the television area 420, the tablet 100 outputs an input signal to the television 200 in accordance with the drag operation position, as in the first embodiment. In the example illustrated in FIGS. 4 and 6, the tablet 100 outputs an input signal to the television 200 to cause the television 200 to select the object 2 c and move it in the drag direction.

  Here, the points different from the first embodiment are as follows. The coordinate processing unit 21 of the tablet 100 determines that the touch position coordinates (for example, the point R in FIG. 5) are the objects in the television area 420 based on the positions of the objects in the television area 420 identified by the position identifying unit 20. It is detected that it overlaps with. Then, the coordinate processing unit 21 converts the touch position coordinates of the touch operation performed on any object into operation position coordinates in the television coordinate system.

  In response to this, the signal output unit 22 generates the operation position coordinates converted by the coordinate processing unit 21 as an input signal and outputs the input signal to the television 200. In the present embodiment, the signal output unit 22 uses the input signal as a signal for requesting the tablet 100 to transmit the object placed at the position of the display unit 42 specified by the operation position coordinates. Output.

  When the signal receiving unit 51 of the television 200 receives the input signal, first, the object processing unit 52 is placed at a position specified by the operation position coordinates (Xtv_r, Ytv_r) as shown in FIG. Identify 2c and select it. Further, the object transmitting / receiving unit 50 transmits an object (icon image) to the tablet 100 in accordance with the request for the input signal. In the example illustrated in FIG. 5, the object 2 c selected by the object processing unit 52 is read from the storage unit 41 and transmitted to the tablet 100 via the communication unit 43.

  As described above, in this embodiment, the signal output unit 22 of the tablet 100 outputs an input signal that requests an object on the television 200 side selected by the user. In the above description, the requested input signal is output when the drag operation is started.

  However, the timing at which the signal output unit 22 outputs the requested input signal is not limited to the above. For example, the signal output unit 22 continues to move while the object 2c is in the television area 420, and outputs the requested input signal when the end of the object 2c contacts the end of the television area 420 on the inside. May be. Alternatively, the signal output unit 22 may output the requested input signal when about half the area of the object 2c protrudes from the television area 420 (see FIG. 12). Alternatively, the signal output unit 22 may output the requested input signal when the touch position (point R in FIG. 13) in the user's drag operation approaches the boundary of the television area 420 (see FIG. 13). . Alternatively, the signal output unit 22 may output the requested input signal when the object 2c is completely out of the television area 420 (see FIGS. 16 and 17). It should be noted that the time when the half area of the object 2c protrudes outside the TV area 420 (FIG. 12) and the time when the touch position R of the drag operation approaches the boundary of the TV area 420 (FIG. 13) do not necessarily coincide. .

  Here, it is assumed that the drag operation is continued and the object 2c continues to move out of the television area 420.

  In the present embodiment, while the object processing unit 52 of the television 200 moves the object 2c according to the input signal from the tablet 100, the object position determination unit 53 determines the position of the object 2c with respect to the television area 420 of the television coordinate system. Is monitoring. Then, the object position determination unit 53 determines that the object 2c has arrived at a position where the object 2c cannot be appropriately displayed by protruding from the frame of the television area 420. Then, the object position determination unit 53 notifies the tablet 100 that the object 2c cannot be appropriately displayed via the communication unit 43. For example, the object position determination unit 53 determines that the object cannot be properly displayed and notifies the timing when about half the area of the object 2c protrudes outside the television area 420 (see FIGS. 12 and 13). And

  However, the timing at which the object position determination unit 53 performs the notification is not limited to the above. For example, the object position determination unit 53 may perform the determination and the notification when the end of the object 2c comes into contact with the end of the television area 420 inside. Alternatively, the object position determination unit 53 may perform the determination and notification when the touch position (point R in FIG. 13) in the user's drag operation approaches the boundary of the television area 420 (see FIG. 13). . Alternatively, the object position determination unit 53 may perform the determination and notification when the object 2c is completely out of the television area 420 (see FIGS. 16 and 17). However, it is desirable that the requested input signal is output by the signal output unit 22 prior to the determination and notification.

  When the communication unit 14 receives the notification from the television 200, the signal output unit 22 of the tablet 100 stops outputting the operation position coordinates of the touch operation received by the input receiving unit 23, that is, the input signal. This is because it is considered that there is no meaning in transmitting an input signal of a drag operation on the object even though the television 200 has notified that the object cannot be displayed properly. However, even when it is determined that the object cannot be displayed properly, a part of the object may still be displayed on the display unit 42 of the television 200.

  Therefore, even if the signal output unit 22 receives the notification from the television 200 at the time of FIG. 12 and FIG. 13, even if a part of the object is displayed in the television area 420 (see, for example, FIG. 15). May continue to sequentially transmit the relative coordinates of the touch position (coordinate values of the television coordinate system). And the signal output part 22 should just continue the said transmission until just before the time of the object 2c going out of the television area | region 420 completely (refer FIG. 16 and FIG. 17).

  Similar to the object position determination unit 53, the object processing unit 52 can grasp the object position and the operation position outside the television area 420 in the television coordinate system with the upper left corner of the television area 420 as the origin. That is, it is possible to understand the position of the device that protrudes from the television area 420 and the arrangement of objects associated therewith.

  Therefore, the object processing unit 52 can continue the object movement and display processing until the object completely protrudes from the screen of the display unit 42 in accordance with the input signal (relative coordinates) continuously output.

  When the communication unit 14 receives the notification from the television 200, the object processing unit 26 of the tablet 100 executes a process of arranging the object 2c (icon image) acquired from the television 200 in advance on the screen of the display unit 12. To do. Then, as shown in FIG. 14, the display control unit 24 displays the object 2c processed by the object processing unit 26 on the display unit 12 of the tablet 100 (the dot pattern object 2c in FIG. 14). Thus, in a case where the drag operation extends from the inside of the television area 420 to the outside of the television area 420, the tablet 100 can take over the display of the object 2c that can no longer be properly displayed on the television 200.

  Here, the timing at which the object processing unit 26 starts displaying the object 2c is not limited to the above. For example, without receiving a notification from the television 200, the object processing unit 26 starts displaying on the display unit 12 when the object 2c is completely out of the television area 420 (see FIGS. 16 and 17). May be. Alternatively, the object processing unit 26 may start display on the display unit 12 when the end of the object 2c comes into contact with the end of the television area 420 on the inside. Alternatively, the object processing unit 26 may start display on the display unit 12 when about half the area of the object 2c protrudes from the television area 420 (see FIGS. 12 and 13). As a matter of course, prior to the display of the object 2c by the object processing unit 26, the signal output unit 22 requests the object 2c from the television 200, and the object transmitting / receiving unit 25 receives the object 2c from the television 200.

  After the object 2c is displayed on the display unit 12, the object processing unit 26 continues to move the object 2c according to the user's drag operation (for example, FIG. 14 (FIG. 15), FIG. 16 (FIG. 17), (See FIG. 18).

  When the input receiving unit 23 no longer detects a touch operation (drag operation), the input receiving unit 23 receives this as the end of the drag operation, that is, as a drop operation.

  Here, as shown in FIG. 18, it is assumed that the user's drag operation on the input unit 11 is finished outside the television area 420 (display unit 42). In response to the drop operation received by the input receiving unit 23, the object processing unit 26 stops the movement of the object 2c and displays the object 2c at the position where the drag operation has ended (see FIG. 18). On the other hand, the signal output unit 22 generates and outputs an input signal that requests the television 200 to transmit entity data (for example, a moving image corresponding to an icon image) associated with the object 2c.

  When the signal reception unit 51 of the television 200 receives the input signal, the object transmission / reception unit 50 reads out the actual data corresponding to the object 2c from the storage unit 41 and transmits it to the tablet 100.

  If the object transmission / reception unit 25 of the tablet 100 receives the entity data and stores it in the storage unit 15, transmission / reception of the entity data of the object 2c accompanying the user's drop operation is completed.

[Flow of processing to realize pseudo touch operation]
FIG. 19 is a flowchart showing a process flow of the tablet and the television according to the embodiment of the present invention.

  When the touch operation (drag operation) by the user is started in the television area 420, the tablet 100 executes a coordinate conversion process similar to that in the first embodiment (S201). That is, S101 to S105 are executed. Subsequently, the signal output unit 22 transmits the operation position coordinates in the television coordinate system indicating the drag start position coordinates as an input signal (first input signal) for requesting an object to the television 200 (S202).

  When the signal receiving unit 51 of the television 200 receives the input signal, the object processing unit 52 selects an object (for example, the object 2c) designated by the operation position coordinates according to the input signal. On the other hand, the object transmitting / receiving unit 50 transmits the selected object 2c to the tablet 100 in accordance with the input signal (S203).

  The object transmitting / receiving unit 25 of the tablet 100 receives the transmitted object 2c and stores it in the storage unit 15 (S204).

  On the other hand, the coordinate processing unit 21 and the signal output unit 22 continue to output the input signal (second input signal) as in the first embodiment while the drag operation is performed in the television area 420 (S205). . That is, S107 to S109 are executed.

  The object processing unit 52 of the television 200 moves the object 2c according to the input signal continuously output from the tablet 100 (S206). The object position determination unit 53 monitors the arrangement of the object 2c.

  The object continues to move while the object 2c is displayed in the television area 420 (NO in S207) and while the output of the input signal from the tablet 100 is continued (NO in S208). (S206). On the other hand, when the output of the input signal from the tablet 100 is stopped (YES in S208) while the object 2c is displayed in the TV area 420 (NO in S207), the object processing unit 52 performs the first embodiment. Similarly, the movement of the object 2c is completed at the position where the drag operation is completed (S209).

  On the other hand, in S207, it is assumed that the object position determination unit 53 determines that the object 2c cannot be displayed on the display unit 42 due to the object 2c protruding outside the television area 420 as the object 2c moves. In this case (YES in S207), the object position determination unit 53 notifies the tablet 100 that the object 2c cannot be displayed properly (S210). The object processing unit 52 hides the object 2c at this time (S211), or moves the object 2c until it completely goes out of the television area 420.

  The signal output unit 22 continues the signal output process of S205 until a notification that display is impossible is received from the television 200 (NO in S212). On the other hand, when a notification indicating that display is impossible is received from television 200 (YES in S212), signal output unit 22 may stop signal output. Alternatively, the signal output may be continued until the object 2c is completely out of the television area 420.

  When the notification that display is impossible is received from the television 200, the object processing unit 26 displays the object 2c received in S204 on the display unit 12 (S213). The object processing unit 26 arranges the object 2c based on the current touch position (for example, the point R in FIG. 15) (see FIG. 14).

  In this way, the display of the object 2 c is handed over from the television 200 to the tablet 100 in accordance with the drag operation straddling the boundary of the television area 420.

  When the drag operation is continuously performed on the object 2c that is displayed on the display unit 12 side (YES in S214), the object processing unit 26 displays the object 2c according to the drag position. Move (S215).

  On the other hand, when the input receiving unit 23 detects that the drag operation has been completed outside the TV area 420 (NO in S214), the signal output unit 22 requests the entity data. An input signal (third input signal) to be generated is generated (S216). Specifically, the signal output unit 22 transmits an input signal requesting entity data (such as a moving image) associated with the selected object 2c to the television 200.

  When the signal receiving unit 51 of the television 200 receives the input signal, the object transmitting / receiving unit 50 transmits the entity data of the object 2c to the tablet 100 via the communication unit 43 (S217).

  The object transmitting / receiving unit 25 of the tablet 100 receives the entity data of the object 2c via the communication unit 14, and stores it in the storage unit 15 (S218).

  According to the above configuration and method, transmission / reception of an object can be realized between the television 200 and the tablet 100 in accordance with a drag operation straddling the boundary of the television area 420. Furthermore, the display of the object that visually represents the transmission / reception of the object can be inherited from the television 200 by the tablet 100.

  The tablet 100 can be remotely operated so as to process a predetermined object with respect to the television 200 only by transmitting the operation position coordinates to the television 200. As a result, it is possible to reduce the communication load between devices and realize the tablet 100 with excellent responsiveness.

  Further, when the input and display mechanism of the operation device is realized by a touch panel like the tablet 100, there are the following merits.

  When the user wants to capture an object displayed on the screen of the television 200 into the tablet 100 at hand, the user does not need to operate a dedicated remote controller or the like of the television 200. According to the tablet 100 at hand in the present invention, it is possible to cause the TV 200 to transmit the object selected by the touch operation to the tablet 100 only by performing a touch operation (drag operation) on the input unit 11. is there. Specifically, the user may select an object displayed in the television area 420 and drag and drop it outside the television area 420.

  Thus, even if the television 200 is a device that does not support touch operation, an operation device that can perform a pseudo touch operation on such a device can be realized. Moreover, since the display of the object accompanying the movement of the object is taken over from the television 200 to the tablet 100, the user can intuitively remotely operate the television 200 through a visually and natural user interface.

<< Embodiment 3 >>
Other embodiments relating to the operating device of the present invention will be described below with reference to FIGS. 10, 11, and 20 to 27.

  In the second embodiment, the operations of the tablet 100 and the television 200 in the case where the touch operation extends from the television area 420 to the outside of the television area 420 have been described. In the third embodiment, operations of the tablet 100 and the television 200 in a case where the touch operation extends from the outside of the television area 420 to the inside of the television area 420 will be described.

[Tablet structure]
The principal part structure of the tablet 100 in this embodiment is as showing in FIG. The difference between the tablet 100 in the present embodiment and the tablet 100 in the second embodiment is as follows.

  In the present embodiment, when the input reception unit 23 detects a user's touch operation, the object transmission / reception unit 25 transmits an object operated by the touch operation to the television 200 at a predetermined timing. In the present embodiment, the touch operation includes a drag operation that extends from outside the television area 420 into the television area 420. Hereinafter, all areas other than the TV area 420 in the tablet area 120 are referred to as non-TV areas.

  Further, in the present embodiment, the object transmission / reception unit 25, when the input reception unit 23 detects the completion of the user's touch operation in the television area 420, the entity data of the selected object at a predetermined timing. Is transmitted to the television 200. The completion of the touch operation is, for example, a drop operation after the drag operation.

  In the present embodiment, the object processing unit 26 visually produces on the display unit 12 that the object displayed on the display unit 12 of the tablet 100 has been transmitted and received from the tablet 100 to the television 200. For this purpose, the object processing unit 26 supplies the object that has been displayed on the display unit 12 so far to the television 200 by the object transmission / reception unit 25 in accordance with the user's touch operation, and then hides the object at a predetermined timing. To.

  In the present embodiment, the object displayed on the tablet 100 must be displayed on the display unit 42 of the television 200 at a predetermined timing in accordance with the user's touch operation on the tablet 100. In the present embodiment, the signal output unit 22 generates an input signal for displaying an object on the display unit 42 of the television 200 and transmits the input signal to the television 200 via the communication unit 14.

[Configuration of TV]
The main configuration of the television 200 in the present embodiment is as shown in FIG. The difference between the television 200 in the present embodiment and the television 200 in the second embodiment is as follows.

  In the present embodiment, the object transmitting / receiving unit 50 receives an object or entity data transmitted from the tablet 100 at a predetermined timing.

  In the present embodiment, when the signal receiving unit 51 receives an operation position coordinate in the television coordinate system from the tablet 100 as an input signal, the object processing unit 52 displays the screen of the display unit 42 designated by the operation position coordinate. The object received by the object transmitting / receiving unit 50 is displayed at the upper position. Thereafter, when the input signal is continuously received, the object processing unit 52 continues the display while moving the object according to the relative coordinates.

[Explanation of operation for realizing pseudo touch operation-specific example]
A method for moving and displaying an object displayed on the tablet 100 on the television 200 using the tablet 100 will be described with reference to FIGS.

  As illustrated in FIG. 20, the object processing unit 26 of the tablet 100 displays the objects (for example, the objects 2e to 2j) stored in the storage unit 15 by superimposing them on the live view image in which the television 200 is captured. Can do.

  The object processing unit 26 may display an object on the object placement area 4 by providing an object placement area 4 for placing an object as shown in FIGS. 20 and 21 (for example, the objects 2e to 2g). ) An object may be displayed at an arbitrary position on the non-TV area in the live view image (for example, objects 2h to 2j).

  As shown in FIG. 21, in the present embodiment, the position specifying unit 20 recognizes an area other than the TV area 420 in the object arrangement area 4 or the tablet area 120 (live view image area) as a non-TV area. To do. Specifically, in FIG. 21, the position specifying unit 20 recognizes an area indicated by a solid line as a TV area 420 and recognizes the other areas as non-TV areas. That is, in the live view image of FIG. 20, even if the screen area of the display unit 42 is an area where the object placement area 4 overlaps, the position specifying unit 20 is not the TV area 420 for that area. It is the structure recognized as a non-television area | region.

  Note that the object processing unit 26 has the following advantages in providing the object placement area 4. For example, when the user's hand supporting the tablet 100 moves steadily, the live view image displayed on the display unit 12 also moves in small increments according to the movement. Accordingly, naturally, the position of the television area 420 also changes in small increments, and the non-television area is not stable, and thus there is a problem that an area for displaying an object cannot be secured stably on the tablet 100 side. Further, when the television 200 is greatly reflected in the imaging unit 13 and the television area 420 occupies most of the tablet area 120, it is difficult to secure an area for displaying an object on the tablet 100 side. Even in these cases, the object processing unit 26 can provide the object placement area 4 to stably secure a non-television area, that is, an area where the object can be displayed on the tablet 100 side.

  When the input receiving unit 23 of the tablet 100 receives a drag operation on an object in the non-TV area (for example, on the object 2e in FIG. 20), first, the object processing unit 26 first selects the object 2e at the drag operation start position. Next, the object 2e is moved in accordance with the drag operation position.

  The position specifying unit 20 monitors the position of the selected object 2e and the position of the non-television area. In this embodiment, while the object 2e is partially in the non-television area (see FIG. 22), the object processing unit 26 continues to move and display the object 2e on the display unit 12 side.

  Here, the points different from the first and second embodiments are as follows. In the present embodiment, when an object displayed on the display unit 12 is selected by a user's touch operation, the object transmitting / receiving unit 25 transmits the selected object 2e (icon image) to the television 200. In the present embodiment, the object transmitting / receiving unit 25 transmits the selected object to the television 200 when the input receiving unit 23 detects the start of a drag operation on the object.

  However, the timing at which the object transmitting / receiving unit 25 transmits the selected object is not limited to the above. For example, the object transmitting / receiving unit 25 continues to move while the object 2e is in the non-television area, and when the end of the object 2e comes into contact with the end of the television area 420 inside the non-television area, the object 2e is moved to the television 200. May be sent to. Alternatively, the object transmission / reception unit 25 may transmit the object 2e to the television 200 when about half the area of the object 2e protrudes into the television area 420 (see FIG. 22). Alternatively, the object transmitting / receiving unit 25 moves the object 2e to the television 200 when the touch position (point R in FIG. 23) in the user's drag operation reaches the boundary between the television area 420 and the non-television area (see FIG. 23). May be sent to. Alternatively, the object transmission / reception unit 25 may transmit the object 2e to the television 200 when the object 2e completely enters the television area 420 (see FIGS. 24 and 25). However, the object transmitting / receiving unit 25 completes the transmission of the object 2e by the time when the television 200 displays the object 2e.

  It should be noted that the time when the half area of the object 2e protrudes outside the TV area 420 (FIG. 22) and the time when the touch position R of the drag operation approaches the boundary of the TV area 420 (FIG. 23) do not necessarily coincide. .

  Here, it is assumed that the drag operation continues and the object 2e continues to move from the non-television area (object arrangement area 4) toward the television area 420.

  In the present embodiment, as described above, while the object processing unit 26 moves the object 2e according to the drag operation from the user, the position specifying unit 20 monitors the positional relationship between the object 2e and the television area 420. ing. Then, the position specifying unit 20 displays the object 2e when the object 2e completely leaves the non-television area (see FIGS. 24 and 25, that is, when the object 2e completely enters the television area 420). The part 42 determines that the object 2e has reached the position to be displayed.

  Here, as illustrated in FIG. 25, the coordinate processing unit 21 of the tablet 100 acquires the touch position coordinates (Xtab_r, Ytab_r) of the touch position (point R) when the position specifying unit 20 determines that display is not possible. Then, the coordinate processing unit 21 converts the touch position coordinates (Xtab_r, Ytab_r) into operation position coordinates (Xtv_r, Ytv_r) in the television coordinate system. Then, the signal output unit 22 outputs the converted operation position coordinates (Xtv_r, Ytv_r) to the television 200 as an input signal. Here, the object processing unit 26 may hide the object 2e displayed on the display unit 12 of the tablet 100 (see FIG. 24).

  The timing at which the signal output unit 22 outputs the operation position coordinate input signal is not limited to the above. For example, as illustrated in FIG. 23, the signal output unit 22 may start outputting an input signal when the touch position R reaches the boundary between the non-TV area and the TV area 420.

  On the other hand, the signal receiving unit 51 of the television 200 receives the operation position coordinates (Xtv_r, Ytv_r) of the touch position R shown in FIG. 25 as an input signal. In response to this input signal, the object processing unit 52 arranges the object 2e at the position specified by the operation position coordinates on the screen of the display unit 42 of the television 200 (see FIG. 24). The object 2e is transmitted in advance by the object transmitting / receiving unit 25 of the tablet 100, and is received by the object transmitting / receiving unit 50 and stored in the storage unit 41.

  Thereafter, the movement of the object 2e based on the drag operation is realized on the television area 420 by the cooperation of the tablet 100 and the television 200 as in the first embodiment.

  When the input receiving unit 23 no longer detects the drag operation, the input receiving unit 23 receives this as the end of the drag operation, that is, as a drop operation.

  Here, as shown in FIG. 26, it is assumed that the user's drag operation on the input unit 11 is completed on the television area 420 (display unit 42). In response to the drop operation detected by the input receiving unit 23, the signal output unit 22 of the tablet 100 stops the output of the relative coordinate input signal for moving the object 2e. Thereby, the signal reception part 51 of the television 200 detects the stop of an input signal. In response to this, the object processing unit 52 stops the movement of the object 2e and displays the object 2e at the position where the input signal has stopped (see FIG. 26).

  On the other hand, when the input receiving unit 23 detects the drop operation, the object transmitting / receiving unit 25 of the tablet 100 transmits the entity data associated with the object 2e selected so far to the television 200. If the object transmission / reception unit 50 of the television 200 receives the entity data and stores it in the storage unit 41, transmission / reception of the entity data of the object 2e accompanying the user's drop operation is completed.

[Flow of processing to realize pseudo touch operation]
FIG. 27 is a flowchart showing a process flow of the tablet and the television according to the embodiment of the present invention.

  Assume that a drag operation is started on an object in a non-television area. The input receiving unit 23 detects and receives the start of the drag operation by the user (S301). If the received touch position coordinates overlap with an object on the non-television area, the object processing unit 26 selects the touched object (for example, object 2e). Here, the position specifying unit 20 of the tablet 100 monitors the positional relationship between the selected object 2e and the TV area 420 or the non-TV area. Specifically, it is monitored whether the object 2e is arranged on the TV area 420 or on the non-TV area.

  Subsequently, the object transmitting / receiving unit 25 transmits the selected object 2e (for example, an icon image) to the television 200 (S302). The object transmitting / receiving unit 50 of the television 200 stores the object 2e received from the tablet 100 in the storage unit 41 (S303).

  Here, if the drag operation continues (YES in S304), the object processing unit 26 moves the object 2e according to the current drag position (S305).

  If part of the object 2e remains on the non-television area (NO in S306), the object processing unit 26 continues to move the object as long as the drag operation is continued (YES in S304, S305). ). On the other hand, when the drag operation is completed before the object 2e has completely moved out of the non-TV area (NO in S304), the object processing unit 26 performs object selection or movement processing on the object 2e. Completion (S315).

  On the other hand, when the position specifying unit 20 determines that the object 2e has completely moved out of the non-TV area and has reached the TV area 420 (YES in S306), the coordinate processing unit 21 is the same as that in the first embodiment. Similar coordinate conversion processing is executed (S307). The signal output unit 22 inputs the operation position coordinates in the television coordinate system converted by the coordinate processing unit 21 with respect to the touch position R (see FIG. 25) when the object 2e reaches the television area 420 as an input signal ( 4th input signal). For the television 200, the touch position R in FIG. 25 is the drag start position. When the signal receiving unit 51 of the television 200 receives the operation position coordinates as an input signal (fourth input signal), the object processing unit 52 places the object 2e at the screen position of the display unit 42 designated by the operation position coordinates. Arrange and display (S308). The object 2e is acquired in advance in S303.

  On the other hand, the object processing unit 26 of the tablet 100 hides the object 2e that has completely reached the television area 420 (S309). In this way, the display of the object 2e when the object 2e moves from the tablet area 120 to the television area 420 is inherited from the tablet 100 to the television 200 in accordance with the drag operation.

  As in the first embodiment, the coordinate processing unit 21 and the signal output unit 22 continuously convert the current drag position coordinates into relative coordinates in the television coordinate system, and convert the input signal (fifth input signal) to the television 200. (S310). Similar to the first embodiment, the object processing unit 52 of the television 200 moves the object 2e according to the relative coordinates supplied as the input signal (fifth input signal) (S311). S310 and S311 are repeatedly executed while the drag operation is continued (YES in S312).

  On the other hand, when the input receiving unit 23 receives the end of the drag operation, that is, the drop operation on the TV area 420 (NO in S312), the signal output unit 22 stops the output of the input signal. In response to this, the object processing unit 52 of the television 200 stops the movement of the object 2e (see FIG. 26). Further, the object transmitting / receiving unit 25 of the tablet 100 transmits the entity data of the selected object 2e to the television 200 (S313). The object transmitting / receiving unit 50 of the television 200 stores the entity data received from the tablet 100 in the storage unit 41 (S314).

  According to the above configuration and method, object transmission / reception can be realized between the tablet 100 and the television 200 in accordance with a drag operation straddling the boundary between the non-television region and the television region 420. Furthermore, the display of the object that visually represents the transmission / reception of the object can be inherited from the tablet 100 by the television 200.

  After the object moves to the television area 420, the tablet 100 can be remotely operated so as to process a predetermined object with respect to the television 200 only by transmitting the operation position coordinates to the television 200. As a result, it is possible to reduce the communication load between devices and realize the tablet 100 with excellent responsiveness.

  Further, when the input and display mechanism of the operation device is realized by a touch panel like the tablet 100, there are the following merits.

  When the user wants the television 200 to capture an object displayed on the screen of the tablet 100 at hand, the user does not need to operate a dedicated remote controller for the television 200. According to the tablet 100 at hand in the present invention, the object selected by the touch operation can be transmitted from the tablet 100 and received by the television 200 simply by performing a touch operation (drag operation) on the input unit 11. It is. Specifically, the user may select an object displayed outside the television area 420 and drag and drop it into the television area 420. In addition, after the drop operation, the position of the object (icon image) received by the television 200 is intuitively moved to a desired position on the display unit 42 by a simple drag operation. Can do. In addition, when there are a plurality of devices to be transmitted (for example, the television 200 and the notebook computer 200a), it is intuitively possible to simply drag the object toward the area where the screen of the device of the transmission destination is reflected. You can specify the destination device.

  Thus, even if the television 200 is a device that does not support touch operation, an operation device that can perform a pseudo touch operation on such a device can be realized. In addition, since the display of the object accompanying the movement of the object is taken over from the tablet 100 to the television 200, the user can intuitively remotely operate the television 200 through a visually and natural user interface.

<< Embodiment 4 >>
Other embodiments relating to the operating device of the present invention will be described below with reference to FIGS. 10, 11, and 28 to 34.

  In the second and third embodiments, the operations of the tablet 100 and the television 200 in the case where the object moves in one direction due to the touch operation straddling the television region 420 and the non-television region are described. In the fourth embodiment, operations of the tablet 100 and the television 200 in a case where an object moves bidirectionally by simultaneously performing a touch operation on each of the television area 420 and the non-television area will be described.

[Composition of tablet and TV]
The principal part structure of the tablet 100 in this embodiment is as showing in FIG. The main configuration of the television 200 in the present embodiment is as shown in FIG. The tablet 100 and the television 200 in the present embodiment are different from the tablet 100 and the television 200 in the first to third embodiments as follows.

  In the present embodiment, the input receiving unit 23 receives a touch operation on an object in the television area 420 displayed on the display unit 42 of the television 200, and in the non-TV area displayed on the display unit 12. Accept touch operations on objects.

  At this time, the object transmission / reception unit 25 of the tablet 100 transmits the object selected in the non-TV area to the TV 200 and receives the object selected in the TV area 420 from the TV 200. On the contrary, the object transmitting / receiving unit 50 of the television 200 transmits the object selected in the television area 420 to the tablet 100 and receives the object selected in the non-television area from the tablet 100. That is, the tablet 100 and the television 200 exchange objects.

  The position specifying unit 20 of the tablet 100 monitors the overlapping of two objects selected simultaneously in the TV area 420 and the non-TV area. In the present embodiment, the time when the position specifying unit 20 determines that two objects overlap each other by a certain amount or more is set as the object exchange timing.

  In the present embodiment, it is conceivable that an object selected in the television area 420 cannot be displayed in the television area 420 before two objects overlap. In such a case, as shown in the second embodiment, the tablet 100 and the television 200 operate in cooperation to connect objects to each other. And the position specific | specification part 20 can monitor the overlap condition of two objects continuously.

[Explanation of operation for realizing pseudo touch operation-specific example]
A method for exchanging and displaying the object displayed on the tablet 100 and the object displayed on the television 200 using the tablet 100 will be described with reference to FIGS.

  As shown in FIG. 28, for example, as shown in the touch scene 3e, the user touches an object in the television area 420 with the index finger. At the same time, as shown in the touch scene 3f, the user touches an object in the non-TV area with the thumb. Here, the object touched in the touch scene 3e is the object 2c, and is actually an object displayed on the display unit 42 of the television 200 when touched. The object touched in the touch scene 3f is the object 2h, and is an object displayed on the display unit 12 of the tablet 100 when touched.

  As shown in FIG. 29, the input receiving unit 23 receives simultaneously the input of the touch position R1 by the touch scene 3e and the input of the touch position R2 by the touch scene 3f in the input unit 11. Here, the term “simultaneously” includes not only the case where the touched moments are completely the same, but also the case where the moments touched are deviated.

  The position specifying unit 20 grasps the TV area 420, the non-TV area, and the position of each object on each area. Therefore, each part of the control unit 10 confirms that the object 2h in the non-TV area is selected at the same time that the object in the TV area 420 is selected based on the two touch positions R1 and R2 detected by the input receiving unit 23. I can understand. Note that each unit of the control unit 10 does not need to know at this time until which object displayed on the television 200 side is selected. It is only necessary to understand that the object on the television 200 side has been selected.

  As described above, when the object 2c on the television 200 side and the object 2h on the tablet 100 side are simultaneously selected, the position specifying unit 20 starts to monitor the overlapping state of the selected object 2c and the object 2h. In addition, the signal output unit 22 and the object transmission / reception unit 25 communicate with the television 200 to exchange the selected object with the television 200.

  More specifically, first, the coordinate processing unit 21 converts the touch position coordinates (Xtab_r1, Ytab_r1) of the touch position R1 into operation position coordinates (Xtv_r1, Ytv_r1) in the television coordinate system. The signal output unit 22 transmits the converted operation position coordinates (Xtv_r1, Ytv_r1) to the television 200 as an input signal. This input signal is a signal for requesting the selected object 2c (icon image) as in the second embodiment. When the signal receiving unit 51 of the television 200 receives the input signal, the object transmitting / receiving unit 50 transmits the object 2c to the tablet 100. Here, the object processing unit 52 selects the object 2c.

  Next, the object transmitting / receiving unit 25 of the tablet 100 transmits the object 2h overlapping the touch position R2 to the television 200. Here, the object processing unit 26 selects the object 2h.

  When the object transmitting / receiving unit 50 of the television 200 receives the object 2h from the tablet 100, the object transmitting / receiving unit 50 stores the object 2h in the storage unit 41. When the object transmitting / receiving unit 25 of the tablet 100 receives the object 2c from the television 200, the object transmitting / receiving unit 25 stores the object 2c in the storage unit 15. This completes the object exchange.

  As shown in FIG. 30, the object 2c on the television 200 side that is dragged with the index finger moves in the same procedure as in the first embodiment. If half the area of the object 2c protrudes from the television area 420 before it is determined that the object 2c has sufficiently overlapped with the object 2h, the subsequent display is similar to that of the second embodiment. The object processing unit 26 may take over. Even in this case, the position specifying unit 20 can continuously monitor the position of the object 2c.

  On the other hand, the object 2h on the tablet 100 side that is dragged with the thumb moves and is displayed on the screen of the display unit 12 according to a normal drag operation until it is determined that the object 2c is sufficiently overlapped with the object 2c. Continue (see FIG. 30). The movement and display processing of the object 2h is executed by the object processing unit 26. In the present embodiment, unlike the third embodiment, even if the object 2h completely enters the television area 420 until it is determined that the object 2h overlaps the object 2c sufficiently, the object processing unit 26 is displayed on the display unit 12. The display of the object 2h is continued. That is, the display of the object on the tablet 100 side is not transferred to the television 200 before the objects overlap. By doing in this way, it is possible to continue monitoring the overlap of objects with a simple process on the tablet 100 side. However, the present invention is not limited to such a configuration. Similarly to the third embodiment, when the object 2h completely enters the television area 420 until it is determined that the object 2c sufficiently overlaps with the object 2c, the object 2h The configuration may be such that the television 200 takes over the display and movement. Even in this case, it is possible to monitor the overlap of objects displayed on the display unit 42 of the television 200 on the tablet 100 side.

  Thus, the object 2c on the TV 200 side and the object 2h on the tablet 100 side move so as to approach each other and overlap each other by the drag operation (or pinch-in operation) of the user.

  Then, as shown in FIGS. 31 and 32, it is assumed that both objects have reached the overlapping position. The position specifying unit 20 determines that both objects have sufficiently overlapped because the object 2c and the object 2h overlap each other by a certain area (for example, the hatched area in FIG. 32). The overlap threshold value referred to by the position specifying unit 20 may be set as appropriate.

  When the position specifying unit 20 detects the overlap of two objects, the signal output unit 22 stops the operation position coordinates of the object 2c so far and transmits another input to the television 200. Send a signal. Another input signal is an input signal that instructs the television 200 to display the replaced object (the object 2h that has been displayed on the tablet 100 side so far) on the display unit 42 of the television 200.

  When the signal receiving unit 51 of the television 200 receives the other input signal, the object processing unit 52 hides the object 2c if it has been displayed according to the input signal so far, The object 2h exchanged in advance is displayed on the display unit 42. In the present embodiment, the object processing unit 52, for example, at the time when the drag operation (touch scene 3e) is started, the object 2h is located at the position (see FIGS. 28 and 29) where the object 2c before replacement is displayed. Is displayed (see FIG. 33). However, the configuration of the object processing unit 52 is not limited to this. For example, the object processing unit 52 sets the object at the position of the original object 2c (see FIGS. 31 and 32) when the overlapping of the objects is detected (when the signal receiving unit 51 receives the other input signal). 2h may be displayed. Alternatively, the object 2h may be displayed at the position of the screen of the display unit 42 of the television 200 corresponding to the position of the object 2h displayed on the tablet 100 until just before the overlapping of the objects is detected. However, in this case, it is necessary for the tablet 100 to convert the touch position coordinates of the thumb that has selected the object 2h in the same manner as in the first embodiment, and to supply the converted coordinates to the television 200. The signal output unit 22 may transmit the operation position coordinates in the television coordinate system corresponding to the touch position coordinates of the thumb to the television 200 as another input signal.

  On the other hand, when the position specifying unit 20 detects the overlap of two objects, the object processing unit 26 of the tablet 100 executes the following display processing. The object processing unit 26 hides the object 2h displayed on the display unit 12 before overlapping. Then, the object processing unit 26 displays the object 2c exchanged with the television 200 on the display unit 12 instead. In the present embodiment, the object processing unit 26, for example, at the time when the drag operation (touch scene 3f) is started, the object 2c is located at the position (see FIGS. 28 and 29) where the object 2h before replacement is displayed. Is displayed (see FIG. 33). However, the configuration of the object processing unit 26 is not limited to this. For example, the object processing unit 26 may display the object 2c at the position of the object 2c and the object 2h (see FIGS. 31 and 32) at the time when the overlapping of the objects is detected.

  Furthermore, it is assumed here that the index finger and the thumb that have been performing the drag operation have left the input unit 11 after the position specifying unit 20 detects the overlapping of the objects. The input receiving unit 23 of the tablet 100 detects that a series of drag operations started in the touch scenes 3e and 3f has ended. In this case, the actual data of the exchanged object may be exchanged by the object transmission / reception unit 25 of the tablet 100 and the object transmission / reception unit 50 of the television 200. Specifically, when the drag operation ends, the signal output unit 22 transmits an input signal requesting the actual data of the object 2c to the television 200. The object transmission / reception unit 50 of the television 200 transmits the actual data of the object 2c in accordance with the input signal. The object transmitting / receiving unit 25 transmits the entity data of the object 2h to the television 200. The object transmitting / receiving unit 50 receives the entity data of the object 2h from the tablet 100. Thereby, the exchange of the entity data of the object is completed.

[Flow of processing to realize pseudo touch operation]
FIG. 34 is a flowchart showing a process flow of the tablet and the television according to the embodiment of the present invention.

  It is assumed that the drag operation is simultaneously started on both the non-TV area object and the TV area 420 object. The input receiving unit 23 detects and receives the start of these two drag operations (S401).

  The tablet 100 transmits the object 2h selected in the non-TV area to the TV 200 and requests the object selected in the TV area 420 (S402). Specifically, first, the object processing unit 26 selects the object 2h if the touch position coordinate received by the input receiving unit 23 overlaps the object on the non-television area. Then, the object transmission / reception unit 25 transmits the object 2h to the television 200. On the other hand, the coordinate processing unit 21 converts the touch position coordinates received on the television area 420 into operation position coordinates in the television coordinate system, as in the first embodiment. The signal output unit 22 transmits the operation position coordinates to the television 200 as an input signal. This input signal becomes an input signal (sixth input signal) for requesting an object (first object).

  Upon receiving the object 2h from the tablet 100, the object transmitting / receiving unit 50 of the television 200 stores it in the storage unit 41 (S403). When the signal receiving unit 51 of the television 200 receives the input signal, the object processing unit 52 selects the object 2c arranged at the position specified by the operation position coordinates as the input signal. The object transmitting / receiving unit 50 transmits the selected object 2c to the tablet 100 according to the input signal (S403). The object transmitting / receiving unit 25 of the tablet 100 stores the object 2c received from the television 200 in the storage unit 15 (S404). This completes the object exchange.

  Subsequent to S402, the position specifying unit 20 specifies the position of the object 2h selected on the non-TV area and the position of the object 2c selected on the TV area 420, and monitors the overlapping state of these positions. (S405).

  If the drag operation accepted in S401 continues, the selected object is moved in cooperation with the tablet 100 and the television 200 according to each drag operation (S406). The signal output unit 22 transmits an input signal (operation position coordinates and relative coordinates; seventh input signal) for moving the object 2 c on the television area 420 to the television 200. The object processing unit 52 of the television 200 moves the object 2c according to the input signal (S407). The object processing unit 26 of the tablet 100 moves the object 2h on the tablet area 120.

  Here, it is assumed that the position specifying unit 20 determines that the object 2c and the object 2h overlap each other at a ratio of a certain area (for example, FIG. 32). In this case (YES in S408), the signal output unit 22 inputs an input signal (eighth input) for instructing to display the replaced object (that is, the object 2h here; the second object) on the television 200 side. Signal) is transmitted to the television 200 (S409). When the signal receiving unit 51 of the television 200 receives the input signal, the object processing unit 52 displays the object 2h received in S403 on the display unit 42 (S410). Here, when the object processing unit 52 has displayed the object 2c so far, it is preferable to hide the object 2c. The position on the display unit 42 where the object processing unit 52 displays the object 2h is not particularly limited here.

  On the other hand, in the tablet 100, when the objects overlap (YES in S408), the object processing unit 26 hides the object 2h that has been displayed on the display unit 12 so far, and the object 2c received in S404 is displayed. This is displayed on the display unit 12 (S411). The position on the display unit 12 where the object processing unit 26 displays the object 2c is not particularly limited here.

  Further, when the input receiving unit 23 detects the end of the drag (pinch-in) operation, the object transmitting / receiving unit 25 transmits the actual data of the replaced object 2h to the television 200 (similar to S313 in FIG. 27). The object transmitting / receiving unit 50 of the television 200 stores the entity data received from the tablet 100 in the storage unit 41 (similar to S314 in FIG. 27). Further, the signal output unit 22 transmits an input signal (a ninth input signal) for requesting entity data associated with the exchanged object 2c to the television 200 (similar to S216 in FIG. 19). The object transmitting / receiving unit 50 of the television 200 transmits the entity data of the object 2c to the tablet 100 (similar to S217 in FIG. 19). The object transmitting / receiving unit 25 receives the entity data of the object 2c and stores it in the storage unit 15 (similar to S218 in FIG. 19). In this way, the exchange of entity data is also completed.

  According to the above configuration and method, it is possible to realize object exchange between the tablet 100 and the television 200 in accordance with a drag operation that is simultaneously performed on the non-television region and the television region 420. Furthermore, it is possible to display an object that visually represents bidirectional transmission / reception of the object in cooperation between the tablet 100 and the television 200.

  After the respective objects in the non-TV area and the TV area 420 are overlapped by the drag operation, the tablet 100 sends an input signal for instructing the display of the object exchanged in advance to the TV 200. Just send it. The tablet 100 implements object exchange with the television 200 by remote control using an input signal. The tablet 100 and the television 200 perform data transmission / reception on object or entity data that actually needs to be exchanged. However, it is not necessary to perform large-capacity data communication for the purpose of sharing the touch position or data exchange timing. As a result, it is possible to reduce the communication load between devices and realize the tablet 100 with excellent responsiveness.

  Further, when the input and display mechanism of the operation device is realized by a touch panel like the tablet 100, there are the following merits.

  When the user wants to exchange an object displayed on the screen of the tablet 100 at hand and an object displayed on the screen of the television 200, the user does not need to operate a remote control dedicated to the television 200 or the like. According to the tablet 100 at hand in the present invention, the two objects selected by the touch operation can be exchanged between the tablet 100 and the television 200 simply by touching the input unit 11 (drag operation). Because you can. Specifically, the user selects an object displayed in the television area 420, selects an object displayed on the non-television area, and drags (or pinch-in) so that they are brought close to each other. do it.

  Thus, even if the television 200 is a device that does not support touch operation, an operation device that can perform a pseudo touch operation on such a device can be realized. In addition, since the display of the object accompanying the movement of the object is performed in cooperation between the tablet 100 and the television 200, the user can remotely operate the television 200 intuitively through a visually and natural user interface. Become.

<< Modification 1 >>
The first embodiment has a configuration in which the tablet 100 causes the television 200 to process an object displayed on the television 200 by remotely operating the television 200 in accordance with a touch (drag) operation on the television area 420 of the tablet 100. .

  In the second and third embodiments, the tablet 100 remotely controls the TV 200 in accordance with a touch (drag) operation that crosses the boundary between the TV area 420 and the non-TV area of the tablet 100. It is the structure which implement | achieves transmission / reception (movement of one direction) of objects between.

  In the fourth embodiment, the tablet 100 remotely operates the TV 200 in accordance with the touch (drag) operation received simultaneously in the TV area 420 and the non-TV area of the tablet 100, so In this configuration, object exchange (bidirectional movement) was realized.

  However, in addition to any of the above-described configurations, the tablet 100 of the present invention transmits an object displayed on the tablet 100 to the television 200 in accordance with a touch operation on the non-TV area of the tablet 100. The television 200 may be remotely operated so that the transmitted object is processed by the television 200.

  FIG. 35 is a diagram illustrating an example of the display screen of the display unit 12 when the input receiving unit 23 receives a touch operation on the non-TV area of the tablet 100.

  In a state where the display control unit 24 of the tablet 100 displays a live view image in which the television 200 is displayed, the input reception unit 23 receives an operation on an object (for example, the object 2h) on the non-TV area. Here, the user performs a touch operation with a gesture that emphasizes the intention to select the object 2h. For example, a touch operation such as a long press operation, a double tap operation, or a touch operation that draws a slide or a circle repeatedly in a certain narrow area (an operation for strongly pressing an image) is performed.

  In this way, when the input receiving unit 23 receives an operation of strongly pressing an object on the non-television area while the live view image of the television 200 is displayed, each unit of the control unit 10 causes the pressed object 2h. For displaying on the TV 200. Specifically, the object transmitting / receiving unit 25 transmits the selected object 2h to the television 200. The signal output unit 22 generates an input signal (tenth input signal) for instructing to display the object 2 h on the display unit 42 of the television 200, and transmits the input signal to the television 200. The object processing unit 26 hides the object 2h that has been displayed so far.

  On the other hand, the object transmitting / receiving unit 50 of the television 200 receives the object 2 h from the tablet 100 and stores it in the storage unit 41. When the signal receiving unit 51 receives the input signal, the object processing unit 52 displays the received object 2h on the display unit 42 according to the input signal (see FIG. 36). FIG. 36 is a diagram illustrating an example of a display screen of the display unit 12 after the object processing unit 52 displays the object 2h on the display unit 42. As shown in FIG. 36, the object processing unit 26 of the tablet 100 hides the selected object 2h, while the object processing unit 52 displays the object 2h on the display unit 42 of the television 200.

  During or before and after the series of display cooperation described above, the object transmitting / receiving unit 25 of the tablet 100 may transmit the entity data of the object 2h to the object transmitting / receiving unit 50 of the television 200.

  According to the above configuration, even when the TV area 420 is not sufficiently secured on the tablet area 120, the target object is transmitted from the tablet 100 to the TV 200 by the intuitive touch operation for the user, and the TV 200 side Can be displayed. As a situation where the TV area 420 is not secured, for example, only a part of the screen of the display unit 42 is shown in the live view image, and the TV area 420 is not large enough to drag and drop the object on the tablet 100 side. A situation is assumed.

<< Modification 2 >>
The second embodiment has a configuration in which the tablet 100 receives an object transmitted to the television 200 by remotely operating the television 200 in accordance with a drag operation extending from the television area 420 of the tablet 100 to the non-TV area.

  However, instead of (or in addition to) the above configuration, the tablet 100 of the present invention receives an object displayed on the television 200 from the television 200 in accordance with a touch operation on the television area 420 of the tablet 100, and receives it. The structure displayed on the display part 12 of the tablet 100 may be sufficient.

  FIG. 37 is a diagram illustrating an example of a display screen of the display unit 12 when the input receiving unit 23 receives a touch operation on the TV area 420 of the tablet 100.

  In a state where the display control unit 24 of the tablet 100 displays a live view image in which the television 200 is displayed, the input reception unit 23 receives an operation on an object (for example, the object 2c) on the television area 420. Here, the user performs a touch operation with a gesture that emphasizes the intention to select, on the object 2c. For example, a touch operation such as a long press operation, a double tap operation, or a touch operation that draws a slide or a circle repeatedly in a certain narrow area (an operation for strongly pressing an image) is performed.

  As described above, when the input receiving unit 23 receives an operation of strongly pressing an object on the TV area 420 (for example, a long press operation) while the live view image of the TV 200 is displayed, each unit of the control unit 10. Performs an operation for acquiring the long-pressed object 2c from the television 200 and causing the display unit 12 to display the object 2c. Specifically, first, the coordinate processing unit 21 converts the long-pressed touch position coordinates into operation position coordinates in the television coordinate system, as in the first embodiment. The signal output unit 22 transmits the converted operation position coordinates to the television 200 as an input signal (first input signal). The signal output unit 22 continuously transmits an input signal (an eleventh input signal) while the input receiving unit 23 detects a long press operation.

  When the signal receiving unit 51 of the television 200 receives the input signal, the object transmitting / receiving unit 50 transmits the object 2c specified by the operation position coordinates to the tablet 100. Then, the object processing unit 52 does not move the selected object 2c while the signal receiving unit 51 receives the input signal, but adds a visual effect (animation) that changes the shape of the object 2c. (See FIG. 38). FIG. 38 is a diagram illustrating an example of a display screen of the display unit 12 when the object processing unit 52 adds a visual effect to the object 2c displayed on the display unit 42. For example, while the area of the object 2c of the input unit 11 is long pressed by the user, the object processing unit 52 generates an animation (a visual effect that reduces the shape) of the object 2c with respect to the object 2c. To add.

  Next, when the user releases the finger after the long press operation, the input receiving unit 23 of the tablet 100 detects that the long press operation has ended. Accordingly, the signal output unit 22 stops outputting the input signal. The object processing unit 52 of the television 200 may hide the object 2c displayed on the display unit 42 of the television 200 as the input signal is stopped.

  On the other hand, in the tablet 100, the object processing unit 26 starts displaying the object 2c. Since the object transmission / reception unit 25 has received the object 2c in advance from the object transmission / reception unit 50 of the television 200, the object processing unit 26 can start displaying the object 2c when the long press operation is completed. Here, it is desirable that the object processing unit 26 takes over the animation of the television 200. Specifically, as shown in FIG. 39, the object processing unit 26 takes over the shape in which the object processing unit 52 of the television 200 displayed the object 2c immediately before the long press operation is ended, The display of 2c is started. The immediately preceding shape may be specified by the position specifying unit 20 from the live view image. Then, the object processing unit 26 adds an animation (visual effect for enlarging the shape) such that the object 2c jumps out of the television 200 in the back to the object 2c. For example, as shown in FIG. 40, the object processing unit 52 enlarges the size while changing the shape when the display is started. Furthermore, as shown in FIG. 41, the object processing unit 52 continues to change and enlarge the shape of the object 2c, and adds an animation as if the object 2c pops out. Then, the object processing unit 52 gradually restores the shape of the object 2c and finally, as shown in FIG. 42, is almost the same as the state displayed on the display unit 42 of the television 200 (see FIG. 37). In the state, the object 2c is displayed.

  During or before and after the series of display cooperation described above, the object transmitting / receiving unit 25 of the tablet 100 may receive the entity data of the object 2c from the object transmitting / receiving unit 50 of the television 200.

  According to the above configuration, even when a non-TV area is not sufficiently secured on the tablet area 120, the user can receive a target object from the TV 200 and display it on the tablet 100 side with an intuitive touch operation. Can do. As a situation where the non-TV area is not secured, for example, a situation is assumed in which the screen of the display unit 42 is reflected in the live view image and the non-TV area is not large enough to drag and drop the object on the TV 200 side. Is done.

[Additional notes]
Note that the input receiving unit 23 of the tablet 100 may receive a gesture that encloses an object in a circle as a touch operation involving a gesture that emphasizes the intention to select. When the input receiving unit 23 receives a simple tap operation on the television area 420 as an operation on the television 200, the input receiving unit 23 reacts to a finger touch unintended by the user, and the television 200 is unintentionally activated. There is a possibility of erroneous operation such as remote operation. However, when a touch operation on the television 200 is received by an intentional gesture of encircling an object to be received, there is an advantage that it is difficult to cause an erroneous operation. In addition, it is difficult to distinguish whether the gesture of enclosing a circle is a drag operation or a circle-enclosed gesture in an area where the selectable object is displayed on the display unit 12 side of the tablet 100 (such as the object placement area 4). May cause inconvenience. However, the operation target area when selecting an object to be received is the TV area 420, and the TV area 420 is a single live view image acquired by the imaging unit 13, and a drag operation that moves the image is performed. Therefore, there is an advantage that it can be recognized easily and accurately without erroneously recognizing the gesture of enclosing it in a circle.

  In the above-described embodiments, while the object (icon image) is moved by the drag operation, the display control unit 24 does not always display the live view image, but one frame at the time when the drag operation is started. It may be replaced with (captured image) and displayed on the display unit 12. A user who operates the tablet 100 in an attempt to remotely operate the television 200 is considered to focus on the display unit 42 of the television 200, not the display unit 12 of the tablet 100. Therefore, even if the display content of the display unit 12 is different from the display content of the display unit 42, the user does not feel great discomfort.

  According to the above configuration, the television area 420 is not changed little by little when the tablet 100 held by the user moves in an unstable manner. Therefore, it is possible to greatly reduce the processing load of calculation when the position specifying unit 20 specifies the position of each region and each object, or when the coordinate processing unit 21 performs coordinate conversion.

  It is assumed that the imaging unit 13 cannot capture the entire area of the screen of the display unit 42 of the television 200, that is, the live view image includes only a part of the screen of the display unit 42. In this case, the position specifying unit 20 is based on the specification information (number of inches, vertical and horizontal sizes, screen aspect ratio, etc.) of the display unit 42 of the TV 200 registered in advance, and the TV area that could not be captured in the live view image. Complementing 420, the origin (0, 0) of the television coordinate system and the entire region of the television region 420 can be defined in the tablet coordinate system.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

[Example of software implementation]
Finally, each block of the tablet 100 and the television 200, in particular, the position specifying unit 20, the coordinate processing unit 21, the signal output unit 22, the input receiving unit 23, the display control unit 24, the object transmitting / receiving unit 25, the object processing unit 26, and The object transmitting / receiving unit 50, the signal receiving unit 51, the object processing unit 52, the object position determining unit 53, and the display control unit 54 may be configured by hardware logic, or realized by software using a CPU as follows. May be.

  That is, the tablet 100 and the television 200 include a CPU (central processing unit) that executes instructions of a control program that realizes each function, a ROM (read only memory) that stores the program, and a RAM (random access memory) that expands the program. ), A storage device (recording medium) such as a memory for storing the program and various data. An object of the present invention is a recording medium in which a program code (execution format program, intermediate code program, source program) of a control program for the tablet 100 or the television 200, which is software that realizes the above-described functions, is recorded in a computer-readable manner. Can also be achieved by supplying the program to the tablet 100 or the television 200 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

  Examples of the recording medium include a tape system such as a magnetic tape and a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, and an optical disk such as a CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  Further, the tablet 100 and the television 200 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Further, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered Trademark), 802.11 wireless, HDR (High Data Rate), mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

  The present invention can be suitably used for an operating device for remotely operating a display device and a remote operating system including each device.

2a to 2j Object 4 Object arrangement area (non-television area / second area)
DESCRIPTION OF SYMBOLS 10 Control part 11 Input part 12 Display part 13 Imaging part 14 Communication part 15 Storage part 20 Position specific part (position specific means)
21 Coordinate processing unit (coordinate processing means)
22 Signal output section (signal output means)
23 Input reception part (input reception means)
24 display control unit (display control means)
25 Object transmission / reception unit (transmission means / reception means / transmission / reception means)
26 Object processing unit (object processing means)
40 control unit 41 storage unit 42 display unit 43 communication unit 50 object transmission / reception unit (transmission means / reception means / transmission / reception means)
51 Signal reception unit (signal reception means)
52 Object processing unit (object processing means)
53 Object position determination unit (determination means)
54 Display control unit (display control means)
100 tablet (operating device)
120 Tablet area 200 Television (display device)
200a notebook computer (display device)
200b Digital photo frame (display device)
420 TV area (first area)

Claims (21)

An operating device for remotely operating a display device,
Display control means for displaying image data obtained by imaging the display device on a display unit of the operation device;
Position specifying means for specifying the position of the first region representing the display unit of the display device in the display unit of the operation device from the displayed image data;
Input receiving means for receiving an operation signal input to the first region;
Based on the position of the first area specified by the position specifying means, the operation position coordinate indicating the position where the operation signal is received on the display unit of the operation device is set as the operation position coordinate on the display unit of the display device. Coordinate processing means to convert;
An operation device comprising: signal output means for outputting operation position coordinates converted by the coordinate processing means as an input signal to the display device. The display unit of the operation device is formed integrally with an input unit for inputting the operation signal, constitutes a touch panel,
The input receiving means receives an operation signal input by a touch operation on the touch panel,
The operation device according to claim 1, wherein the coordinate processing unit converts operation position coordinates on the touch panel of the contact operation into operation position coordinates on a display unit of the display device.   The signal output means outputs an input signal instructing the display device to select an object displayed on the display unit of the display device that overlaps the operation position coordinates converted by the coordinate processing means. The operating device according to claim 1 or 2.   The signal output means instructs the display device to move the selected object displayed on the display unit of the display device to a position indicated by the operation position coordinate converted by the coordinate processing means. The operating device according to any one of claims 1 to 3, wherein an input signal is output. In addition to the first area, the input receiving means further receives an operation signal input to a second area other than the first area on the display unit of the operating device.
The signal output means is
An input signal for instructing the display device to transmit an object displayed on the display unit of the display device, which overlaps the operation position coordinates converted by the coordinate processing means, to the operation device; and
Instructing the display device to move the object selected by the operation signal to the position indicated by the operation position coordinate converted by the coordinate processing means in the second region of the display unit of the operation device. The operating device according to any one of claims 1 to 4, wherein at least one of the input signals is output.   6. The operating device according to claim 5, wherein the input receiving means receives an operation signal input by a drag operation from the first area to the second area. The operating device is
An object processing means for processing an object displayed on the display unit of the operation device;
While the input accepting means accepts an operation signal in the first area,
The coordinate processing means converts the operation position coordinates of the drag operation on the display unit of the operation device into operation position coordinates on the screen of the display device,
The signal output means is
A first input signal for instructing the display device to select an object displayed on the display unit of the display device that overlaps the operation position coordinate converted by the coordinate processing means and to transmit to the operation device. Output,
A second input signal for instructing the display device to move the selected object displayed on the display unit of the display device to the position indicated by the operation position coordinate converted by the coordinate processing means is output. And
While the input accepting means accepts an operation signal in the second area,
The object processing means displays and moves the object received from the display device in response to the first input signal at a position indicated by the operation position coordinates of the drag operation on the display unit of the operation device. The operating device according to claim 6. When the input receiving means stops receiving an operation signal in the second area,
The signal output means outputs a third input signal instructing the display device to transmit substance data associated with the object selected in accordance with the first input signal to the operation device. The operating device according to claim 7, wherein   6. The operating device according to claim 5, wherein the input receiving means receives an operation signal input by a drag operation from the second area to the first area. The operating device is
An object processing means for processing an object displayed on the display unit of the operating device;
Transmission means for transmitting the object stored in the operation device to the display device,
While the input accepting means accepts an operation signal in the second area,
The object processing means selects an object that overlaps the operation position coordinate of the drag operation on the display unit of the operation device, and the operation position coordinate of the drag operation on the display unit of the operation device indicates the selected object. Move it to the position
The transmission means transmits the object selected by the object processing means to the display device,
While the input accepting means accepts an operation signal in the first area,
The coordinate processing means converts the operation position coordinates of the drag operation on the display unit of the operation device into operation position coordinates on the display unit of the display device,
The signal output means is
A fourth input signal for instructing the display device to display the selected object at the position of the display unit of the display device indicated by the operation position coordinates converted by the coordinate processing means;
Outputting a fifth input signal for instructing the display device to move the object displayed in accordance with the fourth input signal to the position indicated by the operation position coordinate converted by the coordinate processing means; The operating device according to claim 9. When the input receiving means stops receiving an operation signal in the first area,
11. The operating device according to claim 10, wherein the transmission unit transmits entity data associated with the object selected by the object processing unit to the display device.   The input receiving means receives an operation signal input by a first drag operation started on the first area and receives an operation signal input by a second drag operation started on the second area. The operating device according to claim 5. The operating device is
An object processing means for processing an object displayed on the display unit of the operating device;
Transmission means for transmitting the object stored in the operation device to the display device,
The position specifying means specifies the position of the first object that moves from the first area to the second area in accordance with the first drag operation on the display unit of the operating device, and also performs the second drag operation. The position of the second object that moves from the second area to the first area along with the position on the display unit of the operating device is identified,
Until the position specifying means determines that the first object and the second object overlap each other,
The signal output means is
Instructing the display device to transmit, to the operation device, the first object displayed on the display unit of the display device that overlaps the operation position coordinates of the first drag operation converted by the coordinate processing means. To output a sixth input signal,
A seventh input signal for instructing the display device to move the selected first object displayed on the display unit of the display device to a position indicated by the operation position coordinate converted by the coordinate processing means. Output
The transmission means transmits the second object selected by the second drag operation to the display device,
When the position specifying means determines that the first object and the second object overlap,
The signal output means is
Outputting an eighth input signal for instructing the display device to display the second object displayed on the display device of the operation device on the display device of the display device;
13. The operating device according to claim 12, wherein the object processing means displays the first object received from the display device in response to the sixth input signal on a display unit of the operating device. When the input receiving means stops receiving operation signals in the first area and the second area,
The signal output means outputs a ninth input signal for instructing the display device to transmit the entity data associated with the first object to the operation device;
The operation device according to claim 13, wherein the transmission unit transmits entity data associated with the second object to the display device. The operating device is
An object processing means for processing an object displayed on the display unit of the operation device;
When the input receiving means receives an operation signal input to the second area,
The object processing means hides the object selected by the operation signal in the second area of the display unit of the operation device,
15. The signal output means outputs a tenth input signal that instructs the display device to display the selected object on a display unit of the display device. The operating device according to any one of the above. The operating device is
An object processing means for processing an object displayed on the display unit of the operation device;
The input receiving means receives an operation signal input by an operation on an object on the first area,
The signal output means is
A first input signal for instructing the display device to select an object displayed on the display unit of the display device that overlaps the operation position coordinate converted by the coordinate processing means and to transmit to the operation device. Output,
An eleventh input signal for instructing the display device to give a visual effect of changing or reducing the shape to the selected object displayed on the display unit of the display device;
The object processing means displays the object received from the display device in response to the first input signal at the position indicated by the operation position coordinates of the operation on the display unit of the operation device, and changes or enlarges the visual The operating device according to claim 1, wherein an operation effect is imparted.   A display device that operates according to an input signal output from the operating device according to any one of claims 1 to 16. The operating device according to any one of claims 1 to 16,
A remote operation system comprising: a display device that operates in accordance with an input signal output from the operation device. A control method of an operating device for remotely operating a display device,
A display control step of displaying image data obtained by imaging the display device on a display unit of the operation device;
A position specifying step for specifying the position of the first region representing the display unit of the display device in the display unit of the operation device from the image data displayed in the display control step;
An input receiving step for receiving an operation signal input to the first region;
Based on the position of the first region specified in the position specifying step, the operation position coordinate indicating the position at which the operation signal is received on the display unit of the operation device is represented as the operation position coordinate on the display unit of the display device. A coordinate processing step to convert to
And a signal output step of outputting the operation position coordinates converted in the coordinate processing step as an input signal to the display device.   The control program for functioning a computer as each means of the operating device of any one of Claim 1-16.   A computer-readable recording medium on which the control program according to claim 20 is recorded.

Images