JP2014137667A - Control device and control method - Google Patents

Abstract

An object of the present invention is to enable control according to an operator's gesture without performing complicated processing such as capturing a change in hand shape.
A control apparatus captures an operator and captures a motion of a specific part of the operator from a camera as an imaging unit that generates a video signal and a video indicated by the video signal generated by the camera. A target part position detection unit 102 that detects the mouse, a mouse event conversion unit 103 that generates a mouse event when the mouse moves according to the movement detected by the target part position detection unit 102, and a mouse event conversion unit 103 And a display unit 106 for displaying a graphical user interface screen according to the mouse event.
[Selection] Figure 1

Description

  The present invention relates to a control device and a control method.

Currently, TV receivers and the like are generally operated from a remote place using a remote controller. In recent years, with the improvement of the performance of a CPU incorporated in a TV receiver or the like, a more convenient and convenient operation method has been sought.
As one of them, a method using a gesture has been proposed. For example, there is a technique for causing an operation target device to recognize an operation to be performed by the user based on the movement and shape of the user's hand. Patent Document 1 discloses a method of detecting the movement and shape of a user's hand taken by a camera and controlling a target device based on the locus and shape.
Patent Document 2 discloses a method for suppressing erroneous recognition by providing an object identifying unit that identifies whether an object has a predetermined shape from a plurality of images.

JP 2002-83302 A International Publication No. 2011/027397

  When using a gesture in this way, it is necessary to recognize a gesture corresponding to the operation of the device to be controlled and connect it to a specific operation of a specific device. For this reason, it is necessary to accurately recognize the gesture, and the user needs to know the correspondence between the gesture and the operation for each device.

  Also, when the contents operated by gestures have a wide range of options, for example, when trying to distinguish the options by the shape of the hand, it is necessary to accurately determine the shape, and recognition of practical accuracy is performed. There are difficulties.

  Therefore, the present invention is characterized in that control according to the gesture of the operator can be performed without performing complicated processing such as capturing a change in the shape of the hand.

  The control device according to an aspect of the present invention captures an operator, generates an image signal indicating an image of the operator, and an image indicated by the image signal generated by the image capturing unit. A target part position detecting unit for detecting a movement of a specific part of the operator; a mouse event converting unit for generating a mouse event when the mouse moves in accordance with the movement detected by the target part position detecting unit; and the mouse And a display unit for displaying a graphical user interface screen in accordance with the mouse event generated by the event conversion unit.

  A control method according to an aspect of the present invention includes: an imaging process for capturing an operator and generating a video signal indicating an image of the operator; and an image indicated by the video signal generated in the imaging process, A target part position detection process for detecting movement of a specific part of the operator, a mouse event conversion process for generating a mouse event when the mouse moves according to the movement detected in the target part position detection process, and the mouse A display process for displaying a graphical user interface screen in accordance with a mouse event generated in the event conversion process.

  According to one aspect of the present invention, control according to an operator's gesture can be performed without performing complicated processing such as capturing a change in the shape of a hand.

2 is a block diagram schematically showing the configuration of a control device according to Embodiment 1. FIG. 4 is a flowchart illustrating an example of processing in the control device according to the first embodiment. 6 is a schematic diagram illustrating an example of a template according to Embodiment 1. FIG. FIG. 5 is a block diagram schematically showing a configuration of a control device according to a second embodiment. FIG. 10 is a block diagram schematically showing a configuration of a control system according to a third embodiment.

Embodiment 1 FIG.
FIG. 1 is a block diagram schematically showing a configuration of a control device 100 according to the first embodiment. The control device 100 includes a camera 101, a target part position detection unit 102, a mouse event conversion unit 103, an input unit 104, a graphical user interface creation unit (hereinafter referred to as GUI creation unit) 105, and a display unit 106. Prepare.

The camera 101 is an imaging unit that captures an image of an operator and generates a video signal indicating the image of the operator. In particular, the camera 101 captures an image so as to include a target portion for detecting movement, such as an operator's hand. The camera 101 gives a video signal indicating the captured video to the target part position detection unit 102.
The target part position detection unit 102 detects the movement of the target part of the operator from the video indicated by the video signal given from the camera 101. For example, first, the target part position detection unit 102 specifies the target part of the operator from the video indicated by the video signal given from the camera 101. Then, the target part position detection unit 102 specifies the direction and speed (magnitude) of the movement of the specified target part as a motion vector on the predetermined two-dimensional coordinates in the video. The target part position detection unit 102 gives the identified motion vector to the mouse event conversion unit 103.

  The mouse event conversion unit 103 generates a mouse event when the mouse moves in accordance with the movement detected by the target part position detection unit 102. For example, the mouse event conversion unit 103 generates a mouse event equivalent to a mouse event generated when the mouse moves based on the motion vector given from the target part position detection unit 102. Then, the mouse event conversion unit 103 gives the generated mouse event to the GUI creation unit 105.

  The input unit 104 receives an instruction input. The input unit 104 can be configured by a mouse or a keyboard. Then, the input unit 104 gives the input instruction to the GUI creation unit 105 as a mouse event or a keyboard event.

The GUI creation unit 105 generates a graphical user interface (hereinafter referred to as GUI) in accordance with at least one of the mouse event given from the mouse event conversion unit 103 and the mouse event or keyboard event given from the input unit 104. create. Then, the GUI creation unit 105 gives the created GUI to the display unit 106.
The display unit 106 displays the GUI screen given from the GUI creation unit 105. Here, since the GUI given from the GUI creation unit 105 is created according to the mouse event generated by the mouse event conversion unit 103, the display unit 106 displays the GUI according to the mouse event generated by the mouse event conversion unit 103. The screen will be displayed.

FIG. 2 is a flowchart illustrating an example of processing in the control device 100.
First, the camera 101 captures an image including the operator and provides the image signal to the target part position detection unit 102 (S10).

  The target part position detection unit 102 specifies the operator's target part from the video indicated by the video signal given from the camera 101, and the specified target part on the predetermined two-dimensional coordinate axis in the video. The position at is detected (S11). Since it is common to use a hand as a target in a gesture, the target part is a hand. In addition, various developments have been made for hand position detection methods and the like. Here, as the method for detecting the position of the hand itself, it is assumed that an existing method is utilized and a method considered to be optimal is used. For example, US Pat. No. 5,594,469, Freeman et al. The hand position detection method described in Mitsubishi Electric Information Technology Center America, “Hand gesture machine control system” can be used. Hereinafter, an example of a hand position detection method based on the description in US Pat. No. 5,594,469 will be described.

  The operator causes the target part position detection unit 102 to recognize the hand that is the target part to be monitored by holding the palm of the hand toward the camera 101. The target part position detection unit 102 detects the position of the hand using the correlation of the local orientation. Specifically, it is as follows.

First, the operator uses the input unit 104 to indicate the initial position of the hand while displaying on the display unit 106 the video indicated by the video signal including the hand sent from the camera 101. The target part position detection unit 102 converts a portion corresponding to the range indicated by the initial position in the video indicated by the video signal sent from the camera 101 into a digital signal using a video quantizer (not shown). Then, the target part position detection unit 102, in the digitalized video, the differential value dI / dX of the image intensity of the pixel in the x-axis direction, and the differential value dI / dY of the image intensity in the y-axis direction orthogonal to the x-axis. Is calculated for the entire digitally converted screen. Further, the target part position detection unit 102 generates a contrast image by taking the square root of the value obtained by adding (dI / dX) ² and (dI / dY) ² . Here, the target part position detection unit 102 performs threshold processing on the contrast image, thereby ignoring the low-contrast portion and removing noise. Here, the value of each pixel constituting the contrast image is defined as a contrast value.

  Next, the target part position detection unit 102 has a value obtained by dividing the differential value in the x-axis direction by the corresponding contrast value and a value obtained by dividing the differential value in the y-axis direction by the corresponding contrast value for the detection target part. A value θ that is cos θ and sin θ is obtained as a function of position. That is, the target part position detection unit 102 defines sin θ. This set of data pairs (cos θ, sin θ) constitutes a template for detecting the position of the hand. FIG. 3 is a schematic diagram showing an example of a template configured as described above. Then, the target part position detection unit 102 stores the template configured as described above in the memory 102a serving as a template storage unit.

  Returning to the description of FIG. 2, the target part position detection unit 102 detects the movement of the hand (S12). As for the method of detecting hand movements, a method that seems to be optimal will be used by utilizing existing methods. For example, the hand movement detection method described in the above-mentioned US Pat. No. 5,594,469 can be used. Hereinafter, an example of a hand movement detection method based on the description of US Pat. No. 5,594,469 will be described.

  The target part position detection unit 102 generates a target image consisting of a set of data pairs (cos θ, sin θ) from the video signal from the camera 101 in the same procedure as that for forming a template. Here, the target image may be generated for all the ranges (full screen) indicated by the video signal from the camera 101, or for a predetermined range around the position where the previous hand position was detected. May be. Then, when the detection of the hand movement is started, the target part position detection unit 102 detects the initial position of the hand using the template created in the preparation stage using the correlation between the target image and the template. This detection is performed by the following process, for example.

First, for each of the template and the target image, the target part position detection unit 102 sets a value obtained by dividing the differential value in the x-axis direction by the contrast value and a value obtained by dividing the differential value in the y-axis direction by the contrast value. Aggregating into large vectors, the normalized correlation value between the two vectors is calculated by the following equation (1).

  Further, in the area obtained by dividing the video indicated by the video signal sent from the camera 101, the ratio of the portion where the contrast value exceeds the threshold is calculated, and the ratio is close to the ratio calculated in the template. For example, the part is specified by a threshold or the like, and is set as a candidate for a region including a position where the hand is the target part. When the position of the hand is determined by the combination of the correlation values as described above, the movement of the hand can be tracked by monitoring the similar correlation value in the peripheral region of the determined position. Can get the movements.

  Then, the target part position detection unit 102 calculates a motion vector at regular time intervals from the detected hand position information, and gives the calculated motion vector to the mouse event conversion unit 103.

  Next, the mouse event conversion unit 103 converts the direction and speed of the motion vector given from the target part position detection unit 102 into the direction and speed of moving the mouse (S13).

In the mouse, an event occurs when the current coordinates (pointer position) of the mouse or the state of the button determined by the movement detected by the optical sensor changes. In this embodiment, attention is paid only to the movement of the hand as a gesture, and attention is paid to the direction and speed of moving the hand. The mouse event conversion unit 103 generates an equivalent mouse event by converting the first direction of movement after the hand is stopped for a certain period of time into the direction of mouse movement and the speed of the hand into the speed of moving the mouse. To do.
At this time, the hand is swung in a desired direction, in other words, reciprocated. However, the mouse event conversion unit 103 ignores the movement of returning the hand to the vicinity of the original position, and sets the motion vector. Does not generate the corresponding mouse event. In other words, the operator temporarily stops the hand, moves the hand in the direction in which the mouse is desired to move, moves it in the direction to return the hand without stopping the hand, returns it to the vicinity of the stop position of the original hand, Further, by moving the hand in the direction in which the mouse is desired to move, the movement when returning the hand is ignored, and the pointer can be moved in a desired direction. The detection of the action of returning the hand is applied when the above-described motion vector changes by 90 degrees or more, which is a predetermined angle, within a predetermined time. Here, the motion vector is ignored, for example, for motion vectors after a point in time when a motion vector changed by 90 degrees or more is detected. The motion vector is ignored until, for example, a motion vector whose direction has changed 90 degrees or more within a predetermined time is detected again. In the case of ignoring the motion vector, a motion vector in a predetermined angle range (for example, an angle smaller than 90 degrees) is calculated with respect to the motion vector in the first direction after the hand has been stopped for a certain time. It may be done.

In general, the operator first moves the mouse quickly to the vicinity of the target position, and then moves the mouse slowly toward the target position. Also in this embodiment, with such movement in mind, when the movement of the hand is fast, the change (movement amount) of the pointer per unit movement amount of the motion vector is increased, and the movement of the hand is slow Improves the operability by reducing the change in the pointer position per unit movement amount of the motion vector. In other words, the mouse event conversion unit 103 increases the movement amount of the pointer per unit movement amount of the motion vector detected by the target part position detection unit 102 as the speed indicated by the motion vector increases. Generate a mouse event. For example, the mouse event conversion unit 103 increases the movement amount of the mouse per unit movement amount of the motion vector detected by the target part position detection unit 102 as the speed indicated by the motion vector increases. Generate a mouse event.
In order to avoid the movement of the pointer due to an extra hand movement, the mouse event conversion unit 103 determines that such a motion vector is used when the magnitude (speed) of the motion vector is equal to or less than a predetermined threshold. Ignore and do not generate mouse events based on such motion vectors.

Next, the GUI creation unit 105 creates a GUI according to the mouse event given from the mouse event conversion unit 103 (S14). Here, the GUI creation unit 105 creates a GUI in which a cursor (pointer) moves on a two-dimensional coordinate axis in response to a hand movement (gesture) imaged by the camera 101. Then, the GUI creation unit 105 gives the created GUI to the display unit 106.
The display unit 106 displays the GUI screen given from the GUI creation unit 105 (S15). Thereby, the movement of the cursor corresponding to the gesture is displayed on the GUI screen.

  In addition, after the cursor is moved to a position corresponding to an option on the GUI, the selection item is determined by left clicking with a normal mouse. However, in the present embodiment, the gesture detection is simplified by using the gesture limited to the movement of the mouse. Therefore, the input of the operation based on the shape of the hand, which has been conventionally proposed as a method of using the gesture, is not detected. For this reason, for example, when the cursor is moved over a desired option and then stops for a certain period of time, the GUI creation unit 105 regards that option as being clicked and generates the mouse event. Alternatively, the operator may input an operation such as a click through the input unit 104.

  In this embodiment, only the direction and speed of hand movement is detected as a gesture, converted into a mouse event and used as a mouse input of the GUI. Therefore, even with a conventional GUI, simple gesture detection is possible. An operation using a gesture can be performed.

  In addition, it is possible to realize a more natural and smooth movement of the mouse cursor by observing a change in the motion vector of the target part and eliminating a movement that does not lead to a mouse event, such as an action of returning the hand.

Embodiment 2. FIG.
FIG. 4 is a block diagram schematically showing the configuration of the control device 200 according to the second embodiment. The control device 200 includes a camera 101, a target part position detection unit 102, a mouse event conversion unit 203, an input unit 104, a GUI creation unit 205, a display unit 106, a microphone 207, a voice recognition unit 208, A keyboard event conversion unit 209. The control device 200 according to the second embodiment is further provided with a processing point in the mouse event conversion unit 203 and the GUI creation unit 205, and a microphone 207, a voice recognition unit 208, and a keyboard event conversion unit 209. This is different from the control device 100 according to the first embodiment.

The microphone 207 is a sound collection unit that acquires sound and generates a sound signal indicating the acquired sound. The microphone 207 gives the generated voice signal to the voice recognition unit 208.
The voice recognition unit 208 recognizes a language from the voice indicated by the voice signal given from the microphone 207, and generates character string data indicating a character string of the recognized language. Then, the voice recognition unit 208 gives the character string data indicating the predetermined character string to the mouse event conversion unit 203, and gives the other character string data to the keyboard event conversion unit 209. For example, the voice recognizing unit 208 gives a character string indicating a mouse operation such as “left click” and “right click” to the mouse event conversion unit 203, and converts a character string indicating a keyboard operation into a keyboard event conversion Part 209.
The keyboard event conversion unit 209 generates a keyboard event corresponding to the character string indicated by the character string data given from the voice recognition unit 208. The keyboard event conversion unit 209 gives the generated keyboard event to the GUI creation unit 205.

  The mouse event conversion unit 203 performs the same processing as the mouse event conversion unit 103 in Embodiment 1, and generates a mouse event corresponding to the character string indicated by the character string data given from the voice recognition unit 208. For example, when the mouse event conversion unit 203 acquires character string data indicating “right click” or “left click” from the voice recognition unit 208, the mouse event conversion unit 203 generates a mouse event of “right click” or “left click”. Then, the mouse event conversion unit 203 gives the generated mouse event to the GUI creation unit 205.

  The GUI creation unit 205 is at least one of a mouse event given from the mouse event conversion unit 203, a mouse event or key event given from the input unit 104, and a keyboard event given from the keyboard event conversion unit 209. In response to this, a GUI is created. Then, the GUI creation unit 205 gives the created GUI to the display unit 106.

  In the first embodiment, a method of using a gesture instead of using a mouse to move the pointer position on the GUI screen has been introduced. However, character input may be required on the GUI. In the second embodiment, by having the above-described configuration, when the operator moves the cursor to a desired position using a gesture, for example, “click” means that a left click is performed. As shown, the system shown in the first embodiment can be changed. In the second embodiment, it is possible to deal with variations of mouse events such as “right click”.

Embodiment 3 FIG.
FIG. 5 is a block diagram schematically showing the configuration of the control system 330 according to the third embodiment. The control system 330 includes a control device 300 and a control target device 340. For example, the control device 300 and the control target device 340 are configured to be able to communicate by connecting to a network (not shown).

  The control device 300 includes a camera 101, a target part position detection unit 102, a mouse event conversion unit 203, an input unit 104, a GUI processing unit 310, a display unit 106, a microphone 207, a voice recognition unit 208, A keyboard event conversion unit 209 and a communication unit 311 are provided. The control device 300 according to the third embodiment is different from the control device 200 according to the second embodiment in that a GUI processing unit 310 is provided instead of the GUI creation unit 205 and a communication unit 311 is further provided.

  The GUI processing unit 310 receives the GUI sent from the control target device 340 via the communication unit 311, gives the received GUI to 106, and causes the display unit 106 to display the screen of this GUI. In addition, the GUI processing unit 310 is at least one of a mouse event given from the mouse event conversion unit 203, a mouse event or key event given from the input unit 104, and a keyboard event given from the keyboard event conversion unit 209. One is transmitted to the control target device 340 via the communication unit 311. For example, if the control system 330 is compatible with VNC (Virtual Network Computing), the GUI processing unit 310 can be realized by a VNC client program. Further, if the control system 330 is compatible with X-Window, the GUI processing unit 310 can be realized by mounting an X server.

  The communication unit 311 is an interface for communicating with the control target device 340. For example, the communication unit 311 can be realized by a NIC (Network Interface Card).

  The control target device 340 includes a control unit 341, a GUI creation unit 342, and a communication unit 343.

The control unit 341 controls the entire control target device 340.
The GUI creation unit 342 creates a GUI in response to a mouse event or key event given from the control device 300. Then, the GUI creation unit 342 transmits the created GUI to the control device 300 via the communication unit 343. For example, if the control system 330 is compatible with VNC, the GUI creation unit 342 can be realized by a VNC server program. If the control system 330 is compatible with X-Window, the GUI creation unit 342 can be realized by installing an X client.

  The communication unit 343 is an interface for communicating with the control device 300. For example, the communication unit 343 can be realized by a NIC.

  The control device 300 according to the third embodiment displays a GUI screen of the control target device 340, which is another device connected to the network, on the display unit 106, and controls another device at a remote location. Can do. Specifically, the control device 300 is implemented by introducing VNC or X-Window that is widely used in PC applications.

  For example, in VNC, a PC serving as a server sends the image as bitmap data to a client device. On the client device side, the received bitmap data is expanded on the frame buffer and displayed on the display screen. On the other hand, a mouse and a keyboard are connected to the client device, and the client device can operate the PC while watching the screen by sending a mouse event and a keyboard event to the server using a protocol called RFB via the network.

  Therefore, by replacing the GUI creation unit 205 in the second embodiment with, for example, the GUI processing unit 310 realized by a VNC client program, the operation of the control target device 340 is also performed using the gesture and voice input interface. Can do.

  Moreover, when using X-Window, it is realizable with the same structure. For example, an X server may be mounted as the GUI processing unit 310, and the GUI creation unit 342 in the control target device 340 may be configured as an X client. Also in the X server and the X client, the keyboard and mouse input are acquired by the X server and transmitted to the client, as in the VNC. The X client can be used for controlling the device itself having the X server, or can be connected to an X client for controlling other devices connected to the network to control the target device.

  As described above, since the input interface based on the gesture and voice recognition introduced in the first and second embodiments is obtained as data converted into the mouse event and the keyboard event, it is connected to the network by using the VNC or the X-Window system. It can also be easily deployed to operate the user interface of another device.

  According to the first to third embodiments, information acquired by a gesture is limited to a movement amount and direction of a specific part such as a hand. And the change on the preset 2D coordinate axis is converted into the format of the event that occurs when the mouse is moved, and it is transmitted to the control object, so that complicated processing such as capturing the shape change of the hand It is possible to move the GUI cursor displayed on the screen based on the converted event without performing gesture detection accompanied with. As a result, selection of operations and the like are performed, and thus a GUI created on the premise that the mouse is used can be used as it is to make a gesture-compatible device.

  In addition, the voice recognition unit 208 can input characters, and the gesture recognition can be simplified by performing an operation such as clicking a mouse event in gesture recognition by voice recognition.

  Furthermore, by using the events generated by the gesture and voice recognition as keyboard events and mouse events such as VNC, it is possible to use the input interface including operations of other devices connected to the network.

  100, 200, 300 control device, 101 camera, 102 target part position detection unit, 103, 203 mouse event conversion unit, 104 input unit, 105, 205, 342 graphical user interface creation unit, 106 display unit, 207 microphone, 208 audio Recognition unit, 209 keyboard event conversion unit, 310 graphical user interface processing unit, 311, 343 communication unit, 340 control target device, 341 control unit.

Claims (13)

An imaging unit that images the operator and generates a video signal indicating the video of the operator;
A target part position detection unit that detects a movement of the specific part of the operator from the video indicated by the video signal generated by the imaging unit;
A mouse event conversion unit that generates a mouse event when the mouse moves according to the movement detected by the target part position detection unit;
And a display unit for displaying a graphical user interface screen in accordance with the mouse event generated by the mouse event conversion unit. The target part position detection unit detects the movement of the specific part of the operator as a motion vector on a predetermined two-dimensional coordinate,
The mouse event conversion unit generates the mouse event by converting the direction and speed indicated by the motion vector detected by the target part position detection unit into the direction and speed of movement of the mouse. The control device according to claim 1, wherein The mouse event conversion unit is a motion vector detected by the target part position detection unit when the specific part of the operator starts moving from a stationary state based on the motion vector detected by the target part position detection unit. The specified direction is the direction in which the mouse has moved, and the speed corresponding to the speed specified by the motion vector detected by the target site position detector after that time is the speed at which the mouse has moved. The control device according to claim 2, wherein: When the direction specified by the motion vector detected by the target part position detection unit changes by a predetermined angle or more within a predetermined time, the mouse event conversion unit The control device according to claim 2 or 3, wherein a mouse event is not generated based on a motion vector. The mouse event conversion unit, the faster the speed indicated by the motion vector detected by the target part position detection unit, the faster the pointer per unit movement amount indicated by the motion vector detected by the target part position detection unit The control device according to any one of claims 2 to 4, wherein the mouse event is generated so that the movement amount of the mouse increases. A sound collection unit for acquiring sound;
A speech recognition unit that recognizes a language from the speech acquired by the sound collection unit and generates character string data indicating a character string of the recognized language;
The mouse event conversion unit, when the character string indicated by the character string data generated by the voice recognition unit is a predetermined character string, the character indicated by the character string data generated by the voice recognition unit The control device according to any one of claims 1 to 5, wherein a mouse event corresponding to the column is generated. When the character string indicated by the character string data generated by the voice recognition unit is not a predetermined character string, the character string indicated by the character string data generated by the voice recognition unit is input with the keyboard A keyboard event conversion unit for generating a keyboard event of the case,
The control according to claim 6, wherein the display unit displays a screen of a graphical user interface in accordance with the mouse event generated by the mouse event conversion unit and the keyboard event generated by the keyboard event conversion unit. apparatus. A graphical user interface generating unit that generates a graphical user interface according to the mouse event generated by the mouse event converting unit;
The control device according to claim 1, wherein the display unit displays a screen of a graphical user interface generated by the graphical user interface generation unit. In accordance with the mouse event generated by the mouse event conversion unit and the keyboard event generated by the keyboard event conversion unit, further comprising a graphical user interface generation unit for generating a graphical user interface,
The control device according to claim 7, wherein the display unit displays a graphical user interface screen generated by the graphical user interface generation unit. A communication unit that communicates with the control target device;
A mouse event generated by the mouse event conversion unit is transmitted to the control target device via the communication unit, and a graphical user interface generated by the control target device according to the transmitted mouse event, A graphical user interface processing unit for receiving via the communication unit,
The control device according to claim 1, wherein the display unit displays a screen of a graphical user interface received by the graphical user interface processing unit. A communication unit that communicates with the control target device;
The mouse event generated by the mouse event conversion unit and the keyboard event generated by the keyboard event conversion unit are transmitted to the control target device via the communication unit, and the transmitted mouse event and keyboard event are transmitted. And a graphical user interface processing unit that receives a graphical user interface generated by the control target device via the communication unit,
The control device according to claim 7, wherein the display unit displays a screen of a graphical user interface received by the graphical user interface processing unit. The control device according to claim 10 or 11, wherein the graphical user interface processing unit corresponds to a VNC client or an X-Window X server. An imaging process of imaging the operator and generating a video signal indicating the operator's video;
From the video shown by the video signal generated in the imaging process, the target part position detection process for detecting the movement of the specific part of the operator,
A mouse event conversion process for generating a mouse event when the mouse moves according to the movement detected in the target part position detection process;
A display process for displaying a graphical user interface screen in accordance with the mouse event generated in the mouse event conversion process.

Images