JP2015122124A - Information apparatus with data input function by virtual mouse - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information apparatus having a function for inputting data related to a GUI operation such as a pointer moving operation and a click operation without a physical mouse by operating a nonexistent virtual mouse with feeling of operating a normal mouse and detecting motion of a hand and motion of fingertips.SOLUTION: A data input method comprises the steps of: analyzing data where positions of a hand and fingertips of an operator are detected; detecting movement of the hand operating a nonexistent virtual mouse, and a position and an operation of each fingertip; and outputting a movement command for moving a pointer on a display when detecting the movement of the hand in a state of detecting a specific shape by each fingertip and a click command when detecting depression operation by the fingertips, to an information processing part.

Description

  The present invention operates a virtual mouse that does not exist physically, detects the movement of the hand of the operator and the movement of the fingertip, and executes screen operations such as a movement operation and a click operation of the displayed pointer. The present invention also relates to an information device that realizes data input equivalent to normal mouse input.

  2. Description of the Related Art In recent years, with the advancement of semiconductor technology, a portable personal computer for mobile use (hereinafter abbreviated as a mobile personal computer) having a small size, light weight and high performance has been spread. Further, screen operations and data input means in application software installed in a mobile personal computer are generally operations based on a graphical user interface (GUI) such as click and drag, etc., using a mouse in addition to a keyboard.

  For this reason, for example, when using a mobile PC, a normal mouse is wired or wirelessly connected at home or at home, and a GUI is operated. It is customary to operate the GUI using a device. This is because it avoids carrying a normal mouse as a separate unit in addition to the mobile personal computer to the destination, and it seems that it is originally more comfortable to operate with a normal mouse.

  As a method for improving this, there is a technique in which, for example, a hand movement direction is analyzed from an image captured by a camera, and command input processing is executed based on the analysis information (for example, Patent Document 1). This technology makes it possible to identify four types, for example, from bottom to top, from left to right, from top to bottom, and from right to left as directions in which a hand or finger as a recognition target object crosses the recognition area (hotspot). When three regions are set as regions, 64 (= 4 × 4 × 4) ways of identification are possible, and these operations are associated with 64 GUI operation commands. Thus, the user can perform a predetermined command to operate the screen without using a physical mouse or the like by performing a predetermined gesture operation on the camera.

  However, although the technique described in Patent Document 1 seems to be suitable for remote control of the volume and channel of a TV being viewed without a remote control device, for example, the hand or finger is recognized. This is a gesture input that requires a large amount of movement to identify the direction across the area, which is very different from the general sense of operating a mouse. Further, the operator needs to learn many predetermined gesture operations, which is insufficient as a technique for replacing a physical mouse.

JP 2007-34525 A

  Therefore, an object of the present invention is to operate a virtual mouse that does not physically exist as if it were a normal mouse operation, detect the movement of its hand and fingertip, and move or click the displayed pointer. An object of the present invention is to provide an information device such as a mobile personal computer that executes a screen operation such as an operation and realizes a data input equivalent to a normal mouse input.

  In order to solve the above-mentioned problem, the present invention analyzes video data from a video camera included in its own device that captures an operator's hand and fingertips, determines operation contents for an invisible virtual mouse, and displays it. An information device for inputting data relating to a pointer movement and a click operation, wherein the video data is analyzed to detect movement of an operator's hand operating the virtual mouse, and the video data is analyzed And fingertip detection means for detecting the position and movement of one or more fingertips that operate the virtual mouse, and the fingertip detection means detects a predetermined specific shape by the one or more fingertips. When the movement detecting means detects the movement of the hand in a state, the pointer being displayed is moved according to the movement of the hand, and the fingertip detecting means is predetermined by the one or more fingertips. When the reciprocating movement of one or more fingertips is detected in a state where the specific shape is detected, the screen being displayed is scrolled, and the fingertip detecting means is predetermined by the one or more fingertips. When a pressing operation by one of the one or more fingertips is detected in a state where the specified shape is detected, it is determined that a click operation has been performed, and a predetermined process corresponding to the click operation is executed It is characterized by.

  According to the present invention, since a virtual mouse that does not physically exist is operated as if a normal mouse was operated, it is not necessary to memorize a special gesture command. Therefore, there is an effect that it is possible to provide an information device such as a mobile personal computer having a comfortable operability equivalent to that of a normal mouse even in a situation where a normal mouse is not carried at the destination.

Overall explanatory diagram of the present invention Block diagram of the device of the present invention Operation flowchart of the device of the present invention

  Hereinafter, as an embodiment of the present invention, a case where the present invention is applied to a mouse input unit of an information device or a communication device will be described as an example with reference to the drawings.

  FIG. 1 is an overall explanatory view of the present invention, wherein 1 is an information device to which the present invention is applied (hereinafter abbreviated as this device), 2 is a video camera incorporated in the device 1, and 30 is the device 1. , A pointer displayed on the display unit 30, 32 and 33 a click area displayed on the display unit 30, 40 an operator's hand operating the virtual mouse, and 41 a hand The fingertip of 40 forefinger, 42 the fingertip of the middle finger of hand 40, 50 a virtual mouse, 51 a left click switch of the virtual mouse, and 52 a right click switch of the virtual mouse. Although not shown, there is a scroll key between the left click switch 51 and the right click switch 52 of the virtual mouse 50 as in a normal mouse.

  50 to 52 indicated by broken lines are not physically present. While looking at the pointer 31 displayed on the display unit 30, the operator imagines that the virtual mouse 50 that is not physically present is being grasped with the hand 40, moves the virtual mouse 50 on the desk with the hand 40, and moves the fingertip 41 and 42 are used to perform a click operation and a screen scroll operation. The hand 40 and fingertips 41 and 42 of the operator who operates the mouse are photographed by the video camera 2, and the video data from the video camera 2 is processed and analyzed to analyze the movement of the hand 40 and the fingertip. Left-click or right-click operations by 41 and 42 are detected. The pointer 31 moves according to the detection results.

  FIG. 1 shows an example in which the pointer 31 displayed on the display unit 30 is moved to the click area 32 of YES and clicked. Note that the position where the pointer 31 is first displayed is appropriately set by processing of application software such as the center of the display unit 30, the top and bottom, left and right edges, and the center of a predetermined click area.

  FIG. 2 is a block diagram of the apparatus 1, wherein 11 is a fingertip detection unit, 12 is a specific shape detection unit, 13 is a movement detection unit, 14 is a mouse setting unit, 15 is a click determination unit, and 16 is a mouse interface. , 17 are information processing units.

  The fingertip detection unit 11 analyzes the video data of each fingertip that operates the virtual mouse 50 taken by the video camera 2, detects the coordinates where the hand or each fingertip is located, and the specific shape detection unit 12, movement detection The detected data is notified to the unit 13 and the click determination unit 15.

  The specific shape detection unit 12 analyzes the data from the fingertip detection unit 11 to determine the shape of each finger, and when detecting a specific shape registered in advance (for example, Janken's face) Notify the mouse interface 16.

  The movement detection unit 13 analyzes the data from the fingertip detection unit 11, detects the movement of the hand 40 that operates the virtual mouse 50, and notifies the mouse interface 16 of the movement direction and movement amount.

  The mouse setting unit 14 operates the virtual mouse 50 such as the function of the left click switch 51 and the right click switch 52 of the virtual mouse 50 and the movement ratio of the pointer 31 with respect to the movement amount of the hand 40 (that is, the movement amount of the virtual mouse 50). It is means for setting such data. The data to be set in the mouse setting unit 14 may be input by an input device other than the virtual mouse 50 such as a keyboard (not shown), or a predetermined shape by each fingertip via the specific shape detection unit 12. It is also possible to input predetermined data corresponding to the above.

  The click determination unit 15 analyzes the data from the fingertip detection unit 11 to monitor the pressing operation of the fingertip 41 or the fingertip 42. When the pressing operation is detected, the left click switch 51 or the right click switch 52 of the virtual mouse 50 is detected. It is determined which of the buttons is clicked, and the determination result is notified to the mouse interface 16. For example, when the pressing operation by the fingertip 41 is detected, it is determined that the left click switch 51 is clicked. When the pressing operation by the fingertip 42 is detected, it is determined that the right click switch 52 is clicked.

  The mouse interface 16 outputs the movement data of the pointer 31, the click command, and the scroll command to the information processing unit 17 based on the data input from the specific shape detection unit 12, the movement detection unit 13, and the click determination unit 15.

  For example, in the shape of Janken, the virtual mouse 50 is gripped by the hand 40 and the fingertip 41 of the right index finger and the fingertip 42 of the right hand ring finger are set at the positions of the left click switch 51 and the right click switch 52 of the virtual mouse 50. This movement is defined as movement or dragging of the virtual mouse 50 to which the pointer 31 should be moved. In this case, movement in a shape other than choki (for example, par or goo) moves the virtual mouse 50 without moving the pointer 31 (corresponding to an operation of lifting the mouse back to the original position in normal mouse operation). It becomes. As a result, the operator only has to move each fingertip with the feeling of operating a normal mouse, so the feeling of strangeness is small.

  The information processing unit 17 is information processing means using application software in general that involves input of movement data, a click command, and a scroll command using a mouse.

  The movement data, click command, and scroll command input to the information processing unit 17 are the same as data from a normal physical mouse, and the mouse corresponding unit (not shown) of the information processing unit 17 is a mouse interface. The pointer 31 being displayed is moved according to the movement data and the click command input from 16, and the shape of the pointer 31 being displayed is transformed into a shape corresponding to a click operation or a drag operation. Further, the displayed image is scrolled according to the scroll command input from the mouse interface 16.

  FIG. 3 is an operation flowchart of the apparatus 1. Hereinafter, the operation flow of the present apparatus 1 will be described with reference to FIGS. This flow starts when the power of the apparatus 1 is turned on and the information processing unit 17 waits for data input from the mouse (S300). In this state, the pointer 31 is displayed at a home position or a specific position designated by the application software running the information processing unit 17.

  When the hand 40 of the operator who is operating the apparatus 1 and video data of the fingertip (captured by the video camera 2) are input to the fingertip detection unit 11 (S310), the fingertip detection unit 11 analyzes the video data. The position of the hand 40 and each fingertip is detected (S311).

  The specific shape detection unit 12 monitors whether or not a predetermined specific shape (for example, choki) is detected from the coordinate data at which each fingertip is detected detected by the fingertip detection unit 11, and detects the specific shape (S320, YES). ), It is determined that the operator is holding the virtual mouse 50, and the process proceeds to S330. If the specific shape is not detected (S320, NO), the process returns to S310.

  In S330, the movement detection unit 13 monitors the movement of the hand 40 from the history of coordinate data where the hand 40 is notified, which is notified from the fingertip detection unit 11, and detects the movement of the hand 40 (S330, YES). The movement data including the movement direction and the movement amount of 40 is output to the information processing unit 17 via the mouse interface 16 (S331), and the process proceeds to S340. Note that the value of the movement amount included in the movement data is a value obtained by multiplying the movement amount of the hand 40 by a parameter relating to movement set in the mouse setting unit 14. The information processing unit 17 relatively moves the displayed pointer 31 according to the input movement data. If NO in S330, the process skips S331 and proceeds to S340.

  In S340, the click determination unit 15 monitors the pressing operation of each fingertip from the history of the coordinate data where the fingertip of the index finger 41 and the middle finger 42 are notified from the fingertip detecting unit 11, and detects the pressing operation (YES in S340). ) A click command corresponding to right click or left click is output to the information processing unit 17 via the mouse interface 16 (S341), and the process proceeds to S350.

  Note that the click operation performed by the operator includes a single click operation, a double click operation, and a drag operation with the click switch held down, but these are identified by the information processing unit 17 (not shown). Further, the right click and the left click may be identified based on whether the fingertip that has been pressed is the fingertip 41 or the fingertip 42. If NO in S340, the process skips S341 and proceeds to S350.

  In S350, when the mouse interface 16 detects a reciprocating motion in the vertical or front-rear direction corresponding to the operation of the scroll key (not shown) of the virtual mouse 50 by the fingertip 41 or the fingertip 42 (S350, YES), the mouse interface 16 Referring to the mouse setting unit 14, the command related to the screen scroll is output to the information processing unit 17 (S351), and the process returns to S310.

  Here, the scroll direction of the screen may be defined as, for example, upward scrolling of the screen when the reciprocating motion of the fingertip 41 is detected, and downward scrolling of the screen when the reciprocating motion of the fingertip 42 is detected. If NO in S350, S351 is skipped, the process returns to S310, and S310 and subsequent steps are repeated.

  As described above, the video data from the video camera that shot the hand and fingertip of the operator who operates the virtual mouse that does not exist is analyzed by image processing technology, and the same movement data, click command, The method for outputting the scroll command to the information processing unit 17 has been described.

  In this method, as a specific method for detecting the position of each fingertip of the operator, for example, by using image recognition processing technology, a nail part may be extracted and the center position determined as the fingertip position. . Moreover, what is necessary is just to let the position of a hand be the position of the center of each detected said fingertip, or a gravity center, for example.

  Furthermore, as a method for determining the position of the fingertip by a simple process, an object having a feature point such as nail art is attached to each nail of each fingertip, a picture or mark having a feature point is drawn, or there is a feature point A glove or finger sack may be worn, and the feature points may be detected by image processing technology.

  By the way, in the embodiment of the present invention, as a method for detecting the position and movement of each fingertip, a method for analyzing and detecting video data from the video camera 2 has been described as an example. It is not limited to this. Although a detailed description is omitted, for example, instead of the video camera 2, a method of detecting the operation content on the virtual mouse 50 by using data from a sensor that directly detects the position of each fingertip can be applied. For example, a magnetic material or a wireless tag may be attached to the nail or glove of each fingertip, and their position and operation may be detected by a predetermined sensor (for example, a hall element or a wireless tag reader).

  It is also possible to freely define a specific shape identified by the specific shape detector 12 (not shown). For example, in addition to goo, choki, and par with each fingertip of the hand 40, the home position of the pointer 31 can be achieved by various shapes or operations such as an OK sign with the thumb and forefinger, and a shape in which a predetermined number of fingers are extended from the shape of goo. It is possible to define various screen operation commands such as moving to, displaying / stopping the pointer 31, and enlarging / reducing the image displayed on the display unit 30.

DESCRIPTION OF SYMBOLS 1 ... Apparatus which applied this invention 2 ... Video camera 11 ... Fingertip detection part 12 ... Specific shape detection part 13 ... Movement detection part 14 ... Mouse setting part 15 ... Click Judgment part 16 ... Mouse interface 17 ... Information processing part 30 ... Display part 31 ... Pointer 32 ... Click area (YES)
33 ... Click area (NO)
40 ... Hand for operating virtual mouse 41 ... Fingertip 42 for index finger operating virtual mouse ... Fingertip 50 for middle finger operating virtual mouse ... Virtual mouse 51 ... Left click switch of virtual mouse 52 ... Virtual mouse right click switch

Claims (1)

Analyzes the video data from the video camera provided in the own device that captures the operator's hand and fingertips, determines the operation contents for the virtual mouse that does not exist, and inputs data related to the movement of the pointer and the click operation An information device,
Movement detection means for detecting movement of an operator's hand operating the virtual mouse by analyzing the video data, and detecting positions and movements of one or more fingertips operating the virtual mouse by analyzing the video data Fingertip detecting means for performing,
When the movement detection unit detects the movement of the hand in a state where the fingertip detection unit detects a predetermined specific shape by the one or more fingertips, a pointer being displayed is displayed according to the movement of the hand. A screen that is being displayed when the fingertip detecting unit detects any reciprocating movement of the one or more fingertips while the fingertip detecting unit is detecting a predetermined specific shape by the one or more fingertips. When a pressing operation by any one of the one or more fingertips is detected in a state where the fingertip detecting means detects a predetermined specific shape by the one or more fingertips, a click operation is performed. An information device having a data input function using a virtual mouse, wherein the information processing unit determines that the click operation is performed, and executes a predetermined process corresponding to the click operation.

Images