JP7179334B2 - GESTURE RECOGNITION DEVICE AND PROGRAM FOR GESTURE RECOGNITION DEVICE - Google Patents

Description

The present invention relates to a gesture recognition device and a program for a gesture recognition device.

In recent years, gesture recognition devices have been developed that recognize gestures made by a user's fingers or the like, and thereby accept various operations. As such a gesture recognition device, for example, Patent Document 1 discloses a device that captures the movement of a user's fingers or the like with a camera and recognizes gestures from the captured image, thereby enabling on-vehicle equipment to be operated without touching it. is disclosed.

JP 2015-184841 A

However, in the gesture recognition device that captures the movement of the user's fingers with a camera or the like, as shown in Prior Document 1, the imaging range of the camera or the like cannot be seen. It is difficult to grasp the initial position suitable for Therefore, by moving fingers in inappropriate positions, gestures may not be recognized, or gestures not intended by the user may be erroneously recognized.

The present invention has been made in view of such problems, and recognizes a user's gesture based on a captured image. The primary object of the present invention is to provide a gesture recognition device capable of reducing erroneous recognition.

As a result of intensive studies to solve the above problem, the inventors of the present invention have found that when the hand making a gesture is too close to the camera, the finger protrudes from the angle of view (imaging range) of the camera. is not recognized, but if the hand is too far from the camera, the pixel resolution of the camera is insufficient, and the shape of the user's hand is not recognized. Therefore, by notifying the user when the user's finger does not protrude from the imaging range and is at a position where the pixel resolution of the imaging means is sufficient, the user can grasp the position suitable for starting the gesture, The inventors have found that erroneous recognition of gestures can be reduced, leading to the present invention.

That is, a gesture recognition apparatus according to the present invention includes imaging means for imaging a user's hand, and recognizes a gesture of the user's hand based on the captured image captured by the imaging means. an initial position determination unit that recognizes the shape of the user's hand in the image and determines whether the user's hand is in an initial position suitable for starting a gesture based on the result; and a notifying unit that notifies the user that a gesture may be started when it is determined that there is a gesture, and when the initial position determining unit recognizes a hand shape with at least one finger raised, the It is characterized by determining that the user's hand is in the initial position.

In this case, the position where the hand shape with at least one finger is recognized is set as the initial position, and when the user's hand is in such an initial position, it is notified that the gesture can be started. As a result, the user can easily grasp the appropriate position (appropriate sense of distance from the imaging means) to start the gesture, and misrecognition of the gesture can be reduced. Here, by setting "the position where the shape of a hand with at least one finger raised" is the initial position, the user can move to a position that is neither too close nor too far from the imaging means, i.e., within the angle of view of the imaging means. Since the gesture can be started at a position where the finger fits and the pixel resolution of the imaging means is sufficient for recognizing the finger, erroneous recognition of the gesture can be reduced.

The gesture recognition device preferably determines that the user's hand is at the initial position when the initial position determination unit recognizes a hand shape in which a plurality of fingers are held apart from each other. It is more preferable to determine that the user's hand is in the initial position when a hand shape in which the fingers of the user are held apart from each other is recognized.
If the distance from the imaging means is the same, misrecognition can be reduced as the gesture is performed at a position closer to the center of the imaging range of the imaging means. Therefore, by setting a position where a hand shape can be recognized with a plurality of fingers spaced apart from each other as an initial position, and notifying that the hand is at this position, a position closer to the center of the imaging range of the imaging means can be obtained. Gestures can be started, and misrecognition of gestures can be further reduced. By setting a position where a hand shape with five fingers held apart from each other can be recognized as the initial position, a gesture can be performed at a position where erroneous recognition of the gesture can be further reduced.
Further, in this way, even if the user creates a hand shape with five fingers spaced apart from each other, if the hand is tilted rather than directly facing the imaging means, the captured image will show Since the fingers overlap and five fingers are not recognized, it is not determined that the user's hand is in the initial position. The hand shape of the user with five fingers spaced apart from each other is placed facing the imaging means, and it is determined that the user's hand is in the initial position only when the five fingers are recognized in the captured image. Gestures can be made at positions where erroneous recognition of gestures can be further reduced.

Preferably, in the gesture recognition device, the initial position determination unit determines that the user's hand is at the initial position when the area of the recognized hand shape is within a predetermined range.
In this way, the user can grasp a more appropriate sense of distance from the imaging means and can start the gesture at a more appropriate position, thereby further reducing erroneous recognition of the gesture.

It is preferable that the gesture recognition device determines that the hand of the user is in the initial position when continuously recognizing a hand shape with at least one finger raised for a predetermined time or longer.
In this way, the gesture can be started in a state in which the user's hand is stable at the initial position, thereby further reducing erroneous recognition of the gesture.

The gesture recognition device includes a gesture recognition unit that recognizes a gesture of the user's hand based on the captured image, and the gesture recognition unit recognizes the gesture after it is determined that the user's hand is at the initial position. It is preferable to
In this way, the gesture is not recognized when the user's hand is not in the initial position, so misrecognition due to the gesture being performed in an inappropriate position can be further reduced.

In the gesture recognition device, the notification unit may notify the user of guidance information for guiding the user's hand to the initial position when it is determined that the user's hand is not at the initial position. preferable.
With such a configuration, the user can more easily grasp the initial position suitable for the gesture operation.

Further, a program for a gesture recognition device of the present invention is a program for a gesture recognition device that includes imaging means for imaging a user's hand, and recognizes a gesture of the user's hand based on an imaged image taken by the imaging means. and recognizing the shape of the user's hand reflected in the captured image, and when the recognized hand shape is a hand shape with at least one finger raised, the user's hand is suitable for starting a gesture. a function as an initial position determination unit that determines that the user's hand is in the initial position, and a notification unit that notifies the user that a gesture may be started when the user's hand is determined to be in the initial position. It is characterized by causing a computer to exhibit functions.

A program for such a gesture recognition device can achieve the same effect as the gesture recognition device described above.

According to the present invention, the user's gesture is recognized based on the captured image, the user can grasp the initial position suitable for starting the gesture, and the erroneous recognition of the gesture can be reduced. A recognition device can be provided.

1 is a perspective view schematically showing the configuration of a gesture recognition device according to one embodiment of the present invention; FIG. The figure which shows the equipment structure of the gesture recognition apparatus of the same embodiment. FIG. 2 is a functional block diagram of the gesture recognition device of the same embodiment; FIG. 4 is an operation flow diagram of the gesture recognition device of the same embodiment; FIG. 4 is a diagram for explaining the operation of the gesture recognition device of the same embodiment; FIG. 4 is a diagram for explaining the operation of the gesture recognition device of the same embodiment; FIG. 4 is a diagram for explaining the operation of the gesture recognition device of the same embodiment;

A gesture recognition device 100 according to an embodiment of the present invention will be described below with reference to the drawings.

A gesture recognition device 100 according to the present embodiment is for assisting computer operations using gestures. Specifically, it replaces input operations performed by buttons and voice with user's gestures, recognizes the user's hand gestures, and performs predetermined outputs according to the gestures. More specifically, as shown in FIG. 1, the gesture recognition device 100 includes imaging means 1 for imaging a user's hand, and a computer integrally having a computer body 2 and a display 3 . An operation target such as an icon or a map is displayed on the display 3, and the user can operate the operation target displayed on the display 3 in a non-contact manner by performing a predetermined gesture within the imaging area of the imaging means 1. It has become. Note that the gesture recognition device 100 of the present embodiment does not include a distance sensor or the like for measuring the distance between the imaging means 1 and the user's hand.

The imaging means 1 captures images of a user's hand (including fingers) at regular time intervals, and sequentially outputs image data representing the captured images to the computer main body 2 . This imaging means 1 is specifically equipped with an infrared camera. Here, the number of effective pixels is configured to output image data of 160×120 at a frame rate of 30 fps, but the configuration is not limited to this. The imaging means 1 is of a so-called attachment type, and is attached to the lower portion of the display 3 so that the infrared camera faces the user. Note that the imaging means 1 is not limited to an infrared camera, and may be a full-color image sensor or a sensor having an array of heat detection elements. Moreover, it is not limited to the attachment type, and may be integrated with the display 3 . The imaging means 1 may be attached to the upper portion of the display 3, or may be installed at a location away from the display 3 at hand of the user.

Structurally, the computer main body 2 includes a CPU 201, a memory 202, an input/output interface 203, etc., as shown in FIG. Based on various application software (hereinafter referred to as a program) stored in the memory 202, the computer main body 2 can generate image data as shown in FIG. It is configured to implement functions as a reception unit 21, a gesture recognition unit 22, a command output unit 23, a display control unit, and the like. Each part will be described below.

The image data reception unit 21 receives image data output by the output unit of the imaging unit 1 and transmits the image data to the gesture recognition unit 22 .

The gesture recognition unit 22 recognizes gestures made by the user based on the captured image indicated by the received image data. Specifically, the gesture recognition unit 22 processes the received image data, for example, the outline of the user's hand, the position of the fingertip, the position of the crotch of the finger, the position of the finger joint, the angle of the finger joint, and the hand in the image. The position of the center of gravity, the area of the hand, etc. are recognized, and based on these, the gesture made by the user is recognized. Then, gesture data indicating the recognized gesture is transmitted to the command output unit 23 .

Gestures that can be recognized by the gesture recognition unit 22 include, for example, up/down/left/right flicking and swiping actions, clockwise/counterclockwise circle actions, push/pull actions, pinch-in/pinch-out actions, and finger gestures. It is a gesture by the movement of the hands (including fingers) such as posing actions such as standing up and making an OK sign.

The command output unit 23 outputs a command corresponding to the gesture indicated by the received gesture data to a control unit (not shown) as command data. A command indicated by the command data is a command for operating an operation target displayed on the display 3 . For example, it is an arbitrary operation that can be set by the user, such as selecting an icon displayed on the display 3, switching screens, enlarging/reducing/scrolling an image or map.

The display 3 displays an image indicated by display data output from the computer body 2 on its screen. This display 3 is not limited to being integrated with the computer main body 2 , and may be separated from the computer main body 2 .

Thus, in the gesture recognition device 100 of the present embodiment, the computer main body 2 is further provided with the functions of the initial position determination section 24 and the notification section 25 .

The initial position determination unit 24 recognizes the shape of the user's hand appearing in the captured image captured by the imaging unit 1, and based on the result, determines whether the user's hand is suitable for starting a gesture (that is, when the gesture is performed). It is determined whether it is in the initial position (the possibility of misrecognition is low).

Specifically, the initial position determining section 24 receives image data from the image data receiving section 21 and processes it to recognize the user's hand shape. More specifically, from the image indicated by the received image data, for example, the outline of the user's hand, the position of the fingertips, the position of the crotch of the fingers, the center of gravity of the hand and the area of the hand in the image, etc. are recognized. Based on this, the user's hand shape is recognized. Then, when the recognized hand shape satisfies a predetermined condition stored in the memory 202 in advance, it is determined that the user's hand is in the initial position, and determination result data indicating that is sent to the notification unit 25 and the gesture recognition unit 22 . Output.

The initial position determination unit 24 determines that the user's hand is in the initial position when the recognized hand shape is a hand shape in which a plurality of (specifically, five) fingers are separated from each other. More specifically, the fingertips of a plurality of fingers (specifically, five fingers) and the four crotches (that is, roots) between the plurality of fingers are reflected in the captured image (that is, the image capturing means 1). If the hand shape is within the angle of view), it is determined that the user's hand is in the initial position.

The initial position determination unit 24 of the present embodiment recognizes the above-described hand shape, and if the area (the area surrounded by the contour of the recognized hand in the captured image) is within a predetermined range, the user's hand is It is determined that it is in the initial position. Here, it is determined that the user's hand is in the initial position when the area of the recognized hand shape is in the range of 200 pixels or more and 8000 pixels or less. It is preferable to determine that the user's hand is in the initial position when the area (number of pixels) of the hand shape with respect to the number of effective pixels of the sensor of the imaging means 1 used is 40% or less, and is 35% or less. is more preferable, and 30% or less is more preferable. By doing so, the user can start the gesture at the initial position with less erroneous recognition. That is, in this way, even if the hand slightly approaches the imaging means 1 side after the gesture is started, the hand is less likely to protrude from the angle of view of the imaging means 1, so that erroneous recognition of the gesture can be further reduced.

Furthermore, in the present embodiment, the initial position determination unit 24 determines that the user's hand is positioned at the initial position when the recognized hand shape continuously satisfies the above conditions for a predetermined period of time or more (specifically, 0.1 seconds or more). I am trying to determine that it is in

On the other hand, if the hand shape recognized based on the image data does not satisfy a predetermined condition stored in advance in the memory 202, the initial position determination unit 24 determines that the user's hand is not in the initial position, and notifies the user of the fact. The determination result data shown is output to the notification unit 25 . Further, in this case, the initial position determination unit 24 generates guidance information for guiding the user's hand to the initial position based on the recognized hand shape, and outputs this as guidance data to the notification unit 25 .

When the initial position determination unit 24 determines that the user's hand is at the initial position, the notification unit 25 notifies the user that the gesture may be started at that position. That is, it notifies that the user is urged to make a gesture at the initial position.

Specifically, when receiving determination result data indicating that the user's hand is in the initial position from the initial position determination unit 24, the notification unit 25 transmits notification data indicating that the gesture may be started to the notification unit 4. do. Then, the informing means 4 outputs that the gesture may be started. Here, the notification unit 25 uses the user's sense of hearing to notify that the gesture can be started, and a speaker (not shown) provided in the computer body 2 is used as the notification means 4. . More specifically, the speaker, which is the notification means 4, emits a sound indicating that the gesture may be started when the notification data is received. The ``sound indicating that the gesture can be started'' is, for example, ``Please start the gesture'', ``Gesture input is possible'', ``Please operate at that position'', etc. It may be a prompting voice, a beep sound, or the like.

On the other hand, when the notification unit 25 receives the determination result data indicating that the user's hand is not in the initial position and the guidance data, the notification unit 25 transmits notification data indicating the guidance information to the notification unit 4 . Then, guidance information is output from the notification means 4 (speaker in this case). The guidance information to be output may be, for example, information related to hand movements such as "Please move your hand a little more to the right" or "Please do not move your hand", or "Please open your hand firmly". or "Please point your hand straight at the camera."

Further, in this embodiment, the gesture recognition unit 22 recognizes a gesture made by the user after receiving determination result data indicating that the user's hand is in the initial position. Specifically, when receiving the determination result data, the gesture recognition unit 22 processes the received image data using this as a trigger, and recognizes the gesture made by the user. Functionally, the gesture recognition unit 22 does not recognize a gesture before the initial position determination unit 24 determines that the user's hand is in the initial position, and recognizes the gesture after the determination is made. It has become.

Next, the operation of the gesture recognition device 100 having such a configuration will be described below with reference to FIGS. 4 to 7. FIG. Here, the display 3 displays an operation target such as a map and a message such as "Please open your hand toward the camera."

First, the user places his or her hand over a position considered to be within the imaging range of the camera (imaging means 1), and then opens the hand toward the camera. Then, the initial position determination unit 24 recognizes the user's hand shape based on the image data output from the camera (step S1).

The initial position determination unit 24 determines whether the user's hand is in the initial position (step S2). The initial position determining unit 24 determines that the recognized hand shape is a hand shape in which a plurality of (specifically, five) fingers are held apart from each other, and the fingertips of the plurality of (five) fingers are If the hand shape includes four crotches (that is, roots) between a plurality of fingers, it is determined that the user's hand is in the initial position (proceed to step S4). On the other hand, if the recognized hand shape does not satisfy the condition, it is determined that the user's hand is not in the initial position, and guidance information for guiding the user's hand to the initial position is output (step S3). , the recognition of the user's hand shape is continued (return to step S1).

For example, when the user's hand is at position P (a position that is an appropriate distance from the camera and is directly in front of the camera) shown in FIG. Image data as shown in b) is transmitted. The initial position determination unit 24 determines from the image data that the hand shape is a hand shape in which five fingers are set apart from each other, and that the hand shape includes the fingertips of the five fingers and four crotches between them. and determines that the user's hand is in the initial position.

On the other hand, when the user's hand is at position Q shown in FIG. Image data as shown in (b) of is transmitted. Also, when the user's hand is in the R position (a position where the distance from the camera is appropriate, but is shifted downward when viewed from the camera), the initial position determination unit 24 displays a position as shown in FIG. 6(c). Image data is sent. In these cases, the initial position determining unit 24 determines from the image data that the hand has a hand shape in which five fingers are spaced apart from each other and includes the fingertips of the five fingers and the four crotches therebetween. The shape cannot be recognized, and it is determined that the user's hand is not in the initial position.

When the user's hand is at position S (a position too close to the camera) shown in FIG. sent. In this case, the initial position determination unit 24 determines from the image data that the hand shape is a hand shape in which five fingers are spaced apart from each other and includes the fingertips of the five fingers and the four crotches therebetween. It is determined that the user's hand is not in the initial position. When the user's hand is at position T (a position too far from the camera) shown in FIG. sent. In this case, the initial position determination unit 24 cannot detect the fingertips and finger cuffs from the image data because the hand shown is too small. It is determined that the user's hand is not in the initial position because the hand shape including the fingertips and the four crotches therebetween cannot be recognized.

When it is determined that the user's hand is in the initial position, the notification unit 25 notifies the user that the gesture may be started (step S4). Specifically, a voice such as "Please start the gesture at that position" is emitted from the notification means 4 (speaker) to urge the user to start the gesture.

Then, when the user makes a gesture at the P position (initial position), the gesture recognition unit 22 recognizes the gesture (step S5). Then, the command output unit 23 outputs a command corresponding to the gesture, and the operation target displayed on the display 3 is operated (step S6).

According to the gesture recognition device 100 of the present embodiment configured as described above, the initial position is a position where a hand shape with five fingers held apart from each other is recognized, and the user's hand is positioned at such an initial position. Since it is notified that the gesture may be started in a certain case, the user can easily grasp the position suitable for starting the gesture, and erroneous recognition of the gesture can be reduced. Here, by setting "a position where a hand shape with five fingers held apart from each other is recognized" as the initial position, the user can move to a position that is neither too close nor too far from the imaging means 1, i.e. A gesture can be started at a position close to the center of the angle of view of the imaging means 1, where the fingers fit within the angle of view and where the pixel resolution of the imaging means 1 is sufficient for recognizing the fingers. erroneous recognition can be prevented with high accuracy.

It should be noted that the present invention is not limited to the above embodiments.

In the above embodiment, when the recognized hand shape is a hand shape in which a plurality of (specifically, five) fingers are separated from each other, the initial position determination unit 24 determines that the user's hand is in the initial position. However, it is not limited to this. The initial position determination unit 24 may determine that the user's hand is in the initial position when the recognized hand shape is a hand shape with one finger up.

In the above embodiment, the initial position determination unit 24 determines that the user's hand is in the initial position when the recognized hand shape continuously satisfies a predetermined condition for a predetermined time or longer. Not exclusively. In another embodiment, the initial position determination unit 24 may immediately determine that the user's hand is in the initial position when the recognized hand shape satisfies a predetermined condition (that is, the hand shape).

In the above embodiment, the computer main body 2 exhibits all the functions of the image data reception unit 21, the gesture recognition unit 22, the command output unit 23, the initial position determination unit 24, and the notification unit 25, but this is not the only option. not. Some or all of these functions may be exhibited by the imaging means 1 .

In the above-described embodiment, the notification unit 25 utilizes the user's sense of hearing. For example, the display 3 may be used as the notification means 4 to display information indicating the start of the gesture and hand guidance information on the screen.

Although the imaging means 1 of the above embodiment is configured to output image data with an effective pixel count of 160×120 at a frame rate of 30 fps, the present invention is not limited to this. For example, it may be configured to output 16×16 image data at a frame rate of 10 fps. In the case of using the imaging means 1 having a low number of effective pixels in this way, the effects of the above-described gesture recognition device 100 are remarkably exhibited.

In addition, the present invention is not limited to the above-described embodiments, and it goes without saying that various modifications are possible without departing from the spirit of the present invention.

DESCRIPTION OF SYMBOLS 100... Gesture recognition apparatus 1... Imaging means 2... Computer main body 21... Image data reception part 22... Gesture recognition part 23... Command output part 24... Initial position determination part 25 ... Notification unit 3 ... Display 4 ... Notification means

Claims (8)

A gesture recognition device comprising imaging means for capturing an image of a user's hand, and recognizing a gesture of the user's hand based on an image captured by the imaging means,
an initial position determination unit that recognizes the user's hand shape reflected in the captured image and determines whether the user's hand is in an initial position suitable for starting a gesture based on the result;
a notification unit that notifies the user that a gesture may be started when it is determined that the user's hand is in the initial position;
The gesture recognition device determines that the hand of the user is in the initial position when the initial position determination unit recognizes a hand shape with at least one finger up. 2. The gesture recognition device according to claim 1, wherein the initial position determination unit determines that the hand of the user is in the initial position when a hand shape in which a plurality of fingers are held apart from each other is recognized. 3. The gesture recognition device according to claim 2, wherein the initial position determination unit determines that the user's hand is in the initial position when the initial position determination unit recognizes a hand shape in which five fingers are spaced apart from each other. 4. The initial position determining unit determines that the hand of the user is at the initial position when the area of the recognized hand shape is within a predetermined range. gesture recognizer. 5. The initial position determination unit determines that the hand of the user is in the initial position when the hand shape with at least one finger raised is continuously recognized for a predetermined time or longer. 2. The gesture recognition device according to item 1. a gesture recognition unit that recognizes a hand gesture of the user based on the captured image;
The gesture recognition device according to any one of claims 1 to 5, wherein the gesture recognition unit recognizes the gesture after it is determined that the user's hand is in the initial position. 7. The notification unit according to any one of claims 1 to 6, wherein, when it is determined that the user's hand is not at the initial position, the notification unit notifies the user of guidance information for guiding the user's hand to the initial position. 2. The gesture recognition device according to claim 1. A program for a gesture recognition device comprising imaging means for imaging a user's hand, and recognizing a gesture of the user's hand based on an image captured by the imaging means,
recognizing the hand shape of the user reflected in the captured image, and moving the user's hand to an initial position suitable for starting a gesture when the recognized hand shape is a hand shape with at least one finger raised; A function as an initial position determination unit that determines that there is
and a function as an informing unit for informing the user that a gesture may be started when it is determined that the user's hand is in the initial position. program.

Images