JP2018195053A - Image processing apparatus, image processing program, and gesture recognition system - Google Patents

Abstract

To provide an image processing apparatus configured to accurately determine a shape of a finger included in an image when an object is image-recognized, an image processing program, and a gesture recognition system.SOLUTION: An image recognition unit 12 executes image recognition processing for recognizing a hand image, which is an area where an object is included in image information D, on the image information D acquired in an image acquisition unit 11. A finger shape determination unit 13 determines a shape of a finger of a hand, on the basis of a distance in at least a depth direction of an image coordinate system including the hand image recognized by the image recognition unit 12.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing program, and a gesture recognition system that recognize an object.

  Conventionally, a gesture recognition system has been developed in which a user's gesture operation is determined by recognizing an image captured by a camera or the like, and a device is controlled based on the determination result. As an image processing apparatus for recognizing a captured image, the technique of Patent Document 1 is well known. Patent Document 1 determines the shape of a finger of a hand by specifying the number of fingers and the distance from a specific point on the image plane to the fingertip based on the result of image processing of a captured image.

Japanese Patent No. 3863809

  By the way, for example, a hand shape with an open finger may be reflected like a pointing hand image depending on the image capturing direction. For example, when a hand shape with an open finger is photographed from the palm or back of the hand, it can be recognized as an image with the finger open, whereas when the hand is photographed from the side, the fingers overlap in the depth direction of image capture. Are lined up. Therefore, there is a possibility that the hand shape with the finger open is erroneously determined as the pointing shape.

  An object of the present invention is to provide an image processing apparatus, an image processing program, and a gesture recognition system capable of accurately determining the shape of a finger appearing on an image when recognizing an object.

  An image processing apparatus that solves the problem includes an image acquisition unit that acquires image information representing an image in which the object is reflected, from an image capturing unit that captures a hand as an object of image recognition, and the image information An image recognition process for recognizing a hand image that is an area in which the object is reflected in the image information acquired by the image acquisition unit, and the hand recognized by the image recognition unit And a finger shape determining unit that determines the shape of the finger of the hand based on at least a distance in a depth direction of an image coordinate system in which an image is shown.

  According to this configuration, a hand image as an object of image recognition is detected through image recognition processing, and the shape of the finger of the hand shown on the image is determined based on at least the distance in the depth direction of the image coordinate system in the hand image. Determine. As described above, in the case of this configuration, the shape of the finger shown in the hand image is determined using the distance in the depth direction of the image coordinate system, rather than simply recognizing the shape of the finger from the planar image of the image coordinate system. Therefore, it is possible to accurately determine the shape of the finger of the hand shown on the image.

  In the image processing device, the image photographing unit obtains the image information including three-dimensional distance information representing a distance from an object for each pixel, and the image recognition unit obtains the image obtained from the image photographing unit. It is preferable to execute the image recognition based on the three-dimensional distance information included in the information. According to this configuration, the image information including the three-dimensional distance information is subjected to image recognition processing, and the human hand shown on the image is acquired as a stereoscopic image. Therefore, it is possible to accurately determine the shape of the finger shown on the image using the stereoscopic image.

  In the image processing device, the finger shape determination unit obtains a distance between a part of the object image in the hand image having a closest distance in the depth direction of the image coordinate system and a fingertip reflected in the hand image. It is preferable to determine the shape of the finger of the hand from the distance. According to this configuration, it is possible to accurately determine the shape of the finger using the characteristic shape of the hand that is taken when the pointing operation is performed.

  In the image processing apparatus, it is preferable that the finger shape determination unit determines the shape of the finger of the hand from an array of distances in the depth direction of the image coordinate system in the hand image. According to this configuration, it is possible to determine the shape of the finger of the hand shown on the image by a simple process of confirming the arrangement of the distance in the depth direction of the image coordinate system in the hand image.

  In the image processing apparatus, the finger shape determination unit determines the shape of the finger of the hand based on a distribution of the distance or an inclination of a change in the distance in the depth direction of the image coordinate system in the hand image. Is preferred. According to this configuration, it is possible to determine the shape of the finger of the hand shown on the image by a simple process of confirming the distance distribution or the gradient of the distance change for the distance in the depth direction of the image coordinate system in the hand image. It becomes possible.

  An image processing program that solves the problem includes a first step of acquiring image information representing an image in which the object is reflected from an image photographing unit that captures a hand as an object of image recognition; and the image information A second step of executing image recognition processing for recognizing a hand image that is an area in which the object is reflected in the image information acquired in the first step, and the hand recognized in the second step. And causing the computer to execute a third step of determining the shape of the finger of the hand based on at least the distance in the depth direction of the image coordinate system in which the image is shown.

  A gesture recognition system that solves the problem includes an image capturing unit that captures a hand as an object of image recognition, and an image acquisition unit that acquires image information representing an image in which the object is reflected from the image capturing unit. An image recognition unit that executes image recognition processing for recognizing a hand image that is an area in which the object is reflected in the image information, with respect to the image information acquired by the image acquisition unit, and the image recognition unit The finger shape determination unit that determines the shape of the finger of the hand based on at least the distance in the depth direction of the image coordinate system in which the hand image recognized by the user is recognized, and the hand on the image specified by the finger shape determination unit A gesture control unit that determines a gesture operation using a finger of a user's hand based on the movement of the finger and controls the device based on the determination result;

  According to the present invention, when recognizing an object, the shape of a finger shown on the image can be accurately determined.

1 is a configuration diagram of an image processing device and a gesture recognition system according to a first embodiment. The perspective view of the driver's seat which shows the example of arrangement | positioning of an image pick-up part. FIG. The three-dimensional model figure of a three-dimensional distance image. (A), (b) is explanatory drawing of the method of determination of the shape of a finger | toe. 6 is a flowchart executed during image processing. (A), (b) is explanatory drawing of the method of the determination of the shape of the finger | toe shown in 2nd Embodiment. (A), (b) is explanatory drawing of the method of determination of the shape of a finger | toe.

(First embodiment)
Hereinafter, a first embodiment of an image processing device, an image processing program, and a gesture recognition system will be described with reference to FIGS.

  As shown in FIG. 1, the vehicle 1 includes a gesture recognition system 3 that recognizes a gesture operation by a user and controls the device 2. The gesture recognition system 3 of this example recognizes a user's gesture operation based on the image information D photographed by the image photographing unit 4 and controls the device 2 in a manner corresponding to the gesture operation. The device 2 is preferably an in-vehicle device 5, for example. The in-vehicle device 5 includes, for example, a center display, a car navigation system, an air conditioner, a volume device, an audio device, and the like.

  The image photographing unit 4 is capable of photographing an object for image recognition (such as a user's hand), and obtains image information D including, for example, three-dimensional distance information Da between the objects (distance image camera). ). The range image camera may be, for example, a TOF (Time of Flight) camera or a TOF sensor. The distance image camera, for example, emits infrared rays from the camera, and outputs a distance image, that is, image information D with distance information, from the time delay of the reflected light. As described above, the image information D of the present example includes the three-dimensional distance information Da including the distance data between the object.

  The gesture recognition system 3 includes an image processing device 8 that performs image recognition processing on the image information D input from the image capturing unit 4 and detects a gesture operation. The image processing device 8 detects the user's gesture operation by recognizing the user's hand (fingertip) in the image captured by the image capturing unit 4 and tracking the movement of the hand (fingertip). As the image recognition processing, various methods can be adopted as long as they are image recognition methods capable of detecting an object (human body) from an image.

  The gesture operation is preferably a hand gesture operation using a user's hand, for example. In the case of hand gesture operation, for example, the device 2 is controlled to a predetermined mode by moving the index finger to a predetermined mode. The hand gesture operation may be an operation using only hand movement and a combination of hand movement and voice, for example. In the case of an operation in which hand movement and voice are combined, for example, the device 2 is controlled by pointing to the device 2 and uttering specific operation contents by voice.

  The image processing apparatus 8 includes an image acquisition unit 11 that acquires image information D from the image capturing unit 4. The image acquisition unit 11 acquires image information D representing an image in which the object is reflected from the image photographing unit 4 that captures a hand as an object for image recognition. In the case of this example, the image acquisition unit 11 acquires image information D showing the user's hand from the image capturing unit 4.

  The image processing apparatus 8 includes an image recognition unit 12 that performs image recognition processing on the image information D acquired by the image acquisition unit 11. The image recognition unit 12 performs image recognition processing for recognizing the hand image R ′, which is the region R in which the object is captured, in the image information D on the image information D acquired by the image acquisition unit 11. In the case of this example, the image recognition unit 12 recognizes the user's hand image R ′ shown on the image through image recognition processing, and extracts the fingertip and the grip portion of the hand from the hand image R ′.

  The image processing apparatus 8 includes a pointing image detection function that can accurately detect the shape of the finger of the hand shown in the image recognized by the image recognition unit 12. By the way, when the user performs a gesture operation by pointing (with one finger up), it is necessary to recognize an image of the shape of the finger at this time. However, for example, when an image of a hand with a finger spread out is taken, if the hand is taken from the side, the fingers are overlapped, and this may be detected as a pointing state. This example is a technique for coping with this.

  The image processing apparatus 8 includes a finger shape determination unit 13 that determines the shape of the finger of the hand in the hand image R ′ recognized by the image recognition unit 12. The finger shape determination unit 13 in this example determines the shape of the finger of the hand based on at least the distance in the depth direction of the image coordinate system in which the hand image R ′ recognized by the image recognition unit 12 is reflected. In the case of this example, the finger shape determination unit 13 determines the shape of the finger of the hand from the distance L between the fingertip and the grip portion in the image coordinate system.

  The image processing apparatus 8 includes a gesture control unit 14 that controls the operation of a gesture operation using image recognition. The gesture control unit 14 determines a gesture operation based on the hand image R ′ whose finger shape is specified by the finger shape determination unit 13 and controls the device 2 based on the determination result. The gesture control unit 14 outputs a control command Sr corresponding to the recognized gesture operation to the body ECU 15 and controls the device 2 through the body ECU 15.

Next, operations and effects of the gesture recognition system 3 (image processing apparatus 8) according to the embodiment of the present invention will be described with reference to FIGS.
As shown in FIG. 2, the image capturing unit 4 starts image capturing when the vehicle 1 is operated to power on (for example, ACC on or IG on), for example. FIG. 2 is an example in which the image photographing unit 4 photographs a human hand or arm from below. In this case, the image photographing unit 4 is preferably arranged in the vicinity of a touch pad installed on a center console or the like, for example. When the user controls the device 2 by a gesture operation, if the user is a driver, the user reaches forward from the driver's seat and moves the index finger along a predetermined mode.

  The image capturing unit 4 outputs image information D including the acquired three-dimensional distance information Da to the image processing device 8. The image processing device 8 acquires the image information D output from the image capturing unit 4 by the image acquisition unit 11.

As shown in FIGS. 3 and 4, the image recognition unit 12 performs image recognition processing on the image information D including the three-dimensional distance information Da to recognize an object on the image. In the case of this example, first, the image recognition unit 12 performs a process of identifying all regions (candidate regions) R that are likely to be human bodies reflected in an image, so-called labeling. As an example of the method for specifying the candidate region R, the following (I) to (IV) are assumed.
(I) Skin color area (II) Brightness exceeds threshold value (III) Area surrounded by edges in edge detection (IV) Area having area equal to or greater than threshold value (IV) Distance from image capturing unit 4 is the same Region in which the subject exists within a range that can be regarded as an image The image recognition unit 12 labels the extracted region (candidate region) R. In the case of this example, a label is attached to the region R in which the hand is reflected, and it is determined whether or not the label is a human hand. The image recognition unit 12 recognizes the label as a candidate for the hand image R ′ if the condition that the label is a hand shape is satisfied.

  Note that when the image recognition unit 12 extracts a candidate for the three-dimensional hand image R ′ from the image information D, the image recognition unit 12 recognizes a part where the edge is raised in the hand image R ′ as a fingertip candidate (circle shown in FIG. 3). Then, the image area of the area where these edges are connected is recognized as a palm candidate (square mark shown in FIG. 3), and the gravity center position of the palm is recognized as a hand gravity center candidate (triangle mark shown in FIG. 3).

  As shown in FIG. 4, the finger shape determination unit 13 recognizes a portion where an edge is raised in the three-dimensional hand image R ′ extracted by the image recognition unit 12 as “tip P1 of the pointing finger”. In addition, the finger shape determination unit 13 determines a location where the distance in the depth direction of the image coordinate system in the three-dimensional hand image R ′ is the closest in the depth direction of the image coordinate system as “finger gripping portion P2”. Recognize as The finger shape determination unit 13 calculates a distance L (three-dimensional distance) between the “tip finger tip P1” and the “finger gripping portion P2” in the depth direction of the image coordinate system. The threshold value Lk is compared.

  As shown in FIG. 5A, when the three-dimensional hand image R ′ is a hand pointing operation, the distance L between the tip P1 of the pointing finger and the finger gripping portion P2 becomes large. . Therefore, the finger shape determination unit 13 confirms that the distance L between the tip P1 of the index finger and the grip portion P2 of the finger is equal to or greater than the threshold Lk. Recognize as “Insertion image”.

  On the other hand, as shown in FIG. 5B, when the three-dimensional hand image R ′ is an operation in which the finger of the hand is opened, the distance between the tip P1 of the index finger and the grip portion P2 of the finger. L becomes smaller. For this reason, the finger shape determination unit 13 confirms that the distance L between the tip P1 of the index finger and the grip portion P2 of the finger is less than the threshold Lk. It is recognized as a "pointing image". That is, when the distance L is less than the threshold value Lk, the hand image R ′ is not recognized as a “pointing image”.

  The image recognizing unit 12 confirms whether or not the region R obtained by the image recognition process, that is, the hand image R ′ is certain as a recognition target. As a method for confirming the recognition target, for example, if the recognition target is a finger, a method for determining whether or not the contour of the finger satisfies a prescribed condition is assumed.

  If the recognition target is certain, the image recognition unit 12 determines the determination result that the region R is the pointing hand image R ′. Then, the image recognition unit 12 recognizes the pointing hand image R ′ shown on the image by repeating the above image recognition processing in a predetermined cycle. The gesture control unit 14 determines a gesture operation from the movement of the hand (forefinger) detected by the above image recognition process, and controls the in-vehicle device 5 in a mode corresponding to the gesture operation.

A summary of the operation of the image processing apparatus 8 of this example will be described with reference to the flowchart shown in FIG.
In step 101, the image acquisition unit 11 acquires, from the image capturing unit 4, image information D (three-dimensional distance information Da) in which at least a part of the human body (in this example, a hand) is reflected as an object for image recognition. Step 101 corresponds to the first step.

  In step 102, the image recognition unit 12 executes image recognition processing for recognizing the hand image R ′, which is the region R in which the object is captured, in the image information D on the image information D acquired by the image acquisition unit 11. To do. Here, for example, a region R that is likely to be a human body reflected in an image is extracted, and a label is attached to it to determine whether or not it is a hand image R ′. If the labeled image satisfies the condition of the hand image R ′, the image is recognized as a candidate for the hand image R ′. Step 102 corresponds to the second step.

  In step 103, the finger shape determination unit 13 determines the shape of the finger of the hand from the image-recognized hand image R '. In this example, the finger shape determination unit 13 recognizes the hand on the image as a “pointing image” if the distance L between the tip P1 of the index finger and the grip portion P2 of the finger is equal to or greater than the threshold Lk. To do. Step 103 corresponds to the third step.

In step 104, the image recognition unit 12 confirms whether the hand image R ′ specified in step 103 is certain as a recognition target.
In step 105, if the recognized image is certain, the gesture control unit 14 determines the user's gesture operation based on the image, and controls the device 2 (the in-vehicle device 5) according to the determination result. Thereby, the user can operate the device 2 through a simple hand gesture. Therefore, the convenience of operation of the device 2 is improved.

  In the present example, the 3D distance information Da captured by the image capturing unit 4 is subjected to image recognition processing to recognize the hand image R ′ on the image, and at least the depth direction ( Based on the distance in the Z-axis direction), the shape of the finger of the hand shown on the image is determined. As described above, in this example, the finger shape is not simply recognized from the planar image of the image coordinate system, but the finger shape shown in the hand image R ′ is determined using the distance in the depth direction of the image coordinate system. . Accordingly, the shape of the finger of the hand shown on the image can be accurately determined.

  The image capturing unit 4 is a distance measuring type that acquires three-dimensional distance information between the object and the object. The image recognition unit 12 performs image recognition based on the three-dimensional distance information Da acquired from the image photographing unit 4. In the case of this example, the 3D distance information Da is subjected to image recognition processing, and the hands and arms shown on the image are acquired as a stereoscopic image. Therefore, the shape of the finger shown on the image can be accurately determined using this type of three-dimensional distance information Da.

  The finger shape determination unit 13 extracts a three-dimensional object set part 20 (corresponding to a component part of the object) from the hand image R ′, and in this set part 20 in the depth direction (Z-axis direction) of the image coordinate system. The distance L between the closest part and the fingertip shown in the hand image R ′ is obtained, and the shape of the finger of the hand is determined from this distance L. Therefore, it is possible to accurately determine the shape of the finger using the shape characteristic unique to the hand taken when the pointing operation is performed.

(Second Embodiment)
Next, a second embodiment will be described with reference to FIGS. In addition, 2nd Embodiment is an Example which changed the determination method of the finger | toe shape of 1st Embodiment. Therefore, the same parts as those in the first embodiment are denoted by the same reference numerals, detailed description thereof is omitted, and only different parts are described in detail.

  As shown in FIGS. 7A and 7B and FIGS. 8A and 8B, the finger shape determination unit 13 confirms the distance in the depth direction of the image coordinate system in the hand image R ′ and determines these depths. The shape of the finger of the hand is determined from the array of distances in the direction. Examples of the arrangement of distances in the depth direction in the image coordinate system of the hand image R ′ include, for example, the distribution of distances in the depth direction in the image coordinate system (see FIGS. 7A and 7B) and the depth direction in the image coordinate system. There is a gradient ΔLz of distance change (see FIGS. 8A and 8B).

  In the example of FIGS. 7A and 7B, the finger shape determination unit 13 confirms the distance distribution in the distance in the depth direction of the image coordinate system in the hand image R ′. At this time, if the finger shape determination unit 13 recognizes a distance distribution in which the distance in the depth direction suddenly increases and the distance is wide, the hand image R ′ is recognized as a “pointing image” (FIG. 7A). )reference). On the other hand, the hand image R ′ that does not take such a distance distribution is recognized as not a “pointing image” (see FIG. 7B).

  In the case of the example of FIGS. 8A and 8B, the finger shape determination unit 13 confirms the slope ΔLz of the distance change in the distance in the depth direction of the image coordinate system in the hand image R ′. At this time, the finger shape determination unit 13 recognizes the hand image R ′ as a “pointing image” when the gradient ΔLz of the distance change in the depth direction is less than the threshold (see FIG. 8A). The reason why such an inclination ΔLz is taken is that the gripping part of the hand exists in front of the fingertip as a characteristic of the pointing hand. On the other hand, the finger shape determination unit 13 recognizes that the hand image R ′ is not a “pointing image” when the gradient ΔLz of the distance change in the depth direction is equal to or greater than the threshold value. (See FIG. 8 (b)).

  In the case of this example, the finger shape determination unit 13 confirms the distance in the depth direction of the image coordinate system in the hand image R ′, and an array of these distances in the depth direction (in this example, the distribution of distance and the change in distance). The shape of the finger of the hand is determined from the inclination ΔLz). Therefore, the finger shape of the hand shown on the image can also be determined by a simple process of confirming the arrangement of the distances in the depth direction of the image coordinate system in which the hand image R ′ is shown.

The embodiment is not limited to the structure described so far, and may be modified as follows.
In each embodiment, the distance L between the tip P1 of the index finger and the grip portion P2 of the finger may be a distance in the depth direction on the three-dimensional image. That is, the depth direction of the image coordinate system is not limited to the direction along the Z axis of the image coordinate system, and may be a direction facing a predetermined amount with respect to the Z axis. In short, the depth direction of the image coordinate system may be a direction along a line intersecting the image plane (XY plane).

-In each embodiment, the attachment position of the image pick-up part 4 can be changed into other positions, such as an instrument panel, for example.
In each embodiment, the image capturing unit 4 is not limited to one, and a plurality of image capturing units 4 may be provided.

In each embodiment, the image capturing unit 4 is not limited to a TOF camera (TOF sensor), and various range image cameras can be used.
In each embodiment, the image information D may be changed as appropriate according to the type of the image capturing unit 4 to be used.

In each embodiment, the image information D may be changed to various information as long as it is three-dimensional information with distance information.
In each embodiment, various methods such as image processing, image processing algorithm, pattern matching, and image feature extraction can be adopted as the image recognition processing. The image processing is processing for extracting information by performing processing on an image, for example. The image processing algorithm is processing for specifying a shape by performing mathematical processing conversion on a computer, for example. Pattern matching is a process for recognizing an image by comparing an acquired image with a template, for example. Image feature extraction is processing for extracting features from an image and specifying a shape.

In each embodiment, the finger shape to be determined is not limited to the pointing shape, and can be changed to other shapes such as a finger shape protruding two fingers.
In each embodiment, the finger shape determination method is not limited to a method for confirming the distance difference between the fingertip and the gripping portion of the finger or a method for confirming the array of distances in the depth direction in the image coordinate system. The method can be appropriately changed to other methods.

In each embodiment, the gesture operation may be, for example, an operation for opening / closing a power slide door or an operation used when the vehicle 1 is parked automatically.
In each embodiment, the gesture recognition system 3 and the image processing device 8 are not limited to being applied to the vehicle 1, and may be used for other devices and apparatuses.

  DESCRIPTION OF SYMBOLS 1 ... Vehicle, 2 ... Apparatus, 3 ... Gesture recognition system, 4 ... Image pick-up part, 5 ... In-vehicle apparatus, 8 ... Image processing apparatus, 11 ... Image acquisition part, 12 ... Image recognition part, 13 ... Finger shape determination part, 14: Gesture control unit, 20: Collective part, D: Image information, Da ... Three-dimensional image information, R ... Area, R '... Hand image, L ... Distance, Lk ... Threshold, P1 ... Tip of index finger, P2 ... Finger grip part.

Claims (7)

An image acquisition unit that acquires image information representing an image in which the object is captured, from an image capturing unit that captures a hand as an object of image recognition;
An image recognition unit that executes an image recognition process for recognizing a hand image that is an area in which the object is reflected in the image information, with respect to the image information acquired by the image acquisition unit;
An image processing apparatus comprising: a finger shape determination unit that determines a finger shape of the hand based on at least a distance in a depth direction of an image coordinate system in which the hand image recognized by the image recognition unit is reflected. The image photographing unit obtains the image information including three-dimensional distance information representing a distance between the object for each pixel,
The image processing apparatus according to claim 1, wherein the image recognition unit performs the image recognition based on the three-dimensional distance information included in the image information acquired from an image capturing unit. The finger shape determination unit obtains a distance between a portion of the object image in the hand image where the distance in the depth direction of the image coordinate system is closest and a fingertip reflected in the hand image, and the hand shape is determined from the distance. The image processing apparatus according to claim 1, wherein the shape of the finger is determined. The image processing apparatus according to claim 1, wherein the finger shape determination unit determines a shape of the finger of the hand from an array of distances in a depth direction of an image coordinate system in the hand image. The image according to claim 4, wherein the finger shape determination unit determines the shape of the finger of the hand based on a distribution of the distance or an inclination of a change in the distance in the depth direction of the image coordinate system in the hand image. Processing equipment. A first step of acquiring image information representing an image in which the object is captured from an image capturing unit that captures a hand as an object of image recognition;
A second step of executing image recognition processing for recognizing a hand image that is an area in which the object is reflected in the image information, with respect to the image information acquired in the first step;
An image processing program for causing a computer to execute a third step of determining a shape of a finger of the hand based on at least a distance in a depth direction of an image coordinate system in which the hand image recognized in the second step is reflected. An image capturing unit that captures a hand as an object of image recognition;
An image acquisition unit for acquiring image information representing an image in which the object is reflected from the image capturing unit;
An image recognition unit that executes an image recognition process for recognizing a hand image that is an area in which the object is reflected in the image information, with respect to the image information acquired by the image acquisition unit;
A finger shape determination unit that determines a finger shape of the hand based on at least a distance in a depth direction of an image coordinate system in which the hand image recognized by the image recognition unit is reflected;
A gesture control unit that determines a gesture operation using the finger of the user's hand based on the movement of the finger of the hand on the image specified by the finger shape determination unit, and controls the device based on the determination result; Gesture recognition system provided.