KR20110080871A - Portable information terminal and content control method using rear finger movement and gesture recognition - Google Patents

Abstract

The present invention recognizes finger movements and finger gestures of a hand held by the portable information terminal by using an image acquisition device provided at the rear of the portable information terminal including an image recognition device and a memory. The present invention relates to a method of controlling available content, the method comprising: a first step of acquiring an image including a finger region of a hand held by the portable information terminal using the image acquisition device; Detecting a finger region from the acquired image and extracting features for finger movement and finger gesture recognition; A third step of recognizing a finger gesture using the feature extracted in the third step to determine whether a finger movement shape of the hand holding the portable information terminal is a finger gesture or a finger movement; A fourth step of detecting a position of a finger from the acquired image when it is determined that the finger movement is performed in the third step; A fifth step of detecting an area change of a finger object region from the acquired image when it is determined that the finger movement is performed in the third step; A sixth step of performing recognition on finger movement by using the feature extracted in the third step, the position of the finger detected in the fourth step, and the area change of the finger object region detected in the fifth step; And a seventh step of controlling the portable information terminal and the content by using the result of the recognition performed in the third step or the result of the recognition performed in the sixth step.

Description

TECHNICAL FOR CONTROLLING A PORTABLE INFORMATION TERMINAL AND CONTENTS THEREOF USING RECOGNIZING FINGER MOTIONS AND GESTURES ON THE BACK OF THE PORTABLE INFORMATION TERMINAL}

The present invention relates to a portable information terminal itself and to a method for controlling content that can be used as a portable information terminal. A method for recognizing and controlling content available to a portable information terminal and a portable information terminal.

In the past, in order to control the contents available by the portable information terminal itself and the portable information terminal, it was common to use a user interface such as a touch screen or a keypad provided in the front portion (front) of the portable information terminal. As portable information terminals are widely used today, the demand for technology development for a method of controlling the portable information terminals under various conditions of use is increasing. However, there is currently no prior art related to a method of controlling a content available to a portable information terminal or a portable information terminal by recognizing finger movements or finger gestures of a hand held by the portable information terminal.

In order to solve this problem, the present invention recognizes finger movements and finger gestures of a hand held by a portable information terminal using an image acquisition device provided at a rear portion of the portable information terminal, thereby enabling content to be used as a portable information terminal and a portable information terminal. It is an object to provide a method of controlling.

The present invention recognizes finger movements and finger gestures of a hand held by the portable information terminal by using an image acquisition device provided at the rear of the portable information terminal including an image recognition device and a memory. A method of controlling available content, the method comprising: a first step of acquiring an image including a finger region of a hand held by the portable information terminal using the image acquisition device; Detecting a finger region from the acquired image and extracting features for finger movement and finger gesture recognition; A third step of recognizing a finger gesture using the feature extracted in the third step to determine whether a finger movement shape of the hand holding the portable information terminal is a finger gesture or a finger movement; A fourth step of detecting a position of a finger from the acquired image when it is determined that the finger movement is performed in the third step; A fifth step of detecting an area change of a finger object region from the acquired image when it is determined that the finger movement is performed in the third step; A sixth step of performing recognition on finger movement by using the feature extracted in the third step, the position of the finger detected in the fourth step, and the area change of the finger object region detected in the fifth step; And a seventh step of controlling the portable information terminal and the content by using the result of the recognition performed in the third step or the result of the recognition performed in the sixth step. Here, the finger gesture refers to a specific movement shape of a finger previously defined among the movement shapes of the finger and stored in the memory of the portable information terminal, and the finger movement refers to the movement shape of the finger instead of the finger gesture among the movement shapes of the finger. Will be pointing.

Further, according to an embodiment of the present invention, the feature extracted in the third step includes the degree of finger bending of the hand holding the portable information terminal, and the position of the finger detected in the fourth step is a two-dimensional plane or a multi-dimensional space region. The change in the area of the finger object region detected in the fifth step may be a change in the area of the finger object region in the acquired image according to the change of the distance between the image capturing apparatus and the finger.

In addition, according to an embodiment of the present invention, the portable information terminal may further include a touch screen or a keypad, and when the portable information terminal and the content are controlled by the control method, the touch screen or the keypad may be a portable information terminal and the content. When the command for controlling the portable information terminal and the content is not controlled by the user, the touch screen or keypad may receive a command for controlling the portable information terminal and the content from the user. You can make it as possible.

According to the present invention, the user's convenience can be increased by controlling the portable information terminal itself and the contents available to the portable information terminal in a non-contact manner with a simple finger operation, and a conventional portable information terminal control method such as a touch screen or a keypad and the like. The function is provided as it is, but in addition, the user can operate and control the portable information terminal with only one hand holding the portable information terminal, thereby providing a more convenient and efficient user environment.

1 is a block diagram of a system for performing a portable information terminal and a content control method using finger movement and gesture recognition according to an embodiment of the present invention.
2 is a flowchart of a method for performing a portable information terminal and a content control method using finger movement and gesture recognition according to an embodiment of the present invention.
3 is a block diagram of a system for performing a portable information terminal and a content control method using finger movement and gesture recognition according to another embodiment of the present invention.
4 is a flowchart of a method for performing a portable information terminal and a content control method using finger movement and gesture recognition according to another embodiment of the present invention.
FIG. 5 is a diagram illustrating an embodiment for acquiring a finger gesture image when a portable information terminal is held by a right hand.
FIG. 6 is a diagram illustrating an embodiment of obtaining a finger gesture image when a portable information terminal is held by a left hand.
7 is a diagram illustrating a method of executing a command according to a movement of a center coordinate of a finger position when a portable information terminal is held by a right hand according to a preferred embodiment of the present invention.
8 is a diagram illustrating a method of executing a command according to a change in the area of a finger object area when a portable information terminal is held by a right hand according to a preferred embodiment of the present invention.
9 is a diagram illustrating a method of executing an instruction according to a movement of a center coordinate of a finger position when a portable information terminal is held by a left hand according to a preferred embodiment of the present invention.
FIG. 10 illustrates a method of executing a command according to a change in the area of a finger object area when a portable information terminal is held by a left hand according to a preferred embodiment of the present invention.

In the present specification, the finger gesture refers to a specific movement shape of a finger previously defined among the movement shapes of the finger and stored in the memory of the portable information terminal, and the finger movement refers to a finger other than the finger gesture among the movement shapes of the finger. Will point to the shape of the movement.

Further, in the present invention, the feature used for the movement or gesture recognition of a two-dimensional plane or a multi-dimensional finger includes a degree of bending of a finger and a shape of a symbolized finger within a range that can be photographed by an image photographing device provided in a portable information terminal. The location of the finger center coordinates in the two-dimensional plane or the multi-dimensional space area, the distance of the continuous finger movement, that is, the area change of the finger object area in the image according to the distance between the image capturing device and the finger. Finger movements or finger gestures recognized using such a feature perform a series of processes for operating and executing a pre-mapped portable information terminal control command or a control command for embedded related content.

A portable information terminal and a content control method using finger movement and gesture recognition will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a system for performing a portable information terminal and a content control method using finger movement and gesture recognition according to an embodiment of the present invention, and FIG. 2 is a finger movement and gesture according to an embodiment of the present invention. A flowchart of a portable information terminal and content control method using recognition.

As shown in FIG. 1, the system 100 for performing a portable information terminal and a content control method according to the present invention includes a finger region image acquisition unit 101, a finger region detection and feature extraction unit 103, and a finger. The gesture and finger motion determination unit 105, the finger position detection unit 107, the finger object area area change detection unit 109, the finger motion recognition unit 111, and the portable information terminal and the content control unit 113 are included.

In addition, the portable information terminal and the content control method 200 according to the present invention, as shown in Figure 2, using the image acquisition device to obtain an image including the finger region of the hand holding the portable information terminal In a first step S201, a finger region is detected from the obtained image, a second step S203 for extracting features for finger movement and finger gesture recognition, and a feature extracted for the finger gesture using the features extracted in the third step. In the third step (S205) of determining whether the shape of the finger movement of the hand holding the portable information terminal is a finger gesture or a finger movement by performing the recognition, if it is determined that the finger movement in the third step, In step S207 of detecting a position of a finger from the captured image, when it is determined that the finger movement is performed in the third step, the acquired image Using the fifth step S209 of detecting the area change of the finger object area, the feature extracted in the third step, the position of the finger detected in the fourth step, and the area change of the finger object area detected in the fifth step A sixth step (S211) of performing a recognition on a finger movement, and a seventh step of controlling the portable information terminal and contents using the result of the recognition performed in the third step or the result of the recognition performed in the sixth step. It may include (S213).

The finger region image acquisition unit 101 performs a first step (S201) of acquiring an image including a finger region of a hand held by the portable information terminal using an image acquisition device provided in the portable information terminal. The image acquisition device included in the portable information terminal is generally a built-in type provided in the rear portion (back side), front portion (front side), or side portion (side) of the portable information terminal. The invention can be applied. In addition, the present invention can be applied regardless of whether the image capturing apparatus is a single camera, a multiple camera, or a stereo camera. In the embodiment of the present invention, it is assumed that the image capturing apparatus is provided on the rear portion (back side) of the portable information terminal. In relation to image acquisition, FIG. 5 illustrates an embodiment for acquiring a finger gesture image when a portable information terminal is held by a right hand, and FIG. 6 is a finger gesture image when a portable information terminal is held by a left hand. Figure is a diagram showing an embodiment for obtaining the.

The finger region detection and feature extractor 103 detects a finger region from the image acquired by the finger region image acquirer 101 and performs a second step S203 of extracting a feature for finger movement and finger gesture recognition. do. Here, the image includes not only a two-dimensional plane image, but also a three-dimensional image which can know the depth (distance) information on the two-dimensional plane. In the present invention, in order to recognize the image of the object to reflect the various characteristics of the object, from the acquired object image information using the Hada-like feature-based AdaBoost learning algorithm, Template Matching, etc. Detection and feature detection using a feature extraction algorithm that includes Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), Independent Component Analysis (ICA), and 2-D Discrete Cosine Transform (2D-DCT). Extract In particular, in detecting the finger region, color information, contour features, and the like may be utilized, and as a feature related to the shape of a finger, a finger bending degree of a hand holding a portable information terminal may be used.

The finger gesture and the finger motion determination unit 105 recognize the finger gesture by using the feature extracted from the finger region detection and feature extraction unit 103 so that the shape of the finger movement of the hand holding the portable information terminal is changed. A third step (S205) is performed to determine whether it is a finger gesture or a finger movement. As described above, the finger gesture in the present specification refers to a specific movement shape of a finger which is previously defined among the movement shapes of the finger and stored in the memory of the portable information terminal. It may be mapped to a specific control command and stored in the memory of the portable terminal in advance. The purpose of symbolizing the finger gesture is to allow a user to directly execute a specific control command by taking a variety of finger gestures by using the index finger having a relatively high degree of freedom of movement when the user is simply holding the portable information terminal. In the third step, a pattern recognition process is performed on which finger gesture corresponds to the shape of the finger motion in the finger region image, and when a corresponding finger gesture exists, a specific control command is mapped to the shape of the user's finger motion. The method determines the symbolized finger gesture to control the portable information terminal and the content (S213). If the corresponding finger gesture does not exist, the user's finger movement shape is determined to be a continuous finger movement and additional features are determined. The process proceeds to the extraction step (S207, S209).

When it is determined that the finger movement is continuous in the third step S205, the finger position detector 107 performs a fourth step S207 of detecting the position of the finger from the acquired image. In the fourth step S207, as a feature for recognizing the movement of the finger in the finger movement recognition step S211, the position of the finger center coordinates in the two-dimensional plane or the multi-dimensional space area is provided.

When it is determined that the finger movement is continuous in the third step S205, the finger object region area change detection unit 109 performs a fifth step S209 of detecting an area change of the finger object area from the obtained image. . In the fifth step S209, as a feature for recognizing the movement of a finger in the finger movement recognizing step S211, the area change of the finger object region in the acquired image according to the distance between the image acquisition device and the finger is changed. to provide.

In the order in which the fourth step S207 and the fifth step S209 are performed, the fifth step S209 may be performed after the fourth step S207 is performed, and the fifth step S209 may be performed. Thereafter, the fourth step S207 may be performed.

The finger motion recognition unit 111 detects the feature extracted by the finger region detection and feature extraction unit 103, the position of the finger detected by the finger position detection unit 107, and the finger object region area change detection unit 109. A sixth step S211 of performing finger recognition using a change in the area of the finger object region is performed. This is a step of recognizing the degree of finger movement in the continuous movement of the finger. After detecting the corresponding finger region by using the finger movement image captured by the image capturing apparatus, the area of the detected finger object region is the maximum- The image is normalized to a finger object image having a specific size by using minimum information and a rotation angle of the image, and the center point coordinates of the corresponding object image are obtained using the normalized object image. Thereafter, the corresponding center point is obtained based on the detected finger region to track the degree of change of the position coordinate according to the change of the center point. Here, the change of the center point means the change of the position of the finger object, and the recognition result of the change of the position of the finger object is mapped with a control command such as up / down / left / right / selection (click), and the touch is performed. It can be presented to the user with visual effects such as cursor movement or mouse pointing in the screen panel.

In the finger gesture recognition step S205 or the finger motion recognition step S211, various objects are applied by applying pattern recognition algorithms such as K-NN, support vector machine (SVM), and embedded hidden markov model (EHMM) based on the extracted feature. An image recognition result is derived, and the present invention includes an overall technique of detecting a corresponding region using a variety of pattern recognition algorithms, and interpreting and recognizing the region in multiple dimensions. In addition, the present invention recognizes a hand shape that is robust to angle and self occlusion using multi-dimensional finger movement or gesture estimation, and recognizes finger gesture and finger movement by analyzing the motion shape of the finger. In general, in order to stabilize the recognition result, temporal continuity must be taken into consideration.Motion Energy Image (MEI), which is a binary image representing temporal position change of finger movement, and a MotionHistory Image (MHI) pattern representing temporal lapse information of the motion HMM recognition method that expresses continuous posture as a state change by using is used in the final process of recognition. In order to remove the noise of the input image, one erosion and morphology operations are performed, and a method of using a dilation morphology operation to prevent the finger region from shrinking is also used. Can be. In addition, in order to effectively express and recognize body poses and motions by extracting feature values from a visual input image, finger motion shape feature extraction methods include the Motion Energy Image (MEI) and Motion History Image (MHI) patterns, and Ross Cutler. It may include a method of (Ross Cutler), a cardboard model (cardboard model). In addition, according to the present invention, the features used for finger movement and finger gesture recognition include, as described above, the degree of bending of the finger of the hand holding the portable information terminal, the shape of a predefined finger, two-dimensional plane, or multi-dimensional. The location of the finger center coordinates in the space area or the area change of the finger object area in the acquired image according to the distance between the finger and the image capturing device may be included.

3 is a block diagram 300 of a system for performing a portable information terminal and a content control method using finger movement and gesture recognition according to another embodiment of the present invention, and FIG. 4 is a finger movement according to another embodiment of the present invention. And a method 400 for performing a portable information terminal and a content control method using gesture recognition.

3 and 4, according to an exemplary embodiment of the present invention, the finger gesture recognition may be integrated into the finger gesture and the finger motion recognition step S411 and performed by the finger gesture and the finger motion recognition unit 311. In this case, the final recognition of the finger gesture may not be performed by the finger gesture and the finger movement determining unit 305 for determining whether the motion shape of the finger is the finger gesture or the finger movement (S405).

The portable information terminal and the content control unit 113 control the portable information terminal and the content by using the result of the recognition performed in the finger gesture recognition step S205 or the result of the recognition performed in the finger motion recognition step S211. Step S213 is performed.

According to the present invention, when a finger movement shape is recognized as a finger gesture, a specific control command mapped to the recognized finger gesture is directly executed. On the other hand, when it is recognized as a finger movement, five categories of control commands, for example, up / down / left / right / selection (click), may be executed for controlling the portable information terminal and embedded related contents. In this case, the characteristics of the image capturing apparatus provided in the portable information terminal and the range of finger movement or finger gesture expression are considered.

7 is a diagram illustrating a method of executing a command according to a movement of a center coordinate of a finger position when a portable information terminal is held by a right hand according to a preferred embodiment of the present invention. A diagram illustrating a method of executing an instruction. FIG. 9 is a diagram illustrating a method of executing a command according to a movement of a center coordinate of a finger position when a portable information terminal is held by a left hand according to a preferred embodiment of the present invention, and FIG. A diagram illustrating a method of executing an instruction.

Hereinafter, the configuration and operation of the present invention shown in and described with reference to FIGS. 7 to 10 will be described in at least one embodiment. Control commands by finger movements and gestures can be further expanded and changed in order of command mapping. However, the technical spirit of the present invention and its core configuration and operation are not limited thereto.

7 to 10 illustrate preferred embodiments of keypad operation of a portable information terminal, and control commands such as up / down / left / right / selection (click) in operating a general dial keypad composed of 3 × 4 keys. The configuration method of the is disclosed.

As shown in FIG. 7 or 9, in generating a left / right movement command, a user photographs a finger region moved in a left / right direction by using an image acquisition device provided in a portable information terminal and captures a finger region from the captured image. After the feature region is detected, the coordinates of the center point of the object image are obtained using the normalized object image. Thereafter, the amount of change of the center coordinate is observed and recognized, and the movement of the corresponding finger is performed as a left / right movement command, thereby controlling or executing the portable information terminal and embedded related contents. In addition, after observing the change in the coordinates of the center point, the step of recognizing the movement or gesture of the finger as a left / right command when the duration of a certain time or a predetermined frame is shown using a time stamp or gesture spotting, etc. Include.

Reference is made to FIG. 8 or 10 with respect to generating an up / down movement command using the finger's movement feature in a similar manner. When the distance between the image capturing device and the corresponding finger is close, the area of the finger object area is very large, and when the distance is far, the area of the finger area is detected to be narrower, and thus the change in the area of the finger object area is detected. A degree can be classified into a new feature in addition to the center point coordinate feature. Therefore, in the step of generating the up / down control command, if the area is wide according to the area of the corresponding finger object image, that is, if the distance between the image capturing device and the corresponding finger is close, the motion of the finger is reduced. The "up" movement is defined as "up" movement, but the area of the object image corresponding to the intermediate stage between "up" and "down" in modeling the object feature using the area of the object image is defined as "up". Control commands corresponding to, "middle" and "bottom" can be generated independently. In addition, after observing the change in the area of the finger object image by using a time stamp or gesture spotting after observation, the movement or gesture of the corresponding finger may be recognized as a left / right command. Can be.

In addition, the control point for selecting or executing a portable information terminal and embedded related content, ie, using a center-based coordinate feature and an area of an object image in controlling a selection function similar to a window-based click or double-click function, may be used. The control command may be generated by using a sudden movement of a finger or a gesture. As an example, as shown in FIG. 7 or 9, the finger of the corresponding hand held by the portable information terminal is moved by moving the center coordinates of the finger in a state where the distance between the image acquisition device and the finger is far from the corresponding finger. The extent of the change in the area of the object image can be confirmed. In addition, since the change of the center coordinates can be observed by moving the finger to the left / right, the system recognizes the change in the area of the center coordinates or the object area and recognizes the portable information terminal and the related contents. It is used as a control command for selection or execution.

Image recognition technology for finger gestures or finger movements can be applied to an image-based object recognition technology including a two-dimensional or three-dimensional gesture image as described above, and according to the present invention, The image recognition technology may utilize all conventional image recognition technologies using a single / multi camera or a stereo camera provided in the portable information terminal. In addition, the present invention may include reconstructing the two-dimensional plane and the multi-dimensional finger model in order to track and analyze the multi-dimensional phase movement distance of the continuous finger by extracting the finger region from the entire image.

Furthermore, the system according to the present invention can acquire an image according to the present invention so that a user can execute a control command based on an image acquisition device, combined with a user interface represented by a touch screen and a keypad for inputting system control and operation commands. The apparatus may further include an input mode determiner configured to determine whether a control command is input through the device. In the input mode determination unit, when a control command using finger gesture recognition is input, a user cannot input a control command through a touch screen or a keypad, and a visual expression (mouse) on a touch screen panel through a control command according to the present invention. Pointing, etc.). In other words, the portable information terminal and the content control method using the finger movement and gesture recognition according to the present invention, when the portable information terminal and the content is controlled by the control method according to the present invention, the touch screen or keypad controls the portable information terminal and the content. In order to be able to receive a command for the user from the user, and when the portable information terminal and the content is not controlled using the control method according to the present invention, the touch screen or keypad is a user command for controlling the portable information terminal and content It may further include a function to be in a state that can be input from.

With respect to the present invention, it will be understood from the foregoing description that the foregoing and equivalents thereof may be embodied in various forms. Thus, the true scope of the present invention includes the following claims and any equivalents that may be associated with such person skilled in the art, and are not limited by the specific embodiments described herein.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Also, in this specification, the singular forms are intended to include the plural forms as well, unless the context clearly indicates that the plural is not.

101: finger region image acquisition unit
103: finger region detection and feature extraction unit
105: finger gesture and finger movement determination unit
107: finger position detection unit
109: finger object region area change detection unit
111: finger motion recognition unit
113: portable information terminal and content control unit
S201: finger area image acquisition step
S203: finger region detection and feature extraction step
S205: finger gesture and finger movement judgment step
S207: Finger Position Detection Step
S209: Finger object region area change detection step
S211: Finger Movement Recognition Step
S213: portable information terminal and content control step

Claims (6)

Content that can be used as a portable information terminal and a portable information terminal by recognizing finger movements and finger gestures of a hand held by the portable information terminal by using an image acquisition device provided at a rear portion of the portable information terminal including an image recognition device and a memory. As a method of controlling
A first step of acquiring an image including a finger region of a hand holding the portable information terminal using the image acquisition device;
Detecting a finger region from the acquired image and extracting features for finger movement and finger gesture recognition;
A third step of recognizing a finger gesture by using the feature extracted in the third step to determine whether a finger movement shape of the hand holding the portable information terminal is a finger gesture or a finger movement;
A fourth step of detecting a position of a finger from the acquired image when it is determined that the finger movement is performed in the third step;
A fifth step of detecting an area change of a finger object region from the acquired image when it is determined that the finger movement is performed in the third step;
A sixth step of performing recognition on finger movement by using the feature extracted in the third step, the position of the finger detected in the fourth step, and the area change of the finger object region detected in the fifth step; And
A seventh step of controlling the portable information terminal and the content by using the result of the recognition performed in the third step or the result of the recognition performed in the sixth step,
The finger gesture refers to a specific movement shape of a finger previously defined among the movement shapes of the finger and stored in the memory of the portable information terminal, and the finger movement refers to the movement shape of the finger instead of the finger gesture among the movement shapes of the finger. A portable information terminal and content control method using finger movement and gesture recognition, characterized in that. The method of claim 1,
The feature extracted in the third step includes a finger bending degree of the hand holding the portable information terminal, wherein the portable information terminal and the content control method using finger movement and gesture recognition. The method of claim 1,
The position of the finger detected in the fourth step is the position of the finger center coordinates in the two-dimensional plane or the multi-dimensional space region. The method of claim 1,
The change in the area of the finger object area detected in the fifth step is a change in the area of the finger object area in the acquired image according to the change of the distance between the image capturing device and the finger. And content control method. The method of claim 1,
The feature extracted in the third step includes the degree of bending of the finger of the hand holding the portable information terminal,
The position of the finger detected in the fourth step is the position of the finger center coordinates in the two-dimensional plane or the multi-dimensional space region,
The change in the area of the finger object area detected in the fifth step is a change in the area of the finger object area in the acquired image according to the change of the distance between the image capturing device and the finger. And content control method. The method according to any one of claims 1 to 5,
The portable information terminal further includes a touch screen or a keypad,
A state in which the touch screen or keypad cannot receive a command for controlling the portable information terminal and contents from a user when controlling the portable information terminal and contents by the control method according to any one of claims 1 to 5. When the portable information terminal and the content are not controlled by the control method according to any one of claims 1 to 5, the touch screen or the keypad inputs commands for controlling the portable information terminal and the content from the user. A portable information terminal and content control method using finger movement and gesture recognition, characterized in that the state can be received.

Images