KR20190069023A - Method of Providing Touchless Input Interface Based on Hand Recognition and The Apparatus Applied Thereto - Google Patents

Abstract

Disclosed are a method and apparatus for providing a contactless input interface based on hand recognition. The method comprises the steps of: obtaining an image including a hand of a user using a camera; extracting and binarizing a hand candidate region, which is a region determined by human skin color in the image; extracting an outline of the binarized hand candidate region and detecting a hand outline coincident with a hand shape from the outline; determining fingertip from the hand outline; and judging an input operation of a user using the fingertip.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a touch-

The present invention relates to a method and apparatus for providing a hand-recognition based non-contact type input interface.

The contents described in this section merely provide background information on the present embodiment and do not constitute the prior art.

2. Description of the Related Art Various types of interface technologies using a touch screen, a sensor, and a camera included in a mobile terminal have been researched as a mobile terminal such as a smart phone has become popular.

In recent years, in order to provide various user interface environments, a touch screen in which a touch pad is combined with a display module is employed in a smart device. Thus, a user can input various user commands or select a function while viewing a screen implemented in the display module .

In a smart device using a conventional touch screen, a user may input a character by inputting a character by touching the keyboard with a user by displaying a keyboard together with a site or a messenger screen on which a character is to be input. In this case, since a small screen of the smart device needs to be divided, it is easy to generate an error due to a small keyboard screen, and it is difficult to operate the keyboard in a situation where foreign substances are present on the hand and the touch is difficult. In addition, since the device can be operated only by touching through a finger or a touch pen, frequent touches may deteriorate the liquid crystal damage or recognition of a specific portion. Thus, there is a need to implement a new input method that allows a user to enter text without touching the touch screen.

The conventional non-contact input interface recognizes a hand using a camera to recognize a gesture of a hand to perform a specific operation or to implement a virtual mouse or a virtual keyboard. However, in the case of such a conventional non-contact type input interface, expensive devices such as a kinetic device and a depth camera should be separately provided in order to recognize a hand. In addition, even in the case of a technique that implements a hand-based input interface using only a two-dimensional camera, most of the techniques for recognizing a hand based on skin color information are difficult to use the interface when the face and the hand are overlapped and recognized by the camera There is a problem.

The main object of the present invention is to provide a method and apparatus for providing a non-contact type input interface capable of recognizing a hand using only a two-dimensional camera provided in an existing smart device and inputting characters through a keyboard.

According to an embodiment of the present invention, there is provided a method of acquiring an image including a user's hand using a camera; Extracting and binarizing a hand candidate region which is an area judged to be a human skin color in the image; Extracting a contour of the binarized hand candidate region and detecting a hand contour line that matches the contour line; Determining a fingertip point from the hand contour; And determining a user's input operation using the fingertip. The present invention also provides a method for providing a non-contact type input interface.

The method for providing the non-contact type input interface according to the present embodiment may further include the following features.

The image may be a YUV data format including brightness information and color difference information.

In the process of extracting and binarizing the hand candidate region, a range similar to a human skin color can be extracted from the image into the hand candidate region by designating a range of U and V values.

The method comprising: setting the current pixel as a contour line when the difference between the U or V value of the current pixel and the next pixel of the image is equal to or greater than a predetermined value in the process of detecting the hand contour, If the difference between the V values is less than a predetermined value, the U, V, or Y values of the current pixel and the previous pixel are compared to set a contour line by applying a sobel mask to the previous pixel .

According to another embodiment of the present invention, there is provided a display device comprising: a display unit; An image acquisition unit for acquiring an image including a user's hand using a camera; An operation recognition unit for recognizing the hand from the image and determining an input operation of the user; And a control unit receiving the input operation of the user and generating character input control information and performing screen control, wherein the motion recognizing unit extracts a hand candidate region, which is an area judged as a human skin color in the image, Wherein the input operation of the user is determined using the finger end point by extracting a contour of the binarized hand candidate region, detecting a hand contour corresponding to a hand shape from the contour line to determine a finger end point, Thereby providing an input interface device.

According to the present embodiment, it is possible to provide a keyboard interface capable of correctly recognizing a finger operation of a user through a two-dimensional camera without using expensive equipment and inputting a character even if the user does not touch the touch screen .

In addition, even if the image recognized by the two-dimensional camera includes a skin color area other than a hand-like hand, the hand area included in the image can be accurately recognized.

1 is a block diagram briefly showing a basic configuration of a non-contact type input interface providing apparatus according to an embodiment of the present invention.
2 is a diagram illustrating a method for providing a non-contact subscription interface based on hand recognition according to an embodiment of the present invention.
3 is a flowchart illustrating a method for providing a non-contact type input interface based on a hand recognition according to an embodiment of the present invention.
4 is a diagram illustrating a binarized hand candidate region according to an embodiment of the present invention.
5 is a view illustrating an outline of a hand candidate region extracted according to an embodiment of the present invention.
6 is a diagram illustrating a process of detecting a hand contour according to an exemplary embodiment of the present invention.
FIG. 7 is a flowchart illustrating a hand outline extraction method according to an embodiment of the present invention.
8 is a diagram illustrating pixels forming data of an input image.
9 is a diagram illustrating a YUV data format of an input image.

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. It should be noted that, in adding reference numerals to the constituent elements of the drawings, the same constituent elements are denoted by the same reference symbols as possible even if they are shown in different drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

In describing the components of the present invention, terms such as first, second, A, B, (a), and (b) may be used. These terms are intended to distinguish the constituent elements from other constituent elements, and the terms do not limit the nature, order or order of the constituent elements. Throughout the specification, when an element is referred to as being "comprising" or "comprising", it means that it can include other elements as well, without excluding other elements unless specifically stated otherwise . In addition, '... Quot ;, " module ", and " module " refer to a unit that processes at least one function or operation, and may be implemented by hardware or software or a combination of hardware and software.

1 is a block diagram briefly showing a basic configuration of a non-contact type input interface providing apparatus according to an embodiment of the present invention.

1, the apparatus 100 for providing a non-contact type input interface according to the present embodiment can be applied to, for example, a mobile device such as a smart phone and a computer, and includes an image acquisition unit 110, A control unit 130 and a display unit 140. [ Also, the non-contact type input interface through hand recognition can be applied to a mobile device equipped with a touch screen having a conventional multi-touch function.

The image acquiring unit 110 includes a camera capable of acquiring an image including a hand to perform an input interface in a non-contact manner. The image acquiring unit 110 acquires an image from the camera and transmits the acquired image to the motion recognizing unit 120. For example, the image acquiring unit 110 may include a front camera of a smart phone so that a user can acquire a screen for moving a finger to input a character according to a keyboard arrangement displayed on the screen.

The motion recognition unit 120 analyzes the change of the shape, position, size, and speed of the hand based on the image acquired through the image acquisition unit 110, and determines the input operation of the user according to the change analysis. For example, a hand-shaped guide may be provided to a user in advance by a motion recognition unit 120 for recognizing a hand, and the motion recognition unit 120 may provide a hand-shaped guide to a user. When the outline extracted from the image matches a guide, . The process of recognizing the hand from the image and determining the input operation by the motion recognition unit 120 will be described in detail with reference to FIG. 3 to FIG. 9. FIG.

When receiving the input operation from the operation recognition unit 120, the control unit 130 controls the screen displayed on the display unit 140 so that the user can perform a desired screen operation according to the received input operation. That is, the control unit 130 recognizes the content of the input operation of the user and generates character input control information, and transmits a screen control signal to the display unit 140 according to the generated character input control information, Control the screen.

2, at least one of letters, numbers, symbols, graphics, and icons for inputting text through the display unit 140 is arranged in a predetermined array (not shown) And receives the input operation from the operation recognition unit 120, and performs non-contact text input through the keyboard screen.

For example, when a search input window of a web browser is selected or an application capable of text input such as a messenger is executed, the control unit 130 automatically overlays and displays a keyboard screen for inputting text on the screen together with the application, The controller 130 recognizes that the key of the corresponding keyboard is touched through the change of the shape, position, size, and speed of the hand such that the text corresponding to the key of the touched keyboard is displayed through the display unit 140 .

The display unit 140 displays various screens to be provided to the user and displays a controlled screen according to the screen control signal received from the control unit 130. [ For example, as shown in FIG. 2D, the display unit 140 displays a keyboard screen for text input when an application capable of text input is executed, Receives a control signal to which a text corresponding to a key of the key is input, and displays the corresponding text.

2 is a diagram illustrating a method for providing a non-contact subscription interface based on hand recognition according to an embodiment of the present invention.

First, a hand is recognized through camera shooting as shown in FIG. 2 (a). A front camera built into the smartphone can be used to recognize the hand. The background is deleted as shown in FIG. 2 (b) in order to recognize the hand from the image containing the hand. The finger end is recognized by applying an algorithm for recognizing the finger end from the image in which the background is deleted (FIG. 2 (c)), and the user's finger end point is positioned at a specific position on the keyboard of the screen, The text can be displayed and used as an input tool (Fig. 2 (d)).

3 is a flowchart illustrating a method for providing a non-contact type input interface based on a hand recognition according to an embodiment of the present invention.

4 is a diagram illustrating a binarized hand candidate region according to an embodiment of the present invention.

5 is a view illustrating an outline of a hand candidate region extracted according to an embodiment of the present invention.

6 is a diagram illustrating a process of detecting a hand contour according to an exemplary embodiment of the present invention.

Hereinafter, a method for recognizing and controlling a user's input operation for providing a hand-recognition-based non-contact type input interface will be described with reference to FIGS. 3 to 6. FIG.

An image including a hand is acquired from a camera provided in the apparatus (S310). At this time, it is possible to place an image of the hand-shaped guide on the screen and request the user to photograph the hand so that the hand can be positioned on the guide, thereby acquiring the image. For example, an image is acquired from a camera provided in a mobile device such as a smart phone. The camera can be a general two-dimensional camera whose depth estimation is impossible, and acquires data on a two-dimensional plane every frame. Each data is transmitted in the order of each pixel forming the image of each frame from the upper left to the lower right. At this time, the image is not a RGB color format but a YUV data format including brightness information and color difference information. In addition, the image of the RGB color format can be converted into the YUV format and used. Here, the Y value of the YUV data format data indicates the brightness information of the pixel, the U value indicates a blue color difference signal, and the V value indicates a red color difference signal.

In step S320, a hand candidate region is extracted from the acquired image and binarized. The hand candidate region refers to a region estimated to be skin color as a result of analyzing the color of an image. When a body portion other than a hand is included in the image, the corresponding portion also corresponds to the hand candidate region. A U value and a V value range for each pixel of the image are specified to extract a pixel estimated as a skin color from the image. The range of the U value and the V value can be designated as a value obtained through experiments. The extracted result is binarized so as to be able to distinguish the extracted hand candidate region from other regions. Referring to FIG. 4, when the acquired image includes both the hand and the face, it is difficult to correctly recognize the hand because the face and the hand overlap each other in the result of binarizing the extracted hand candidate region. Therefore, the following processes are additionally performed so that they can be distinguished and recognized.

Next, the contour of the binarized hand candidate region is extracted, and a hand contour corresponding to the hand shape is detected from the contour (S330). In step S330, the outline of the entire hand candidate region is extracted to recognize only the hand from the binarized hand candidate region. In order to extract the contour, the original image (YUV format image) is used, and the contour line is extracted only for the pixels of the hand candidate region in the original image, thereby reducing the amount of data operation compared to processing all the pixels. A detailed process of extracting contour lines for the hand candidate region will be described with reference to FIG. FIG. 5 shows the outline of the hand candidate region estimated from skin color in the image, and it can be recognized that the hand and face can be distinguished from each other.

When the contour of the hand candidate region is extracted, the portion corresponding to the hand-shaped guide is determined, and the matching portion is detected as the contour of the hand. The previously input guide is composed of a set of data composed of points or lines, which is compared with the contour of the extracted hand candidate region, and a portion with a high matching rate is determined as a hand contour. Referring to FIG. 6, red represents data for the guide. In the case of a binarized image, for example, a contour line has a value of 1 and a non-contour line has a value of 0. The portion of 0% of the guide data and the portion of 0% of the contour of the hand candidate region are calculated, and the portion indicating the highest matching ratio is determined as a hand, and a portion of the hand candidate region contour It can be judged as a hand contour line. Here, since the portion outside the contour line is not an area within the hand, it is possible to reduce the amount of data calculation by calculating the matching rate with the portion having a value of 0 within the contour line, excluding the contour line.

A finger end point is determined from the detected hand contour (S340). In the data of the hand shape guide, an index indicating a finger end point is displayed in a portion corresponding to a finger end point, and a coordinate of a hand outline coinciding with the position of the index is determined as a finger end point. For example, an index can be displayed in a manner that a portion corresponding to the fingertip in the guide data is displayed using a number other than 0 and 1.

The user's input is determined using the determined finger end point (S350). When the hand contour is detected and the fingertip point is determined, the contour and fingertip are continuously tracked along the hand shape when the hand moves in the image. The change of the finger curve is continuously tracked in the contour of the hand, and when it is judged that the finger is bent through the change, it is judged that there is an input operation in the corresponding part. For example, when a keyboard on which characters are arranged is displayed on the screen and the size of a specific finger curve is reduced by a predetermined value or more on a hand contour line, an operation of bending the finger occurs and an input is made to the character positioned at the end point of the finger .

In step S350, a process for determining whether a user's input operation has occurred is as follows.

First, the center of gravity is obtained by averaging the pixel coordinates of the detected hand contour.

Next, the distance between the calculated center of gravity and each finger end point is calculated. The distance from the thumb to the small finger to the end point of each of the five fingers is referred to as L1 to L5. In this case, when the finger is bent and the input operation is performed, the region estimated from the finger position to the skin color is reduced in the image. In the hand contour represented by binarization, the length of the finger is recognized as being shortened, .

It is possible to judge that the input operation of the user has occurred with respect to the point where the change of the distance according to the motion of the hand is greatest by judging the change of L1 to L5. For example, if the change of L1, L2, L4, and L5 is constant and the change of L3 is larger than these changes, it can be determined that the input operation has been performed on the character where the end point of the middle finger is located. When you move your fingers, your thumb and index finger can move without moving the other finger, but the other fingers are accompanied by the movement of the other finger reflexively. Therefore, it is determined that the input operation has occurred to the finger having the largest change in the distance from the finger with the greatest motion, that is, the finger to the finger end, by comparing the distances to all finger end points.

FIG. 7 is a flowchart illustrating a hand outline extraction method according to an embodiment of the present invention.

8 is a diagram illustrating pixels forming data of an input image.

9 is a diagram illustrating a YUV data format of an input image.

Hereinafter, with reference to Figs. 7 to 9, a method of detecting a contour line from a hand candidate region and detecting a contour line of a hand corresponding to the hand contour line from the detected contour line will be described in detail.

Referring to FIG. 8, one frame of the acquired image is transferred in order from the top left to the bottom right. In the figure, this order is indicated by a number increasing sequentially from 0. Such a frame is composed of a YUV format indicating brightness information and color difference information as shown in FIG. The YUV format has Y values of the corresponding pixel size, and the four pixels forming the adjacent squares share a U value and a V value to produce different colors in this combination.

First, it is determined whether the U value and the V value of the current pixel (nth pixel) satisfy the specified range in order to extract a portion estimated to be a skin color (S710). The U value and the V value are color information, which means the difference between the blue signal components and the red signal components, respectively, and a range similar to a human skin color can be derived by designating a range of the corresponding component. For example, if the byte array data received from the camera of the mobile device is '&' calculated at 0xff in JAVA and 105 <U <13, V> 130 can be specified to extract the skin color. The larger the V value (red signal), the closer the color to the red color. However, in limiting the U value, the data can be selected to some extent. Since human skin color is more redder than thought, Therefore, no upper limit is placed on the V value. Even when a red object exists as a background, the outline can be extracted through the following processing.

When the U value U _n and the V value V _n of the current pixel satisfy the range, the U value U _{n + 1} or the V value V _{n + 1} with the next pixel (n + 1 th pixel) And determines whether the difference is equal to or greater than a specific value du or dv (S720). Here, du and dv are values determined by experiments. If the difference (U _{n + 1} -U _n ) or the difference (V _{n + 1} -V _n ) between the U value of the current pixel and the next pixel is du or dv or more, It is determined that the object is changed and the n-th pixel is determined as the outline (S730).

If it is determined in step S720 that the difference between the U value and the V value is smaller than du and dv, the difference (U _n -U _n-1 ) between the current pixel and the U value of the previous pixel (n-1th pixel) (V _n -V _n-1 ) or the difference in Y value (Y _n -Y _n-1 ) with du, dv, and dy (S740). If the difference between the U value of the current pixel and the next pixel and the difference of the V value is less than du and dv, this case may be the case where the object has not changed or is not identifiable. At this time, it is determined whether the difference between the Y value (Y _n-1 ), the U value (U _n-1 ), or the V (V _n-1 ) value between the pixel and the immediately preceding pixel is more than du, dv or dy. Here, dy is a value determined by experiments.

If any one of the difference between the Y value, the U value and the V value of the current pixel and the previous pixel is more than du, dv, and dy, the Sobel mask is applied to the YUV value of the previous pixel (S750).

If the result of applying the Sobel mask is SY _n-1 , SU _n-1 , SV _n-1 and SY _n-1 , SU _n-1 or SV _n-1 is larger than the specified values sy, , the (n-1) th pixel is determined as a contour line (S770). Here, sy, su, and sv correspond to the values determined by the experiment.

The above process can be repeated for all the pixels to extract the contour line from the image.

When the U and V values are U = V = 128 or near, the brightness is very bright or very dark, that is, the part where the Y value has the minimum value or has the maximum value and can not be identified. In this case, the test is not performed until an identifiable pixel is found, and a check is made between the immediately preceding pixel and the presently identifiable pixel, which were not identifiable.

The foregoing description is merely illustrative of the technical idea of the present embodiment, and various modifications and changes may be made to those skilled in the art without departing from the essential characteristics of the embodiments. Therefore, the present embodiments are to be construed as illustrative rather than restrictive, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be construed according to the following claims, and all technical ideas within the scope of equivalents thereof should be construed as being included in the scope of the present invention.

As described above, the hand recognition-based non-contact input interface providing method and hand outline extraction method described in FIGS. 3 and 7 can be implemented by a program and recorded in a computer-readable recording medium. A program for implementing a similar mathematical problem retrieval method according to an embodiment of the present invention is recorded, and a computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like. The computer readable recording medium may also be distributed over a networked computer system so that computer readable code is stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present embodiment can be easily inferred by programmers in the technical field to which the present embodiment belongs.

100: Non-contact input interface providing device

Claims (8)

A process of obtaining an image including a user's hand using a camera;
Extracting and binarizing a hand candidate region which is an area judged to be a human skin color in the image;
Extracting a contour of the binarized hand candidate region and detecting a hand contour line that matches the contour line;
Determining a fingertip point from the hand contour; And
The process of determining the input operation of the user using the fingertip
The method comprising the steps of: The method according to claim 1,
Wherein the image comprises a YUV data format including brightness information and color difference information. 3. The method of claim 2,
Extracting the hand candidate region and binarizing the hand candidate region,
And a range of U and V values is specified to extract a portion similar to a human skin color from the image to the hand candidate region. 3. The method of claim 2,
The step of detecting the hand contour includes:
Setting the current pixel as a contour if the difference between the current pixel of the image and the U or V value of the next pixel is greater than a predetermined value; And
If the difference between the U or V value of the current pixel and the next pixel is equal to or less than a preset value, the U, V, or Y value of the current pixel is compared with the previous pixel, mask) to set the outline
The method comprising the steps of: The method according to claim 1,
Wherein the step of determining the fingertip point comprises:
Wherein an index indicating a fingertip portion is displayed in the hand shape and a coordinate of the hand contour line coinciding with the position of the index is determined as the finger end point The method according to claim 1,
Wherein the step of determining the input operation of the user comprises:
Calculating an average of pixel coordinates of the hand contour line to obtain a center of gravity;
Calculating a distance between the center of gravity and each finger end point; And
A step of determining that a user's input operation has occurred with respect to a point where a change in the distance between the center of gravity and the fingertip point is largest,
The method comprising the steps of: A display section;
An image acquisition unit for acquiring an image including a user's hand using a camera;
An operation recognition unit for recognizing the hand from the image and determining an input operation of the user; And
A control unit for receiving the input operation of the user and generating character input control information and performing screen control;
Lt; / RTI &gt;
Wherein the motion recognition unit comprises:
Extracting and binarizing a hand candidate region, which is an area judged by human skin color in the image, extracting a contour line of the binarized hand candidate region, determining a finger end point by detecting a hand contour line corresponding to the hand shape from the contour line, Wherein the input operation of the user is determined using the finger end point. A computer program for creating a scenario recorded on a computer-readable recording medium including computer program instructions executable by a processor, the computer program causing a computing device, when executed by a processor of the computing device,
A process of obtaining an image including a user's hand using a camera;
Extracting and binarizing a hand candidate region which is an area judged to be a human skin color in the image;
Extracting a contour of the binarized hand candidate region and detecting a hand contour line that matches the contour line;
Determining a fingertip point from the hand contour; And
The process of determining the input operation of the user using the fingertip
The computer program product comprising:

Images