JP6994206B2 - An electronic terminal device that recognizes a user's hand gesture and executes a control command and its operation method - Google Patents

Description

The present invention relates to an electronic terminal device that recognizes a user's hand gesture and executes a control command, and an operation method thereof.

Recently, as various electronic terminal devices equipped with touch screens have been released, various interface technologies for controlling such electronic terminal devices have appeared. The electronic terminal device equipped with the touch screen operates in a form of recognizing a user's touch input pattern input on the touch screen and executing a control command corresponding to the pattern.

Further, recently, with the advent of VR (Virtual Reality) devices and the like, research on interface technology that enables the device to be controlled only by a user's gesture without another controller is being actively researched.

Such an interface technology captures a pattern of a user's hand through a predetermined camera, and then issues an appropriate control command corresponding to the pattern of the user's hand in the captured image on the electronic terminal device. Is configured to apply to.

In this way, the interface technology that allows the user to control the device simply by creating a specific hand pattern is said to be highly useful in next-generation multimedia devices that do not use another controller, such as VR devices. In that respect, the demand for technology tends to increase gradually.

In the present invention, when the user takes a predetermined hand pattern, an image for the user's hand is acquired through the camera, joint points for the user's hand are extracted from the image, and gesture confirmation is performed for the user's hand area. Appropriate control command corresponding to the pattern of the user's hand by checking which joint point of each joint point belongs to the reference circle after generating a predetermined reference circle for An electronic terminal device capable of executing the above and an operation method thereof are presented.

The electronic terminal device that recognizes the user's hand gesture and executes the control command according to the embodiment of the present invention is preliminarily corresponding to a plurality of preset control commands different from each other and each of the plurality of control commands. Multiple n (n is a natural number of 2 or more) -dimensional gesture judgment reference vectors that are different from each other-the plurality of gesture judgment reference vectors are vectors having the codes of "0" and "1" as components. A stored gesture judgment reference vector storage unit, an image acquisition unit that captures a user's hand (Hand) through a camera and acquires an image including the user's hand, and a user's hand area from the image. A joint point extraction unit that detects and extracts n (n is a natural number of 2 or more) joint points from the user's hand area, and a virtual reference circle for gesture confirmation for the user's hand area. The reference circle generation unit that generates the By assigning a code of "1" to the located point and assigning a code of "0" to the point located outside the virtual reference circle, it was assigned to each of the n joint points. The similarity between each of the plurality of gesture judgment reference vectors stored in the gesture vector generation unit that generates the n-dimensional gesture vector having the code as a component and the gesture judgment reference vector storage unit and the gesture vector is determined. After the calculation, the first gesture determination is performed from the similarity confirmation unit for confirming the first gesture determination reference vector whose similarity is maximized among the plurality of gesture determination reference vectors, and the gesture determination reference vector storage unit. A control command execution unit that executes the first control command after confirming the first control command stored corresponding to the reference vector with the control command corresponding to the pattern of the user's hand is included.

Further, the operation method of the electronic terminal device that recognizes the user's hand gesture and executes the control command according to the embodiment of the present invention includes a plurality of preset control commands different from each other and each of the plurality of control commands. Multiple n (n is a natural number of 2 or more) dimensions that are preset to correspond to each other and differ from each other. Gesture judgment reference vector-The plurality of gesture judgment reference vectors have codes of "0" and "1" as components. The step of holding the gesture judgment reference vector storage unit in which the user's hand is stored, the step of photographing the user's hand through the camera and acquiring the image including the user's hand, and the step of acquiring the image including the user's hand from the image. Step to detect the hand area of the user and extract n (n is a natural number of 2 or more) joint points from the user's hand area, and generate a virtual reference circle for gesture confirmation for the user's hand area. Step, check whether the n joint points are located inside the virtual reference circle, and for the points located inside the virtual reference circle among the n joint points. By assigning the code of "1" and assigning the code of "0" to the points located outside the virtual reference circle, the code assigned to each of the n joint points is included as a component. A step of generating an n-dimensional gesture vector, after calculating the similarity between each of the plurality of gesture judgment reference vectors stored in the gesture judgment reference vector storage unit and the gesture vector, the plurality of gestures. A step of confirming the first gesture judgment reference vector in which the similarity is calculated to the maximum among the judgment reference vectors, and a first stored from the gesture judgment reference vector storage unit corresponding to the first gesture judgment reference vector. (1) The step includes a step of executing the first control command after confirming the control command with the control command corresponding to the pattern of the user's hand.

In the present invention, when the user takes a predetermined hand pattern, an image for the user's hand is acquired through the camera, joint points for the user's hand are extracted from the image, and gesture confirmation is performed for the user's hand area. Appropriate control command corresponding to the pattern of the user's hand by checking which joint point of each joint point belongs to the reference circle after generating a predetermined reference circle for It is possible to present an electronic terminal device capable of executing the above and an operation method thereof.

It is a figure which shows the structure of the electronic terminal apparatus which recognizes a user's hand gesture and executes a control command by one Embodiment of this invention. It is a figure for demonstrating the operation of the electronic terminal apparatus which recognizes a user's hand gesture and executes a control command by one Embodiment of this invention. It is a figure for demonstrating the operation of the electronic terminal apparatus which recognizes a user's hand gesture and executes a control command by one Embodiment of this invention. It is a figure for demonstrating the operation of the electronic terminal apparatus which recognizes a user's hand gesture and executes a control command by one Embodiment of this invention. It is a figure for demonstrating the operation of the electronic terminal apparatus which recognizes a user's hand gesture and executes a control command by one Embodiment of this invention. It is a flowchart which shows the operation method of the electronic terminal apparatus which recognizes a user's hand gesture and executes a control command by one Embodiment of this invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Such description is not intended to limit the invention to any particular embodiment and must be understood as including all modifications, equivalents or alternatives contained within the ideas and technical scope of the invention. It doesn't become. Similar reference numerals will be attached to similar components in the description of each drawing, and unless otherwise defined, all terms used herein, including technical or scientific terms, are the present invention. It has the same meaning as what is generally understood by those with ordinary knowledge in the technical field to which it belongs.

In this document, when a component is referred to as "contains" a component, this does not exclude other components unless otherwise stated, and may further include other components. do. Also, in various embodiments of the invention, each component, functional block or means may be composed of one or more subcomponents, the electrical, electronic and mechanical functions performed by each component. May be realized by various known elements or mechanical elements such as electronic circuits, integrated circuits, ASICs (Application Specific Integrated Circuits), etc., each of which is individually realized or two or more integrated into one. May be realized.

On the other hand, the blocks of the attached block diagram and the steps of the flowchart are installed in the processor and memory of a device capable of data processing such as a general-purpose computer, a special computer, a portable notebook computer, and a network computer, and perform specified functions. It can be interpreted as meaning the computer program instructions to be executed. Since these computer program instructions can be stored in the memory provided in the computer device or in the computer-readable memory, the functions described in the blocks of the block diagram or the steps of the flowchart are produced in the product including the instruction means to execute it. May be done. In addition, each block or step can represent part of a module, segment, or code that contains one or more executable instructions to perform a particular logical function. It should also be noted that in some alternative embodiments, the functions mentioned in the blocks or steps can be performed out of a defined order. For example, the two blocks or steps shown below may be executed substantially simultaneously or in reverse order, and in some cases may be executed with some blocks or steps omitted. You may.
FIG. 1 is a diagram showing a structure of an electronic terminal device that recognizes a user's hand gesture and executes a control command according to an embodiment of the present invention.

Referring to FIG. 1, the electronic terminal device 110 according to the present invention has a gesture determination reference vector storage unit 111, an image acquisition unit 112, a joint point extraction unit 113, a reference circle generation unit 114, a gesture vector generation unit 115, and a similarity degree. It includes a confirmation unit 116 and a control command execution unit 117.

In the gesture determination reference vector storage unit 111, a plurality of preset control commands different from each other and a plurality of different n (n is a natural number of 2 or more) preset to correspond to each of the plurality of control commands. ) Dimensional gesture judgment reference vector and is stored.
Here, the plurality of gesture determination reference vectors mean a vector having the codes of "0" and "1" as components.
In relation to this, when n is 21, information is stored in the gesture determination reference vector storage unit 111 as shown in Table 1 below.

イメージ取得部１１２は、カメラを介してユーザの手（Ｈａｎｄ）を撮影して前記ユーザの手が含まれたイメージを取得する。

The image acquisition unit 112 photographs the user's hand (Hand) through the camera and acquires an image including the user's hand.

The joint point extraction unit 113 detects the user's hand region from the image, and extracts n (n is a natural number of 2 or more) joint points from the user's hand region.

Here, the joint point extraction unit 113 can extract the n joint points from the user's hand region by utilizing CPM (Convolutional Pose Machines) technology.

CPM is a model in which one person is trained to learn a network that outputs a person's posture (Pose), and is a multi-state for understanding the relationship between human joints and refining the output. It is a model characterized by being configured in the network form of the above and evaluated (Evaluation). It has a network structure that outputs the Heatmap of each joint using CNN (Convolutional Neural Networks), and the result of Stage 1 is used again as the input of Stage 2 together with the image feature (Feature), and thus iteratively. The final result of processing the same Heatmap result over many stages is the output of the CPM network. In this way, the joints are searched for through a part of the image area in the first half of the Stage, and the relationship between the joints different from each other is taken into consideration due to the influence of the larger Receptive Field in the second half of the Stage, so that the heatmap result is more accurate. Is because it can be obtained.

At this time, the joint point extraction unit 113 utilizes such a CPM technique to obtain 21 joint points (1, 2, 3) which are the main joint points, as shown in FIG. 2, from the user's hand area. 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21) can be extracted.

At this time, according to one embodiment of the present invention, the joint point extraction unit 113 corresponds to each of the n joint points based on the positions of the n joint points in the user's hand region. You can assign an order.

In connection with this, as shown in FIG. 2, the joint point extraction unit 113 may assign the order from 1 to 21 one by one according to the position of the 21 joint points in the user's hand area. can.
The reference circle generation unit 114 generates a virtual reference circle for gesture confirmation for the user's hand area.

In connection with this, the reference circle generation unit 114 can generate a virtual reference circle 310 for gesture confirmation with respect to the user's hand region, as shown by reference numeral 310 in FIG.

The gesture vector generation unit 115 confirms whether or not the n joint points are located inside the virtual reference circle, and positions the n joint points inside the virtual reference circle among the n joint points. By assigning a code of "1" to the points to be used and a code of "0" to the points located outside the virtual reference circle, the codes assigned to each of the n joint points. Generates an n-dimensional gesture vector having.

At this time, according to one embodiment of the present invention, the gesture vector generation unit 115 vectorizes the code assigned to each of the n joint points according to the order corresponding to each of the n joint points. The gesture vector can be generated by designating each of the components of.

In connection with this, as shown in FIG. 3, when 21 joint points are extracted and the order from 1 to 21 is assigned, the gesture vector generation unit 115 is virtual among the 21 joint points. A code of "1" can be assigned to a point located inside the reference circle 310, and a code of "0" can be assigned to a point located outside the virtual reference circle 310.

Then, the gesture vector generation unit 115 sequentially designates the code assigned to each joint point as a component of the vector according to the order of the 21 joint points, thereby [1 1 1 0 1 1 1 1 1 1 1 1 It is possible to generate a 21-dimensional gesture vector of [1 1 0 0 0 1 0 0 0 1 0 0 0].

The similarity confirmation unit 116 calculates the similarity between each of the plurality of gesture determination reference vectors stored in the gesture determination reference vector storage unit 111 and the gesture vector, and then the plurality of gesture determination reference vectors. Of these, the first gesture determination reference vector in which the similarity is calculated to the maximum is confirmed.

At this time, according to one embodiment of the present invention, the similarity confirmation unit 116 calculates a Hadamard product between each of the plurality of gesture determination reference vectors and the gesture vector, and the Hadamard product. The number of existing code values of "1" in the components constituting the vector calculated by the above calculation can be calculated as the similarity.

Here, the Hadamard product means an operation of multiplying each component in a vector or matrix of the same magnitude, and when the vectors [ab c] and [x yz] exist, the Hadamard product between the two vectors is. It is calculated in [ax by cz].

In connection with this, as in the above-mentioned example, the gesture vector generation unit 115 generates a 21-dimensional gesture vector [1 1 1 0 1 1 1 1 1 1 1 0 0 0 1 0 0 0 1 0 0 0]. Assuming that the information is stored in the gesture judgment reference vector storage unit 111 as shown in Table 1, the similarity confirmation unit 116 determines the first gesture stored in the gesture judgment reference vector storage unit 111. The reference vectors [1 0 1 1 1 1 0 0 0 0 0 0 1 0 1 0 0 0 0 1 1 1] and [1 1 1 0 1 1 1 1 1 1 1 0 0 0 1 0 0 0 1 0 0 0 0 ] And the Adamal product is calculated to calculate the vector [1 0 1 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0], and then the calculated vector is constructed. The gesture judgment reference vector [1 0 1 1 1 1 1 0 0 0 0 0 0 1 0 1 0 0 0 0 1 1 1] and [1], which is the number of existing code values of “1”, are set to It can be calculated as the degree of similarity with the gesture vector of 1 1 0 1 1 1 1 1 1 1 0 0 0 1 0 0 0 1 0 0 0].

In this way, the similarity confirmation unit 116 can calculate all the similarity between the gesture determination reference vector stored in the gesture determination reference vector storage unit 111 and the gesture vector.

After that, the similarity confirmation unit 116 confirms the gesture determination reference vector in which the similarity is calculated to the maximum among the gesture determination reference vectors stored in the gesture determination reference vector storage unit 111 in the first gesture determination reference vector. can do.

The control command execution unit 117 confirms the first control command stored corresponding to the first gesture judgment reference vector from the gesture judgment reference vector storage unit 111 to the control command corresponding to the pattern of the user's hand. , The first control command is executed.

In relation to this, when the first gesture judgment reference vector is [1 1 1 1 1 1 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1], the control command execution unit 117 is shown in Table 1. "Control command 3" stored corresponding to the gesture judgment reference vector [1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1] from the gesture judgment reference vector storage unit 111 such as Is confirmed by the control command corresponding to the pattern of the user's hand, and then the "control command 3" can be executed.
At this time, according to one embodiment of the present invention, the control command execution unit 117 can include the control command selection message display unit 120 and the selection control command execution unit 121.

When the control command selection message display unit 120 confirms in the similarity confirmation unit 116 that two or more gesture determination reference vectors having the maximum similarity calculated among the plurality of gesture determination reference vectors exist, the gesture determination criterion After confirming the two or more control commands stored corresponding to the two or more gesture judgment reference vectors from the vector storage unit 111, it is possible to select one of the two or more control commands. Display the selection message to be instructed on the screen.

When the second control command is selected from the two or more control commands by the user, the selection control command execution unit 121 confirms the second control command with the control command corresponding to the pattern of the user's hand. , The second control command is executed.

For example, as a result of confirming the gesture judgment reference vector in which the similarity is calculated to the maximum by the similarity confirmation unit 116, [1 0 1 1 1 1 1 0 0 0 0 0 0 1 0 1 0 0 0 0 1 1 1] and [ When the similarity to the gesture judgment reference vector of 1 1 1 1 1 1 1 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1] is calculated to the maximum, the control command selection message display unit 120 is shown in Table 1. After confirming "control command 1" and "control command 3" from the gesture judgment reference vector storage unit 111 such as, select one of "control command 1" and "control command 3". A selection message instructing is displayed on the screen.

At this time, if the user sees the selection message and selects "control command 3", the selection control command execution unit 121 converts the "control command 3" into a control command corresponding to the pattern of the user's hand. After confirmation, "control instruction 3" can be executed.

As a result, the electronic terminal device 110 according to the present invention acquires an image of the user's hand through the camera when the user takes a predetermined hand pattern, and extracts joint points for the user's hand from the image. After generating a predetermined reference circle for gesture confirmation for the user's hand area, the user's hand is confirmed by checking which joint point of each joint point belongs to the reference circle. Appropriate control commands corresponding to the pattern of can be executed. Thereby, the user can easily control the electronic terminal device 110 simply by taking a predetermined gesture by using a hand without another controller.

It is said that the electronic terminal device 110 according to the present invention adopts a method of determining a gesture corresponding to a pattern of a user's hand by confirming which joint point is included or not included in the reference circle. In terms of points, it is necessary to appropriately specify the size and position of the reference circle according to the size of the image taken by the user's hand.

In connection with this, according to one embodiment of the present invention, the reference circle generation unit 114 is configured to appropriately specify the size and position of the reference circle, and may include the generation unit 118 and the arrangement unit 119. can.

The generation unit 118 generates a virtual square area surrounding the user's hand area, compares the virtual square area with a reference square area having a preset pattern and size, and compares the reference square area with each other. After confirming the horizontal axis size ratio and the vertical axis size ratio of the virtual square area to the area, the horizontal axis size ratio is calculated with respect to the horizontal axis diameter of the reference circle having a preset pattern and size. The virtual reference circle is generated by multiplying the diameter of the vertical axis of the reference circle by the size ratio of the vertical axis to change the pattern and size of the reference circle.

Then, the arrangement unit 119 confirms the first point corresponding to the first relative ratio set in advance with respect to the left side point on the horizontal axis of the virtual square area, and on the vertical axis of the virtual square area, the arrangement unit 119 confirms. After confirming the second point corresponding to the second relative ratio set in advance with respect to the upper end point, the intersection of the vertical line passing through the first point and the horizontal line passing through the second point is the virtual reference. By setting it at the center point of the circle, the virtual reference circle is arranged on the virtual square area.
In relation to this, the operations of the generation unit 118 and the arrangement unit 119 will be described with reference to FIGS. 3 to 5 as follows.
As shown in FIG. 3, the generation unit 118 can generate a virtual square area 320 that surrounds the user's hand area in the image.

Then, as shown in the drawing reference numeral 410 of FIG. 4, the generation unit 118 compares the virtual square region 320 with the reference square region 411 having a preset pattern and size, and compares the reference square region 411 with each other. It is possible to confirm the horizontal axis size ratio and the vertical axis size ratio of the virtual square area 410 with respect to.

After that, as shown in the drawing reference numeral 420 of FIG. 4, the generation unit 118 multiplies the horizontal axis diameter of the reference circle 421 having a preset pattern and size by the horizontal axis size ratio, and the reference circle 421. By multiplying the vertical axis diameter of the vertical axis by the vertical axis size ratio to change the pattern and size of the reference circle 421, a virtual reference circle 310 can be generated.

For example, when the horizontal axis size ratio is 1.3 and the vertical axis size ratio is 1.8, the generation unit 118 multiplies the horizontal axis diameter of the reference circle 421 by 1.3 and the vertical axis. By multiplying the diameter by 1.8, a virtual reference circle 310 can be generated.

When the virtual reference circle 310 is generated in this way, as shown in FIG. 5, the arrangement unit 119 has a first relative ratio preset with respect to the left side point on the horizontal axis of the virtual square region 320. The first point 510 corresponding to the above can be confirmed, and the second point 520 corresponding to the second relative ratio set in advance with respect to the upper end point can be confirmed on the vertical axis of the virtual square area 320.

For example, when the first relative ratio is 3/5 and the second relative ratio is 4/7, the arrangement unit 119 corresponds to 3/5 with respect to the left side point on the horizontal axis of the virtual square area 320. The point corresponding to 4/7 can be confirmed at the second point 520 on the vertical axis of the virtual square area 320 with respect to the upper end point.

After that, as shown in FIG. 5, the arrangement unit 119 sets the intersection 530 where the vertical line passing through the first point 510 and the horizontal line passing through the second point 520 intersect as the center point of the virtual reference circle 310. , The virtual reference circle 310 can be placed on the virtual square area 320. As a result, the virtual reference circle 310 is arranged at an appropriate position on the user's hand area with an appropriate size and pattern, as shown in FIG.
FIG. 6 is a flowchart showing an operation method of an electronic terminal device that recognizes a user's hand gesture and executes a control command according to an embodiment of the present invention.

In step (S610), a plurality of preset control commands different from each other and a plurality of different n (n is a natural number of 2 or more) dimensional gestures preset to correspond to each of the plurality of control commands. It holds a gesture judgment reference vector storage unit in which a judgment reference vector (the plurality of gesture judgment reference vectors are vectors having codes of "0" and "1" as components) is stored.
In step (S620), the user's hand is photographed through the camera to acquire an image including the user's hand.
In step (S630), the user's hand region is detected from the image, and n (n is a natural number of 2 or more) joint points are extracted from the user's hand region.
In step (S640), a virtual reference circle for gesture confirmation is generated for the user's hand area.

In step (S650), it is confirmed whether or not the n joint points are located inside the virtual reference circle, and the n joint points are located inside the virtual reference circle among the n joint points. By assigning a code of "1" to a point and a code of "0" to a point located outside the virtual reference circle, the code assigned to each of the n joint points is assigned. Generates an n-dimensional gesture vector having as a component.

In step (S660), after calculating the similarity between each of the plurality of gesture determination reference vectors stored in the gesture determination reference vector storage unit and the gesture vector, the plurality of gesture determination reference vectors of the plurality of gesture determination reference vectors are calculated. Among them, the first gesture judgment reference vector in which the similarity is calculated to the maximum is confirmed.

In step (S670), after confirming the first control command stored corresponding to the first gesture judgment reference vector from the gesture judgment reference vector storage unit to the control command corresponding to the pattern of the user's hand, The first control command is executed.

At this time, according to one embodiment of the present invention, in step (S630), the order corresponding to each of the n joint points is based on the position of the n joint points in the user's hand region. Can be assigned.

At this time, in step (S650), the code assigned to each of the n joint points is specified for each of the vector components according to the order corresponding to each of the n joint points. The gesture vector can be generated.

Further, according to one embodiment of the present invention, in step (S660), the Hadamard product between each of the plurality of gesture determination reference vectors and the gesture vector is calculated, and the calculation is performed by the calculation of the Hadamard product. The number of existing code values of "1" in the components constituting the vector can be calculated as the similarity.

Further, according to one embodiment of the present invention, in step (S640), a virtual square area surrounding the user's hand area is generated, and the virtual square area has a preset pattern and size. After comparing the reference square area with each other and confirming the horizontal axis size ratio and the vertical axis size ratio of the virtual square area with respect to the reference square area, the reference circle having a preset pattern and size By multiplying the horizontal axis diameter by the horizontal axis size ratio and multiplying the vertical axis diameter of the reference circle by the vertical axis size ratio to change the pattern and size of the reference circle, the virtual In the step of generating the reference circle of, and in the horizontal axis of the virtual square area, the first point corresponding to the first relative ratio set in advance with respect to the left side point is confirmed, and the vertical axis of the virtual square area is confirmed. In, after confirming the second point corresponding to the second ratio set in advance with respect to the upper end point, the intersection of the vertical line passing through the first point and the horizontal line passing through the second point is the virtual intersection. By setting the center point of the reference circle, the step of arranging the virtual reference circle on the virtual square region can be included.

Further, according to one embodiment of the present invention, in the operation method of the electronic terminal device, the gesture determination reference vector in which the similarity is calculated to the maximum is 2 among the plurality of gesture determination reference vectors in step (S660). When it is confirmed that the above exists, after confirming two or more control commands stored corresponding to the two or more gesture judgment reference vectors from the gesture judgment reference vector storage unit, among the two or more control commands. A step of displaying a selection message on the screen instructing to select any one of the control commands, and when the second control command is selected from the two or more control commands by the user, the second control command. Can be further included in the step of executing the second control command after confirming with the control command corresponding to the pattern of the user's hand.

As described above, the operation method of the electronic terminal device that recognizes the user's hand gesture and executes the control command according to the embodiment of the present invention has been described with reference to FIG. Here, in the operation method of the electronic terminal device that recognizes the user's hand gesture and executes the control command according to the embodiment of the present invention, the user's hand gesture described with reference to FIG. 1 is recognized and the control command is executed. In order to correspond to the configuration relating to the operation of the electronic terminal device 110, a more detailed description thereof will be omitted.

Even if the operation method of the electronic terminal device that recognizes a user's hand gesture and executes a control command according to an embodiment of the present invention is realized by a computer program stored in a storage medium for execution by coupling with a computer. good.

Further, the operation method of the electronic terminal device that recognizes a user's hand gesture and executes a control command according to an embodiment of the present invention is realized in a program instruction form that can be executed via various computer means and can be read by a computer. It may be recorded on a medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be those specially designed and configured for the present invention, or those known and usable by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and floptic discs. Includes magnetic-optical media such as disk) and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory and the like. Examples of program instructions include machine language code, such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

As described above, the present invention has been described by means of specific matters such as specific components and limited embodiments and drawings, which are provided to aid in a more general understanding of the invention. However, the present invention is not limited to the above-described embodiment, and any person who has ordinary knowledge in the field to which the present invention belongs can make various modifications and modifications from such a description.

Therefore, the idea of the present invention should not be limited to the described embodiments, and is not limited to the scope of claims described later, but anything having a modification equal to or equivalent to the scope of claims. Can be said to belong to the category of the idea of the present invention.

110 ・ ・ ・ Electronic terminal device that recognizes the user's hand gesture and executes control commands 111 ・ ・ ・ Gesture judgment reference vector storage unit 112 ・ ・ ・ Image acquisition unit 113 ・ ・ ・ Joint point extraction unit 114 ・ ・ ・ Reference Circle generation unit 115 ・ ・ ・ Gesture vector generation unit 116 ・ ・ ・ Similarity confirmation unit 117 ・ ・ ・ Control command execution unit 118 ・ ・ ・ Generation unit 119 ・ ・ ・ Arrangement unit 120 ・ ・ ・ Control command selection message display unit 121・ ・ ・ Selection control command execution unit

Claims (12)

A plurality of preset control commands different from each other and a plurality of different n (n is a natural number of 2 or more) dimensions gesture determination presets corresponding to each of the plurality of control commands-the plurality. Gesture judgment reference vector of is a vector having a code of "0" and "1" as components-and a gesture judgment reference vector storage unit, which stores
An image acquisition unit that captures a user's hand (Hand) through a camera and acquires an image including the user's hand.
A joint point extraction unit that detects the user's hand area from the image and extracts n (n is a natural number of 2 or more) joint points from the user's hand area.
A reference circle generator that generates a virtual reference circle for gesture confirmation for the user's hand area,
It is confirmed whether or not the n joint points are located inside the virtual reference circle, and among the n joint points, "1" is given to the point located inside the virtual reference circle. By assigning the code of "0" to the point located outside the virtual reference circle, the n-dimensional code having the code assigned to each of the n joint points is used as a component. Gesture vector generator, which generates gesture vectors,
After calculating the similarity between each of the plurality of gesture determination reference vectors stored in the gesture determination criterion vector storage unit and the gesture vector, the similarity is the maximum among the plurality of gesture determination criterion vectors. A similarity confirmation unit for confirming the first gesture determination reference vector calculated in the above, and a first control command stored from the gesture determination reference vector storage unit corresponding to the first gesture determination reference vector are transmitted by the user. An electronic terminal device that recognizes a user's hand gesture and executes a control command, including a control command execution unit that executes the first control command after confirming the control command corresponding to the pattern of the hand. The joint point extraction unit is
Based on the position of the n joint points in the user's hand area, the order corresponding to each of the n joint points is assigned.
The gesture vector generation unit is
Claimed to generate the gesture vector by designating one code assigned to each of the n joint points as a component of the vector according to the order corresponding to each of the n joint points. An electronic terminal device that recognizes the user's hand gesture according to 1 and executes a control command. The similarity confirmation unit is
A code value of "1" exists in a component that calculates a Hadamard product between each of the plurality of gesture determination reference vectors and the gesture vector and constitutes a vector calculated by the calculation of the Hadamard product. An electronic terminal device that recognizes the user's hand gesture according to claim 1 and executes a control command by calculating the number of vectors to be performed as the similarity. The reference circle generator is
A virtual square area surrounding the user's hand area is generated, the virtual square area and a reference square area having a preset pattern and size are compared with each other, and the virtual square area with respect to the reference square area is compared with each other. After confirming the horizontal axis size ratio and the vertical axis size ratio of the square area, multiply the horizontal axis diameter of the reference circle having a preset pattern and size by the horizontal axis size ratio, and then multiply the reference circle. By multiplying the vertical axis diameter of the vertical axis by the vertical axis size ratio to change the pattern and size of the reference circle, the generation unit that generates the virtual reference circle, and the horizontal axis of the virtual square region. In, the first point corresponding to the first relative ratio set in advance with respect to the left side point is confirmed, and in the vertical axis of the virtual square area, it corresponds to the second relative ratio preset with reference to the upper end point. After confirming the second point, the intersection of the vertical line passing through the first point and the horizontal line passing through the second point is set as the center point of the virtual reference circle, thereby setting the virtual reference circle. An electronic terminal device for recognizing a user's hand gesture according to claim 1 and executing a control command, including an arrangement unit for arranging the above-mentioned virtual square area. The control command execution unit is
When it is confirmed in the similarity confirmation unit that there are two or more gesture judgment reference vectors having the maximum similarity calculated among the plurality of gesture judgment reference vectors, the gesture judgment reference vector storage unit indicates that the two or more gesture judgment reference vectors are present. After confirming two or more control commands stored corresponding to the gesture judgment reference vector, a selection message instructing to select one of the two or more control commands is displayed on the screen. When the second control command is selected from the two or more control commands by the control command selection message display unit and the user, the second control command is confirmed by the control command corresponding to the pattern of the user's hand. An electronic terminal device that recognizes the user's hand gesture according to claim 1 and executes a control command, including a selection control command execution unit that executes the second control command. A plurality of preset control commands different from each other and a plurality of different n (n is a natural number of 2 or more) dimensions gesture determination presets corresponding to each of the plurality of control commands-the plurality. The gesture judgment reference vector of is a vector having the code of "0" and "1" as components-the step of holding the gesture judgment reference vector storage unit in which the code is stored.
A step of photographing a user's hand (Hand) through a camera and acquiring an image including the user's hand.
A step of detecting the user's hand region from the image and extracting n (n is a natural number of 2 or more) joint points from the user's hand region.
A step of generating a virtual reference circle for gesture confirmation for the user's hand area,
It is confirmed whether or not the n joint points are located inside the virtual reference circle, and among the n joint points, "1" is given to the point located inside the virtual reference circle. By assigning the code of "0" to the point located outside the virtual reference circle, the n-dimensional code having the code assigned to each of the n joint points is used as a component. Steps to generate gesture vectors,
After calculating the similarity between each of the plurality of gesture determination reference vectors stored in the gesture determination criterion vector storage unit and the gesture vector, the similarity is the maximum among the plurality of gesture determination criterion vectors. The step of confirming the first gesture judgment reference vector calculated in the above, and the first control command stored corresponding to the first gesture judgment reference vector from the gesture judgment reference vector storage unit are the pattern of the user's hand. A method of operating an electronic terminal device that recognizes a user's hand gesture and executes a control command, including a step of executing the first control command after confirming with the control command corresponding to the above. The extraction step is
Based on the position of the n joint points in the user's hand area, the order corresponding to each of the n joint points is assigned.
The step of generating the gesture vector is
Claimed to generate the gesture vector by designating one code assigned to each of the n joint points as a component of the vector according to the order corresponding to each of the n joint points. 6. An operation method of an electronic terminal device that recognizes a user's hand gesture and executes a control command according to 6. The step to confirm is
A code value of "1" exists in a component that calculates a Hadamard product between each of the plurality of gesture determination reference vectors and the gesture vector and constitutes a vector calculated by the calculation of the Hadamard product. The operation method of an electronic terminal apparatus which recognizes a user's hand gesture according to claim 6 and executes a control command, which calculates the number of vectors to be performed as the similarity. The step of generating the reference circle is
A virtual square area surrounding the user's hand area is generated, the virtual square area and a reference square area having a preset pattern and size are compared with each other, and the virtual square area with respect to the reference square area is compared with each other. After confirming the horizontal axis size ratio and the vertical axis size ratio of the square area, multiply the horizontal axis diameter of the reference circle having a preset pattern and size by the horizontal axis size ratio, and then multiply the reference circle. In the step of generating the virtual reference circle by multiplying the vertical axis diameter by the vertical axis size ratio to change the pattern and size of the reference circle, and on the horizontal axis of the virtual square region. , Confirm the first point corresponding to the first relative ratio set in advance based on the left side point, and correspond to the second relative ratio preset in reference to the upper end point on the vertical axis of the virtual square area. After confirming the second point, the virtual reference circle is set by setting the intersection of the vertical line passing through the first point and the horizontal line passing through the second point as the center point of the virtual reference circle. The operation method of the electronic terminal apparatus which recognizes a user's hand gesture according to claim 6 and executes a control command, including the step of arranging on the virtual square area. When it is confirmed that there are two or more gesture judgment reference vectors having the maximum similarity calculated among the plurality of gesture judgment reference vectors in the confirmation step, the two or more gestures are performed from the gesture judgment reference vector storage unit. After confirming two or more control commands stored corresponding to the judgment reference vector, a selection message instructing to select one of the two or more control commands is displayed on the screen. When the step and the second control command among the two or more control commands are selected by the user, the second control command is confirmed to the control command corresponding to the pattern of the user's hand, and then the second control is performed. The method of operating an electronic terminal device for recognizing a user's hand gesture and executing a control command according to claim 6, further comprising a step of executing the command. A computer-readable recording medium on which a computer program for executing the method according to any one of claims 6 to 10 by coupling with a computer is recorded. A computer program stored in a storage medium for executing the method according to any one of claims 6 to 10 by coupling with a computer.

Images