TWI611340B - Method for determining non-contact gesture and device for the same - Google Patents

Abstract

The invention relates to a non-contact gesture judging method and device thereof, which has an image sensor for detecting image sensing information, and uses an inertial sensor to detect an inertial sensing signal, and then the processor determines whether to detect Detecting object information and whether there is an inertial event, thereby eliminating image sensing information generated in the same and/or adjacent image frame as the inertial event, so that the image sensing information used to determine the gesture is not Infrared events affect the image sensing information to improve the correctness of the gestures judged.

Description

Non-contact gesture judgment method and device thereof

The invention relates to a non-contact gesture determination method and device thereof, and relates to a method and device for judging a non-contact gesture through an optical image sensing method.

With the advancement of technology, the input method of electronic products has also improved, from the traditional physical key input, to the virtual touch panel input, and recently developed a non-contact input mode, the user does not need to touch the electronic device. The electronic device detects the gesture of the user operating in the air through the non-contact sensor, and recognizes and executes the corresponding instruction. Especially for devices with Augmented Reality (AR) functionality, the use of contactless inputs makes the input more intuitive and more convenient to use. Augmented reality with electronic devices (especially wearable electronic devices) can be derived from a variety of different applications, such as games, teleconferencing, navigation maps, and the like.

The non-contact input method is used to capture information such as the moving distance, speed, angle, and the like of the palm or finger swiped by the user in the air through the image sensor, and determine the corresponding gesture to trigger the corresponding instruction. However, the electronic device is worn by the user or held by the user, and is easily moved by the user's own unintentional motion, so that relative movement between the electronic device and the non-moving finger or the palm may occur. The sensor will therefore be judged as the user's palm Or the movement of the finger, and the gesture of misjudgment, and then trigger the instructions that the user does not want to do, thus causing great trouble to the use.

In view of this, the present invention is directed to the case of misjudging a contactless gesture to solve the problems in the prior art.

In order to achieve the above object, the technical means adopted by the present invention is to create a non-contact gesture determination method, which comprises: a. detecting an image sensing information within a sensing range of a non-contact sensing device, And detecting an inertial sensing signal of the non-contact sensing device; b. determining, according to the inertial sensing signal, whether an inertial event occurs in the non-contact sensing device, and determining whether to detect according to the image sensing information Go to an object information; c. When it is determined that the inertia event has not occurred and the object information is detected, the gesture is judged according to at least one piece of the image sensing information and the corresponding gesture is output.

Further, in conjunction with the foregoing non-contact gesture determination method, the present invention creates a non-contact sensing device, which includes: a processor; an inertial sensor connected to the processor; at least one image sense And a processor coupled to the processor, the processor performing the steps in the non-contact gesture determination method of the present invention.

The invention has the advantages that the gesture of the inertial sensor is matched and the occurrence of the inertia event is judged to make the output gesture not be affected by the inertia event, thereby avoiding misjudgment gestures and false touch commands.

The present invention further employs a technical means for creating a non-contact gesture determination method, which is applied to a portable electronic device, and the portable electronic device includes a gesture detection unit for detecting a user's gesture operation, and a first sensor for detecting motion of the portable electronic device, the method comprising: a. determining that the portable electronic device is moving according to an output of the first sensor; and b. after step a, The interrupt triggers the gesture operation of the user.

10‧‧‧ processor

11‧‧‧ storage unit

20‧‧‧Inertial Sensor

30‧‧‧Image Sensor

40‧‧‧ storage unit

BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a block diagram of a first embodiment of the apparatus of the present invention.

Figure 2 is a block diagram of a second embodiment of the apparatus of the present invention.

Figure 3 is a flow chart of the method of the present invention.

4 is a flow chart of a first embodiment of a method of the present invention.

Figure 5 is a flow chart of a second embodiment of the method of the present invention.

Figure 6 is a flow chart of a third embodiment of the method of the present invention.

Figure 7 is a flow chart of a fourth embodiment of the method of the present invention.

Figure 8 is a timing diagram of an embodiment of the present invention.

Figure 9 is a flow chart of another method of the present invention.

The technical means adopted by the present invention for achieving the intended purpose of the invention are further explained below in conjunction with the drawings and the embodiments of the present invention.

As shown in FIG. 1 , the non-contact sensing device of the present invention comprises a processor 10 , an inertial sensor unit (IMU) 20 and at least one image sensor 30 .

In one embodiment (as shown in FIG. 1), a memory unit 11 is built into the processor 10; in another embodiment (as shown in FIG. 2), the processor 10 is coupled to an external storage unit 40. connection.

The inertial sensor 20 is coupled to the processor 10, and the inertial sensor 20 transmits the detected inertial sensing signal S _I to the processor 10. The inertial sensor 20 can be various detectable A sensor that measures the motion of an object. In one embodiment, the inertial sensor 20 is an accelerometer (G-sensor, Accelerometer); in one embodiment, the inertial sensor 20 is a gyroscope; in an embodiment, The inertial sensor 20 is a magnetometer.

The image sensor 30 transmits the detected non-contact image sensing information S _G to the processor 10 . In one embodiment, the non-contact sensing device of the present invention has a plurality of image sensors 30 for capturing image sensing information at different angles and positions, thereby improving image accuracy or for other image processing applications. . In an embodiment, the image sensor 30 is an optical sensor.

Referring to FIG. 3, the non-contact gesture judging method of the present invention comprises the following steps:

Detecting the image sensing information S _G and an inertial measurement signal S _I (S1): corresponding to the inertial sensor 20 detects through to the non-contact sensing means worn on the user of the inertia sensing signal S _I And detecting, by the image sensor 30, the image sensing information S _G located in the sensing range thereof. In an embodiment, the step S1 detects the inertial sensing signal S _I and the image sensing information S _G in an image frame.

Determining whether an object information is detected according to the image sensing information S _G, determining whether an inertia event occurs according to the inertial sensing signal S _I (S2): the processor 10 determines whether an inertia event occurs according to the inertial sensing signal S _I And determining whether an object information is detected according to the image sensing information S _G . If it is determined that the inertia event has not occurred and the object information is detected, the process proceeds to step S3, and if any of the conditions does not match, the process returns to step S1. In an embodiment, it is determined whether the inertial event has not occurred and the object information is detected in the same image frame. Determining whether an inertial event occurs may be achieved by a plurality of determining methods. In an embodiment, after the sensing amount of the inertial sensing signal S _I is compared with a threshold value, when the sensing amount of the inertial sensing signal S _I is greater than the threshold When the value is determined, it is determined that the inertia event has occurred. Determining whether the object information is detected may also be achieved by a plurality of judging methods. In an embodiment, the image sensing information S _{G is} compared with a preset object specific feature (such as a fingertip, a palm, etc.) to determine whether it is One of the default objects, if it is, is judged to have object information.

When it is determined that the inertia event has not occurred and the object information is detected, the gesture is judged and the corresponding gesture is output (S3): when the processor 10 determines that neither the inertia event nor the object information is detected, then At least one piece of the image sensing information performs a gesture determination and outputs a corresponding gesture.

In an embodiment, referring to FIG. 4, step S2 includes steps S21A and S22A: the processor 10 determines whether an inertia event occurs according to the inertial sensing signal S _I (S21A), and if so, returns to the step. The S1A continues to detect the image sensing information S _G and the inertial sensing signal S _I . If not, the process proceeds to step S22A. The processor 10 then determines whether an object information is detected according to the image sensing information S _G ( S22A), if no, return to step S1A, and if yes, proceed to step S30A. Step S3 includes steps S30A, S31A, and S32A: the processor 10 marks the image frame determined to be YES in step S22A in the storage unit 11, 40 (S30A); then the processor 10 follows the marked image map. The image sensing information in the frame determines whether a gesture is satisfied (S31A), and if so, outputs a corresponding gesture (S32A), and if not, returns to step S1A.

In another embodiment, referring to FIG. 5, step S2 includes steps S21B, S20B, and S22B: the processor 10 determines, according to the inertial sensing signal S _I , whether an inertia event occurs (S21B), and if so, Marking the image frame in which the inertia event occurs is recorded in the storage unit 11, 40 (S20B), and then returning to step S1B to continue detecting the image sensing information S _G and the inertial sensing signal S _I , if Otherwise, the process proceeds to step S22B. The processor 10 then determines whether an object information is detected based on the image sensing information S _G (S22B). If not, the process returns to step S1B, and if yes, the process proceeds to step S31B. Step S3 includes steps S31B and S32B: the processor 10 determines whether a gesture is satisfied according to the image sensing information in the unmarked image frame (S31B), and if so, outputs a corresponding gesture (S32B), if not Then, it returns to step S1B.

In an embodiment, referring to FIG. 6 , step S2 includes steps S21C and S22C: the processor 10 determines, according to the image sensing information S _G , whether an object information is detected (S21C), and if not, Returning to step S1C, the image sensing information S _G and the inertial sensing signal S _{I are} further detected. If yes, the process proceeds to step S22C. The processor 10 then determines whether an inertia event occurs according to the inertial sensing signal S _I . (S22C), if yes, return to step S1C, and if no, proceed to step S30C. Step S3 includes steps S30C, S31C, and S32C: the processor 10 marks and records the image frame in the step S22C as NO in the storage unit 11, 40 (S30C); then the processor 10 is based on the marked The image sensing information in the image frame determines whether a gesture is satisfied (S31C), and if so, outputs a corresponding gesture (S32C), and if not, returns to step S1C.

In another embodiment, referring to FIG. 7, step S2 includes steps S21D, S20D, and S22D: the processor 10 determines, according to the image sensing information S _G , whether an object information (S21D) is detected. Then, the process proceeds to step S22D. If no, the process continues to detect the image sensing information S _G and the inertial sensing signal S _I in step S1D. The processor 10 then determines whether the occurrence occurs according to the inertial sensing signal S _I . An inertia event (S22D), if yes, first marks the image frame in which the inertia event occurred and records in the storage unit 11, 40 (S20D), and then returns to step S1D, and if not, proceeds to step S31D. Step S3 includes steps S31D and S32D: the processor 10 determines whether a gesture is satisfied according to the image sensing information in the unmarked image frame (S31D), and if so, outputs a corresponding gesture (S32D), if not Then, it returns to step S1D.

As shown in FIG. 4 and FIG. 5, it is determined whether an inertial event occurs according to the inertial sensing signal S _I. If the inertial event has occurred, the image sensing information S _G has been subjected to the inertia event. The effect does not truly reflect the manipulation or gesture that the user wants to perform, so there is no need to further determine whether an object information is detected. In addition, as shown in FIG. 6 and FIG. 7 , it is first determined whether an object information is detected according to the image sensing information S _G. If it is determined that there is no object information, there is no need for further recognition of the gesture, so of course There is no need to judge whether an inertia event has occurred. Therefore, in accordance with the foregoing embodiment shown in FIG. 4 to FIG. 7, the processor 10 can determine whether to perform another determination or directly return to step S1 according to one of the determination results, so as to reduce the load of the processor 10, and Reduce energy consumption.

In the embodiment shown in FIG. 5 and FIG. 7 , the image frame in which the inertia event occurs is marked, and the time point before and after the occurrence of the inertia event also affects the correctness of the judgment gesture, and the unmarked is determined. When the image sensing information in the image frame satisfies a gesture (S31B) (S31D), the processor 10 will simultaneously exclude the marked image frame and the other image frames adjacent thereto, and then The image sensing information in the plurality of image frames that are not affected by the inertia event is judged to conform to a gesture. In this way, the influence of the inertia event on the gesture judgment can be effectively eliminated, so as to improve the correctness of the gesture judgment.

The method of the present invention will be described below in conjunction with the foregoing embodiments using a specific embodiment:

Referring to FIG. 8 and FIG. 4 and FIG. 6 , a plurality of image frames F1 F F6 are generated in a detection period, wherein the image frame F1 is in accordance with the steps (S21A, S22A) or steps (S21C, S22C). When it is determined that an object information G1 is detected and an inertia event has not occurred, steps S30A and S30C are performed to mark the image frame F1, and then steps S31A and S31C are executed to determine the marked image in the image frame F1. Whether the sensing information conforms to a gesture, and if it is determined that the image sensing information is still insufficient to conform to a gesture, return to steps S1A and S1C to continue detecting; then the image frame F2 and the front The image frame F1 is the same. Only the object information G2 is detected and no inertia event occurs. Therefore, the image frame F2 is also marked according to the foregoing steps. When the steps S31A and S31C are executed, the marked two are marked. The image sensing information in the image frames F1 and F2 is combined to determine whether the gesture conforms to a gesture. If yes, the steps S32A and S32C are performed to output the corresponding gesture. If not, the process returns to the steps S1A and S1C to continue the detection. When the image frame F3 determines that an inertial event I3 occurs according to steps S21A and S22C, it returns to steps S1A and S1C to continue detecting; when the image frame F4 determines that an inertial event I4 occurs according to steps S21A and S22C, Steps S1A and S1C continue to detect; then, the image frame F5 is the same as the image frame F1, and only one object information G5 is detected without an inertia event, so the image frame F5 is also marked according to the foregoing steps. When the steps S31A and S31C are performed, the image sensing information in the three image frames F1, F2, and F5 are combined to determine whether the gesture conforms to a gesture. If yes, steps S32A and S32C are performed to output the signal. Corresponding gestures, do not meet go back to step S1A, S1C continue detection. In an embodiment, when the steps S31A and S31C are performed, the image frames F2 and F5 adjacent to the unmarked image frames F3 and F4 may be excluded together, only in the image frame F1. The image sensing information is used to determine whether a gesture is met.

Then, the image frame F6 is the same as the image frame F1. Only an object information G6 is detected and no inertia event occurs. Therefore, the image frame F6 is also marked according to the foregoing steps. At this time, steps S31A and S31C are performed. The image sensing information in the four image frames F1, F2, F5, and F6 that have been marked is combined to determine whether the gesture conforms to a gesture. If yes, the steps S32A and S32C are performed to output the corresponding gesture. Then, the process returns to steps S1A and S1C to continue the detection. In an embodiment, when the steps S31A and S31C are performed, the image frames F2 and F5 adjacent to the unmarked image frames F3 and F4 may be excluded, and only the image frames F1 and F6 may be excluded. The image in the sense information is used to determine whether a gesture is met.

Referring to FIG. 8 and FIG. 5 and FIG. 7 , a plurality of image frames F1 F F6 are generated in a detection period, wherein the image frame F1 is in accordance with the steps (S21B, S22B) or steps (S21D, S22D) determining that an object information G1 is detected and no inertia event occurs, performing steps S31B, S31D to determine whether the image sensing information in the unmarked image frame F1 conforms to a gesture, and the judgment is still insufficient. After the gesture is met, the process returns to the steps S1B and S1D to continue the detection; then the image frame F2 is the same as the image frame F1, and only one object information G2 occurs without an inertia event, so the steps are also performed according to the foregoing steps. S31B and S31D, combining the image sensing information in the two unmarked image frames F1 and F2 to determine whether the gesture conforms to a gesture, and if yes, executing steps S32B and S32D to output the corresponding gesture, if not, returning Go to step S1B, S1D to continue detecting; when the image frame F3 determines that an inertial event I3 occurs according to steps S21B, S22D, mark the image frame F3 and return to steps S1B, S1D to continue detecting; when the image The frame F4 determines that an inertial event I4 occurs according to steps S21B and S22D, marks the image frame F4 and returns to steps S1B and S1D to continue detecting; then the image frame F5 is compared with the image frame F1. Only an object information G5 is detected and no inertia event occurs. Therefore, steps S31B and S31D are also performed, and the image sensing information in the unmarked three image frames F1, F2, and F5 is combined to determine whether the gesture conforms to a gesture. If yes, steps S32B and S32D are executed to output the corresponding gesture. If not, the process returns to steps S1B and S1D to continue the detection. In an embodiment, when the steps S31B and S31D are performed, the image frames F2 and F5 adjacent to the marked image frames F3 and F4 may be excluded, only in the image frame F1. The image sensing information is used to determine whether a gesture is met.

Then, the image frame F6 is the same as the image frame F1. Only one object information G6 is detected and no inertia event occurs. Therefore, when steps S31B and S31D are also performed, the unmarked four image frames F1 are displayed. The image sensing information in F2, F5, and F6 is combined to determine whether the gesture conforms to a gesture. If yes, steps S32B and S32D are performed to output the corresponding gesture. If not, the processing returns to steps S1B and S1D to continue detecting. In an embodiment, when the steps S31B and S31D are performed, the image frames F2 and F5 adjacent to the marked image frames F3 and F4 may be excluded, and only the image frames F1 and F6 may be used. The image in the sense information is used to determine whether a gesture is met.

In summary, according to the device and method of the present invention, the influence of inertial events such as vibration, movement, and shaking caused by the user on the non-contact gesture judgment can be further effectively eliminated, and the image sense is not affected by the inertia event. The information is measured to determine the non-contact gesture, and thus the correctness of the determined contactless gesture can be improved.

In another embodiment, as shown in FIG. 9, the method of the present invention is applied to a portable electronic device, and the portable electronic device includes a gesture detecting unit for detecting a gesture operation of a user. And a first sensor for detecting motion of the portable electronic device itself, the method comprising: determining that the portable electronic device is moving according to an output of the first sensor (S1E); and after step S1E The interrupt triggers the gesture operation of the user (S2E). The portable electronic device can be a non-contact sensing device, and the gesture detecting unit can include one or more of the foregoing image sensors 30, and the first sensor can include one or more of the foregoing The sensing device 20, when determining that the portable electronic device is in motion, may mean that the sensing amount of the sensing signal detected by the first sensor is greater than a preset threshold; the interruption triggers the user's Gesture operation, which can abandon the gesture operation detected by the pen.

The above is only the embodiment of the present invention, and is not intended to limit the scope of the present invention. The present invention has been disclosed by the embodiments, but is not intended to limit the invention, and any one of ordinary skill in the art, In the scope of the technical solutions of the present invention, equivalent modifications may be made to the equivalents of the embodiments of the present invention without departing from the technical scope of the present invention. Any simple modifications, equivalent changes and modifications made to the above embodiments are still within the scope of the technical solutions of the present invention.

Claims (24)

A non-contact gesture determination method includes: a. detecting an image sensing information in a sensing range of a non-contact sensing device, and detecting an inertial sensing signal of the non-contact sensing device itself b. determining, according to the inertial sensing signal, whether an inertial event occurs in the non-contact sensing device, determining whether an object information is detected according to the image sensing information; c. determining that the inertia event does not occur and detecting When the object information is detected, the gesture is judged according to at least one piece of the image sensing information and the corresponding gesture is output. The non-contact gesture judging method according to claim 1, wherein in step b, it is first determined whether the inertia event occurs, and if it is determined that the inertia event does not occur, further determining whether the object information is detected; If the inertia event occurs, return to step a. The non-contact gesture judging method according to claim 2, wherein when it is determined that the inertia event does not occur, and further determining whether the object information is detected, if it is determined that the object information is not available, the process returns to step a. The non-contact gesture judging method according to claim 1, wherein in step b, it is first determined whether the object information is detected, and if it is determined that the object information is detected, further determining whether the inertia event occurs; If it is determined that the object information is not detected, then return to step a. The non-contact gesture judging method according to claim 4, wherein when it is determined that the object information is detected and further determining whether the inertia event occurs, if it is determined that the inertia event occurs, the process returns to step a. The non-contact gesture judging method according to any one of claims 1 to 5, wherein the step c further comprises the following steps: c1. In an image frame, when the step b determines that the inertia event does not occur and detects When the object information is obtained, the image frame is marked; c2. determining whether the image sensing information in the marked image frame corresponds to a gesture, and if not, returning to step a, and if yes, executing step c3; C3. Output the corresponding gesture. The non-contact gesture judging method according to any one of claims 1 to 5, wherein: step b further comprises the step of: marking an image frame when determining that the inertia event occurs in an image frame; Step c further includes the following steps: c1. determining whether the image sensing information in the unmarked image frame conforms to a gesture, and if not, returning to step a, and if yes, performing step c2; c2. output corresponding The gesture. The non-contact gesture judging method according to claim 7, wherein: in step c1, the image frame adjacent to the marked image frame is excluded, and then the image frame remaining in the excluded image frame is excluded. The image sensing information is used to determine whether a gesture is met. The non-contact gesture determination method according to any one of claims 1 to 5, wherein determining whether the inertia event occurs in step b means that the sensing amount of the inertial sensing signal is compared with a threshold value, if If the amount of inertia sensing signal exceeds the threshold, it is determined that the inertia event occurs. The non-contact gesture judging method according to any one of claims 1 to 5, wherein determining whether the object information is detected in step b is determining whether the image sensing signal conforms to one of the preset object features. . A non-contact sensing device includes: a processor; an inertial sensor connected to the processor; at least one image sensor connected to the processor; wherein the processor is executed The following steps are: a. receiving one of the non-contact image sensing information detected by the image sensor, and receiving the detection by the inertial sensor to detect the non-contact sensing device itself a sensing signal; b. determining whether the non-contact sensing device generates an inertial event according to the inertial sensing signal, determining whether the object information is detected according to the image sensing information; c. determining that the inertia event does not occur And when the object information is detected, the gesture is judged according to at least one piece of the image sensing information and the corresponding gesture is output. The non-contact sensing device of claim 11, wherein the inertial sensor is an accelerometer, a gyroscope, or a magnetometer. The non-contact sensing device of claim 11 or 12, further comprising a storage unit built into the processor or connected to the processor in an external manner. The non-contact sensing device of claim 13, wherein the step c performed by the processor further comprises the following steps: c1. In an image frame, when the processor determines in the foregoing step b that the inertia does not occur When the event is detected and the object information is detected, the image frame is marked in the storage unit; c2. The processor determines whether the image sensing information in the marked image frames matches a gesture, and if not, Going back to step a, if yes, executing step c3; c3. outputting the corresponding gesture. The non-contact sensing device of claim 13, wherein the step b and the step c performed by the processor further comprise the following steps: Step b further comprising the steps of: in an image frame, when the processor determines When the inertia event occurs, the image frame is marked in the storage unit; step c further includes the following steps: c1. The processor determines whether the image sensing information in the unmarked image frame conforms to a gesture, if Otherwise, return to step a, and if yes, execute step c2; c2. Output the corresponding gesture. The non-contact sensing device of claim 15, wherein in the step c1 performed by the processor, the processor excludes the image frame adjacent to the marked image frame, and then removes the image frame. The image sensing information in the remaining image frames is used to determine whether a gesture is met. The non-contact sensing device of claim 11 or 12, wherein in the step b executed by the processor, it is first determined whether the inertia event occurs, and if it is determined that the inertia event does not occur, further determining whether to detect Go to the object information; if it is determined that the inertia event occurs, return to step a. The non-contact sensing device of claim 17, wherein in the step b executed by the processor, when it is determined that the inertia event does not occur, and further determining whether the object information is detected, if the object is not determined For information, go back to step a. The non-contact sensing device of claim 11 or 12, wherein in step b performed by the processor, it is first determined whether the object information is detected, and if it is determined that the object information is detected, further It is determined whether the inertia event occurs; if it is determined that the object information is not detected, then return to step a. The non-contact sensing device of claim 19, wherein in the step b executed by the processor, when it is determined that the object information is detected, and further determining whether the inertia event occurs, if the inertia is determined When the event occurs, go back to step a. The non-contact sensing device of claim 11 or 12, wherein determining whether the inertial event occurs in step b performed by the processor means that the amount of the inertia sensing signal is compared with a threshold value, If the amount of the inertia sensing signal exceeds the threshold, it is determined that the inertia event occurs. The non-contact sensing device of claim 11 or 12, wherein determining whether the object information is detected in step b performed by the processor means determining whether the image sensing signal meets one of the presets Object characteristics. A non-contact gesture determination method is applied to a portable electronic device. The portable electronic device includes a gesture detection unit for detecting a gesture operation of a user, and a first sensor for detecting The movement of the portable electronic device includes: a. determining that the portable electronic device is moving according to an output of the first sensor; and b. after step a, interrupting a gesture operation that triggers the user. The non-contact gesture judging method according to claim 23, wherein the interrupt described in step b triggers the gesture operation of the user, and the gesture operation is discarded.