WO2021227651A1 - Finger action acquisition method and finger action acquisition device - Google Patents

Abstract

A finger action acquisition method and a finger action acquisition device, which relate to the technical field of human-machine interaction. The finger action acquisition method comprises: a light source (2), worn on the forearm or wrist of a user, emitting light towards the fingers; an optical sensor, worn on the forearm or wrist of the user, sensing reflected light, the reflected light being the light after the light is reflected by fingers; and generating information or an instruction according to the reflected light sensed by the optical sensor. By wearing the optical sensor and the light source (2) on one side of the wrist or forearm, part or all of the fingers can be clearly illuminated and sensed, and since the fingertips almost completely determine the finger action with respect to the position of the wrist, by using this method, the problems both of the convenience of external devices and of the accuracy of an indirect sensing method are solved, and the scope of application is wide.

Description

Finger motion collection method and finger motion collection equipment Technical field

The invention relates to the technical field of human-computer interaction, in particular to a finger motion collection method and a finger motion collection device.

Background technique

It is an important idea to realize the output of instructions by recognizing gestures to realize human-computer interaction. However, in the existing technology, part of the technical solution is to put the equipment outside the body, such as installing in a TV, air conditioner, and range hood. The gesture recognition device on the Internet can ensure recognition accuracy and richness, but people need to stay near the device, which is not convenient enough and has a small application range.

Another part of the technical solution is to use wearable devices, but in these wearable devices, finger movements are recognized through indirect methods, for example: the pressure sensor detects the undulation of the wrist skin caused by the finger movements to recognize the finger movements, and the electromyography Sensors sense changes in the electrical skin signal of the wrist caused by finger movements to identify finger movements, optical sensors sense changes in the texture and shape of the skin surface of the wrist caused by finger movements to recognize finger movements, and angiography technology senses the wrist caused by finger movements Changes in the shape of the subcutaneous blood vessel of the lower arm or forearm are used to recognize finger movements. The above-mentioned indirect methods for recognizing finger movements have problems such as poor accuracy and poor versatility; A wearable device with an optical sensor. The light source can be positioned on the back side of the wearable device or the side facing the skin, and the optical sensor can be positioned near the light source. During operation, light can be emitted from the light source and use optical The sensor performs sensing, and the changes in the sensed light can be used to recognize user gestures; the light emitted from the light source can be reflected from the wearer’s skin, and optical sensors can be used to sense the reflected light, and the wearer can make gestures in a specific way At this time, due to muscle contraction, device shifting, skin stretching, or the distance between the optical sensor and the wearer’s skin, the reflected light may change noticeably, and the recognized gesture can be interpreted as being used to interact with the wearable It can be seen that this technical solution uses an indirect method to recognize finger movements, which also has problems such as poor accuracy.

technical problem

The purpose of the present invention is to provide a finger motion collection method, which aims to solve the problems that the finger motion collection method in the prior art cannot simultaneously achieve convenient use, wide application range, and high recognition accuracy.

Technical solutions

To achieve the above objective, the present invention adopts the following technical solutions: the finger motion collection method includes a light source worn on the user's forearm or wrist emitting light toward the finger; an optical sensor worn on the user's forearm or wrist detects the reflected light, The reflected light is the light reflected by the finger; information or instructions are generated based on the reflected light sensed by the optical sensor.

A further technical solution of the present invention: the light source and the optical sensor are arranged at the same position, or separately arranged at different positions.

A further technical solution of the present invention: the light is visible light or invisible light.

A further technical solution of the present invention: after the optical sensor detects the reflected light, it recognizes the spatial position signal of certain points on the user's finger corresponding to the reflected light, wherein the spatial position signal includes a three-dimensional position signal, a two-dimensional position signal Position signal and/or one-dimensional position signal, generate information or instructions based on the spatial position signal; or/and, identify the light intensity signal of the reflected light, and generate information or instructions based on the light intensity signal; or/and , Identify the texture image signal of the finger skin corresponding to the reflected light, and generate information or instructions based on the texture image signal of the finger skin; or/and identify the shape image signal of the finger skin corresponding to the reflected light, based on The image signal of the shape of the finger skin is used to generate information or instructions.

A further technical solution of the present invention: the method for detecting the reflected light by the optical sensor includes: collecting the reflected light by using a monocular camera, a binocular camera, a multi-eye camera, a TOF camera or/and a 3D structured light camera.

A further technical solution of the present invention: the information generated based on the reflected light sensed by the optical sensor is the information of the finger that can be directly sensed by the optical sensor, including the spatial position information of certain points on the finger, and the finger Part of the shape information, finger texture information, this information can be used to be recorded, displayed on the screen, or displayed in a VR scene; or, the information generated based on the reflected light sensed by the optical sensor is the entire hand The information includes hand movement information and position information of various parts of the hand. The corresponding relationship between the position of the finger relative to the wrist and the hand movement is stored through a database or machine learning algorithm, and the hand is obtained by sensing the position of the end of the finger This information can be used to be recorded, displayed on the screen, or displayed in a VR scene.

A further technical solution of the present invention: the instructions generated based on the reflected light sensed by the optical sensor include: discrete instructions, continuous instructions, or combined instructions; wherein the discrete instructions include button instructions, mouse click instructions; continuous instructions include mouse Move; combined instructions include mouse movement and click.

A further technical solution of the present invention: the light source and the optical sensor are arranged on the forearm or wrist on one side of the user, and the optical sensor senses the reflected light of the finger on that side; or, the forearms on both sides of the user Or the wrist is provided with a matching light source and optical sensor, and the optical sensor on the same side senses the reflected light of the finger on that side; or, the light source and optical sensor are provided on the forearm or wrist of the user, And the optical sensor simultaneously senses the reflected light of the finger on the user's side and the reflected light of the skin of the user's other side of the hand; or, the light source and the optical sensor are arranged on the forearm or wrist of the user's side, And the light source and the optical sensor are provided with two sets, which are respectively used to sense the reflected light from the user's hand on this side and the other side; or, the forearms or wrists on both sides of the user are respectively provided with matching use A light source and an optical sensor, and the optical sensor senses the reflected light from the user's hand on one side and the other side at the same time.

A further technical solution of the present invention: It also includes a feedback method for letting the user know the degree of similarity between his finger action and his original intention, and the feedback method includes visual feedback, auditory feedback or/and tactile feedback.

A further technical solution of the present invention: the visual feedback includes: generating a signal based on the reflected light sensed by the optical sensor, recognizing the signal, and displaying an image including the position of the user's finger relative to each key on the keyboard on the screen Information, the user uses the finger to tap the keyboard, the optical sensor senses the changing reflected light to generate a signal, continues to recognize the signal, and synchronously displays the continuous image information of the user’s finger on the screen. When the difference between the signal of the tap action and the preset signal of correctly tapping the keyboard does not exceed a preset threshold, the corresponding key on the screen is displayed in one color; when the signal of the finger tapping action in the image information is consistent with the correct tapping of the keyboard When the difference of the preset signal exceeds a preset threshold, the corresponding key on the screen is displayed in another color, and the color display is different when the deviation direction of the finger tapping action and the preset action of tapping the keyboard is different .

A further technical solution of the present invention: the auditory feedback includes: generating a signal of a finger tapping action based on the reflected light sensed by the optical sensor, and when the signal of the finger tapping action is consistent with a preset of correctly tapping the keyboard When the signal difference does not exceed a preset threshold, the sound device is controlled to emit a prompt sound; when the difference between the signal of the finger tapping action and the preset signal of correctly tapping the keyboard exceeds a preset threshold, the sound is controlled When the device emits another prompt sound, and the deviation direction of the finger tapping action is different from the preset action of tapping the keyboard correctly, the prompt tone is also different.

A further technical solution of the present invention: the tactile feedback includes: generating a finger tapping action signal based on the reflected light sensed by the optical sensor. When the signal difference does not exceed a preset threshold, the haptic device is controlled to emit a vibration signal; when the difference between the signal of the finger tapping action and the preset signal of correctly tapping the keyboard exceeds a preset threshold, the tactile sense is controlled When the device sends out another kind of vibration signal, and the deviation direction of the finger tapping action is different from the preset action of tapping the keyboard correctly, the vibration signal is also different.

A further technical solution of the present invention: in the visual feedback, when the deviation direction of the finger tapping action relative to the preset action of correctly tapping the keyboard is toward the left, right, upper, and lower sides, the corresponding on the screen The keys are displayed in different colors A, B, C, and D respectively.

A further technical solution of the present invention: in the auditory feedback, when the deviation direction of the finger tapping action relative to the preset action of the correct tapping on the keyboard is respectively toward the left, right, upper, and lower sides, the sound device is controlled respectively Different prompt tone A, prompt tone B, prompt tone C, and prompt tone D are emitted.

A further technical solution of the present invention: in the tactile feedback, when the deviation directions of the finger tapping action relative to the preset action of the correct tapping on the keyboard are respectively toward the left, right, upper, and lower sides, the tactile device is controlled respectively Send out different vibration signal A, vibration signal B, vibration signal C, and vibration signal D.

A further technical solution of the present invention: it also includes a position sensor, an acceleration sensor or a speed sensor worn on the forearm or wrist of the user.

The object of the present invention is to provide a finger motion collection device using the above-mentioned finger motion collection method, including a housing that can be worn on the forearm or wrist of a user; a light source mounted on the housing and capable of emitting light toward the user's fingers; An optical sensor on the housing that can sense the reflected light from the user's finger.

A further technical solution of the present invention: the casing is provided with a wristband, the user wears the wristband to set the casing on the user's forearm or wrist, and the casing is located on the user's forearm or wrist facing the palm surface side.

Beneficial effect

The beneficial effects of the present invention are: in the present invention, the optical sensor and light source are worn on one side of the wrist or forearm, which can clearly illuminate and sense part or all of the finger, and because of the position of the end of the finger relative to the wrist The finger motion is almost completely determined, so the use of this method not only solves the problem of the convenience of the external device, but also solves the problem of the accuracy of the indirect sensing method, and has a wide range of applications.

Description of the drawings

Figure 1 is a schematic diagram of a specific embodiment of the finger motion collection device of the present invention;

Fig. 2 is a schematic diagram of tactile feedback in a specific embodiment of the finger motion collection device of the present invention.

Embodiments of the present invention

The specific embodiments of the present invention will be further described below in conjunction with the accompanying drawings.

It is an important idea to realize the output of instructions by recognizing gestures to realize human-computer interaction. However, in the prior art, part of the technical solution is to put the device outside the body, which can ensure the recognition accuracy and richness, but people need Staying near the device is not convenient enough; another part of the technical solution is to use wearable devices, but in these wearable devices, finger movements are recognized through indirect methods, for example: the pressure sensor detects the undulation of the wrist skin caused by the finger movement To recognize finger motions, to recognize finger motions by sensing changes in the electrical skin signal of the wrist caused by finger motions by electromyography sensors, to recognize finger motions by sensing changes in the texture and shape of the skin surface of the wrist caused by finger motions through optical sensors, and to recognize finger motions through angiography The technology detects changes in the shape of the subcutaneous blood vessels of the wrist or forearm caused by finger movements to recognize finger movements, etc. The above-mentioned indirect methods to recognize finger movements have problems such as poor accuracy.

Therefore, in the present invention, as shown in FIG. 1, a finger motion collection method includes a light source 2 worn on the user's forearm or wrist emitting light toward the finger; an optical sensor worn on the user's forearm or wrist detects the reflection Light, the reflected light is the light reflected by the finger; information or instructions are generated based on the reflected light sensed by the optical sensor.

The finger movement collection method is used to determine and recognize the user's finger movement, and output information or instructions. In this specific embodiment, the optical sensor and the light source 2 are worn on the side of the user's wrist or forearm facing the palm, as shown in FIG. 1, The light source emits light toward the finger, and the optical sensor senses the light reflected by the finger, and generates information or instructions according to the sensed light.

Of course, the optical sensor and the light source can also be worn on the user's wrist or other side of the forearm, as long as the optical sensor can sense the light emitted by the light source reflected by the finger. More specifically, the light source and the optical sensor are arranged at the same position. For example, the light source and optical sensor are worn in the middle position of the user’s wrist or forearm facing the palm, or set in different positions separately. The optical sensor is correspondingly worn on the side of the user's wrist or forearm facing the little finger or thumb.

A finger motion collection device using the above-mentioned finger motion collection method, comprising a housing 1 that can be worn on the forearm or wrist of a user; a light source 2 mounted on the housing 1 and capable of emitting light toward the user's fingers; mounted on the housing An optical sensor that can sense the reflected light of the user’s finger. By wearing the device, it can be used to realize human-computer interaction between humans and other devices, such as switching devices, switching device screens, sound adjustments, and text input. , Game control, etc., you can also use the wearable device as a signal or instruction output object to control some functions on the device, such as device switch operation, opening or closing of the application in the device, screen switching in the application, and sound in the application Adjustments and so on.

In the present invention, the optical sensor and light source 2 are worn on one side of the wrist or forearm, which can clearly illuminate and sense part or all of the finger, and because the position of the end of the finger relative to the wrist almost completely determines the finger action Therefore, the use of this method not only solves the problem of the convenience of in vitro equipment, but also solves the problem of the accuracy of the indirect sensing method, and has a wide range of applications.

Preferably, the casing 1 is provided with a wristband 3, and the user wears the wristband 3 to set the casing 1 on the user's forearm or wrist, and the casing is located on the side of the user's forearm or wrist facing the palm surface As shown in Figure 1, the light source emits light to illuminate the finger, and the sensor senses the light reflected by the finger to generate a signal or instruction; in another specific embodiment, the device can be worn on the forearm due to different usage scenarios or user habits. On one side of the palm, the light source and optical sensor are on the side of the device facing the fingers.

Specifically, an important use scenario of the method and device provided by the present invention is to connect the device with smart glasses, and output signals or instructions to the smart glasses through finger motions.

Specifically, the light emitted by the light source can be visible light or invisible light; it can be a single light source or a combination of multiple light sources; it can be a simple homogeneous light, or it can be modulated with a certain structure. Light.

Optionally, the method for detecting the reflected light by the optical sensor includes: collecting the reflected light with a monocular camera, a binocular camera, a multi-camera camera, a TOF camera or/and a 3D structured light camera, and so on.

Furthermore, in a specific embodiment, a structured light depth vision device is used to realize the emission and sensing of light. The device includes a light source 2 and an optical sensor. The principle is that the emitted light has a certain structure through modulation. The sensor senses the above-mentioned light reflected by the object, and determines the shape information of the surface of the object according to the structural change of the reflected light; in another specific embodiment, a TOF device can be used to realize the emission and sensing of light, and the device includes a light source 2 and The principle of the optical sensor is to illuminate the finger through the light source 2, sense the reflected light through the optical sensor, and calculate the time it takes for the light to be emitted from emission to being sensed to determine the distance between the light source 2, the sensor and some points on the finger; In a specific embodiment, the light source 2 plus a camera can be used to obtain the light reflected by the finger, and information or instructions can be generated according to the reflected light. It can be a single camera or multiple cameras. Increasing the number of cameras can improve the recognition accuracy. ; The above scheme can be used alone or in combination of several.

Specifically, after the optical sensor detects the reflected light, it recognizes the spatial position signal of certain points on the user's finger corresponding to the reflected light, where the spatial position signal includes a three-dimensional position signal, a two-dimensional position signal, and/or One-dimensional position signal, generating information or instructions based on the spatial position signal; or, identifying the light intensity signal of the reflected light, and generating information or instructions based on the light intensity signal; or, identifying the light intensity signal corresponding to the reflected light The texture image signal of the finger skin generates information or instructions based on the texture image signal of the finger skin; or, the shape image signal of the finger skin corresponding to the reflected light is recognized, and the information or command is generated based on the shape image signal of the finger skin. Instructions, etc. The device can generate information or instructions based on a certain type of information, or it can generate information or instructions based on a combination of several types of information.

In a specific embodiment, the information generated based on the reflected light sensed by the optical sensor is the information of the finger that can be directly sensed by the optical sensor, including the spatial position information of certain points on the finger, and the information of the finger part. Shape information, texture information of the finger part, which can be used to be recorded, displayed on the screen or displayed in the VR scene;

In another specific embodiment, the information generated based on the reflected light sensed by the optical sensor is information of the entire hand, including hand motion information and position information of various parts of the hand, which is stored by a database or a machine learning algorithm The corresponding relationship between the position of the finger relative to the wrist and the hand movement. The hand movement information is obtained by sensing the position of the end of the finger. This information can be used to be recorded, displayed on the screen or displayed in the VR scene; although the sensor Only the finger or part of the finger can be sensed, but there is a strong correlation between the position of the finger relative to the wrist and the hand movement. The correspondence between the position of the finger relative to the wrist and the hand movement is stored through a database or machine learning algorithm , You can get hand information by sensing finger information.

The instructions generated based on the reflected light sensed by the optical sensor include: discrete instructions, continuous instructions, or combined instructions; among them, discrete instructions include button instructions, mouse click instructions; continuous instructions include mouse movement; combined instructions include mouse movement And click.

In a specific embodiment, the finger volleys or taps the keyboard action on the desktop. The device senses the reflected light of the finger to understand the finger action, and outputs the corresponding key command. The device can obtain the spatial position information of the end of the finger. By contrast, you can understand the intention of the finger to hit the keyboard. The device will move when worn on the wrist, but the topological relationship between the points formed by the finger ends has strong stability in a single user, which can be corrected by the above topological relationship The effect of the device's wearing position on the recognition result.

In a specific embodiment, a machine learning algorithm can be used to realize the association between finger motion information and keystroke instructions. After wearing the device, perform a keystroke action to collect finger information and a keystroke within a period of time collected by the device The instruction is a set of data, collect multiple sets of the above data, edit the machine learning algorithm at the same time, use the above data to train the algorithm, after the algorithm's recognition accuracy reaches the expected requirement, the algorithm can be used to realize the instruction output; the above data can be collected It can be collected from the same person or from different people. The wider the scope of collection and the more data collected, the higher the versatility and accuracy of the trained algorithm.

In a specific embodiment, after wearing the device, first adjust the device to output mouse instructions, and then move a certain finger or a few fingers to realize the instruction output of moving the cursor. The instruction of clicking the left mouse button is realized by tapping with another finger to realize the instruction of clicking the right mouse button.

Specifically, the light source and the optical sensor are arranged on the forearm or wrist on one side of the user, and the optical sensor senses the reflected light from the finger on that side; or, the forearms or the wrist on both sides of the user are respectively arranged There is a matching light source and optical sensor, and the optical sensor on the same side senses the reflected light of the finger on that side; or, the light source and optical sensor are arranged on the forearm or wrist of the user side, and the optical sensor Simultaneously sense the reflected light of the finger on the user's side and the reflected light of the skin of the user's other side of the hand; or, the light source and the optical sensor are arranged on the forearm or wrist of the user's side, and the light source and There are two sets of optical sensors, which are respectively used to sense the reflected light from the user’s hand on one side and the other side; or the forearms or wrists on both sides of the user are respectively provided with matching light sources and optical sensors, And the optical sensor simultaneously senses the reflected light from the user's hand on one side and the other side.

It should be noted that whether the device is worn with one hand or with both hands, whether the device is worn with one hand or both hands is sensed, and whether the two devices work independently or in linkage. The above-listed are non-limiting examples. There can be other multiple combinations under the enlightenment, which can be flexibly adopted according to the needs.

Optionally, in addition to the light source 2, the optical sensor, the device can integrate a motherboard, a processor, a battery, a display screen, a buzzer, a sound device, a wired transmission interface, a wireless transmission module, etc. Of course, the display screen can be set on the wrist The palm can also be set on the back of the wrist. The buzzer feeds back information through touch or hearing, and the sound device feeds back information through hearing or touch. The wired transmission interface and wireless transmission module are used to interact with other devices.

In a specific embodiment, in order to let the user know how similar his finger action is to his original intention, a feedback method is added, and the feedback method includes visual feedback, auditory feedback, or/and tactile feedback.

Wherein, optionally, a signal is generated based on the reflected light sensed by the optical sensor, the signal is recognized, and image information including the position of the user's finger relative to each key on the keyboard is displayed on the screen, and the user uses the finger to tap Keyboard action, the optical sensor senses the changing reflected light to generate a signal, continuously recognizes the signal, and synchronously displays the continuous image information of the user’s finger on the screen. When the difference between the preset signal of the keyboard does not exceed a preset threshold, the corresponding key on the screen is displayed in a color, such as green; when the signal of the finger tapping action in the image information is different from the preset signal of correctly tapping the keyboard When the difference exceeds a preset threshold, the corresponding key on the screen is displayed in another color, such as red, and the color display is different when the deviation direction of the finger tapping action and the preset action of correctly tapping the keyboard is different, such as :

When the finger tapping action is shifted to the left side of the key relative to the preset action of tapping the keyboard correctly, the corresponding key on the screen is displayed in a gradual red that gradually deepens from right to left;

When the finger tapping action is biased to the right side of the key relative to the preset action of tapping the keyboard correctly, the corresponding key on the screen will be displayed in a gradual red that gradually deepens from left to right;

When the finger tapping action is biased to the upper side of the key relative to the preset action of tapping the keyboard correctly, the corresponding key on the screen is displayed in a gradual red that gradually deepens from bottom to top;

When the finger tapping action is shifted to the lower side of the key relative to the preset action of tapping the keyboard correctly, the corresponding key on the screen is displayed in a gradual red that gradually deepens from top to bottom.

Wherein, optionally, the auditory feedback includes: generating a finger tapping action signal based on the reflected light sensed by the optical sensor, and when the finger tapping signal is compared with a preset signal for correctly tapping the keyboard When the difference does not exceed a preset threshold, the sound device is controlled to emit a prompt sound, such as a crisp, short and pleasant sound; when the signal of the finger tapping action is different from the preset signal of correctly tapping the keyboard When the difference exceeds a preset threshold, the sound device is controlled to emit another prompt sound, such as a muddy, relatively long sound, and when the direction of deviation between the finger tapping action and the preset action of correctly tapping the keyboard is different, The prompt tones are also different, such as:

When the finger tapping action is biased to the left side of the key relative to the preset action of tapping the keyboard correctly, a sound that can be distinguished from right to left is set as the prompt sound;

When the finger tapping action is biased to the right side of the button relative to the preset action of tapping the keyboard correctly, a sound that can be distinguished from left to right is set as the prompt sound;

When the finger tapping action is biased to the upper side of the key relative to the preset action of tapping the keyboard correctly, a sound that can be distinguished from near to far is set as the prompt sound;

When the finger tapping action is biased to the lower side of the key relative to the preset action of tapping the keyboard correctly, a sound that can be distinguished from far to near is set as the prompt sound;

The above-mentioned prompt tone can be emitted by the sound device on the device or by other devices.

Optionally, as shown in FIG. 2, the tactile feedback includes: generating a signal of a finger tapping action based on the reflected light sensed by the optical sensor. When the difference between the preset signal of the keyboard does not exceed a preset threshold, the haptic device is controlled to send out a vibration signal, such as a crisp and short vibration; when the signal of the finger tapping action is different from the preset signal of correctly tapping the keyboard When the difference exceeds a preset threshold, the haptic device is controlled to send another vibration signal, such as a turbid, long-lasting vibration, and when the deviation direction of the finger tapping action is different from the preset action of correctly tapping the keyboard, the vibration The signal is also different. For example, arrange 5 miniature buzzers in the device according to the cross shape:

When the finger tapping action is biased to the left side of the key relative to the preset action of tapping the keyboard correctly, the middle buzzer and the left buzzer will vibrate in sequence;

When the finger tapping action is biased to the right side of the key relative to the preset action of tapping the keyboard correctly, the middle buzzer and the right buzzer will vibrate in sequence;

When the finger tapping action is biased to the upper side of the key relative to the preset action of tapping the keyboard correctly, let the middle buzzer and the upper buzzer vibrate in sequence;

When the finger tapping action is biased to the lower side of the key relative to the preset action of tapping the keyboard correctly, the middle buzzer and the lower buzzer will vibrate in sequence;

It should be noted that the above feedback methods can be used alone or in combination.

In a specific embodiment, a position sensor, an acceleration sensor, or a speed sensor may be provided in the device.

More specifically, buttons, touchpads, touch screens, scroll wheels, voice recognition modules, etc. can be integrated on the device to assist in operating the device. The device can be turned on and off. The device can be operated by buttons, touch, scroll, voice, etc., or It can be realized by some special hand movements, for example, when you make an OK action with your fingers, the device starts up, when you clenched your fist, the device shuts down, and when the device switches from one function to another, you can use buttons or touch , Scrolling, voice, etc. It can also be achieved by some special hand movements. For example, if the index finger and middle finger tap twice at the same time, you can switch from the keyboard output function to the mouse output function. The device can use Wired or wirelessly connected to one or more devices to realize the function of outputting information or instructions to the device. To switch from the state of outputting information or instructions to one device to the state of outputting information or instructions to another device, you can It can be realized by buttons, touch, scroll, voice, etc. It can also be realized by some special hand movements. For example, if the index finger taps 4 times in a row, you can switch from the state of connecting to the TV to the state of connecting to the speaker.

For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description, and therefore it is intended to fall within the claims. All changes within the meaning and scope of the equivalent elements of are included in the present invention. Any reference signs in the claims should not be regarded as limiting the claims involved.

... In addition, it should be understood that although this specification is described in accordance with the implementation manners, not each implementation manner only includes an independent technical solution. This narration in the specification is only for the sake of clarity, and those skilled in the art should regard the specification as a whole The technical solutions in the various embodiments can also be appropriately combined to form other implementations that can be understood by those skilled in the art.

Claims (14)

A method for collecting finger movements, characterized in that it comprises: a light source worn on the forearm or wrist of a user emits light toward the finger; an optical sensor worn on the forearm or wrist of the user senses the reflected light, and the reflected light is said The light after the light is reflected by the finger; information or instructions are generated based on the reflected light sensed by the optical sensor. The finger motion collection method according to claim 1, wherein the light source and the optical sensor are arranged at the same position, or separately arranged at different positions. The finger motion collection method according to claim 1, wherein the light is visible light or invisible light. The finger motion collection method according to claim 1, wherein after the optical sensor detects the reflected light, it recognizes the spatial position signal of a certain point on the user's finger corresponding to the reflected light, wherein the spatial position The signal includes a three-dimensional position signal, a two-dimensional position signal and/or a one-dimensional position signal, and information or instructions are generated based on the spatial position signal; or/and the light intensity signal of the reflected light is identified, and the light intensity signal is based on the light intensity signal. Generate information or instructions; or/and, identify the texture image signal of the finger skin corresponding to the reflected light, and generate information or instructions based on the texture image signal of the finger skin; or/and, identify the texture image signal corresponding to the reflected light The shape image signal of the finger skin generates information or instructions based on the shape image signal of the finger skin. The finger motion collection method according to claim 1, wherein the method for detecting the reflected light by the optical sensor comprises: adopting a monocular camera, a binocular camera, a multi-eye camera collection, a TOF camera or/and a 3D structured light The camera collects the reflected light. The finger motion collection method according to claim 1, wherein the information generated based on the reflected light sensed by the optical sensor is the information of the finger that can be directly sensed by the optical sensor, including certain information on the finger. The spatial position information of some points, the shape information of the finger part, and the texture information of the finger part, which can be used to be recorded, displayed on the screen or displayed in the VR scene; or, based on the reflected light sensed by the optical sensor The information generated is the information of the entire hand, including hand movement information and position information of various parts of the hand. The corresponding relationship between the position of the finger relative to the wrist and the hand movement is stored through the database or machine learning algorithm. Measure the position of the end of the finger to obtain hand movement information, which can be used to be recorded, displayed on the screen, or displayed in a VR scene. The finger motion collection method according to claim 1, wherein the instructions generated based on the reflected light sensed by the optical sensor include: discrete instructions, continuous instructions, or combined instructions; wherein the discrete instructions include button instructions, Mouse click instructions; continuous instructions include mouse movements; combined instructions include mouse movements and clicks. The finger motion collection method according to any one of claims 1-7, wherein the light source and the optical sensor are arranged on the forearm or wrist of the user, and the optical sensor senses the finger on the side. Or, the forearms or wrists on both sides of the user are respectively provided with a matching light source and optical sensor, and the optical sensor on the same side senses the reflected light of the finger on that side; or, the light source and optical sensor The sensor is arranged on the forearm or wrist of one side of the user, and the optical sensor senses the reflected light of the finger on the user's side and the reflected light of the hand skin of the user's other side at the same time; or, the light source and optics The sensor is arranged on the forearm or wrist of the user, and the light source and the optical sensor are arranged in two sets, which are respectively used to sense the reflected light of the skin of the user’s hand and the other side; or, the user’s The forearms or wrists on both sides are respectively provided with matching light sources and optical sensors, and the optical sensors simultaneously sense the reflected light from the user's hand skin on one side and the other side. The finger action collection method according to any one of claims 1-7, further comprising a feedback method for letting the user know how similar his finger action is to his original intention, and the feedback method includes visual feedback, auditory feedback or / And tactile feedback. The finger motion collection method according to claim 9, wherein the visual feedback comprises: generating a signal based on the reflected light sensed by the optical sensor, recognizing the signal, and displaying on the screen a signal that includes the user’s finger relative to each other. For the image information of each key position on the keyboard, the user uses his finger to tap the keyboard. The optical sensor senses the changing reflected light to generate a signal, continuously recognizes the signal, and synchronously displays a continuous image of the user's finger on the screen Information. When the difference between the signal of the finger tapping action in the image information and the preset signal of correctly tapping the keyboard does not exceed a preset threshold, the corresponding key on the screen is displayed in one color; when the finger tapping in the image information When the difference between the signal of the action and the preset signal of correctly hitting the keyboard exceeds a preset threshold, the corresponding key on the screen is displayed in another color, and the deviation of the finger hitting action from the preset action of correctly hitting the keyboard When the direction is different, the color display is also different; the auditory feedback includes: generating a finger tapping action signal based on the reflected light sensed by the optical sensor. When the signal difference does not exceed a preset threshold, the sound device is controlled to emit a prompt sound; when the difference between the signal of the finger tapping action and the preset signal of the correct tapping of the keyboard exceeds a preset threshold, the control The sound device emits another prompt tone, and the deviation direction of the finger tapping action is different from the preset action of tapping the keyboard correctly, the prompt tone is also different; the tactile feedback includes: based on the reflection sensed by the optical sensor The light is used to generate the signal of the finger tapping action. When the difference between the signal of the finger tapping action and the preset signal of correctly tapping the keyboard does not exceed a preset threshold, the haptic device is controlled to emit a vibration signal; when the finger When the difference between the signal of the tapping action and the preset signal of correctly tapping the keyboard exceeds a preset threshold, the haptic device is controlled to emit another vibration signal, and the deviation of the finger tapping action from the preset action of correctly tapping the keyboard When the direction is different, the vibration signal is also different. The finger action collection method according to claim 10, wherein the deviation direction of the finger tapping action relative to the preset action of correctly tapping the keyboard in the visual feedback is respectively toward the left side, the right side, the upper side, and On the lower side, the corresponding keys on the screen are displayed in different colors A, B, C, and D; in the auditory feedback, the deviation directions of the finger tapping action relative to the preset action of correctly tapping the keyboard are respectively When facing the left, right, upper and lower sides, control the sound equipment to emit different prompts A, B, C, and D respectively; in the tactile feedback, when the finger tapping action is relative to the correct When the deviation directions of the preset actions of tapping the keyboard are respectively toward the left, right, upper, and lower sides, the haptic device is controlled to send different vibration signals A, vibration signals B, vibration signals C, and vibration signals D respectively. The finger motion collection method according to any one of claims 1-7, further comprising a position sensor, an acceleration sensor or a speed sensor worn on the forearm or wrist of the user. A finger action collection device using the finger action collection method according to any one of claims 1-12, characterized by comprising: a housing that can be worn on the forearm or wrist of a user; mounted on the housing and A light source that can emit light toward the user's finger; an optical sensor that is installed on the housing and can sense the reflected light of the user's finger. The finger motion collection device according to claim 13, wherein a wristband is provided on the housing, and the user wears the wristband so that the housing is set on the user’s forearm or wrist, and the housing is located on the user’s small arm or wrist. The side of the arm or wrist facing the palm surface.