CN110942040B - Gesture recognition system and method based on ambient light - Google Patents

Abstract

The present invention relates to the technical field of gesture recognition, and aims to provide a low-cost, high-accuracy gesture recognition system and method based on ambient light, which can overcome the requirements of similar systems for light sources and provide more abundant use scenarios. The technical solution adopted is: including data acquisition terminal, gesture recognition server and application terminal, data acquisition terminal includes multiple photoelectric receivers, signal amplification module, analog-to-digital conversion module and signal processing module; photoelectric receiver is used to capture gesture generated When the optical signal changes, the output terminals of the photoelectric receiver are connected to the input terminals of the signal amplification module, the output terminals of the signal amplification module are connected to the input terminals of the analog-to-digital conversion module, and the output terminals of the analog-to-digital conversion module are connected to the input terminals of the signal processing module. The data information processed by the signal processing module is transmitted to the gesture recognition server for recognition, and the gesture recognition server outputs the recognition information and sends it to the application end, and the application end displays the recognition information in real time.

Description

A gesture recognition system and method based on ambient light

technical field

The present invention relates to the technical field of gesture recognition, in particular to a gesture recognition system and method based on ambient light.

Background technique

The popularity of smart devices and the continuous innovation of human-computer interaction complement each other. Traditional smart homes or smart buildings usually use IoT technology to network smart devices, and send instructions through smart terminals to interact with them. This traditional interaction method is gradually developing towards a natural interaction method with lower learning costs. Voice assistants such as smart speakers such as Tmall Genie and smart devices such as Siri are common in the market. As one of the natural interaction methods, voice recognition has gradually matured and been widely used. Although there are products such as Kinect and Leap on the market, gesture recognition is still rarely used in daily life.

Common methods for realizing gesture recognition include using technologies such as images, acoustics, radio frequency, and visible light. But image and acoustic technologies require image capture and audio capture, which raises security and privacy concerns. Radio frequency technology not only requires the use of relatively complex technical means at the transmitting end and receiving end, but also is susceptible to interference from various electromagnetic fields. Ambient light has the advantages of wide spectrum, almost everywhere, safety, no privacy concerns, and convenient collection.

Tianxing Li et al. proposed and built a gesture reconstruction method and system Aili based on LED light in "Reconstructing Hand Poses Using Visible Light". The system uses a modulated LED array as a light source and a photodiode as a visible light sensor to realize 3D gesture reconstruction. However, this system and many similar systems in the field of visible light communication have the following disadvantages:

1) It is necessary to install additional modulation equipment at the light source, which is inconvenient to install and increases the system cost;

2) Due to the need for modulation, the light source type of the system can only be LED, but in fact the current penetration rate of LED is not high;

3) The main work of the system is 3D reconstruction. If it is used in gesture recognition scenarios, additional work is still required.

Contents of the invention

The purpose of the present invention is to provide a low-cost, high-accuracy gesture recognition system and method based on ambient light, which can overcome the requirements of similar systems for light sources, and provide more diverse usage scenarios.

In order to achieve the purpose of the above invention, the technical solution adopted by the present invention is: a gesture recognition system based on ambient light, including a data collection terminal, a gesture recognition server and an application terminal, the data collection terminal includes a plurality of photoelectric receivers, signal An amplification module, an analog-to-digital conversion module and a signal processing module; the photoelectric receiver is used to capture the light signal changes generated by gestures, and a plurality of the photoelectric receivers form a combined arrangement; the output terminals of the photoelectric receiver are all connected to the signal amplification The input end of the module is connected, the output end of the signal amplification module is connected with the input end of the analog-digital conversion module, the output end of the analog-digital conversion module is connected with the input end of the signal processing module, and the data processed by the signal processing module The information is transmitted to the gesture recognition server for recognition, and the gesture recognition server outputs the recognition information and sends it to the application end, and the application end displays the recognition information in real time.

Further, the gesture recognition server includes a data preprocessing unit and a deep learning network model unit, the data preprocessing unit decodes the data packets output by the signal processing module and restores them into multi-channel data, and the deep learning network model unit Recognizing and classifying the restored multi-channel data; the multi-channel data and the recognition and classification results of the deep learning network model unit are used as the recognition information output by the gesture recognition server.

Further, the combination arrangement is a rectangular array arrangement, a trapezoidal arrangement or a dispersed arrangement.

Further, the deep learning network model unit is a gated recurrent unit.

Further, the application end is a web page front end.

A gesture recognition method based on ambient light, comprising the following recognition steps:

S1: Multiple photoelectric receivers capture the light signal changes generated by gestures at different hand positions in real time, and convert the collected multiple light signals into current signals respectively;

S2: The signal amplification module converts the current signal into a voltage signal and amplifies it;

S3: The amplified voltage signal is converted into a digital signal through an analog-to-digital conversion template;

S4: Multi-channel digital signals are merged and encoded by the signal processing module, and then transmitted to the gesture recognition server;

S5: The data preprocessing unit decodes the received original data and restores it to multi-channel data;

S6: The restored multi-channel data is input into the deep learning network model unit to complete the recognition and classification of gestures;

S7: At the same time, the restored multi-channel data and the recognition and classification results are used as the input of the application side, and the application side displays the multi-channel data and the recognized gestures in real time.

Further, the gesture recognition method further includes a training step before recognition, and the training step is: the gesture recognition server performs training through a backpropagation algorithm according to different gesture actions, and establishes a deep network model.

The beneficial effects of the present invention are embodied in:

1. The photoelectric receiver can perform gesture recognition only with ambient light, there is no limit to the light source, and the application scenarios are more extensive;

2. There is no need to install additional modulation light source equipment, which reduces the system cost and makes it easier to use and install;

3. The photoelectric receiver is very sensitive and can detect small changes in light intensity, so it can identify nuanced movements;

4. Multiple photoelectric receivers are combined and arranged to form multi-channel photosensitive data, which can improve the accuracy of gesture recognition and reduce the recognition delay.

Description of drawings

Fig. 1 is a system block diagram of the present invention;

Fig. 2 is a block diagram of the recurrent neural network model of the present invention;

Fig. 3 is a schematic diagram of the layout of the photoelectric receiver of the present invention;

Fig. 4 is a schematic diagram of the deep learning gesture training framework of the present invention;

Fig. 5 is a schematic diagram of gesture training actions.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

As shown in Figure 1, a gesture recognition system based on ambient light includes a data collection terminal, a gesture recognition server, and an application terminal. The data collection terminal includes a plurality of photoelectric receivers, a signal amplification module, an analog-to-digital conversion module, and processing module;

The photoelectric receiver is used to receive ambient light in any free space, and can capture the light signal changes generated by gestures, and a plurality of the photoelectric receivers are arranged in combination; the photoelectric receiver converts the captured light signal into a current signal , the output terminals of the photoelectric receiver are connected to the input terminals of the signal amplification module;

The small current signal of the signal amplifier is converted into a voltage signal, and then amplified, and the output end of the signal amplification module is connected to the input end of the analog-to-digital conversion module;

The digital-to-analog conversion module converts the analog voltage signal output by the signal amplifier into a digital digital signal, the output end of the analog-to-digital conversion module is connected to the input end of the signal processing module, and the signal processing module is used to convert the analog The digital signals output by the digital conversion module are combined and encoded; when the gesture recognition server sends a request to the signal processing module, the signal processing module sends a data packet to the gesture recognition server;

代表向量逐元素相乘；门控循环单元共包括了两个门，后缀为u代表更新门，后缀为r代表重置门，最后通过后缀为c的状态更新运算计算出下一时刻的神经网络迭代向量；The gesture recognition server includes a data preprocessing unit and a deep learning network model unit. In this embodiment, the deep learning network model unit is a gated loop unit; as shown in Figure 2, wherein h represents the iteration of the neural network Vector, t is the time point, v represents the input vector, U and W are the parameter matrix of the gated recurrent unit, σ and tanh in the box represent the sigmoid and tanh activation functions respectively, g is the output vector of each gate,

Represents the multiplication of vectors element by element; the gated cycle unit includes two gates, the suffix u represents the update gate, the suffix r represents the reset gate, and finally calculates the neural network at the next moment through the state update operation with the suffix c iteration vector;

The data preprocessing unit decodes the data packets output by the signal processing module and restores them into multi-channel data, the restored multi-channel data is used as the input of the deep learning network model unit, and then the restored multi-channel data is identified And classification, the specific classification of gestures is shown in Figure 5, including seven gestures of natural drooping of five fingers, natural spreading of palms and clenching of fists.

The multi-channel data and the recognition and classification results of the deep learning network model unit are used as the recognition information output by the gesture recognition server, and the gesture recognition server outputs the recognition information and sends it to the application end. In this embodiment, the application The terminal is the front end of the web page, which can display the voltage value of each channel to the user, and can also be used to display the final gesture recognition result and related interactive applications.

In this embodiment, the photoelectric receivers are photodiodes or phototransistors, the number of which is preferably eight, and the photoelectric receivers are arranged in three combinations on the user's hand, as shown in Figure 3, in Figure 3 In the hand a of , the photoelectric receivers are arranged in a 2*4 rectangular array, and the distance between every two adjacent photoelectric receivers is 5cm. The advantage of this method is that it is symmetrical and easy to deploy;

In the hand b in Figure 3, the photoelectric receivers are arranged in a trapezoidal shape. There are three photoelectric receivers on the upper side, which are arranged on the index finger, middle finger and ring finger of the hand. There are five photoelectric receivers on the lower side, which are evenly distributed laterally. The distance between the upper and lower photoelectric receivers is 5cm; the advantage of this method is that the left and right hands can be switched conveniently without adjusting the arrangement, and it can adapt to different hand shapes. It is possible to capture as many tiny movement changes as possible in the corresponding space of gestures in this embodiment;

In the hand c of Fig. 3, the photoelectric receivers are arranged in a dispersed manner, and the photoelectric receivers are respectively installed near the fingertips of each finger and on the palm. This arrangement can capture the gestures recognized by this embodiment to the greatest extent Changes in the light signal, however, can only be used to recognize single-hand gestures due to their asymmetrical characteristics.

A gesture recognition method based on ambient light, comprising the following recognition steps:

S1: Multiple photoelectric receivers capture the light signal changes generated by gestures at different hand positions in real time, and convert the eight collected light signals into current signals respectively;

S2: The signal amplification module converts the current signal into a voltage signal and amplifies it;

S3: The amplified voltage signal is converted into a digital signal through an analog-to-digital conversion template;

S4: The eight-channel digital signals are merged and encoded by the signal processing module, and then transmitted to the gesture recognition server;

S5: The data preprocessing unit decodes the received original data and restores it to eight-channel data;

S6: The restored eight-channel data is input into the deep learning network model unit to complete the recognition and classification of gestures;

S7: At the same time, the restored eight-channel data and the recognition and classification results are used as the input of the application side, and the application side displays the eight-channel data and recognized gestures in real time.

The gesture recognition method also includes a training step before recognition, the training step is: the gesture recognition server trains through a backpropagation algorithm according to different gesture actions, and establishes a deep network model; its training principle is: the main The approach is to iteratively update the parameters of the parameter matrix through the backpropagation algorithm, mainly using multi-classification cross entropy as the loss function to perform backpropagation iterations. As shown in Figure 4, each V represents the photoelectric receiver data input to the neural network at each moment, and the output of the neural network (that is, RNN Output) is obtained after the neural network operation, and the input can be obtained after passing through the fully connected layer and the softmax function The gesture corresponding to the photoelectric receiver data; the gesture output by the algorithm is compared with the correct gesture. If it is correct, the parameters of the neural network weight matrix will not be adjusted, and if it is wrong, the neural network weight matrix will be updated through the back propagation algorithm.

It should be noted that, for the sake of simple description, all the aforementioned method embodiments are expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action sequence. Because according to the application, certain steps may be performed in other order or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification belong to preferred embodiments, and the actions and units involved are not necessarily required by this application.

Claims (7)

1. A non-contact gesture recognition device based on ambient light, comprising:

a data acquisition terminal: the system comprises 8 photoelectric receivers, a controller and a display, wherein the 8 photoelectric receivers are used for receiving an ambient light signal and generating a current signal, and the ambient light signal is generated in one or more non-contact gestures under an ambient light scene; the 8 photoelectric receivers are arranged in a trapezoidal shape and located below the hand, the upper side of the 8 photoelectric receivers is provided with 3 photoelectric receivers which are arranged at positions corresponding to the index finger, the middle finger and the ring finger of the hand, and the lower side of the 8 photoelectric receivers is provided with 5 photoelectric receivers which are transversely and uniformly distributed;

the signal amplification module: the photoelectric receiver is used for converting the current signal into a voltage signal and amplifying the voltage signal, and the input end of the signal amplification module is connected with the output end of the photoelectric receiver;

an analog-to-digital conversion module: the analog-to-digital conversion module is used for converting the voltage signal into a digital signal, and the input end of the analog-to-digital conversion module is connected with the output end of the signal amplification module;

the signal processing module: the photoelectric receivers are used for receiving the 8 photoelectric signals generated by the photoelectric receivers;

a gesture recognition server: the data preprocessing unit is used for decoding the data output by the signal processing module and restoring the data into a plurality of groups of data;

deep learning network model unit: the data processing system is used for identifying and classifying the restored groups of data in real time;

an application end: the gesture recognition server is used for displaying the one or more non-contact gestures output by the gesture recognition server in real time.

2. The device according to claim 1, wherein the deep learning network model unit is a gate control cycle unit, the deep learning network model unit is trained in advance based on a non-contact gesture interpretation data set in a historical environment light scene, and the training algorithm is a back propagation algorithm.

3. The device according to claim 1, wherein the application terminal is a front end of a web page.

4. The ambient light-based non-contact gesture recognition device according to claim 1, wherein the 8 photo receivers are arranged in a 2 x 4 rectangular array and located below the hand, and the distance between every two adjacent photo receivers is 5cm.

5. The device according to claim 1, wherein the 8 photoelectric receivers are distributed and located under the hand, and the photoelectric receivers are respectively located near the fingertip of each finger and at the palm position.

6. A non-contact gesture recognition method based on ambient light is characterized by comprising the following steps: the method comprises the following identification steps:

s1:8 photoelectric receivers capture the light signal change generated by the gesture action at different hand positions in real time, and collect a plurality of collected lights

The signals are respectively converted into current signals; the 8 photoelectric receivers are arranged in a trapezoid manner and positioned below the hand, the upper side of the 8 photoelectric receivers is provided with 3 photoelectric receivers which are arranged at positions corresponding to the index finger, the middle finger and the ring finger of the hand, and the lower side of the 8 photoelectric receivers is provided with 5 photoelectric receivers which are transversely and uniformly distributed;

s2: the signal amplification module converts the current signal into a voltage signal and amplifies the voltage signal;

s3: the amplified voltage signal is converted into a digital signal through an analog-to-digital conversion template;

s4: the multi-channel digital signals are merged and coded by the signal processing module and then transmitted to the gesture recognition server;

s5: the data preprocessing unit decodes the received original data and restores the original data into multi-channel data;

s6: inputting the restored multi-channel data into a deep learning network model unit to finish the recognition and classification of gestures;

s7: and meanwhile, the restored multi-channel data and the recognition and classification results are used as the input of the application terminal, and the application terminal displays the data of the multiple channels and recognized gestures in real time.

7. The ambient light-based non-contact gesture recognition method of claim 6, wherein: the deep learning network model unit is a gating cycle unit, and the deep learning network model unit further comprises a training step before recognition, wherein the training step comprises the following steps: and the gesture recognition server trains through a back propagation algorithm according to different gesture actions to establish a deep network model.

Images