CN112347951B - Gesture recognition method, device, storage medium and data glove - Google Patents

Abstract

The invention provides a gesture recognition method, device, storage medium and data glove. The method includes: acquiring sensor data collected by each sensor of the data glove when the data glove completes the current action, and all sensor data form an input data; using principal component analysis The feature extraction method is used to extract the input data to obtain the second feature data; the second feature data is input into the trained multi-class SVM classifier to determine the gesture corresponding to the current action; when the trained multi-class SVM classifier cannot recognize the current action , preprocess the input data to obtain the preprocessed data; input the preprocessed data into the trained gesture recognition model, and output the gesture corresponding to the current action. The gesture recognition model is based on convolutional neural network and long-term and short-term Memory recurrent network built. The technical solution of the present invention can ensure the accuracy of gesture recognition while increasing the speed of gesture recognition.

Description

Gesture recognition method, device, storage medium and data glove

technical field

The present invention relates to the technical field of gesture recognition, in particular to a gesture recognition method, device, storage medium and data glove.

Background technique

Sign language is to use gestures to measure movements, and to simulate images or syllables according to gesture changes to form certain meanings or words. It is a hand language for people who are hearing impaired or unable to speak to communicate and exchange ideas. Therefore, recognizing gestures is very important for communicating with sign language users. At present, the following two methods are often used to recognize gestures.

One is to use a camera to capture gestures, analyze the captured pictures, and recognize gestures. However, the camera has high requirements for light in the process of capturing gestures, and poor lighting conditions will affect the accuracy of gesture recognition.

The other is to obtain the hand surface electromyogram when the hand completes the gesture, and then analyze and process the electromyogram to recognize the gesture, but the existing algorithm for recognizing gestures based on the electromyogram is relatively complicated and inefficient .

Contents of the invention

The problem solved by the present invention is how to balance the efficiency and accuracy of gesture recognition.

To solve the above problems, the present invention provides a gesture recognition method, device, storage medium and data glove.

In a first aspect, the present invention provides a gesture recognition method, including:

Obtain the sensor data collected by each sensor of the data glove when the data glove completes the current action, and all the sensor data form an input data;

performing feature extraction on the input data by principal component analysis to obtain second feature data;

Inputting the second feature data into the trained multi-class SVM classifier to determine the gesture corresponding to the current action;

When the trained multi-class SVM classifier cannot recognize the current action, preprocessing the input data to obtain preprocessed data;

Input the preprocessed data into the trained gesture recognition model, and output the gesture corresponding to the current action, wherein the gesture recognition model is established based on a convolutional neural network and a long-term short-term memory cycle network.

Further, before inputting the second feature data into the trained multi-class SVM classifier, it includes:

respectively acquiring the input data when the data glove completes different calibration actions;

respectively amplifying and filtering each of the input data to obtain filtered input data;

performing feature extraction on all the filtered input data by using principal component analysis to obtain the first feature data;

Using the first feature data to train a multi-class SVM classifier to obtain the trained multi-class SVM classifier.

Further, the feature extraction of all the filtered input data by using principal component analysis method includes:

calculating an average of all said filtered input data;

separately determining the difference between each of said filtered input data and said mean value, and determining a covariance matrix based on all said differences;

Calculate eigenvalues and eigenvectors according to the covariance matrix, and determine a principal component matrix according to the eigenvectors;

The first feature data is determined according to the principal component matrix and the difference.

Further, the calibration action is in one-to-one correspondence with the gesture template, and the training of multi-class SVM classifiers using the feature data includes:

For any of the gesture templates, the first feature data corresponding to the gesture template is used as a positive set, and the first feature data other than the positive set is used as a negative set, and the corresponding positive set and all The above negative set is a training set;

The training set is input into the multi-class SVM classifier, the multi-class SVM classifier includes a plurality of classification functions, each of the classification functions processes the training set respectively, and outputs one of the first classifications respectively value;

determining the maximum value of the first classification value and the classification function corresponding to the maximum value, and corresponding the classification function to the gesture template;

Each of the first feature data is processed sequentially, and the gesture templates are in one-to-one correspondence with the classification functions.

Further, inputting the second feature data into the trained multi-class SVM classifier, and determining the gesture corresponding to the current action includes:

Inputting the second feature data into the trained multi-class SVM classifier, each of the classification functions processes the second feature data respectively, and outputs a second classification value respectively;

determining the maximum value and the second maximum value among all the second classification values, comparing the maximum value and the second maximum value among the second classification values with preset thresholds, and when the maximum value is greater than or equal to When the preset threshold is smaller than the preset threshold, it is determined that the gesture template corresponding to the classification function that outputs the maximum value is the gesture corresponding to the current action.

Further, the failure of the trained multi-class SVM classifier to identify the current action includes that both the maximum value and the second maximum value of the second classification value are greater than or equal to the preset threshold.

Further, before inputting the preprocessed data into the trained gesture recognition model, it includes:

Acquire the input data when the data glove completes different calibration actions, each of the input data includes all the sensor data corresponding to one gesture template;

performing preprocessing on all the input data respectively to obtain the preprocessed data;

Constructing a gesture recognition model based on a convolutional neural network and a long-term short-term memory cycle network, using the preprocessed data to train the gesture recognition model, and obtaining the trained gesture recognition model.

Further, each of the input data includes all the sensor data corresponding to one of the calibration actions, and performing preprocessing on all the input data respectively, and obtaining the preprocessed data includes:

For any of the input data, a time synchronization mechanism based on time slot channel hopping is used to synchronize all sensor data corresponding to the input data to obtain synchronized sensor data;

filtering the synchronized sensor data by using a Butterworth bandpass filter to obtain filtered sensor data;

A sliding window is used to intercept the filtered sensor data to obtain multiple data segments, and all the data segments form the preprocessed data.

Further, the training of the gesture recognition model using the preprocessed data includes a forward propagation step, and the forward propagation step includes:

Input the preprocessed data into the gesture recognition model, and output the probability that the calibration action is each gesture template;

The gesture template with the highest probability is determined to be the predicted gesture.

Further, the gesture recognition model includes two one-dimensional convolutional layers, a maximum pooling layer, a flattening layer, an LSTM layer, two fully connected layers, a Softmax layer and an output layer connected in sequence, and the preprocessing After the data is input into the gesture recognition model, the probability of outputting the calibration action as each gesture template includes:

The preprocessed data is input into the first one-dimensional convolution layer, and the two one-dimensional convolution layers perform feature extraction on the preprocessed data to obtain the third feature data, and all the The third feature data constitutes a feature map;

Inputting the feature map into the maximum pooling layer, performing feature extraction on each sub-region of the feature map, and obtaining fourth feature data;

Inputting the fourth feature data into the flattening layer, shaping the fourth feature data into a one-dimensional vector, inputting the one-dimensional vector into the LSTM layer for processing, and outputting the processed data;

The processed data is sequentially input into the two fully connected layers and the Softmax layer, and the output of the calibration action is the probability of each of the gesture templates;

The output layer outputs the gesture template with the highest probability, and the gesture template with the highest probability is the predicted gesture.

Further, the gesture recognition model also includes a plurality of drop-off layers, wherein two of the drop-off layers are arranged between the second one-dimensional convolutional layer and the maximum pooling layer, and two of the drop-off layers It is set between the LSTM layer and the first fully connected layer.

Further, the training of the gesture recognition model using the preprocessed hand data further includes:

The backpropagation step includes performing a cross-entropy loss according to the calibration action and the predicted gesture, and optimizing the gesture recognition model according to the loss;

The forward propagation step and the backward propagation step are repeated cyclically until the loss no longer decreases, and a stable gesture recognition model is obtained.

In a second aspect, the present invention provides a gesture recognition device, comprising:

The acquisition module is used to acquire the sensor data collected by each sensor of the data glove when the data glove completes the current action, and all the sensor data form an input data;

A feature extraction module, configured to extract features from the input data using principal component analysis to obtain second feature data;

A first processing module, configured to input the second feature data into a trained multi-class SVM classifier to determine the gesture corresponding to the current action;

A preprocessing module, configured to preprocess the input data to obtain preprocessed data when the trained multi-class SVM classifier cannot recognize the current action;

The second processing module is used to input the preprocessed data into the trained gesture recognition model, and output the gesture corresponding to the current action, wherein the gesture recognition model is based on convolutional neural network and long short-term memory cycle network built.

In a third aspect, the present invention provides a gesture recognition device, including a memory and a processor;

The memory is used to store computer programs;

The processor is configured to implement the above gesture recognition method when executing the computer program.

In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned gesture recognition method is realized.

In a fifth aspect, the present invention provides a data glove, including a glove, a plurality of sensors, and the gesture recognition device as described above, a plurality of the sensors are respectively electrically connected to the gesture recognition device, and a plurality of the sensors are respectively It is arranged on the glove and is suitable for detecting motion data of each finger.

Further, the sensor includes a plurality of fully flexible capacitive sensors and/or a plurality of piezoresistive sensors;

A plurality of the fully flexible capacitive sensors are respectively arranged at the back of each finger and the back of the thumb at the tiger's mouth of the glove, and are respectively used to measure the motion data during the flexion or extension of the fingers, and the motion data during the lateral motion of the thumb ;

A plurality of the piezoresistive sensors are respectively arranged at the joints inside each finger of the glove and between two adjacent fingers, and are respectively used to measure the movement data of each finger joint and the abduction or adduction of the fingers. motion data.

The beneficial effects of the gesture recognition method, device, storage medium and data glove of the present invention are: the sensor data collected by each sensor when the data glove completes the current action is acquired, and all the sensor data corresponding to the current action form one input data. The principal component analysis method is used to extract the features of the input data, and the feature data is obtained and input to the trained multi-class SVM classifier to recognize the gesture corresponding to the current action. The multi-class SVM classifier is fast, simple and efficient. When the multi-class SVM classifier is difficult to recognize the current action, the input data corresponding to the current action is preprocessed, and the preprocessed data is input into the trained gesture recognition model, and the gesture corresponding to the current action is output. The gesture recognition model is based on Deep learning model construction, which can accurately recognize gestures. The technical solution of the present invention first uses multi-class SVM classifiers to recognize gestures, and the recognition speed is fast. When multi-class SVM classifiers are difficult to recognize gestures, a gesture recognition model is used to recognize gestures, which can ensure recognition accuracy while improving recognition speed.

Description of drawings

Fig. 1 is a schematic diagram of the back structure of a data glove according to an embodiment of the present invention;

Fig. 2 is a schematic diagram of the front structure of a data glove according to an embodiment of the present invention;

Fig. 3 is a schematic diagram of a circuit connection of a data glove according to an embodiment of the present invention;

Fig. 4 is a schematic diagram of sensor signals under different gestures according to an embodiment of the present invention;

5 is a schematic flow diagram of a multi-class SVM classifier training method according to an embodiment of the present invention;

6 is a schematic flow chart of a gesture recognition model training method according to an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of a gesture recognition model according to an embodiment of the present invention

FIG. 8 is a schematic flowchart of a gesture recognition method according to an embodiment of the present invention;

FIG. 9 is a schematic structural diagram of a gesture recognition model according to an embodiment of the present invention.

Explanation of reference signs:

10-glove; 20-fully flexible capacitive sensor; 30-piezoresistive sensor; 40-base.

Detailed ways

In order to make the above objects, features and advantages of the present invention more comprehensible, specific embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.

It should be noted that the terms "first" and "second" in the description and claims of the present invention and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein.

As shown in Figures 1 and 2, a data glove provided by the present invention includes a glove 10, a plurality of sensors and a gesture recognition device as described below, a plurality of the sensors are respectively electrically connected to the gesture recognition device, And a plurality of the sensors are respectively arranged on the glove 10, and are suitable for detecting motion data of each finger.

Preferably, the sensor includes a plurality of fully flexible capacitive sensors 20 and/or a plurality of piezoresistive sensors 30;

A plurality of fully flexible capacitive sensors 20 are respectively arranged at the back of each finger and the back of the thumb of the glove 10, and are respectively used to measure the motion data during the flexion or extension of the fingers, and the motion data during the lateral movement of the thumb. sports data;

A plurality of piezoresistive sensors 30 are respectively arranged at the joints inside each finger of the glove 10 and between two adjacent fingers, and are respectively used to measure the movement data of each finger joint and the abduction or adduction of the fingers. Received exercise data.

Specifically, as shown in FIG. 3 , the output terminals of the piezoresistive sensors 30 or the fully flexible capacitive sensors 20 at each fingertip on the data glove are electrically connected to the input terminals of the amplifier, and the output terminals of the amplifier are electrically connected to the input terminals of the filter. connected, the output end of the filter is electrically connected to the input end of the AD conversion processor, and the output end of the AD conversion processor is electrically connected to the communication device. Among them, the amplifier is used to amplify the signal detected by the sensor, the filter is used to filter the amplified signal, the AD conversion processor is used to convert the filtered signal into a digital signal, and the communication device is used to transmit the converted digital signal to the host Machine and so on to process and recognize gestures.

Amplifiers, filters, AD conversion processors and communication devices can be integrated on a circuit board, and the circuit board can be installed on any convenient wearing and moving position through the base 40 as required, such as: the back of the wrist, the back of the palm of the data glove, the small On the arm and on the boom etc., the base 40 can be manufactured by 3D printing.

When the piezoresistive sensor 30 detects pressure, or the fully flexible capacitive sensor 20 detects stretch, the voltage signal changes. As shown in Figure 4, when the data glove adopts a fully flexible capacitive sensor 20, when multiple fingers move at the same time, the fully flexible capacitive sensor 20 on each fingertip records the movement data of each finger respectively, and the movement data of all fingers under one gesture The motion data is combined into one input data, and the data acquisition channels of the five fingers are independent of each other. When a finger moves, the voltage signal detected by the sensor changes, generating a voltage peak. For example, in Figure 4, the thumb corresponding to the gesture "A" does not move, and no voltage peak is generated, and the other four fingers bend to generate a voltage peak respectively; the thumb, index finger and middle finger corresponding to gesture "3" do not move, and no voltage is generated peaks, while the ring finger and little finger flex, respectively, to produce voltage peaks.

As shown in Figure 5, a kind of multiclass SVM (Support Vector Machine, support vector machine) classifier training method that the embodiment of the present invention provides, comprises:

Step 110, respectively acquiring the input data when the data glove completes different calibration actions;

Step 120, respectively amplifying and filtering each of the input data to obtain filtered input data;

Step 130, using principal component analysis to perform feature extraction on all the filtered input data to obtain the first feature data;

Step 140, using the first feature data to train a multi-class SVM classifier to obtain the trained multi-class SVM classifier.

In this embodiment, the input data is amplified and filtered to reduce noise data. The principal component analysis method is used to extract the feature data in the input data corresponding to each gesture template, which is used to train a multi-class SVM classifier, which can improve the training speed. A trained multi-class SVM classifier is able to classify and recognize gestures.

Preferably, the feature extraction of all the filtered input data using the principal component analysis method includes:

Compute the average of all said filtered input data.

Specifically, the filtered input data corresponding to each gesture template is combined into a set, S={s ₁ , s ₂ , s ₃ ...s _n }, where s _i is the filtered input data corresponding to the i-th gesture template data. The average value of all filtered input data is calculated using a first formula comprising:

Wherein, n is the number of gesture templates, and S _avg is the average value of all filtered input data.

Differences between each of said filtered input data and said mean value are respectively determined, and a covariance matrix is determined from all said differences.

Specifically, a second formula is used to determine the difference between each of the filtered input data and the average value, and the second formula includes:

δ _i =s _i -S _avg ,

Wherein, δ _i is the difference between the filtered input data corresponding to the i-th gesture template and the average value.

The third formula is adopted to calculate the covariance matrix, and the third formula includes:

Among them, M is the covariance matrix, and δ _i ^T is the transpose of the difference vector δ _i .

Calculate eigenvalues and eigenvectors according to the covariance matrix, and determine a principal component matrix according to the eigenvectors;

Specifically, the fourth formula is used to calculate the eigenvalue and the eigenvector, and the fourth formula includes:

M×v _i =λ _i ×v _i (i=1,2,3...k),

Among them, (λ ₁ ,λ ₂ ,λ ₃ ...λ _k ) are eigenvalues respectively, (v ₁ ,v ₂ ,v ₃ ...v _k ) are eigenvectors, and the principal component matrix is V={v ₁ , v ₂ , v ₃ ... v _k }.

The first feature data is determined according to the principal component matrix and the difference.

Specifically, adopt the fifth formula to project the input data into the principal component matrix, and the fifth formula includes:

y _i =V ^T ×δ _i ,

Among them, y _i is the feature data corresponding to the difference vector δ _i .

Preferably, the calibration action is in one-to-one correspondence with the gesture template, and the training of a multi-class SVM classifier using the feature data includes:

For any of the gesture templates, the first feature data corresponding to the gesture template is used as a positive set, and the first feature data other than the positive set is used as a negative set, and the corresponding positive set and all The above negative set is a training set;

The training set is input into the multi-class SVM classifier, the multi-class SVM classifier includes a plurality of classification functions, each of the classification functions processes the training set respectively, and outputs one of the first classifications respectively value;

determining the maximum value of the first classification value and the classification function corresponding to the maximum value, and corresponding the classification function to the gesture template;

Each of the first feature data is processed sequentially, and the gesture templates are in one-to-one correspondence with the classification functions.

Specifically, the multi-class SVM classifier uses a one-to-all support vector machine to obtain the input data corresponding to the gesture template to train the multi-class SVM classifier. The multi-class SVM classifier includes multiple classification functions, and how many gesture templates are constructed. A classification function, a classification function of the multi-class SVM classifier can distinguish the corresponding gesture template from other gesture templates, and the classification function corresponds to the gesture template one by one. When the trained multi-class SVM classifier is used to recognize gestures, after determining the classification function corresponding to the hand movement, the gesture template can be quickly determined, and the gesture corresponding to the hand movement can be recognized, which is simple, efficient and fast.

As shown in FIG. 6, a method for training a gesture recognition model provided by an embodiment of the present invention includes:

Step 210, acquiring the input data when the data glove completes different calibration actions, each input data includes all the sensor data corresponding to one gesture template;

Step 220, performing preprocessing on all the input data respectively to obtain the preprocessed data;

Step 230, constructing a gesture recognition model based on a convolutional neural network and a long-term short-term memory cycle network, using the preprocessed data to train the gesture recognition model, and obtaining the trained gesture recognition model.

In this embodiment, the input data when the data glove completes the calibration actions corresponding to different gesture templates is obtained, and the gesture recognition model is trained using the input data. less data. When the gesture recognition model based on deep learning is used to recognize gestures, the accuracy is high.

Preferably, each of the input data includes all the sensor data corresponding to one of the calibration actions, and performing preprocessing on all the input data respectively, and obtaining the preprocessed data includes:

For any of the input data, a time synchronization mechanism based on time-slotted channel hopping-based (TSCH) is used to synchronize all sensor data corresponding to the input data to obtain synchronized sensor data;

The synchronized sensor data is filtered by a Butterworth bandpass filter to obtain filtered sensor data.

Specifically, since the human hand movement frequency is usually lower than 10 Hz, after the data glove acquires the sensor data, it uses a Butterworth bandpass filter with a cutoff frequency of 0.5 Hz to 10 Hz for filtering, which can remove irrelevant energy. And, due to strong noise from biometrics and data glove wearing loose fold errors, the similarity of the same activity of the collected sensor data may be low, and the average Pearson (Pearson) correlation coefficient between sensor data of the same activity is about 0.2-0.4. A Butterworth bandpass filter can increase similarity by taking out noise from other channels, increasing the Pearson correlation of the same activity to 0.7-0.8.

A sliding window is used to intercept the filtered sensor data to obtain multiple data segments, and all the data segments form the preprocessed data.

Specifically, a sliding window with a fixed time of 4 seconds can be used to intercept filtered sensor data, and there is a partial overlap of 50% between two adjacent sliding windows. If the sampling frequency of sensor data is 128Hz, combined with the East China window Design, relative to having 512 samples at 128Hz sampling frequency, will reshape the obtained time series data into a 2D matrix of 16×32=512 samples, and use this 2D matrix as the input of the gesture recognition model.

Preferably, the training of the gesture recognition model using the preprocessed hand data includes a forward propagation step, and the forward propagation step includes:

Input the preprocessed data into the gesture recognition model, and output the probability that the calibration action is each gesture template;

The gesture template with the highest probability is determined to be the predicted gesture.

Preferably, as shown in Figure 7, the gesture recognition model includes two one-dimensional convolutional layers (Conv1D), a maximum pooling layer (Max Pooling), a flattening layer (flatten layer), an LSTM (Long Short -TermMemory, long-term short-term memory cycle network) layer, two fully connected layers (Fully Connected Layer), Softmax layer and output layer (Output Layer), the data after the described preprocessing is input into the gesture recognition model, output The probability that the calibration action is each gesture template includes:

The preprocessed data is input into the first one-dimensional convolution layer, and the two one-dimensional convolution layers perform feature extraction on the preprocessed data to obtain the third feature data, and all the The third feature data constitutes a feature map.

Specifically, the preprocessed data is reshaped from one-dimensional time series data to a two-dimensional matrix to meet the input size requirements of the next-dimensional convolutional layer in the Tensorflow Keras (machine learning framework) computing framework, where one dimension is the time step , and the other dimension is the features at each time step. Using two one-dimensional convolutional layers can improve the robustness of the extracted features. The one-dimensional convolutional layer is a variant of CNN (Convolutional Neural Networks, convolutional neural network), which is specially used to process sequence and time series data. In a 1D convolutional layer, the convolutional filters only move along the time direction of the data, thus, a 1D convolutional layer is able to derive features from fixed-length segments of data. When applied to recognize gestures, CNN has two advantages over other models, local dependence and scale invariance. Local dependence means that nearby signals are likely to be related, while scale invariance means that the scales of different steps or frequencies are different. Change.

Inputting the feature map into the maximum pooling layer, performing feature extraction on each sub-region of the feature map, and obtaining fourth feature data.

Specifically, after the convolution is completed, the maximum pooling layer is applied to extract the most important features from each region in the feature map output by the one-dimensional convolutional layer, which can reduce the number of features and speed up the training process.

Inputting the fourth feature data into the flattening layer, shaping the fourth feature data into a one-dimensional vector, inputting the one-dimensional vector into the LSTM layer for processing, and outputting the processed data.

Specifically, since the output processed by the one-dimensional convolution layer and the maximum pooling layer is a two-dimensional matrix, and the input size of the LSTM layer is required to be a one-dimensional vector, the flattening layer is used to convert the second output of the maximum pooling layer The feature data is re-flattened into a one-dimensional vector, which is input to the LSTM layer as one-dimensional time series data. The flattening layer can be regarded as a bridging layer between CNN and LSTM layers, which is used to convert two-dimensional data into one-dimensional data and unify LSTM layers without losing information.

The processed data is sequentially input into the two fully connected layers and the Softmax layer, and the probability that the calibration action is each of the gesture templates is output.

Specifically, the rectified linear unit (ReLu) activation is used in the first fully connected layer, and the Softmax activation is used in the second fully connected layer to output the category label, which is gesture. ReLu is used as an activation function in the deep learning model, which can meet the requirements of fast convergence and solve the problem of gradient disappearance, mainly because its gradient is non-saturated, which can greatly speed up the convergence speed of gradient descent, and pass the gradient to 0 or 1 to solve the problem of gradient disappearance.

The output layer outputs the gesture template with the highest probability, and the gesture template with the highest probability is the predicted gesture.

Specifically, the Softmax layer is often used in the last layer of the model, and is an activation function commonly used in classification problems. The action corresponding to the output gesture template is the probability of each category label, and the sum of the probabilities of all category labels is 1, of which the highest probability is The class label is the model's predicted class label, which is the model's predicted gesture.

Preferably, the gesture recognition model further includes a plurality of dropout layers, wherein two dropout layers are arranged between the second one-dimensional convolutional layer and the maximum pooling layer, and the two dropout layers are arranged between the second one-dimensional convolutional layer and the maximum pooling layer. The first exfoliation layer is disposed between the LSTM layer and the first fully connected layer.

Specifically, training a neural network with a relatively small data set can lead to overfitting of the training data because the model learns the statistical noise in the training data, and when the trained model is evaluated on test or new data, it will perform poorly. poor performance. Therefore, in order to prevent overfitting and reduce generalization errors, a shedding layer is introduced into the deep learning framework, and the model learns robust features. The shedding rate of the shedding layer can be set to 0.5, which means that the shedding layer randomly selects 50% of the input units and is set to zero.

Preferably, the training of the gesture recognition model using the preprocessed hand data further includes:

The backpropagation step includes performing a cross-entropy loss according to the calibration action and the predicted gesture, and optimizing the gesture recognition model according to the loss;

The forward propagation step and the backward propagation step are repeated cyclically until the loss no longer decreases, and a stable gesture recognition model is obtained.

Specifically, according to the loss, the parameters of the gesture recognition model are driven and optimized. After multiple iterative optimizations of the gesture recognition model, when the loss no longer decreases, the gesture recognition model reaches a stable state.

As shown in Figure 8, a gesture recognition method provided by an embodiment of the present invention includes:

Step 310, acquire the sensor data collected by each sensor of the data glove when the data glove completes the current action, and all the sensor data form an input data;

Step 320, using principal component analysis to perform feature extraction on the input data to obtain second feature data;

Step 330: Input the second feature data into the trained multi-class SVM classifier to determine the gesture corresponding to the current action.

Specifically, the second feature data is input into a trained multi-class SVM classifier, and the trained multi-class SVM classifier includes a plurality of classification functions, and each of the classification functions processes the feature data respectively, respectively A second classification value is output, and the classification function is in one-to-one correspondence with the gesture template.

determining a maximum value and a second maximum value among all the second classification values, and comparing the maximum value and the second maximum value with preset thresholds, and when the maximum value is greater than or equal to the preset threshold and When the second largest value is smaller than the preset threshold, it is determined that the gesture template corresponding to the classification function that outputs the largest value is the gesture corresponding to the current action.

Step 340, when the trained multi-class SVM classifier cannot recognize the current action, perform preprocessing on the input data to obtain preprocessed data.

Specifically, when both the maximum value and the second maximum value are greater than or equal to the preset threshold, preprocessing is performed on the input data to obtain preprocessed data.

Step 350, input the preprocessed data into the trained gesture recognition model, and output the gesture corresponding to the current action, wherein the gesture recognition model is established based on a convolutional neural network and a long-term short-term memory cycle network.

In this embodiment, the sensor data collected by each sensor when the data glove completes the current action is obtained, and all the sensor data corresponding to the current action form one input data. The principal component analysis method is used to extract the features of the input data, and the feature data is obtained and input to the trained multi-class SVM classifier to recognize the gesture corresponding to the current action. The multi-class SVM classifier is fast, simple and efficient. When the multi-class SVM classifier is difficult to recognize the current action, the input data corresponding to the current action is preprocessed, and the preprocessed data is input into the trained gesture recognition model, and the gesture corresponding to the current action is output. The gesture recognition model is based on Deep learning model construction, which can accurately recognize gestures. The technical solution of the present invention first uses multi-class SVM classifiers to recognize gestures, and the recognition speed is fast. When multi-class SVM classifiers are difficult to recognize gestures, a gesture recognition model is used to recognize gestures, which can ensure recognition accuracy while improving recognition speed.

As shown in Figure 9, a gesture recognition device provided by an embodiment of the present invention includes:

The acquisition module is used to acquire the sensor data collected by each sensor of the data glove when the data glove completes the current action, and all the sensor data form an input data;

A feature extraction module, configured to extract features from the input data using principal component analysis to obtain second feature data;

A first processing module, configured to input the second feature data into a trained multi-class SVM classifier to determine the gesture corresponding to the current action;

A preprocessing module, configured to preprocess the input data to obtain preprocessed data when the trained multi-class SVM classifier cannot recognize the current action;

The second processing module is used to input the preprocessed data into the trained gesture recognition model, and output the gesture corresponding to the current action, wherein the gesture recognition model is based on convolutional neural network and long short-term memory cycle network built.

A gesture recognition device provided by another embodiment of the present invention includes a memory and a processor; the memory is used to store a computer program; and the processor is used to realize the gesture as described above when the computer program is executed recognition methods. The device can be a computer, a server, and the like.

Still another embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored on the storage medium, and when the computer program is executed by a processor, the above-mentioned gesture recognition method is implemented.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented through computer programs to instruct related hardware, and the programs can be stored in a computer-readable storage medium. During execution, it may include the processes of the embodiments of the above-mentioned methods. Wherein, the storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM) or a random access memory (Random Access Memory, RAM) and the like. In this application, the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple on a network unit. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present invention. In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

Although the disclosure of the present invention is as above, the protection scope of the disclosure of the present invention is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will all fall within the protection scope of the present invention.

Claims (13)

1. A method of gesture recognition, comprising:

acquiring sensor data acquired by each sensor of the data glove when the data glove finishes the current action, wherein all the sensor data form one input data;

performing feature extraction on the input data by adopting a principal component analysis method to obtain second feature data;

inputting the second characteristic data into a trained multi-class SVM classifier, and determining a gesture corresponding to the current action;

when the trained multi-class SVM classifier cannot recognize the current action, preprocessing the input data to obtain preprocessed data;

inputting the preprocessed data into a trained gesture recognition model, and outputting a gesture corresponding to the current action, wherein the gesture recognition model is established based on a convolutional neural network and a long-term and short-term memory cyclic network;

Before the second characteristic data is input into the trained multi-class SVM classifier, the method comprises the following steps: respectively acquiring the input data of the data glove when different calibration actions are completed; amplifying and filtering each input data respectively to obtain filtered input data; performing feature extraction on all the filtered input data by adopting a principal component analysis method to obtain first feature data; training a multi-class SVM classifier by adopting the first characteristic data to obtain the trained multi-class SVM classifier;

the feature extraction of all the filtered input data by adopting the principal component analysis method comprises the following steps: calculating the average value of all the filtered input data; respectively determining differences between the filtered input data and the average value, and determining a covariance matrix according to all the differences; calculating a characteristic value and a characteristic vector according to the covariance matrix, and determining a principal component matrix according to the characteristic vector; determining the first feature data from the principal component matrix and the difference value;

the calibrating actions are in one-to-one correspondence with the gesture templates, and the training the multi-class SVM classifier by adopting the first characteristic data comprises the following steps: for any gesture template, taking the first characteristic data corresponding to the gesture template as a positive set, taking the first characteristic data except the positive set as a negative set, and taking the corresponding positive set and negative set as a training set; inputting the training set into the multi-class SVM classifier, wherein the multi-class SVM classifier comprises a plurality of classification functions, each classification function respectively processes the training set and respectively outputs a first classification value; determining a maximum value in the first classification value and the classification function corresponding to the maximum value, and corresponding the classification function to the gesture template; processing the first characteristic data in sequence, and enabling the gesture templates to correspond to the classification functions one by one;

Inputting the second characteristic data into a trained multi-class SVM classifier, wherein determining the gesture corresponding to the current action comprises: inputting the second characteristic data into the trained multi-class SVM classifier, wherein each classification function respectively processes the second characteristic data and respectively outputs a second classification value; and determining the maximum value and the next largest value in all the second classification values, comparing the maximum value and the next largest value in the second classification values with a preset threshold value respectively, and determining the gesture template corresponding to the classification function outputting the maximum value as the gesture corresponding to the current action when the maximum value is greater than or equal to the preset threshold value and the next largest value is smaller than the preset threshold value.

2. The gesture recognition method of claim 1, wherein the inability of the trained multi-class SVM classifier to recognize the current action includes a maximum value and a next-largest value of the second classification values being greater than or equal to the preset threshold.

3. The gesture recognition method according to claim 1 or 2, wherein before inputting the preprocessed data into the trained gesture recognition model, comprising:

Respectively acquiring the input data of the data glove when different calibration actions are completed;

preprocessing all the input data respectively to obtain preprocessed data;

and constructing a gesture recognition model based on a convolutional neural network and a long-term and short-term memory cyclic network, and training the gesture recognition model by adopting the preprocessed data to obtain the trained gesture recognition model.

4. A gesture recognition method according to claim 3, wherein each of the input data includes all of the sensor data corresponding to one of the calibration actions, the preprocessing of all of the input data, respectively, includes:

for any input data, synchronizing all sensor data corresponding to the input data by adopting a time synchronization mechanism based on time slot channel hopping to obtain synchronized sensor data;

filtering the synchronized sensor data by using a Butterworth band-pass filter to obtain filtered sensor data;

and intercepting the filtered sensor data by adopting a sliding window to obtain a plurality of data segments, wherein all the data segments form the preprocessed data.

5. The method of claim 4, wherein training the gesture recognition model using the preprocessed data comprises a forward propagating step that includes:

inputting the preprocessed data into the gesture recognition model, and outputting the probability that the calibration action is each gesture template;

and determining the gesture template with the highest probability as a predicted gesture.

6. The gesture recognition method of claim 5, wherein the gesture recognition model comprises two one-dimensional convolution layers, a max pooling layer, a flattening layer, an LSTM layer, two full connection layers, a Softmax layer, and an output layer connected in sequence, the inputting the preprocessed data into the gesture recognition model, and outputting the probability that the calibration action is each gesture template comprises:

inputting the preprocessed data into a first one-dimensional convolution layer, and carrying out feature extraction on the preprocessed data by two one-dimensional convolution layers to obtain third feature data, wherein all the third feature data form a feature map;

inputting the feature map into the maximum pooling layer, and extracting features of each sub-region of the feature map to obtain fourth feature data;

Inputting the fourth characteristic data into the flattening layer, shaping the fourth characteristic data into a one-dimensional vector, inputting the one-dimensional vector into the LSTM layer for processing, and outputting processed data;

inputting the processed data into the two full-connection layers and the Softmax layer in sequence, and outputting the probability that the calibration action is each gesture template;

and the output layer outputs the gesture template with the highest probability, and the gesture template with the highest probability is the predicted gesture.

7. The method of claim 6, wherein the gesture recognition model further comprises a plurality of shedding layers, wherein two of the shedding layers are disposed between a second one of the one-dimensional convolution layers and the max pooling layer, and two of the shedding layers are disposed between the LSTM layer and a first one of the fully-connected layers.

8. The method of claim 7, wherein training the gesture recognition model using the preprocessed hand data further comprises:

a back propagation step, comprising cross entropy loss according to the calibration action and the predicted gesture, and optimizing the gesture recognition model according to the loss;

And repeating the forward propagation step and the backward propagation step in a circulating way until the loss is not reduced any more, and obtaining a stable gesture recognition model.

9. A gesture recognition apparatus, comprising:

the acquisition module is used for acquiring sensor data acquired by each sensor of the data glove when the data glove finishes the current action, and all the sensor data form one input data;

the feature extraction module is used for carrying out feature extraction on the input data by adopting a principal component analysis method to obtain second feature data;

the first processing module is used for inputting the second characteristic data into a trained multi-class SVM classifier and determining a gesture corresponding to the current action;

the preprocessing module is used for preprocessing the input data to obtain preprocessed data when the trained multi-class SVM classifier cannot recognize the current action;

the second processing module is used for inputting the preprocessed data into a trained gesture recognition model and outputting a gesture corresponding to the current action, wherein the gesture recognition model is built based on a convolutional neural network and a long-term and short-term memory cyclic network;

The first processing module is specifically configured to: respectively acquiring the input data of the data glove when different calibration actions are completed; amplifying and filtering each input data respectively to obtain filtered input data; performing feature extraction on all the filtered input data by adopting a principal component analysis method to obtain first feature data; training a multi-class SVM classifier by adopting the first characteristic data to obtain the trained multi-class SVM classifier;

the feature extraction module is specifically configured to: calculating the average value of all the filtered input data; respectively determining differences between the filtered input data and the average value, and determining a covariance matrix according to all the differences; calculating a characteristic value and a characteristic vector according to the covariance matrix, and determining a principal component matrix according to the characteristic vector; determining the first feature data from the principal component matrix and the difference value;

the calibration actions are in one-to-one correspondence with the gesture templates, and the first processing module is specifically further configured to: for any gesture template, taking the first characteristic data corresponding to the gesture template as a positive set, taking the first characteristic data except the positive set as a negative set, and taking the corresponding positive set and negative set as a training set; inputting the training set into the multi-class SVM classifier, wherein the multi-class SVM classifier comprises a plurality of classification functions, each classification function respectively processes the training set and respectively outputs a first classification value; determining a maximum value in the first classification value and the classification function corresponding to the maximum value, and corresponding the classification function to the gesture template; processing the first characteristic data in sequence, and enabling the gesture templates to correspond to the classification functions one by one;

The first processing module is specifically further configured to: inputting the second characteristic data into the trained multi-class SVM classifier, wherein each classification function respectively processes the second characteristic data and respectively outputs a second classification value; and determining the maximum value and the next largest value in all the second classification values, comparing the maximum value and the next largest value in the second classification values with a preset threshold value respectively, and determining the gesture template corresponding to the classification function outputting the maximum value as the gesture corresponding to the current action when the maximum value is greater than or equal to the preset threshold value and the next largest value is smaller than the preset threshold value.

10. A gesture recognition apparatus comprising a memory and a processor;

the memory is used for storing a computer program;

the processor being configured to implement the gesture recognition method of any one of claims 1 to 8 when the computer program is executed.

11. A computer readable storage medium, characterized in that the storage medium has stored thereon a computer program which, when executed by a processor, implements the gesture recognition method according to any of claims 1 to 8.

12. A data glove comprising a glove, a plurality of sensors and a gesture recognition device according to claim 10, wherein the plurality of sensors are respectively electrically connected with the gesture recognition device, and the plurality of sensors are respectively arranged on the glove and are suitable for detecting movement data of each finger.

13. The data glove of claim 12, wherein the sensor comprises a plurality of fully flexible capacitive sensors and/or a plurality of piezoresistive sensors;

the plurality of fully flexible capacitive sensors are respectively arranged at the back of each finger of the glove and the back tiger mouth of the thumb and are respectively used for measuring movement data in the process of buckling or stretching of the finger and movement data in the process of transverse movement of the thumb;

the piezoresistive sensors are respectively arranged at joints on the inner sides of the fingers of the glove and between two adjacent fingers and are respectively used for measuring motion data of each finger joint and motion data of abduction or adduction of the fingers.

Images