CN115343704A - Hand gesture recognition method for FMCW millimeter wave radar based on multi-task learning - Google Patents

Abstract

The invention provides a gesture recognition method of an FMCW millimeter wave radar based on multitask learning, which comprises the following steps: acquiring a target characteristic vector time sequence of a gesture to be recognized; inputting the target characteristic vector time sequence into a multi-task learning gesture recognition model, and outputting a gesture classification result; the multi-task learning gesture recognition model is obtained by training a sample target characteristic vector time sequence, a corresponding gesture category label and a corresponding gesture track label, and the target characteristic vector time sequence and the sample target characteristic vector time sequence are obtained by processing gesture recognition original data acquired by an FMCW millimeter wave radar through a preset algorithm. The multi-dimensional characteristics of the user gestures are effectively and conveniently acquired, and the accuracy of gesture recognition classification is improved through multi-task learning.

Description

Hand gesture recognition method for FMCW millimeter wave radar based on multi-task learning

technical field

The present invention relates to the technical field of gesture recognition, in particular to a gesture recognition method for FMCW (Frequency Modulated Continuous Wave, Frequency Modulated Continuous Wave) millimeter-wave radar based on multi-task learning.

Background technique

In recent years, human-computer interaction technology has been gradually applied to daily life. Gesture recognition, as one of the most intuitive means of human interaction, enables users to interact with machines in a more natural way through palm or finger movements. become a research hotspot. Existing gesture recognition schemes can be divided into gesture recognition methods based on visual sensors, inertial sensors, and wireless radio frequency sensors according to data sources. The gesture recognition scheme based on optical sensors uses the camera to capture the scene video containing gestures, and then uses computer algorithms to identify, extract and classify the gesture features in the image. The hardware system is relatively simple, and the collected images and videos can provide high space Resolution information, but also because of the need to process two-dimensional or even three-dimensional image data, the amount of data is large and the calculation cost is high, and the required high power consumption makes it difficult to apply to portable devices. In addition, gesture recognition based on optical sensors is sensitive to ambient light, prone to visual blind spots and light occlusion, which limits the scope of use of users, and there is a risk of user privacy leakage. The inertial sensor can measure the acceleration of the object, and can measure the object, but it requires the user to wear the device all the time while performing the action, and the user experience is poor.

With the development of radio frequency technology and integrated circuits, gesture recognition methods based on wireless signals such as radar and WiFi have gradually attracted the attention of scholars. Compared with optical sensors and inertial sensors, wireless signals have lower environmental requirements. Among them, millimeter wave radar has many unique advantages. First of all, the radar transmits and receives signals, and collects gesture information without being affected by weather conditions and ambient light, which greatly improves the application scenarios that the system can cover. Radar can also propagate through occlusions, making gesture interaction possible under partial or complete occlusions. Secondly, millimeter-wave radar receives echoes with lower energy consumption than optical solutions, and can be better integrated into embedded devices, and its convenience is better than wearable device solutions. At the same time, radar signals are more secure and confidential, and basically do not reveal user privacy. In addition, mmWave radar sensors have a narrower beam narrower and higher spatial resolution than other low-band RF sensors. Because of the many advantages mentioned above, millimeter wave radar has attracted the attention of researchers in recent years. However, due to the inherent differences in the design of data collection, feature extraction and fusion, and gesture classification algorithm design, the gesture recognition method based on millimeter wave radar has inherent differences. Therefore, the development of its algorithm is still not very mature, and there are still some limitations. For example, the information provided by radar cannot be fully utilized, and the size and time complexity of the recognition model are relatively large.

From the two aspects of feature extraction method and gesture classification method, the radar-based gesture recognition method has achieved certain results in the technical research and application process of radar gesture recognition, but there are still some challenges:

(1) Construction of labeled dataset. When using deep learning for gesture recognition, sufficient gesture data is the basis for model training, but based on the existing hardware platform, the acquisition of gesture data takes a lot of time and is very expensive.

(2) Multi-dimensional gesture feature extraction. The existing algorithms use the micro-Doppler spectrum and the range-Doppler spectrum as network input data, and the amount of data is large and there is no correspondence between the characteristics of each target point, and there is a lack of utilization and fusion of azimuth and elevation angle information. Failure to make full use of the information provided by the radar affects the accuracy of gesture recognition.

(3) Classifier design. When using deep learning to classify radar gestures, a single type of recognition model is often used. The recognition algorithm has a large amount of calculation and high model complexity, and the effective features of the input data cannot be fully extracted, resulting in low gesture classification accuracy.

To sum up, how to effectively and conveniently extract multi-dimensional gesture features, and how to design a classifier to improve the accuracy of gesture classification are still problems to be solved urgently by those skilled in the art.

Contents of the invention

The present invention provides a gesture recognition method of FMCW millimeter wave radar based on multi-task learning, which is used to solve the problems that the existing multi-dimensional gesture feature extraction is not effective and convenient, and the accuracy rate of gesture classification is not high.

The present invention provides a kind of gesture recognition method based on the FMCW millimeter-wave radar of multi-task learning, comprising:

Obtain the time series sequence of the target feature vector of the gesture to be recognized;

Inputting the sequence sequence of the target feature vector into a multi-task learning gesture recognition model, and outputting gesture category results;

Wherein, the multi-task learning gesture recognition model is obtained by training based on the sample target feature vector sequence, the corresponding gesture category label and the corresponding gesture trajectory label, the target feature vector sequence and the sample target feature vector sequence The sequences are obtained by processing the raw gesture recognition data acquired by the FMCW millimeter-wave radar through a preset algorithm.

According to the multi-task learning-based FMCW millimeter-wave radar gesture recognition method provided by the present invention, the multi-task learning gesture recognition model includes two learning tasks of gesture recognition classification and gesture trajectory reconstruction in the training process.

According to a gesture recognition method of FMCW millimeter-wave radar based on multi-task learning provided by the present invention, the overall loss function of the multi-task learning gesture recognition model in the training process is two parts of the gesture recognition classification part and the reconstructed gesture track part. The weighting of the loss function of each part.

According to the gesture recognition method of a FMCW millimeter-wave radar based on multi-task learning provided by the present invention, the training network results of the multi-task learning gesture recognition model in the training process include:

Sequentially connected one-dimensional CNN (Convolutional Neural Network, convolutional neural network) layer, LSTM (Long Short-Term Memory, long-term short-term memory network) layer and one-dimensional transposed convolutional layer;

And, sequentially connected one-dimensional CNN layer, LSTM layer and fully connected layer.

According to a gesture recognition method of FMCW millimeter-wave radar based on multi-task learning provided by the present invention, the one-dimensional CNN layer has 128 convolution kernels, the fully connected layer has a softmax activation function, and the one-dimensional transpose The convolution uses 128 deconvolution kernels of size 7×1.

According to the multi-task learning-based FMCW millimeter-wave radar gesture recognition method provided by the present invention, the target feature vector time series and the sample target feature vector time series are both acquired by the FMCW millimeter-wave radar through a preset algorithm It is obtained after processing the raw data of gesture recognition, including:

A specific multi-dimensional fusion feature extraction algorithm is used to extract the spatial and temporal features of the gesture signal acquired by the FMCW millimeter-wave radar, and the distance, angle and speed information at each moment are one-to-one in the form of vector feature fusion. Obtain the time series of target feature vectors and the time series of sample target feature vectors.

The present invention also provides a gesture recognition method for FMCW millimeter-wave radar based on multi-task learning, wherein the target feature vector time series and the sample target feature vector time series are all gestures acquired by the FMCW millimeter wave radar through a preset algorithm It is obtained after identifying the raw data for processing, including:

For the distance value in the original feature of gesture recognition acquired by FMCW millimeter-wave radar for each frame, first perform fast-time FFT on the intermediate frequency signal, and then detect the possible target distance through CFAR after removing static clutter;

Then use the Capon algorithm to calculate the azimuth angle spectrum at the corresponding distance and detect the possible target azimuth angle at a specific distance through CFAR (Constant False Alarm Rate, constant false alarm rate);

Use the Capon algorithm to calculate the pitch angle spectrum at the corresponding distance and azimuth angle and detect the pitch angle of the target at a specific distance and azimuth angle through CFAR;

Finally, beamforming is used to calculate the target velocity of each target point;

Construct target distance, target azimuth, target pitch angle and target velocity into target gesture recognition features, perform Doppler FFT processing, and then based on density clustering, extract cluster centers as target feature vector time series and sample target feature vector time series sequence.

According to the gesture recognition device of a FMCW millimeter-wave radar based on multi-task learning provided by the present invention, it includes:

An acquisition unit, configured to acquire a time series sequence of target feature vectors of gestures to be recognized;

A recognition unit, configured to input the time series sequence of the target feature vector into the multi-task learning gesture recognition model, and output gesture category results;

Wherein, the multi-task learning gesture recognition model is obtained by training based on the sample target feature vector sequence, the corresponding gesture category label and the corresponding gesture trajectory label, the target feature vector sequence and the sample target feature vector sequence The sequences are obtained by processing the raw gesture recognition data acquired by the FMCW millimeter-wave radar through a preset algorithm.

The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and operable on the processor. When the processor executes the program, the multi-based Steps in a task-learning approach to gesture recognition for FMCW millimeter-wave radar.

The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the gesture of the FMCW millimeter-wave radar based on multi-task learning as described in any one of the above is realized Identify the steps of the method.

The gesture recognition method of the FMCW millimeter-wave radar based on multi-task learning provided by the present invention obtains the time-series sequence of the target feature vector of the gesture to be recognized; the time-series sequence of the target feature vector is input into the multi-task learning gesture recognition model, and the gesture category result is output ; Wherein, the multi-task learning gesture recognition model is obtained by training based on the sample target feature vector sequence, the corresponding gesture category label and the corresponding gesture track label, the target feature vector sequence and the sample target feature vector The time series are all obtained by processing the raw gesture recognition data acquired by the FMCW millimeter-wave radar through a preset algorithm. It realizes the effective and convenient acquisition of multi-dimensional features of user gestures, and improves the accuracy of gesture recognition classification through multi-task learning.

Description of drawings

In order to more clearly illustrate the present invention or the technical solutions in the prior art, the accompanying drawings that need to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings in the following description are the present invention. For some embodiments of the invention, those skilled in the art can also obtain other drawings based on these drawings without creative effort.

Fig. 1 is the schematic flow chart of the gesture recognition method of the FMCW millimeter-wave radar based on multi-task learning provided by the present invention;

Fig. 2 is the physical structural diagram of the FMCW millimeter-wave radar hardware platform provided by the present invention;

Fig. 3 is the schematic diagram of the interface of the present invention to the parameter configuration of radar transmission signal;

4 is a schematic diagram of a multi-task learning network provided by the present invention;

Fig. 5 is a schematic diagram of a multi-task learning network framework provided by the present invention;

Fig. 6 is the flow chart of multi-dimensional feature extraction provided by the present invention;

Fig. 7 is the variation of verification accuracy in the training process provided by the present invention;

Fig. 8 is the confusion matrix of the test set classification result provided by the present invention;

FIG. 9 is a schematic structural diagram of a gesture recognition device based on a multi-task learning FMCW millimeter-wave radar provided by the present invention;

Fig. 10 is a schematic structural diagram of an electronic device provided by the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention , but not all examples. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

In the existing gesture recognition and classification technologies, there are generally problems that the multi-dimensional gesture feature extraction is not effective and convenient, and the accuracy of gesture classification is not high. The gesture recognition method of the FMCW millimeter-wave radar based on multi-task learning of the present invention will be described below in conjunction with FIG. 1 . Fig. 1 is the schematic flow chart of the gesture recognition method of the FMCW millimeter-wave radar based on multi-task learning provided by the present invention, as shown in Fig. 1, the method comprises:

Step 110, acquiring a time series sequence of target feature vectors of gestures to be recognized.

Specifically, the time series sequence of the target feature vector of the gesture to be recognized is obtained, that is, the target feature vector obtained after the feature vector formed by the distance, speed, azimuth and pitch angle of the gesture to be recognized is extracted by multi-dimensional feature fusion, and then from the time series The multi-frame target feature vectors are spliced together to obtain a time series sequence of target feature vectors for identifying the corresponding gesture types.

Step 120, input the time series sequence of the target feature vector into the multi-task learning gesture recognition model, and output gesture category results;

Wherein, the multi-task learning gesture recognition model is obtained by training based on the sample target feature vector sequence, the corresponding gesture category label and the corresponding gesture trajectory label, the target feature vector sequence and the sample target feature vector sequence The sequences are obtained by processing the raw gesture recognition data acquired by the FMCW millimeter-wave radar through a preset algorithm.

Specifically, the target feature vector sequence is input into the multi-task learning gesture recognition model, and the predicted label is output, that is, the gesture category recognition result corresponding to the input target feature vector sequence. Among them, the multi-task learning gesture recognition model is obtained after training a large number of target feature vector time series samples and gesture category labels and gesture trajectory labels corresponding to the samples one by one. The higher the accuracy rate is used. The training model used in the embodiment of the present invention is a multi-task learning type, that is, in addition to outputting gesture classification results, the model can also output results similar to trajectory reconstruction. Adjust the parameters to be adjusted in , so that the model obtained after model training is used to recognize gestures more accurately. What needs to be explained here is that both the time series of target feature vectors and the time series of sample target feature vectors are obtained by processing the original gesture recognition data acquired by the FMCW millimeter-wave radar through a preset algorithm.

What needs to be further explained here is that this method uses the hardware experiment platform and software of FMCW millimeter-wave radar to collect data, and then designs a multi-dimensional fusion feature extraction method for the collected radar gesture signals. On the basis of the data set, the gesture recognition algorithm research is carried out, and a multi-task-based neural network structure is proposed, which makes full use of the parameter information extracted by the radar to achieve higher classification accuracy and faster classification speed.

Fig. 2 is the physical structure diagram of the FMCW millimeter-wave radar hardware platform provided by the present invention, as shown in Fig. 2, the FMCW millimeter-wave radar platform adopted in the embodiment of the present invention is mainly composed of two parts, including the IWR 1443 radar module of Texas Instruments And DCA1000 data acquisition board. Among them, the IWR1443 radar development board is mainly responsible for generating and transmitting chirp waves, receiving and demodulating target echoes, and obtaining intermediate frequency signals carrying target information. The DCA1000 data acquisition board is responsible for transmitting the intermediate frequency signal to the personal computer for the next step of data processing. In terms of software, Fig. 3 is a schematic diagram of the interface of the present invention for configuring the parameters of the radar transmission signal. As shown in Fig. 3, the embodiment of the present invention adopts the mmWave studio host computer software developed by TI Company to configure the radar, and collect data and store it in the PC end.

Since the gain of the pattern of the commercial radar drops rapidly, the requirements of the gesture recognition application are not taken into account. For the gesture and gesture recognition application scenarios to be adopted in the present invention, an antenna optimization algorithm is designed to optimize the pattern of the array antenna to improve the sensitivity. The pattern gain of the target area of interest reduces the interference of other areas, and it is expected that by analyzing the configuration parameters of the radar signal acquisition platform, the optimal configuration of the FMCW millimeter-wave radar system for gesture recognition can be found to maximize the use of radar performance and hereby Based on the collected gesture data, a gesture data set is constructed.

The gesture recognition method of the FMCW millimeter-wave radar based on multi-task learning provided by the embodiment of the present invention obtains the time-series sequence of the target feature vector of the gesture to be recognized; inputs the time-series sequence of the target feature vector into the multi-task learning gesture recognition model, and outputs the gesture Category results; wherein, the multi-task learning gesture recognition model is obtained by training based on the sample target feature vector time series and the corresponding gesture category label, and the target feature vector time series and the sample target feature vector time series are both It is obtained by processing the raw gesture recognition data acquired by the FMCW millimeter-wave radar through a preset algorithm. It realizes the effective and convenient acquisition of multi-dimensional features of user gestures, and improves the accuracy of gesture recognition classification through multi-task learning.

Based on the above embodiments, in this method, the multi-task learning gesture recognition model includes two learning tasks of gesture recognition classification and gesture trajectory reconstruction during the training process.

Specifically, the multi-task learning gesture recognition model in the training process includes two learning tasks of gesture recognition classification and gesture trajectory reconstruction, that is, the labels used in the training process include gesture category labels corresponding to the time sequence of sample target gesture feature vectors, It also includes gesture trajectory labels corresponding to the time series of sample target gesture feature vectors, which are used to incorporate the parameters to be adjusted in the adjustment model network while "correcting" the predicted labels and reconstructing the trajectory.

Based on the above embodiment, in this method, the overall loss function of the multi-task learning gesture recognition model in the training process is the weighting of the loss functions of the gesture recognition classification part and the gesture trajectory reconstruction part.

Specifically, the data after the compressed representation is used as the input to reconstruct the gesture trajectory part and the gesture recognition classification part, and the weighted sum of the loss functions of the reconstructed gesture trajectory part and the gesture recognition classification part is used as the overall loss function, aiming to pass The feature extraction module in the reconstruction task assists the network to perform better feature extraction on the input data to improve the performance of the gesture recognition classification part.

Based on the foregoing embodiments, in this method, the training network results of the multi-task learning gesture recognition model during the training process include:

Sequentially connected one-dimensional CNN layer, LSTM layer and one-dimensional transposed convolutional layer;

And, sequentially connected one-dimensional CNN layer, LSTM layer and fully connected layer.

Specifically, Fig. 4 is a schematic diagram of a multi-task learning network provided by the present invention. As shown in Fig. 4, the training network results of the multi-task learning gesture recognition model in the training process include: sequentially connected one-dimensional CNN layer, LSTM layer and 1D transposed convolution layer (1D CNN Transpose); and, sequentially connected 1D CNN layer, LSTM layer and fully connected layer. The schematic diagram of the network framework is shown in Figure 4. The neural network structure used in the present invention can be mainly divided into four stages: feature extraction, encoding, classification and reconstruction. In the feature extraction stage, the one-dimensional convolution operation is used to extract appropriate and sufficient features from the input time series trajectory such as distance and angle. The extracted features are mapped to the hidden layer through the LSTM layer in the encoding stage, and the LSTM in the model converts the input data. is the learned compressed representation, which is used as input to the classification and reconstruction stages. The classification stage consists of a fully connected layer that computes the conditional probability of the output based on the learned input trajectory features by applying a softmax function. In the reconstruction or decoding stage, the compressed representation of the input trajectory is fed to several 1D transposed convolutional layers, which attempt to reconstruct the input feature trajectory.

Based on the above embodiment, in this method, the one-dimensional CNN layer has 128 convolution kernels, the fully connected layer has a softmax activation function, and the one-dimensional transposed convolution layer uses 128 convolution kernels with a size of 7× 1 deconvolution kernel.

Specifically, FIG. 5 is a schematic diagram of a multi-task learning network framework provided by the present invention. As shown in FIG. 5, the input of a one-dimensional CNN is 4×100, and this layer has 128 convolution kernels. The one-dimensional CNN layer is followed by an LSTM layer, and the output of the LSTM layer is then input to a fully connected layer with a softmax activation function. The output of the fully connected layer is then input to seven one-dimensional transposed convolutional layers. The first two layers use 32 deconvolution kernels with a size of 7×1, and the next two layers use 64 deconvolution kernels with a size of 7×1. Deconvolution kernel, the next two layers use 128 deconvolution kernels with a size of 7×1, and the last one-dimensional transposed convolution layer uses 128 deconvolution kernels with a size of 7×1. Additionally, all convolutional and transposed layers use rectified linear units (ReLU) as activation functions. To avoid model overfitting, dropout is used in the last 1D transposed convolutional layer of the model.

Based on the above-mentioned embodiments, in this method, the target feature vector time series and the sample target feature vector time series are both obtained by processing the original gesture recognition data acquired by the FMCW millimeter-wave radar through a preset algorithm, specifically including :

A specific multi-dimensional fusion feature extraction algorithm is used to extract the spatial and temporal features of the gesture signal acquired by the FMCW millimeter-wave radar, and the distance, angle and speed information at each moment are one-to-one in the form of vector feature fusion. Obtain the time series of target feature vectors and the time series of sample target feature vectors.

Specifically, a multi-dimensional fusion feature extraction method is designed, which extracts the spatial and temporal features of gesture signals, and integrates the distance, angle, and speed information at each moment in the form of vectors to solve the problem. It solves the problem of using pictures as network input data and there is no correspondence between the features of each target point, and more accurately reflects the changing law of the hand over time during movement.

Based on the above embodiments, in this method, the target feature vector time series and the sample target feature vector time series are both obtained by processing the original gesture recognition data acquired by the FMCW millimeter-wave radar through a preset algorithm, Specifically include:

For the distance value in the original feature of gesture recognition acquired by FMCW millimeter-wave radar for each frame, first perform fast-time FFT on the intermediate frequency signal, and then detect the possible target distance through CFAR after removing static clutter;

Then use the Capon algorithm to calculate the azimuth angle spectrum at the corresponding distance and detect the possible target azimuth angle at a specific distance through CFAR;

Use the Capon algorithm to calculate the pitch angle spectrum at the corresponding distance and azimuth angle and detect the pitch angle of the target at a specific distance and azimuth angle through CFAR;

Finally, beamforming is used to calculate the target velocity of each target point;

Construct target distance, target azimuth, target pitch angle and target velocity into target gesture recognition features, perform Doppler FFT processing, and then based on density clustering, extract cluster centers as target feature vector time series and sample target feature vector time series sequence.

Specifically, in order to make full use of the effective information provided by the radar and integrate each feature, the embodiment of the present invention designs a multi-dimensional feature fusion gesture feature extraction method, so that the distance, speed, and orientation of each target at each moment Angle and pitch angle features correspond to each other, which more accurately embodies the changing law of the hand over time during movement, reduces the input size of the classification model, and reduces the calculation amount of the forward propagation of the classification model. Figure 6 provides the present invention with The flow chart of multi-dimensional feature extraction, the specific steps are shown in Figure 6. In each frame, first perform a fast-time FFT on the intermediate frequency signal, remove the static clutter, and then use CFAR to detect the distance of the possible target, and then use the Capon algorithm to calculate the azimuth angle spectrum at the corresponding distance and use CFAR to detect the specific distance. The azimuth of the target may exist. Similarly, the elevation angle of the target is detected at the corresponding distance and azimuth angle. After calculating the distance, azimuth angle, and elevation angle of the target, beamforming is performed on the 8 echo signals, and then the slow time dimension Carry out Doppler FFT, calculate the velocity of each target point, and cluster according to the position of the target point in each frame in space, and use the cluster center as the position of the target in the current frame to construct the feature vector.

After preprocessing the raw radar echo signal, each gesture is represented by the time series trajectory of distance, angle, and velocity, and the distance, azimuth, pitch angle, and velocity vector of 100 consecutive frames are used as the classifier The input builds the gesture dataset. In order to make full use of the extracted gesture feature vectors, a network framework based on multi-task learning is used for gesture recognition.

The key points of the embodiment of the present invention are as follows: 1. Firstly, a multi-dimensional fusion feature extraction method is designed, which extracts the spatial and temporal features of the gesture signal, and corresponds the distance, angle, and speed information at each moment Feature fusion in the form of vectors solves the problem of using pictures as network input data and there is no correspondence between the features of each target point, and more accurately reflects the changing law of hands over time during movement; using Kalman filtering to While smoothing the gesture trajectory, fine-tuning the result by changing the parameters, increasing the number of samples while ensuring the filtering effect, and realizing sample enhancement; 2. Secondly, a multi-task learning network framework is designed, and the overall network framework It is divided into two task modules of trajectory reconstruction and gesture classification. Through the reconstruction task, the shared part of the network is assisted to better extract features from the input data. First, the features of the input information are extracted through a one-dimensional convolutional layer, and then the LSTM layer is used to encode the extracted information, that is, the compressed representation; the next step is to use the compressed and represented data as the input of the reconstruction part and the classification part, and The weighted sum of the loss functions of the reconstruction part and the classification part is used as the overall loss function, which aims to assist the feature extraction module in the network to better extract features from the input data through the reconstruction task, so as to improve the performance of the classification part.

Compared with the existing technology, the advantages of the embodiments of the present invention are: first, a multi-dimensional fusion feature extraction method is designed to extract the spatial and temporal features of the gesture signal, and the distance, angle, and speed of each moment One-to-one correspondence of information is carried out in the form of vector feature fusion, which solves the problem of using pictures as network input data and there is no correspondence between the features of each target point, and more accurately reflects the changing law of hands over time during movement; the experiment It shows that there are obvious differences in the trajectories of different gestures, and they all show different characteristics.

Experimental verification process:

Use the designed multi-task learning model to train and classify the gesture data set, use 80% of the total samples to train the model, and use the remaining 20% of the samples for testing. According to the used network structure and loss function, the present invention adopts Adam optimizer, the learning rate is set to 0.0005, the epoch is set to 50, the batchsize is set to 32, then the network is trained and tested, and the experimental results are counted. Finally, the average recognition accuracy of the model for the 6 gestures is 99%. Figure 7 shows the variation of verification accuracy during the training process provided by the present invention, and Figure 8 shows the confusion matrix of the test set classification results provided by the present invention. As shown in Figure 8, the overall recognition accuracy of the eight types of gestures finally converges at 99%. Among them, the misclassified ones are mainly the third type of gestures sliding to the left and the fourth type of gestures sliding to the right. According to the analysis, the main reason may be this. The two types of gestures differ only in the azimuth feature, while the rest of the feature trajectories are basically the same, so the probability of confusion is greater.

The multi-task network structure in the present invention is compared with the single-task network structure that removes the reconstruction module and the work of Infineon Company. In order to verify the recognition performance, the present invention adopts the following parameters to analyze from two aspects of classification performance and time complexity respectively: Algorithms for comparison:

(1) Accuracy: The ratio of the number of correctly classified samples in the test set to the total number of samples can reflect the ability of the recognition algorithm to judge the data set.

(2) Floating Point Operations (FLOPs): It is the calculation amount, which can be used to measure the complexity of the model.

(3) Number of parameters: The number of neural network parameters can measure the size of the model.

Table 1 shows the performance comparison of different methods, using the samples collected in the present invention to verify the recognition scheme, and the comparison of classification performance and algorithm complexity is shown in Table 1.

Table 1 Performance comparison of different methods

It can be seen from Table 1 that in terms of classification performance, compared with the single-task module without the reconstruction module, the average recognition accuracy of the multi-task network structure used in the present invention is increased by 3% for the same data set, indicating that the multi-task Compared with the network structure without adding multi-task modules, the structure can better extract the spatial features and temporal features in the gesture parameters with the assistance of the reconstruction task; while the method used by Infineon company only reached 87.5% The main reason for the recognition accuracy is that it uses the range Doppler image sequence as the network input, and only uses the distance information and velocity information of gestures, but fails to use the horizontal angle information, so it is difficult to recognize the radial velocity and radial velocity and radius information with angle changes. Gestures with insignificant changes in distance, resulting in a decrease in the overall recognition accuracy. In terms of algorithm time complexity, the number of parameters is used to represent the size of the classification model, and FLOPs is used to evaluate the calculation amount of the model during forward propagation. It can be seen that the work of the present invention can have the same time complexity compared with the single-task network. Higher recognition accuracy; compared with 3D CNN, the number of parameters, model parameters and FLOPs are less while having higher recognition accuracy, which shows that the algorithm proposed by the present invention is better than the method used by Infineon. This algorithm is compared with other algorithms, and the influencing factors of classification performance are analyzed. The experimental results show that the average recognition accuracy of the present invention can reach 99%, which is better than the compared algorithms.

The gesture recognition device of the FMCW millimeter-wave radar based on multi-task learning provided by the present invention is described below. The gesture recognition device of the FMCW millimeter-wave radar based on multi-task learning described below is the same as the FMCW millimeter-wave radar based on multi-task learning described above. The gesture recognition methods of the radar can be referred to in correspondence with each other.

FIG. 9 is a schematic structural diagram of a multi-task learning-based FMCW millimeter-wave radar gesture recognition device provided by the present invention. As shown in FIG. 9, the device includes an acquisition unit 910 and a recognition unit 920, wherein,

The acquiring unit 910 is configured to acquire a time series sequence of target feature vectors of gestures to be recognized;

The recognition unit 920 is configured to input the time series sequence of the target feature vector into a multi-task learning gesture recognition model, and output gesture category results;

Wherein, the multi-task learning gesture recognition model is obtained by training based on the sample target feature vector sequence, the corresponding gesture category label and the corresponding gesture trajectory label, the target feature vector sequence and the sample target feature vector sequence The sequences are obtained by processing the raw gesture recognition data acquired by the FMCW millimeter-wave radar through a preset algorithm.

The gesture recognition device of the FMCW millimeter-wave radar based on multi-task learning provided by the present invention obtains the target feature vector time sequence sequence of the gesture to be recognized; the target feature vector time sequence sequence is input into the multi-task learning gesture recognition model, and the gesture category result is output ; Wherein, the multi-task learning gesture recognition model is obtained by training based on the sample target feature vector sequence, the corresponding gesture category label and the corresponding gesture track label, the target feature vector sequence and the sample target feature vector The time series are all obtained by processing the raw gesture recognition data acquired by the FMCW millimeter-wave radar through a preset algorithm. It realizes the effective and convenient acquisition of multi-dimensional features of user gestures, and improves the accuracy of gesture recognition classification through multi-task learning.

On the basis of the above embodiments, in the device, the multi-task learning gesture recognition model includes two learning tasks of gesture recognition classification and gesture trajectory reconstruction during the training process.

On the basis of the above embodiments, in the device, the overall loss function of the multi-task learning gesture recognition model during training is the weighting of the loss functions of the gesture recognition classification part and the gesture trajectory reconstruction part.

On the basis of the foregoing embodiments, in this device, the training network results of the multi-task learning gesture recognition model during the training process include:

Sequentially connected one-dimensional CNN layer, LSTM layer and one-dimensional transposed convolutional layer;

And, sequentially connected one-dimensional CNN layer, LSTM layer and fully connected layer.

On the basis of the above embodiment, in this device, the one-dimensional CNN layer has 128 convolution kernels, the fully connected layer has a softmax activation function, and the one-dimensional transposed convolution layer uses 128 convolution kernels of size It is a 7×1 deconvolution kernel.

On the basis of the above-mentioned embodiments, in the device, the target feature vector time series and the sample target feature vector time series are all obtained by processing the gesture recognition raw data acquired by the FMCW millimeter wave radar through a preset algorithm. , including:

A specific multi-dimensional fusion feature extraction algorithm is used to extract the spatial and temporal features of the gesture signal acquired by the FMCW millimeter-wave radar, and the distance, angle and speed information at each moment are one-to-one in the form of vector feature fusion. The time series of target feature vectors and the time series of sample target feature vectors are obtained.

On the basis of the above-mentioned embodiments, in the device, the target feature vector time series and the sample target feature vector time series are all obtained by processing the gesture recognition raw data acquired by the FMCW millimeter wave radar through a preset algorithm. , including:

For the distance value in the original feature of gesture recognition acquired by FMCW millimeter-wave radar for each frame, first perform fast-time FFT on the intermediate frequency signal, and then detect the possible target distance through CFAR after removing static clutter;

Then use the Capon algorithm to calculate the azimuth angle spectrum at the corresponding distance and detect the possible target azimuth angle at a specific distance through CFAR;

Use the Capon algorithm to calculate the pitch angle spectrum at the corresponding distance and azimuth angle and detect the pitch angle of the target at a specific distance and azimuth angle through CFAR;

Finally, beamforming is used to calculate the target velocity of each target point;

Construct target distance, target azimuth, target pitch angle and target velocity into target gesture recognition features, perform Doppler FFT processing, and then based on density clustering, extract cluster centers as target feature vector time series and sample target feature vector time series sequence.

FIG. 10 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG. 10 , the electronic device may include: a processor (processor) 1010, a communication interface (CommunicationsInterface) 1020, a memory (memory) 1030, and a communication bus 1040, wherein , the processor 1010 , the communication interface 1020 , and the memory 1030 communicate with each other through the communication bus 1040 . The processor 1010 can call the logic instructions in the memory 1030 to execute the gesture recognition method of the FMCW millimeter-wave radar based on multi-task learning, the method includes: acquiring the target feature vector sequence sequence of the gesture to be recognized; Sequentially input the multi-task learning gesture recognition model, and output the gesture category result; wherein, the multi-task learning gesture recognition model is trained based on the sample target feature vector sequence, the corresponding gesture category label and the corresponding gesture track label, so Both the target feature vector time series and the sample target feature vector time series are obtained by processing the gesture recognition raw data acquired by the FMCW millimeter-wave radar through a preset algorithm.

In addition, the above-mentioned logic instructions in the memory 1030 may be implemented in the form of software functional units and be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-OnlyMemory), random access memory (RAM, RandomAccessMemory), magnetic disk or optical disk and other media that can store program codes.

On the other hand, the present invention also provides a computer program product, the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer During execution, the computer can perform the gesture recognition method of the FMCW millimeter-wave radar based on multi-task learning provided by the above-mentioned methods, the method includes: obtaining the target feature vector sequence sequence of the gesture to be recognized; inputting the target feature vector sequence sequence A multi-task learning gesture recognition model that outputs gesture category results; wherein, the multi-task learning gesture recognition model is obtained by training based on the sample target feature vector sequence, the corresponding gesture category label and the corresponding gesture track label, and the target Both the time series of feature vectors and the time series of feature vectors of the sample target are obtained by processing the original gesture recognition data acquired by the FMCW millimeter-wave radar through a preset algorithm.

In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it is implemented to perform the above-mentioned FMCW millimeter-wave radar based on multi-task learning. A gesture recognition method, the method comprising: obtaining a target feature vector time series sequence of a gesture to be recognized; inputting the target feature vector time series sequence into a multi-task learning gesture recognition model, and outputting a gesture category result; wherein, the multi-task learning gesture recognition The model is trained based on the sample target feature vector time series, the corresponding gesture category label and the corresponding gesture track label. The target feature vector time series and the sample target feature vector time series are both FMCW It is obtained after processing the raw gesture recognition data acquired by the millimeter-wave radar.

The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without any creative efforts.

Through the above description of the implementations, those skilled in the art can clearly understand that each implementation can be implemented by means of software plus a necessary general hardware platform, and of course also by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic discs, optical discs, etc., including several instructions to make a computer device (which may be a personal computer, server, or network device, etc.) execute the methods described in various embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims (10)

1. a gesture recognition method based on the FMCW millimeter-wave radar of multi-task learning, it is characterized in that, comprising: Obtain the time series sequence of the target feature vector of the gesture to be recognized; Inputting the sequence sequence of the target feature vector into a multi-task learning gesture recognition model, and outputting gesture category results; Wherein, the multi-task learning gesture recognition model is obtained by training based on the sample target feature vector sequence, the corresponding gesture category label and the corresponding gesture trajectory label, the target feature vector sequence and the sample target feature vector sequence The sequences are all obtained by processing the raw gesture recognition data acquired by the FMCW millimeter-wave radar through a preset algorithm. 2. the gesture recognition method of the FMCW millimeter-wave radar based on multi-task learning according to claim 1, is characterized in that, described multi-task learning gesture recognition model comprises gesture recognition classification and reconstruction gesture track two in training process learning assignment. 3. the gesture recognition method of the FMCW millimeter-wave radar based on multitask learning according to claim 2, is characterized in that, the total loss function of described multitask learning gesture recognition model in training process is gesture recognition classification part and weight The weighting of the loss function of the two parts of the gesture trajectory part. 4. the gesture recognition method of the FMCW millimeter-wave radar based on multitask learning according to claim 1, is characterized in that, the training network result of described multitask learning gesture recognition model in training process comprises: Sequentially connected one-dimensional CNN layer, LSTM layer and one-dimensional transposed convolutional layer; And, sequentially connected one-dimensional CNN layer, LSTM layer and fully connected layer. 5. the gesture recognition method of the FMCW millimeter-wave radar based on multi-task learning according to claim 4, is characterized in that, described one-dimensional CNN layer has 128 convolution cores, and described fully connected layer has softmax activation function, The one-dimensional transposed convolution layer uses 128 deconvolution kernels with a size of 7×1. 6. the gesture recognition method of the FMCW millimeter-wave radar based on multi-task learning according to claim 1, is characterized in that, described target feature vector sequence and described sample target feature vector sequence all are by preset algorithm The gesture recognition raw data acquired by the FMCW millimeter wave radar is processed, including: A specific multi-dimensional fusion feature extraction algorithm is used to extract the spatial and temporal features of the gesture signal acquired by the FMCW millimeter-wave radar, and the distance, angle and speed information at each moment are one-to-one in the form of vector feature fusion. Obtain the time series of target feature vectors and the time series of sample target feature vectors. 7. the gesture recognition method of the FMCW millimeter-wave radar based on multi-task learning according to claim 6, is characterized in that, described target feature vector sequence and described sample target feature vector sequence all are by preset algorithm The gesture recognition raw data acquired by the FMCW millimeter wave radar is processed, including: For the distance value in the original feature of gesture recognition acquired by FMCW millimeter-wave radar for each frame, first perform fast-time FFT on the intermediate frequency signal, and then detect the possible target distance through CFAR after removing static clutter; Then use the Capon algorithm to calculate the azimuth angle spectrum at the corresponding distance and detect the possible target azimuth angle at a specific distance through CFAR; Use the Capon algorithm to calculate the pitch angle spectrum at the corresponding distance and azimuth angle and detect the pitch angle of the target at a specific distance and azimuth angle through CFAR; Finally, beamforming is used to calculate the target velocity of each target point; Construct target distance, target azimuth, target pitch angle and target velocity into target gesture recognition features, perform Doppler FFT processing, and then based on density clustering, extract cluster centers as target feature vector time series and sample target feature vector time series sequence. 8. A gesture recognition device based on the FMCW millimeter-wave radar of multi-task learning, it is characterized in that, comprising: An acquisition unit, configured to acquire a time series sequence of target feature vectors of gestures to be recognized; A recognition unit, configured to input the time series sequence of the target feature vector into the multi-task learning gesture recognition model, and output gesture category results; Wherein, the multi-task learning gesture recognition model is obtained by training based on the sample target feature vector sequence, the corresponding gesture category label and the corresponding gesture trajectory label, the target feature vector sequence and the sample target feature vector sequence The sequences are obtained by processing the raw gesture recognition data acquired by the FMCW millimeter-wave radar through a preset algorithm. 9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, characterized in that, when the processor executes the program, any of the following claims 1 to 7 are realized. A step of the gesture recognition method of the FMCW millimeter-wave radar based on multi-task learning. 10. A non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the multi-task-based learning as described in any one of claims 1 to 7 is realized The steps of the gesture recognition method for the FMCW millimeter wave radar.

Images