CN111506190A - Real-time continuous prediction method for human motion intention based on myoelectricity - Google Patents

Abstract

The invention is a real-time continuous prediction method of human body motion intention based on myoelectricity. The method is specifically as follows: extracting the EMG signal and the IMU signal of the unilateral lower limb of the human body, and setting the evaluation threshold; respectively selecting the IMU signal collected by any electrode of the thigh and the calf, and obtaining the knee joint angle at the current moment through the lower limb kinematics solution algorithm. ; Use the sliding time window method to resample the lower limb EMG signal and the knee joint angle at the current moment; establish five LSTM angle predictors with the same network structure, and compare them with the real-time knee joint angle to obtain the evaluation standard root mean square error RMSE, when When the root mean square error RMSE of the evaluation standard is less than the set threshold, the output angle of the predictor is accurate. The training data of the present invention is randomly shuffled in units of samples, which can reduce the influence of overfitting of the parameters of the prediction model caused by the fixed data sequence.

Description

A real-time continuous prediction method of human motion intention based on electromyography

technical field

The invention relates to the technical field of real-time continuous prediction of human motion intention, and is a real-time continuous prediction method of human motion intention based on myoelectricity.

Background technique

As a product of a high degree of integration of disciplines in various fields, robots have greatly liberated productivity since their birth. Auxiliary robots, that is, exoskeleton robots and powered prosthetics, have great research and development value in the fields of elderly care and disabled, military and other fields. The uniqueness of the exoskeleton robot is that its working space is highly overlapped with the human body, and the robot body needs to be highly coordinated with the human body. When the running speed or trajectory of the exoskeleton manipulator deviates from the human body's motion intention, it is easy to hinder the normal activities of the human body and even cause injury. . The traditional exoskeleton becomes a passive exoskeleton, which can only passively accept external control instructions and cannot actively judge the timing of assistance. Human-robot Interaction (HRI) technology makes the exoskeleton transition from passive to active understanding of the human body's movement intention by processing human-related movement information, and provide appropriate assistance according to the human body's movement intention.

At present, the bioelectric signals that have attracted much attention mainly include electroencephalography (EEG), electrooculography (EOG) and electromyography (EMG). Compared with the other two electrical signals, EMG signals are highly correlated and relatively It has the characteristics of stability, and has convenient conditions such as non-invasive surface acquisition. It has been widely used in related fields such as medical diagnosis, motion analysis and interactive games.

The use of EMG signals to perceive human behavior intentions has always occupied an important seat in the field of human-computer interaction research at home and abroad. The human body has an electromyography-mechanical delay phenomenon, that is, the EMG signal is earlier than the action, and the lead time is about 25-125ms. This feature provides a possibility to predict the kinematic parameters of the human joint before the action. Brantley et al. proposed in 2017 that continuous EMG signals can be used to estimate the kinematic parameters of the lower extremity knee and ankle joints. In 2014, Jimson et al. introduced EMD into the reference item when estimating the angle of the finger joint based on the EMG time-domain feature and signal envelope. The results show that the estimation accuracy of the joint angle is improved after the introduction of EMD. Domestically, in 2015, based on Hill's muscle model, Han Jianda and others from Shenyang Institute of Automation used only the wavelength and zero-crossing rate characteristics of the single-channel EMG signal of the biceps brachii to estimate the angle of the upper limb movement with the help of the IMU sensor. In closed-loop prediction, it fits very well, while in open-loop prediction it eventually tends to diverge. In 2017, Xia Peng et al. of Shanghai Jiaotong University used the EMG signals of the biceps, triceps, anterior deltoid, posterior deltoid, and middle deltoid, using Convolutional Neural Network (CNN) and recursion. The combined model of neural network (Recurrent Neural Network, RNN) has estimated the motion trajectory of the upper limbs of the human body, and achieved good results.

In the process of EMG signal processing, the extraction of robust features and the running of related programs will take time, and the EMG signal analysis results corresponding to the sampling time cannot be well matched to the motion state at the time of outputting the results. In addition, the mechanical and control hysteresis of the mobility aid itself can exacerbate the problem.

SUMMARY OF THE INVENTION

In order to solve the problem that the EMG signal analysis result cannot be well matched with the motion state at the time of outputting the result, and there is mechanical hysteresis and control hysteresis, the present invention provides a real-time continuous prediction method of human motion intention based on electromyography. The following technical solutions have been proposed:

A method for real-time continuous prediction of human motion intention based on electromyography, comprising the following steps;

Step 1: Extract the EMG signal and IMU signal of the unilateral lower limb of the human body, and set the evaluation threshold;

Step 2: Select the IMU signal collected by any electrode of the thigh and calf respectively, and obtain the knee joint angle at the current moment through the lower limb kinematics solution algorithm;

Step 3: Use the sliding time window method to resample the lower limb EMG signal and the knee joint angle at the current moment;

Step 4: Establish five LSTM angle predictors with the same network structure and correspond to the five motion modes; put the EMG signal and the knee joint angle at the current moment into the corresponding LSTM angle predictor for training according to the labeled motion mode label, The knee joint angle vector of the same data length after the timestamp is shifted back by a certain time is used as the predictor training label;

Step 5: According to the trained LSTM angle predictor, determine the current motion mode through the motion mode classification algorithm, and input the data to the corresponding LSTM predictor for real-time angle prediction according to the current motion mode;

Step 6: Move the time stamp of the predictor output angle value forward for a certain time as a whole, and compare it with the real-time knee joint angle to obtain the evaluation standard root mean square error RMSE, when the evaluation standard root mean square error RMSE is less than the set threshold , the predictor output angle is accurate.

Preferably, the step 1 is specifically:

Attach the signal acquisition device to the rectus femoris, vastus lateralis, vastus medialis, tibialis anterior muscle, semitendinosus, long head of biceps femoris, lateral gastrocnemius and medial gastrocnemius muscle to extract EMG signals and IMU signal, and set the evaluation threshold;

Video recording is performed, and the motion mode label corresponding to each data point is judged through the video.

Preferably, the signal acquisition device adopts the Delsys Trigno EMG acquisition system, the system includes a wireless communication base station and 16 wireless EMG electrodes, and each electrode has a built-in EMG and IMU signal collector.

Preferably, the step 2 is specifically:

Select the IMU signal collected by any one of the electrodes on the thigh and calf respectively, and transmit the collected fixed length data of the IMU signal to the development board through the wireless module. Knee joint angle, the frequency of the knee joint angle at the current moment is consistent with the frequency of the IMU signal.

Preferably, both ends of the sliding time window correspond to the synchronization points of the EMG signal and the joint angle signal, and the step length of the sliding time window ensures that the EMG and angle data in the sliding time window are synchronized with each other. and resample the knee joint angle at the current moment.

Preferably, five LSTM angle predictors with the same network structure are established and correspond to five motion modes; the EMG signal and the knee joint angle at the current moment are put into the corresponding LSTM angle predictor for training according to the labeled motion mode label, The knee joint angle vector of the same data length after the timestamp is shifted back by 27ms-300ms is used as the predictor training label, and the k-fold cross-validation method is used. One sample is taken as the test set each time, and the remaining k-1 samples are used as the training set. Repeat the experiment k times, so that each sample can be used as a test set.

Preferably, before each training starts, the training data are randomly shuffled in units of samples.

Preferably, the step 5 is specifically as follows: each time the EMG signal and the IMU signal collect fixed-length data, the data is transmitted to the development board through the wireless module, and the motion pattern is determined according to the EMG and IMU signals by the motion pattern classifier mounted on the board. Save, solve the real-time knee joint angle through the lower limb kinematics solution algorithm equipped with the development board and save it, and input the calculated real-time knee joint angle and the corresponding EMG signal to the corresponding LSTM angle predictor through the classified motion mode Make an angle prediction and save the result.

Preferably, the step 6 is specifically: extracting the prediction result of the LSTM angle predictor and the lower limb kinematics solution algorithm to solve the real-time knee joint angle, moving the time stamp of the angle value output by the LSTM angle predictor forward by 27ms-300ms as a whole, and Calculate the real-time knee joint angle with the lower extremity kinematics solution algorithm, and obtain the evaluation standard root mean square error RMSE, which is expressed by the following formula:

Among them, θ _p is the prediction result of the LSTM angle predictor, θ _r is the real-time knee joint angle, and n is the number of trials;

When the evaluation standard root mean square error RMSE is less than the set threshold, the predictor output angle is accurate; when the evaluation standard root mean square error RMSE is greater than the set threshold, re-collect data for training.

Preferably, the development board is equipped with an NVIDIA Pascal architecture GPU, two Denver 64-bit CPUs and a quad-core A57 composite processor, and is equipped with a Linux Ubuntu operating system and a communication interface.

The present invention has the following beneficial effects:

The knee joint angle obtained by the real-time calculation of the present invention predicts the knee joint angle of 27-300 ms, and the function of the predicted value is to provide motion parameter output and at the same time make up for the system delay in the decision command transmission process. The training data of the present invention is randomly shuffled in units of samples, which can reduce the influence of overfitting of the parameters of the prediction model due to the fixed data sequence. LSTM filters relevant information by setting a "gate structure" to solve the long-term dependency problem in recurrent neural networks, and at the same time, it also largely eliminates the problems of gradient explosion and gradient disappearance during the training process of recurrent neural network models.

Description of drawings

Fig. 1 is the flow chart of the real-time continuous prediction method of human motion intention based on myoelectricity;

Fig. 2 is a schematic diagram of the position where the signal acquisition device is pasted;

Figure 3 is the verification experimental environment;

Figure 4 shows the lower limb joint angle prediction model network constructed based on LSTM.

Detailed ways

The present invention is described in detail below with reference to specific embodiments.

Specific embodiment one:

As shown in FIG. 1 , the present invention provides a method for real-time continuous prediction of human motion intention based on myoelectricity, comprising the following steps:

A method for real-time continuous prediction of human motion intention based on electromyography, comprising the following steps;

Step 1: Extract the EMG signal and IMU signal of the unilateral lower limb of the human body, and set the evaluation threshold; the step 1 is specifically:

As shown in Figure 2, the signal acquisition device is attached to the rectus femoris, vastus lateralis, vastus medialis, tibialis anterior, semitendinosus, biceps femoris long head, lateral gastrocnemius and medial gastrocnemius muscles on the unilateral lower limb of the human body Eight muscles extract EMG signals and IMU signals, and set evaluation thresholds;

Video recording is performed, and the motion mode label corresponding to each data point is judged through the video.

As shown in FIG. 3 , the present invention aims at five lower limb movement modes, namely flat ground, up stairs, down stairs, up slope and down slope, and develops knee joint angle prediction models for different movement modes based on long short-term memory neural network. , which can predict the future knee joint angle based on the EMG signal at the current moment and the knee joint angle, and perceive and predict the motion intention of the lower limbs of the human body from the perspective of quantitative analysis.

Long Short-term Memory (LSTM) neural network is one of the variants of recurrent neural network. It has relatively mature applications in the fields of speech recognition and language translation, and is very suitable for processing time series information. LSTM filters relevant information by setting a "gate structure" to solve the long-term dependency problem in recurrent neural networks, and at the same time, it also largely eliminates the problems of gradient explosion and gradient disappearance during the training process of recurrent neural network models. The lower limb joint angle prediction model network based on LSTM proposed by the present invention consists of an EMG signal feature extractor and an LSTM angle predictor. The model network structure is shown in Figure 4. The EMG signal feature extractor is used, and the linear layer outputs the extracted EMG Feature vector. The LSTM angle predictor is used as a lower limb joint angle predictor. The current joint angle and the feature vector obtained in the first part are used as input, and the output is the joint angle at the future time.

The signal acquisition device adopts the Delsys Trigno EMG acquisition system, the system includes a wireless communication base station and 16 wireless EMG electrodes, and each electrode has a built-in EMG and IMU signal collector.

Step 2: Select the IMU signal collected by any electrode of the thigh and calf respectively, and obtain the knee joint angle at the current moment through the lower limb kinematics solution algorithm;

The step 2 is specifically:

Select the IMU signal collected by any electrode of the thigh and calf, and transmit the collected fixed-length data of the collected IMU signal to the NVIDIA Jetson TX2 development board through the wireless module, and carry it on the lower limb kinematics solution of the NVIDIA Jetson TX2 development board. The algorithm obtains the knee joint angle at the current moment, and the frequency of the knee joint angle at the current moment is consistent with the frequency of the IMU signal. The NVIDIA Jetson TX2 development board is equipped with an NVIDIA Pascal architecture GPU, 2 Denver 64-bit CPUs and a quad-core A57 composite processor, and is equipped with a Linux Ubuntu operating system and a communication interface.

Step 3: Use the sliding time window method to resample the lower limb EMG signal and the knee joint angle at the current moment; both ends of the sliding time window correspond to the synchronization points of the EMG signal and the joint angle signal, and the step length of the sliding time window ensures the The EMG and angle data in the time window are synchronized with each other, and the sliding time window method is used to resample the lower limb EMG signal and the knee joint angle at the current moment.

Step 4: Establish five LSTM angle predictors with the same network structure and correspond to the five motion modes; put the EMG signal and the knee joint angle at the current moment into the corresponding LSTM angle predictor for training according to the labeled motion mode label, The knee joint angle vector of the same data length after the timestamp is shifted back by 27ms-300ms is used as the predictor training label; five LSTM angle predictors with the same network structure are established and correspond to the five motion patterns; the EMG signal is compared with the current The knee joint angle at the moment is put into the corresponding LSTM angle predictor for training according to the marked motion mode label, and the knee joint angle vector of the same data length after the timestamp is moved back by 27ms-300ms is used as the predictor training label, and k-fold cross-validation is used. method, each time a sample is taken as the test set, the remaining k-1 samples are used as the training set, and the experiment is repeated k times, so that each sample can be used as a test set. Before each training starts, the training data is randomly shuffled in units of samples, which can reduce the overfitting effect of the fixed data sequence on the parameters of the prediction model.

Step 5: Determine the current motion pattern of the trained LSTM angle predictor through the motion pattern classification algorithm, and input the data to the corresponding LSTM predictor for real-time angle prediction according to the current motion pattern;

The step 5 is specifically: the EMG signal and the IMU signal are each collected fixed-length data and transmitted to the NVIDIA Jetson TX2 development board through the wireless module, and judged according to the EMG and IMU signals by the motion mode classifier mounted on the NVIDIA Jetson TX2 board. The motion mode is saved, and the real-time knee joint angle is solved and saved by the lower limb kinematics solution algorithm equipped with the NVIDIA Jetson TX2 development board. The corresponding LSTM angle predictor performs angle prediction and saves the result.

Step 6: Move the time stamp of the predictor output angle value forward by 27ms-300ms as a whole, and compare it with the real-time knee joint angle to obtain the evaluation standard root mean square error RMSE. When the evaluation standard root mean square error RMSE is less than the set threshold , the predictor output angle is accurate.

The step 6 is specifically: extracting the prediction result of the LSTM angle predictor and the lower limb kinematics solution algorithm to solve the real-time knee joint angle, moving the time stamp of the angle value output by the LSTM angle predictor forward as a whole by 27ms-300ms, and combining it with the lower limb movement. Learn the solution algorithm to solve the real-time knee joint angle, and obtain the evaluation standard root mean square error RMSE, which is expressed by the following formula:

Among them, θ _p is the prediction result of the LSTM angle predictor, θ _r represents the real-time knee joint angle, n=k;

When the evaluation standard root mean square error RMSE is less than the set threshold, the predictor output angle is accurate; when the evaluation standard root mean square error RMSE is greater than the set threshold, re-collect data for training.

The NVIDIA Jetson TX2 development board is equipped with an NVIDIA Pascal architecture GPU, 2 Denver 64-bit CPUs and a quad-core A57 composite processor, and is equipped with a Linux Ubuntu operating system and a communication interface.

The above is only a preferred embodiment of a method for real-time continuous prediction of human motion intention based on myoelectricity. The technical solutions under the idea all belong to the protection scope of the present invention. It should be pointed out that for those skilled in the art, some improvements and changes without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims (10)

1. a kind of human body motion intention real-time continuous prediction method based on myoelectricity, is characterized in that: comprise the following steps; Step 1: Extract the EMG signal and IMU signal of the unilateral lower limb of the human body, and set the evaluation threshold; Step 2: Select the IMU signal collected by any electrode of the thigh and calf respectively, and obtain the knee joint angle at the current moment through the lower limb kinematics solution algorithm; Step 3: Use the sliding time window method to resample the lower limb EMG signal and the knee joint angle at the current moment; Step 4: Establish five LSTM angle predictors with the same network structure and correspond to the five motion modes; put the EMG signal and the knee joint angle at the current moment into the corresponding LSTM angle predictor for training according to the labeled motion mode label, The knee joint angle vector of the same data length after the timestamp is shifted back by a certain time is used as the predictor training label; Step 5: According to the trained LSTM angle predictor, determine the current motion mode through the motion mode classification algorithm, and input the data to the corresponding LSTM predictor for real-time angle prediction according to the current motion mode; Step 6: Move the time stamp of the predictor output angle value forward for a certain time as a whole, and compare it with the real-time knee joint angle to obtain the evaluation standard root mean square error RMSE, when the evaluation standard root mean square error RMSE is less than the set threshold , the predictor output angle is accurate. 2. a kind of human body motion intention real-time continuous prediction method based on myoelectricity according to claim 1, is characterized in that: described step 1 is specifically: Attach the signal acquisition device to the rectus femoris, vastus lateralis, vastus medialis, tibialis anterior muscle, semitendinosus, long head of biceps femoris, lateral gastrocnemius and medial gastrocnemius muscle to extract EMG signals and IMU signal, and set the evaluation threshold; Video recording is performed, and the motion mode label corresponding to each data point is judged through the video. 3. a kind of human body motion intention real-time continuous prediction method based on myoelectricity according to claim 2, is characterized in that: described signal acquisition equipment adopts Delsys Trigno myoelectricity acquisition system, and described system comprises wireless communication base station and 16 Wireless EMG electrodes, each electrode has built-in EMG and IMU signal collectors. 4. a kind of human body motion intention real-time continuous prediction method based on myoelectricity according to claim 1, is characterized in that: described step 2 is specifically: Select the IMU signal collected by any one of the electrodes on the thigh and calf respectively, and transmit the collected fixed length data of the IMU signal to the development board through the wireless module. Knee joint angle, the frequency of the knee joint angle at the current moment is consistent with the frequency of the IMU signal. 5. a kind of human body motion intention real-time continuous prediction method based on myoelectricity according to claim 1, is characterized in that: the synchronizing point of EMG signal and joint angle signal corresponding to both ends of sliding time window, the step of sliding time window The EMG and angle data in the time window after length-preserving sliding are synchronized with each other, and the sliding time window method is used to resample the EMG signal of the lower extremity and the knee joint angle at the current moment. 6. a kind of human body motion intention real-time continuous prediction method based on myoelectricity according to claim 1, is characterized in that: establish the LSTM angle predictor of five identical network structures, and correspond with five kinds of motion patterns; The signal and the knee joint angle at the current moment are put into the corresponding LSTM angle predictor for training according to the marked motion mode label, and the knee joint angle vector of the same data length after the timestamp is shifted back by 27ms-300ms is used as the predictor training label, using k Fold cross-validation method, each time one sample is taken as the test set, the remaining k-1 samples are used as the training set, and the experiment is repeated k times, so that each sample can be used as a test set. 7 . The method for real-time continuous prediction of human motion intention based on myoelectricity according to claim 6 , wherein: before each training starts, the training data is randomly shuffled in units of samples. 8 . 8. a kind of real-time continuous prediction method of human body motion intention based on myoelectricity according to claim 1, is characterized in that: described step 5 is specifically: described EMG signal and IMU signal each collect fixed length data to transmit through wireless module To the development board, the motion pattern classifier installed on the board is used to determine the motion pattern according to the EMG and IMU signals and save it, and the real-time knee joint angle is calculated and saved by the lower limb kinematics solution algorithm mounted on the development board. The real-time knee joint angle and the corresponding EMG signal are input to the corresponding LSTM angle predictor through the classified motion pattern for angle prediction, and the result is saved. 9. a kind of human body motion intention real-time continuous prediction method based on myoelectricity according to claim 1, is characterized in that: described step 6 is specifically: the prediction result of extracting LSTM angle predictor and lower limb kinematics solving algorithm solve Real-time knee joint angle, the LSTM angle predictor output angle value timestamp is moved forward as a whole by 27ms-300ms, and the real-time knee joint angle is solved with the lower limb kinematics solution algorithm to obtain the evaluation standard root mean square error RMSE, which is expressed by the following formula Evaluation standard mean square error:

Among them, θ _p is the prediction result of the LSTM angle predictor, θ _r is the real-time knee joint angle, and n is the number of trials; When the evaluation standard root mean square error RMSE is less than the set threshold, the predictor output angle is accurate; when the evaluation standard root mean square error RMSE is greater than the set threshold, re-collect data for training. 10. a kind of human body motion intention real-time continuous prediction method based on myoelectricity according to claim 4 or 8, is characterized in that: described development board is equipped with NVIDIA Pascal architecture GPU, 2 Denver 64-bit CPUs and quad-core A57 Composite processor with Linux Ubuntu operating system and communication interface.

Images