CN116459114A - Home-based hand function rehabilitation training system and rehabilitation training method for stroke patients - Google Patents

Abstract

The invention provides a home hand function rehabilitation training system and a rehabilitation training method for cerebral apoplexy patients, wherein the system comprises orthopedic gloves, an upper limb movement data acquisition module and a software platform; interacting with an interactive hand function rehabilitation training game in a software platform through the orthopedic glove, and capturing gesture perception information of a cerebral apoplexy patient in the interaction process; the upper limb movement data acquisition module captures an upper limb movement video in the interaction process of cerebral apoplexy; and the software platform recognizes the upper limb movement posture of the cerebral apoplexy patient according to the gesture sensing information and the upper limb movement video, feeds back the recognized compensatory movement posture to the cerebral apoplexy patient in real time, and guides the cerebral apoplexy patient to actively correct the compensatory movement posture. The invention provides a home hand function rehabilitation training system for a cerebral apoplexy patient, which can guide the recovery of hand functions of the cerebral apoplexy patient.

Description

Home-based hand function rehabilitation training system and rehabilitation training method for stroke patients

technical field

The present invention relates to the field of medical devices, and more specifically, to a home-based hand function rehabilitation training system and rehabilitation training method for stroke patients.

Background technique

According to statistics, up to 85% of stroke survivors have upper limb (hand and arm) dysfunction after stroke. Upper extremity rehabilitation after stroke is a long-term process. Although the ideal course of upper extremity rehabilitation should be to achieve the maximum recovery of upper limbs, stroke patients have been receiving rehabilitation treatment in the hospital. However, due to the constraints of medical and health resources, economy and other aspects, the upper limb rehabilitation service needs of most stroke patients cannot be met, and community/family-based rehabilitation has become a trend. On the other hand, the upper limb training of stroke patients usually follows the training rule of "from near to far, from coarse to fine", and the sequence of upper limb joint rehabilitation is also "from near to far". Therefore, hand function recovery tends to be slower and poorer in stroke patients compared with proximal joints of the upper extremity, such as the shoulder and elbow.

The home-based hand function rehabilitation system based on rehabilitation robots can meet the rehabilitation needs of stroke patients to a certain extent. However, the existing rehabilitation treatment system is often costly. In addition, compared with the traditional rehabilitation in the hospital, in the robot-assisted rehabilitation treatment system, due to the lack of professional physicians’ guidance, stroke patients may adopt compensatory exercises during rehabilitation training. Compensatory exercise refers to the new movement pattern formed by stroke patients through the adjustment or replacement of the remaining movement components after stroke, that is, the original hand function is replaced, replaced or completely replaced by different end-effectors or limb functions. Clinical studies have shown that this compensatory behavior can hinder the progress of upper limb rehabilitation in stroke patients and may lead to new orthopedic problems.

Contents of the invention

Aiming at the technical problems existing in the prior art, the present invention provides a home-based hand function rehabilitation training system and rehabilitation training method for stroke patients.

According to the first aspect of the present invention, a home-based hand function rehabilitation training for stroke patients is provided, including orthopedic gloves, an upper limb movement data acquisition module and a software platform, the software platform is integrated with an interactive hand function rehabilitation training game and a compensation recognition feedback module;

The orthopedic glove is used to interact with the interactive hand function rehabilitation training game in the software platform, capture gesture perception information of stroke patients during the interaction process, and send the gesture perception information to the software platform through wireless communication;

The upper limb movement data acquisition module is used to capture the upper limb movement video of the stroke patient during the interaction process with the interactive hand function rehabilitation training game, and send the upper limb movement video to the software platform;

The software platform is used to identify the upper limb movement posture of the stroke patient according to the gesture perception information and the upper limb movement video, and feed back the recognized compensatory movement posture to the stroke patient in real time, and guide the stroke patient to actively correct the compensatory movement posture.

On the basis of the above technical solution, the present invention can also make the following improvements.

Optionally, the orthopedic glove includes a soft baseball glove, a cable tie, a microcontroller with a Bluetooth module and a six-axis attitude sensor, a linear motor, and a curvature sensor fixed on each finger of the orthopedic glove;

The user wears the orthopedic glove, and the microcontroller drives the user's finger to stretch or contract by controlling the linear motor to pull the strip;

Each of the bending sensors is used to sense the corresponding finger bending data of the user's hand, and send the finger bending data to the software platform through the Bluetooth module;

The six-axis posture sensor is used to acquire the palm posture data of the user, and send the palm posture data to the software platform through the bluetooth module.

Optionally, the upper limb movement data acquisition module includes two RGB cameras fixed at different positions, one of which is arranged directly in front of the stroke patient, and the other is arranged on the top of the stroke patient's head. The two cameras are respectively used to shoot the front-view movement video and the top-view movement video of the stroke patient during the interaction process, and transmit the movement video to the software platform through the local area network.

Optionally, the software platform is configured to recognize the upper limb movement posture of a stroke patient according to the gesture perception information and the upper limb movement video, including:

According to the stroke patient's interaction with the rehabilitation training game, the bending data of each finger, the palm posture data, and the obtained frontal view motion image and overlooking motion image of the stroke patient, the motion feature values of multiple key parts of the stroke patient's upper limbs are obtained;

Based on the motion feature value, a multi-label classifier is used to classify and identify non-compensated and multiple typical compensated motion postures;

Wherein, the various typical compensatory movement postures include trunk forward leaning, trunk rotation, shoulder lifting and elbow underextension.

Optionally, according to the stroke patient's interaction with the rehabilitation training game, the bending data of each finger, the palm posture data, and the acquired frontal view motion image and overlooking motion image of the stroke patient, the motion feature values of multiple key parts of the stroke patient's upper limbs are obtained, including:

Based on the upper limb motion images of stroke patients captured by two cameras, the two-dimensional coordinate information of the key points of the user's upper body skeleton is obtained from two different perspectives based on OpenPose;

According to the two-dimensional coordinate information of the key points of the upper body bones of the user, based on two camera calibration parameters and the least square method, obtain the corresponding three-dimensional coordinate information of the key points of the upper body bones of the user;

Based on the three-dimensional coordinate information of the key points of the human skeleton and the constructed kinematics model of the upper body of the human body, the movement angle values of multiple key parts of the upper limbs of the stroke patient are obtained.

Optionally, the acquisition of the movement angle values of multiple key parts of the stroke patient's upper limbs includes shoulder joint adduction/abduction angle values, shoulder joint internal rotation/external rotation angle values, shoulder joint extension angle values, elbow joint bending angle values, trunk forward flexion angle values, trunk lateral flexion angle values, and trunk rotation angle values.

Optionally, according to the two-dimensional coordinate information of the key points of the upper body bones of the user, the corresponding three-dimensional coordinate information of the key points of the upper body bones of the user is obtained based on two camera calibration parameters and the least square method, including:

For each camera, the coordinate transformation relationship between its coordinates in the pixel coordinate system and the world coordinate system can be expressed by formula (1):

Among them, (X _w , Y _w , Z _w ) are coordinates in the world coordinate system; R, T are the rotation matrix and translation matrix of the world coordinate system relative to the camera coordinate system; Z _c is the distance from the object to the optical center; f is the focal length of the camera; (u ₀ , v ₀ ) is the coordinates of the center point of the image;

The conversion of the coordinates in the pixel coordinate system of the two cameras to the coordinates in the world coordinate system is shown in formula (2):

Equation (2) can be decomposed into 6 equations, among which, K ₁ M ₁ , K ₂ M ₂ are obtained through camera calibration, which are known quantities; (u ₁ , v ₁ ), (u ₂ , v ₂ ) are pixel coordinates, and the original color image can be directly obtained after OpenPose processing, which are known quantities; the remaining 5 unknown quantities, Z _c1 , Z _c2 , X _w , Y _w , Z _w , are obtained by using the least squares method to obtain the optimal solution, which is The three-dimensional coordinates (X _w , Y _w , Z _w ) of bone key points in the world coordinate system.

Optionally, also include:

Realize displaying the motion video with the upper limb bone key point mark and the recognized compensatory motion posture on the interactive interface of the software platform;

Wherein, if one or more compensatory motion postures are recognized, the type of compensatory motion posture is displayed on the interactive interface of the software platform in the form of text; if the recognition result is an uncompensated motion posture, then no compensation is displayed on the interactive interface of the software platform.

According to the second aspect of the present invention, there is provided a home-based hand function rehabilitation training method for stroke patients, comprising:

Based on the interaction between orthopedic gloves and interactive hand function rehabilitation training games, capture the gesture perception information of stroke patients;

Capture the upper limb movement video of stroke patients during the interaction with the interactive hand function rehabilitation training game;

According to the gesture perception information and the upper limb movement video, the stroke patient's upper limb movement posture is recognized, and the recognized compensatory movement posture is fed back to the stroke patient in real time, and the stroke patient is guided to actively correct the compensatory movement posture.

Optionally, according to the gesture perception information and the upper limb movement video, identifying the stroke patient's upper limb movement posture includes:

According to the stroke patient's interaction with the rehabilitation training game, the bending data of each finger, the palm posture data, and the obtained frontal view motion image and overlooking motion image of the stroke patient, the motion feature values of multiple key parts of the stroke patient's upper limbs are obtained;

Based on the motion feature value, a multi-label classifier is used to classify and identify non-compensated and multiple typical compensated motion postures;

Wherein, the various typical compensatory movement postures include trunk forward leaning, trunk rotation, shoulder lifting and elbow underextension.

The home-based hand function rehabilitation training system and rehabilitation training method for stroke patients provided by the present invention have the following beneficial effects:

(1) A simple orthopedic glove that can provide driving force for the fingers of stroke patients is designed. With the driving force provided by the glove, stroke patients can freely complete finger flexion and extension. In the process of hand rehabilitation training for stroke patients, the orthopedic glove can be used to well train the patient's hand joint mobility, hand muscle ability, and finger coordination ability.

(2) The hand function rehabilitation training game is designed based on the basic training actions of daily living activities, which can not only improve the enthusiasm of stroke patients to participate in rehabilitation training, but also help stroke patients meet the needs of hand function rehabilitation training in daily life scenes as much as possible.

(3) The software platform can guide stroke patients to actively correct compensatory movement postures in the process of hand function rehabilitation training, so as to better complete the rehabilitation training tasks with the help of the rehabilitation training system and speed up the upper limb rehabilitation process.

Description of drawings

Fig. 1 is a schematic structural diagram of a home-based hand function rehabilitation training system for stroke patients provided by the present invention;

Fig. 2 is the schematic diagram of orthopedic glove structure;

Fig. 3 is the schematic flow chart of obtaining the three-dimensional coordinates of the key points of the human upper body skeleton by the upper limb movement data acquisition module;

FIG. 4 is a schematic diagram of the process of acquiring the user's upper limb movement features;

Fig. 5 is a schematic flow chart of user's compensatory motion gesture recognition;

Fig. 6 is the overall rendering of the present invention;

Fig. 7 is a schematic flowchart of a home-based hand function rehabilitation training method for stroke patients provided by the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention. In addition, the technical features in each embodiment or a single embodiment provided by the present invention can be combined arbitrarily with each other to form a feasible technical solution. This combination is not restricted by the sequence of steps and/or structural composition mode, but must be based on the realization of those skilled in the art. When the combination of technical solutions is contradictory or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Based on the problems in the background technology, the present invention combines robot-assisted hand function rehabilitation training for stroke patients with compensatory movement posture recognition and feedback functions, which can not only meet the hand function rehabilitation training needs of stroke patients in the home environment, but also guide stroke patients to actively correct compensatory movement postures during rehabilitation training, thereby reducing the adverse effects of compensatory movements on rehabilitation training and reducing dependence on clinical therapists.

Figure 1 is a home-based hand function rehabilitation training system for stroke patients provided by the present invention. The system mainly includes orthopedic gloves, an upper limb movement data acquisition module and a software platform. The software platform integrates an interactive hand function rehabilitation training game and a compensation recognition feedback module.

The orthopedic glove is used to interact with the interactive hand function rehabilitation training game in the software platform, capture gesture perception information of stroke patients during the interaction process, and send the gesture perception information to the software platform through wireless communication.

The upper limb movement data acquisition module is used to capture the upper limb movement video of stroke patients during the interaction process with the interactive hand function rehabilitation training game, and send the upper limb movement video to the software platform.

The software platform is used to identify the upper limb movement posture of the stroke patient according to the gesture perception information and the upper limb movement video, and feed back the recognized compensatory movement posture to the stroke patient in real time, and guide the stroke patient to actively correct the compensatory movement posture.

It can be understood that the rehabilitation training system provided by the present invention includes orthopedic gloves, an upper limb movement data acquisition module, and a software platform integrating an interactive hand function rehabilitation training game and a compensation recognition feedback module.

The orthopedic glove, as a system actuator, performs information interaction with the software platform through the Bluetooth module, and is used to provide assistance for hand training actions in the process of hand function rehabilitation training for stroke patients. The upper limb motion data acquisition module is used to capture the motion video of the stroke patient's trunk, shoulder joint and elbow joint, and transmit the motion video to the software platform. The software platform is used to receive the sensor data sent by the orthopedic glove, realize the real-time interaction between the orthopedic glove and the rehabilitation training game, and record the training information at the same time; it is used to process the upper limb movement video of the stroke patient collected by the upper limb movement data acquisition module, and realize the classification and recognition of the compensatory movement posture; it is used to give the stroke patient real-time feedback on the compensatory movement posture, and guide the stroke patient to actively correct the compensatory movement posture.

As an example, the orthopedic glove includes a soft baseball glove, a cable tie, a microcontroller with a Bluetooth module and a six-axis attitude sensor, a linear motor, and a curvature sensor fixed on each finger of the orthopedic glove.

The user wears the orthopedic glove, and the micro-controller drives the user's finger to stretch or contract by controlling the linear motor to pull the strip; each of the curvature sensors is used to sense the corresponding finger bending data of the user's hand, and send the finger bending data to the software platform through the Bluetooth module; the six-axis attitude sensor is used to obtain the user's palm posture data, and send the palm posture data to the software platform.

It can be understood that, referring to FIG. 2 , it is a schematic structural diagram of an orthopedic glove. The orthopedic glove is mainly made of a specially tailored baseball glove, and a cable tie is used to simulate the tendon of a human hand. In order to drive the extension and contraction of the five fingers, the cable tie is fixed on each finger of the baseball glove through the cable tie claw. The back of the four fingers except the thumb is fixed with the cable tie. The end of the cable tie is glued and then fixed on the linear motor; In order to ensure the linear motion track of the motor, use the motor fixing slot (3D printing) to fix the motor. The movement of the motor is controlled using physical buttons through microcontroller programming. In order to realize the interaction with the rehabilitation training game and obtain the hand movement data associated with different compensatory movement modes, a bending sensor is fixed on each finger of the baseball glove, and the microcontroller obtains the current bending angle data of the finger and palm posture data, and transmits them to the computer through the Bluetooth module. The linear motor and microcontroller are powered by a rechargeable lithium battery.

As an example, the upper limb movement data acquisition module includes two RGB cameras fixed at different positions, one of which is arranged directly in front of the stroke patient, and the other is arranged on the top of the stroke patient's head, and the two cameras are respectively used to shoot the front view movement video and the top view movement video of the stroke patient during the interaction process, and transmit the movement video to the software platform through the local area network.

It is understandable that the upper limb movement data acquisition module consists of two common RGB cameras (camera models) fixed at different positions. A camera is arranged directly in front of the user, facing the user at a slight angle. Another camera is placed above the user's head, looking down at the user. The layout of the two cameras ensures that the movement of the shoulder joint, elbow joint and torso can be clearly captured when the user completes hand function rehabilitation training with this system.

The upper limb motion data acquisition module transmits the captured shoulder joint, elbow joint and trunk motion videos to the software platform through the local area network, and the software platform further processes and calculates the motion videos to obtain motion feature values that can be used for automatic classification and recognition of compensatory motion postures.

Understandably, the software platform integrates hand function rehabilitation training games and compensatory recognition feedback functions. The hand function rehabilitation training game provides an immersive hand function rehabilitation training environment for stroke patients. The compensatory recognition feedback function classifies and recognizes the compensatory movement postures adopted by stroke patients in the process of hand function rehabilitation training in real time, and gives real-time feedback to stroke patients according to the recognition results, and guides stroke patients to actively correct compensatory movement postures.

As an embodiment, the software platform, the software platform, is used to recognize the upper limb movement posture of the stroke patient according to the gesture perception information and the upper limb movement video, including: according to the stroke patient’s interaction with the rehabilitation training game, the bending data of each finger, the palm posture data, and the obtained front-view motion image and overlooking motion image of the stroke patient, obtain the motion feature values of multiple key parts of the stroke patient’s upper limb; Classification and recognition are carried out; wherein, the multiple typical compensatory movement postures include trunk forward tilt, trunk rotation, shoulder lifting and elbow underextension.

It can be understood that the present invention combines the kinematic analysis method of upper limb joints and torso of stroke patients with the principal component analysis method of multiple motion variables of upper limb joints and torso to obtain the user's motion characteristic value. As an example, the acquisition of motion feature values of multiple key parts of the stroke patient’s upper limbs based on the bending data of each finger, palm posture data, and the acquired front-facing moving images and overlooking moving images of the stroke patient during the interaction with the rehabilitation training game includes: based on the upper limb moving images of the stroke patient captured by two cameras, acquiring two-dimensional coordinate information of key points of the user’s upper body bones based on two different perspectives based on OpenPose; The least squares method is used to obtain the corresponding three-dimensional coordinate information of the key points of the user's upper body skeleton; based on the three-dimensional coordinate information of the key point of the human skeleton and the constructed human upper body kinematics model, the movement angle values of multiple key parts of the upper limb of the stroke patient are obtained.

It should be noted that based on the OpenPose open source library, the BODY_25 output model was selected to obtain the three-dimensional coordinates of the key points of the upper body bones of stroke patients (points 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, and 12 in the OpenPose output model).

Wherein, according to the two-dimensional coordinate information of the key points of the upper body bones of the user, based on two camera calibration parameters and the least square method, the corresponding three-dimensional coordinate information of the key points of the upper body bones of the user is obtained, including:

For each camera, the coordinate transformation relationship between its coordinates in the pixel coordinate system and the world coordinate system can be expressed by formula (1):

Among them, (X _w , Y _w , Z _w ) are coordinates in the world coordinate system; R, T are the rotation matrix and translation matrix of the world coordinate system relative to the camera coordinate system; Z _c is the distance from the object to the optical center; f is the focal length of the camera; (u ₀ , v ₀ ) is the coordinates of the center point of the image;

The conversion of the coordinates in the pixel coordinate system of the two cameras to the coordinates in the world coordinate system is shown in formula (2):

Equation (2) can be decomposed into 6 equations, among which, K ₁ M ₁ , K ₂ M ₂ are obtained through camera calibration, which are known quantities; (u ₁ , v ₁ ), (u ₂ , v ₂ ) are pixel coordinates, and the original color image can be directly obtained after OpenPose processing, which are known quantities; the remaining 5 unknown quantities, Z _c1 , Z _c2 , X _w , Y _w , Z _w , are obtained by using the least squares method to obtain the optimal solution, which is The three-dimensional coordinates (X _w , Y _w , Z _w ) of bone key points in the world coordinate system.

Through the above method, according to the motion posture and motion video of the stroke patient during the interaction process, the three-dimensional coordinate information of the key points of the human skeleton is obtained, and based on the three-dimensional coordinate information of the key points of the human skeleton, combined with the constructed kinematic model, the user's motion feature value is solved.

Wherein, the acquisition of the motion feature values of multiple key parts of the upper limbs of stroke patients includes the angle values of shoulder joint adduction/abduction angle, shoulder joint internal rotation/external rotation angle, shoulder joint protraction angle, elbow bending angle, trunk forward flexion angle, trunk side flexion angle, and trunk rotation angle, and other key motion angle values of upper limbs.

It is understandable that the software platform integrates hand function rehabilitation training games and compensatory recognition feedback functions. The hand function rehabilitation training games provide stroke patients with an immersive hand function rehabilitation training environment. The compensatory recognition feedback function classifies and recognizes the compensatory movement postures adopted by stroke patients in the process of hand function rehabilitation training in real time, and gives real-time feedback to stroke patients according to the recognition results, and guides stroke patients to actively correct compensatory movement postures.

Table 1 is the specific content of the rehabilitation training game in the present invention. In order to cover the hand function rehabilitation needs in the daily life scenes of stroke patients as much as possible, the hand function rehabilitation training game was developed based on the basic training actions of daily life activities.

Table 1 The specific content of the rehabilitation training game

Through the analysis of CAHAI (The Chedoke Arm and Hand Activity Inventory) and other related research materials, four basic training actions for daily living activities are summarized: arm extension, grasping, pinching, and moving and manipulating objects. On the basis of fully considering the above four basic training actions of daily living activities, six kinds of hand function rehabilitation training games were finally developed. The highlight of the rehabilitation training game designed by this system is to start from the daily life scenes of stroke patients, through the summary of relevant knowledge, determine the four basic movements that must be involved in the hand movement in daily life activities, and then develop rehabilitation training games with the four basic movements as the core, hoping to help stroke patients recover more comprehensively the demand for hand motor function in daily life scenes.

As shown in Fig. 3, Fig. 4 and Fig. 5, the data processing flow chart of the compensatory motion posture recognition and feedback module in the present invention uses the motion characteristic values of the user's shoulder joint, elbow joint and trunk as input, and uses a multi-label classifier to classify and identify uncompensated and four typical compensatory motion postures, wherein the four typical compensatory motion postures include trunk leaning forward, trunk rotation, shoulder lifting, and insufficient elbow extension. If one or several compensatory movement postures are identified, the type of compensatory movement posture will be displayed on the software interface in the form of information enhancement. If the recognition result is an uncompensated movement posture, "Uncompensated" will be displayed directly.

Specifically, the OpenPose open source library is used to process two color images from two cameras at the same time to obtain the two-dimensional coordinate information of key points of human bones. Then, based on the camera calibration parameters and the least square method, the three-dimensional coordinate information of the bone key points at this moment is obtained. Next, build a human kinematics model, and calculate the movement angle values of the human shoulder joint, elbow joint, and trunk based on the camera rotation matrix and the Euler angle calculation method, including: shoulder joint adduction/abduction angle values, shoulder joint internal rotation/external rotation angle values, shoulder joint extension angle values, elbow joint bending angle values, trunk flexion angle values, trunk side flexion angle values, and trunk rotation angle values. These angle values are represented as a feature vector, which is used as input to the recognition model for automatic classification of compensatory motion poses. Considering the simultaneous occurrence of multiple compensatory movement postures, the multi-label k-nearest neighbor algorithm was selected to train the classification model, and the upper limb compensatory movement postures of stroke patients were classified and recognized based on the trained classification model.

Fig. 6 is an overall effect diagram of the present invention. As shown in Fig. 6, the user interacts with the rehabilitation training game with the assistance of orthopedic gloves to complete the hands-on functional rehabilitation training. At the same time, two RGB cameras capture the user's motion video in real time, and the motion video with the key points of the upper limb bones is displayed on the interactive interface in real time after being processed by the background algorithm of the software platform. The recognition results of the compensatory motion posture are also displayed on the interactive interface in the form of information enhancement, giving users real-time feedback and guiding the user to actively correct the compensatory motion posture. During the feedback process, if one or several compensatory motion postures are recognized, the type of compensatory motion posture will be displayed on the software interface in the form of text. If the recognition result is an uncompensated movement posture, "Uncompensated" will be displayed directly. Stroke patients correct and adjust their own exercise postures according to the compensatory exercise postures displayed on the software platform.

Fig. 7 is a home-based hand function rehabilitation training method for stroke patients provided by the present invention. The method is applied to a home-based hand function rehabilitation training system for stroke patients. The method includes:

S1, based on the interaction between orthopedic gloves and an interactive hand function rehabilitation training game, the gesture perception information of stroke patients is captured.

S2, capturing the upper limb movement video of the stroke patient during the interaction with the interactive hand function rehabilitation training game.

S3. Identify the upper limb movement posture of the stroke patient according to the gesture perception information and the upper limb movement video, and feed back the recognized compensatory movement posture to the stroke patient in real time, and guide the stroke patient to actively correct the compensatory movement posture.

Wherein, the recognition of the upper limb movement posture of the stroke patient according to the gesture perception information and the upper limb movement video includes: obtaining the motion feature values of multiple key parts of the stroke patient's upper limbs according to the stroke patient's frontal view motion image and the top view motion image; based on the motion feature values, using a multi-label classifier to classify and identify uncompensated and typical compensated motion postures.

The acquisition of motion feature values of multiple key parts of the stroke patient's upper limbs based on the stroke patient's front-facing motion images and overlooking motion images includes: acquiring two-dimensional coordinate information of human bone key points from two different perspectives based on the stroke patient's upper limb motion images collected by two cameras; obtaining corresponding three-dimensional coordinate information of human bone key points based on the two-dimensional coordinate information of human bone key points, based on two camera calibration parameters and the least square method; The movement angle values of multiple key parts of the patient's upper limbs. Based on the movement angle values of multiple key parts of the stroke patient's upper limbs, the patient's compensatory movement posture is identified.

It can be understood that the home-based hand function rehabilitation training method for stroke patients provided by the present invention corresponds to the home-based hand function rehabilitation training system for stroke patients provided in the above-mentioned embodiments. The relevant technical features of the home-based hand function rehabilitation training method for stroke patients can refer to the relevant technical features of the home-based hand function rehabilitation training system for stroke patients, and will not be repeated here.

A home-based hand function rehabilitation training system and rehabilitation training method for stroke patients provided by the embodiment of the present invention has beneficial effects:

(1) The designed and developed orthopedic glove can not only provide specific motor assistance for the hand motor dysfunction of stroke patients, but also has low manufacturing cost. Hand dysfunction caused by stroke includes stiffness, atrophy, and poor coordination of hand muscles. After the stroke recovery period, most stroke patients are able to regain the ability to flex their fingers voluntarily, but the extension function of the fingers is still limited due to improper flexor muscle activation and inability to activate the extensor muscles. Based on this, a simple orthopedic glove that can provide driving force for the fingers of stroke patients is designed. With the driving force provided by the glove, stroke patients can freely complete finger flexion and extension. In the process of hand rehabilitation training for stroke patients, the orthopedic glove can be used to well train the patient's hand joint mobility, hand muscle ability, and finger coordination ability.

(2) The hand function rehabilitation training game is designed based on the basic training actions of daily living activities, which can not only improve the enthusiasm of stroke patients to participate in rehabilitation training, but also help stroke patients meet the needs of hand function rehabilitation training in daily life scenes as much as possible.

(3) Design and develop a compensatory recognition and feedback module, which can guide stroke patients to actively correct compensatory movement postures in the process of hand function rehabilitation training, so as to better complete rehabilitation training tasks with the rehabilitation training system and speed up the upper limb rehabilitation process.

It should be noted that, in the foregoing embodiments, descriptions of each embodiment have their own emphases, and for parts that are not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special purpose computer, an embedded computer or other programmable data processing equipment to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing equipment produce means for realizing the functions specified in one or more procedures of the flow chart and/or one or more blocks of the block diagram.

These computer program instructions can also be stored in a computer-readable memory capable of directing a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means that implement the functions specified in one or more flows of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to generate computer-implemented processing, so that the instructions executed on the computer or other programmable equipment provide steps for realizing the functions specified in one flow or multiple flows of the flow chart and/or one or more square blocks of the block diagram.

While preferred embodiments of the invention have been described, additional changes and modifications to these embodiments can be made by those skilled in the art once the basic inventive concept is understood. Therefore, it is intended that the appended claims be construed to cover the preferred embodiment as well as all changes and modifications which fall within the scope of the invention.

Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and equivalent technologies thereof, the present invention also intends to include these modifications and variations.

Claims (10)

1. The home hand function rehabilitation training system for the cerebral apoplexy patient is characterized by comprising orthopedic gloves, an upper limb movement data acquisition module and a software platform, wherein the software platform is integrated with an interactive hand function rehabilitation training game and a compensation identification feedback module;

the orthopedic glove is used for interacting with the interactive hand function rehabilitation training game in the software platform, capturing gesture perception information of a cerebral apoplexy patient in the interaction process, and sending the gesture perception information to the software platform in a wireless communication mode;

the upper limb movement data acquisition module is used for capturing upper limb movement videos of a cerebral apoplexy patient in the interaction process with the interactive hand function rehabilitation training game and sending the upper limb movement videos to the software platform;

the software platform is used for identifying the upper limb movement posture of the cerebral apoplexy patient according to the gesture sensing information and the upper limb movement video, feeding back the identified compensatory movement posture to the cerebral apoplexy patient in real time, and guiding the cerebral apoplexy patient to actively correct the compensatory movement posture.

2. The rehabilitation training system according to claim 1, characterized in that said orthopedic glove comprises a soft baseball glove, a tie, a microcontroller with a bluetooth module and a six-axis attitude sensor, a linear motor and a curvature sensor fixed on each finger of said orthopedic glove;

the user wears the orthopedic glove, and the microcontroller drives the fingers of the user to extend or retract by controlling the linear motor to pull the rolling belt;

each bending sensor is used for sensing corresponding finger bending data of a user hand and sending the finger bending data to the software platform through a Bluetooth module;

the six-axis gesture sensor is used for acquiring palm gesture data of a user and sending the palm gesture data to the software platform through the Bluetooth module.

3. The rehabilitation training system according to claim 1, wherein the upper limb movement data acquisition module comprises two RGB cameras fixed at different positions, one of the cameras is arranged right in front of the cerebral apoplexy patient, the other camera is arranged on the top of the cerebral apoplexy patient, the two cameras are respectively used for shooting forward-looking movement videos and overlooking movement videos of the cerebral apoplexy patient in the interaction process, and the movement videos are transmitted to the software platform through a local area network.

4. The rehabilitation training system according to claim 3, wherein the software platform is configured to identify an upper limb movement posture of a cerebral apoplexy patient according to the gesture sensing information and the upper limb movement video, and comprises:

according to the bending data and palm posture data of each finger and the obtained front-view moving image and overlook moving image of the cerebral apoplexy patient in the process of interacting with a rehabilitation training game, obtaining the movement characteristic values of a plurality of key parts of the upper limb of the cerebral apoplexy patient;

based on the motion characteristic values, classifying and identifying uncompensated motion gestures and various typical compensated motion gestures by utilizing a multi-label classifier _；

Wherein the plurality of typical compensatory exercise positions include trunk forward tilting, trunk rotation, shoulder lifting, and elbow underextension.

5. The rehabilitation training system according to claim 4, wherein the acquiring motion feature values of a plurality of key parts of the upper limb of the cerebral apoplexy patient from the bending data of each finger, the palm posture data, and the acquired front view moving image and top view moving image of the cerebral apoplexy patient during the interaction with the rehabilitation training game comprises:

acquiring two-dimensional coordinate information of key points of bones of the upper body of a user at two different visual angles based on OpenPose based on upper limb moving images of a cerebral apoplexy patient captured by the two cameras;

according to the two-dimensional coordinate information of the key points of the upper body bones of the user, based on two camera calibration parameters and a least square method, obtaining corresponding three-dimensional coordinate information of the key points of the upper body bones of the user;

and acquiring the motion angle values of a plurality of key parts of the upper limb of the cerebral apoplexy patient based on the three-dimensional coordinate information of the key points of the human skeleton and the constructed human upper body kinematic model.

6. The rehabilitation training system according to claim 5, wherein the calculating the corresponding three-dimensional coordinate information of the upper body bone key point of the user based on the two camera calibration parameters and the least square method according to the two-dimensional coordinate information of the upper body bone key point of the user comprises:

for each camera, the coordinate in the pixel coordinate system and the coordinate conversion relationship in the world coordinate system thereof can be represented by the formula (1):

wherein, (X _w ,Y _w ,Z _w ) Is the coordinate under the world coordinate system; r, T are the rotation matrix and translation matrix of the world coordinate system relative to the camera coordinate system respectively; z is Z _c Distance from the object to the optical center; f is the focal length of the camera; (u) ₀ ,v ₀ ) Coordinates of a center point of the image; dx, dy is the physical size of each pixel of the photosensitive device in the camera; (u, v) is coordinates in a pixel coordinate system; k, M are camera internal parameters and external parameters respectively;

the conversion from coordinates in the pixel coordinate system of the two cameras to coordinates in the world coordinate system is shown in the formula (2):

formula (2) can be decomposed into 6 equations, where K ₁ M ₁ ,K ₂ M ₂ Obtained through camera calibration, is a known quantity; (u) ₁ ,v ₁ ),(u ₂ ,v ₂ ) The pixel point coordinates are obtained by directly obtaining an original color image after OpenPose processing, and the original color image is a known quantity; residual Z _c1 ,Z _c2 ,X _w ,Y _w ,Z _w 5 unknowns, and obtaining an optimal solution by adopting a least square method to obtain three-dimensional coordinates (X _w ,Y _w ,Z _w )。

7. The rehabilitation training system according to claim 5, wherein the acquisition of the movement angle values of the plurality of key parts of the upper limb of the cerebral apoplexy patient includes a shoulder adduction/abduction angle value, a shoulder internal rotation/external rotation angle value, a shoulder extension angle value, an elbow joint bending angle value, a trunk anteversion angle value, a trunk lateral version angle value, and a trunk rotation angle value.

8. The rehabilitation training system according to claim 5, further comprising:

displaying the motion video carrying the upper limb skeleton key point mark and the identified compensatory motion gesture on an interactive interface of the software platform;

if one or more compensatory motion gestures are identified, displaying the compensatory motion gesture types on an interactive interface of the software platform in a text form; and if the recognition result is the uncompensated motion gesture, displaying the uncompensated motion gesture on an interactive interface of the software platform.

9. A method for rehabilitation training of home hand function of a cerebral apoplexy patient, which is applied to the rehabilitation training system for home hand function of a cerebral apoplexy patient according to any one of claims 1 to 8, the method comprising:

capturing gesture perception information of a cerebral apoplexy patient in the process of interaction between the orthopedic glove and the interactive hand function rehabilitation training game;

capturing upper limb movement videos of a cerebral apoplexy patient in the interaction process with an interactive hand function rehabilitation training game;

and identifying the upper limb movement posture of the cerebral apoplexy patient according to the gesture sensing information and the upper limb movement video, and feeding back the identified compensatory movement posture to the cerebral apoplexy patient in real time to guide the cerebral apoplexy patient to actively correct the compensatory movement posture.

10. The rehabilitation training method according to claim 9, characterized in that the recognition of the upper limb movement posture of the cerebral apoplexy patient according to the gesture sensing information and the upper limb movement video comprises:

according to the bending data and palm posture data of each finger and the obtained front-view moving image and overlook moving image of the cerebral apoplexy patient in the process of interacting with a rehabilitation training game, obtaining the movement characteristic values of a plurality of key parts of the upper limb of the cerebral apoplexy patient;

based on the motion characteristic values, classifying and identifying uncompensated motion gestures and various typical compensated motion gestures by utilizing a multi-label classifier _；

Wherein the plurality of typical compensatory exercise positions include trunk forward tilting, trunk rotation, shoulder lifting, and elbow underextension.