CN113499173B - Real-time instance segmentation-based terrain identification and motion prediction system for lower artificial limb - Google Patents

Abstract

The invention provides a real-time instance segmentation-based terrain recognition and motion prediction system for lower limb prostheses, comprising: a binocular camera collects terrain information around the lower limb prosthesis, an image acquisition and information transmission hardware platform collects the information obtained by the binocular camera, and uploads the information. To the cloud server, the image acquisition and information transmission hardware platform is electrically connected to the lower limb prosthesis controller, and the lower limb prosthesis controller controls the movement mode of the lower limb prosthesis; the cloud server is deployed with a target terrain monitoring module, an instance segmentation module, a feature matching module and a position size The computing module, the terrain information is fed back to the image acquisition and information transmission hardware platform after being processed by the above modules in turn. The invention realizes the intelligent optimization and improvement of the multi-motion mode lower limb prosthesis, so that it can accurately perceive the surrounding terrain distribution in real time, correct and adjust the motion control strategy in time, and improve the exercise ability and wearing comfort of the lower limb prosthesis wearer.

Description

Real-time Instance Segmentation Based Terrain Recognition and Motion Prediction System for Lower Limb Prosthesis

technical field

The invention relates to the field of active environment perception of lower limb prostheses in a complex terrain environment, in particular to a terrain recognition and motion prediction system for lower limb prostheses based on real-time instance segmentation.

Background technique

The lower limb prosthesis is used to assist the disabled in sports and rehabilitation training. It is an external power mechanical disabled device designed according to the ergonomic structure characteristics of the disabled, which integrates machinery, electronics, sensors, intelligent control, transmission and other technologies. A series of auxiliary functions such as support, protection, and movement. The level of intelligent control and comfort of lower limb prostheses largely determines the exercise ability of the wearer.

In the Chinese invention patent document with the notification number CN209422174U, a power prosthetic environment recognition system that integrates vision is disclosed, including a prosthesis body, a power module, a motion sensing module, a visual detection module, and a control module. The power module is used to make the prosthesis body Motion, the motion sensing module is used to obtain the state information of the prosthesis body, the visual detection module is used to obtain the surrounding environment information of the prosthesis body, and the control module can judge the road conditions around the human body through the acquired state information of the prosthesis body and the surrounding environment information of the prosthesis body and obstacle information, predict the movement trend of the prosthesis, and judge the movement intention of the human body, so as to control the power module to make the prosthesis body move reasonably, so as to assist the patient to adapt to different road conditions or cross obstacles; , Perceive the movement intention of the human body in advance and continuously detect the road conditions around the human body. The data feedback is real-time and stable, which is convenient for patients to use.

At present, traditional lower limb prostheses have low intelligence, single movement mode, and prolonged strategy switching time; inaccurate recognition of the surrounding terrain environment, prolonged terrain recognition time, and no ability to predict the terrain; poor practicability in continuously changing complex terrain environments, etc. series of questions. These problems not only prevent the wearer of the lower limb prosthesis from recovering better movement ability, but even put the wearer at risk of injury.

Contents of the invention

Aiming at the defects in the prior art, the object of the present invention is to provide a lower limb prosthetic terrain recognition and motion prediction system based on real-time instance segmentation.

According to the real-time instance segmentation-based lower limb prosthesis terrain recognition and motion prediction system provided by the present invention, the local end includes a lower limb prosthesis, a lower limb prosthesis controller, a binocular camera, an image acquisition and information transmission hardware platform, and the cloud includes a cloud server;

The binocular camera is installed on the lower limb prosthesis to collect terrain information around the lower limb prosthesis. The image acquisition and information transmission hardware platform collects the information obtained by the binocular camera and uploads it to the cloud server. The lower limb prosthesis controller controls the lower limb prosthesis. movement of the prosthesis;

The cloud server is deployed with a target terrain monitoring module, an instance segmentation module, a feature matching module, and a position and size calculation module. After the terrain information is processed by the above modules in turn, it is fed back to the lower limb prosthesis controller.

Preferably, the lower limb prosthesis includes a knee joint and an ankle joint, the knee joint has one degree of freedom, and the ankle joint has two active degrees of freedom of flexion and extension and external rotation/varus.

Preferably, the binocular camera is installed at the front of the knee joint to collect RGB images of the terrain around the lower limb prosthesis, and the configuration of the binocular camera is: 1280*720 resolution, 30 frames/second sampling rate.

Preferably, the image acquisition and information transmission hardware platform is located in the internal gap of the lower limb prosthetic body, receives the RGB image collected by the binocular camera in real time, transmits it to the cloud server through the wireless network, and receives the terrain information fed back by the cloud server in real time, and outputs it to the lower limb Prosthetic motion controller.

Preferably, the cloud server is equipped with a CPU and a GPU, the CPU has the characteristics of multi-thread processing, and provides services for multiple users at the same time; the GPU speeds up RGB image processing.

Preferably, the lower limb prosthesis controller controlling the movement of the lower limb prosthesis includes the following steps:

Step S1: Based on the IMU, collect the movement data of the lower limbs of healthy people in different terrain environments, and store them in the lower limb prosthetic controller;

Step S2: the lower limb prosthesis controller controls and receives the current terrain data of the lower limb prosthesis from the cloud server;

Step S3: The lower limb prosthesis controller controls the lower limb prosthesis to adopt a motion state suitable for the terrain according to the motion data of the human lower limb;

Step S4: For terrain or obstacles that still have a certain distance, the lower limb prosthetic controller predicts whether it may pass and the estimated arrival time according to the current direction and speed of movement, so as to achieve a smoother state switching from the software.

Preferably, the target detection module uses the trained and optimized YoloV4-Lite convolutional neural network based on MobileNet to preliminarily identify different terrain information in the RGB image, use Box to mark them respectively, and output them to the instance segmentation module.

Preferably, the instance segmentation module uses a trained Deep Snake convolutional neural network based on circular convolution, uses the Extreme Points method to extract the initial contour of the Box obtained from target detection, and then uses the Deep Snake network to deform the initial contour to obtain an accurate fit The terrain contours of each terrain are output to the feature matching module.

Preferably, the feature matching module uses the Speeded Up Robust Features method to perform feature extraction and feature matching on the pixels within the range of the same type of terrain contour at the same moment processed by the binocular camera and processed by the instance segmentation module, respectively, according to the same Feature point parallax, combined with binocular camera parameters to calculate the depth information of feature points, and then calculate its three-dimensional coordinates.

Preferably, for the recognized terrain, the position and size calculation module calculates its position and length and width range according to the coordinates of the feature points located on the edge of the contour, and for the recognized steps, obstacles, etc., calculates its length, width and height according to the coordinates of its contour corner points size and location information.

Compared with the prior art, the present invention has the following beneficial effects:

1. The lower limb prosthesis terrain recognition and motion prediction system based on real-time instance segmentation of the present invention realizes the intelligent optimization and improvement of multi-motion mode lower limb prostheses, enabling it to accurately perceive the surrounding terrain distribution in real time, and timely correct and coordinate motion control strategies. Adjustment to improve the movement ability and wearing comfort of the lower limb prosthetic wearer.

Description of drawings

Other characteristics, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

1 is a schematic structural diagram of a lower limb prosthesis terrain recognition and motion prediction system based on real-time instance segmentation according to an embodiment of the present application;

Fig. 2 is a schematic diagram of the depth separable convolution in the lower limb prosthesis terrain recognition and motion prediction system based on real-time instance segmentation according to the embodiment of the present application;

Fig. 3 is the YOLOV4 network structure diagram in the lower limb prosthesis terrain recognition and motion prediction system based on real-time instance segmentation in the embodiment of the present application;

Fig. 4 is the Deep Snake network structure diagram based on circular convolution in the lower limb prosthesis terrain recognition and motion prediction system based on real-time instance segmentation in the embodiment of the present application;

Fig. 5 is a schematic diagram of binocular vision positioning in China of the lower limb prosthesis terrain recognition and motion prediction system based on real-time instance segmentation according to the embodiment of the present application.

Detailed ways

The present invention will be described in detail below in conjunction with specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

A terrain recognition and motion prediction system for lower limb prosthesis based on real-time instance segmentation, referring to Figure 1, includes a local terminal and a cloud. The local terminal includes a lower limb prosthesis, a lower limb prosthetic controller, a binocular camera, an image acquisition and information transmission hardware platform, and a cloud Including cloud servers.

The lower limb prosthesis has multi-motion modes. The limbs and feet of the prosthesis are made of 3D-printed ABS resin materials. The lower limb prosthesis includes knee joints and ankle joints. The knee joints and ankle joints are made of 1060 aluminum alloy materials. The knee joint and ankle joint of the lower limb prosthesis are driven by Maxon motors. The knee joint has one degree of freedom, and the ankle has two active degrees of freedom of flexion and extension and external rotation/varus.

The binocular camera is installed at the front end of the knee joint on the lower limb prosthesis, and is fixed by an electric pan/tilt to collect RGB images of the terrain around the lower limb prosthesis. The configuration of the binocular camera is: 1280*720 resolution, 30 frames per second Sampling Rate. The image acquisition and information transmission hardware platform is located in the internal gap of the lower limb prosthetic body. It receives the RGB image collected by the binocular camera in real time, transmits it to the cloud server through the wireless network, and receives the terrain information fed back by the cloud server in real time, and outputs it to the lower limb prosthesis motion controller. The lower limb prosthetic controller controls the movement mode of the lower limb prosthesis.

The lower limb prosthesis controller controls the movement of the lower limb prosthesis including the following steps:

Step S1: Based on the IMU, collect the movement data of the lower limbs of healthy people walking and running on sand, grass, and asphalt, as well as up and down ramps, steps, and across obstacles of different widths and heights, and store them in the lower limb prosthesis controller;

Step S2: the lower limb prosthesis controller controls and receives the current terrain data of the lower limb prosthesis from the cloud server;

Step S3: The lower limb prosthesis controller controls the lower limb prosthesis to adopt a motion state suitable for the terrain according to the motion data of the human lower limb;

Step S4: For terrain or obstacles that still have a certain distance, the lower limb prosthetic controller predicts whether it may pass and the estimated arrival time according to the current direction and speed of movement, and prepares for the movement strategy switch to achieve a smoother state from the software Toggle for improved prosthetic comfort.

The cloud server is deployed with target terrain monitoring module, instance segmentation module, feature matching module and position size calculation module. After the terrain information is processed by the above modules in sequence, it is fed back to the image acquisition and information transmission hardware platform. The image acquisition and information transmission hardware platform The lower limb prosthetic controller issues control commands.

The cloud server is equipped with a CPU and a GPU. The CPU has the characteristics of multi-thread processing and provides services for multiple users at the same time; the GPU speeds up RGB image processing. Considering the increase in the number of users and the problem of failure, its performance has strong scalability and a redundant backup mechanism.

The target terrain detection module needs to perform fine-grained image detection, design and build a detection framework based on deep convolutional neural network with good real-time performance and high accuracy, and complete the determination of the category and precise outline of the target object through the convolutional neural network. Collect the detection data set of the target terrain, complete the manual labeling, divide the training set, verification set, test set, and data preprocessing. Design and build a deep convolutional neural network with a suitable structure, and design a suitable loss function to train the deep convolutional neural network on the data set. According to the training results, the hyperparameters and network structure are continuously adjusted, aiming to train a deep convolutional neural network with good real-time performance and high recognition accuracy. The present invention uses the improved Mobilenet-based YoloV4-Lite to identify grass, sand, asphalt ground, and steps. The biggest feature of Mobilenet is the use of depth-separable convolution. This channel-by-channel linear convolution can guarantee feature Under the premise of extracting the effect, the amount of parameters is greatly reduced, as shown in Figure 2.

Referring to Figure 3, the input of the original YOLOV4 network is an RGB image with a height of 416 and a width of 416, and features are extracted through the CSPDarknet-53 backbone network. Three feature layers of different sizes are used to achieve multi-scale feature fusion, regression is performed on the offset of the target position relative to the default rectangular box (default box), and a classification confidence is output at the same time. The original YOLOV4 network structure is complex and the prediction speed is slow. In order to meet the detection speed requirements in hand-eye coordination, the network structure needs to be adjusted. Considering the trade-off between accuracy and speed, the MobileNet network is used as the backbone network to extract features, but the resolution of the input image is not changed, and the same number of features is used for target detection.

The instance segmentation module needs to perform further background segmentation on a series of rectangular boxes and terrain category labels obtained by the target terrain detection module to obtain the precise outline of each target terrain, and the algorithm requires low computational complexity and good real-time performance. As shown in Figure 4, the Deep Snake algorithm represents the shape of the object as its outermost contour, and the parameter amount is very small, which is close to the rectangular box representation. First, connect the midpoints of the four sides of the recognized target terrain rectangle to obtain a rhombus, input the four vertices of the rhombus into the Deep Snake network, and predict the top, bottom, left, and bottom points of the target terrain in the image. The Offsets of the four rightmost limit pixels, the rhombus can be transformed according to the Offsets to obtain the coordinates of these four limit pixels; secondly, a line segment is extended on each point, and then each line segment is connected to obtain an irregular eight The polygon, this octagon is used as the initial outline; later, considering that the outline of the object can be regarded as a Cycle Graph, the number of adjacent nodes of each node is fixed at 2, and the adjacent order is fixed, so that the convolution kernel can be defined to use the volume Contour iterative optimization problem of product processing, so the deep convolutional network based on circular convolution is used to fuse and predict the features of the current contour point, and the Offsets pointing to the contour of the object are obtained by mapping, and the coordinates of the contour point are deformed by using Offsets; finally, the initial The contour is the starting point, and the above process is repeated until the calculated contour fits the actual edge of the target object well. At this time, the predicted Offsets is less than the threshold, and the result consisting of the terrain category label and the coordinate set representing the contour point can be output.

In addition, the foreground occlusion phenomenon is very common for objects whose terrain belongs to the background category. To solve this problem, the group detection method can be used to divide the discontinuous parts of the terrain truncated by foreground objects into different target detection rectangles, and then in each Contours are extracted respectively in the rectangular frame.

The feature matching module and the position and size calculation module use the Speeded Up Robust Features method to perform feature extraction and feature matching on the pixels within the range of the same type of terrain contour collected by the binocular camera and processed by the instance segmentation module at the same time. The parallax of the same feature point is combined with the parameters of the binocular camera to calculate the depth information of the feature point, and then calculate its three-dimensional coordinates. The principle of binocular vision positioning is shown in Figure 5. For the recognized grassland, sandy land, asphalt ground, etc., its position and length and width range can be calculated according to the coordinates of the feature points located on the edge of the contour. High size and location information.

Those skilled in the art know that, in addition to realizing the system provided by the present invention and its various devices, modules, and units in a purely computer-readable program code mode, the system provided by the present invention and its various devices can be completely programmed by logically programming the method steps. , modules, and units implement the same functions in the form of logic gates, switches, ASICs, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by the present invention can be regarded as a hardware component, and the devices, modules, and units included in it for realizing various functions can also be regarded as hardware components. The structure; the devices, modules, and units for realizing various functions can also be regarded as not only the software modules for realizing the method, but also the structures in the hardware components.

In the description of this application, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", The orientation or positional relationship indicated by "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the application and simplifying the description, rather than indicating or implying the referred device Or elements must have a certain orientation, be constructed and operate in a certain orientation, and thus should not be construed as limiting the application.

Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. In the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims (6)

1. A real-time instance segmentation based lower limb prosthesis terrain identification and motion prediction system, characterized by: the local end comprises a lower limb prosthesis, a lower limb prosthesis controller, a binocular camera and an image acquisition and information transmission hardware platform, and the cloud end comprises a cloud server;

the binocular camera is installed on the lower limb artificial limb and used for collecting topographic information around the lower limb artificial limb, the image collection and information transmission hardware platform is used for collecting information obtained by the binocular camera and uploading the information to the cloud server, the image collection and information transmission hardware platform is electrically connected with the lower limb artificial limb controller, and the lower limb artificial limb controller is used for controlling the motion mode of the lower limb artificial limb;

a target terrain monitoring module, an instance segmentation module, a feature matching module and a position size calculation module are deployed on the cloud server, and terrain information is processed by the modules in sequence and then fed back to an image acquisition and information transmission hardware platform;

the target terrain detection module preliminarily identifies different terrain information in the RGB image by using a trained and optimized YoloV4-Lite convolutional neural network based on MobileNet, marks the terrain information respectively by using Box, and outputs the terrain information to the example segmentation module;

the example segmentation module uses a trained Deep Snake convolutional neural network based on cyclic convolution to extract an initial contour of a Box obtained by target detection by using an Extreme Points method, and further uses the Deep Snake network to deform the initial contour to obtain a terrain contour accurately fitting various terrains, and the terrain contour is output to the feature matching module;

the feature matching module is used for respectively extracting Features and matching the Features of pixels, collected by a binocular camera and processed by the example segmentation module, in the same-time same-type terrain contour range by using a Speeded Up Robust Features method, calculating feature point depth information by combining binocular camera parameters according to the same feature point parallax, and further calculating three-dimensional coordinates of the pixels;

the position size calculation module calculates the position, length, width and range of the identified terrain according to the coordinates of the feature points positioned at the edges of the contour, and calculates the length, width, height and position information of the identified steps, obstacles and the like according to the coordinates of the corner points of the contour.

2. A real-time instance segmentation based lower limb prosthetic topography recognition and motion prediction system according to claim 1, wherein: the lower limb prosthesis comprises a knee joint and an ankle joint, wherein the knee joint has one degree of freedom, and the ankle joint has two active degrees of freedom of flexion and extension and external rotation/internal inversion.

3. A real-time instance segmentation based lower limb prosthetic terrain identification and motion prediction system according to claim 1, wherein: the binocular camera is arranged at the front end of the knee joint part and used for acquiring RGB images of terrain around the lower limb prosthesis, and the configuration of the binocular camera is as follows: 1280 × 720 resolution, 30 frames/second sampling rate.

4. A real-time instance segmentation based lower limb prosthetic topography recognition and motion prediction system according to claim 1, wherein: the image acquisition and information transmission hardware platform is located in a gap in the lower limb artificial limb structure body, receives RGB images acquired by the binocular camera in real time, transmits the RGB images to the cloud server through a wireless network, receives terrain information fed back by the cloud server in real time, and outputs the terrain information to the lower limb artificial limb motion controller.

5. A real-time instance segmentation based lower limb prosthetic terrain identification and motion prediction system according to claim 1, wherein: the cloud server is provided with a CPU and a GPU, the CPU has the characteristic of multithreading processing and provides service for multiple users at the same time; the GPU accelerates the RGB image processing speed.

6. A real-time instance segmentation based lower limb prosthetic terrain identification and motion prediction system according to claim 1, wherein: the lower limb prosthesis controller controls the lower limb prosthesis to move comprises the following steps:

step S1: acquiring motion data of the lower limbs of the healthy person in different environments based on the IMU, and storing the motion data in a lower limb prosthesis controller;

step S2: the lower limb prosthesis controller controls and receives the current topographic data of the lower limb prosthesis sent by the cloud server;

and step S3: the lower limb prosthesis controller controls the lower limb prosthesis to adopt a motion state suitable for the terrain according to the motion data of the lower limbs of the human body;

and step S4: for terrain or obstacles that are still at a distance, the lower limb prosthesis controller predicts whether a passing and predicted arrival time is likely based on the current direction and speed of motion to achieve a smoother state switch from software.

Images