CN113742992A - Master-slave control method based on deep learning and application - Google Patents

Abstract

According to the problems existing in the existing solutions, the present invention proposes a master-slave control method and application based on deep learning. After establishing the motion coordinate system of the execution unit, a master-slave mapping relationship is established, and the master-slave mapping relationship is used to describe the Perform the kinematic forward and inverse operations of the master-slave hand corresponding to the execution unit to construct a master-slave heterogeneous model; perform real-time tracking of the target under the master-slave relationship to obtain the tracking result data, and finally perform deep learning through the convolutional neural network to complete all the steps. Describe the control process of the execution unit. The invention can facilitate the operator in the master-slave control mode according to the assistance of the target tracking algorithm, improve the manipulation accuracy of the flexible robot hand, facilitate the operation in the complex and narrow operation space, thus reduce the operation risk and relieve the pressure of the doctor.

Description

Master-slave control method and application based on deep learning

technical field

The invention relates to the technical field of deep learning, in particular to a target tracking algorithm (GOTURN) and its application.

Background technique

At present, surgical robots have penetrated into all aspects of surgical planning, minimally invasive positioning, and non-invasive treatment. The use of surgical robots for surgery can greatly reduce the pain caused by excessive trauma to patients after surgery, facilitate wound recovery and reduce postoperative complications. Most developed countries in the world already have relatively mature medical remote control operating systems , These more advanced surgical robots provide intuitive visual feedback and convenient operation, which makes the development of medical surgical robots enter an epoch-making and rapid development period, but in the actual operation process, it is necessary to avoid certain important organs and tissues or Blood vessels, which have certain requirements for the flexibility of the manipulator. The Da Vinci surgical robot is an existing solution. Its main principle is to allow doctors to control the robotic arm (slave hand) to enter the human body to complete complex surgical operations by manipulating the operating table (master hand). The advantages of this existing scheme are as follows:

It perfectly combines the advantages of open surgery and minimally invasive surgery, allowing doctors to complete complex and large-scale operations.

The flexible and fast surgical robotic arm can respond quickly according to the operation of the doctor, helping the doctor to easily complete various complex surgical actions.

The high-precision and stable surgical robotic arm can ensure the accuracy and safety of the operation, and the corresponding algorithm can also eliminate the shaking of the doctor's hand and prevent misoperation. High-performance surgical robotic arms can make surgery more convenient and easier.

However, the Da Vinci surgical robot still has some shortcomings that cannot be ignored. When it is necessary to avoid organs or perform surgery in a small space (such as the oral cavity), the manipulator still has the problem of insufficient distal accuracy.

SUMMARY OF THE INVENTION

According to the problems existing in the existing solutions, the present invention proposes a master-slave control method and application based on deep learning. The GOTURN algorithm based on offline learning is used to assist the surgical robot, so that the operator can track the algorithm according to the target in the master-slave control mode. It can accurately manipulate the flexible robotic hand to perform surgery in complex and narrow surgical spaces, thereby reducing the risk of surgery and relieving the pressure of doctors.

The technical scheme of the present invention is:

A master-slave control method based on deep learning, including:

Establish the motion coordinate system of the execution unit;

Establish a master-slave mapping relationship, describe the kinematic forward and inverse operations of the master-slave hand corresponding to the execution unit through the master-slave mapping relationship, and build a master-slave heterogeneous model;

Real-time tracking of the target is carried out under the master-slave relationship, and the tracking result data is obtained;

Deep learning is performed through a convolutional neural network to complete the control process of the execution unit.

The "establish a master-slave mapping relationship, describe the kinematic forward and inverse operations of the master-slave hand corresponding to the execution unit through the master-slave mapping relationship, and construct a master-slave heterogeneous model", including:

A mapping relationship is established from the Cartesian space coordinate system to describe the forward and inverse kinematics operations of the master and slave hands, and the inverse Jacobian matrix is used to solve the problem;

The speed of the end of the main hand: ΔX=J(θ)·Δθ;

Speed from the end of the hand: Δθ=J(θ) ^-1 ΔX;

为关节角速度矢量，J(θ)∈R^6×6为雅克比矩阵，

为末端速度矢量。in,

is the joint angular velocity vector, J(θ)∈R ^6×6 is the Jacobian matrix,

is the terminal velocity vector.

The "real-time tracking of the target under the master-slave relationship to obtain tracking result data" includes:

The steps of real-time tracking of targets using the GOTURN algorithm.

The "Deep Learning through Convolutional Neural Networks" includes:

The convolutional layers are pre-trained on ImageNet, the network is trained with a learning rate of 1e-5, and the other hyperparameters are taken from the default values of CaffeNet;

Each training example is alternately taken from the training set and video cropped using the GOTURN algorithm.

The "establishing the motion coordinate system of the execution unit" and the "establishing a master-slave mapping relationship" describe the kinematic forward and inverse operations of the master-slave hand corresponding to the execution unit through the master-slave mapping relationship to construct a master-slave heterogeneous type. model”, including: Flexibility test steps:

At the same time, one end of the execution unit is driven to perform a bending motion to perform a flexibility test, and the connection end is defined as the distal end;

Determine r, L, θ, δ, d as the distance between the line and the central axis, the length of the flexible joint, the rotation angle in the y direction, the rotation angle in the z direction, and the position of the distal end in the plane x _b Oy _b projection;

The position projection can be calculated by:

Among them, d _x , dy represent the position projections on the x and _y axes, respectively, d _x , _dy can be measured during the operation, and the posture formula of the distal end of the flexible joint is:

Attach the jaws to the distal end, the coordinates of the jaw tips are:

Among them, s represents the connection length between the rigid rod and the machining spring, and Lg represents the length of the clamping jaw; when s=0, the execution unit is regarded as a bendable joint.

After the "flexibility test steps", it includes: performance test steps:

A drive device for providing rotational power is configured and connected with the execution unit;

Make sure that the execution unit is in a bent state, and at the same time load a tensile force on the distal end of the gripper, and perform a performance test on the execution unit.

The "real-time tracking of the target under the master-slave relationship to obtain tracking result data" includes: smoothing steps of the tracking effect:

First model the center of the bounding box in the current frame (c' _x , c' _y ) relative to the center of the bounding box in the previous frame (c _x , c _y ):

c' _x =c _x +w·Δx

c' _y = _cy +h·Δy

where w and h are the width and height of the bounding box of the previous frame, respectively; Δx and y are random variables;

Again, the size change is simulated by:

w'= _w ·γw

h'= _h ·γh

where w' and h' are the current width and height of the bounding box, w and _h are the previous width and height of the bounding box, and γw and _γh are random variables;

In the training set, use _γw and _γh with a mean of 1 to model the Laplacian distribution, and augment the training set with random targets extracted from the Laplacian distribution;

The scale parameter of the Laplace distribution is:

对于边界框大小的变化：

For motion in the center of the bounding box:

For bounding box size changes:

After the "deep learning through convolutional neural network", it includes: using the PD link to eliminate the accumulated master-slave follow-up error and the steps of eliminating the doctor's hand shake;

In "Using PD Links to Eliminate Accumulated Master-Slave Following Errors":

The regulation rule is:

分别表示主、从手末端执行器位姿速度，X_m,X_s分别表示主、从手末端执行器位姿，同时k_p和k_d分别代表比例参数和微分参数；in

respectively represent the pose and velocity of the master and slave hand end effectors, X _m , X _s represent the master and slave hand end effector poses respectively, while k _p and k _d represent proportional parameters and differential parameters respectively;

In "Remove Doctor's Hand Shake", two sliding mean filters will be applied to the master hand and the slave hand respectively, as follows:

N _i is the filtering calculation result, n is the average digital filter order (i≥n), and i is the i-th sampling period.

An application of the above-mentioned master-slave control method based on deep learning in the direction of medical flexible robots.

The beneficial effects of the present invention are:

In the present invention, after establishing the motion coordinate system of the execution unit, a master-slave mapping relationship is established, and the master-slave mapping relationship is used to describe the kinematic forward and inverse operations of the master-slave hand corresponding to the execution unit, and a master-slave heterogeneous model is constructed; The real-time tracking of the target is carried out under the master-slave relationship to obtain the tracking result data, and finally deep learning is carried out through the convolutional neural network to complete the control process of the execution unit. The invention can facilitate the operator in the master-slave control mode according to the assistance of the target tracking algorithm, improve the manipulation accuracy of the flexible robot hand, facilitate the operation in the complex and narrow operation space, thus reduce the operation risk and relieve the pressure of the doctor.

At the same time, the tracking algorithm used in the present invention can be trained in an offline manner, which ensures that the user can control the network in a feed-forward manner without online fine-tuning, and the tracker can run at a speed of 100fps, thereby enabling the network to track in real time target.

Description of drawings

FIG. 1 is a schematic diagram of a coordinate system of an execution unit.

Figure 2 is a flow chart of the method of the present invention.

Detailed ways

The present application will be further described below with reference to the accompanying drawings.

The core idea of the present invention is: since the flexible surgical manipulator can use its flexibility to perform surgical operations in a narrow and complex surgical environment; therefore, the master-slave control system uses the target tracking algorithm to track the surgical area in real time and feed back depth information, It is convenient for doctors to control in real time. In actual operation, the user can control the flexible surgical manipulator through the AR image obtained by target tracking, so as to operate in the accurate surgical area (target tracking area) and successfully complete the operation.

Specific embodiment 1: In this embodiment, the execution unit may be one of a flexible robot or a flexible manipulator or a flexible mechanism such as a memory alloy-driven surgical robot or a pneumatic flexible surgical robot based on a force feedback device; preferably, The flexible manipulator can be: such as Chinese Patent: CN207055546, the structure of the minimally invasive manipulator involved in the disclosed "A Minimally Invasive Manipulator Structure".

As shown in Figures 1 to 2, the specific steps of the deep learning-based master-slave control method of the present invention are as follows:

Step 1: Establish the motion coordinate system of the flexible manipulator. The coordinate system of the flexible joint is shown in Figure 1, where x _b , y _b , and z _b respectively represent the base coordinate system of the flexible joint, which coincides with the world coordinate system. And x _e , y _e , z _e and x _g , y _g , z _g represent the coordinate systems of the end effector and the gripper, respectively.

Step 2: According to the coordinate system of step 1, in order to jointly drive the execution unit, the four steel wires are distributed on the outside of the processed spring, and the other end (referred to as the distal end) of the flexible joint is driven to bend by pulling the four steel wires together. sports. Determine r, L, θ, δ, d as the distance between the line and the central axis, the length of the flexible joint, the rotation angle in the y direction, the rotation angle in the z direction, and the projection of the position of the distal end in the plane x _b Oy _b . Therefore, the position projection can be calculated by the following formula (1-1):

Among them, d _x , dy represent the position projections on the x and _y axes, respectively, d _x , _dy can be measured during the operation, and the posture formula (1-2) of the distal end of the flexible joint is:

Connect the jaws to the distal end of the flexible joint, and the coordinates of the jaw tips are obtained from formula (1-3).

where s is the length of the connection between the rigid rod and the machined spring, and Lg is the length of the jaws. The flexible bendable joint ensures that the gripper can achieve a full range of bending motion. The length of the bendable segment depends on the connection distance between the processing spring and the rigid rod. The bending curvature of the flexible joint increases gradually with the increase of the connection distance. When s = When 0, the entire flexible joint is considered as a bendable joint, and conversely, when s = 16 mm, the entire flexible joint will be considered as a rigid body.

Step 3: Verify the performance of the manipulator through experiments. Four harmonic servo motors (RSF-100-E050-C) are configured to provide rotational power. In the tensile experiments, the flexible joint was controlled into a bent state while a tensile force was applied to the distal end of the gripper; the flexible joint consisted of a machined spring, an elastic backbone, and a rigid base rod. Therefore, when the coupling distance between the machined spring and the rigid rod is zero, the flexible joint will be the weakest rigid state. In this experiment, the flexible joint was bent to 60° along the x-axis and y-axis, respectively, and 3.5N, 5N tensile forces were applied at the distal end of the surgical manipulator, with 5 trajectories for each tensile force. The obtained test data are as follows (1-1):

Table (1-1) Tensile experiment of surgical manipulator

According to the data in Table (1-1), when the tensile force is 3.5N, the surgical manipulator can achieve good rigidity retention; when the flexible joint is bent 60° along the y-axis under a load of 5N, one trajectory fails; when the flexible joint is bent 60° along the y-axis; Both tests failed when bent at a coupling angle of 60° along the x- and y-axes under a 5N load. Since the geometry of the gripper is flat on the upper/lower side and curved on the left/right side, the surgical manipulator maintains a stable load capacity when attached to the upper/lower side (bending along the x-axis), and the left/right side is curved. / The load on the right side will slide due to the shape of the connecting plane. During the experiments, the flexible joints did not produce significant shape deformation on all paths.

Step 4: Establish a master-slave mapping relationship. The robot of the present invention is a master-slave heterogeneous type, and a mapping relationship needs to be established from the Cartesian space coordinate system to describe the kinematic forward and inverse operations of the master-slave hand. The key point of the master-slave control based on the Cartesian space coordinate system is To solve the inverse kinematics, we will use the inverse Jacobian matrix to solve it.

The mapping of joint space velocity end to Cartesian space velocity is (Equation 1-4):

为关节角速度矢量，J(θ)∈R^6×6为雅克比矩阵，

为末端速度矢量，将机器人末端和关节角在短时间内的位移和速度分别替代瞬时末端速度和关节速度，即把式(1-4)转变为式(1-5)：in,

is the joint angular velocity vector, J(θ)∈R ^6×6 is the Jacobian matrix,

is the terminal velocity vector, the displacement and velocity of the robot terminal and the joint angle in a short time are replaced by the instantaneous terminal velocity and joint velocity respectively, that is, formula (1-4) is transformed into formula (1-5):

ΔX=J(θ)·Δθ Equation (1-5)

In the same way, the inverse kinematics equation (1-6) can be obtained from equation (1-5):

Δθ=J(θ) ^-1 ·ΔX Equation (1-6)

The velocity of the end of the master hand can be obtained from Equation (1-5). After passing through the master-slave mapping relationship, the velocity of the end of the slave hand can be obtained, and the joint velocity can be obtained according to Equation (1-6).

Step 5: Use the GOTURN algorithm to track the target in real time under the master-slave relationship. According to the master-slave mapping relationship obtained in step 4, it is assumed that in the previous frame, the tracker predicted that the target is located in the target frame centered on c=(c _x , c _y ), with a width of w and a height of h. At the current moment, a new target frame is intercepted according to c obtained in the previous frame and taken as the center, with a width of k _1w and a height of k _1h . This cropping method allows the network to identify which object is being tracked in the image, and how the motion state of the target object changes by comparing the width and height of the frames before and after. Then, in the current frame, a region is cropped with c'=(c' _x , c' _y ) as the center, where c' is the current position of the target object. The cropping width and height of the current frame are respectively k _2w and k _2h , where w and h are the width and height of the target box in the previous frame, and k ₂ defines the search range of the target object. For fast-moving objects, the size of the search region can be increased at the cost of increasing network complexity; for occlusion problems, it can be optimized by training.

Step 6: Smoothing of the tracking effect. First, model the center of the bounding box in the current frame (c' _x , c' _y ) relative to the center of the bounding box in the previous frame (c _x , c _y ), and obtain equations (1-7) and (1) -8):

c' _x =c _x +w·Δx Equation (1-7)

c' _y = _cy +h·Δy Formula (1-8)

where w and h are the width and height of the bounding box of the previous frame, respectively. Δx and y are random variables that capture changes in the position of the bounding box relative to its size. In the training set, the position change of the target can be modeled by a Laplacian distribution with both Δx and Δy having a mean of 0, which has a higher probability for smaller movements than for larger movements.

Likewise, dimensional changes are simulated by the following equations (1-9) and (1-10):

w'= _w ·γw Formula (1-9)

h'=h·γ _h formula (1-10)

对于边界框中心的运动；和

对于边界框大小的变化。通过约束随机裁剪，使得它必须在每个维度中包含至少一半的目标对象。而且还得限制大小变化，如γ_w,γ_h∈(0.6,1.4)，以避免网络过度地拉伸或收缩边界框。where w' and h' are the current width and height of the bounding box, _w and h are the previous width and height of the bounding box, and γw and _γh are random variables used to record the size changes of the bounding box. In the training set, modeling the Laplacian distribution with _γw and _γh with a mean of 1 makes it possible to have a higher probability of having similar bounding boxes in the two frames before and after. In order to teach the network to prefer small movements over large ones, augmenting the training set with random targets drawn from the Laplacian distribution described above can be achieved. Because these training samples are sampled from a Laplacian distribution, motions of small magnitude will be sampled more than motions of large magnitude, so the network will learn to favor small motions more, all else being equal rather than large movements. Experiments show that the Laplacian cropping procedure improves tracker performance compared to the standard uniform cropping procedure used in classification tasks. Also note that the scale parameter of the Laplace distribution is obtained through cross-validation as

for the motion of the center of the bounding box; and

For bounding box size changes. Random cropping by constraining it so that it must contain at least half of the target objects in each dimension. And it is necessary to limit the size changes, such as γ _w , γ _h ∈ (0.6, 1.4), to avoid the network stretching or shrinking the bounding box excessively.

Step 7: Deep Learning through Convolutional Neural Networks. The convolutional layers in the algorithm network need to be pre-trained on ImageNet, the network is trained with a learning rate of 1e-5, and other hyperparameters are taken from the default values of CaffeNet. To train the network, each training example is alternately taken from a training set of videos and images. When using a video training example, a video needs to be randomly selected, and a pair of consecutive frames in this video is randomly selected. Then, crop the video according to the method described in step 5. For better results, random cropping of the current frame is also required to augment the dataset with additional examples. After training on the video, randomly sample an image and repeat the process described above. Every time a video or image is sampled, new random samples are dynamically generated to create more diversity during training.

Step 8: Use the PD link to eliminate the accumulated master-slave following error. In this link, the use of proportional coefficient k _p and differential coefficient k _d for adjustment can quickly make the system reach a steady state, so that the flexible manipulator at the end of the slave hand can quickly and accurately follow the change of the pose coordinates of the end of the master hand. The regulation law is as formula (1-11):

分别表示主、从手末端执行器位姿速度，X_m,X_s分别表示主、从手末端执行器位姿，同时k_p和k_d都不应该过大，k_p过大会使系统动态性能变差，k_d过大会使系统抑制干扰能力降低。in

Respectively represent the pose and velocity of the master and slave hand end effectors, X _m , X _s represent the master and slave hand end effector poses respectively, at the same time k _p and k _d should not be too large, if k _p is too large, it will make the system dynamic performance If k _d is too large, it will reduce the ability of the system to suppress interference.

Step 9: Eliminate the influence of the doctor's hand shaking (the shaking will be reflected to the slave hand through the master-slave mapping). In the present invention, the master hand and the slave hand are respectively filtered by two sliding mean values (the sliding mean algorithm has better suppression of periodic interference) to filter out the jitter, and it is set to perform continuous sampling within a certain period of time, and then each calculation is performed once. The measurement data only needs to be sampled once, and its specific calculation is shown in formula (1-12):

In the i-th sampling cycle, each discrete point obtained by sampling is calculated by the above formula to obtain the mean sampling result before entering the next sampling; wherein, N _i is the filtering calculation result, n is the mean value digital filter order ( i≥n).

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

Claims (10)

1. The master-slave control method based on deep learning is characterized by comprising the following steps:

establishing a motion coordinate system of an execution unit;

establishing a master-slave mapping relation, describing the kinematic forward and inverse operation of a master hand and a slave hand corresponding to the execution unit through the master-slave mapping relation, and constructing a master-slave heterogeneous model;

tracking the target in real time under the master-slave relation to obtain tracking result data;

and carrying out deep learning through a convolutional neural network to complete the control process of the execution unit.

2. The deep learning-based master-slave control method according to claim 1, wherein the establishing of the master-slave mapping relationship, the describing of the kinematic forward-inverse operation of the master-slave hand corresponding to the execution unit through the master-slave mapping relationship, and the establishing of the master-slave heterogeneous model comprises:

establishing a mapping relation from a Cartesian space coordinate system to describe the kinematics positive and inverse operation of the master hand and the slave hand, and solving by using an inverse Jacobian matrix;

velocity of the end of the master hand: Δ X ═ J (θ) · Δ θ;

velocity from the end of the hand: Δ θ ═ J (θ)^-1·ΔX；

Wherein,

is the angular velocity vector of the joint, J (theta) is epsilon R^6×6The matrix of the Jacobian is obtained,

is the terminal velocity vector.

3. The master-slave control method based on deep learning of claim 1, wherein the "real-time tracking of the target under master-slave relationship to obtain tracking result data" comprises:

and utilizing a GOTURN algorithm to perform real-time tracking of the target.

4. The master-slave control method based on deep learning of claim 1, wherein the deep learning through convolutional neural network comprises:

pre-training the convolutional layer on ImageNet, training the network by using the learning rate of 1e-5, and taking other hyper-parameters from the default value of CaffeNet;

each training example is alternately taken from the training set and video cropping is performed using the GOTURN algorithm.

5. The deep learning based master-slave control method according to claim 4, comprising:

random cropping of the current frame is also required, with additional examples to augment the data set.

6. The deep learning-based master-slave control method according to claim 1, wherein the establishing of the motion coordinate system of the execution unit and the establishing of the master-slave mapping relationship describe the kinematics positive and inverse operation of the master and slave hands corresponding to the execution unit through the master-slave mapping relationship, and the establishing of the master-slave heterogeneous model comprises: a flexibility testing step:

simultaneously driving one end of the execution unit to perform bending motion to perform flexibility test, wherein the connecting end is defined as a far end;

determining r, L, theta, delta, d as the distance of the line from the central axis, the length of the flexible joint, the rotation angle in the y-direction, the rotation angle in the z-direction, respectivelyAnd distal end in plane x_bOy_bPosition projection of (1);

the position projection can be calculated by:

wherein d is_x,d_yRepresenting the projection of the position on the x, y axis, respectively, d_x,d_yCan measure in the operation process, the posture formula of the far end of the flexible joint is as follows:

attaching a jaw to the distal end, the jaw tip having coordinates of:

wherein s represents the length of the connection between the rigid rod and the machining spring, and Lg represents the length of the clamping jaw; when s is 0, the execution unit is considered as a bendable joint.

7. The master-slave control method based on deep learning of claim 6, wherein after the "flexible testing step", the method comprises: and (3) performance testing:

the driving device is configured to provide rotary power and is connected with the execution unit;

and ensuring that the execution unit is in a bending state, and loading tensile force at the far end of the clamp holder to perform performance test on the execution unit.

8. The deep learning-based master-slave control method according to claim 1, wherein the step of performing real-time tracking of the target in a master-slave relationship to obtain tracking result data comprises: smoothing of tracking effect:

first, the current frame (c'_x,c'_y) Center of middle bounding box relative to previous frame (c)_x,c_y) The center of the middle bounding box is modeled:

c'_x＝c_x+w·Δx

c'_y＝c_y+h·Δy

where w and h are the width and height, respectively, of the bounding box of the previous frame; Δ x and y are random variables;

also, the dimensional change was simulated by:

w′＝w·γ_w

h'＝h·γ_h

where w 'and h' are the current width and height of the bounding box, w and h are the previous width and height of the bounding box, γ_wAnd gamma_hA random variable;

in the training set, γ with an average value of 1 is used_wAnd gamma_hPerforming Laplace distribution modeling, and expanding a training set by using a random target extracted from the Laplace distribution;

the proportion parameters of the Laplace distribution are as follows:

for motion of the bounding box center:

for variations in bounding box size:

9. the master-slave control method based on deep learning of claim 1, wherein after deep learning by convolutional neural network, the method comprises: eliminating the master-slave following error accumulated continuously and eliminating the hand shake of the doctor by utilizing a PD link;

in the process of eliminating the continuously accumulated master-slave following errors by utilizing the PD link, the method comprises the following steps:

the regulation and control rule is as follows:

wherein

Respectively represents the pose speed, X, of the master and slave hand-end actuators_m,X_sRespectively representing the poses of the master hand end effector and the slave hand end effector, and k is the same time_pAnd k_dRespectively representing a proportional parameter and an integral parameter;

in the process of eliminating hand shake of doctors, two times of sliding mean filtering are respectively adopted for the master hand and the slave hand, specifically:

N_ifor the filtering calculation result, n is the order of the mean digital filter (i is more than or equal to n), and i is the ith sampling period.

10. Use of the master-slave control method based on deep learning according to claim 1 in the direction of a medical flexible robot.

Images