CN116460860B - A model-based offline reinforcement learning control method for robots - Google Patents

Abstract

The invention discloses a model-based robot offline reinforcement learning control method, and belongs to the field of robot control. The method comprises the following steps: deep learning respectively establishes a depth kinematic model based on jacobian and a depth dynamics model based on Lagrange, which are corresponding to the mechanical arm of the robot, a depth transfer model is established by the depth kinematic model based on jacobian and the depth dynamics model based on Lagrange, the depth transfer model is used for establishing a Markov decision process model of a mechanical arm track tracking task, offline reinforcement learning based on the model is carried out through a Soft Actor-Critic reinforcement learning algorithm to obtain a control strategy, and the mechanical arm is controlled by combining a traditional calculation moment controller. The method greatly reduces the sample complexity of the reinforcement learning control of the robot, improves the precision of the track tracking task, and has stronger generalization and robustness.

Description

A model-based offline reinforcement learning control method for robots

Technical Field

The present invention relates to the field of high-precision trajectory tracking tasks of robots, and in particular to a model-based offline reinforcement learning control method for robots.

Background Art

Reinforcement learning algorithms provide a powerful framework to solve sequential decision-making problems, and recent deep learning techniques have accelerated the development of model-free reinforcement learning algorithms. However, these algorithms are rarely directly applied to real-world physical systems, especially robotic systems, because they have high sample complexity and the intermediate policies during training may be harmful to the robotic system and the environment.

In contrast, the model-based reinforcement learning algorithm simulates and plans trajectories by learning the state transition model of the system and environment, thus reducing the sample complexity of the algorithm. For the robot's trajectory tracking control task, traditional model-based dynamic control methods, such as augmented PD and calculated torque control, have the advantages of higher tracking accuracy and lower control energy consumption. The existing model-based robot control method requires accurate acquisition of the robot's kinematic model and dynamic model first, and the controller parameters need to be manually adjusted based on experience. However, as robots become more and more complex, how to obtain accurate kinematic models and dynamic models corresponding to the robot and automatically obtain the controller parameters to achieve high-precision robot control is a problem that needs to be solved.

In view of this, the present invention is proposed.

Summary of the invention

The purpose of the present invention is to provide a model-based offline reinforcement learning control method for a robot, which can obtain the corresponding kinematic model and dynamic model of the robot through deep learning and automatically obtain the parameters of the controller, and combine it with a traditional computational torque controller to achieve high-precision trajectory tracking tasks of the robot in the joint space and operation space, and well solve the above-mentioned technical problems existing in the prior art.

The objective of the present invention is achieved through the following technical solutions:

A model-based offline reinforcement learning control method for a robot, characterized in that it is used to control a mechanical arm as a robot, comprising the following steps:

Step S1, establishing a Jacobian-based deep kinematic model and a Lagrangian-based deep dynamic model corresponding to the robotic arm through deep learning; wherein the deep kinematic model is used to predict the position and posture of the end effector of the robotic arm and calculate the Jacobian matrix corresponding to the robotic arm; the deep dynamic model is used to predict the joint angle, angular velocity and angular acceleration of the robotic arm to obtain the state change of the joint space of the robotic arm, and obtain the control torque of the torque controller when the robotic arm performs a trajectory tracking task;

Step S2, establishing a random excitation trajectory model described by a finite Fourier series, controlling the robotic arm with the random excitation trajectory given by the random excitation trajectory model as the expected motion trajectory, measuring and collecting the actual motion trajectory of the robotic arm as a training data set, and respectively training the Jacobian-based deep kinematic model and the Lagrangian-based deep dynamics model established in step S1;

Step S3, establishing a Markov decision process model for the robot arm trajectory tracking task, wherein the state transition model in the Markov decision process model is a deep transfer model for simulating the robot arm motion trajectory constructed by combining the trained Jacobi-based deep kinematic model and the Lagrangian-based deep dynamics model;

Step S4, according to the Markov decision process model, the control parameters of the torque controller are calculated offline through the Soft Actor-Critic reinforcement learning algorithm as the control strategy, the simulated motion trajectory data of the manipulator in which the deep transfer model interacts with the control strategy offline is collected, and the actor network and the critic network of the Soft Actor-Critic reinforcement learning algorithm are updated until the optimal control strategy is obtained;

Step S5, the calculation torque controller calculates the specific control torque of the robot arm according to the optimal control strategy obtained in step S4 to control the robot arm.

Compared with the prior art, the model-based robot offline reinforcement learning control method provided by the present invention has the following beneficial effects:

Through deep learning, a deep kinematic model and a deep dynamic model corresponding to the robot arm are established. With the offline learning of the SoftActor-Critic reinforcement learning algorithm, the model is combined with the traditional computational torque controller to control the robot arm to perform high-precision trajectory tracking tasks in the joint space and operation space. This method greatly reduces the sample complexity of robot reinforcement learning control, improves the accuracy of trajectory tracking tasks, and has strong generalization and robustness.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

FIG1 is a schematic flow chart of a model-based offline reinforcement learning control method for a robot according to an embodiment of the present invention.

FIG2 is a specific flow chart of a model-based robot offline reinforcement learning control method provided in an embodiment of the present invention.

DETAILED DESCRIPTION

The following is a clear and complete description of the technical solutions in the embodiments of the present invention in combination with the specific content of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

First, the terms that may be used in this article are explained as follows:

The term “and/or” means that either or both of them can be realized at the same time. For example, X and/or Y means both “X” or “Y” and “X and Y”.

The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

Unless otherwise specified or limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this article can be understood according to specific circumstances.

The orientation or position relationship indicated by terms such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc. are based on the orientation or position relationship shown in the drawings and are only for the convenience and simplification of description, and do not explicitly or implicitly indicate that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation of this document.

The model-based robot offline reinforcement learning control method provided by the present invention is described in detail. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If the specific conditions are not specified in the embodiments of the present invention, they are carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer. The reagents or instruments used in the embodiments of the present invention that do not indicate the manufacturer are all conventional products that can be purchased commercially.

As shown in FIG. 1 and FIG. 2 , an embodiment of the present invention provides a model-based offline reinforcement learning control method for a robot, which is used to control a robot arm as a robot, and includes the following steps:

Step S1, establishing a Jacobian-based deep kinematic model and a Lagrangian-based deep dynamic model corresponding to the robotic arm through deep learning; wherein the deep kinematic model is used to predict the position and posture of the end effector of the robotic arm and calculate the Jacobian matrix corresponding to the robotic arm; the deep dynamic model is used to predict the joint angle, angular velocity and angular acceleration of the robotic arm to obtain the state change of the joint space of the robotic arm, and obtain the control torque of the torque controller when the robotic arm performs a trajectory tracking task;

Step S2, establishing a random excitation trajectory model described by a finite Fourier series, controlling the robotic arm with the random excitation trajectory given by the random excitation trajectory model as the expected motion trajectory, measuring and collecting the actual motion trajectory of the robotic arm as a training data set, and respectively training the Jacobian-based deep kinematic model and the Lagrangian-based deep dynamics model established in step S1;

Step S3, establishing a Markov decision process model for the robot arm trajectory tracking task, wherein the state transition model in the Markov decision process model is a deep transfer model for simulating the robot arm motion trajectory constructed by combining the trained Jacobi-based deep kinematic model and the Lagrangian-based deep dynamics model;

Step S4, according to the Markov decision process model, the control parameters of the torque controller are calculated offline through the Soft Actor-Critic reinforcement learning algorithm as the control strategy, the simulated motion trajectory data of the manipulator in which the deep transfer model interacts with the control strategy offline is collected, and the actor network and the critic network of the Soft Actor-Critic reinforcement learning algorithm are updated until the optimal control strategy is obtained;

Step S5, the calculation torque controller calculates the specific control torque of the robot arm according to the optimal control strategy obtained in step S4 to control the robot arm.

Preferably, in step S1 of the above method, a Jacobian-based deep kinematic model corresponding to the robotic arm is established through deep learning in the following manner, including:

The forward kinematics model of the robotic arm is determined as: ;in, is the position of the end effector of the robot arm in the operating space, is the degree of freedom of the robot's operating space; is the joint angle of the robot arm, is the degree of freedom of the robot joint space;

The velocity relationship between the joints of the robot and the end effector of the robot is expressed by the differential of the kinematic equation f with respect to time: ;in, It is the derivative of the position of the end effector of the robot arm in the operating space with respect to time; is the Jacobian matrix actually corresponding to the robotic arm, is the degree of freedom of the robot's operating space, is the degree of freedom of the robot joint space; is the angular velocity of the robot joint;

Determine the position and posture of the end effector of the robotic arm in the operating space as , where p is the position of the end effector of the robot arm; is the quaternion representing the posture of the end effector of the robot arm, ,in represents the rotation axis of the quaternion, , and are the three components of the rotation axis, represents the angle of rotation around the axis of rotation, Satisfy constraints ; Quaternion The relationship between the time derivative and the angular velocity ω of the end effector of the manipulator in the operating space is: , that is, the relationship is:

;

In the above relationship, , and are the components of the angular velocity of the end effector of the robot arm in the x-axis, y-axis and z-axis respectively;

A fully connected deep neural network is established to learn the forward kinematics model of the robotic arm, and the corresponding deep kinematics model of the robotic arm is obtained as follows: ,in, are the network parameters of the deep kinematics model; is the kinematic equation derived from deep learning; q is the joint angle of the robot arm;

Applying the chain rule to the derivatives of each layer in the deep kinematic model, iteratively calculating the derivatives of the output of the deep neural network with respect to the input to recover and learn the Jacobian matrix, the Jacobian matrix After the following calculation, it is expressed as:

;

in, are the network parameters of the deep kinematics model; The partial derivatives of the kinematic equations derived from deep learning for the joint angles; is the i-th network layer of the deep kinematics model, , ; , , The weights of the deep kinematics model applied to the input of the previous layer of the deep neural network; for deviation; and They are the nonlinear activation function and its derivative of the deep neural network respectively.

Preferably, in the above method, the training data set of the deep kinematic model , where q is the joint angle of the robot arm; is the joint angular velocity of the robot arm, x is the position of the end effector of the robot arm in the operation space; is the derivative of the position of the end effector of the robot arm in the operating space with respect to time;

The loss function during the training of the deep kinematics model is:

;

in, are the network parameters of the deep kinematics model; It is a training dataset for deep kinematics models; is the size of the training dataset; is the generalized inverse of the Jacobian matrix; is the size of the training dataset; the first loss term is the mean square error between the actual position and predicted position of the robot in the operating space; the second loss term is the mean square error between the actual speed of the robot's operating space and the predicted speed; the last loss term It is the Jacobian matrix that makes the deep neural network more accurate by fitting the angular velocity of the robot arm joints.

Preferably, in step S1 of the above method, a Lagrangian-based deep dynamics model corresponding to the robotic arm is established through deep learning in the following manner, including:

The forward dynamics model and the inverse dynamics model of the robot arm are determined as follows:

;

;

in, They are the joint angle, joint angular velocity and joint angular acceleration of the robot arm; are the forces and moments acting on the joints of the robot arm;

The generalized coordinates of the robot are selected as the joint angles of the robot, and the Lagrangian function is defined as ,in, is the kinetic energy of the robotic arm; is the potential energy of the robot arm; is the mass matrix of the robot; the superscript T indicates the transposed matrix;

To ensure the mass matrix of the robot arm The positive definite symmetry of the mass matrix Perform Chelesky decomposition to obtain ,in, is a non-negative diagonal lower triangular matrix, the superscript T represents the transposed matrix, and the non-negative diagonal lower triangular matrix is learned using a deep neural network and the potential energy of the robotic arm To fit the Lagrangian function , combined with the Euler-Lagrange equation , where F is the generalized force and torque, and the deep forward dynamics model and deep inverse dynamics model that constitute the deep dynamics model corresponding to the robotic arm are obtained as follows:

;

;

in, , and They are the joint angle, joint angular velocity and joint angular acceleration of the robot arm respectively; is the mass matrix of the robot; is the derivative of the mass matrix of the robot with respect to time; are the Coriolis force and the centripetal force; is a conservative force including gravity and spring force; Output torque for the joints of the robot arm; is the joint friction of the robot arm, which is obtained by introducing a priori friction model of the robot arm composed of Coulomb friction, viscous friction and Stribeck friction. The priori friction model of the robot arm is:

;

in, is the Coulomb friction force; is the viscous friction coefficient; is the maximum static friction; υ and δ are the correlation coefficients of the Stribeck friction model respectively; is the angular velocity of the i-th joint of the robot.

Preferably, in the above method, the training data set of the deep dynamics model is , where q is the joint angle of the robot arm; is the joint angular velocity of the robot arm; is the joint angular acceleration of the robot arm; is the joint torque of the robot arm;

The loss function of the deep dynamics model is:

;

in, It is a training dataset for deep dynamics models; is the size of the training dataset for the deep dynamics model; is the angular acceleration of the robot joint Regression loss, , are the predicted value and actual value of the joint angular acceleration of the robot arm at time t; is the joint torque Regression loss, , are the predicted value and actual value of the joint torque of the robot arm at time t respectively; is the multi-step prediction loss obtained by numerical integration, is the total number of steps predicted, , are the actual values of the joint angle and joint angular velocity of the robot arm at time t+1, and They are the predicted values of the robot arm joint angle and joint angular velocity at time t+1 respectively.

Preferably, in step S2 of the above method, a random excitation trajectory model described by a finite Fourier series is established in the following manner, including:

For joint i of the robot at time t, the random excitation trajectory model Defined as:

;

in, and are the amplitude of cosine and the amplitude of sine respectively; is the frequency of the cosine and the frequency of the sine; is the offset of the joint angle of the robot arm. The amplitude of the sine, the amplitude and frequency of the cosine and the offset of the joint angle are randomly selected to ensure that the joint angle, joint angular velocity and joint angular acceleration of the robot arm are within a safe range. is the number of Fourier series, randomly selected from 1 to 3, is the variable during accumulation, and t is the time value at time t.

Preferably, in step S3 of the above method, a Markov decision process model of the robot arm trajectory tracking task is established in the following manner, wherein the state transition model of the Markov decision process model is a deep transition model for simulating the robot arm motion trajectory constructed by combining a trained Jacobi-based deep kinematic model and a Lagrangian-based deep dynamics model, including:

Modeling the robot's trajectory tracking task as a finite-time discounted discrete Markov decision process ;in, is the state space, is the state at time t; is the action space; is the action at time t; It is a deep transition model as a state transition model; is the reward function; is the discount factor;

A deep transfer model simulating the motion trajectory of the robotic arm is constructed by combining the trained Jacobian-based deep kinematic model and the Lagrangian-based deep dynamics model in the following manner, including:

According to the joint angle and joint angular velocity of the robot at time t Joint torque with the robot arm The predicted values of the joint angle and joint angular velocity of the manipulator at time t+1 are obtained by using the fourth-order Runge-Kutta numerical integration method combined with the Lagrangian-based deep dynamics model. ;

The Jacobian-based deep kinematic model is used to predict the joint angles and joint angular velocities of the robot at time t+1. The predicted values of the position and velocity of the end effector of the robot arm at time t+1 are obtained as input ;

According to the predicted values of the joint angle, joint angular velocity of the robot arm and the position and velocity of the end effector at time t+1 , combined with the expected trajectory of the robot at time t+1, the error value required to complete the trajectory tracking task is calculated to form the state of the robot trajectory tracking task at time t+1 ;

Use the robot arm trajectory at time t+1 to track the status of the task Building a deep transfer model , in this deep transfer model and are the states of the robot arm trajectory tracking task at time t and time t+1, respectively. is the action output by the control strategy at time t.

Preferably, in step S4 of the above method, the control parameters of the torque controller are calculated offline by the Soft Actor-Critic reinforcement learning algorithm according to the Markov decision process model as the control strategy, the simulated motion trajectory data of the manipulator in which the deep transfer model interacts offline with the control strategy is collected, and the actor network and the critic network of the Soft Actor-Critic reinforcement learning algorithm are updated until the optimal control strategy is obtained, including:

The state space and action space of the Markov decision process model are respectively set according to the trajectory tracking task performed by the robot arm in the joint space and the trajectory tracking task performed in the operation space, and the model-based offline reinforcement learning is performed through the Soft Actor-Critic reinforcement learning algorithm, and the control parameters of the torque controller are output and calculated as the control strategy;

Collect the simulated motion trajectory data of the robot arm that interacts with the control strategy offline , the actor network and critic network of the Soft Actor-Critic reinforcement learning algorithm are updated until the optimal control strategy is obtained.

Preferably, in the above method, the state space and action space of the Markov decision process model are respectively set according to the trajectory tracking task performed by the robot arm in the joint space and the trajectory tracking task performed in the operation space in the following manner, including:

Set the state of the robot arm joint space for trajectory tracking task at time t for:

;

in, , They are the joint angle error and joint angular velocity error of the robot arm respectively; is the accumulated value of the joint angle error of the robot arm; , and are the joint angle, joint angular velocity, and joint angular acceleration of the desired trajectory of the robot arm, respectively;

The action used by the robot arm joint space to perform trajectory tracking tasks The computational torque controller is designed as follows:

;

in, and are the joint angles and joint angular velocities of the robot arm, respectively; is the mass matrix of the robot; are the Coriolis force and the centripetal force; is a conservative force including gravity and spring force; is the joint friction of the robot arm; , and They are the joint angle error, joint angular velocity error and cumulative value of joint angle error of the robot arm respectively; , They are respectively used to calculate the control parameters of the torque controller;

Set the state of the trajectory tracking task in the robot operation space at time t for:

;

in, , are the posture error and velocity error vector of the end effector of the robot arm respectively; , They are the position error and linear velocity error of the end effector of the robot arm respectively; , They are the attitude error and angular velocity error of the end effector of the robot arm respectively; is the accumulated value of the position error of the end effector of the robot arm; , and are the position, velocity and acceleration of the desired trajectory of the robot end effector respectively; and are the position and linear velocity of the end effector of the robot arm, respectively; , and are the position, linear velocity and linear acceleration of the desired trajectory of the robot end effector, respectively; is the quaternion representing the posture of the end effector of the robot arm, and are the rotation axis of the quaternion and the angle of rotation around the rotation axis respectively; is the quaternion of the desired trajectory of the robot end effector, and are the rotation axis of the quaternion and the angle of rotation around the rotation axis respectively; and are the first-order derivative and the second-order derivative of the quaternion of the desired trajectory of the robot end effector with respect to time; and The angular velocity of the end effector of the robot arm and the angular velocity of the desired trajectory respectively; the subscript t of the above parameters indicates that the parameter is the parameter at time t;

The action used by the robot to perform trajectory tracking tasks in the operating space is The acceleration-based operating space calculation torque controller is designed as follows:

;

in, and are the joint angles and joint angular velocities of the robot arm, respectively; is the mass matrix of the robot; are the Coriolis force and the centripetal force; is a conservative force including gravity and spring force; is the joint friction of the robot arm; is the reference acceleration of the manipulator in the operating space, , are the posture error and velocity error vector of the end effector of the robot arm respectively; are the network parameters of the deep kinematics model; is the generalized inverse matrix of the Jacobian matrix of the robot; , Calculate the control parameters of the torque controller and select the control torque of the null space The form is:

;

in, and are the joint angles and joint angular velocities of the robot arm, respectively; is the initial value of the robot arm joint angle; is the identity matrix; are the network parameters of the deep kinematics model; is the generalized inverse matrix of the Jacobian matrix of the robot; is the Jacobian matrix of the robot; , They are respectively used to calculate the control parameters of the torque controller;

The reward function is set as a segmented reward function, which is:

;

in, and are the weights of different control accuracies, including The term is used to quickly reduce the strategy exploration error, including The term is used to make the strategy learn actions that improve accuracy; β is the value for adjusting the proportion of different weights, and the value of β is 0.75; , and In the trajectory tracking task of the robot arm joint space, they are the joint angle error, joint angular velocity error and the cumulative value of the joint angle error of the robot arm; in the trajectory tracking task of the robot arm operation space, they are the posture error, velocity error and cumulative value of the posture error of the robot arm end effector;

The actor network and critic network of the Soft Actor-Critic reinforcement learning algorithm are updated as follows:

The Soft Actor-Critic reinforcement learning algorithm fits the state-action value function through the critic network To evaluate the strategy, update it by minimizing the error of the Bellman equation as follows: ,in, is the state of the robot trajectory tracking task at time t+1, The action output by the control strategy at time t+1; is the state-action value function at time t; To control strategy is the expected value under the condition; t represents the time t; is the discount factor, and the superscript t represents the tth power of the discount factor; is the reward function; is the expected value under the condition of the state and action at time t+1; is the state-action value function at time t+1;

The actor network of the Soft Actor-Critic reinforcement learning algorithm is updated by minimizing the KL divergence of the policy as follows:

;

in, is a sample in the distribution of control strategies; Π is the distribution of control strategies represented by the actor network; To minimize the KL divergence of the strategy; is the state of the robot trajectory tracking task at time t; is the quaternion representing the posture of the end effector of the robot arm; is the control strategy before updating; is the state-action value function before updating; is the partition function used to normalize the distribution;

The optimal control strategy derived from the Soft Actor-Critic reinforcement learning algorithm is to introduce the maximum entropy objective while maximizing the expected reward. for:

;

Where T is the length of the robot arm's motion trajectory; is the expected value under the condition of state and action at time t; is the state of the robot trajectory tracking task at time t; is the action of the control strategy output at time t; is the reward function; is the discount factor; is the entropy of the control strategy; is the regularization coefficient of entropy, which uses adaptive updating to adjust the proportion of entropy in the objective function to control the randomness of the strategy.

Preferably, in step S5 of the above method, the control parameter of the calculated torque controller output by the calculated torque controller according to the optimal control strategy obtained in step S4 is , combined with the calculated torque controller, the specific control torque of the robotic arm is calculated.

In the control method of the embodiment of the present invention, by introducing prior knowledge, the deep kinematic model and deep dynamic model obtained by deep learning are both gray box models, which improves the generalization of the model and obtains interpretable deep kinematic model and deep dynamic model, which are used to simulate the robot's motion trajectory and optimize the robot's control torque. The present invention greatly reduces the sample complexity of robot reinforcement learning control, improves the accuracy of trajectory tracking, and has strong generalization and robustness.

In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the model-based robot offline reinforcement learning control method provided by the embodiment of the present invention is described in detail with specific examples below.

Example 1

As shown in Figures 1 and 2, an embodiment of the present invention provides a model-based offline reinforcement learning control method for a robot, which establishes a deep kinematic model and a deep dynamic model corresponding to a robot arm as a robot through deep learning, and realizes the combination with a traditional computational torque controller, thereby completing the high-precision trajectory tracking task of the robot arm in the joint space and the operation space. The method includes the following steps:

First, a Jacobian-based deep kinematic model corresponding to the robotic arm is established through deep learning, including:

The forward kinematics model of the robot arm is:

;

in, is the position and posture of the end effector of the robot arm in the operating space, that is, the posture, is the joint angle of the robot arm, is the degree of freedom of the robot's operating space, is the degree of freedom of the robot joint space;

The kinematic equation f, differentiated with respect to time, describes the velocity relationship between the joints of the robot and the end effector of the robot as follows:

;

in, It is the derivative of the position of the end effector of the robot arm in the operating space with respect to time; is the Jacobian matrix actually corresponding to the robotic arm, is the degree of freedom of the robot's operating space, is the degree of freedom of the robot joint space; is the angular velocity of the robot joint. Using numerically robust quaternions To represent the posture of the end effector; the quaternion is expressed as , and satisfy the constraints ,in represents the rotation axis of the quaternion, , and are the three components of the rotation axis, represents the angle of rotation around the rotation axis. Therefore, the position of the end effector of the robot arm in the operation space can be expressed as , where p is the position of the end effector of the robot arm. Quaternion The relationship between the time derivative and the angular velocity ω of the operating space is: ,Right now:

;

In the above relationship, , and are the components of the angular velocity of the end effector of the robot arm in the x-axis, y-axis and z-axis respectively;

A fully connected deep neural network is established to learn the forward kinematics model of the robotic arm, and the corresponding deep kinematics model of the robotic arm is obtained as follows: ,in, are the network parameters of the deep kinematics model; is the kinematic equation derived from deep learning; q is the joint angle of the robot arm;

The chain rule is applied to the derivatives of each layer of the obtained deep kinematic model, and the derivatives of the output of the deep neural network with respect to the input are iteratively calculated to recover and learn the Jacobian matrix. The learned Jacobian matrix After the following calculation, it is expressed as:

;

in, are the network parameters of the deep kinematics model; The partial derivatives of the kinematic equations derived from deep learning for the joint angles; is the i-th network layer of the deep kinematics model, , ; , , The weights of the deep kinematics model applied to the input of the previous layer of the deep neural network; for deviation; and They are the nonlinear activation function and its derivative of the deep neural network respectively.

Training dataset for the above deep kinematics model , the training data set consists of the joint angle q and joint angular velocity of the robot , the position x of the end effector in the operating space and the derivative of the position with respect to time composition.

Specifically, the training data set of the above-mentioned deep kinematic model is obtained by first establishing a random excitation trajectory model described by a finite Fourier series, using the random excitation trajectory given by the random excitation trajectory model as the expected motion trajectory to control the robotic arm, and measuring and collecting the actual motion trajectory of the robotic arm.

Preferably, a random excitation trajectory model described by a finite Fourier series is established in the following manner, including:

For joint i of the robot at time t, the random excitation trajectory model Defined as:

;

in, and are the amplitude of cosine and the amplitude of sine respectively; is the frequency of the cosine and the frequency of the sine; is the offset of the joint angle of the robot arm. The amplitude of the sine, the amplitude and frequency of the cosine and the offset of the joint angle are randomly selected to ensure that the joint angle, joint angular velocity and joint angular acceleration of the robot arm are within a safe range. is the number of Fourier series, randomly selected from 1 to 3, is the variable during accumulation, and t is the time value at time t.

The loss function for training the deep kinematics model is set as:

;

in, are the network parameters of the deep kinematics model; It is a training dataset for deep kinematics models; is the size of the training dataset; is the generalized inverse of the Jacobian matrix; is the size of the training dataset; the first loss term is the mean square error between the actual position and predicted position of the robot in the operating space; the second loss term is the mean square error between the actual speed of the robot's operating space and the predicted speed; the last loss term It is the Jacobian matrix that makes the deep neural network more accurate by fitting the angular velocity of the robot arm joints.

Secondly, a deep dynamics model corresponding to the robotic arm is established through deep learning, including:

The forward dynamics model and inverse dynamics model of the robot arm are expressed as follows:

;

;

in, They are the joint angle, angular velocity and angular acceleration of the robot arm, are the forces and moments acting on the joints of the robot arm.

The dynamic model of the robotic arm is derived based on Lagrangian mechanics to establish a deep dynamic model.

The Lagrangian function is defined as ,in is the kinetic energy of the robot arm, is the potential energy of the robot arm, is the mass matrix of the robot; the superscript T indicates the transposed matrix;

To ensure the mass matrix of the robot arm The positive definite symmetry of the mass matrix Perform Chelesky decomposition to obtain ,in, is a lower triangular matrix with non-negative diagonal, and the superscript T indicates the transposed matrix;

Learning non-negative diagonal lower triangular matrices using deep neural networks and the potential energy of the robotic arm To fit the Lagrangian function , combined with the Euler-Lagrange equation , where F is the generalized force and torque, and the deep forward dynamics model and deep inverse dynamics model that constitute the deep dynamics model corresponding to the robotic arm are obtained as follows:

;

;

in, , and They are the joint angle, joint angular velocity and joint angular acceleration of the robot arm respectively; is the mass matrix of the robot; is the derivative of the mass matrix of the robot with respect to time; are the Coriolis force and the centripetal force; is a conservative force including gravity and spring force; Output torque for the joints of the robot arm; is the joint friction of the robot arm, which is obtained by introducing a priori friction model of the robot arm composed of Coulomb friction, viscous friction and Stribeck friction. The priori friction model of the robot arm is:

;

in, is the Coulomb friction force; is the viscous friction coefficient; is the maximum static friction; υ and δ are the correlation coefficients of the Stribeck friction model respectively; is the angular velocity of the i-th joint of the robot.

Training dataset for the above deep dynamics model , the training data set consists of the joint angle q and joint angular velocity of the robot , joint angular acceleration and joint torque composition.

Specifically, the training data set of the above-mentioned deep dynamics model is obtained by first establishing a random excitation trajectory model described by a finite Fourier series, controlling the robot arm using the random excitation trajectory given by the random excitation trajectory model as the expected motion trajectory, and measuring and collecting the actual motion trajectory of the robot arm. The method of establishing the random excitation trajectory model described by a finite Fourier series is the same as that mentioned in the above-mentioned deep kinematics model, and will not be repeated here.

The loss function of the deep dynamics model is set as:

;

in, It is a training dataset for deep dynamics models; is the size of the training dataset for the deep dynamics model; is the angular acceleration of the robot joint Regression loss, , are the predicted value and actual value of the joint angular acceleration of the robot arm at time t; is the joint torque Regression loss, , are the predicted value and actual value of the joint torque of the robot arm at time t respectively; is the multi-step prediction loss obtained by numerical integration. This loss term can improve the multi-step prediction accuracy of the deep dynamics model for the subsequent strategy optimization process in reinforcement learning. is the total number of steps predicted, , are the actual values of the joint angle and joint angular velocity of the robot arm at time t+1, and They are the predicted values of the robot arm joint angle and joint angular velocity at time t+1 respectively.

Then, a Markov decision process model of the robot arm trajectory tracking task is established. The state transition model in the Markov decision process model is a deep transition model for simulating the robot arm motion trajectory constructed by combining the trained Jacobi-based deep kinematic model and the Lagrangian-based deep dynamics model, including:

The robot trajectory tracking task is modeled as a finite-time discounted discrete Markov decision process. ;in, is the state space, is the action space, is the state at time t, is the action at time t, It is a deep transition model as a state transition model. is the reward function, is the discount factor;

A deep transfer model simulating the motion trajectory of the robotic arm is constructed by combining the trained Jacobi-based deep kinematic model and the Lagrangian-based deep dynamics model as follows, including:

According to the joint angle and joint angular velocity of the robot at time t Joint torque with the robot arm The predicted values of the joint angle and joint angular velocity of the manipulator at time t+1 are obtained by using the fourth-order Runge-Kutta numerical integration method combined with the Lagrangian-based deep dynamics model. ;

The Jacobian-based deep kinematic model is used to predict the joint angles and joint angular velocities of the robot at time t+1. The predicted values of the position and velocity of the end effector of the robot arm at time t+1 are obtained as input ;

According to the predicted values of the joint angle, joint angular velocity of the robot arm and the position and velocity of the end effector at time t+1 , combined with the expected trajectory of the robot at time t+1, the error value required to complete the trajectory tracking task is calculated to form the state of the robot trajectory tracking task at time t+1 ;

Use the robot arm trajectory at time t+1 to track the status of the task Building a deep transfer model , in this deep transfer model and are the states of the robot arm trajectory tracking task at time t and time t+1, respectively. is the action output by the control strategy at time t.

The above-mentioned deep transfer model is used to simulate the motion trajectory of the robotic arm and provide offline interaction data for the model-based reinforcement learning method to optimize the control strategy.

Finally, according to the Markov decision process model of the robot arm trajectory tracking task, the model-based offline reinforcement learning method includes:

The state space, action space and reward function of the Markov decision process model are set respectively, and the Soft Actor-Critic (SAC) reinforcement learning method is combined to implement a model-based offline reinforcement learning method to obtain the control parameters of the computational torque controller as the optimization control strategy.

Set the state of the trajectory tracking task in the robot joint space at time t to:

;

in, , They are the joint angle error and joint angular velocity error of the robot arm respectively; is the accumulated value of the joint angle error of the robot arm; , and are the joint angles, joint angular velocities, and joint angular accelerations of the desired trajectory of the robot arm, respectively.

The computational torque controller is used to solve the motion of the trajectory tracking task in the joint space of the manipulator. The form of the computational torque controller is:

;

in, and are the joint angles and joint angular velocities of the robot arm, respectively; is the mass matrix of the robot; are the Coriolis force and the centripetal force; is a conservative force including gravity and spring force; is the joint friction of the robot arm; , and They are the joint angle error, joint angular velocity error and cumulative value of joint angle error of the robot arm respectively; , They are respectively used to calculate the control parameters of the torque controller;

Set the state of the trajectory tracking task of the robot operation space at time t to:

;

in, , are the posture error and velocity error vector of the end effector of the robot arm respectively; , They are the position error and linear velocity error of the end effector of the robot arm respectively; , They are the attitude error and angular velocity error of the end effector of the robot arm respectively; is the accumulated value of the position and posture error of the end effector of the robot arm; , and are the position, velocity and acceleration of the desired trajectory of the robot end effector respectively; and are the position and linear velocity of the end effector of the robot arm, respectively; , and are the position, linear velocity and linear acceleration of the desired trajectory of the robot end effector, respectively; is the quaternion representing the posture of the end effector of the robot arm, and are the rotation axis of the quaternion and the angle of rotation around the rotation axis respectively; is the quaternion of the desired trajectory of the robot end effector, and are the rotation axis of the quaternion and the angle of rotation around the rotation axis respectively; and are the first-order derivative and the second-order derivative of the quaternion of the desired trajectory of the robot end effector with respect to time; and The angular velocity of the end effector of the robot arm and the angular velocity of the desired trajectory respectively; the subscript t of the above parameters indicates that the parameter is the parameter at time t.

The action design used by the robot arm to perform trajectory tracking tasks in the operating space is an acceleration-based operating space calculation torque controller in the following form:

;

in, and are the joint angles and joint angular velocities of the robot arm, respectively; is the mass matrix of the robot; are the Coriolis force and the centripetal force; is a conservative force including gravity and spring force; is the joint friction of the robot arm; is the reference acceleration of the manipulator in the operating space, , are the posture error and velocity error vector of the end effector of the robot arm respectively; are the network parameters of the deep kinematics model; is the generalized inverse matrix of the Jacobian matrix of the robot; , Calculate the control parameters of the torque controller and select the control torque of the null space The form is:

;

in, and are the joint angles and joint angular velocities of the robot arm, respectively; is the initial value of the robot arm joint angle; is the identity matrix; are the network parameters of the deep kinematics model; is the generalized inverse matrix of the Jacobian matrix of the robot; is the Jacobian matrix of the robot; , They are respectively used to calculate the control parameters of the torque controller;

Similar to the trajectory tracking task setting in joint space, the policy learning output calculates the control parameters of the torque controller instead of the specific joint control torque.

The reward function is set as a segmented reward function, which is:

;

in, and are the weights of different control accuracies, including The term is used to quickly reduce the strategy exploration error. When the error is large, it is mainly composed of The item provides a reward value, which enables the strategy to explore actions with rapidly reduced errors; The term is used to make the strategy learn actions that improve accuracy. When the error is small, it is mainly composed of The term provides a reward value, which enables the strategy to learn actions that further improve accuracy; β is the value for adjusting the proportion of different weights, and β is set to 0.75. By adjusting the proportion of different weights through β, the high-precision tracking task of the desired trajectory is finally completed; , and In the trajectory tracking task of the robot arm joint space, they are the joint angle error, joint angular velocity error and the cumulative value of the joint angle error of the robot arm; in the trajectory tracking task of the robot arm operation space, they are the posture error, velocity error and the cumulative value of the posture error of the robot arm end effector.

The Soft Actor-Critic reinforcement learning algorithm, namely the SAC algorithm, is used as the strategy optimization algorithm of the present invention, and the strategy output calculates the control parameters of the torque controller , combined with the calculated torque controller, the final control torque is calculated, and the robot arm is controlled according to the final control torque.

The SAC algorithm introduces the maximum entropy objective while maximizing the expected reward to balance exploration and optimization and improve the performance and robustness of the strategy. The optimal strategy is:

;

Where T is the length of the robot arm's motion trajectory; is the expected value under the condition of state and action at time t; is the state of the robot trajectory tracking task at time t; is the action of the control strategy output at time t; is the reward function; is the discount factor; is the entropy of the control strategy; is the regularization coefficient of entropy, which uses adaptive updating to adjust the proportion of entropy in the objective function to control the randomness of the strategy.

The SAC algorithm uses a critic network to fit the state-action value function , which is used to evaluate the policy and is updated by minimizing the error of the Bellman equation as follows: ,in, is the state of the robot trajectory tracking task at time t+1, The action output by the control strategy at time t+1; is the state-action value function at time t; To control strategy is the expected value under the condition; t represents the time t; is the discount factor, and the superscript t represents the tth power of the discount factor; is the reward function; is the expected value under the condition of the state and action at time t+1; is the state-action value function at time t+1.

The actor network of the SAC algorithm is updated by minimizing the KL divergence of the following strategies:

;

in, is a sample in the distribution of control strategies; Π is the distribution of control strategies represented by the actor network; To minimize the KL divergence of the strategy; is the state of the robot trajectory tracking task at time t; is the quaternion representing the posture of the end effector of the robot arm; is the control strategy before updating; is the state-action value function before updating; is the partition function used to normalize the distribution.

The specific implementation process of the method is:

First, the actual motion trajectory data of the robot is collected for deep learning to establish a deep kinematic model and a deep dynamic model corresponding to the robot. The deep kinematic model is used to predict the position and posture of the end effector of the robot and calculate the Jacobian matrix corresponding to the robot. The deep dynamic model is used to predict the joint angle, angular velocity and angular acceleration of the robot to obtain the state change of the joint space of the robot, and obtain the control torque of the torque controller when the robot performs trajectory tracking tasks.

Then, a random excitation trajectory model described by a finite Fourier series is established, the random excitation trajectory given by the random excitation trajectory model is used as the expected motion trajectory to control the robotic arm, the actual motion trajectory of the robotic arm is measured and collected as a training data set, and the Jacobian-based deep kinematic model and the Lagrangian-based deep dynamics model established in step S1 are trained respectively;

Secondly, a Markov decision process model of the robot arm trajectory tracking task is established, wherein the state transition model in the Markov decision process model is a deep transfer model for simulating the robot arm motion trajectory constructed by combining the trained Jacobi-based deep kinematic model and the Lagrangian-based deep dynamics model;

Afterwards, according to the Markov decision process model, the control parameters of the torque controller are calculated offline through the Soft Actor-Critic reinforcement learning algorithm as the control strategy, the simulated motion trajectory data of the manipulator that interacts with the control strategy offline in the deep transfer model is collected, and the actor network and the critic network of the Soft Actor-Critic reinforcement learning algorithm are updated until the optimal control strategy is obtained;

Finally, the calculated torque controller calculates the specific control torque of the robot arm according to the optimal control strategy obtained in step S4 to control the robot arm.

Compared with the prior art, the beneficial effects of the present invention are as follows:

(1) The control method of the present invention is a learning method for a robot based on a Jacobian deep kinematic model and a Lagrangian deep dynamic model. Different from the traditional method, this method introduces prior knowledge, such as: the deep kinematic model introduces the prior knowledge of the Jacobian model for the velocity relationship of the robot arm; the deep dynamic model introduces the prior knowledge of the structure of the dynamic model (including the prior friction model) and the physical constraints such as the energy conservation and motion equation constraints of the robot arm system, rather than the information of the specific robot itself. The deep kinematic model and deep dynamic model corresponding to the robot arm are learned through a data-driven network model, without the need for a complex modeling process, and have high model accuracy and generalization. The deep kinematic model is used to predict the position and posture of the end effector of the robot arm in real time and calculate the Jacobian matrix. The deep dynamic model learns each component of the model in an unsupervised manner to predict the state change of the robot joint space. The deep transfer model composed of the deep kinematic model and the deep dynamic model can be used to simulate the motion trajectory of the robot, and then used to optimize the control strategy, and combined with the traditional computational torque control law to achieve higher control accuracy.

(2) The control method of the present invention is a model-based offline reinforcement learning method suitable for high-precision trajectory tracking of robots. The method can complete high-precision trajectory tracking tasks in joint space and operation space. By designing the state space, action space and reward function, and combining with the traditional computational torque controller, the rapid convergence and high-precision performance of the offline trajectory tracking strategy are achieved, while ensuring the stability and safety of the control strategy.

Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims (10)

1. A model-based offline reinforcement learning control method for a robot, characterized in that it is used to control a mechanical arm as a robot, comprising the following steps: Step S1, establishing a Jacobian-based deep kinematic model and a Lagrangian-based deep dynamic model corresponding to the robotic arm through deep learning; wherein the deep kinematic model is used to predict the position and posture of the end effector of the robotic arm and calculate the Jacobian matrix corresponding to the robotic arm; the deep dynamic model is used to predict the joint angle, angular velocity and angular acceleration of the robotic arm to obtain the state change of the joint space of the robotic arm, and obtain the control torque of the torque controller when the robotic arm performs a trajectory tracking task; Step S2, establishing a random excitation trajectory model described by a finite Fourier series, controlling the robotic arm with the random excitation trajectory given by the random excitation trajectory model as the expected motion trajectory, measuring and collecting the actual motion trajectory of the robotic arm as a training data set, and respectively training the Jacobian-based deep kinematic model and the Lagrangian-based deep dynamics model established in step S1; Step S3, establishing a Markov decision process model for the robot arm trajectory tracking task, wherein the state transition model in the Markov decision process model is a deep transfer model for simulating the robot arm motion trajectory constructed by combining the trained Jacobi-based deep kinematic model and the Lagrangian-based deep dynamics model; Step S4, according to the Markov decision process model, the control parameters of the torque controller are calculated offline through the Soft Actor-Critic reinforcement learning algorithm as the control strategy, the simulated motion trajectory data of the manipulator in which the deep transfer model interacts with the control strategy offline is collected, and the actor network and the critic network of the Soft Actor-Critic reinforcement learning algorithm are updated until the optimal control strategy is obtained; Step S5, the calculation torque controller calculates the specific control torque of the robot arm according to the optimal control strategy obtained in step S4 to control the robot arm. 2. The model-based offline reinforcement learning control method for robots according to claim 1, characterized in that, in step S1, a Jacobian-based deep kinematic model corresponding to the robot arm is established by deep learning in the following manner, including: The forward kinematics model of the robotic arm is determined as: ;in, is the position of the end effector of the robot arm in the operating space, is the degree of freedom of the robot's operating space; is the joint angle of the robot arm, is the degree of freedom of the robot joint space; The velocity relationship between the joints of the robot and the end effector of the robot is expressed by the differential of the kinematic equation f with respect to time: ;in, It is the derivative of the position of the end effector of the robot arm in the operating space with respect to time; is the Jacobian matrix actually corresponding to the robotic arm, is the degree of freedom of the robot's operating space, is the degree of freedom of the robot joint space; is the angular velocity of the robot joint; Determine the position and posture of the end effector of the robotic arm in the operating space as , where p is the position of the end effector of the robot arm; is the quaternion representing the posture of the end effector of the robot arm, ,in represents the rotation axis of the quaternion, , and are the three components of the rotation axis, represents the angle of rotation around the axis of rotation, Satisfy constraints ; Quaternion The relationship between the time derivative and the angular velocity ω of the end effector of the manipulator in the operating space is for , that is, the relationship for: ; in, , and are the components of the angular velocity of the end effector of the robot arm in the x-axis, y-axis and z-axis respectively; A fully connected deep neural network is established to learn the forward kinematics model of the robotic arm, and the corresponding deep kinematics model of the robotic arm is obtained as follows: ,in, are the network parameters of the deep kinematics model; is the kinematic equation derived from deep learning; q is the joint angle of the robot arm; The chain rule is applied to the derivatives of each layer in the deep kinematic model, and the derivatives of the output of the deep neural network with respect to the input are iteratively calculated to recover and learn the Jacobian matrix. The learned Jacobian matrix After the following calculation, it is expressed as: ; in, are the network parameters of the deep kinematics model; The partial derivatives of the kinematic equations derived from deep learning for the joint angles; is the i-th network layer of the deep kinematics model, , ; , , The weights of the deep kinematics model applied to the input of the previous layer of the deep neural network; for deviation; and They are the nonlinear activation function and its derivative of the deep neural network respectively. 3. The model-based offline reinforcement learning control method for robots according to claim 2, characterized in that the training data set of the deep kinematic model , where q is the joint angle of the robot arm; is the joint angular velocity of the robot arm, x is the position of the end effector of the robot arm in the operation space; is the derivative of the position of the end effector of the robot arm in the operating space with respect to time; The loss function during the training of the deep kinematics model is: ; in, are the network parameters of the deep kinematics model; It is a training dataset for deep kinematics models; is the size of the training dataset; is the generalized inverse of the Jacobian matrix; the first loss term is the mean square error between the actual position and predicted position of the robot in the operating space; the second loss term is the mean square error between the actual speed of the robot's operating space and the predicted speed; the last loss term It is the Jacobian matrix that makes the deep neural network more accurate by fitting the angular velocity of the robot arm joints. 4. The model-based offline reinforcement learning control method for robots according to any one of claims 1 to 3, characterized in that, in step S1, a Lagrangian-based deep dynamics model corresponding to the robot arm is established by deep learning in the following manner, including: The forward dynamics model and the inverse dynamics model of the robot arm are determined as follows: ; ; in, They are the joint angle, joint angular velocity and joint angular acceleration of the robot arm; is the torque acting on the joint of the robot arm; The generalized coordinates of the robot are selected as the joint angles of the robot, and the Lagrangian function is defined as ,in, is the kinetic energy of the robotic arm; is the potential energy of the robot arm; is the mass matrix of the robot; the superscript T indicates the transposed matrix; To ensure the mass matrix of the robot arm The positive definite symmetry of the mass matrix Perform Chelesky decomposition to obtain ,in, is a non-negative diagonal lower triangular matrix, the superscript T represents the transposed matrix, and the non-negative diagonal lower triangular matrix is learned using a deep neural network and the potential energy of the robotic arm To fit the Lagrangian function , combined with the Euler-Lagrange equation , where F is the generalized force and torque, and the deep forward dynamics model and deep inverse dynamics model that constitute the deep dynamics model corresponding to the robotic arm are obtained as follows: ; ; in, , and They are the joint angle, joint angular velocity and joint angular acceleration of the robot arm respectively; is the mass matrix of the robot; is the derivative of the mass matrix of the robot with respect to time; are the Coriolis force and the centripetal force; is a conservative force including gravity and spring force; Output torque for the joints of the robot arm; is the predicted value of the joint friction of the robot arm. The joint friction of the robot arm is obtained by introducing a priori friction model of the robot arm composed of Coulomb friction, viscous friction and Stribeck friction. The priori friction model of the joint of the robot arm is: ; in, is the Coulomb friction force; is the viscous friction coefficient; is the maximum static friction; υ and δ are the correlation coefficients of the Stribeck friction model respectively; is the angular velocity of the i-th joint of the robot. 5. The model-based offline reinforcement learning control method for robots according to claim 4, characterized in that the training data set of the deep dynamics model , where q is the joint angle of the robot arm; is the joint angular velocity of the robot arm; is the joint angular acceleration of the robot arm; is the joint torque of the robot arm; The loss function of the deep dynamics model is: ; in, It is a training dataset for deep dynamics models; is the size of the training dataset for the deep dynamics model; is the angular acceleration of the robot joint Regression loss, , are the predicted value and actual value of the joint angular acceleration of the robot arm at time t; is the joint torque Regression loss, , are the predicted value and actual value of the joint torque of the robot arm at time t respectively; is the multi-step prediction loss obtained by numerical integration, is the total number of steps predicted, , are the actual values of the joint angle and joint angular velocity of the robot arm at time t+1, and They are the predicted values of the robot arm joint angle and joint angular velocity at time t+1 respectively. 6. The model-based offline reinforcement learning control method for robots according to any one of claims 1 to 3, characterized in that, in step S2, a random excitation trajectory model described by a finite Fourier series is established in the following manner, including: For joint i of the robot at time t, the random excitation trajectory model Defined as: ; in, and are the amplitude of cosine and the amplitude of sine respectively; is the frequency of the cosine and the frequency of the sine; is the offset of the joint angle of the robot arm. The amplitude of the sine, the amplitude and frequency of the cosine and the offset of the joint angle are randomly selected to ensure that the joint angle, joint angular velocity and joint angular acceleration of the robot arm are within a safe range. is the number of Fourier series, randomly selected from 1 to 3, is the variable during accumulation, and t is the time value at time t. 7. The model-based offline reinforcement learning control method for robots according to any one of claims 1 to 3, characterized in that, in the step S3, a Markov decision process model of the robot trajectory tracking task is established in the following manner, and the state transition model of the Markov decision process model is a deep transition model for simulating the motion trajectory of the robot arm constructed by combining a trained Jacobi-based deep kinematic model and a Lagrangian-based deep dynamics model, comprising: Modeling the robot's trajectory tracking task as a finite-time discounted discrete Markov decision process ;in, is the state space, is the state at time t; is the action space; is the action at time t; It is a deep transition model as a state transition model; is the reward function; is the discount factor; A deep transfer model simulating the motion trajectory of the robotic arm is constructed by combining the trained Jacobian-based deep kinematic model and the Lagrangian-based deep dynamics model in the following manner, including: According to the joint angle and joint angular velocity of the robot at time t Joint torque with the robot arm The predicted values of the joint angle and joint angular velocity of the manipulator at time t+1 are obtained by using the fourth-order Runge-Kutta numerical integration method combined with the Lagrangian-based deep dynamics model. ; The Jacobian-based deep kinematic model is used to predict the joint angles and joint angular velocities of the robot at time t+1. The predicted values of the position and velocity of the end effector of the robot arm at time t+1 are obtained as input ; According to the predicted values of the joint angle, joint angular velocity of the robot arm and the position and velocity of the end effector at time t+1 , combined with the expected trajectory of the robot at time t+1, the error value required to complete the trajectory tracking task is calculated to form the state of the robot trajectory tracking task at time t+1 ; Use the robot arm trajectory at time t+1 to track the status of the task Building a deep transfer model , in this deep transfer model and are the states of the robot arm trajectory tracking task at time t and time t+1, respectively. is the action output by the control strategy at time t. 8. The model-based offline reinforcement learning control method for robots according to any one of claims 1 to 3, characterized in that in the step S4, the control parameters of the torque controller are calculated offline by the Soft Actor-Critic reinforcement learning algorithm as the control strategy according to the Markov decision process model, the simulated motion trajectory data of the robot arm that interacts with the control strategy offline by the deep transfer model is collected, and the actor network and the critic network of the Soft Actor-Critic reinforcement learning algorithm are updated until the optimal control strategy is obtained, including: The state space and action space of the Markov decision process model are respectively set according to the trajectory tracking task performed by the robot arm in the joint space and the trajectory tracking task performed in the operation space, and the model-based offline reinforcement learning is performed through the Soft Actor-Critic reinforcement learning algorithm, and the control parameters of the torque controller are output and calculated as the control strategy; Collect the simulated motion trajectory data of the robot arm that interacts with the control strategy offline , the actor network and critic network of the Soft Actor-Critic reinforcement learning algorithm are updated until the optimal control strategy is obtained. 9. The model-based offline reinforcement learning control method for robots according to claim 8 is characterized in that, in the method, the state space and action space of the Markov decision process model are respectively set according to the trajectory tracking task performed by the robot arm in the joint space and the trajectory tracking task performed in the operation space in the following manner, including: Set the state of the robot arm joint space for trajectory tracking task at time t for: ; in, , They are the joint angle error and joint angular velocity error of the robot arm respectively; is the accumulated value of the joint angle error of the robot arm; , and are the joint angle, joint angular velocity, and joint angular acceleration of the desired trajectory of the robot arm, respectively; The action used by the robot arm joint space to perform trajectory tracking tasks The computational torque controller is designed as follows: ; in, and are the joint angles and joint angular velocities of the robot arm, respectively; is the mass matrix of the robot; are the Coriolis force and the centripetal force; is a conservative force including gravity and spring force; is the joint friction of the robot arm; , and They are the joint angle error, joint angular velocity error and cumulative value of joint angle error of the robot arm respectively; , They are respectively used to calculate the control parameters of the torque controller; Set the state of the trajectory tracking task in the robot operation space at time t for: ; in, , are the posture error and velocity error vector of the end effector of the robot arm respectively; , They are the position error and linear velocity error of the end effector of the robot arm respectively; , They are the attitude error and angular velocity error of the end effector of the robot arm respectively; is the accumulated value of the position error of the end effector of the robot arm; , and are the position, velocity and acceleration of the desired trajectory of the robot end effector respectively; and are the position and linear velocity of the end effector of the robot arm, respectively; , and are the position, linear velocity and linear acceleration of the desired trajectory of the robot end effector, respectively; is the quaternion representing the posture of the end effector of the robot arm, and are the rotation axis of the quaternion and the angle of rotation around the rotation axis respectively; is the quaternion of the desired trajectory of the robot end effector, and are the rotation axis of the quaternion and the angle of rotation around the rotation axis respectively; and are the first-order derivative and the second-order derivative of the quaternion of the desired trajectory of the robot end effector with respect to time; and The angular velocity of the end effector of the robot arm and the angular velocity of the desired trajectory respectively; the subscript t of the above parameters indicates that the parameter is the parameter at time t; The action used by the robot to perform trajectory tracking tasks in the operating space is The acceleration-based operating space calculation torque controller is designed as follows: ; in, and are the joint angles and joint angular velocities of the robot arm, respectively; is the mass matrix of the robot; are the Coriolis force and the centripetal force; is a conservative force including gravity and spring force; is the joint friction of the robot arm; is the reference acceleration of the manipulator in the operating space, , are the posture error and velocity error vector of the end effector of the robot arm respectively; are the network parameters of the deep kinematics model; is the generalized inverse matrix of the Jacobian matrix of the robot; , Calculate the control parameters of the torque controller and select the control torque of the null space The form is: ; in, and are the joint angles and joint angular velocities of the robot arm, respectively; is the initial value of the robot arm joint angle; is the identity matrix; are the network parameters of the deep kinematics model; is the generalized inverse matrix of the Jacobian matrix of the robot; is the Jacobian matrix of the robot; , They are respectively used to calculate the control parameters of the torque controller; The reward function is set as a segmented reward function, which is: ; in, and are the weights of different control accuracies, including The term is used to quickly reduce the strategy exploration error, including The term is used to make the strategy learn actions that improve accuracy; β is the value for adjusting the proportion of different weights, and the value of β is 0.75; , and In the trajectory tracking task of the robot arm joint space, they are the joint angle error, joint angular velocity error and the cumulative value of the joint angle error of the robot arm; in the trajectory tracking task of the robot arm operation space, they are the posture error, velocity error and cumulative value of the posture error of the robot arm end effector; The actor network and critic network of the Soft Actor-Critic reinforcement learning algorithm are updated as follows: The Soft Actor-Critic reinforcement learning algorithm fits the state-action value function through the critic network To evaluate the strategy, update it by minimizing the error of the Bellman equation as follows: ,in, is the state of the robot trajectory tracking task at time t+1, The action output by the control strategy at time t+1; is the state-action value function at time t; To control strategy is the expected value under the condition; t represents the time t; is the discount factor, the superscript t represents the tth power of the discount factor, and t is the time value at time t; is the reward function; is the expected value under the condition of the state and action at time t+1; is the state-action value function at time t+1; The actor network of the Soft Actor-Critic reinforcement learning algorithm is updated by minimizing the KL divergence of the policy as follows: ; in, is a sample in the distribution of control strategies; Π is the distribution of control strategies represented by the actor network; To minimize the KL divergence of the strategy; is the state of the robot trajectory tracking task at time t; is the quaternion representing the posture of the end effector of the robot arm; is the control strategy before updating; is the state-action value function before updating; is the partition function used to normalize the distribution; The optimal control strategy derived from the Soft Actor-Critic reinforcement learning algorithm is to introduce the maximum entropy objective while maximizing the expected reward. for: ; Where T is the length of the robot's motion trajectory; is the expected value under the condition of state and action at time t; is the state of the robot trajectory tracking task at time t; is the action of the control strategy output at time t; is the reward function; is the discount factor at time t; is the entropy of the control strategy; is the regularization coefficient of entropy, which uses adaptive updating to adjust the proportion of entropy in the objective function to control the randomness of the strategy. 10. The model-based robot offline reinforcement learning control method according to claim 8, characterized in that in step S5, the control parameters of the calculated torque controller output by the calculated torque controller according to the optimal control strategy obtained in step S4 are , combined with the calculated torque controller, the specific control torque of the robotic arm is calculated.