CN118818968A - A quadruped robot motion control method based on deep reinforcement learning - Google Patents

Abstract

A four-foot robot motion control algorithm based on deep reinforcement learning comprises the following specific steps: s1, establishing a model of a quadruped robot, wherein the model comprises a dynamics model for simulating the quadruped robot and a driver model for identifying and simulating a motor driver of the quadruped robot, and the driver model adopts an experience driver model; s2, describing the motion process of the quadruped robot as a Markov process, designing a reward function, optimizing the motion strategy of the quadruped robot by using a near-end strategy optimization algorithm of a multi-loss function in the simulation environment established in the S1 by using a deep reinforcement learning algorithm, and training to obtain a motion controller; and S3, deploying the motion controller obtained through training on the quadruped robot. The invention can automatically learn the motion strategy in the simulation, reduce the difference from the simulation to the reality and realize the robust motion of the quadruped robot.

Description

A quadruped robot motion control method based on deep reinforcement learning

Technical Field

The present invention belongs to the technical field of robot control and relates to a quadruped robot motion control method based on deep reinforcement learning.

Background Art

With the continuous development of current robot control technology, the application of quadruped robots has become increasingly widespread. Compared with wheeled and tracked robots, quadruped robots have higher degrees of freedom and discrete footholds, showing great advantages in complex terrain operations. They can be widely used in search and rescue, reconnaissance, industrial inspection, and exploration of unknown environments.

However, the high degree of freedom of quadruped robots also brings great challenges to motion control. In recent years, many model-based methods have been applied to the motion control problem of quadruped robots, but such control methods often require careful design of various scenarios, and it is difficult to avoid corner cases. In contrast, reinforcement learning methods can autonomously learn a motion controller through trial and error, which can achieve good control effects in a variety of scenarios. This method often needs to be trained in a simulator first and then deployed on a real quadruped robot. However, since most simulators cannot fully simulate the complexity of the real environment, these controllers often suffer from relatively large performance losses during the sim-to-real transfer process.

Summary of the invention

The present invention provides a quadruped robot motion control algorithm based on deep reinforcement learning, which can automatically learn motion strategies in simulation, reduce the difference between simulation and reality, and realize robust motion of the quadruped robot.

The technical solution adopted by the present invention is:

A quadruped robot motion control algorithm based on deep reinforcement learning. The specific steps are as follows:

S1, establishing a model of a quadruped robot, including a dynamic model for simulating the quadruped robot and a driver model for identifying and simulating a motor driver of the quadruped robot, wherein the driver model adopts an empirical actuator model (EAM);

S2. Describe the motion process of the quadruped robot as a Markov process, design a reward function, use a deep reinforcement learning algorithm in the simulation environment established in S1, use the Multi-Loss Proximal Policy Optimization (MLPPO) algorithm to optimize the motion strategy of the quadruped robot, and train the motion controller.

S3. Deploy the trained motion controller to the quadruped robot.

Further, step S1 specifically includes the following steps:

S11. Establish a dynamic model of the quadruped robot, including the base mass and inertia tensor of the quadruped robot, the mass and inertia tensor of each joint link, the installation position and limit of each joint, and the collision model of each joint;

S12. Establish a driver model for the quadruped robot. The mathematical expression of the empirical driver model is as follows:

Among them, q _t and is the position and velocity of the joint at time t, _tin is the input delay of the joint, is the expected joint position given at time tt _in , and is the proportional derivative gain, is the expected output torque of the joint; t _out is the output delay of the driver torque, τ _m is the external characteristic curve of the motor; the external characteristic curve of the motor is the curve of the change of the maximum torque output by the motor with the motor speed.

Furthermore, the model generated in step S11 is described by a Unified Robotics Description Format (URDF) file, and the robot model is simulated using Multi-Joint dynamics with Contact (MuJoCo).

Further, step S2 specifically includes the following steps:

S21. Describe the motion process of the quadruped robot as a Markov decision process (MDP), including the state space Action Space State transfer function And the reward function At time t, the parameterized strategy π _θ generates actions based on the historical state The environment updates its state based on the state transition function And calculate the reward The goal of MDP is to maximize the discounted reward and in is the mathematical expectation, γ is the discount coefficient of the reward;

S22, collecting training data in the simulation environment, in each environment step, collecting the current environment state, giving the action of the current frame by the strategy, converting the action into joint torque through the empirical driver model, running the simulation to obtain the next frame state, calculating the reward value based on the two-frame state and action, and saving each state in the cache;

S23. After collecting a certain number of states, the policy is updated using Multi-Loss Proximal Policy Optimization (MLPPO). A parameterized policy can be expressed as the conditional probability of an action for a state, p _θ (a _t |s _t ), where θ is the policy parameter. The optimization objective of MLPPO is

minL _ppo +w _symmetry L _symmetry +w _smooth L _smooth

Among them, L _ppo is the loss function of standard PPO;

Among them, A _t is the advantage at time t, θ′ is the parameter of the strategy when collecting data, and ε is the clipping ratio; L _symmetry and L _smooth are special objective functions designed for quadruped robots, which are symmetric loss and smooth loss respectively, and w _symmetry and w _smooth are the weights of these two objective functions respectively. These two objective functions can be expressed as

in, and Represents the symmetric mapping of states and actions respectively.

Furthermore, the state space in step S21 Including robot linear velocity command _cx , _cy , angular velocity command _cr , 3D base linear velocity v, 3D base angular velocity ω, 12D joint angle q, 12D joint angular velocity As well as the base roll angle ψ _x and pitch angle ψ _y .

Furthermore, the action space in step S21 are the desired angles of the 12 joint angles.

Furthermore, the reward function in step S21 is is the weighted sum of a series of rewards These include rewards for tracking a given velocity command, penalties for power and joint motion, and rewards for base attitude and stable motion.

Further, step S3 specifically includes the following steps:

S31, obtaining various state quantities required by the strategy network from the onboard sensors of the quadruped robot, wherein the robot angular velocity and posture are obtained from the inertial measurement unit of the robot; each joint angle and joint angular velocity are obtained from the joint encoder, and the robot linear velocity is obtained from the state estimator;

S32. The strategy network is inferred in real time on the motion controller of the quadruped robot at a fixed frequency to generate actions, i.e., the desired joint positions, which are then sent to the joint motors to achieve whole-body control.

Beneficial effects of the present invention:

1. It can automatically learn motion strategies in simulation, reduce the difference between simulation and reality, and realize robust motion of quadruped robots.

2. Use the empirical actuator model (EAM) to identify the actual robot's actuator and motion strategy training to reduce the gap between simulation and reality.

3. The quadruped robot can accurately track the given speed command through the reinforcement learning method. Through the reinforcement learning framework of multiple loss functions, the symmetry and smoothness of the strategy are optimized while maximizing the reward function. Therefore, the strategy can drive the quadruped robot to run at a high speed of 4.2m/s and achieve a speed tracking error of less than 0.07m/s within a wide range of commands. Moreover, the strategy has excellent symmetry, smoothness and aesthetics, and exceeds the model-based controller in terms of energy efficiency.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic structural diagram of the present invention.

DETAILED DESCRIPTION

The present invention is further described below in conjunction with specific embodiments, but the present invention is not limited to these specific embodiments. Those skilled in the art should recognize that the present invention covers all possible alternatives, improvements and equivalents within the scope of the claims.

In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "clockwise", "counterclockwise" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, the meaning of "multiple" is two or more, unless otherwise clearly defined.

In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

Referring to FIG1 , the present invention provides a quadruped robot motion control algorithm based on deep reinforcement learning, and the specific steps are as follows:

S1, establishing a model of a quadruped robot, including a dynamic model for simulating the quadruped robot and a driver model for identifying and simulating a motor driver of the quadruped robot, wherein the driver model adopts an empirical actuator model (EAM);

The specific steps include:

S11. Establish a dynamic model of the quadruped robot, including the base mass and inertia tensor of the quadruped robot, the mass and inertia tensor of each joint link, the installation position and limit of each joint, and the collision model of each joint; the generated model is described by a unified robot description format URDF file, and the robot model is simulated using contact multi-joint dynamics MujoCo.

S12. Establish a driver model for the quadruped robot. The mathematical expression of the empirical driver model is as follows:

Among them, q _t and is the position and velocity of the joint at time t, _tin is the input delay of the joint, is the expected joint position given at time tt _in , and is the proportional derivative gain, is the expected output torque of the joint. t _out is the output delay of the driver torque, and τ _m is the external characteristic curve of the motor. The external characteristic curve of the motor is the curve of the maximum torque output by the motor versus the motor speed.

The quadruped robot joint motor operation data is collected at a frequency of 1000HZ to obtain a data set. The data set is traversed through t _in ∈ [0, t _max ], t _out ∈ [0, t _max ], and the linear regression is performed by minimizing the mean square error to obtain and The set of parameters with the smallest mean square error is taken as the identification value of the joint motor.

S2, describe the motion process of the quadruped robot as a Markov process, design a reward function, use a deep reinforcement learning algorithm in the simulation environment established in S1, use a proximal strategy optimization algorithm with multiple loss functions to optimize the motion strategy of the quadruped robot, and train the motion controller;

The specific steps include:

S21. Describe the motion process of the quadruped robot as a Markov decision process (MDP), including the state space Action Space State transfer function And the reward function At time t, the parameterized strategy π _θ generates actions based on the historical state The environment updates its state based on the state transition function And calculate the reward The goal of MDP is to maximize the discounted reward and in is the mathematical expectation, and γ is the discount factor of the reward. The state space Including robot linear velocity command _cx , _cy , angular velocity command _cr , 3D base linear velocity v, 3D base angular velocity ω, 12D joint angle q, 12D joint angular velocity And the base roll angle ψ _x and pitch angle ψ _y . Action Space are the desired angles of the 12 joint angles.

The reward function of this embodiment is the weighted sum of a series of rewards These include rewards for tracking a given velocity command, penalties for power and joint action, and rewards for base posture and stable motion. The reward function is shown in Table 1 below, where _ti is the swing time of the i-th leg.

Table 1 Reward function

S22, collecting training data in the simulation environment, in each environment step, collecting the current environment state, giving the action of the current frame by the strategy, converting the action into joint torque through the empirical driver model, running the simulation to obtain the next frame state, calculating the reward value based on the two-frame state and action, and saving each state in the cache;

S23. After collecting a certain number of states, use Multi-Loss Proximal Policy Optimization (MLPPO) to update the policy. A parameterized policy can be expressed as the conditional probability of an action for a state p _θ (a _t |s _t ), where θ is the parameter of the policy. The optimization goal of MLPPO is

minL _ppo +w _symmetry L _symmetry +w _smooth L _smooth

Among them, L _ppo is the loss function of standard PPO;

Among them, A _t is the advantage at time t, θ′ is the parameter of the strategy when collecting data, and ε is the clipping ratio; L _symmetry and L _smooth are special objective functions designed for quadruped robots, which are symmetric loss and smooth loss respectively, and w _symmetry and w _smooth are the weights of these two objective functions respectively. These two objective functions can be expressed as

in, and They represent the symmetric mapping of states and actions respectively. The symmetric mapping relationship is shown in Table 2.

Table 2 Symmetric mapping of states and actions

S3. Deploy the trained motion controller to the quadruped robot.

The specific steps include the following:

S31, obtaining various state quantities required by the strategy network from the onboard sensors of the quadruped robot, wherein the robot angular velocity and posture are obtained from the inertial measurement unit of the robot; each joint angle and joint angular velocity are obtained from the joint encoder, and the robot linear velocity is obtained from the state estimator;

S32, the strategy network is inferred in real time on the motion controller of the quadruped robot at a fixed frequency of 100HZ to generate actions, that is, the expected joint positions, which are then sent to the joint motors to achieve whole-body control.

Claims (8)

1. A quadruped robot motion control algorithm based on deep reinforcement learning. The specific steps are as follows: S1, establishing a model of a quadruped robot, including a dynamic model for simulating the quadruped robot and a driver model for identifying and simulating a motor driver of the quadruped robot, wherein the driver model adopts an empirical driver model; S2, describe the motion process of the quadruped robot as a Markov process, design a reward function, use a deep reinforcement learning algorithm in the simulation environment established in S1, use a proximal strategy optimization algorithm with multiple loss functions to optimize the motion strategy of the quadruped robot, and train the motion controller; S3. Deploy the trained motion controller to the quadruped robot. 2. According to the deep reinforcement learning-based quadruped robot motion control algorithm of claim 1, it is characterized in that: step S1 specifically comprises the following steps: S11. Establish a dynamic model of the quadruped robot, including the base mass and inertia tensor of the quadruped robot, the mass and inertia tensor of each joint link, the installation position and limit of each joint, and the collision model of each joint; S12. Establish an empirical driver model for the quadruped robot. The mathematical expression of the empirical driver model is as follows: Among them, q _t and are the position and velocity of the joint at time t, _tin is the input delay of the joint, is the expected joint position given at time tt _in , and is the proportional derivative gain, is the expected output torque of the joint; t _out is the output delay of the driver torque, τ _m is the external characteristic curve of the motor, and the external characteristic curve of the motor is the curve of the change of the maximum torque output by the motor with the motor speed. 3. According to the deep reinforcement learning-based quadruped robot motion control algorithm of claim 2, it is characterized in that: the model generated in step S11 is described by a unified robot description format file, and the robot model is simulated using contact multi-joint dynamics. 4. According to the deep reinforcement learning-based quadruped robot motion control algorithm of claim 1, it is characterized in that: step S2 specifically comprises the following steps: S21. Describe the motion process of the quadruped robot as a Markov process, including the state space Action Space State transfer function And the reward function At time t, the parameterized strategy π _θ generates actions based on the historical state The environment updates the state based on the state transition function And calculate the reward The goal of MDP is to maximize the discounted reward and in is the mathematical expectation, γ is the discount coefficient of the reward; S22, collecting training data in the simulation environment, in each environment step, collecting the current environment state, giving the action of the current frame by the strategy, converting the action into joint torque through the empirical driver model, running the simulation to obtain the next frame state, calculating the reward value based on the two-frame state and action, and saving each state in the cache; S23. After collecting a certain number of states, the proximal strategy with multiple loss functions is used to optimize the update strategy. A parameterized strategy can be expressed as the conditional probability of the action for the state p _θ (a _t |s _t ), where θ is the parameter of the strategy. The optimization goal of MLPPO is min L _ppo +w _symmetry L _symmetry +w _smooth L _smooth Among them, L _ppo is the loss function of standard PPO; Among them, A _t is the advantage at time t, θ′ is the parameter of the strategy when collecting data, and ε is the clipping ratio; L _symmetry and L _smooth are special objective functions designed for quadruped robots, which are symmetric loss and smooth loss respectively, and w _symmetry and w _smooth are the weights of these two objective functions respectively; these two objective functions can be expressed as in, and Represents the symmetric mapping of states and actions respectively. 5. A quadruped robot motion control algorithm based on deep reinforcement learning according to claim 4, characterized in that: the state space in step S21 Including robot linear velocity command _cx , _cy , angular velocity command _cr , 3D base linear velocity v, 3D base angular velocity ω, 12D joint angle q, 12D joint angular velocity As well as the base roll angle ψ _x and pitch angle ψ _y . 6. A quadruped robot motion control algorithm based on deep reinforcement learning according to claim 4, characterized in that: the action space in step S21 are the desired angles of the 12 joint angles. 7. A quadruped robot motion control algorithm based on deep reinforcement learning according to claim 4, characterized in that: the reward function in step S21 is the weighted sum of a series of rewards These include rewards for tracking a given velocity command, penalties for power and joint motion, and rewards for base attitude and stable motion. 8. The quadruped robot motion control algorithm based on deep reinforcement learning according to claim 1, characterized in that: step S3 specifically includes the following steps: S31, obtaining various state quantities required by the strategy network from the onboard sensors of the quadruped robot, wherein the robot angular velocity and posture are obtained from the inertial measurement unit of the robot; each joint angle and joint angular velocity are obtained from the joint encoder, and the robot linear velocity is obtained from the state estimator; S32. The strategy network is inferred in real time on the motion controller of the quadruped robot at a fixed frequency to generate actions, i.e., the desired joint positions, which are then sent to the joint motors to achieve whole-body control.