CN102402712B - A Neural Network-Based Initialization Method for Robot Reinforcement Learning - Google Patents

Abstract

The invention proposes a neural network-based initialization method for robot reinforcement learning. The neural network has the same topological structure as the robot workspace, and each neuron corresponds to a discrete state of the state space. First, the neural network is evolved according to the known part of the environmental information until it reaches an equilibrium state. At this time, the output value of each neuron represents the maximum cumulative return that the corresponding state can obtain by following the optimal strategy. Then define the initial value of the Q function as the immediate return of the current state plus the maximum discounted cumulative return that the subsequent state can obtain by following the optimal strategy. The neural network maps the known environmental information into the initial value of the Q function, thereby integrating the prior knowledge into the robot learning system and improving the learning ability of the robot in the initial stage of reinforcement learning. Compared with the traditional Q learning algorithm, the present invention can effectively Improve the learning efficiency in the initial stage and speed up the algorithm convergence speed.

Description

A Neural Network-Based Initialization Method for Robot Reinforcement Learning

technical field

The invention relates to a method for integrating prior knowledge into a learning system of a mobile robot and for initializing a Q value in the robot reinforcement learning process, and belongs to the technical field of machine learning.

Background technique

With the continuous expansion of the application field of robots, the tasks faced by robots are becoming more and more complex. Although researchers can pre-program the repetitive behaviors that robots may perform in many cases, the behavior design changes to achieve the overall desired behavior It is becoming more and more difficult for designers to make reasonable predictions about all the behaviors of robots in advance. Therefore, an autonomous robot capable of perceiving the environment must be able to learn new behaviors online by interacting with the environment, so that the robot can choose the optimal action to achieve the goal according to a specific task.

Reinforcement learning uses a trial-and-error method similar to human thinking to discover the optimal behavior strategy, and has shown good learning performance in robot behavior learning. The Q-learning algorithm is a reinforcement learning method for solving Markov decision-making problems with incomplete information. According to the environment state and the immediate return obtained from the previous step of learning, the mapping strategy from the state to the action is modified so that the cumulative return value obtained by the behavior from the environment The maximum, so as to obtain the optimal behavior policy. The standard Q-learning algorithm generally initializes the Q value to Q or a random number. The robot has no prior knowledge of the environment, and the initial stage of learning can only choose actions randomly. Therefore, the algorithm converges slowly in complex environments. In order to increase the convergence speed of the algorithm, researchers have proposed many methods to improve Q-learning, improve the learning efficiency of the algorithm, and improve the learning performance.

Usually, the method to accelerate the convergence speed of Q-learning mainly includes two aspects: one method is to design an appropriate reward function, and the other method is to initialize the Q function reasonably. At present, researchers have proposed many improved Q-learning algorithms to enable robots to obtain more effective returns in the process of reinforcement learning, mainly including: associative Q-learning algorithm, lazy Q-learning algorithm, Bayesian Q-learning algorithm, etc. Its main purpose is to integrate the implicit information valuable to the robot into the reward function, thereby accelerating the convergence speed of the algorithm. Associative Q-learning compares the current reward with the immediate reward of the past moment, and selects the action with a greater reward value. The learning ability of the system can be improved through the method of associating rewards, and the number of iteration steps required to obtain the optimal value can be reduced. The goal of lazy Q-learning is to provide a method to predict the immediate reward of the state. The principle of information delay is used in the learning process to predict new targets if necessary. The action comparator checks the expected reward of each situation and chooses Action execution with maximum expected reward. Bayesian Q-learning uses the probability distribution to describe the uncertainty estimation of the robot state-action on the Q value. In the learning process, the distribution of the Q value at the previous moment needs to be considered, and the previous distribution is updated using the experience learned by the robot. The Bayesian variable represents the maximum cumulative return of the current state, and the Bayesian method essentially improves the exploration strategy of Q-learning and improves the performance of Q-learning.

Since the reinforcement signals in standard reinforcement learning are all scalar values calculated by the state value function, it is impossible to integrate human knowledge and behavior patterns into the learning system. In the process of robot learning, people often have experience and knowledge in related fields. Therefore, feeding back human cognition and intelligence to the robot in the form of strengthened signals during the learning process can reduce the dimension of the state space and speed up the convergence of the algorithm. speed. In view of the problems existing in the human-computer interaction process of standard reinforcement learning, Thomaz et al. gave external reinforcement signals in real time during the robot reinforcement learning process, and humans adjusted the training behavior according to their own experience to guide the robot to perform forward-looking exploration. Arsenio proposed a learning strategy for online and automatic labeling of training data. During the human-computer interaction process, specific events are triggered to obtain training data, thereby embedding the instructor into the feedback loop of reinforcement learning. Mirza et al. proposed an architecture based on interaction history. Robots can use the historical experience of social interaction with humans for reinforcement learning, enabling robots to gradually acquire appropriate behaviors in simple games with humans.

Another way to improve the performance of Q-learning algorithm is to integrate prior knowledge into the learning system and initialize the Q-value. At present, the methods for initializing Q value mainly include approximate function method, fuzzy rule method, potential function method and so on. The approximate function method uses intelligent systems such as neural networks to approximate the optimal value function, maps prior knowledge into reward function values, and enables the robot to learn on a subset of the entire state space, thereby speeding up the convergence of the algorithm. The fuzzy rule method establishes a fuzzy rule base according to the initial environmental information, and then uses fuzzy logic to initialize the Q value. The fuzzy rules established by this method are all artificially set according to the environmental information, which often cannot objectively reflect the environmental state of the robot, resulting in the instability of the algorithm. The potential function method defines the corresponding state potential function in the entire state space. Each potential energy value corresponds to a discrete state value in the state space, and then uses the state potential function to initialize the Q value. The Q value of the learning system can be expressed as the initial value Plus the amount of change for each iteration. Among the various behaviors of the robot, the robot must abide by a series of codes of conduct. The robot emerges with corresponding behavior and intelligence through cognition and interaction. The Q value initialization of the robot reinforcement learning is to map the prior knowledge into the corresponding robot behavior. . Therefore, how to obtain the regular expression form of prior knowledge, especially the machine reasoning of field experts' experience and common sense knowledge, and the fusion technology of man-machine intelligence that transforms human cognition and intelligence into machine calculation and reasoning are robot behaviors. Learn about urgent problems to solve.

Contents of the invention

Aiming at the research status and existing deficiencies of the existing robot reinforcement learning technology, the present invention proposes a neural network-based robot reinforcement learning initialization method that can effectively improve the learning efficiency in the initial stage and accelerate the convergence speed. The method can be initialized by Q value Integrating the prior knowledge into the learning system optimizes the learning of the initial stage of the robot, thus providing a better learning basis for the robot.

In the neural network-based robot reinforcement learning initialization method of the present invention, the neural network has the same topology as the robot workspace, and each neuron corresponds to a discrete state in the state space. First, the neural network is evolved according to the known partial environmental information until it reaches an equilibrium state. At this time, the output value of each neuron represents the maximum cumulative return that its corresponding state can obtain, and then the immediate return of the current state is added to the subsequent The state follows the optimal strategy to obtain the maximum discounted cumulative return (the maximum cumulative return multiplied by the conversion factor), which can set a reasonable initial value for Q(s, a) of all state-action pairs. Through the initialization of the Q value, the previous Integrate empirical knowledge into the learning system, and optimize the learning of the robot in the initial stage, so as to provide a better learning basis for the robot; specifically, the following steps are included:

(1) Establish a neural network model

The neural network has the same topology as the structural space where the robot works. Each neuron is only connected to the neurons in its local neighborhood. The connection forms are the same, the connection weights are equal, and the information transmission between neurons is bidirectional. The neural network has a highly parallel architecture, and each neuron corresponds to a discrete state of the robot's workspace. The entire neural network consists of N×N neurons in a two-dimensional topology, and the neural network evolves according to each discrete The input of the state updates the state in its neighborhood until the neural network reaches the equilibrium state. When the equilibrium state is reached, the output value of the neuron in the neural network forms a single-peak surface, and the value of each point on the surface indicates that the corresponding state can be obtained. The maximum cumulative return of ;

(2) Design return function

During the learning process, the robot can move in 4 directions. In any state, choose 4 actions: up, down, left, and right. The robot chooses an action according to the current state. If the action makes the robot reach the goal, the immediate reward is 1. If the robot collides with objects or other robots, the immediate reward is -0.2, and if the robot is moving in free space, the immediate reward is -0.1;

(3) Calculate the initial value of the maximum cumulative return

When the neural network reaches the equilibrium state, define the maximum cumulative reward V ^* _Init (s _i ) corresponding to the state of each neuron is equal to the output value x _i of the neuron, and the relationship formula is as follows:

${\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; Init</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&LeftArrow;</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>$

In the formula, x _i is the output value of the i-th neuron when the neural network reaches the equilibrium state, and V ^* _Init (s _i ) is the maximum cumulative return that can be obtained from the state s _i following the optimal strategy;

(4) Q value initialization

The initial value of Q(s _i , a) is defined as the immediate reward r obtained by choosing action a in state s _i plus the maximum discounted cumulative reward of the subsequent state:

${\mathrm{<span\; class="notranslate">\; Q</span>}}_{\mathrm{<span\; class="notranslate">\; Init</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&gamma;</span>}{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; Init</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>$

In the formula, s _j is the subsequent state generated by the robot choosing action a in state s _i , Q _Init (s _i , a) is the initial Q value of the state-action pair (s _i , a); γ is the conversion factor, Select gamma = 0.95;

(5) Neural Network-Based Robot Reinforcement Learning Steps

(a) The neural network evolves according to the initial environment information until it reaches an equilibrium state;

(b) The initial value of the maximum cumulative return obtained by state s _i is defined as the neuron output value x _i , and the relationship formula is as follows:

${\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; Init</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&LeftArrow;</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}$

(c) Initialize the Q value according to the following rules: $Q_{\mathrm{Init}}({\mathrm{the\; s}}_i,a)=r+\mathrm{\&gamma;}{V}_{\mathrm{Init}}^{*}\left({\mathrm{the\; s}}_j\right),$

(d) observe the current state s _t ;

(e) Continue to explore in the complex environment, select and execute an action a _t in the current state s _t , update the environment state to a new state s' _t , and receive an immediate reward r _t ;

(f) observe the new state s'_t;

(g) Update the value of table item Q(s _t , a _t ) according to the following formula:

${\mathrm{<span\; class="notranslate">\; Q</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; Q</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&gamma;</span>}\underset{{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}}\mathrm{<span\; class="notranslate">\; max</span>}{\mathrm{<span\; class="notranslate">\; Q</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>$

In the formula, α _t is the learning rate, the value range is (0, 1), usually 0.5, and decays with the learning process; Q _t-1 (s _t , a _t ) and Q _t-1 (s' _t , a' _t ) are the values of the state-action pair (s _t , a _t ) and (s' _t , a' _t ) at time t-1 respectively, and a' _t is the value selected under the new state s' _t Actions;

(h) Judging whether the robot has reached the target or the learning system has reached the set maximum number of learning times, the set maximum number of learning times should ensure that the learning system converges within the maximum number of learning times, if the two meet one of them, then the learning ends, otherwise Return to step (d) to continue learning.

The invention maps the known environmental information into the initial value of the Q function through the neural network, thereby integrating prior knowledge into the robot learning system, improving the learning ability of the robot in the initial stage of reinforcement learning, and compared with the traditional Q learning algorithm, it can Effectively improve the learning efficiency in the initial stage and accelerate the convergence speed of the algorithm.

Description of drawings

Figure 1 is a schematic diagram of the i-th neuron neighborhood structure.

Fig. 2 is a schematic diagram of neuron output values in the neighborhood of the target point of the robot.

Figure 3 is a schematic diagram of the initial value V* _Init (s, a) of the maximum cumulative return.

Fig. 4 is a schematic diagram of neuron output values when the neural network reaches an equilibrium state.

FIG. 5 is a schematic diagram of a robot planning path obtained by existing Q-learning.

Fig. 6 is a schematic diagram of the convergence process of the existing Q-learning algorithm.

Fig. 7 is a schematic diagram of the robot planning path of the present invention.

Fig. 8 is a schematic diagram of the learning convergence process of the present invention.

Detailed ways

The invention initializes the reinforcement learning of the robot based on the neural network. The neural network and the robot workspace have the same topological structure. When the neural network reaches the equilibrium state, the output value of the neuron represents the maximum cumulative return of the corresponding state, and the immediate return of the current state and the The maximum discounted cumulative reward of the successor state obtains the initial value of the Q function. The prior knowledge can be integrated into the learning system through the Q value initialization, and the learning of the initial stage of the robot can be optimized, so as to provide a better learning basis for the robot; specifically, the following steps are included:

1 Neural Network Model

The neural network has the same topology as the robot workspace, and each neuron corresponds to a discrete state of the robot workspace. All neurons are only connected to neurons in their local neighborhood, and their connection forms are the same. The whole neural network consists of N×N neurons to form a two-dimensional topology. The neural network has a highly parallel architecture, all connection weights are equal, and the propagation of information between neurons is bidirectional. During the evolution process of the neural network, the state in its neighborhood is updated according to the input of each discrete state, and the whole neural network can be regarded as a discrete-time dynamical system.

During the evolution of the neural network, the external input of the neural network is generated according to the mapping of the target point and obstacle position information in the neural network topology. The neurons corresponding to the obstacle area have negative external inputs, and the target point neurons have positive external inputs. external input. The neural network evolves according to the external input, and the output value of the neuron at the target point position can be gradually attenuated and propagated to the entire state space through the local connection of the neuron until it reaches an equilibrium state. The S-type activation function ensures that the neuron at the target point has the global maximum positive neuron output value, and the output value of the neuron in the obstacle area is suppressed to zero. After the neural network reaches equilibrium, the output values of all neurons form a single-peak surface, and the value of each point on the surface represents the maximum cumulative return available for its corresponding state.

Assuming that the robot workspace consists of 20×20 squares, the neural network has the same topology as the robot workspace, and also contains 20×20 neurons, and each neuron corresponds to a discrete state of the workspace. Each neuron is only connected to neurons in its local neighborhood, and the connection form of the i-th neuron to neurons in its neighborhood is shown in Figure 1. The whole neural network consists of 20×20 neurons to form a two-dimensional topology. Neural networks have a highly parallel architecture where all connections are equally weighted. In the process of neural network evolution, all neurons are both input neurons and output neurons, and the transmission of information between neurons is bidirectional. The entire neural network can be regarded as a discrete-time dynamic system.

The i-th neuron of the neural network corresponds to the i-th discrete state of the structure space, then the discrete-time dynamic equation of the i-th neuron is:

${\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; ifi</span>}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; i</span>}}_{<span\; class="notranslate">\; *</span>}\\ \mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; otherwise</span>}\end{array}\right.$

In the formula, i _* is the index value of the target neuron, x _i (t) is the output value of the i-th neuron at time t, N is the number of neurons in the neighborhood of the i-th neuron, I _i (t) is the external input of the i-th neuron at time t, f is the activation function, w _ij is the connection weight from the j-th neuron to the i-th neuron, and the calculation formula is as follows:

${\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; e</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&eta;</span>}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}& \mathrm{<span\; class="notranslate">\; if</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; <figure-callout\; id="0"\; label="r"\; filenames="HDA0000088055920000012.png,HDA0000088055920000013.png"\; state="\{\{state\}\}">r</figure-callout></span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; if</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; ></span>\mathrm{<span\; class="notranslate">\; r</span>}\end{array}\right.$

In the formula, |ij| is the Euclidian distance between the vectors x _i and x _j in the structural space. Since each neuron is only connected to neurons in its local neighborhood, r takes a value of 1. In order to ensure that the neural network reaches an equilibrium state The neuron output forms a single-peak surface, and the value range of η is (1, 2), obviously w _ij =w _ji , that is, w _ij is symmetrical; the neuron activation function selects a S-type function, which is defined as follows:

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; ifx</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; 0</span>}\\ \mathrm{<span\; class="notranslate">\; x</span>}& \mathrm{<span\; class="notranslate">\; if</span>}\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; 1</span>}\\ \mathrm{<span\; class="notranslate">\; 1</span>}& \mathrm{<span\; class="notranslate">\; ifx</span>}<span\; class="notranslate">\; \&Greater\; Equal;</span>\mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right.$

In the formula, k is the slope of the linear model, and the value range is (0, 1). f(x) ensures that the output value of the neuron at the positive position of the target point can be gradually attenuated and propagated to the entire state space, and the target point has a global The largest positive neuron output value, the output value of the neuron in the obstacle area is suppressed to zero; the external input of the i-th neuron is generated by the mapping of the target point and obstacle position information in the neural network topology, defined as follows:

${\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; V</span>}& \mathrm{<span\; class="notranslate">\; if</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; et</span>}\\ <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; V</span>}& \mathrm{<span\; class="notranslate">\; if</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; <figure-callout\; id="0"\; label="obstacle"\; filenames="HDA0000088055920000012.png,HDA0000088055920000013.png"\; state="\{\{state\}\}">obstacle</figure-callout></span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; otherwise</span>}\end{array}\right.$

In the formula, V is a relatively large constant. In order to ensure that the target point neuron has the global maximum neuron output value, and the obstacle area neuron has the global minimum neuron output value, the value of V should be greater than the sum of neuron input , the value range is a real number greater than 4.

2 Return function design

During the learning process, the robot can move in 4 directions. In any state, you can choose 4 actions: up, down, left, and right. The robot chooses an action according to the current state. If the action makes the robot reach the goal, the immediate reward is 1. If the robot and An immediate reward of -0.2 is obtained for collisions with obstacles or other robots, and -0.1 if the robot is moving in free space.

3 Calculate the initial value of the maximum cumulative return

According to the mapping of the target point and obstacle position information in the neural network topology, the external input of the neural network is generated. The neuron corresponding to the obstacle area has a negative external input, and the target point neuron has a positive external input. The neural network evolves according to the external input, and the output value of the neuron at the target point position can be gradually attenuated and propagated to the entire state space through the local connection of the neuron until it reaches an equilibrium state. The S-type activation function ensures that the neuron at the target point has the global maximum positive neuron output value, and the output value of the neuron in the obstacle area is suppressed to zero. After the neural network reaches the equilibrium state, the output values of all neurons form a single-peak surface, as shown in Figure 2, the value of each point on the surface represents the maximum cumulative return available for its corresponding state.

The cumulative reward obtained by the robot starting from any initial state s _t is defined as follows:

${\mathrm{<span\; class="notranslate">\; V</span>}}^{\mathrm{<span\; class="notranslate">\; \&pi;</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&gamma;</span>}{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&gamma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&Lambda;</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{<span\; class="notranslate">\; \&infin;</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&gamma;</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}}$

In the above formula, π is the control strategy, r is the obtained immediate reward sequence, γ is the conversion factor, and the value range is (0, 1). Here, γ=0.95 is selected; then the robot starts from the state s and follows the optimal strategy to obtain The maximum cumulative return V ^* (s) for is calculated as follows:

${\mathrm{<span\; class="notranslate">\; V</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; \&pi;</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}{\mathrm{<span\; class="notranslate">\; V</span>}}^{\mathrm{<span\; class="notranslate">\; \&pi;</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span><span\; class="notranslate">\; (</span><span\; class="notranslate">\; \&ForAll;</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span>$

When the neural network reaches the equilibrium state, define the maximum cumulative reward V ^* _Init (s _i ) corresponding to the state of each neuron is equal to the output value x _i of the neuron, and the relationship formula is as follows:

${\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; Init</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&LeftArrow;</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; .</span>$

In the formula, _xi is the output value of the i-th neuron when the neural network reaches the equilibrium state, and V ^* _Init (s _i ) is the maximum cumulative reward that can be obtained from the state s _i following the optimal strategy.

4 Neural Network-Based Reinforcement Learning for Robots

4.1 Traditional Q-learning algorithm

In the Markov decision-making process, the robot perceives the surrounding environment through sensors to know the current state, and chooses the action to be performed currently. The environment responds to the action and gives an immediate reward, and generates a subsequent state. The task of robot reinforcement learning is to obtain an optimal strategy to enable the robot to obtain the maximum discounted cumulative return from the current state. The cumulative reward obtained by a robot following any policy π from any initial state is defined as:

${\mathrm{<span\; class="notranslate">\; V</span>}}^{\mathrm{<span\; class="notranslate">\; \&pi;</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&equiv;</span>{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&gamma;</span>}{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&gamma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&Lambda;</span>}<span\; class="notranslate">\; \&equiv;</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{<span\; class="notranslate">\; \&infin;</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; \&gamma;</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; i</span>}}$

In the formula, r _t is the immediate return at time t, γ is the conversion factor, and the value range is (0, 1). Here, γ=0.95 is selected.

The optimal strategy π* for the robot to obtain the maximum cumulative reward starting from the state s is defined as follows:

${\mathrm{<span\; class="notranslate">\; \&pi;</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; \&equiv;</span>\underset{\mathrm{<span\; class="notranslate">\; \&pi;</span>}}{\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}{\mathrm{<span\; class="notranslate">\; V</span>}}^{\mathrm{<span\; class="notranslate">\; \&pi;</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span><span\; class="notranslate">\; (</span><span\; class="notranslate">\; \&ForAll;</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; )</span>$

The maximum cumulative return that the robot can obtain from the state s following the optimal strategy π* is defined as V*(s), then the value of the Q function is the immediate return of the current state plus the maximum discounted cumulative return of the subsequent state, and the calculation formula is as follows :

Q(s, a)≡(1-α _t )Q(s, a)+α _t (r(s, a)+γV*(s′))

In the formula, α _t is the learning rate, and the value range is (0, 1). Usually, the initial value of α _t is 0.5, and it decays with the number of learning times; the relationship between V*(s') and Q(s', a') The formula is as follows:

$\mathrm{<span\; class="notranslate">\; V</span>}<span\; class="notranslate">\; *</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; a</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}{\mathrm{<span\; class="notranslate">\; max</span>}\mathrm{<span\; class="notranslate">\; Q</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span>}$

Then Q(s _t , a _t ) is updated according to the following rules:

${\mathrm{<span\; class="notranslate">\; Q</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; Q</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; +</span>\underset{{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}{\mathrm{<span\; class="notranslate">\; \&gamma;</span>}\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}{\mathrm{<span\; class="notranslate">\; Q</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>$

In the formula, Q _t-1 (s _t , a _t ) and Q _t-1 (s' _t , a' _t ) are state-action pairs (s _t , a _t ) and (s' _t , a' _t ) at time t-1, a' _t is the action selected in the new state s' _t .

4.2 Q value initialization

The neural network is evolved according to the known environmental information until it reaches an equilibrium state. At this time, it is defined that the maximum cumulative reward that can be obtained for each discrete state is equal to the output value of its corresponding neuron. Then add the immediate reward obtained by executing the selected action from the current state to the maximum discounted cumulative reward obtained by the subsequent state following the optimal strategy, and then set a reasonable initial value for Q(s _i , a) of all state-action pairs . The calculation formula of the initial value of Q(s _i , a) is as follows:

${\mathrm{<span\; class="notranslate">\; Q</span>}}_{\mathrm{<span\; class="notranslate">\; Init</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&gamma;</span>}{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; Init</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>$

In the formula, r is the immediate reward obtained by choosing action a in state s _i , γ is the conversion factor, and the value range is (0, 1), here choose γ=0.95; s _j is the action selected by the robot in state s _i The subsequent state generated by a, Q _Init (s _i , a) is the initial Q value of the state-action pair (s _i , a);

4.3 Q learning algorithm based on neural network of the present invention

(1) The neural network evolves according to the initial environment information until it reaches an equilibrium state.

(2) Use the neuron output value _xi to initialize the maximum cumulative reward that can be obtained from state _si , and the relationship formula is as follows:

${\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; Init</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&LeftArrow;</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}$

(3) Initialize the Q value according to the following rules:

Q _Init (s _i , a)=r+γV _Init *(s _j )

(4) Observe the current state s _t .

(5) Continue to explore in the complex environment, select and execute an action a _t in the current state s _t , update the environment state to a new state s' _t , and receive an immediate reward r _t .

(6) Observe the new state s' _t .

(7) Update the value of table item Q(s _t , a _t ) according to the following formula:

${\mathrm{<span\; class="notranslate">\; Q</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; Q</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; +</span>\underset{{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}{\mathrm{<span\; class="notranslate">\; \&gamma;</span>}\mathrm{<span\; class="notranslate">\; arg</span>}\mathrm{<span\; class="notranslate">\; max</span>}}{\mathrm{<span\; class="notranslate">\; Q</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>$

(8) Judging whether the robot has reached the target or the learning system has reached the maximum number of studies set (the maximum number of studies set can ensure that the learning system converges in the maximum number of studies, in the experimental environment of the present invention, the maximum number of studies is set to 300), if the two meet one of them, then the learning ends, otherwise return to step (4) to continue learning.

In order to illustrate the Q value initialization process of robot reinforcement learning, the robot target point neighborhood is selected for demonstration. When the neural network reaches an equilibrium state, the output values of the neurons in the neighborhood are shown in the nodes in Figure 3. Each node corresponds to a discrete state, and the maximum cumulative return of each state is equal to the output value of the state neurons. The red nodes represents the target state, and gray nodes represent obstacles. Each arrow represents an action. If the robot is directed to the goal state G, the immediate reward obtained is 1, if it collides with obstacles or other robots, the immediate reward obtained is -0.2, and if the robot moves in free space, the immediate reward obtained is is -0.1. γ is the conversion factor, select γ=0.95, and the initial value of the Q function can be obtained according to the Q value initialization formula. The initial Q value of each state-action pair is shown in Figure 4 as the arrow represents the value. After the initialization is completed, the robot can choose the appropriate action in any initial state. When the robot faces a more complex environment, it has a certain purpose in the initial stage of learning, instead of choosing actions completely randomly, thereby speeding up the algorithm convergence. speed.

On the mobile robot environment modeling and exploration software platform established in the laboratory, the simulation experiment was carried out. Fig. 5 is the robot planning path obtained by the existing robot reinforcement learning method; Fig. 6 is the convergence process of the existing robot reinforcement learning algorithm. The learning algorithm starts to converge after 145 times of learning. In the initial stage of learning (such as the first 20 times of learning), the robot basically cannot reach the target point within the maximum number of iterations. This is because the Q value is initialized to 0, so that the robot does not have any prior knowledge and can only choose actions randomly, resulting in low efficiency in the initial stage of learning and slow algorithm convergence.

Fig. 7 is the robot planning path of the present invention; Fig. 8 is the convergence process of the present invention. The learning algorithm starts to converge after 76 times of learning, and the robot can basically reach the target point within the maximum number of iterations in the initial stage of learning. The invention effectively improves the learning efficiency of the robot in the initial stage, and obviously accelerates the learning process. convergence speed.

Claims (1)

1. the robot reinforced learning initialization method based on neural network, neural network has identical topological structure with robot working space, each neuron is corresponding to a discrete state in state space, first according to known component environment information, neural network is developed, until reach equilibrium state, at this moment each neuron output value just represents the cumulative maximum return that its corresponding states can obtain, then the return immediately of current state is added to follow-up state follows the maximum conversion accumulation return that optimal strategy obtains, can be to the right Q (s of all states-move, a) set rational initial value, by Q value initialization, priori can be dissolved in learning system, study to the robot initial stage is optimized, thereby for robot provides a good learning foundation, specifically comprise the following steps:

(1) set up neural network model

Neural network has identical topological structure with the structure space of robot work, each neuron is only connected with the neuron in its local neighborhood, type of attachment is all identical, connection weight all equates, between neuron, the propagation of information is two-way, neural network has the architecture of highly-parallel, each neuron is corresponding to one of robot working space discrete state, whole neural network forms two dimensional topology by N * N neuron, neural network is upgraded the state in its neighborhood according to the input of each discrete state in evolutionary process, until neural network reaches equilibrium state, while reaching equilibrium state, in neural network, neuron output value just forms a single-peaked curved surface, on curved surface, the value of every bit just represents the cumulative maximum return that institute's corresponding states can obtain,

(2) design return function

In learning process, robot can move up 4 sides, can select action according to current state, in free position, can both in 4 actions of advancing, retreat, turn left, turn right, select a suitable action, if this action makes robot arrive target, the return immediately obtaining is 1, if robot and barrier or other robot bump, obtain return immediately for-0.2, if robot moves at free space, the return immediately obtaining is-0.1;

(3) calculate cumulative maximum return initial value

When neural network reaches equilibrium state, define the cumulative maximum return V of the corresponding state of each neuron ^* _init(s _i) equal this neuron output value x _i, its relation formula is as follows:

$V_{\mathrm{Init}}^{*}\left(s_i\right)\&LeftArrow;x_i,$

In formula, xi is neural network i neuronic output valve while reaching equilibrium state, V ^* _init(s _i) be from state s _iset out and follow the obtainable cumulative maximum return of optimal strategy institute;

(4) Q value initialization

Q(s _i, initial value a) is defined as at state s _ilower selection action a obtains returns the maximum conversion accumulation return that r adds follow-up state immediately:

$Q_{\mathrm{Init}}(s_i,a)=r+{\mathrm{\&gamma;V}}_{\mathrm{Init}}^{*}\left(s_j\right)$

In formula, s _jfor robot is at state s _ithe follow-up state that lower selection action a produces, Q _init(s _i, be a) that state-action is to (s _i, initial Q value a); γ is commutation factor, selects γ=0.95;

(5) the robot intensified learning step based on neural network

(a) neural network develops according to initialization context information, until reach equilibrium state;

(b) state s _ithe initial value of the cumulative maximum return obtaining is defined as neuron output value x _i, relation formula is as follows:

$V_{\mathrm{Init}}^{*}\left(s_i\right)\&LeftArrow;x_i$

(c) according to following regular initialization Q value: $Q_{\mathrm{Init}}(s_i,a)=r+{\mathrm{\&gamma;V}}_{\mathrm{Init}}^{*}\left(s_j\right),$

(d) observe current state s _t;

(e) continue to explore in complex environment, at current state s _taction a of lower selection _tand carry out, ambient condition is more

New is new state s' _t, and r is returned in reception immediately _t;

(f) observe new state s' _t;

(g) according to following formula, upgrade list item Q (s _t, a _t) value:

$Q_t(s_t,a_t)=(1-{\mathrm{\&alpha;}}_t)Q_{t-1}(s_t,a_t)+{\mathrm{\&alpha;}}_t(r_t+\mathrm{\&gamma;}\underset{a_t^{\&prime;}}{\arg \max }{Q}_{t-1}(s_t^{\&prime;},a_t^{\&prime;}))$

In formula, α _tfor learning rate, value is 0.5, and decays with learning process; Q _t-1(s _t, a _t) and Q _t-1(s' _t, a' _t) state of being respectively-action is to (s _t, a _t) and (s' _t, a' _t) at t-1 value constantly, a' _tfor at new state s' _tthe action of lower selection;

(h) judge whether robot has arrived the maximum study number of times that target or learning system have reached setting, the maximum study number of times of setting should guarantee that learning system restrains in maximum study number of times, if both meet one, study finishes, otherwise turns back to step (d) continue studying.

Images