JP3703821B2 - Parallel learning device, parallel learning method, and parallel learning program - Google Patents

Description

  The present invention relates to a parallel learning device, a parallel learning method, and a parallel learning program for learning an action policy for achieving a given task.

  Minsky, as well as human society, moves the human mind as various agents collaborate and compete with each other, see intelligence as a collection of simple agents, and as a result of the interaction between agents, It is advocated that it generates behavior. This idea is also attracting attention in the field of computational neuroscience, and even in the study of motor procedure learning, multiple learning modules can simultaneously learn in different coordinate systems and contribute to learning of each series. It is suggested that

Also, several methods for preparing a plurality of learning devices and switching them have already been proposed for a task of learning complex behavior using reinforcement learning. For example, a method of switching a plurality of learners according to a TD error (see Non-Patent Document 1) or a module in which a prediction model to be controlled and a reinforcement learner are paired is used in parallel, and the prediction error of the prediction model is used. A method of switching and combining based on (see Non-Patent Document 2) has been proposed.
SP Singh, “Transfer of learning by composing solutions of elemental sequential tasks”, Machine Learning, 1992, vol. 3, p. 9-p. 44 K. Doya et al., “Multiple Model-Based Reinforcement Learning”, Neural Computation, 2002, vol. 14, p. 1347-p. 1369

  However, in the above conventional method, since each learning device has the same structure and uses the same learning method, the learning efficiency of the entire learning device is not different from the case of learning with one learning device, and there are a plurality of learning devices. It is not possible to train the learners efficiently.

  An object of the present invention is to provide a parallel learning apparatus, a parallel learning method, and a parallel learning program capable of significantly reducing a learning time until a structure suitable for a task is acquired by efficiently learning a plurality of learning means. Is to provide.

A parallel learning device according to the present invention is a parallel learning device that learns an action policy for achieving a given task, and includes an acquisition unit that acquires an external state, and an external state acquired by the acquisition unit. select a plurality of learning means, one of the action policy based on the learning performance of each learning means from among a plurality of action policy in which a plurality of learning means has determined that learns to determine the action policy from a result of learning based Selecting means for selecting one learning means based on the learning performance of each learning means from among the plurality of learning means, the acquiring means acquiring the state of the outside world, and the plurality of learning means Each learning at the same time as other learning means based on the state of the external world acquired by the acquisition means, determining an action policy from the learning result, and the action determined by the selection learning means selected by the selection means Output the strategy, Each of the learning means weights the parameters used for learning according to the similarity between the action policy determined by the learning means and the action policy of one learning means selected by the selection means, using the importance sampling method By repeating the above, the process of correcting the parameters used for learning is repeated .

  In the parallel learning device according to the present invention, the state of the outside world is acquired, a plurality of learning means learn simultaneously based on the acquired state of the outside world, the action policy is determined from the learning result, and the plurality of determined actions One action policy is selected from the policies based on the learning performance of each learning means, and an action according to the selected action policy is executed.

  By repeating the above process, other learning means that have not been selected are also learned from the experience obtained by the action policy determined by the selected learning means, and multiple action strategies for achieving the task are learned. Since the means can learn at the same time, a plurality of learners can be efficiently learned, and the learning time until a structure suitable for a task is acquired can be greatly shortened.

When there is one learning means with the highest learning performance among the plurality of learning means, the selection means selects this learning means, and there are a plurality of learning means with high learning performance, and the learning performance of these learning means is When it is within the predetermined range, it is preferable to select one learning means from these learning means with equal probability . In this case, since one learning means can be selected probabilistically from learning means whose learning performance is within the predetermined range, a plurality of learning means can be efficiently learned.

  Each of the plurality of learning means is preferably different from other learning means in at least one of state expression and learning method. In this case, since learning can be performed using a plurality of learning means having different learning characteristics, for example, the data quickly collected by the learning means having a simple configuration can be used for the learning means having a complicated configuration. The learning speed can be improved and the learning performance can be improved.

  Each of the plurality of learning means includes a calculation means for calculating a value function for evaluating learning performance using a predetermined parameter based on a state of the external world acquired by the acquisition means, and an external environment acquired by the acquisition means Determining means for determining an action policy based on the status function and the value function calculated by the calculating means; an external state acquired by the acquiring means; an action policy determined by the determining means; and an action policy selected by the selecting means It is preferable to include a correction unit that corrects the parameter of the calculation unit based on the above.

  In this case, the parameters used for calculating the value function are corrected based on the acquired external state, the action policy determined based on the external state and the value function, and the selected action policy. Therefore, other learning means that are not selected can be learned from the experience obtained by the action policy determined by the selected learning means.

  Preferably, at least one learning means of the plurality of learning means further includes a storage means for storing the action policy determined by the determination means. In this case, since the learning means includes the storage means, it is possible to handle the partial observation Markov decision problem.

Parallel learning method according to the present invention, the acquisition means, a parallel learning method using a parallel learning device comprising a plurality of learning means and selection means learns the action policy for accomplishing a given task, selecting Each of the selection step in which the means selects one learning means from the plurality of learning means based on the learning performance of each learning means, the acquisition step in which the acquisition means acquires the state of the outside world, and each of the plurality of learning means However, the learning step of learning simultaneously with other learning means based on the state of the external world acquired in the acquisition step, and determining the action policy from the learning result, and the selection means by one learning means selected in the selection step The step of outputting the determined action policy, and each of the plurality of learning means uses the importance sampling method to the action policy and selection step determined by the learning means. By performing weighting parameters used for learning in accordance with the degree of similarity between the action policy of one learning means selected Te, but repeating the step of correcting the parameters used for learning.

A parallel learning program according to the present invention is a parallel learning program for learning an action policy for achieving a given task, and includes an acquisition unit that acquires an external state, and an external environment acquired by the acquisition unit. learns based on the state, a plurality of learning means for determining the action policy from a result of learning, one action policy based from among a plurality of action policy in which a plurality of learning means has determined the learning performance of each learning means The selection unit selects one learning unit based on the learning performance of each learning unit from the plurality of learning units, the acquisition unit acquires the external state, Each of the plurality of learning means learns simultaneously with other learning means based on the state of the external world acquired by the acquisition means, determines an action policy from the learning result, and the selection means selects one learning The action policy determined by the step is output, and each of the plurality of learning means uses the importance sampling method to determine the action policy determined by the learning means and the action policy of the one learning means selected by the selection means. The process of correcting the parameters used for learning is repeated by weighting the parameters used for learning according to the degree of similarity .

  According to the present invention, other learning means that are not selected are also learned from the experience obtained by the action policy determined by the selected learning means, and the plurality of learning means simultaneously execute the action policy for achieving the task. Since learning can be performed, a plurality of learners can be efficiently learned, and the learning time until a structure suitable for a task is acquired can be greatly shortened.

  Hereinafter, a parallel learning apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a learning system using a parallel learning device according to an embodiment of the present invention.

  The learning system shown in FIG. 1 includes a sensor unit 1, a parallel learning device 2, and an actuator unit 3. The parallel learning device 2 includes a normal microcomputer including a ROM (read only memory), a CPU (central processing unit), a RAM (random access memory), an A / D (analog / digital) converter, a D / A ( The state acquisition unit 11, the probabilistic selector 12, the switch 13, and the n learning devices 21 to 2n are configured by a CPU and a parallel learning program that is configured by a digital / analog) converter and stored in the ROM. Learn how to act and act to achieve a given task.

  The sensor unit 1 is composed of various sensors that detect the state of the outside world, and the actuator unit 3 is composed of various actuators that execute a predetermined action according to an action policy for achieving a given task. . For example, when the learning system is configured as an autonomous traveling robot, the sensor unit 1 can be a camera that captures an image of the outside world, a distance sensor that detects a moving distance, a wheel rotation amount, and an amount of movement from an initial position. Odometry to be calculated can be used, and as the actuator unit 3, wheels and motors for moving in an arbitrary direction can be used.

  The sensor unit 1 detects the state of the outside world and outputs it to the state acquisition unit 11. The state acquisition unit 11 acquires the state of the outside world detected by the sensor unit 1 and outputs it to the n learning devices 21 to 2n. Each of the learning devices 21 to 2n includes a corrector 31, a value function unit 32, and a controller 33. However, the specific structures of the value function unit 32 and the controller 33 are different from each other, and at least one of the state expression and the learning method is different from other learning devices. Each of the learning devices 21 to 2n learns simultaneously based on the acquired state of the outside world, determines an action policy from the learned result, and outputs the action policy to the switching device 13.

  Each of the learning devices 21 to 2n performs weighting according to the degree of similarity between the action policy determined by itself and the action policy output from the switcher 13, and corrects the parameters used for learning. Here, each of the learning devices 21 to 2n is preferably weighted using an importance sampling method described later.

  The value function unit 32 calculates a value function for evaluating the learning performance based on the state of the external world from the state acquisition unit 11 using predetermined parameters, and the calculated value function is calculated by the controller 33 and the stochastic selector. 12 is output. The controller 33 determines an action policy based on the state of the outside world from the state acquisition unit 11 and the value function calculated by the value function unit 32, and outputs the determined action policy to the switcher 13. The corrector 31 reads the currently set parameters from the value function unit 32, selects the external state from the state acquisition unit 11, the action policy determined by the controller 33, and the stochastic selector 12 and the switch 13. The parameters of the value function unit 32 are corrected based on the learned learning device action policy, and the parameters are updated.

  The probabilistic selector 12 acquires the value function from the value function unit 32 of each of the learners 21 to 2n, and determines the optimum action policy from the plurality of learners 21 to 2n based on the acquired value function. The operation of the switcher 13 is controlled so as to select the learning device. For example, if there is one learner with the highest learning performance, the stochastic selector 12 selects an action policy of this learner, and there are a plurality of learners with high learning performance, and the learning performance of these learners is When it is within the predetermined range, the operation of the switcher 13 is controlled so that one action policy is selected from the action policies of these learning devices with a substantially equal probability.

  The switcher 13 selects the action policy of the learning device instructed by the probabilistic selector 12 from the plurality of action policies, outputs the selected action policy to each of the learners 21 to 2n, and selects the selected action policy. The actuator unit 3 is caused to execute an action according to the above. The actuator unit 3 executes an action according to the selected action policy. By this action, the state of the outside world changes. By detecting this change by the sensor unit 1 and repeating the above processing, the plurality of learning devices 21 to 2n learn simultaneously.

For example, a plurality of learners M ⁱ (i = 1,..., N) learn a control policy π ⁱ for achieving a given task using a value function method or a policy gradient method, and each learner M ^When the state value function of ⁱ is V ⁱ , the parallel learning device 2 selects the learning device M ⁱ according to the following probability based on the initial observation x ₀ for each episode.

Here, T _Sel is a parameter for controlling the randomness of the selection probability, and if it is large, there is a tendency to select a learning device at random. The action policy of the selected learning device is referred to as a behavior policy and is denoted as π ^Beh . The parallel learning device 2 evaluates the target policy of each learner M ⁱ using episodes obtained using π ^Beh .

  The configuration of the learning device is not particularly limited to the above example, and various modifications are possible. For example, the following external memory may be added. In this case, a partially observable Markov decision process (POMDP: Partially Observable Markov Decision Process) can be handled.

  FIG. 2 is a block diagram showing another configuration of the learning device. The learning device 21a shown in FIG. 2 differs from the learning devices 21 to 2n shown in FIG. 1 in that an external memory 34 is added, and the corrector 31, the value function unit 32, and the controller 33 are the corrector 31a and the value function unit. The points that have been changed to 32a and the controller 33a will be described in detail below.

  The external memory 34 has a storage capacity of 1 bit, stores the action policy determined by the controller 33a, and outputs the stored action policy to the corrector 31a, the value function unit 32a, and the controller 33a. In addition, the external memory 34 receives the external state from the state acquisition unit 11 and can record the external state. The value function unit 32a calculates a value function for evaluating learning performance based on the state of the external world from the state acquisition unit 11 and the action policy from the external memory 34 using predetermined parameters, and calculates the calculated value function. It outputs to the controller 33a and the stochastic selector 12. The controller 33a determines the action policy based on the external state from the state acquisition unit 11, the action policy from the external memory 34, and the value function calculated by the value function unit 32a, and the determined action policy is switched to the switch 13. Output to. The corrector 31a reads the currently set parameters from the value function unit 32a, the state of the external world from the state acquisition unit 11, the action policy from the external memory 34, the action policy determined by the controller 33a, and the switcher 13. The parameters of the value function unit 32a are corrected by updating the parameters of the value function unit 32a based on the action policy of the learning device that is output from.

With the above configuration, the learning unit 21a, in addition to the state o _t of the resulting environmental by the sensor unit 1 at time t, can use the information m _t of the external memory 34, action policy a _t the controller 33a is actually and action output a ^e _t to the environment by the actuator unit 3 to cause a state transition, and a action policy a ^m _t to manipulate the memory bit.

In this case, the observed amount _{x t} to be used in the learning unit 21a is expressed in combination with information _{m t} of the state of the environment _{o t} and the external memory _34, x t _{= (o} t, _{m t)} becomes. Since each bit of the external memory 34 takes 1 or 0, the information m _t is 2 ^{l in} total. Further, since the action policy a ^m _t has an action of setting each bit of the external memory 34 to 1 and an action of setting it to 0, the total number of actions is 2l. ^{Incidentally,} one of the combinations can also be a behavioral measures _{a t} learners 21a, similar primitives and the ^a _{m t} ^a _{e t} in order to suppress the complexity behavior of ^a _{e t} and ^a _{m t} You may make it add as.

  In the present embodiment, the state acquisition unit 11 corresponds to an example of an acquisition unit, the learning devices 21 to 2n and 21a correspond to an example of a learning unit, and the probabilistic selector 12 and the switch 13 are an example of a selection unit. The value function units 32 and 32a correspond to an example of a calculation unit, the controllers 33 and 33a correspond to an example of a determination unit, and the correctors 31 and 31a correspond to an example of a correction unit. The external memory 34 corresponds to an example of a storage unit.

  Next, the parallel learning process of the learning system configured as described above will be described. FIG. 3 is a flowchart for explaining parallel learning processing of the parallel learning apparatus shown in FIG.

First, in step S1, the stochastic selector 12 probabilistically selects one learner based on the learning performance of each of the learners 21 to 2n. Specifically, the stochastic selector 12 selects this learning device when there is one learning device with the highest learning performance, and there are a plurality of learning devices with high learning performance, and the learning performance of these learning devices is If it is within the predetermined range, one learner is selected from these learners with equal probability.

  After the learning device is selected, in step S2, the state acquisition unit 11 acquires the state of the outside world detected by the sensor unit 1, and gives it to the value function unit 32 of each learning device 21 to 2n.

  Next, in step S3, the value function unit 32 of each of the learning devices 21 to 2n calculates a value function based on the state of the external world from the state acquisition unit 11, and outputs the calculated value function to the controller 33. The controller 33 determines an action policy based on the state of the outside world from the state acquisition unit 11 and the value function calculated by the value function unit 32, and outputs the determined action policy to the switcher 13. At this time, the stochastic selector 12 determines the action policy by controlling the switch 13 so that the action policy of the learning device selected in step S1 is output to the actuator unit 3.

  Next, in step S4, the switching unit 13 drives the actuator unit 3 to cause the actuator unit 3 to execute an action in accordance with the behavior policy of the learning device selected by the stochastic selector 12, and the actuator unit 3 selects Execute the action according to the action policy.

  Next, in step S <b> 5, the corrector 31 of each of the learning devices 21 to 2 n reads the current parameters from the value function unit 32, the external state from the state acquisition unit 11, and the action policy determined by the controller 33. And each parameter is correct | amended based on the action policy of the learning device selected by the switch 13, the parameter of the value function part 32 is updated, and the distribution process by an important sampling method is performed.

  Here, the distribution processing by the above-described importance sampling method will be described in detail. In the following description, a case where the learning device 21a having the external memory 34 shown in FIG. 2 is used as the learning devices 21 to 2n will be described as an example.

When the state of the environment at time t is s _t, the parallel learning apparatus 2 receives a part of the sensor unit 1 as o _t, the information of the external memory 34 at that time and m _t, the learner 21~2n The observation amount x _t acquired by x becomes x _t = (o _t , m _t ). At this time, when outputting the action a _t the actuator unit 3 according to action policy [pi, resulting environment is state transition s _{t + 1,} to obtain a scalar reward r _t is the evaluation value. The value V ^π (s) of the state s under the action policy π is given by the following equation.

  Here, R (s) is the revenue observed from the state s, γ is the attenuation rate (0 ≦ γ ≦ 1), and Eπ {} is the expectation when the parallel learning device 2 follows the action policy π. Represents a value. Similarly, the value of executing action a in state s under action policy π is given by:

The above V ^{π is referred} to as a state value function, Q ^π is referred to as an action value function, and both are collectively referred to as a value function. In order to estimate V ^π and Q ^π , consider a case in which another action policy π ′ different from the original action policy π is used. To deal with the difference. Now, let _{m m} the episode obtained by the behavior policy π ′ be h _m , T ^{m be} the time step until the end of episode h _m , and let Pr ^π (h _m ) and Pr ^{π ′} (h _m ) and the probability that the episode h _m occurs when, according to the policy π and π '.

  At this time, the Monte Carlo estimation required after observing M profits is given by the following equation.

Here, R _m is actually obtained profit R _m (s) = r _{tm (s)} + γr _{tm (s) +1} +... + Γ ^{Tm−tm (s) −1} r _Tm−1 and t _m ( s) is a time step when the state s is obtained for the first time in the _mth episode hm. The probability that the episode h _m occurs is given by the following equation.

Here, ρ _t is a coefficient for correcting the difference in action policy, and knowledge of the dynamics of the environment is not required for calculating Pr ^π (h _m ) / Pr ^{π ′} (h _m ), and the ratio of action policies Only needed. If π (s, a)> 0, it is required that π ′ (s, a)> 0.

Next, when the learning devices 21 to 2n use the value function method for reinforcement learning, a method for applying the above-described importance sampling method to the value function method will be specifically described. The value function method is a method of estimating a value Q ^VF defined for a set of state and action using the Bellman equation, and representative methods include Q learning and SARSA. SARSA is policy-on type reinforcement learning, and Q-learning is policy-off type reinforcement learning, which can have a behavior policy and an estimation policy separately.

First, when formulated regarded as state observations, perform an action _{a t} the observation value _{x t,} upon receipt of a reward _{r t} and the next observation value _{x t + 1,} the Q-learning and Sarsa, TD error respectively It is given by the following formula.

Here, Q ^Q and Q ^SARSA are action value functions when Q learning and SARSA are used.

As a method of using the importance sampling method for the value function method, a known method can be used. In this embodiment, the value function is weighted in the form of a lookup table, that is, w _k = Q (x, a). Is assigned, and the action value function in the case of using importance sampling is given by the following equation.

  Here, when the Markov property of the environment is used as in SARSA, the update formula is given by the following formula.

Here, t _m is the time when (x _t , a _t ) = (x, a) first in the m-th episode, T ^VF is a fitness trace, and λ is the fitness decay rate. Yes, α _VF is the learning rate. If the behavior policy and the target policy match, ρ _t = 1, which is a normal SARSA update formula.

  Further, the stochastic action policy is expressed by the following equation using, for example, a Boltzmann distribution.

Here, _TVF is a temperature parameter, and takes a large value in the initial stage of learning, but is controlled so as to take a small value as the learning proceeds. Value function method, environment Markov Decision Process: If a (MDP Markov Decision Process), i.e. in the case of x _{t =} s _t are shown convergence to the optimal policy. Even in a POMDP environment, an optimal probabilistic policy can be obtained by appropriately setting λ within a range that does not have internal variables.

  Next, when the learning devices 21 to 2n use the policy gradient method for reinforcement learning, a method for applying the above-described importance sampling method to the policy gradient method will be specifically described. Conventionally, there has been proposed a method for updating parameters in a gradient direction of an expected value of a reward in a problem with a delay in reward, and various policy gradient methods have been proposed as a trigger.

First, the action policy π ^PG expressed by the parameter w _{k is} improved by the following equation using the gradient of the value function V ^PG in which the expected value is taken as x.

Here, α _PG is a step size parameter, and w is a parameter vector in which w _k are collected. At this time, when the importance sampling method is used, the state value function is given by the following equation.

Here, Pr (h m | _w) is the probability of obtaining an episode h _m with action policy parameterized by the vector w, it is represented by the following expression.

Here, φ (h _m ) and ψ (w, h _m ) are given by the following equations.

The above φ (h _m ) must be sampled from the environment, but Ψ (w, h _m ) can be calculated from the action policy of the parallel learning device 2, so when one episode is obtained, the action policy is improved the direction is as shown in the following equation by differentiating V the (w) at w _k.

  The above Pr (h | w) = Pr (h | w ') can be calculated by multiplying the ratio of the control policy, and the update formula when the policy gradient method is used is given by the following formula.

Here, T _t (k) is a goodness-of-fit trace in the case of the policy gradient method, and ρ _t = 1 when the behavior policy matches the target policy.

Next, in the policy gradient method, it is necessary to express the action policy as a parameter. However, weights are assigned to a set of states and actions as w _k = P (x, a), and Boltzmann as shown in Expression (13). It is expressed by the following formula using the distribution.

Here, P ^PG (x _t , a _t ) is not an action value, and T _PG is a temperature parameter, but takes a constant value unlike the equation (13). At this time, the derivative of the equation (23) is given by the following equation.

In the above policy gradient method, the value function is not explicitly estimated, and the policy is updated online. However, in the present invention, the value function needs to be used to select the learner at the beginning of the episode, and the formula ( The value V ^PG is updated for each episode according to 4).

  Referring to FIG. 3 again, after the distribution processing by the above-described importance sampling method is performed, in step S6, each of the learning devices 21 to 2n determines whether or not the task currently being executed has ended. If the task has not ended, the processes in and after step S2 are repeated. If the task has ended, the process proceeds to step S7.

  When the task is finished, in step S7, the stochastic selector 12 can acquire whether or not the learning is finished for the given task, that is, the learning performance required for the given task. If the learning has not been completed, the process from step S1 is repeated, and the process is terminated when the learning is completed.

  According to the above processing, in the present embodiment, the state acquisition unit 11 acquires the state of the outside world, and the learning devices 21 to 2n simultaneously learn based on the acquired state of the outside world, and the action policy is determined from the learning result. One action policy is selected based on the learning performance of each of the learners 21 to 2n by the probabilistic selector 12 and the switch 13 from among the plurality of determined action policies, and the action according to the selected action policy Is executed by the actuator unit 3, and these processes are repeated. As a result, other learners that have not been selected are also learned from the experience obtained by the action policy determined by the selected learner, and a plurality of learners 21 to 2n are actions for achieving a given task. Since the strategies can be learned at the same time, the plurality of learners 21 to 2n can be efficiently learned, and the learning time until the learners 21 to 2n acquire a structure suitable for the task is greatly reduced. be able to.

  Next, the learning effect of the parallel learning apparatus will be described with a specific example. FIG. 4 is a characteristic diagram showing learning performance when the parallel learning device shown in FIG. 1 is used for controlling an inverted pendulum. The example shown in FIG. 4 controls the movement of the carriage so that the pole provided on the carriage stands upright, and only the position x of the carriage and the angle θ of the pole, which are part of the state variables, can be observed. This is an example in the case of POMDP. Here, the vertical axis in FIG. 4 indicates the total reward for each episode representing learning performance, and the horizontal axis indicates the number of episodes.

  A curve A shown in FIG. 4 represents learning performance when the parallel learning device shown in FIG. 1 is used. As the learning devices 21 to 2n, a learning device using the value function method and not having the external memory 34, the value function method. And a learning device that uses the policy gradient method and does not have the external memory 34, and a learning device that uses the policy gradient method and has the external memory 34, and uses the importance sampling method. The learning performance is shown when learning is performed simultaneously for each learning device.

  On the other hand, the curves B to F are comparative examples, the curve B represents the learning performance when only the learning device using the value function method and not having the external memory 34 is used, and the curve C uses the value function method. The curve D represents the learning performance when only the learning device having the external memory 34 is used, and the curve D represents the learning performance when only the learning device using the policy gradient method and not having the external memory 34 is used. Represents learning performance when only the learning device using the policy gradient method and having the external memory 34 is used, and the curve F represents learning when four learning devices are simultaneously learned without using the importance sampling method. Represents performance.

  From FIG. 4, even when the environment is POMDP, when the parallel learning device shown in FIG. 1 is used (curve A), the learning efficiency is the highest compared to other learning methods (curves BF), and the learning time is reduced. It was found that the learning performance can be shortened most and the reachable learning performance is the highest.

  FIG. 5 is a characteristic diagram showing learning performance when the parallel learning device shown in FIG. 1 is used for traveling control of an autonomous traveling robot. In the example shown in FIG. 5, the autonomous mobile robot reaches the target position while avoiding an obstacle. The vertical axis in FIG. 5 indicates the average reward indicating the learning performance, and the horizontal axis indicates the number of episodes.

  A curve A shown in FIG. 5 represents learning performance when the parallel learning apparatus shown in FIG. 1 is used. As learning devices 21 to 2n, a learning device that performs coarse movement control using the value function method, a value function method is shown. Using a learning device that performs precise movement control using a learning device, a learning device that performs coarse movement control using a policy gradient method, and a learning device that performs precise movement control using a policy gradient method, and using an importance sampling method, 4 The learning performance is shown when learning is performed simultaneously for each learning device.

  On the other hand, curves B and C are comparative examples, and curve B represents learning performance when only a learning device that performs coarse movement control using the value function method is used, and curve C uses value function method. The learning performance when only a learning device that performs precise movement control is used is shown.

  From FIG. 5, when the parallel learning apparatus shown in FIG. 1 is used for an autonomous traveling robot (curve A), the learning efficiency increases sharply as the number of episodes increases compared to other learning methods (curves B and C). It has been found that the learning performance can be shortened most and the learning performance that can be reached is the highest.

  In the above-described embodiment, the autonomous traveling robot or the like has been described. However, the application target of the present invention is not particularly limited to the above example, and can be applied to various types. For example, the parallel learning device of the present invention may be applied to a pet robot or the like, and a human teaching may be introduced as one of a plurality of learning devices. In this case, the pet robot can act as taught by a human and learning of the pet robot itself can be realized at the same time.For example, the owner teaches the pet robot a trick and performs more intelligent behavior by autonomous learning. Can be earned.

  In addition, the parallel learning device of the present invention is applied to the optimal control field, etc. to combine conventional control and machine learning, and what has been used for manipulator control etc. in factories etc. is used as the controller of the learning device You may make it do. In this case, since what has been used so far can be used as it is, it is possible to automatically use better performance acquired by other learning devices while guaranteeing the same performance as before.

  Furthermore, the parallel learning device of the present invention may be applied to a portion that evaluates a large number of learning devices such as evolution robotics. In this field, since a plurality of controllers were evaluated one by one in order, enormous time was required. However, by using the parallel learning device of the present invention, it is possible to evaluate a plurality of learners in parallel. This can greatly reduce the evaluation time.

It is a block diagram which shows the structure of the learning system using the parallel learning apparatus by one embodiment of this invention. It is a block diagram which shows the other structure of a learning device. It is a flowchart for demonstrating the parallel learning process of the parallel learning apparatus shown in FIG. It is a characteristic view showing learning performance at the time of using the parallel learning device shown in Drawing 1 for control of an inverted pendulum. It is a characteristic view showing learning performance at the time of using the parallel learning device shown in Drawing 1 for run control of an autonomous running robot.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Sensor part 2 Parallel learning apparatus 3 Actuator part 11 State acquisition part 12 Probabilistic selector 13 Switch 21-21n, 21a Learner 31, 31a Corrector 32, 32a Value function part 33, 33a Controller 34 External memory

Claims (6)

A parallel learning device that learns action strategies to achieve a given task,
Acquisition means for acquiring the state of the outside world;
A plurality of learning means learns, determines the action policy from a result of learning based on the state of the outside world acquired by the acquisition means,
Selecting means for selecting one action policy based on the learning performance of each learning means from a plurality of action policies determined by the plurality of learning means ,
The selection unit selects one learning unit from the plurality of learning units based on the learning performance of each learning unit, the acquisition unit acquires an external state, and each of the plurality of learning units , Learning simultaneously with other learning means based on the state of the outside world acquired by the acquisition means, determining an action policy from the learning result, and the action determined by the selection learning means selected by the selection means A policy is output, and each of the plurality of learning means uses an importance sampling method according to the similarity between the action policy determined by the learning means and the action policy of one learning means selected by the selection means A parallel learning apparatus that repeats the process of correcting parameters used for learning by weighting parameters used for learning. The selection means selects the learning means when there is one learning means with the highest learning performance from the plurality of learning means, and there are a plurality of learning means with high learning performance and learning of these learning means. 2. The parallel learning apparatus according to claim 1, wherein when the performance is within a predetermined range, one learning means is selected from these learning means with equal probability. Each of the plurality of learning means includes
Calculation means for calculating a value function for evaluating learning performance using a predetermined parameter based on the state of the external world acquired by the acquisition means;
Determining means for determining an action policy based on the state of the outside world acquired by the acquiring means and the value function calculated by the calculating means;
A correction unit that corrects the parameter of the calculation unit based on the state of the outside world acquired by the acquisition unit, the action policy determined by the determination unit, and the action policy selected by the selection unit; The parallel learning apparatus according to claim 1 or 2. 4. The parallel learning apparatus according to claim 3, wherein at least one learning means of the plurality of learning means further includes a storage means for storing the action policy determined by the determination means. A parallel learning method for learning an action policy for achieving a given task using a parallel learning device comprising an acquisition means, a plurality of learning means and a selection means,
A selection step in which the selection means selects one learning means based on the learning performance of each learning means from the plurality of learning means;
The obtaining means for obtaining an external state; and
A learning step in which each of the plurality of learning means learns simultaneously with other learning means based on the state of the outside world acquired in the acquisition step, and determines an action policy from the learning result;
The selection means outputting the action policy determined by the learning means selected in the selection step;
Parameters used for learning by each of the plurality of learning means according to the degree of similarity between the action policy determined by the learning means and the action policy of one learning means selected in the selection step by using an importance sampling method A parallel learning method characterized by repeating the step of correcting parameters used for learning by weighting. A parallel learning program for learning action strategies to accomplish a given task,
An acquisition means for acquiring the state of the outside world;
Learning based on the state of the outside world acquired by the acquisition means, a plurality of learning means for determining an action policy from the learning results;
Causing the computer to function as a selection means for selecting one action policy based on the learning performance of each learning means from the plurality of action policies determined by the plurality of learning means;
The selection unit selects one learning unit from the plurality of learning units based on the learning performance of each learning unit, the acquisition unit acquires an external state, and each of the plurality of learning units , Learning simultaneously with other learning means based on the state of the outside world acquired by the acquisition means, determining an action policy from the learning result, and the action determined by the selection learning means selected by the selection means A policy is output, and each of the plurality of learning means uses an importance sampling method according to the similarity between the action policy determined by the learning means and the action policy of one learning means selected by the selection means A parallel learning program characterized by repeating processing for correcting parameters used for learning by weighting parameters used for learning.