WO2024224501A1 - Action determination method and action determination device - Google Patents

Abstract

This action determination method includes an action determination step, a state change step, and a learning step. In the action determination step, an optimization calculation is executed on the basis of a policy represented by a model applicable to the optimization calculation, and the action of an agent is determined. In the state change step, the action is input into an actual environment, whereby the actual environment and the agent interact, and the state of the agent changes. In the learning step, a value for the state of the agent after the action is acquired as a reward, and the next policy is determined on the basis of the reward. The optimization calculation can modify randomness by using a calculation parameter. The calculation parameter is modified on the basis of a change amount of the actual environment between the learning step and the action determination step.

Description

Behavior determination method and behavior determination device

The present invention relates to a behavior decision-making method and a behavior decision-making device.

Machine learning can be broadly divided into supervised learning, unsupervised learning, and reinforcement learning. Reinforcement learning is said to be suitable for learning autonomous driving and behavior optimization, and is attracting attention.

Reinforcement learning is a learning method in which an agent executes a task by repeatedly interacting with the environment and through trial and error. Interaction refers to the sending and receiving of information between the agent and the environment.

In reinforcement learning, the agent and the environment send and receive information on state, action, and reward. The state is the situation the agent is in, and the action is the agent's behavior. The reward is an evaluation index for the agent's state after the action. The agent inputs actions based on the policy to the environment. The environment inputs the state and reward to the agent according to the agent's actions. The agent and environment repeatedly interact through trial and error to maximize the reward.

For example, Patent Document 1 discloses a reinforcement learning method that applies optimization calculations to the reward function.

Patent No. 7111178

In the reinforcement learning method described in Patent Document 1, the policy is expressed by a probability distribution function (π(a|s)=exp(r _a (s))/Z _R ) with the distribution function Z _R as the denominator. The agent decides on an action according to the policy. If the policy is specified by a probability distribution, the randomness of the action decision is uniquely determined. When the real environment does not change, there is no problem in deciding the action according to the learned policy, but when the real environment changes, the action is decided based on the policy learned in the real environment before the change. In this case, the action decision in the real environment after the change is made based on the policy learned in the real environment before the change, and the validity of the action decision may be reduced. Therefore, the reinforcement learning method described in Patent Document 1 needs to re-learn every time the real environment changes, and cannot adequately respond to changes in the real environment.

The present invention has been made in consideration of the above circumstances, and aims to provide a behavior decision-making method and device that can increase the probability of selecting an action that is not truly optimal during learning by changing the randomness of behavior decisions in response to changes in the real environment, and can increase the possibility of selecting an action that is appropriate for a changed environment, even if the environment changes.

To solve the above problems, the present invention provides the following means.

The behavior decision method according to the first aspect includes a behavior decision step, a state change step, and a learning step. In the behavior decision step, an optimization calculation is performed based on a measure represented by a model applicable to the optimization calculation, and an agent's behavior is decided. In the state change step, the behavior is input into a real environment, causing an interaction between the real environment and the agent, and changing the state of the agent. In the learning step, the value of the agent's state after the behavior is obtained as a reward, and the next measure is decided based on the reward. The optimization calculation can change randomness by a calculation parameter. The calculation parameter is changed based on the amount of change in the real environment between the learning step and the behavior decision step.

The behavior decision device according to the second aspect has a learning unit, a first control unit, and a second control unit. The first control unit executes an optimization calculation and decides an agent's behavior based on a measure represented by a model applicable to the optimization calculation. The second control unit inputs the behavior into a real environment and changes the state of the agent. The learning unit obtains the state of the agent after the behavior as a reward, and decides the next measure based on the reward. The optimization calculation can change randomness by a calculation parameter. The calculation parameter is changed based on the amount of change in the real environment between when the agent's behavior is decided and when the measure is decided.

The behavior decision-making method and behavior decision-making device of the present invention can be adapted to environmental changes by changing the randomness of behavior decisions in response to changes in the environment.

FIG. 2 is a conceptual diagram of reinforcement learning that is a component of the behavior decision-making method according to the first embodiment. FIG. 2 is a conceptual diagram showing state changes in reinforcement learning that is a part of the behavior decision-making method according to the first embodiment. This is an illustration of the probability distribution of solutions obtained by optimization calculations using quantum annealing. This is an image of the probability distribution of actions selected by expressing the policy as an optimization calculation model and performing optimization calculations. This is an image diagram showing the relationship between the action selected by an agent based on a policy and changes in the real environment. FIG. 2 is an example of a flow diagram of reinforcement learning according to the first embodiment. 1 is a block diagram of a behavior determination device according to a first embodiment.

The present embodiment will be described in detail below with reference to the drawings as appropriate. The drawings used in the following description may show enlarged characteristic parts for the sake of convenience in order to make the features of the present embodiment easier to understand, and the dimensional ratios of each component may differ from the actual ones. The materials, dimensions, etc. exemplified in the following description are merely examples, and the present invention is not limited to them, and may be modified as appropriate within the scope of the present invention.

FIG. 1 is a conceptual diagram of reinforcement learning that carries out the behavior decision-making method according to the first embodiment. In reinforcement learning, an agent 1 and a real environment 2 learn by interacting with each other, and a reward r is obtained according to a state S _a of the agent 1 after an action a. The action a of the agent 1 is determined according to a policy π, and the agent 1 performs the determined action a. The action a of the agent 1 can be represented, for example, by an action vector (a ¹ , a ² , a ³ , ..., a ⁿ ). The state S _a of the agent 1 changes depending on the action a. The state S _a of the agent 1 can be represented, for example, by a state vector (S _a0 , S _a1 , S _a2 , ..., S _an ).

For example, a specific explanation will be given using the control of a robot arm as an example. In this case, agent 1 is the control unit of the robot arm. The real environment 2 is the environment in which the robot arm operates. The real environment 2 is, for example, the temperature and humidity at which the robot arm operates, the wear state of parts, the material of the floor on which the robot arm operates, the movement of adjacent robot arms, etc.

For example, if a robot arm operates by passing a current through n conductors connected to the robot arm, then action a of agent 1 corresponds to the action of passing a current through the conductors. For example, one action a of the robot arm is determined by specifying how many A of current should be passed through each of the n conductors.

Each element of the action vector (a ¹ , a ² , a ³ , ..., a ⁿ ) corresponds to an option of how many amperes to pass through the l-th conductor (l=an integer from 1 to n). For example, a ¹ is an option of how many amperes to pass through the first conductor, and a ² is an option of how many amperes to pass through the second conductor. For example, if there are Kn options for the current to pass through the l-th conductor, there are options to apply any one of the currents {I ^l ₁ , I ^l ₂ , ..., I ^l _Kl } to the l-th conductor.

The policy π that determines the behavior of agent 1 is, for example, a guideline for controlling a robot arm. Based on the guideline for controlling the robot arm (policy π), the control of the robot arm (action a) is determined.

When the agent 1 performs an action a, the state S _a of the agent 1 changes. The state S _a of the agent 1 is, for example, the angle of the joint of a robot arm. For example, when a current is applied to each of n conductors connected to the robot arm, the angle of the joint of the robot arm changes, and the state S _a changes.

The reward r is, for example, a value obtained by changing the state S _a of the robot arm. For example, if the precise control of the robot arm is to bend the arm by 30 degrees, the closer the state S _a of the robot arm caused by the action a of the agent 1 is to a state bent by 30 degrees, the higher the reward r will be.

FIG. 2 is a conceptual diagram showing state changes in reinforcement learning, which is the behavior decision-making method according to the first embodiment. In reinforcement learning, the state of agent 1 transitions as a result of learning.

Each of S _a0 , S _a1 , S _a2 , S _a3 , and S _a4 in FIG. 2 represents a state of the agent 1. The state of the agent 1 is selected by applying an optimization calculation to the policy π from the initial state S _a0 , and transitions to, for example, any one of S _a0 , S _a1 , S _a2 , and S _a3 according to the result. For example, in the case of control of a robot arm, transitions from the initial state S _a0 in which the joints are not bent to any one of the next states S _a0 , S _a1 , S _a2 , and S _a3 . The next state may remain the original state S _a0 in which the joints are not bent, or may be any one of the other states S _a1 , S _a2 , and S _a3 in which the joints are bent. The state to which the agent transitions is determined based on the policy π. For example, when transitioning to the state S _a2 in which the joints are bent, transitions to another state S _a0 , S _a3 , and S _a4 depending on the next action. In reinforcement learning, for example, a reward r is calculated for each action a that produces a state transition. Alternatively, the reward r may be calculated after selecting an action and executing the action multiple times according to the same policy (for each episode).

In this embodiment, the policy π is represented by a model that can be calculated by optimization. Then, an action a is determined based on the policy π by performing an optimization calculation. The randomness of the optimization calculation can be changed by calculation parameters. Although details will be described later, by determining an action a based on the policy π using an optimization calculation that can change the randomness by calculation parameters, even if the real environment 2 has changed (fluctuated) since the learning that determined the policy π was performed, the possibility of selecting an action a that matches the environment increases.

The optimization calculations may be performed, for example, using adiabatic quantum computing performed on a quantum computer, quantum annealing, or a genetic algorithm.

For example, when the optimization calculation is performed using quantum annealing, an Ising model or QUBO can be used as a model applicable to the optimization calculation. Furthermore, when the optimization calculation is performed using adiabatic quantum computing, any Hermitian matrix can be used as a model applicable to the optimization calculation. Furthermore, when the optimization calculation is performed using a genetic algorithm, any real-valued function can be used as a model applicable to the optimization calculation.

For example, when the policy π is expressed by the Ising model, the policy π is expressed by the following equation (1).

Here, π(a|S _a ) is an optimization calculation model, and represents the relationship between each element of the behavior vector a. _{σ i} and σ _j are input variables, and each have two values, +1 or -1. h _ij is an interaction parameter. h _ij is expressed, for example, as a function of the state S _a of agent 1. When the options are expressed in one-hot representation, N is expressed as N=K ₁ +K ₂ +...+K _n . _{K l} is the number of options for the amount of current that can be applied to the lth conductor (l=a natural number from 1 to n).

In optimization calculations using quantum annealing, the probability of finding the true optimal value can vary depending on the calculation time. For example, the optimal value found after performing calculations over a nearly infinite period of time will not necessarily be the same as the optimal value found after performing calculations over a short period of time.

3 is an image diagram of the probability distribution of a solution obtained by an optimization calculation using quantum annealing. (a) in FIG. 3 is a probability distribution of a solution output after performing an optimization calculation for a long time, and (b) is a probability distribution of a solution output after performing an optimization calculation for a short time. The true optimal value is v ₀ , and the probability distribution is expressed, for example, by a normal distribution.

When the optimization calculation using quantum annealing is performed for a long time, the probability of selecting the true optimal value _v0 increases, and the probability of outputting a value _v1 other than the true optimal value _v0 as the optimal value decreases. In contrast, when the optimization calculation time using quantum annealing is short, the probability of selecting the true optimal value _v0 decreases, and the probability of outputting a value _v1 other than the true optimal value _v0 as the optimal value increases. This is because the short calculation time in quantum annealing is generally treated as noise. In a short calculation, a value _v1 other than the true optimal value _v0 may be output as the optimal value. In a general optimization calculation, it is problematic to output a value _v1 other than the true optimal value _v0 as the optimal value, but the behavior decision-making method according to this embodiment utilizes this characteristic.

Figure 4 shows an image of the probability distribution of action a selected by performing optimization calculations, with the policy π represented as an optimization calculation model. Figure 4 (a) shows the probability distribution of action a selected after performing optimization calculations based on the policy π for a long period of time, and (b) shows the probability distribution of action a selected after performing optimization calculations based on the policy π for a short period of time.

When the policy π is represented by an optimization calculation model and an action a is obtained by performing an optimization calculation, the probability of selecting an action _a1 other than the optimal action _a0 changes depending on the calculation time of the optimization calculation. For example, the longer the calculation time, the lower the probability of selecting an action _a1 other than the optimal action _a0 , and the shorter the calculation time, the higher the probability of selecting an action _a1 other than the optimal action _a0 . Changing the calculation time in quantum annealing can change the randomness of the optimization calculation.

5 is an image diagram showing the relationship between the action selected by the agent based on the policy and the change in the real environment. The optimal action may change when the real environment 2 changes. For example, even if the action _a0 was optimal in the real environment 2 before the change, the action _a0 ' may become optimal when the real environment 2 changes.

When the randomness of the optimization calculation is small (for example, when the optimization calculation takes a long time), there is a low probability of selecting an action other than the action _a0 that is considered optimal in a given real environment 2. In this case, even when the real environment 2 changes, there is a high probability of continuing to select the action _a0 that is considered optimal in the real environment 2 before the change. In other words, there is a low probability of selecting the action _a0 ' that is optimal in the real environment 2 after the change, and there may be cases where the action _a0 ' that is optimal in the real environment 2 after the change is not performed.

In contrast, when the randomness of the optimization calculation is high (for example, the optimization calculation time is short), there is a high probability of selecting an action other than the action _a0 considered optimal in a given real environment 2. In this case, even if the real environment 2 changes, there is room for selecting an action other than the action _a0 considered optimal in the real environment 2 before the change. In other words, a high probability of selecting an action other than the action _a0 considered optimal in the real environment 2 before the change is equivalent to a high probability of selecting an action _a0 ' considered optimal in the environment after the real environment 2 has changed.

In the behavior decision-making method according to this embodiment, the randomness of the optimization calculation is changed based on the amount of change in the real environment. The randomness of the optimization calculation can be changed by changing the calculation parameters. As described above, when the optimization calculation is performed using quantum annealing, the calculation time of the optimization calculation is one of the calculation parameters.

When optimization calculations are performed using a genetic algorithm, the probability of mutations occurring is one of the calculation parameters that contribute to the randomness of the optimization calculation. The randomness of the optimization calculation can be changed by changing the mutation parameters. When optimization is performed using adiabatic quantum computing, which is performed on a quantum computer, noise such as heat applied to the calculator is one of the calculation parameters that contribute to randomness.

FIG. 6 is an example of a flow diagram of reinforcement learning according to the first embodiment. The behavior decision method according to the first embodiment has a behavior decision step S1, a state change step S2, and a learning step S3.

In the action decision step S1, an optimization calculation is performed based on a policy π represented by a model applicable to the optimization calculation, and an action a of the agent 1 is determined. In the state change step S2, the action a is input to the real environment 2, whereby the real environment 2 and the agent 1 interact with each other, changing the state of the agent 1. In the learning step S3, the value of the state S _a of the agent 1 after the action a is obtained as a reward r, and the next policy π is determined based on the reward r.

The action decision step S1 includes, for example, an environmental change measurement step S11, a calculation parameter setting step S12, and an optimization calculation step S13.

In the environmental change measurement step S11, the real environment 2 at the time of action decision (action decision step S1) is measured, and the amount of change with respect to each parameter of the real environment 2 at the time of learning (learning step S3) is obtained. For example, the temperature at which the robot arm operates, humidity, wear condition of parts, material of the floor on which the robot arm operates, and the amount of change in the movement of an adjacent robot arm are measured. The measurement of the real environment 2 is performed, for example, by a sensor, etc. Each parameter to be measured is set in advance. The amount of environmental change is found, for example, by adding up the amount of change in each measured parameter. Of the parameters, a coefficient may be multiplied to the parameter with the highest importance.

In calculation parameter setting step S12, calculation parameters are set based on the amount of environmental change found in environmental change measurement step S11. The calculation parameters differ depending on the method of optimization calculation. For example, when the optimization calculation is performed using quantum annealing, the calculation time of quantum annealing is one of the calculation parameters. In another example, when the optimization calculation is performed using a genetic algorithm, the probability of mutation is one of the calculation parameters. In another example, when the optimization calculation is performed using adiabatic quantum computing, noise such as heat added to the calculation is one of the calculation parameters.

For example, the amount of environmental change may be input to a sigmoid function and converted into a numerical value (calculation parameter) between 0 and 1. This type of conversion makes it possible to reduce the amount of change in the calculation parameter when the amount of environmental change is small, and to increase the amount of change in the calculation parameter when the amount of environmental change is large.

For example, when optimization calculations are performed using quantum annealing, the greater the amount of change in the environment, the shorter the quantum annealing calculation time will be. Also, for example, when optimization calculations are performed using a genetic algorithm, the greater the amount of change in the environment, the higher the probability of mutations occurring will be. Also, for example, when optimization calculations are performed using adiabatic quantum computing, the greater the amount of change in the environment, the more sources of noise will be added.

In the optimization calculation step S13, optimization calculation is performed according to the calculation parameters determined in the calculation parameter setting step S12. By performing the optimization calculation step S13, an action a is determined. As shown in Fig. 4, the action a selected in the optimization calculation step S13 is not limited to the truly optimal action _a0 .

In the optimization calculation step S13, it is possible to select an action _a1 other than the truly optimal action _a0 in a given real environment 2. For example, the smaller the randomness of the calculation parameters, the lower the probability of selecting an action _a1 other than the truly optimal action _a0 , and the greater the randomness of the calculation parameters, the higher the probability of selecting an action _a1 other than the truly optimal action _a0 . By being able to select an action _a1 other than the truly optimal action _a0 in the optimization calculation step S13, it is also possible to adapt to changes in the real environment 2.

The state change step S2 includes, for example, a behavior step S21 and an operation step S22.

In the action step S21, the action a determined according to the strategy π is input to the agent 1 in the real environment 2. For example, the action a is input to the robot arm by applying a current to each of the conductors that control the movement of the robot arm. The current applied to each is determined based on optimization calculations. The strategy π is represented by an optimization calculation model.

In the action step S22, the agent 1 acts in accordance with the action a determined in the action step S21. For example, in the action step S21, a current is applied to each of the conductors that control the movement of the robot arm, causing the robot arm to act. The action of the agent 1 changes the state S _a of the agent 1. For example, the action of the robot arm changes the state of the robot arm (e.g., the angle of the joint). The interaction between the real environment 2 and the agent 1 changes the state S _a of the agent 1.

The learning step S3 includes, for example, a real-environment measurement step S31, a reward calculation step S32, and a strategy determination step S33.

In the real environment measurement step S31, the real environment 2 in the learning step S3 is measured. The real environment 2 in the learning step S3 does not necessarily match the real environment 2 in the action decision step S1 described above. By measuring the real environment 2 in the learning step S3 where a strategy is decided, the amount of change in the real environment 2 between the learning step S3 and the action decision step S1 can be obtained, and calculation parameters can be set. The real environment measurement step S31 may be performed after the reward calculation step S32.

In the reward calculation step S32, a reward r for the state S _a of the agent 1 after the action a is obtained. The higher the reward r, the more appropriate the action a of the agent 1 was.

In the policy decision step S33, the next policy is decided based on the reward r. For example, in the policy decision step S33, the result of the previous action is reflected and a model of the optimization calculation representing the policy π is remade. The policy π represents the relationship between each action (how it affects the state S _a of the agent 1). If the reward r is high, it can be said that the previous policy π was an appropriate model, and a similar model is created. If the reward r is low, it can be said that the relationship represented by the previous policy π was not appropriate, and a new model that takes the previous action into account is created.

The reinforcement learning according to the first embodiment repeats the action decision step S1, the state change step S2, and the learning step S3 to learn to maximize the value of the data (maximize the reward). The environmental change measurement step S11, the calculation parameter setting step S12, and the real environment measurement step S31 do not have to be performed every time these steps are repeated. For example, the environmental change measurement step S11, the calculation parameter setting step S12, and the real environment measurement step S31 may be performed every time the steps are repeated multiple times, or the environmental change measurement step S11, the calculation parameter setting step S12, and the real environment measurement step S31 may be performed randomly.

The behavior decision-making method according to the first embodiment can also adapt to changes in the real environment 2.

For example, if the policy π is expressed as a certain probability distribution and actions are determined probabilistically according to this policy, the policy will not change even if the environment changes, and the probability that an action will be determined will not change either. Therefore, once reinforcement learning is performed in a specified real environment 2 and a policy π is specified, the method of determining actions will not change even if the environment changes. This policy π can select the optimal action a before the real environment 2 changes, but there is no certainty that action a is optimal even after the real environment 2 changes.

With this policy, even if the real environment 2 changes between when the action is decided and when it is learned, the action a is selected probabilistically according to the probability distribution learned in the real environment 2 before the change. In other words, even if the truly optimal action changes ( _a0 → _a0 ') due to a change in the real environment 2, the probability that _a0 or _a0 ' will be selected does not change from the time the policy was decided. As a result, the agent 1 is more likely to select an action a that is inappropriate in the real environment 2 after the change.

For example, if the degree of wear in the joints of the robot arm worsens, the movement of the robot arm will no longer be smooth. Even if current is applied to each conductor of the robot arm according to the strategy determined in the real environment 2 before wear, it is possible that the robot arm will not reach the desired state in the real environment after wear.

In contrast, in the behavior decision-making method according to the present embodiment, the policy π is represented by a model applicable to optimization calculation. This model is set in the real environment 2 during learning, and the same policy π can be used even if the real environment 2 changes. Behavior decision based on the policy π is performed by optimization calculation. The probability distribution of the behavior a obtained as a result of the optimization calculation changes depending on the calculation parameters. When the real environment 2 has not changed, the probability that the truly optimal behavior a0 in the policy _π is selected can be increased, and when the real environment 2 has changed, the probability that the truly optimal behavior _a0 in the policy π is selected or is reduced, and the probability that another behavior _a1 is selected can be increased.

For example, if the degree of wear at the joints of the robot arm worsens, it is possible to apply current to each conductor of the robot arm under conditions other than those that were considered optimal in the real environment 2 before wear. For example, in the behavior decision-making method according to this embodiment, it is also possible to select, in accordance with the strategy π, to apply to each conductor of the robot arm a greater amount of current than was considered optimal in the real environment 2 before wear. In reality, in a post-wear state after the real environment 2 has changed, it is appropriate to select to apply to each conductor of the robot arm a greater amount of current than was considered optimal in the real environment 2 before wear.

The real environment 2 when deciding on an action does not necessarily match the real environment 2 when the policy π was obtained by learning. Nevertheless, the action is decided based on the policy π learned in the previous real environment 2. In the action decision method according to this embodiment, by changing the randomness of the action decision, the probability of selecting an action other than the action _a0 that is optimal in the real environment 2 before the change increases, and it is possible to respond to changes in the real environment 2. Therefore, when the action decision method according to this embodiment is used, it becomes easier to select an action adapted to the new real environment 2, and learning can be advanced by utilizing the progress of learning so far.

FIG. 7 is a block diagram of the behavior decision device 10 according to the first embodiment. The behavior decision device 10 includes, for example, a learning unit 11, a first control unit 12, a second control unit 13, a memory 14, a measurement unit 15, and a transmission/reception unit 16.

The learning unit 11 acquires the state S _a of the agent 1 after the action a as a reward r, and determines the next policy π based on the reward r. The learning unit 11 performs learning so as to maximize the reward r. The learning unit 11 acquires, for example, the action a and the state S _a of the agent 1 as a result of the action a from the memory 14. The learning unit 11 obtains the reward r based on the acquired action a and state S _a . The learning unit 11 determines the policy π based on the reward r.

The learning unit 11 has, for example, a computing unit (CPU). The learning unit 11 performs, for example, learning step S3.

The first control unit 12 executes an optimization calculation based on a strategy represented by a model applicable to the optimization calculation, and determines the behavior of the agent. The first control unit 12 performs, for example, the behavior determination step S1.

The strategy π determined by the learning unit 11 is transmitted to the first control unit 12. The first control unit 12 sets calculation parameters based on the amount of change in the real environment 2 measured by the measurement unit 15. The first control unit 12 then executes optimization calculations with the set calculation parameters and determines the action a of the agent 1. The first control unit 12 has, for example, a computing unit (CPU).

The second control unit 13 inputs the action a into the real environment 2, and changes the state S _a of the agent 1. The second control unit 13 instructs the agent 1 to perform the action a. The second control unit 13 instructs the agent 1 via, for example, the transmitting/receiving unit 16. The second control unit 13 has, for example, a computing unit (CPU).

The memory 14 stores learning data, a program according to the behavior decision method, and information on environmental changes. The learning data includes, for example, an agent's behavior a, a state S _a of the agent 1 as a result of the agent's behavior a, and a reward r for the state S _a .

The measurement unit 15 is, for example, a sensor. The measurement unit 15 measures the real environment 2. The measurement unit 15 is used, for example, when performing the real environment measurement step S31 and the environmental change amount measurement step S11. The measurement unit 15 measures, for example, each parameter representing the real environment 2.

The transmitting/receiving unit 16 transmits the action a to the device, for example, according to an instruction from the second control unit 13. For example, the transmitting/receiving unit 16 transmits the action a based on the measure π to the control unit of the robot arm. The transmitting/receiving unit 16 also receives the state S _a of the agent 1 as a result of the action a. The transmitting/receiving unit 16 may be wired or wireless. The transmitting/receiving unit 16 is responsible for the state change step S2. The reward r obtained in the reward calculation step S32 is stored in the memory 14, for example, via the transmitting/receiving unit 16.

The behavior decision-making device 10 according to the first embodiment operates according to the behavior decision-making method described above, and is therefore also capable of responding to changes in the real environment 2.

The above describes the embodiments of the present invention in detail with reference to the drawings, but each configuration and their combinations in each embodiment are merely examples, and additions, omissions, substitutions, and other modifications of configurations are possible without departing from the spirit of the present invention.

REFERENCE SIGNS LIST 1 Agent 2 Real environment 10 Action decision device 11 Learning unit 12 First control unit 13 Second control unit 14 Memory 15 Measurement unit 16 Transmitting/receiving unit a Action r Reward π Policy S _a State S1 Action decision step S11 Environment change amount measurement step S12 Calculation parameter setting step S13 Optimization calculation step S2 State change step S21 Action step S22 Operation step S3 Learning step S31 Real environment measurement step S32 Reward calculation step S33 Policy decision step

Claims (5)

an action decision step of performing an optimization calculation and deciding an action of an agent based on a policy represented by a model applicable to the optimization calculation;
a state change step in which the behavior is input into a real environment, whereby the real environment and the agent interact with each other and a state of the agent changes;
A learning step of acquiring a value of the state of the agent after the action as a reward and determining a next policy based on the reward;
The optimization calculation can change randomness by a calculation parameter;
The behavior determination method includes changing the calculation parameters based on an amount of change in the real environment between the learning step and the behavior determination step. The behavior decision-making method according to claim 1, wherein the optimization calculation is an adiabatic quantum calculation performed on a quantum computer. The optimization calculation is quantum annealing,
The method of claim 1 , wherein the model is an Ising model or a QUBO model. The behavior decision-making method according to claim 1, wherein the optimization calculation is performed using a genetic algorithm. A learning unit, a first control unit, and a second control unit,
The first control unit executes an optimization calculation based on a measure represented by a model applicable to the optimization calculation, and determines an action of an agent;
the second control unit inputs the action into a real environment and changes a state of the agent;
the learning unit acquires a state of the agent after performing the action as a reward, and determines a next policy based on the reward;
The optimization calculation can change randomness by a calculation parameter;
The calculation parameters are changed based on an amount of change in the real environment between when the agent's action is determined and when the strategy is determined.

Images