WO2025004859A1 - Learning device, control system, input/output device, learning method, and recording medium - Google Patents

Abstract

In the present invention, a learning device generates a policy, which is a rule for determining a to-be-controlled action, on the basis of a reward value that indicates an evaluation of the action in a state in which a time step is traced back in an episode representing a learning period, and the learning device determines the to-be-controlled action on the basis of the policy.

Description

Learning device, control system, input/output device, learning method, and recording medium

This disclosure relates to a learning device, a control system, an input/output device, a learning method, and a recording medium.

One type of machine learning is reinforcement learning (see, for example, Patent Document 1).

Japanese Patent Application Publication No. 2022-182581

It would be preferable to be able to efficiently perform reinforcement learning using simulation.

One example of the objective of this disclosure is to provide a learning device, a control system, an input/output device, a learning method, and a recording medium that can solve the above-mentioned problems.

According to a first aspect of the present disclosure, the learning device includes a policy generation means for generating a policy, which is a decision rule for the behavior of a controlled object, based on a reward value, which is a value indicating an evaluation of the behavior of the controlled object at a time step back within an episode representing a learning period, and a behavior decision means for deciding the behavior of the controlled object based on the policy.

According to a second aspect of the present disclosure, a control system includes a learning device and a control device, the learning device includes a policy generation means for generating a policy, which is a decision rule for the behavior of the control object, based on a reward value, which is a value indicating an evaluation of the behavior of the control object at a time step back within an episode representing a learning period, and a behavior decision means for deciding the behavior of the control object based on the policy, and the control device controls the control object based on the policy obtained using the learning device.

According to a third aspect of the present disclosure, the input/output device includes a state presenting means for presenting to a user the state of the environment in which the controlled object acts, and a state designation receiving means for receiving a user operation for designating a state in which the simulator that simulates the environment needs to be interrupted.

According to a fourth aspect of the present disclosure, the learning method includes generating a policy, which is a decision rule for the behavior of a controlled object, based on a reward value, which is a value indicating an evaluation of the behavior of the controlled object at a time step back within an episode representing a learning period, and determining the behavior of the controlled object based on the policy.

According to a fifth aspect of the present disclosure, the recording medium stores a program for causing a computer to execute the following: generating a policy, which is a decision rule for the behavior of a controlled object, based on a reward value, which is a value indicating an evaluation of the behavior of the controlled object at a time step back within an episode representing a learning period; and determining the behavior of the controlled object based on the policy.

According to the present disclosure, reinforcement learning using simulation can be performed relatively efficiently.

FIG. 1 illustrates an example configuration of a control system according to some embodiments of the present disclosure. FIG. 2 is a diagram illustrating an example of the configuration of a learning device according to some embodiments of the present disclosure. FIG. 13 is a diagram showing an example of an instantaneous reward value that is the basis for calculating a cumulative reward value. FIG. 13 is a diagram illustrating an example of a cumulative reward value. FIG. 11 is a diagram showing a first example of a probability distribution set for reset destination candidates according to some embodiments of the present disclosure. FIG. 13 is a diagram showing a second example of a probability distribution set for reset destination candidates according to some embodiments of the present disclosure. FIG. 13 is a diagram showing a third example of a probability distribution set for reset destination candidates according to some embodiments of the present disclosure. A diagram showing an example of data input and output in a learning device according to some embodiments of the present disclosure. A diagram showing an example of a processing procedure performed by a learning device according to some embodiments of the present disclosure. FIG. 13 is a diagram illustrating another example of the configuration of a learning device according to some embodiments of the present disclosure. FIG. 2 illustrates another example of a control system configuration according to some embodiments of the present disclosure. FIG. 2 is a diagram illustrating an example of a configuration of an input/output device according to some embodiments of the present disclosure. FIG. 1 is a diagram illustrating an example of a processing procedure in a learning method according to some embodiments of the present disclosure. FIG. 1 illustrates an example of a computer configuration in accordance with at least one embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described, but the following embodiments do not limit the disclosure according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the disclosure.
1 is a diagram illustrating an example of a configuration of a control system according to some embodiments of the present disclosure. In the configuration illustrated in FIG. 1, the control system 1 includes a learning device 100, a control device 200, and a control target 910.

The control system 1 is a system that learns control of a control target 910 and controls the control target 910 based on the learning results.
The learning device 100 learns control over the control object 910. In particular, the learning device 100 learns control over the control object 910 by reinforcement learning using a simulation.

The reinforcement learning referred to here is a machine learning technique that learns a policy, which is the behavioral rule of an agent that performs an action in a certain environment, based on a reward, which represents an evaluation of the action.
The state of the environment is also referred to simply as the state. The environment may include an agent. Therefore, the state may include the state of the agent.

For each step in reinforcement learning, the learning device 100 determines an action based on a policy in the state at that time, simulates the determined action, and calculates the next state, which is the state at the next step. The learning device 100 also calculates a reward value based on the obtained next state, and updates the policy based on the calculated reward value. The update of the policy here can be considered as the generation of a policy. In other words, the learning device 100 can be considered to generate a policy based on a past policy.

Hereinafter, a step in reinforcement learning will be referred to as a time step or simply as a step. Also, below, time will be represented by a time step.
The reinforcement learning method used by the learning device 100 is not limited to a specific type of method. For example, the learning device 100 may learn control of the control target 910 based on a known reinforcement learning method such as Q-learning or SARSA.

For example, the learning device 100 repeats the processing for each step of each episode until the episode ends. An episode here is a unit of time in reinforcement learning, and is a time period from when an agent starts a series of actions until when it ends. For example, the learning device 100 repeats the processing for each step in the episode until a condition that is predetermined as the condition for ending the episode is met.

Furthermore, when the learning device 100 determines that a predetermined condition for interrupting the simulation is satisfied, the learning device 100 interrupts the simulation even in the middle of an episode. The condition for interrupting the simulation is also referred to as an interruption condition. Interrupting the simulation can also be said to interrupt the episode being executed.
This allows the learning device 100 to automatically interrupt the execution of an episode when the learning device 100 falls into a state where policy updating is not expected even if the episode is continued, or when the learning device 100 falls into a state where policy updating is expected to progress slowly.
Policy updates can be thought of as policy improvements. Policy updates can be thought of as progress in reinforcement learning.

The learning device 100 that has suspended the execution of an episode may go back a time step within the episode and resume the execution of the episode. Alternatively, the learning device 100 that has suspended the execution of an episode may start the execution of another episode.
According to the learning device 100, reinforcement learning can be performed more efficiently than in a case where the execution of an episode is continued even when the state in which policy updating is not expected or when the state in which policy updating is expected to progress slowly is reached.

The learning device 100 also stores the state at each of a number of time steps during the execution of an episode. When the learning device 100 suspends the execution of an episode, it selects one of the time steps for which the state is stored, returns to the selected time step, and resumes the execution of that episode. Specifically, the learning device 100 resumes processing for each time step from the state of the selected time step.

Here, if we consider a case where an episode is executed from the beginning every time, the beginning of the episode is a part where learning has already progressed sufficiently, and it is conceivable that the policy update will not proceed any further. By allowing the learning device 100 to select a time step in the middle of an episode as a destination to go back to, not just the beginning of the episode, it is expected that reinforcement learning can be performed more efficiently.
The time step to go back in the episode is also called the reset destination. The candidates for the reset destination are also called reset destination candidates. The reset destination candidates are, for example, executed time steps whose states are stored as time steps at which the simulation can be resumed within the episode that was executed until the simulation was interrupted.
Interrupting the execution of a simulation and changing the state in the simulation to a reset state is also called resetting the simulation.

The control device 200 controls the control target 910 based on the policy obtained by learning using the learning device 100 .
The control object 910 is not limited to a specific one, and may be various objects capable of learning control over the control object 910 using reinforcement learning. For example, the control object 910 may be equipment such as a plant or a power plant, a system such as a manufacturing line in a factory, or a standalone device. Alternatively, the control object 910 may be a moving object such as an automobile, a railroad vehicle, an airplane, a ship, or a self-propelled mobile robot, or a transportation system such as a railroad or air traffic control system.
The controlled object 910 may be configured as a part of the control system 1 , or may be configured external to the control system 1 .

During learning, it is sufficient to have the learning device 100, and it is not necessary to have the control device 200 and the control target 910. Furthermore, during execution of control, it is sufficient to have the control device 200 and the control target 910, and it is not necessary to have the learning device 100.
All of the learning device 100, the control device 200, and the control target 910, or a combination of two of them, may be configured integrally. For example, the learning device 100 and the control device 200 may be configured as a single device. Also, the control device 200 and the control target 910 may be configured as a single device.

FIG. 2 is a diagram showing an example of the configuration of the learning device 100. In the configuration shown in FIG. 2, the learning device 100 includes a communication unit 110, a display unit 120, an operation input unit 130, a memory unit 180, and a processing unit 190. The processing unit 190 includes an action decision unit 191, a simulation unit 192, a measure generation unit 193, a reset decision unit 194, and a resume state decision unit 195.

The communication unit 110 communicates with other devices. For example, the communication unit 110 may receive various information for performing a simulation from other devices. The communication unit 110 may also transmit the measures obtained by learning to the control device 200.

The display unit 120 has a display screen, such as a liquid crystal panel or an LED (Light Emitting Diode) panel, and displays various images. For example, the display unit 120 may display information related to the learning performed by the learning device 100, such as the progress of the learning performed by the learning device 100.

The operation input unit 130 includes input devices such as a keyboard and a mouse, and receives user operations. For example, the operation input unit 130 may receive a user operation to instruct the control unit 910 to start learning control over the control target 910.
The storage unit 180 stores various data. For example, a snapshot of a state in a simulation may be stored. The storage unit 180 is configured using a storage device included in the learning device 100.

The processing unit 190 performs various processes by controlling each unit of the learning device 100. The functions of the processing unit 190 are performed, for example, by a CPU (Central Processing Unit) included in the learning device 100 reading a program from the storage unit 180 and executing it.
The behavior decision unit 191 decides the behavior of the control object 910 in reinforcement learning. The behavior decision unit 191 decides the behavior of the control object 910 based on a policy.
The behavior determining unit 191 corresponds to an example of a behavior determining means.

The simulation unit 192 performs a simulation of the environment. In particular, the simulation unit 192 performs a simulation of the action determined by the action determination unit 191, and calculates a next state which is a state after the action. Here, the control object 910 in the simulation can be regarded as an agent or a part thereof, and the control object 910 may be included in the environment that is the target of the simulation. The state of the control object 910 may be included in the state in the simulation.
The simulation unit 192 corresponds to an example of a simulation means.
The simulation unit 192 may be configured as a part of the learning device 100 , or may be configured external to the learning device 100 .

The policy generator 193 updates the policy based on the reward value. As described above, the reward value is a value indicating an evaluation for an action. The policy is a decision rule for an action. As described above, the update of the policy here can be considered as the generation of the policy.
The policy generation unit 193 may update the policy based on the instantaneous reward value, or may update the policy based on the cumulative reward value. The instantaneous reward value is a value calculated for each step based on the state, action, and next state, or a part of these, and indicates an evaluation of the action in that step. The cumulative reward value is a value calculated by accumulating the instantaneous reward values for multiple steps. When calculating the cumulative reward value, the instantaneous reward value may be multiplied by a coefficient value such as a forgetting coefficient.
Alternatively, the policy generator 193 may update the policy based on a value function value (also referred to as value), which is a prediction of the cumulative reward value, e.g., an expected value of the cumulative reward value at the end of an episode.

The instantaneous reward value, the cumulative reward value, and the value function value are all examples of reward values.
The learning device 100 may use a reward value in which the larger the value, the better the evaluation, or may use a reward value in which the smaller the value, the better the evaluation. In the following, an example will be described in which the learning device 100 uses a reward value in which the larger the value, the better the evaluation.
The combination of the action decision unit 191 and the measure generation unit 193 is also called an agent processing unit.

When the reset determination unit 194 determines that the interruption condition is satisfied, it interrupts the simulation by the simulation unit 192. Specifically, the reset determination unit 194 interrupts the simulation being performed by the simulation unit 192. As described above, the interruption condition is a condition that is determined in advance as a condition for interrupting the simulation of the behavior of the control target 910.
The reset determination unit 194 corresponds to an example of a reset determination means.

As described above for the learning device 100, the reset decision unit 194 can automatically interrupt the execution of an episode if the episode falls into a state where policy updates are not expected to occur even if the episode continues to be executed, or if the episode falls into a state where policy updates are expected to proceed slowly.

The reset determination unit 194 that has suspended the execution of an episode may go back a time step within the episode and restart the simulation by the simulation unit 192 from the state of the time step that was gone back. Alternatively, the reset determination unit 194 that has suspended the execution of an episode may select another episode and cause the simulation unit 192 to perform a simulation of the selected episode.
This allows the learning device 100 to perform reinforcement learning more efficiently compared to a case in which the learning device 100 continues to execute the episode even when the learning device 100 falls into a state in which policy updating is not expected or when the learning device 100 falls into a state in which policy updating is expected to progress slowly.

The restart state determination unit 195 determines a reset destination when the simulation by the simulation unit 192 is interrupted. The simulation unit 192 restarts the simulation from the reset destination state determined by the restart state determination unit 195.
The restart state determination unit 195 corresponds to an example of a restart state determination means.

As described above, it is conceivable that at the beginning of an episode, policy updates may not necessarily progress. As described above for the learning device 100, the restart state determination unit 195 can select not only the beginning of an episode but also any time step in the middle as the destination to go back in the episode, and this is expected to enable the learning device 100 to perform reinforcement learning more efficiently.

When the reset decision unit 194 interrupts the execution of an episode, the simulation unit 192 may resume the simulation from a predetermined state, such as the initial state of the episode.
In this case, the learning device 100 may be configured without including the resume state determination unit 195 .

The reset decision unit 194 may decide whether or not to interrupt the simulation by the simulation unit 192 based on a change in the cumulative reward value.
3 is a diagram showing an example of an instantaneous reward value that is the basis for calculating a cumulative reward value. The horizontal axis of the graph in FIG. 3 indicates a time step, and the vertical axis indicates an instantaneous reward value.
In the example of Fig. 3, the instantaneous reward value can be a positive value or a negative value. The larger the instantaneous reward value (e.g., the larger the positive value and the magnitude), the better the evaluation indicated by the instantaneous reward value. The smaller the instantaneous reward value (e.g., the larger the negative value and the magnitude), the worse the evaluation indicated by the instantaneous reward value.
In the example of FIG. 3, at time step t11, the instantaneous reward value is a positive value, and from time step t11 onwards there are successive time steps in which the instantaneous reward value is a negative value.

Fig. 4 is a diagram showing an example of a cumulative reward value. The horizontal axis of the graph in Fig. 4 indicates a time step, and the vertical axis indicates a cumulative reward value.
Fig. 4 shows the cumulative reward value calculated by accumulating the instantaneous reward values shown in Fig. 3 from the start of the episode. The cumulative reward value reaches a maximum at time step t11, and from the time step after t11, the cumulative reward value continues to decrease.

When the cumulative reward value is small, such as in the state of time step t12, it is possible that an event has occurred that causes the evaluation indicated by the reward value to be poor. For example, if the control object 910 is a railway, it is possible that the intervals between trains have become too close, causing a disruption to the train schedule.

If an event occurs that causes the reward value to be poorly evaluated, continuing to execute the episode will not improve the reward value, and it is thought that going back a time step and restarting the episode will result in more progress in updating the policy.
For example, when the control object 910 is a railway and train intervals are short, causing a disruption in the schedule, it is considered that the evaluation indicated by the reward value will not improve until the short train intervals are resolved. It is considered that the poor evaluation indicated by the reward value will not progress in updating the measures by the measure generation unit 193.

In this case, rather than continuing to execute the episode as is, it is expected that the policy will be updated more effectively if the episode is restarted by going back to a time step before the train intervals become shorter, or before an event occurs that causes the reward value to be evaluated poorly.
In the example of Fig. 4, in the time steps from the time step t11 onwards where the cumulative reward value continues to decrease, it is possible that an event that causes the reward value to be poorly evaluated has already occurred. In this case, it is expected that the policy update will progress more if the episode is executed again going back to time step t11 or an earlier time step.

Therefore, the reset decision unit 194 may decide whether or not to interrupt the simulation by the simulation unit 192 based on the change in the accumulated reward value.
For example, when the reset decision unit 194 determines that the evaluation indicated by the cumulative reward value has deteriorated below a predetermined threshold, the reset decision unit 194 may decide to interrupt the simulation by the simulation unit 192. In the example of Fig. 4, the reset decision unit 194 may compare the magnitude of the decrease amount from the maximum value of the cumulative reward value at the time step t11 with the threshold value d11, and may decide to interrupt the simulation by the simulation unit 192 at or after the end of the time step t12 where the magnitude of the decrease amount is greater than the threshold value d11.

In this case, the reset decision unit 194 may update the threshold value so that the extent of deterioration in evaluation indicated by the threshold value of the cumulative reward value increases as the simulation by the simulation unit 192 progresses. In the example of FIG. 4, the reset decision unit 194 may update the value of the threshold value d11 so that the value of the threshold value d11 increases as the time steps progress.

The starting point from which the reset determination unit 194 calculates the amount of decrease in the cumulative reward value is not limited to the maximum value of the cumulative reward value. For example, the reset determination unit 194 may calculate the amount of decrease from a cumulative reward value of 0 and compare it with a threshold value. Alternatively, the reset determination unit 194 may calculate the amount of decrease from the maximum value of the cumulative reward value and compare it with a threshold value.

Here, it is considered that in the early stages of an episode or in the early stages of reinforcement learning, there are many cases in which the reward value does not become large because learning has not progressed. In this case, by the reset decision unit 194 setting the threshold to a relatively small value and suspending the execution of the episode earlier, it is expected that the action decision unit 191 will try various actions and find an action (or series of actions) that will result in a large reward value at a relatively early stage.

Alternatively, the reset decision unit 194 may decide to interrupt the simulation by the simulation unit 192 when the deterioration of the evaluation indicated by the cumulative reward value continues for a predetermined number of time steps or more, based on the increase or decrease in the cumulative reward value. In the example of FIG. 4, if the number of time steps for interrupting the simulation by the simulation unit 192 is set to 7, the reset decision unit 194 may decide to interrupt the simulation by the simulation unit 192 at or after the end of time step t12 in which the cumulative reward value has decreased seven times consecutively from time step t11 at which the cumulative reward value is maximized.

The reset decision unit 194 may decide whether or not to interrupt the simulation by the simulation unit 192 based on a combination of the amount of decrease in the cumulative reward value and the number of time steps in which the cumulative reward value has continuously decreased. For example, the reset decision unit 194 may decide to interrupt the simulation by the simulation unit 192 when the amount of decrease in the cumulative reward value becomes greater than a predetermined threshold value, and when the number of time steps in which the cumulative reward value has continuously decreased reaches or exceeds a predetermined number.

The reset decision unit 194 may decide whether or not to suspend the simulation by the simulation unit 192 based on the value function value in addition to or instead of the cumulative reward value. The value function value is a predicted value of the cumulative reward value, and it is considered that there is a positive correlation between the value function value and the cumulative reward value. By having the reset decision unit 194 decide whether or not to suspend the simulation by the simulation unit 192 based on the value function value, the simulation by the simulation unit 192 will be suspended when an event occurs that causes the evaluation indicated by the reward value to be a bad evaluation, just as in the case where it is based on the cumulative reward value, and it is expected that the update of the policy will progress.

As in the case of the cumulative reward value described above, if the reset decision unit 194 determines, based on an increase or decrease in the value function value, that the evaluation indicated by the value function value has deteriorated below a predetermined threshold value, the simulation by the simulation unit 192 may be interrupted.
In this case, as in the case of the cumulative reward value described above, the reset determination unit 194 may update the threshold value so that the extent of the deterioration in evaluation indicated by the threshold value of the value function value increases as the simulation by the simulation unit 192 progresses.

Also, as in the case of the cumulative reward value described above, the reset decision unit 194 may decide to interrupt the simulation by the simulation unit 192 if the deterioration of the evaluation indicated by the value function value continues for a predetermined number of time steps or more, based on an increase or decrease in the value function value.
As in the case of the cumulative reward value described above, reset determination unit 194 may determine whether or not to interrupt the simulation by simulation unit 192 based on a combination of the amount of decrease in the value function value and the number of time steps in which the value function value continuously decreases.

If the reset decision unit 194 determines that the current state (the state at the current time step) is similar to one or more states that are pre-set as states requiring interruption to a certain degree or more, the simulation unit 192 may be configured to interrupt the simulation.
For example, when the control target 910 is a railway, the storage unit 180 stores a plurality of patterns of states in which train intervals are short and disruption of the schedule occurs as states requiring interruption. The reset decision unit 194 then compares the current state with each of the states requiring interruption and determines whether the current state is similar to any of the states requiring interruption. If it is determined that the current state is similar to any of the states requiring interruption, the reset decision unit 194 decides to interrupt the simulation by the simulation unit 192.

The method by which the reset determination unit 194 determines whether the current state is similar to the state requiring interruption is not limited to a specific method.
For example, the reset determination unit 194 may calculate the feature amount of each of the two states as a vector, and calculate the similarity of the vectors, such as cosine similarity. The reset determination unit 194 may then compare the calculated similarity with a threshold, and determine that the two states are similar if the similarity is equal to or greater than the threshold. Alternatively, a machine learning model that determines whether or not the two states are similar may be prepared, and the reset determination unit 194 may use this machine learning model to determine whether or not the current state is similar to the state requiring interruption.

When the learning device 100 collects states requiring interruption, the learning device 100 may present the states to the user and accept the user's designation of the states requiring interruption.
For example, the reset determination unit 194 may cause some state, such as a current state or a past state in the simulation, to be displayed on the display unit 120. The reset determination unit 194 may then receive a user operation via the operation input unit 130 instructing whether or not to register the displayed state as a state requiring interruption.

In this case, the combination of the reset determination unit 194 and the display unit 120 corresponds to an example of a state presenting unit, and the combination of the reset determination unit 194 and the operation input unit 130 corresponds to an example of a state designation receiving unit.
When the control target 910 is a railway, the display unit 120 may display the current train operation status in the simulation in the form of a diagram. When the user determines that the displayed operation status is one that requires interruption, the user may perform a user operation indicating this on the operation input unit 130, and the reset determination unit 194 may detect that the user operation has been performed.

The learning device 100 is an example of an input/output device in that it is equipped with a reset decision unit 194, a display unit 120, and an operation input unit 130. Alternatively, the input/output device may be configured as a device separate from the learning device 100. For example, the terminal device of the learning device 100 may have a function of displaying the status according to instructions from the learning device 100, and a function of accepting a user operation specifying a state requiring interruption and notifying the learning device 100.

The restart state determination unit 195 may select, from among the reset destination candidates in the episode that was being executed until the simulation was interrupted, a reset destination candidate that corresponds to a time step with the fewest number of time steps between the time the simulation was interrupted and the time step in which the change in evaluation, indicated by an increase or decrease in the cumulative reward value, has turned from improvement to deterioration, or a time step prior to that. In other words, the restart state determination unit 195 may select a reset destination candidate that corresponds to a time step in which the evaluation was maximally good immediately before the interruption of the simulation, or a time step prior to that.

In the example of FIG. 4, if the reset determination unit 194 interrupts the simulation by the simulation unit 192 at time step t12, time step t11 corresponds to the time step at which the cumulative reward value is maximized immediately before the simulation is interrupted (time step t12). The restart state determination unit 195 may select a reset destination candidate that corresponds to time step t11 or an earlier time step.

In time steps after the time step in which the evaluation was the most recent maximally good evaluation when the simulation was interrupted, the cumulative reward value is continuously decreasing, and it is possible that an event has occurred that causes the evaluation indicated by the reward value to be a bad evaluation. By having the restart state determination unit 195 select a reset destination candidate that corresponds to the time step in which the evaluation was the most recent maximally good evaluation when the simulation was interrupted or an earlier time step, it is expected that the possibility of selecting a time step before the occurrence of an event that causes the evaluation indicated by the reward value to be a bad evaluation will increase.

Alternatively, the resume state determination unit 195 may select one of all reset destination candidates within the episode that was being executed up until the simulation was interrupted as selection targets.
As a result, if a decrease in the cumulative reward value does not necessarily indicate the occurrence of an event that will cause the reward value to have a bad evaluation, the resume state determination unit 195 may be able to select a later reset destination candidate (a reset destination candidate closer to the time the simulation was interrupted).

The resume state determination unit 195 may select one of the multiple reset destination candidates in accordance with a probability distribution set for the multiple reset destination candidates.
Here, when an event that causes the reward value to be evaluated poorly occurs, from the viewpoint of returning to a time step before the event occurred, it is possible to return to as early a time step in the episode as possible (a time step close to the beginning of the episode). On the other hand, from the viewpoint of reducing the number of repetitions of learning at the beginning of the episode, it is possible to return to as late a time step as possible among the time steps that have been executed in the episode. In this way, there is a trade-off between achieving efficiency in reinforcement learning by avoiding a state in which an event that causes the reward value to be evaluated poorly occurs, and achieving efficiency in reinforcement learning by reducing the number of repetitions of learning at the beginning of the episode.

The restart state determination unit 195 therefore probabilistically selects one of the multiple reset destination candidates. This makes it possible to avoid restarting the execution of the episode from the beginning every time the execution of the episode is interrupted. Furthermore, if an event that would cause the reward value to be evaluated poorly has already occurred in the selected reset destination candidate, it is expected that by interrupting the episode one or more times, it will be possible to select a reset destination candidate that was selected before the event that would cause the reward value to be evaluated poorly occurred.

The restart state determination unit 195 may select one of the multiple reset destination candidates in accordance with a uniform probability distribution.
This allows the restart state determination unit 195 to select a reset destination candidate in response to a control object 910 for which the approximate number of time steps to go back to reach a state before the occurrence of an event that caused the evaluation indicated by the reward value to be poorly evaluated is unknown.

5 is a diagram showing a first example of a probability distribution set for reset destination candidates, in which the horizontal axis of the graph in Fig. 5 represents time steps and the vertical axis represents probability.
5, the reset determination unit 194 suspends the simulation by the simulation unit 192 at a time step after time step t25, and the five time steps from time step t21 to t25 are reset destination candidates. The restart state determination unit 195 sets a uniform probability distribution for these reset destination candidates, selecting three time steps from time step t21 to t23 among these five reset destination candidates as reset destination candidates to be selected. The restart state determination unit 195 selects each of the three reset destination candidates with a one-third probability according to the set probability distribution.

Alternatively, the restart state determination unit 195 may set a probability distribution for the reset destination candidates such that the greater the deterioration in evaluation at the time the simulation is interrupted, as indicated by an increase or decrease in the cumulative reward value, the more likely it is that a reset destination candidate with a large number of time steps between the time the simulation is interrupted and the reset destination candidate will be selected, and one of the reset destination candidates may be selected according to the set probability distribution.

6 is a diagram showing a second example of a probability distribution set for reset destination candidates, in which the horizontal axis of the graph in Fig. 6 represents time steps and the vertical axis represents probability.
6, the reset determination unit 194 suspends the simulation by the simulation unit 192 at a time step after time step t25, and the five time steps from time step t21 to t25 are reset destination candidates. The restart state determination unit 195 sets three time steps from time step t21 to t23 out of these five reset destination candidates as reset destination candidates to be selected, and sets a probability distribution for these reset destination candidates.

FIG. 6 shows an example in which the amount of decrease in the cumulative reward value when the simulation is interrupted is relatively small, and a higher probability is set for a reset destination candidate that is closer to the interruption (i.e., a reset destination candidate with fewer time steps between the interruption and the simulation).
The restart state determination unit 195 selects one of the three reset destination candidates according to the set probability distribution. This makes it relatively easy for the restart state determination unit 195 to select a reset destination candidate that is close to the interruption time (i.e., a reset destination candidate with a small time step between the interruption time and the reset destination candidate).

7 is a diagram showing a third example of a probability distribution set for reset destination candidates. The horizontal axis of the graph in Fig. 7 represents time steps, and the vertical axis represents probability.
7, the reset determination unit 194 suspends the simulation by the simulation unit 192 at a time step after time step t25, and the five time steps from time step t21 to t25 are reset destination candidates. The resume state determination unit 195 sets three time steps from time step t21 to t23 out of these five reset destination candidates as reset destination candidates to be selected, and sets a probability distribution for these reset destination candidates.

Fig. 7 shows an example in which the magnitude of the decrease in the cumulative reward value at the time of interruption of the simulation is relatively large. In the example of Fig. 7, a higher probability is set for time step t21, which is a candidate for the reset destination relatively close to the time of interruption, than in Fig. 6. Also, a lower probability is set for time step t23, which is a candidate for the reset destination relatively close to the time of interruption, than in Fig. 6.
The restart state determination unit 195 selects one of the three reset destination candidates according to the set probability distribution. This makes it relatively easy for the restart state determination unit 195 to select a reset destination candidate that is far from the interruption time (i.e., a reset destination candidate with many time steps between the interruption time and the reset destination candidate).

Here, it is particularly desirable to avoid a situation in which the cumulative reward value drops suddenly. In order to increase the possibility of avoiding a situation in which the cumulative reward value drops suddenly, it is possible to take a large number of time steps back to the time when the simulation was interrupted.
On the other hand, a situation where the cumulative reward value gradually decreases is less desirable than a situation where the cumulative reward value rapidly decreases. In this case, in order to reduce the repetition of learning at the beginning of an episode, it is conceivable to make the number of time steps back when the simulation is interrupted relatively small.
It is expected that the learning device 100 can efficiently perform reinforcement learning by the restart state determination unit 195 changing the reset destination candidates that are easy to select depending on the magnitude of the increase or decrease in the cumulative reward value.

FIG. 8 is a diagram showing an example of data input/output in the learning device 100.
In the example of FIG. 8, the action decision unit 191 decides the action of the control target 910 based on the measure, and outputs the decided action to the simulation unit 192 .
The simulation unit 192 simulates the action determined by the action determination unit 191 to calculate a state (next state), and outputs the calculated state to the measure generation unit 193.

The policy generation unit 193 calculates a reward value based on the state acquired from the action decision unit 191, and updates the policy based on the calculated reward value. Depending on the reward value, the policy generation unit 193 may not update the policy (may leave it as is).
In addition, the measure generator 193 outputs the calculated reward value to the reset determiner 194 and the restart state determiner 195 .

The reset decision unit 194 decides whether or not to interrupt the simulation by the simulation unit 192 based on the reward value. If the reset decision unit 194 decides to interrupt the simulation, the restart state decision unit 195 decides the reset destination based on the reward value. The reset decision unit 194 and the restart state decision unit 195 then output an instruction to interrupt the simulation and the reset destination to the simulation unit 192 and the measure generation unit 193.

Here, the reward value used by the policy generation unit 193, the reward value used by the reset determination unit 194, and the reward value used by the restart state determination unit 195 are not limited to a specific one. The reward value used by the policy generation unit 193, the reward value used by the reset determination unit 194, and the reward value used by the restart state determination unit 195 may be an instantaneous reward value, a cumulative reward value, a value function value, or any other reward value, or may be a combination of these.

Furthermore, the reward value used by the policy generation unit 193, the reward value used by the reset determination unit 194, and the reward value used by the restart state determination unit 195 may be the same or different. For example, the policy generation unit 193 may output an instantaneous reward value to the reset determination unit 194, and the reset determination unit 194 may calculate a cumulative reward value based on the instantaneous reward value.

When an interrupt instruction is received, the simulation unit 192 interrupts the simulation and returns the state of the simulation to the designated reset destination.
Furthermore, when an interruption instruction is received, the policy generation unit 193 returns either the policy or the reward value, or both, to the values at the specified reset destination. The policy generation unit 193 may not change either the policy or the reward value when the simulation is interrupted. In this case, the reset determination unit 194 and the restart state determination unit 195 may not output the interruption instruction and the reset destination to the policy generation unit 193.

FIG. 9 is a diagram illustrating an example of a procedure of processing performed by the learning device 100.
9, the simulation unit 192 sets up a simulation (step S101). At the start of an episode, the simulation unit 192 sets the state of the episode that is specified in the episode.
Next, the processing unit 190 executes one step of reinforcement learning (step S102).
Then, the reset determination unit 194 determines whether or not the interruption condition is met (step S103).

When the reset determination unit 194 determines that the interruption condition is met (step S103: YES), the resume state determination unit 195 determines a reset destination (step S111).
After step S111, the process returns to step S101. In this case, in step S101, the simulation unit 192 sets the state in the simulation to the state after reset.

On the other hand, if the reset determination unit 194 determines in step S103 that the interruption condition is not met (step S103: NO), the processing unit 190 determines whether or not the episode end condition is met (step S121).
When the processing unit 190 determines that the episode end condition is not met (step S121: NO), the process returns to step S102.

On the other hand, when it is determined in step S103 that the end condition of the episode is satisfied (step S121: YES), the processing unit 190 determines whether or not the end condition of the reinforcement learning is satisfied (step S131).
When it is determined that the condition for ending the reinforcement learning is not satisfied (step S131: NO), the processing unit 190 selects the next episode (step S141).
After the process of step S141, the process returns to step S101. In this case, the simulation unit 192 sets the state of the episode that is specified in the episode selected by the processing unit 190.
On the other hand, in step S131, if the processing unit 190 determines that the condition for ending the reinforcement learning is satisfied (step S131: NO), the learning device 100 ends the process in FIG.

As described above, the policy generation unit 193 generates a policy, which is a decision rule for behavior, based on a reward value, which is a value indicating an evaluation of the behavior of the control object 910 at a time step back within an episode representing a learning period. The behavior decision unit 191 decides the behavior of the control object based on the policy.

According to the learning device 100, it is expected that reinforcement learning can be performed more efficiently by going back in time steps within an episode.
For example, the learning device 100 may be able to avoid the occurrence of a factor that would cause the evaluation indicated by the reward value to deteriorate by going back to a time step in which the factor occurred that would cause the evaluation indicated by the reward value to deteriorate. This is expected to enable the learning device 100 to generate a measure that will improve the evaluation of the entire episode more quickly.

Furthermore, the reset determination unit 194 suspends the simulation when it is determined that a predetermined condition for suspending the simulation of the behavior of the control target 910 is satisfied. The measure generation unit 193 generates a measure based on a reward value calculated based on the simulation of the behavior of the control target 910.
According to the learning device 100, reinforcement learning can be performed more efficiently than in a case where the execution of an episode is continued even when the state in which policy updating is not expected or when the state in which policy updating is expected to progress slowly is reached.

Furthermore, the simulation unit 192 simulates the behavior of the controlled object 910 and calculates the state of the environment in which the controlled object behaves.
According to the learning device 100, it is expected that reinforcement learning can be performed more efficiently by going back in time steps in a simulation.

In addition, the reset decision unit 194 decides whether or not to interrupt the simulation based on a change in the cumulative reward value, which is the cumulative value of the instantaneous reward value.
This allows the reset decision unit 194 to decide whether to suspend the simulation based on not only the state at one time step but also on a change in state. In this respect, the learning device 100 is expected to be able to appropriately decide whether to suspend the simulation.

Furthermore, when the reset decision unit 194 determines, based on an increase or decrease in the cumulative reward value, that the evaluation indicated by the cumulative reward value has deteriorated below a predetermined threshold, it decides to interrupt the simulation.
According to the learning device 100, it is possible to determine whether or not to interrupt the simulation by the simple process of comparing the cumulative reward value with a threshold value.

Furthermore, the reset determination unit 194 updates the threshold value of the cumulative reward value so that the extent of deterioration in the evaluation indicated by the threshold value increases as the simulation progresses.
As described above, in the early stages of an episode or in the early stages of reinforcement learning, it is considered that there are many cases in which the reward value does not become large because learning has not progressed. In this case, it is expected that the reset decision unit 194 sets the threshold to a relatively small value to hasten the interruption of the execution of the episode, so that the action decision unit 191 will try various actions, and an action (or a series of actions) that will increase the reward value will be found at a relatively early stage. In this respect, it is expected that the learning device 100 can efficiently perform reinforcement learning.

Furthermore, the reset decision unit 194 decides to interrupt the simulation if the deterioration of the evaluation indicated by the cumulative reward value continues for a predetermined number of time steps or more, based on the increase or decrease in the cumulative reward value.
According to the learning device 100, it is possible to determine whether or not to interrupt the simulation by the simple process of counting the number of steps in which the cumulative reward value is continuously decreasing.

Further, the reset decision unit 194 decides whether or not to interrupt the simulation based on a value function value that is a predicted value of a cumulative reward value that is a cumulative value of instantaneous reward values.
This allows the reset decision unit 194 to decide whether to suspend the simulation based on not only the state at one time step but also on a change in state. In this respect, the learning device 100 is expected to be able to appropriately decide whether to suspend the simulation.

Furthermore, when reset decision unit 194 determines, based on an increase or decrease in the value function value, that the evaluation indicated by the value function value has deteriorated below a predetermined threshold value, it decides to interrupt the simulation.
According to the learning device 100, it is possible to determine whether or not to interrupt the simulation by the simple process of comparing the value function value with a threshold value.

Furthermore, the reset determination unit 194 updates the threshold value so that the extent of deterioration in the evaluation indicated by the threshold value of the value function value increases as the simulation progresses.
As described above, in the early stage of an episode or in the early stage of reinforcement learning, it is considered that there are many cases where the reward value does not become large because learning has not progressed. In this case, it is expected that the reset decision unit 194 sets the threshold to a relatively small value and interrupts the execution of the episode earlier, so that the action decision unit 191 tries various actions and finds an action (or a series of actions) that will increase the reward value at a relatively early point in time.
In this respect, the learning device 100 is expected to enable efficient reinforcement learning.

Furthermore, reset decision unit 194 decides to interrupt the simulation when the deterioration of the evaluation indicated by the value function value continues for a predetermined number of time steps or more, based on the increase or decrease in the value function value.
According to the learning device 100, it is possible to determine whether or not to interrupt the simulation by the simple process of counting the number of steps in which the value function value is continuously decreasing.

In addition, the reset decision unit 194 decides to interrupt the simulation when it determines that the current state of the environment in which the control object 910 is acting, as calculated in the simulation, is similar to one or more states that are pre-set as states requiring interruption to a certain extent or more.
The designer who designs learning device 100 only needs to prepare a sample of a state in which a simulation needs to be interrupted, and does not need to design rules for determining whether or not a simulation needs to be interrupted. In this respect, learning device 100 reduces the burden on the designer.

The combination of the reset determination unit 194 and the display unit 120 presents the state to the user. The combination of the reset determination unit 194 and the operation input unit 130 accepts a user operation that specifies a state that requires interruption.
According to the learning device 100, a sample of a state in which a simulation needs to be interrupted can be acquired in response to a user's designation. In this respect, the learning device 100 reduces the burden on the designer of the learning device 100. Furthermore, the user can reflect in the learning device 100 the user's own judgment as to whether or not a simulation needs to be interrupted.

Furthermore, the restart state determining unit 195 determines the state when the simulation is restarted.
As described above, it is considered that the policy update does not necessarily progress at the beginning of an episode. As described above for the learning device 100, the resumption state determination unit 195 can select not only the beginning of an episode but also an intermediate time step as the destination to go back in the episode, and therefore it is expected that the learning device 100 can perform reinforcement learning more efficiently.

The restart state determination unit 195 also selects one of the reset destination candidates based on a probability distribution set for the reset destination candidates, which are executed time steps whose states are stored as time steps at which the simulation can be restarted within the episode that was executed until the simulation was interrupted.

As mentioned above, when an event occurs that causes the reward value to be poorly evaluated, from the perspective of going back to a time step before that event occurred, it is possible to go back to as early a time step as possible in the episode (a time step closest to the beginning of the episode). On the other hand, from the perspective of reducing the number of learning repetitions at the beginning of the episode, it is possible to go back to as late a time step as possible among the time steps that have already been executed in the episode. In this way, there is a trade-off between improving the efficiency of reinforcement learning by avoiding states in which an event has occurred that causes the reward value to be poorly evaluated, and improving the efficiency of reinforcement learning by reducing the number of learning repetitions at the beginning of the episode.

Then, by having the restart state determination unit 195 probabilistically select one of the multiple reset destination candidates, it is possible to avoid restarting the execution of the episode from the beginning of the episode every time the execution of the episode is interrupted. Furthermore, if an event that would cause the reward value to be evaluated poorly has already occurred in the selected reset destination candidate, it is expected that by interrupting the episode one or more times, it will be possible to select a reset destination candidate before the event that would cause the reward value to be evaluated poorly occurred.

Furthermore, the reset determination unit 194 selects one of the multiple reset destination candidates in accordance with a uniform probability distribution.
According to the learning device 100, a reset destination candidate can be selected in response to a control object 910 for which the approximate number of time steps to go back to reach a state before the occurrence of an event that caused the evaluation indicated by the reward value to be poor is unknown.

The reset decision unit 194 also sets a probability distribution for multiple reset destination candidates so that the greater the deterioration in evaluation at the time the simulation is interrupted, which is indicated by an increase or decrease in the cumulative reward value, the more likely it is to select a reset destination candidate with a large number of time steps between the time the simulation is interrupted and the reset destination candidate, and selects one of the reset destination candidates according to the set probability distribution.

As mentioned above, it is particularly desirable to avoid a situation in which the cumulative reward value decreases rapidly. In order to increase the possibility of avoiding a situation in which the cumulative reward value decreases rapidly, it is possible to take a large number of time steps back to the time when the simulation was interrupted.
On the other hand, a situation where the cumulative reward value gradually decreases is less desirable than a situation where the cumulative reward value rapidly decreases. In this case, in order to reduce the repetition of learning at the beginning of an episode, it is conceivable to make the number of time steps back when the simulation is interrupted relatively small.
It is expected that the learning device 100 can efficiently perform reinforcement learning by the restart state determination unit 195 changing the reset destination candidates that are easy to select depending on the magnitude of the increase or decrease in the cumulative reward value.

The reset decision unit 194 also selects, from among the reset destination candidates, a time step in which the change in evaluation, indicated by an increase or decrease in the cumulative reward value, has turned from improvement to deterioration, and which corresponds to the time step with the fewest number of time steps between the time the simulation was interrupted, or a time step earlier than that.

In time steps after the time step in which the evaluation was the most recent maximally good evaluation when the simulation was interrupted, the cumulative reward value is continuously decreasing, and it is possible that an event has occurred that causes the evaluation indicated by the reward value to be a bad evaluation. By having the restart state determination unit 195 select a reset destination candidate that corresponds to the time step in which the evaluation was the most recent maximally good evaluation when the simulation was interrupted or an earlier time step, it is expected that the possibility of selecting a time step before the occurrence of an event that causes the evaluation indicated by the reward value to be a bad evaluation will increase.

10 is a diagram showing another example of the configuration of a learning device according to some embodiments of the present disclosure. In the configuration shown in FIG. 10, a learning device 610 includes a policy generator 611 and an action determiner 612.
In this configuration, the policy generating unit 611 generates a policy, which is a decision rule for behavior, based on a reward value, which is a value indicating an evaluation of the behavior of the controlled object in a state going back a time step in an episode representing a learning period. The behavior deciding unit 612 decides the behavior of the controlled object based on the policy.
The measure generating unit 611 corresponds to an example of a measure generating means, and the action deciding unit 612 corresponds to an example of an action deciding means.

It is expected that the learning device 610 can perform reinforcement learning more efficiently by going back in time steps within an episode.
For example, the learning device 610 may be able to avoid the occurrence of a factor that would cause the evaluation indicated by the reward value to deteriorate by going back to a time step in which the factor caused the evaluation indicated by the reward value to deteriorate. This is expected to enable the learning device 610 to generate a measure that will improve the evaluation of the entire episode more quickly.

The policy generation unit 611 can be realized, for example, by using the functions of the policy generation unit 193 in FIG. 1, etc. The action decision unit 612 can be realized, for example, by using the functions of the action decision unit 191 in FIG. 1, etc.

FIG. 11 is a diagram illustrating another example of a control system configuration according to some embodiments of the present disclosure.
In the configuration shown in FIG. 11, the control system 620 includes a measure generator 622 and an action determiner 623 .

In this configuration, the policy generating unit 622 generates a policy, which is a decision rule for behavior, based on a reward value, which is a value indicating an evaluation of the behavior of the controlled object in a state going back a time step in an episode representing a learning period. The behavior deciding unit 623 decides the behavior of the controlled object based on the policy.
The measure generating unit 622 corresponds to an example of a measure generating means, and the action deciding unit 623 corresponds to an example of an action deciding means.

It is expected that the control system 620 can perform reinforcement learning more efficiently by going back in time steps within an episode.
For example, the control system 620 may be able to avoid the occurrence of a factor that would cause the evaluation indicated by the reward value to deteriorate by going back to a time step in which the factor caused the evaluation indicated by the reward value to deteriorate. This is expected to enable the control system 620 to generate a measure that would improve the evaluation of the entire episode more quickly.

The policy generation unit 622 can be realized, for example, by using the functions of the policy generation unit 193 in FIG. 1, etc. The action decision unit 623 can be realized, for example, by using the functions of the action decision unit 191 in FIG. 1, etc.

12 is a diagram illustrating an example of a configuration of an input/output device according to some embodiments of the present disclosure. In the configuration illustrated in FIG. 12, an input/output device 630 includes a status presenting unit 631 and a status designation receiving unit 632.
With this configuration, the state presenting unit 631 presents to the user the state of the environment in which the controlled object acts. The state designation receiving unit 632 receives a user operation that designates a state in which the simulator that simulates the environment needs to be interrupted.
The status presenting unit 631 is an example of a status presenting unit, and the status designation receiving unit 632 is an example of a status designation receiving unit.

According to the input/output device 630, a sample of a state in which a simulation needs to be interrupted can be acquired in response to a user's designation. In this respect, the input/output device 630 reduces the burden on the designer of the input/output device 630. Furthermore, the user can reflect his/her own judgment as to whether or not a simulation needs to be interrupted in the execution of the simulation.
The state presenting unit 631 can be realized, for example, by using the functions of the reset determining unit 194 and the display unit 120 in Fig. 1. The state designation receiving unit 632 can be realized, for example, by using the functions of the reset determining unit 194 and the operation input unit 130 in Fig. 1.

FIG. 13 is a diagram showing an example of a processing procedure in a learning method according to some embodiments of the present disclosure. The learning method shown in FIG. 13 includes generating a strategy (step S611) and determining an action (step S612).

In generating a policy (step S611), the computer generates a policy, which is a decision rule for the behavior, based on a reward value, which is a value indicating an evaluation of the behavior of the controlled object at a time step back within the episode representing the learning period.
In determining an action (step S612), the computer determines an action of the controlled object based on a strategy.

According to the learning method shown in FIG. 13, it is expected that reinforcement learning can be performed more efficiently by going back a time step within an episode.
For example, according to the learning method shown in Fig. 13, it is possible to avoid the occurrence of a factor that deteriorates the evaluation indicated by the reward value by going back to the time step in which the factor occurred that deteriorates the evaluation indicated by the reward value. As a result, it is expected that the learning method shown in Fig. 13 can generate a policy that improves the evaluation of the entire episode more quickly.

FIG. 14 is a diagram illustrating an example of a computer configuration in accordance with at least some embodiments of the present disclosure.
In the configuration shown in FIG. 14, a computer 700 includes a CPU 710 , a main memory device 720 , an auxiliary memory device 730 , an interface 740 , and a non-volatile recording medium 750 .

Any one or more of the learning device 100, the control device 200, the learning device 610, the learning device 621, the control device 626, and the input/output device 630, or a part of them, may be implemented in the computer 700. In this case, the operation of each of the above-mentioned processing units is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, expands it in the main storage device 720, and executes the above-mentioned processing according to the program. The CPU 710 also secures a storage area corresponding to each of the above-mentioned storage units in the main storage device 720 according to the program. Communication between each device and other devices is executed by the interface 740 having a communication function and communicating according to the control of the CPU 710. The interface 740 also has a port for the non-volatile recording medium 750, and reads information from the non-volatile recording medium 750 and writes information to the non-volatile recording medium 750.

When the learning device 100 is implemented in a computer 700, the operations of the processing unit 190 and each of its units are stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, expands it in the main storage device 720, and executes the above-mentioned processing according to the program.

The CPU 710 also reserves a memory area for the memory unit 180 in the main memory device 720 in accordance with the program. Communication with other devices by the communication unit 110 is achieved by the interface 740 having a communication function and operating under the control of the CPU 710. Display of images by the display unit 120 is achieved by the interface 740 having a display device and displaying various images under the control of the CPU 710. Reception of user operations by the operation input unit 130 is achieved by the interface 740 having an input device and accepting user operations under the control of the CPU 710.

When the control device 200 is implemented in the computer 700, its operation is stored in the form of a program in the auxiliary storage device 730. The CPU 710 reads the program from the auxiliary storage device 730, expands it in the main storage device 720, and executes the above-mentioned processing according to the program.

The CPU 710 also allocates a memory area in the main memory device 720 for the control device 200 to perform processing according to the program. Communication between the control device 200 and other devices is performed by the interface 740, which has a communication function and operates according to the control of the CPU 710. Interaction between the control device 200 and a user is performed by the interface 740, which has an input device and an output device, presenting information to the user via the output device according to the control of the CPU 710, and accepting user operations via the input device.

When the learning device 610 is implemented in the computer 700, the operations of the policy generation unit 611 and the action decision unit 612 are stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, expands it in the main storage device 720, and executes the above-mentioned processing according to the program.

The CPU 710 also allocates a memory area in the main memory device 720 for the learning device 610 to perform processing according to the program. Communication between the learning device 610 and other devices is performed by the interface 740, which has a communication function and operates according to the control of the CPU 710. Interaction between the learning device 610 and a user is performed by the interface 740, which has an input device and an output device, presenting information to the user via the output device according to the control of the CPU 710, and accepting user operations via the input device.

When the learning device 621 is implemented in the computer 700, the operations of the policy generation unit 622 and the action decision unit 623 are stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, expands it in the main storage device 720, and executes the above-mentioned processing according to the program.

The CPU 710 also allocates a memory area in the main memory device 720 for the learning device 621 to perform processing according to the program. Communication between the learning device 621 and other devices is performed by the interface 740, which has a communication function and operates according to the control of the CPU 710. Interaction between the learning device 621 and a user is performed by the interface 740, which has an input device and an output device, presenting information to the user via the output device according to the control of the CPU 710, and accepting user operations via the input device.

When the control device 626 is implemented in the computer 700, its operation is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, expands it in the main storage device 720, and executes the above-mentioned processing according to the program.

The CPU 710 also allocates a memory area in the main memory 720 for the control device 626 to perform processing according to the program. Communication between the control device 626 and other devices is performed by the interface 740, which has a communication function and operates according to the control of the CPU 710. Interaction between the control device 626 and a user is performed by the interface 740, which has an input device and an output device, presenting information to the user via the output device according to the control of the CPU 710, and accepting user operations via the input device.

When the input/output device 630 is implemented in the computer 700, the operations of the state presentation unit 631 and the state designation reception unit 632 are stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads the program from the auxiliary storage device 730, expands it in the main storage device 720, and executes the above-mentioned processing according to the program.

The CPU 710 also allocates a storage area in the main memory device 720 for the I/O device 630 to perform processing according to the program. Communication between the I/O device 630 and other devices is performed by the interface 740, which has a communication function and operates according to the control of the CPU 710. Interaction between the I/O device 630 and a user is performed by the interface 740, which has an input device and an output device, presenting information to the user via the output device according to the control of the CPU 710, and accepting user operations via the input device.

Any one or more of the above-mentioned programs may be recorded on the non-volatile recording medium 750. In this case, the interface 740 may read the program from the non-volatile recording medium 750. The CPU 710 may then directly execute the program read by the interface 740, or may temporarily store the program in the main memory device 720 or the auxiliary memory device 730 and then execute it.

In addition, a program for executing all or part of the processing performed by learning device 100, control device 200, learning device 610, learning device 621, control device 626, and input/output device 630 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform processing of each part. Note that the term "computer system" here includes hardware such as an OS (Operating System) and peripheral devices.
Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs (Read Only Memory), and CD-ROMs (Compact Disc Read Only Memory), as well as storage devices such as hard disks built into computer systems. The above-mentioned program may be for realizing part of the above-mentioned functions, or may be capable of realizing the above-mentioned functions in combination with a program already recorded in the computer system.

　Although the embodiments of this disclosure have been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs that do not deviate from the gist of this disclosure.

Some or all of the above embodiments can be described as follows, but are not limited to the following:

(Appendix 1)
a policy generating means for generating a policy, which is a decision rule for the action, based on a reward value, which is a value indicating an evaluation of the action of the controlled object in a state going back a time step in an episode representing a learning period;
an action decision means for deciding an action of the control target based on the measure;
A learning device comprising:

(Appendix 2)
a reset decision means for suspending the simulation when it is determined that a predetermined condition for suspending the simulation of the behavior of the controlled object is satisfied,
The policy generating means generates the policy based on the reward value calculated based on the simulation.
2. A learning device as described in claim 1.

(Appendix 3)
3. The learning device according to claim 2, further comprising: a simulation means for performing the simulation to calculate a state of an environment in which the controlled object acts.

(Appendix 4)
The reset decision means decides whether or not to interrupt the simulation based on a change in a cumulative reward value, which is a cumulative value of an instantaneous reward value.
4. A learning device according to claim 2 or 3.

(Appendix 5)
The reset decision means decides to interrupt the simulation when it is determined that the evaluation indicated by the cumulative reward value has deteriorated below a predetermined threshold based on the increase or decrease of the cumulative reward value.
5. A learning device as described in claim 4.

(Appendix 6)
The reset determination means updates the threshold value so that a range of deterioration in the evaluation indicated by the threshold value of the cumulative reward value increases as the simulation progresses.
6. A learning device as described in appendix 5.

(Appendix 7)
The reset decision means decides to interrupt the simulation when a deterioration in the evaluation indicated by the cumulative reward value continues for a predetermined number of time steps or more based on an increase or decrease in the cumulative reward value.
7. A learning device according to any one of claims 4 to 6.

(Appendix 8)
The reset determination means determines whether or not to interrupt the simulation based on a value function value that is a predicted value of a cumulative reward value that is a cumulative value of an instantaneous reward value.
8. A learning device according to any one of claims 2 to 7.

(Appendix 9)
The reset decision means decides to interrupt the simulation when it is determined that the evaluation indicated by the value function value has deteriorated below a predetermined threshold based on an increase or decrease in the value function value.
9. A learning device as described in claim 8.

(Appendix 10)
the reset determination means updates the threshold value so that a range of deterioration in the evaluation indicated by the threshold value of the value function value increases as the simulation progresses.
10. The learning device according to claim 9.

(Appendix 11)
The reset decision means decides to interrupt the simulation when a deterioration in the evaluation indicated by the value of the value function continues for a predetermined number of time steps or more based on an increase or decrease in the value of the value function.
11. The learning device according to claim 9 or 10.

(Appendix 12)
the reset decision means decides to interrupt the simulation when it is determined that a current state of the environment in which the control target acts, calculated in the simulation, is similar to any one or more states that are preset as states requiring interruption to a certain degree or more;
12. A learning device according to any one of claims 2 to 11.

(Appendix 13)
a state presenting means for presenting a state of the environment to a user;
a state designation receiving means for receiving a user operation for designating a state in which the interruption is required;
13. The learning device of claim 12, further comprising:

(Appendix 14)
14. The learning device according to any one of appendices 2 to 13, further comprising a restart state determination means for determining a state of an environment in which the controlled object acts when resuming an interrupted simulation.

(Appendix 15)
the restart state determination means selects one of the reset destination candidates based on a probability distribution set for the reset destination candidates, which are executed time steps whose states are stored as time steps at which the simulation can be restarted within the episode that was executed until the simulation was interrupted;
15. A learning device as described in appendix 14.

(Appendix 16)
The restart state determination means selects one of a plurality of reset destination candidates in accordance with a uniform probability distribution.
16. A learning device as described in appendix 15.

(Appendix 17)
The restart state determination means sets a probability distribution for the plurality of reset destination candidates such that the greater the deterioration of the evaluation at the time of interruption of the simulation, which is indicated by an increase or decrease in an accumulated reward value that is an accumulated value of the instantaneous reward value, the easier it is to select a reset destination candidate having a larger number of time steps between the time of interruption of the simulation and the reset destination candidate, and selects one of the reset destination candidates according to the set probability distribution.
16. A learning device as described in appendix 15.

(Appendix 18)
The restart state determination means selects a reset destination candidate corresponding to a time step having the smallest number of time steps between the time of interruption of the simulation and the time step in which the change in the evaluation indicated by the increase or decrease in the cumulative reward value, which is the cumulative value of the instantaneous reward value, has turned from improvement to deterioration, among the reset destination candidates which are executed time steps whose states are stored as time steps at which the simulation can be restarted within the episode that was executed until the interruption of the simulation, or a time step earlier than that.
18. A learning device according to any one of appendices 14 to 17.

(Appendix 19)
A control device that performs control on a control target based on a policy obtained using the learning device according to any one of appendixes 1 to 18.

(Appendix 20)
A learning device and a control device are provided,
The learning device includes:
a policy generating means for generating a policy, which is a decision rule for the action, based on a reward value, which is a value indicating an evaluation of the action of the controlled object in a state going back a time step in an episode representing a learning period;
an action decision means for deciding an action of the control target based on the measure;
Equipped with
The control device controls a control target based on the policy obtained using the learning device.
Control system.

(Appendix 21)
A state presentation means for presenting to a user a state of an environment in which the controlled object acts;
a state designation receiving means for receiving a user operation for designating a state in which the simulator that simulates the environment needs to be interrupted;
An input/output device comprising:

(Appendix 22)
The computer
generating a policy, which is a decision rule for the action, based on a reward value, which is a value indicating an evaluation of the action of the controlled object, in a state going back a time step in the episode representing the learning period;
determining an action of the controlled object based on the policy;
A learning method that includes:

(Appendix 23)
On the computer,
generating a policy, which is a decision rule for the action, based on a reward value, which is a value indicating an evaluation of the action of the controlled object in a state going back in time steps within an episode representing a learning period;
determining an action of the controlled object based on the policy;
A recording medium storing a program for executing the above.

This application claims priority based on Japanese Patent Application No. 2023-103973, filed on June 26, 2023, the entire disclosure of which is incorporated herein by reference.

This disclosure may be applied to a learning device, a control system, an input/output device, a learning method, and a recording medium.

1, 620 Control system 100, 610, 621 Learning device 110 Communication unit 120 Display unit 130 Operation input unit 180 Memory unit 190 Processing unit 191, 612, 623 Action decision unit 192 Simulation unit 193, 611, 622 Measure generation unit 194 Reset decision unit 195 Resume state decision unit 200, 624 Control device 630 Input/output device 631 State presentation unit 632 State designation reception unit 910 Control target

Claims (20)

a policy generating means for generating a policy, which is a decision rule for the action, based on a reward value, which is a value indicating an evaluation of the action of the controlled object in a state going back a time step in an episode representing a learning period;
an action decision means for deciding an action of the control target based on the measure;
A learning device comprising: a reset decision means for suspending the simulation of the behavior of the controlled object when it is determined that a predetermined condition for suspending the simulation of the behavior of the controlled object is satisfied,
The policy generating means generates the policy based on the reward value calculated based on the simulation.
The learning device according to claim 1 . The learning device according to claim 2 , further comprising: a simulation means for performing the simulation to calculate a state of an environment in which the controlled object acts. The reset decision means decides whether or not to interrupt the simulation based on a change in a cumulative reward value, which is a cumulative value of an instantaneous reward value.
The learning device according to claim 2 or 3. The reset decision means decides to interrupt the simulation when it is determined that the evaluation indicated by the cumulative reward value has deteriorated below a predetermined threshold based on the increase or decrease of the cumulative reward value.
The learning device according to claim 4. The reset determination means updates the threshold value so that a range of deterioration in the evaluation indicated by the threshold value of the cumulative reward value increases as the simulation progresses.
The learning device according to claim 5 . The reset decision means decides to interrupt the simulation when a deterioration in the evaluation indicated by the cumulative reward value continues for a predetermined number of time steps or more based on an increase or decrease in the cumulative reward value.
A learning device according to any one of claims 4 to 6. The reset determination means determines whether or not to interrupt the simulation based on a value function value that is a predicted value of a cumulative reward value that is a cumulative value of an instantaneous reward value.
A learning device according to any one of claims 2 to 7. The reset decision means decides to interrupt the simulation when it is determined that the evaluation indicated by the value function value has deteriorated below a predetermined threshold based on an increase or decrease in the value function value.
The learning device according to claim 8. the reset determination means updates the threshold value so that a range of deterioration in the evaluation indicated by the threshold value of the value function value increases as the simulation progresses.
The learning device according to claim 9. The reset decision means decides to interrupt the simulation when a deterioration in the evaluation indicated by the value of the value function continues for a predetermined number of time steps or more based on an increase or decrease in the value of the value function.
The learning device according to claim 9 or 10. the reset decision means decides to interrupt the simulation when it is determined that a current state of the environment in which the control target acts, calculated in the simulation, is similar to any one or more states that are preset as states requiring interruption to a certain degree or more;
A learning device according to any one of claims 2 to 11. a state presenting means for presenting a state of the environment to a user;
a state designation receiving means for receiving a user operation for designating a state in which the interruption is required;
The learning device of claim 12 further comprising: The learning device according to claim 2 , further comprising: a restart state determination means for determining a state of an environment in which the controlled object acts when restarting an interrupted simulation. the restart state determination means selects one of the reset destination candidates based on a probability distribution set for the reset destination candidates, which are executed time steps whose states are stored as time steps at which the simulation can be restarted within the episode that was executed until the simulation was interrupted;
The learning device according to claim 14. The restart state determination means selects one of a plurality of reset destination candidates in accordance with a uniform probability distribution.
The learning device according to claim 15. A learning device and a control device are provided,
The learning device includes:
a policy generating means for generating a policy, which is a decision rule for the action, based on a reward value, which is a value indicating an evaluation of the action of the controlled object in a state going back a time step in an episode representing a learning period;
an action decision means for deciding an action of the control target based on the measure;
Equipped with
The control device controls a control target based on the policy obtained using the learning device.
Control system. A state presentation means for presenting to a user a state of an environment in which the controlled object acts;
a state designation receiving means for receiving a user operation for designating a state in which the simulator that simulates the environment needs to be interrupted;
An input/output device comprising: The computer
generating a policy, which is a decision rule for the action, based on a reward value, which is a value indicating an evaluation of the action of the controlled object, in a state going back a time step in the episode representing the learning period;
determining an action of the controlled object based on the policy;
A learning method that includes: On the computer,
generating a policy, which is a decision rule for the action, based on a reward value, which is a value indicating an evaluation of the action of the controlled object in a state going back in time steps within an episode representing a learning period;
determining an action of the controlled object based on the policy;
A recording medium storing a program for executing the above.

Images