JP2004178492A - Plant simulation method using enhanced learning method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve a problem that a garbage incineration plant or the like shows cumbersome behavior, different behavior is shown even in the same plant, and behavior changes even by the long-term depletion of the plant. <P>SOLUTION: A plant simulation method using an enhanced learning method comprises the steps of: setting a value function to the initial state (S1); using a previously prepared actual plant operation data (S2); obtaining the amount of a state by conducting model calculations to the amount of inputted data operation by using a previously prepared process model (S3); calculating remuneration by using the amount of the operation, the calculated amount of a state, and the actual plant operation data (S4); learning a policy for maximizing a profit being the total of the remuneration by conducting enhanced learning based on the remuneration calculated for a plurality of parameters (S5); obtaining the profit expected in the future derived from an action in a given state as the value function; conducting simulation based on the learned parameter obtained by using the value function. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a plant simulator, an operation method, and a program thereof. In particular, the present invention relates to a simulation method for simulating (simulating) the operation of a plant such as a refuse incineration plant or the like, which requires skilled operator operation, or a plant whose state changes due to long-term operation.
[0002]
[Prior art]
In a refuse incinerator, the refuse to be burned (waste) is composed of various ratios according to its properties. Such variation in the components of the garbage is particularly remarkable when the waste is general waste such as household waste. For this reason, the behavior of the refuse incinerator itself is complicated. In addition, because of the influence of the dispersion of the garbage components and the habit of the operator's operation, the garbage incinerator has a habit in the behavior even if the plant changes or every incinerator in the same plant. There are many. Therefore, considerable skill is required to reliably operate such a waste incinerator. Even in other conventional plants other than refuse incinerators, there are those requiring skill in operation.
[0003]
In other words, in such a plant, the operator needs to be highly skilled because the operator must read and operate the relationship between a large number of controlled state variables by operating a large number of control variables.
[0004]
It is conceivable to use a training simulator to perform such plant operation training, but in order to construct a practical training simulator, it is the basis for simulating the complex behavior of the plant. A plant model is required. For example, Patent Literature 1 describes a method for generating a plant simulation model using a transfer function and PID control. Further, for example, Patent Literature 2 discloses a method of matching an actual plant operation with a plant simulator using an error.
[0005]
Further, in such a plant, there are those in which the state quantities (temperature, pressure, etc.) of the process are difficult to measure, and those in which complicated processes that are actually occurring cannot be fully grasped. In some cases, it is difficult to grasp the type and timing of maintenance required for the plant, or how much time is left until the end of its useful life. Due to these uncertain factors, it was necessary to set an operation period and perform maintenance, etc., with a large margin for safety.
[0006]
In the field of learning algorithms, a technique called reinforcement learning is known. The reinforcement learning method is one of unsupervised learning methods. In an environment, an autonomous agent, which is a learning subject, receives a reward or penalty obtained from the environment. Is a learning method that determines a policy or policy (policy) and maximizes the value that is the expected return (expected value of reward) given by the policy. The environment is complicated and uncertain. It has the feature that the agent can be learned even if it is present (for example, Non-Patent Document 1). As an example using this reinforcement learning method, Patent Document 3 discloses a method for optimizing the route of a dredger, but does not disclose an example used for a process model of a plant or the like.
[0007]
[Patent Document 1]
JP-A-7-64610
[Patent Document 2]
JP-A-10-207507
[Patent Document 3]
JP-A-10-253602
[Non-patent document 1]
The Institute of Electrical Engineers of Japan, GA / Neuro-based learning method and its application research expert committee, “Learning and its algorithm”, Morikita Publishing, August 28, 2002, p. 155-164
[0008]
[Problems to be solved by the invention]
In actual garbage incineration plants, in addition to exhibiting complex behavior depending on the nature of the garbage, if the plants are different, even if the plants are constructed in the same way, or if the incinerators are in the same plant, Also exhibit a different behavior (that is, a behavior in which the operation habit of the operator is multiplied by the habit of the plant and the incinerator), and the behavior also changes with the aging of the plant. When the behavior is complicated as described above, the complexity is not sufficiently represented by a simple model. In addition, the simulation of the behavior of the plant and the incinerator at the time of execution of the simulation under the aging of the actual machine has not been performed. In order to reflect such a state of the actual machine to the process model, it is necessary to sequentially analyze the operation of the plant in order to reduce uncertainties of the plant.
[0009]
[Means for Solving the Problems]
In order to solve such a problem, the present invention uses a reinforcement learning algorithm in an apparatus or the like for simulating the operation of a plant. In the present invention, a space corresponding to the operation amount and the state amount of the plant is set as an environment for reinforcement learning.
[0010]
That is, the present invention provides (a) a step of setting a value function to an initial state, and (b) a model using a process model created in advance for a certain manipulated variable using previously prepared actual plant operation data. Performing a calculation to obtain a state quantity; (c) calculating a reward using the manipulated variable, the calculated state quantity, and actual plant operation data; (d) steps (b) and (c). ) Is repeated for each of the plurality of parameters in the parameter space that defines the relationship between the operation amount and the state amount in the actual plant operation data; and (e) reinforcement learning based on the reward calculated for the plurality of parameters. Learning a measure for maximizing the profit that is the sum of the rewards by using the value function; and (f) obtaining the obtained value function. Used to provide a simulation method of plant operation comprising the steps of performing a simulation based on the learning parameters obtained.
[0011]
Here, the value function is a function used as a learning index, and is a kind of an evaluation function used in the reinforcement learning method. The actual plant operation data is data including an operation amount and a state amount in a state where an actual plant is operating. The operation amount is an amount to be variously adjusted or changed when the plant is operated. For example, in the case of a refuse incinerator, although details will be described later, the amount of refuse is input. The model calculation is a calculation for obtaining a state quantity by calculation according to an operation amount, and is based on a physical theoretical calculation model formula, an empirical formula by fitting numerical values, a theoretical formula based on a work hypothesis, and the like. Its value can be adjusted by some parameter. The state quantity is a numerical value serving as a monitoring item when the plant is operating. For example, the temperature at a certain position of the furnace. When specifically considering the subject that learns inside the computer, consider the subject called “agent”. This implements an agent in a computer as a learning subject in a sense generally used in the field of the reinforcement learning method, but also operates as a subject for performing a simulation later. This is not a function as a single piece of hardware consisting of arithmetic means and storage means, but a function realized in a virtual space at least partially held in a main storage device of a computer. It works as if it works. The reward is an amount that is added when a value function used as a learning index is rewritten in accordance with a state update, is assigned to the state, and represents an incentive given to a learning agent. The parameter space is a mathematical space where parameter values can be taken. This parameter is a parameter of an equation that approximates the measurement data. In general, the present invention causes the agent function to learn the parameter space corresponding to the measurement data as an “environment” in the reinforcement learning method. Here, the computer has at least a calculation unit, a storage unit, an input unit, and an output unit. That is, according to the present invention, the above-described steps (a) to (f) are executed in a computer including an arithmetic unit, a storage unit, an input unit, and an output unit.
[0012]
According to this simulation method, it is possible to cause a computer (agent) to behave as seen in the operation amount and state amount of this plant, and it is possible to simulate an actual plant using the computer.
[0013]
Also, in the present invention, (g) a step of receiving a virtual operation amount; (h) a step of performing the model calculation according to the virtual operation amount based on the learned parameter; And a step of outputting the obtained state quantity.
[0014]
The virtual manipulated variable is a value used as an input value in a simulation, for example, as if the user is an input to an actual plant, or transmitted as a manipulated variable by another computer assuming a computer that executes the method as a plant. Value. If the input is made by the user, the input is made from an input device such as a keyboard and a mouse. The model calculation reflects the result of the learning at that time, and is a model calculation that simulates an actual plant of the plant by reinforcement learning. The state quantities obtained as a result are output using the present simulation method instead of actually operating the actual plant. This output is output to the display device for the user or to another computer. Thus, the simulator of the present invention can be used for training of plant operation by an operator.
[0015]
In the present invention, the step (j) of comparing the parameter values based on the policy after learning at different times and outputting the result of the comparison is performed, wherein the previously prepared actual plant operation data is temporal data. A simulation method of a plant operation that further includes the above method can be provided.
[0016]
The time-dependent data is data of the operation amount and the state amount at a plurality of times. Since the policy after learning gives the optimum parameters at that time, the optimum values for the parameters of the plant model at different times are obtained. By comparing these parameters with each other, a temporal change in the parameters of the plant model can be simulated as a temporal change in the state of the plant. In addition, the data at the time when the optimum value of this parameter is different reflects the actual phenomenon of the plant. Thus, the state of the plant can be analyzed using the parameters of the model.
[0017]
In the present invention, the value function is defined as a value which is a function of the state s and the action a. Function, whereby the reinforcement learning can be performed by the Q-Learning method. If the value function serving as the evaluation function in the reinforcement learning method is an action value function Q (s, a) based on the state s and the action a, the Q-Learning method can be performed.
[0018]
Further, in the present application, the virtual operation amount input means is, for example, an input means provided in an appropriate computer terminal, and is an input means for receiving an input from an operator who receives training. The simulated state quantity output means is an appropriate data output means or display means, for example, as if the trained operator using the virtual manipulated variable input means is in the operating state of the plant. Display means for displaying.
[0019]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0020]
[Embodiment 1]
In the present embodiment, a simulator for simulating the operation of a plant will be described.
[0021]
(Outline of the actual plant)
FIG. 1 illustrates a situation where the simulator of the present invention is used. The refuse incinerator 1 is an example of an actual plant that requires considerable training for operation. Such a plant is operated by an operator (operator) 21 performing various operations. Usually, the operator 21 operates the operator console 22 connected to the plant operation device 2 connected to the refuse incinerator 1. The plant operation device 2 presents various state quantities necessary for the operation of the waste incinerator 1 to the operator through the operator console 22, and the operator grasps the state of the waste incinerator 1 and responds to the state of the waste incinerator 1. To operate the refuse incinerator 1 appropriately by changing the setting of some amount of operation.
[0022]
The monitoring center 4 is connected to the plant operating device 2 via a network 3 such as a dedicated line, monitors the state of the incinerator 1, and is installed to provide a service for supporting operation management of the plant. Therefore, the monitoring center 4 can collect various operation quantities and state quantities of the refuse incinerator 1. The monitoring center 4 has functions such as risk prediction 5, operation diagnosis 6, abnormal failure diagnosis 7, remaining life prediction 8, and operation training 9, in order to support the operation management of the refuse incinerator 1.
[0023]
(Overview of simulator)
FIG. 2 illustrates the configuration of the simulator according to the case of the driving training device that performs the driving training 9. The operator console 22 is connected to a computer serving as a process simulator 92 via the network 3. The process simulator 92 is disposed, for example, at the monitoring center 4, but may be located at any location as long as it can be connected to the network 3. The process simulator 92 includes an actual plant operation data file 94. The actual plant operation data file 94 may be in any location accessible from the process simulator 92. In the actual plant operation data file 94, measured data of the operation amount and the state amount of the waste incinerator 1 are stored in chronological order through the network 3.
[0024]
(Contents of actual measurement data)
The actual measurement data of the manipulated variables and state variables are various values in an actual waste incinerator. Examples of the operation amount are a feeder speed related to the amount of dust, a damper opening degree of a blowing fan, a primary air temperature, a primary air pressure, a smoke exhaust damper opening degree, and the like, which are artificially operated amounts. . The state quantity is, for example, a furnace temperature, a furnace pressure, an exhaust gas temperature, an exhaust gas amount, an exhaust gas pressure, an exhaust gas component (oxygen amount, carbon monoxide amount, and the like), and is a quantity representing a plant state. In addition, a quantity representing the state of the surrounding environment (for example, temperature and humidity) determined by the weather may be a state quantity or an operation quantity. Since these do not directly represent the state of the plant and do not actively operate, they are not considered here, but they can be added to the simulation. In an actual garbage incinerator, the operation is performed while these numerical data change every moment, and the measured data is stored in a plant actual machine operation data file 94.
[0025]
Here, in an actual refuse incinerator, the relationship between the manipulated variable and the state variable has a functional relationship, but the state variable does not depend only on the manipulated variable at that time. For example, the state quantity at a certain time acts as an initial value for the subsequent state quantity and affects the state quantity at the subsequent time. Further, the manipulated variable at a certain time affects a subsequent state variable with a certain delay, for example, an input variable for a first-order delay element. Further, there is not always a deterministic relationship between the operation amount and the state amount. This is because what is operated as the operation amount does not always completely correspond to the operation amount, but is operated with a certain width (for example, if the feeder speed related to the dust input amount is constant). Even so, the actual amount of garbage input will not always be constant.) This is also because the state quantity is influenced by climatic conditions (temperature and humidity in the air) and the nature of the garbage (the type and composition of the garbage, the amount of water contained, and the like). Further, one of the reasons is that a phenomenon (for example, a combustion process) which must be treated as a stochastic phenomenon in terms of engineering also influences.
[0026]
(Agent operation)
FIG. 2 The agent 96 has a reinforcement learning function 962, and changes the state of the agent 96 itself according to the environment according to the reinforcement learning method. The substance of the agent 96 is a virtual thing existing on the computer, and the state of the agent 96 itself is changed by some parameter. The agent 96 performs the reinforcement learning based on the actual plant operation data. In the present embodiment, the state of the agent 96 is determined by a model including a mathematical expression that can express the operation of the plant by a numerical input / output relationship. In addition to the models including mathematical formulas, the present invention also includes models including only numerical values not including mathematical formulas (for example, a simple matrix indicating a correspondence relationship between a vector of operation amount data and a vector of state amount data). Including. In the present embodiment, this mathematical expression model is referred to as a process model 964. Having the process model 964 include some adjustable parameters allows the process model 964 to be tuned to the actual plant. The process model 964 can be changed by changing numerical values (parameters) of the plant model data file 98 that characterize the plant model. In this adjustment operation, when changing the state of the agent 96 itself, the parameter space for adjusting the process model 964 becomes an environment in which the agent 96 learns according to the reinforcement learning method in the present embodiment.
[0027]
(How to build a simulator)
FIG. 3 illustrates a method of constructing the simulator of the present embodiment using the reinforcement learning method. The simulator is constructed using a computer by causing the agent 96 to perform reinforcement learning.
[0028]
First, at the beginning of learning, a process model is created and initialized (step S1). The process model can be performed by a model formula that expresses its characteristics in consideration of various physical phenomena. Although the transfer function of the first-order lag element is described in FIG. 3, other theoretical considerations such as an approximation formula using an autoregressive model, a model expressing the effect of turbulence of combustion by a probability density function, A model created based on the work hypothesis can be used arbitrarily, and may be a model expressed as a combination of a plurality of physical phenomena. When the model is simple and cannot reproduce the complexity that actually occurs in the plant, it is possible to reproduce the unavoidable fluctuations seen in the actual plant by adding an appropriate probability term. The probability term does not need to be particularly considered in the learning stage. In the present embodiment, a model capable of expressing the relationship between the operation amount and the state amount is used. However, as a whole, even if the relationship cannot be modeled, it can be described as an input / output relationship that can be expressed numerically. Good.
[0029]
The initialization is to set the agent 96 to the initial state and to set the action value function Q to be used later to the initial state. The state of the agent 96 is determined by parameters that determine the operation of the agent 96. For example, if the first-order lag element in FIG. 3 is used as a model in which the amount of garbage processed is the manipulated variable and the furnace temperature is the state variable, the state of the agent 96 is determined by a set of the time constant T and the gain G. Is determined. At this stage, the range of the parameter to be considered for the model to be used and the step size of the value are also determined at this stage.
[0030]
Next, data for causing the agent 96 to learn is appropriately sampled from the actual plant operation data file 96 (step S2). Sampling is performed because data with sufficient accuracy is sufficient for an environment for reinforcement learning.
[0031]
Next, model calculation is performed using the process model at that time (step S3). Usually, since the actually measured state quantity is obtained according to the sampled operation quantity of the plant, the state quantity is calculated by calculation based on the process model at that time for the same operation quantity as the actual measurement.
[0032]
Next, a reward is calculated (step S4). For this purpose, for a set of the operation amount and the state amount obtained in the actual plant, the operation amount and the state amount by the above calculation are considered, and a set of the operation amount and the calculation state amount is considered. In general, there is a difference between the actual state quantity of the plant and the state quantity output by the agent 96, even for the same operation quantity. This difference can be attributed to the incompleteness of the model (too simple or the parameter settings not being optimized) as well as the accuracy limits of the manipulated variables in the actual plant and the stochastic behavior of the plant. And fluctuation factors, or errors in the measurement of state quantities. The reward can be determined according to, for example, a difference (residual error) between the actual measurement and the calculated state quantity. For example, taking the absolute value of the residual for all of the sampled measured data,
Equation 1
r = C- | Calculated state quantity-Actual state quantity |
(C: positive constant)
Thus, a reward element r can be determined for each sampling data. The reward data required for subsequent learning is reward for a certain range of parameters in the parameter space (in this example, a range in which the values of the time constant T and the gain G can be taken). A reward for the set of values needs to be sought. In the value of each parameter, for example, the above-mentioned reward element r is added to the sum of the parameters included in the set of parameter values, and data is added in order to prevent the reward of the parameter having a large number of data from becoming an apparently large numerical value. Normalize by dividing by a number. In addition to this example, the reward can be appropriately determined, and an appropriate numerical value representing the difference between the calculation and the actual value for the agent 96 can be used.
[0033]
Then, based on the reward calculated for each parameter in the parameter space, Q-learning, which is a type of reinforcement learning, is performed (step S5). Here, the reason why Q-Learning is adopted is that, in addition to the state of the agent, the action taken by the agent is also subject to learning. Since the behavior is also a target of learning, for example, it is possible to build a simulator with less time delay with good tracking of actual data of a plant that changes relatively quickly with respect to the optimization calculation. . For this reason, in the present embodiment, Q-learning is used, but in the present invention, another reinforcement learning method such as a TD learning method may be used. For example, if the TD learning method is used, the action is not evaluated because the state of the next time is determined based on the state of the agent at that time, and when the actual data of the plant is early, an error occurs due to the time difference. However, when the number of actions is large (for example, when the number of parameters is large, such as when a model formula is created by a higher-order polynomial), the amount of calculation can be reduced and the number of learning iterations can be increased. The error may be small. According to the actor-critic method, which is another example of reinforcement learning, there is an advantage that the amount of calculation can be reduced and stochastic action selection can be performed, similarly to the TD learning method.
[0034]
In Q-Learning, an action value function Q (s, a) is considered for a certain state s and an action a determined for each s. The action value function Q (s, a) is an evaluation value (value) in the Q-Learning method for obtaining an evaluation value using the state s and the action a. In this example, the state s is a state that the agent 96 is currently taking in the two-dimensional space of the time constant T and the gain K. The initial value of Q (s, a) is, for example, 0, but can be generally an arbitrary value (step S0).
[0035]
(About Q-Learning)
When Q-Learning is started (step S50), first, the action a is stochastically determined in the state s based on the policy π at that time (step S52). As a result, the next state s' is determined. Here, the policy π is a function of the state s and the action a. When the policy π allows a plurality of actions, one is stochastically selected from the plurality of actions using an appropriate random number. The action evaluation function Q (s, a) of the action a from the state s is recalculated based on the maximum Q value of the possible actions a 'in the state s'.
[0036]
Next, Q is updated according to equation 2 (step S54). At this time, a reward r in the state s, a discount rate γ (a value of 0 or more and less than 1 determined in advance in step S0), and a learning rate a (a value greater than 0 and 1 or less in advance determined in step S0) are used.
Equation 2
Q (s, a) ← (1-α) Q (s, a) + α [r + γmaxQ (s ′, a ′)]
[0037]
An example of the action a is an action of moving to a new state in any of the eight directions if the time constant T and the gain G can move in eight directions, up, down, left, and right, from the point of the current state. The maximum value (max) is the maximum value of the actions a 'that can be taken for the state s'.
[0038]
When the state changes to s' as a result of the transition according to the action a (step S56), the action evaluation function Q for the action a in the state s is strengthened according to the equation 2, and this is repeated (here, The repetition is not shown), and an optimum state is obtained based on the given actual measurement data. If the discount rate γ is selected to be greater than or equal to 0 and less than 1, the value of Q does not diverge even when it is repeated. As a result, an optimum parameter is obtained by using repetition.
[0039]
Here, whether or not the optimization has actually been performed and the reinforcement learning has been completed is determined based on the fact that the state has not transitioned. Since the action “a” does not change the state, the action a that does not change the state is the optimum action at that point in time. Such a set of parameters in the state s (the time constant T and the gain K in this example) is stored in the plant model data file 98 (FIG. 2) as appropriate. The agent can always recall the operation of the plant at the time when the parameter was created by calling the parameter from the plant model data file 98. As a result, an optimal state (set of parameter values) is determined (S56), and the step of Q-Learning ends (S58).
[0040]
The optimum parameters used for the model are obtained as described above. When the actual plant operation data is updated after the optimization, new sampling is performed and the above process is executed again.
[0041]
In the present embodiment, even when the actual plant operation data is newly updated, the model parameters can be learned at any time using the updated data. This is because the reinforcement learning method itself is a learning method in which learning is performed empirically, and can cope with a case where data is sequentially updated. In the present embodiment, optimization using only two parameters is shown for the sake of illustration. However, as shown in the above example of the actual measurement data of the refuse incinerator, in an actual plant, a large number of operation quantities and state quantities are large. is there. The above advantages of the present invention are extremely effective when trying to simulate an actual plant that needs to be optimized by a more complicated and parameterized formula.
[0042]
In addition, if Q-learning is adopted among the reinforcement learning methods that provide such an advantage, the behavior itself in the parameter space becomes an evaluation target. Is adjusted so as to exhibit a behavior close to that of the actual machine, and a simulator that is closer to the actual one can be constructed.
[0043]
[Example 1]
In the present embodiment, an embodiment in which a plant operation training apparatus is configured by simulating an actual plant with the simulator of the present invention will be described. The operator who receives the training uses the operator console 22 (virtual manipulated variable input means, simulated state quantity output means) of FIG. Send a signal. The operation of the agent 96 is set according to the parameters called from the plant model data file 98, and the agent 96 outputs a signal simulating the behavior of the incinerator 1 in response to a signal from the operator console 22.
[0044]
The output from the agent 96 of the process simulator 92 is displayed on the operator console 22 as if it were an actual operation result of the incinerator 1. This makes it possible to train operators without actually operating the incinerator 1.
[0045]
Here, a case where the actual state of the plant has fluctuation will be described. Fluctuations include stochastic behaviors in which the actual value of the manipulated variable cannot be actually grasped and in which the phenomenon itself fluctuates. The fluctuation distribution and temporal characteristics (temporal fluctuations) Most of them are characterized by: For example, it is possible to express cumulative data over a long period of time with a probability density function (for example, a normal distribution) for a phenomenon exhibiting spectral characteristics such as 1 / f fluctuation. In addition, as an event whose properties fluctuate stepwise at a certain time, it is also possible to assume a normal distribution for the width of the step of the fluctuation and a Poisson distribution for the occurrence probability of the fluctuating event. In this way, it is possible to give a property appropriately mathematically modeled as a stochastic event to a plant operation quantity (for example, garbage properties), a plant model parameter, or a plant state quantity. it can.
[0046]
By appropriately simulating the fluctuation of the actual plant in this way, it is possible to perform an appropriate training for the operator, which is closer to the actual plant. In addition, for the purpose of training, this probability can be set to a probability different from the actual probability and used to enhance the effect of training.
[0047]
[Example 2]
In the present embodiment, an embodiment in which an abnormality diagnosis device is configured in combination with the simulator of the present invention will be described. The parameters such as the time constant T and the gain G obtained by simulating the actual plant not only determine the state of the agent 96 after learning, but also reflect the actual state of the plant. Since this data is stored in the plant model data file 98, information on the operating state of the plant can be obtained by monitoring a change in this value. It is possible to monitor, albeit indirectly, a situation inside the plant, which is unlikely to surface during normal operation. Thus, the abnormality of the plant can be diagnosed using the state quantity other than the measurable state even during the operation of the plant.
[0048]
[Example 3]
In this embodiment, an example will be described in which a driving diagnostic device is configured in combination with the simulator of the present invention. The driving diagnostic device is a device for examining future driving. In other words, according to the actual measurement data of the plant obtained at a certain point in time and the internal state of the plant obtained from the abnormality diagnosis apparatus of the second embodiment based on the data, an operation plan is established for the subsequent operation of the plant. Help.
[0049]
By clarifying the relationship between the change in the time constant T or the gain G of the plant and the manipulated variable or state variable, the relationship between the state inside the plant and the state variable that can be operated or measured from outside is clarified. From this relationship, if the condition of the operation amount that gives the most suitable operation method for the plant is determined, the determination of the operation method that reflects the actual situation better than the case where the quality of the operation method is managed only by the state quantity, That is, driving diagnosis can be performed. In order to do this, a step of obtaining the optimum value of the model parameter of the plant by numerical calculation or the like in advance, a step of obtaining the value of the model parameter in an actual operating condition by the apparatus of the present invention, and a step of further optimizing the model parameter A step of comparing the value with the value of the model parameter is used.
[0050]
[Example 4]
In this embodiment, when the thickness of the waste incinerator decreases due to temporal wear or the like, the temperature difference between the inside and the outside of the furnace is measured as a state quantity. Furthermore, the state quantity time is expressed by an interpolation formula, and the space of values on the curve is defined as a variable parameter space. By simulating an actual plant while updating data at any time by the simulator of the present invention, a change over time of the slope is obtained as a result of reinforcement learning. Since the slope on the curve represents the rate of decrease in the furnace thickness in the past operation history of the actual plant, the life of the furnace when the same operation is continued can also be predicted. That is, the life of the furnace of the plant can be analyzed. Also, within the range where the simulator operates properly, the virtual operation amount is variously changed, and the change in the inclination is observed to analyze how the life of the furnace depends on the operating conditions. It is possible to select the operation method that is used.
[0051]
As described above, if the measurement data of the state quantity and the manipulated variable of the present invention is made time-dependent, and the equation is used for the state quantity of the plant, it is possible to analyze the aging of the plant and the remaining service life.
[0052]
【The invention's effect】
By using the reinforcement learning method in the process simulation in the plant, the operation data can be sequentially reflected. This makes it possible to manufacture a simulator that learns the actual behavior of the plant in consideration of long-term fuel property changes and aging. In addition, by having a process model for each plant and each incinerator, and learning with the reinforcement learning method for each, it is possible to faithfully simulate the behavior at that time taking into account the habit of each plant and each incinerator. . As a result, it is possible to perform operation training that simulates the complicated behavior of the refuse incinerator. Further, if this simulator is used, it is possible to simulate the behavior of a plant by examining an optimal operation method and an optimized operation method that minimizes risk in advance. Further, it is possible to obtain an analysis tool that can reflect measured data at any time and analyze the behavior of the plant.
[Brief description of the drawings]
FIG. 1 is a configuration diagram illustrating a situation in which a simulator of the present invention is used.
FIG. 2 is a configuration diagram illustrating a configuration of a simulator when performing driving training according to the embodiment of the present invention.
FIG. 3 is a flowchart illustrating a method for constructing a simulator according to the present embodiment using a reinforcement learning method.
FIG. 4 is a flowchart illustrating a learning method of a Q-Learning method, which is an example of a reinforcement learning method.
[Explanation of symbols]
1 Garbage incinerator
21 Operator
22 Operator console
2 Plant operation equipment
3 network
4 Monitoring center
92 Process Simulator
94 Plant operation data file
96 agents
98 Plant model data file
962 Reinforcement learning function
964 Process Model

Claims (8)

(A) initializing the value function;
(B) using a previously prepared plant actual machine operation data to execute a model calculation with a process model created in advance for a certain operation amount to obtain a state amount;
(C) calculating a reward using the manipulated variable, the calculated state variable, and the actual plant operation data;
(D) repeating step (b) and step (c) for each of a plurality of parameters in a parameter space that defines a relationship between an operation amount and a state amount in actual plant operation data;
(E) learning, by using the value function, a measure for maximizing the profit, which is the sum of the rewards, by performing reinforcement learning based on the rewards calculated for the plurality of parameters;
(F) performing a simulation based on learned parameters obtained using the obtained value function. (G) receiving a virtual operation amount;
(H) performing the model calculation according to the virtual operation amount based on the learned parameters;
The method according to claim 1, further comprising: (i) outputting a state quantity obtained by the model calculation. The previously prepared plant actual machine operation data is time-lapse data,
The method of simulating plant operation according to claim 1, further comprising: (j) comparing values of parameters based on the policy after learning at different times and outputting a result of the comparison. The value function is an action value function that is a function of the state s and the action a in which the value of adopting the action a in the state s is defined as the value of a future expected profit for a certain action in a given state. The simulation method according to claim 1, wherein the reinforcement learning is performed by a Q-learning method. A program for causing a computer to execute each step according to claim 1. A simulator comprising a computer having arithmetic means, storage means, input means, and output means,
Storing plant operation data prepared in advance in the storage means,
The calculating means is
Using the actual machine operation data, for each of a plurality of parameters in the parameter space that defines the relationship between the manipulated variable and the state quantity in the actual plant machine operation data, model calculation is performed using a process model created in advance for a certain manipulated variable. Executing to obtain a state quantity and repeatedly calculating a reward using the manipulated variable, the calculated state quantity, and the actual plant operation data, and enhancing based on the reward calculated for a plurality of parameters. A strategy for maximizing the profit that is the sum of rewards by performing learning is learned using the value function, and the learned policy is stored in the storage unit.
A plant operation simulator for executing a simulation by executing the model calculation in accordance with an input amount received by the input means, based on a parameter determined from the learned policy. Virtual manipulated variable input means;
Simulated state quantity output means,
The computer calls the learned policy, executes the model calculation according to the virtual operation amount from the virtual operation amount input unit, based on a parameter determined from the learned policy,
7. The plant operation simulator according to claim 6, wherein training of an operator of the plant is performed by outputting a simulated state quantity corresponding to the virtual manipulated variable to the simulated state quantity output means. The plant operation data prepared in advance is time-lapse data, and the values of parameters based on the policy after learning at different times are compared, and the result of the comparison is output to analyze the state of the plant. A plant operation simulator according to claim 6.

Images