CN115167150B - Batch process two-dimensional off-orbit strategy staggered Q learning optimal tracking control method with unknown system dynamics - Google Patents

Abstract

The two-dimensional off-track strategy interleaved Q learning optimal tracking control method for batch processes with unknown system dynamics belongs to the field of industrial process control technology. The specific steps are as follows: Step 1: Establish a nonlinear state space equation for a batch process with unknown dynamics; Step 2: Extend the tracking error as a state variable to the performance index, and construct the performance index of the two-dimensional nonlinear system; Step 3: Define the expression of the two-dimensional optimal value function and the Q function according to the relationship between the performance index and the value function; Step 4: Introduce the interleaved Q iteration algorithm; Step 5: Construct a model network to obtain the initial weights of the neural network; Step 6: Construct a judgment network and a behavior network to obtain the final control strategy. This method solves the problem of over-reliance on system models in traditional industrial processes. At the same time, when the initial parameters of the system are unknown, the industrial process can also proceed smoothly, greatly improving production efficiency and reducing computing costs.

Description

Two-dimensional off-track strategy staggered Q-learning optimal tracking control method for batch processes with unknown system dynamics

Technical Field

The present invention belongs to the technical field of industrial process control, and in particular relates to a two-dimensional off-track strategy interleaved Q-learning optimal tracking control method for a batch process with unknown system dynamics.

Background technique

At a time when product upgrades and demand are increasing dramatically, industrial processes are numerous and extremely complex. In order to meet the different needs of more users, industrial production should gradually tend to high-quality and small-scale production trends. For this reason, batch processes are used to solve various problems in industrial production with their many advantages, and at the same time have become one of the hot research topics in the field of control today. Compared with continuous processes, batch processes have the characteristics of rapidity, low cost, repeated operation, diversity, and multiple stages. However, most batch processes usually rely on the model of the controlled process to design the controller of the system.

However, in practical applications, the batch process is not just a simple production process as we imagine. It involves knowledge from many disciplines and contains many internal and external factors. Due to the diversity of batch processes, various problems will inevitably arise during the operation. Moreover, excessive reliance on the process model during operation will inevitably lead to a decrease in the performance of the process model, making it more difficult to establish an accurate system model of the batch process and resulting in a significant reduction in product accuracy. To this end, in view of the above situation, a two-dimensional off-track strategy interleaved Q learning optimal tracking control method for batch processes with unknown dynamics is designed to study the control problem of the batch process without relying on the process model and the initial parameters of the system.

Summary of the invention

The present invention proposes a two-dimensional off-track strategy interleaved Q-learning optimal tracking control method for a batch process with unknown system dynamics. The method can effectively solve the problem that the system cannot be accurately modeled, reduce the model dependence of the system, and continuously learn only by relying on data in the time direction and batch direction. Even when the initial parameters of the system are unknown, the optimal control strategy can still be obtained, thereby improving the control and tracking performance of the system and accelerating the convergence speed.

The present invention is achieved through the following technical solutions:

The present invention proposes a two-dimensional off-track strategy staggered Q learning optimal tracking control method for a batch process with unknown system dynamics. First, a nonlinear state space equation of a batch process with unknown dynamics is established. Secondly, the tracking error is extended to the performance index as a state variable to construct the performance index of the two-dimensional nonlinear system. According to the relationship between the performance index and the value function, the expression of the two-dimensional optimal value function and the Q function is defined, and the staggered Q iteration algorithm is introduced. Then, a model network, a judgment network and a behavior network are constructed. In this process, the weights in each layer of the neural network are continuously learned and updated through the data in the batch direction, and finally we find the optimal control strategy. The present invention can effectively solve the problem that the system cannot be accurately modeled, greatly reduce the excessive dependence on the system model, and at the same time adopt the Q iteration learning method. When the initial parameters of the system are unknown, the optimal control strategy can be obtained quickly and accurately, the optimal performance of the system is improved, and the convergence speed is accelerated.

Step 1: Establish the nonlinear state space equations of the batch process with unknown dynamics;

The batch process with unknown system dynamics is represented by the nonlinear radial state-space equations, which are expressed as follows:

y(t+1,k)＝f((y(t,k))+g(y(t,k))u(t,k) (1)

Where t represents the time direction, k represents the batch direction, y(t,k) ^∈Rn represents the system state, u(t,k) ^∈Rm represents the system control input, f((y(t,k)) ^∈Rn) represents the f function of y(t,k), g(y(t,k))∈Rn ^×m represents the g function of y(t,k), R represents a real matrix, and n and m represent the appropriate dimensions of the real matrix R;

Step 2: Extend the tracking error as a state variable into the performance index and construct the performance index of the two-dimensional nonlinear system;

The performance index of a two-dimensional nonlinear system is defined as:

Where T is the prediction time domain, y _r (t, k) represents the expected state of the system, u(t, k-1) represents the control input of the system k-1 batch at time t, and R represents the weighted matrix of the corresponding dimension of the control input;

Extended state Q ₁ = ^CT QC, Q ₁ represents the extended state The weight matrix of the corresponding dimension, I represents the identity matrix of appropriate dimension; then the performance index of the two-dimensional nonlinear system can be redefined as:

At the same time, the extended state of the system k batch at time t+1 It can be expressed as:

in, , represents a matrix of appropriate dimension between the expected setpoints of the system, and 0 represents a zero matrix of appropriate dimension;

Step 3: Based on the relationship between the performance index and the value function, define the expressions of the two-dimensional optimal value function and the Q function; according to formula (3), define the two-dimensional optimal value function and the two-dimensional optimal Q function as follows:

in,

By minimizing the right side of formula (5), the optimal control strategy is solved to generate the optimal value function Based on the necessary conditions for optimality, the optimal control strategy u ^* (t, k) can be obtained by taking the derivative of u(t, k); therefore,

When the control strategy u(t,k) reaches the optimal value u ^* (t,k), the optimal value function is equivalent to the optimal Q function, that is:

Step 4: Introduce the interleaved Q iteration algorithm;

Design a Q iterative derivation algorithm, according to formula (8) the optimal Q function It can also be expressed as follows:

Then the optimal control strategy can be described as:

In order to find the optimal solution in (9) and (10), we first set the initial value Q ⁰ () = 0, then the control strategy u ⁰ (t, k) is expressed as:

At the same time, the Q function can be updated as:

When i＝1,2…, the control strategy u ⁰ (t,k) and Q function The iteration will be completed;

and

Step 5: Build a model network and obtain the initial weights of the neural network;

In practical applications, The result is uncertain and it is difficult to get an accurate value; therefore, we will use a neural network to approximate the dynamics of system (4) and calculate the result of;

For the model network, a three-layer neural network is used to identify A nonlinear system of structures, where Represent the number of neuron nodes in the input layer, hidden layer and output layer respectively; let the weight matrix between the input layer and the hidden layer be W _m1 , and the weight matrix between the hidden layer and the output layer be W _m2 ;

Assume that the extended state of the nonlinear system and the control input u(t,k) of the system are known, then the output of the model network can be expressed as:

in,

The error function of the training model network is defined as:

The objective function is minimized:

Among them, _Em (t,k) is the square error of the model network;

The weight update adopts a gradient-based adaptive method, namely:

Wherein, W _m1 (j+1) represents the weight matrix between the input layer and the hidden layer of the model network at the time of iteration j+1, W _m1 (j) represents the weight matrix between the input layer and the hidden layer of the model network at the time of iteration j, W _m2 (j+1) represents the weight matrix between the hidden layer and the output layer of the model network at the time of iteration j+1, W _m2 (j) represents the weight matrix between the hidden layer and the output layer of the model network at the time of iteration j, α _m >0 represents the learning rate of the model network, and j represents the number of internal cycles in the model network;

Once training is complete, the weights of the model network remain unchanged;

Step 6: Construct the judgment network and behavior network to obtain the final control strategy;

The critic network is used to approximate the Q function Structure, the output structure of a three-layer neural network used to judge the network is shown as follows:

Among them, the weight matrix between the input layer and the hidden layer is W _c1 , and the weight matrix between the hidden layer and the output layer is W _c2 ;

Based on formula (14), when the Q function iterates to the i+1th time, the expected value function can be obtained as follows:

The prediction error is defined as:

The objective function can be minimized as:

Where E _c(i+1) (t,k) is the square error when the evaluation network iterates to the i+1th time;

Based on the gradient adaptive rule, the weight matrix can be updated as follows:

In the process of updating the weight matrix, W _c1 (j+1) represents the weight matrix between the input layer and the hidden layer of the evaluation network at the time of iteration j+1, W _c1 (j) represents the weight matrix between the input layer and the hidden layer of the evaluation network at the time of iteration j, W _c2 (j+1) represents the weight matrix between the hidden layer and the output layer of the evaluation network at the time of iteration j+1, W _c2 (j) represents the weight matrix between the hidden layer and the output layer of the evaluation network at the time of iteration j, α _c >0 represents the learning rate of the evaluation network, and j represents the number of inner cycles of weight update in the evaluation network;

Execute the network to set the system status As input, to approximate the control strategy An adaptive linear neural network with a three-layer structure is selected and applied to the execution network;

Where, W _a1 represents the weight matrix of the input layer and the hidden layer, and W _a2 represents the weight matrix of the hidden layer and the output layer;

The prediction error of the behavioral network is defined as:

In order to minimize the objective function, the square error of the behavior network is defined as:

Among them, E _a (t,k) is the square error of the judgment network;

The gradient descent algorithm is used to update the weight matrix in the behavior network;

Among them, _Wa1 (j+1) represents the weight matrix between the input layer and the hidden layer of the behavior network iterated to time j+1, _Wa1 (j) represents the weight matrix between the input layer and the hidden layer of the behavior network iterated to time j, _Wa2 (j+1) represents the weight matrix between the hidden layer and the output layer of the behavior network iterated to time j+1, _Wa2 (j) represents the weight matrix between the hidden layer and the output layer of the behavior network iterated to time j, _αa ＞0 and j are the learning rate and the number of internal cycles of the behavior network, respectively.

Therefore, the final control strategy can be obtained as:

in, The result is:

In summary,

The steps to implement the two-dimensional off-orbit strategy staggered Q learning optimal tracking control method are as follows:

1. Collect system data x(t,k) through the behavior control strategy u(t,k) and store it.

2. Initialize the weights W _m1 and W _m2 in the neural network.

3. Training model network:

(1) By using the measurement data method, the weights W _m1 and W _m2 are trained until e _m (t, k) ≤ ψ _m (t, k), where ψ _m (t, k) is a very small positive number.

(2) Obtain the trained weights W _m1 (j+1), W _m2 (j+1), and set k=0.

4. Staggered iteration:

(1) Set the initial value of the Q function Q ⁰ (·) = 0. When the iteration parameter i = 0, further calculate the control strategy u ⁰ (t, k).

(2) Calculated by formulas (19) and (20) and u(t,k-1)), and use formula (23) to update the weights W _c1 and W _c2 to achieve the purpose of training the judgment network.

(3) Approximately, by formula (24) The weights _Wa1 and _Wa2 are updated once using an adaptive gradient descent algorithm for training the behavior network.

(4) If not satisfied: Where ψ _Q > 0, then return to step 4 (2). If it is satisfied, the final control strategy is obtained.

5. Set t=t+1 and return to step 4 to continue.

The advantages and effects of the present invention are:

The present invention aims at the problem of the difficulty of modeling batch processes with unknown system dynamics and the complex characteristics of unknown system initial parameters, and proposes a two-dimensional off-track strategy interleaved Q learning optimal tracking control method for batch processes with unknown system dynamics; this invention can solve the optimal control strategy by only using the data measured in the batch and time directions of the injection molding process, which is different from the previous method of accurately modeling such batch processes, and is more different from the traditional one-dimensional model. Under the framework of two-dimensional theory, the optimal tracking control of the batch process by the two-dimensional off-track strategy interleaved Q learning algorithm is realized. While establishing the state space model, the actual output tracking error and the deviation of the increment are introduced, and a multi-degree-of-freedom design method is provided for the subsequent controller, and the tracking performance of the controller is guaranteed; no precise modeling process is required, which greatly reduces the model dependence of the system and reduces the calculation cost; when solving the controller gain, the mutual connection between the time direction and the batch direction is taken into account, and the convergence speed is accelerated and the optimal performance is enhanced while obtaining a more accurate control strategy; and the control algorithm does not need an initial allowable control strategy to obtain the optimal value, which accelerates the speed of industrial production and improves the efficiency of industrial production.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 shows the weight training results of the model network;

Figure 2 shows the weight test results of the model network;

Figure 3 shows the weight results of the evaluation network under different batch processes;

Figure 4 shows the weight results of the behavioral network under different batch processes;

Figure 5 is the output tracking curve under the two-dimensional off-track strategy interleaved Q learning algorithm;

Figure 6 shows the control input curve under the two-dimensional off-track strategy interleaved Q learning algorithm;

Detailed ways

In order to further illustrate the present invention, the present invention is described in detail below with reference to the accompanying drawings and examples, but they should not be construed as limiting the protection scope of the present invention.

Embodiment 1:

In the process of injection molding, the injection stage is the first step of the entire injection molding cycle. The time is calculated from the time when the mold starts to close the mold for injection until the mold cavity is filled to about 95%, which is generally about 1-5 seconds. Although the injection time is very short, it is one of the important factors affecting product quality. The more reasonable the control arrangement of the injection time, the more helpful it is to achieve ideal filling of the melt in the mold cavity, and it is of great significance to improve product quality and reduce process errors. Therefore, this paper studies the approximate optimal tracking control of the injection speed in the injection stage of injection molding based on the staggered Q learning algorithm with a dimensional off-track strategy.

After collecting a large amount of experimental data, the open-loop dynamic characteristics between injection velocity (IV) and valve opening (VO) in the actual injection molding process can be described as follows:

Among them, IV(t+1,k) is the injection speed of batch k at time t+1; IV(t,k) is the injection speed of batch k at time t; IV(t-1,k) is the injection speed of batch k at time t-1; VO(t,k) is the valve opening of batch k at time t; VO(t-1,k) is the valve opening of batch k at time t-1.

definition:

According to formula (29) and formula (30), the state space model of the injection stage can be established:

Where, ρ(t,k)=0.001x ₁ (t,k)+0.6.

After repeated experiments, the parameters of the injection phase were determined to be Q ₁ = Q ₂ = diag [6,6] and R = 1. However, in the actual batch production process, the initial parameters of the system are also difficult to obtain. Therefore, in the following study, we proposed a two-dimensional off-track strategy interleaved Q learning algorithm to design the optimal controller for the batch process.

First, a model network with 7 input layers, 20 hidden layers and 6 output layers is established. The initial weight is arbitrarily selected between -0.1 and 0.1, and the learning rate is set. Next, 1000 sets of data are selected to train the model network. After the training, the weights in the network remain unchanged. Secondly, for the judgment network, we select a neural network with 6 input layers, 10 hidden layers and 1 output layer. In addition, the learning rate is set to, and the initial weight is still arbitrarily selected between -0.1 and 0.1. Finally, a behavioral network with 6 input layers, 10 hidden layers and 1 output layer is designed, and the learning rate of the network is set to, and the initial weight is arbitrarily selected between -0.1 and 0.1.

Figure 1 and Figure 2 represent the weight test graph and weight training graph of the model network, respectively. In addition, in the process of simulating the model network, we set the training sample ratio to 0.8, that is, Figure 1 is the training result obtained by the first 80% weight samples, and Figure 2 is the result graph obtained by testing the weight training samples in Figure 1 with the last 20% weight samples. Therefore, it can be seen from the above two figures that, whether it is the weight training graph or the test graph, in the 100 randomly selected sets of weight data, the weight prediction value curve and the measured value curve are basically consistent, and both have achieved good results.

Figures 3 and 4 represent the weight curves of different batches from the hidden layer to the output layer in the judgment network and the behavior network, respectively. In order to show a good control effect, we selected the 4th, 7th, 10th, and 20th batches as representatives. It can be seen that with the increase of batches, the training effect of each weight curve is getting better and better, and each weight curve has reached the optimal value in the 10th batch, indicating that the algorithm proposed in this paper has a very good optimization effect. In subsequent batches, each weight can also reach the optimal value, proving the effectiveness and feasibility of the algorithm.

Figures 5 and 6 show the control input and batch process output curves, where the output setting value of the two-dimensional system is 40 mm/s. As can be seen from Figure 5, in the 4th batch, the output of the batch process is far from the setting value required by industrial production, but as the number of learning times increases, the error between the output value and the setting value becomes smaller and smaller. This shows that the tracking effect of the system is getting better and better, and the effect of weight training is also gradually improving. After the 10th batch, the system output reached the setting value, indicating that the control algorithm is feasible and effective.

In summary, the present invention takes the control design of the dynamic batch process with unknown system as an example to verify the effectiveness and feasibility of the control method proposed by the present invention; this invention can solve the optimal control strategy by only using the data measured in the batch and time directions of the injection molding process, which is different from the previous method of accurately modeling such batch processes; different from the traditional one-dimensional model, no accurate modeling process is required, which greatly reduces the model dependence of the system and reduces the calculation cost; when solving the controller gain, the mutual connection between the time direction and the batch direction is taken into account, and the convergence speed is accelerated and the optimal performance is enhanced while obtaining a more accurate control strategy; and the control algorithm does not require an initial allowable control strategy to obtain the optimal value, which accelerates the speed of industrial production and improves the efficiency of industrial production. The simulation results of the injection stage show that with the increase of batches, the final control strategy converges to the optimal value, and the control effect and tracking effect gradually improve; therefore, the proposal of this invention method provides a new design solution for the control problem of the dynamic batch process with unknown system, which can reduce the system modeling and calculation cost while ensuring the system tracking control effect.

The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims (1)

1. The batch process two-dimensional off-orbit strategy staggered Q learning optimal tracking control method with unknown system dynamics is characterized by comprising the following steps of:

The method comprises the following specific steps:

step one: establishing a batch process nonlinear state space equation with unknown dynamics;

The batch process with unknown system dynamics is represented by a nonlinear radial state space equation, which takes the following form:

y(t+1,k)＝f((y(t,k))+g(y(t,k))u(t,k) (1)

Where t represents the time direction, k represents the batch direction, y (t, k) e R ⁿ represents the system state, u (t, k) e R ^m represents the system control input, f ((y (t, k))eR ⁿ represents the f function of y (t, k), g (y (t, k))eR ^n×m represents the g function of y (t, k), R represents the real matrix, n and m represent the appropriate dimensions of the real matrix R;

Step two: the tracking error is used as a state variable to be expanded into a performance index, and the performance index of the two-dimensional nonlinear system is constructed;

the performance indexes of the two-dimensional nonlinear system are defined as follows:

Wherein T is a prediction time domain, y _r (T, k) represents a desired state of the system, u (T, k-1) represents a control input at a time T of a batch of the system k-1, and R represents a weighting matrix of a corresponding dimension of the control input;

Let the extended state Q ₁＝C^TQC,Q₁ is represented as an extended stateA weighting matrix of the corresponding dimension is provided,I represents an identity matrix of appropriate dimension; the performance index of the two-dimensional nonlinear system can be redefined as:

At the same time, the expansion state of system k batch t+1 time Can be expressed as:

Wherein, Y _r(t+1,k)＝θy_r (t, k), θ represents a matrix of appropriate dimension between the desired settings of the system, and 0 represents a zero matrix of appropriate dimension;

Step three: according to the relation between the performance index and the value function, defining an expression of a two-dimensional optimal value function and a Q function;

According to formula (3), a two-dimensional optimal value function and a two-dimensional optimal Q function are defined as follows:

Wherein,

Solving the optimal control strategy to produce an optimal value function by minimizing the right measurement of equation (5)Based on the optimality requirement, the optimal control strategy u ^* (t, k) can be obtained by deriving u (t, k); thus, the first and second substrates are bonded together,

When the control strategy u (t, k) reaches the optimal value u ^* (t, k), the optimal value function is equivalent to the optimal Q function, i.e.:

Step four: introducing an interleaving Q iteration algorithm;

designing a Q iterative derivation algorithm, optimizing Q function according to formula (8) Can also be expressed in the following form:

The optimal control strategy can be described as:

To find the optimal solution in equations (9) and (10), first, an initial value Q ⁰ (·) =0 is set, and then the control strategy u ⁰ (t, k) is expressed as:

while the Q function may be updated as:

control strategy u ⁰ (t, k) and Q function when i=1, 2 … The iteration will be completed;

And

Step five: constructing a model network to obtain initial weights of the neural network;

in the practical application of the present invention, Is uncertain and it is difficult to obtain an accurate value; therefore we will approximate the dynamics of the system (4) using a neural network and calculateResults of (2);

For model networks, a three-layer neural network is used to identify a model with a model Nonlinear system of structures, whereinThe number of neuron nodes in the input layer, the hidden layer and the output layer are respectively represented; let the weight matrix between the input layer and the hidden layer be W _m1, and the weight matrix between the hidden layer and the output layer be W _m2;

Assume the extended state of a nonlinear system And the control inputs u (t, k) of the system are known, the output of the model network can be expressed as:

Wherein,

Defining an error function of the training model network as:

Objective function minimization:

Wherein E _m (t, k) is the square of the error of the model network;

The weight updating adopts a gradient-based self-adaptive method, namely:

Wherein W _m1 (j+1) represents a weight matrix between the model network input layer and the hidden layer iterated to the moment j+1, W _m1 (j) represents a weight matrix between the model network input layer and the hidden layer iterated to the moment j, W _m2 (j+1) represents a weight matrix between the model network hidden layer and the output layer iterated to the moment j+1, W _m2 (j) represents a weight matrix between the model network hidden layer and the output layer iterated to the moment j, alpha _m > 0 represents a learning rate of the model network, and j represents the number of internal loops in the model network;

Once training is completed, the weights of the model network remain unchanged;

step six: constructing a judging network and a behavior network to obtain a final control strategy;

the evaluation network is used to approximate the Q function The structure, an output structure of a three-layer neural network used for evaluation network is represented as follows:

Wherein, the weight matrix between the input layer and the hidden layer is W _c1, and the weight matrix between the hidden layer and the output layer is W _c2;

Based on equation (14), when the Q function iterates to the i+1 th, the expected value function can be obtained by:

Defining a prediction error as:

The objective function can be minimized as:

Wherein E _c(i+1) (t, k) is the square of the error in the evaluation network from iteration to the (i+1);

based on the gradient-based adaptive rules, the weight matrix may be updated as follows:

In the process of updating the weight matrix, W _c1 (j+1) represents the weight matrix between the input layer and the hidden layer of the evaluation network iterated to the moment j+1, W _c1 (j) represents the weight matrix between the input layer and the hidden layer of the evaluation network iterated to the moment j, W _c2 (j+1) represents the weight matrix between the hidden layer and the output layer of the evaluation network iterated to the moment j+1, W _c2 (j) represents the weight matrix between the hidden layer and the output layer of the evaluation network iterated to the moment j, alpha _c > 0 is the learning rate of the evaluation network, and j is the number of internal loops of weight updating in the evaluation network;

Executing the network to bring the system state As input, to approximate the control strategySelecting an adaptive linear neuron network with a three-layer structure to be applied to an execution network;

Wherein W _a1 represents the weight matrix of the input layer and the hidden layer, and W _a2 represents the weight matrix of the hidden layer and the output layer;

the prediction error of the behavioural network is defined as:

to achieve objective function minimization, the square error of the defined behavior network is:

Wherein E _a (t, k) is the square of the error of the evaluation network;

adopting a gradient descent algorithm to update a weight matrix in the behavior network;

Wherein W _a1 (j+1) represents a weight matrix between the behavioural network input layer and the hidden layer iterated to the moment j+1, W _a1 (j) represents a weight matrix between the behavioural network input layer and the hidden layer iterated to the moment j, W _a2 (j+1) represents a weight matrix between the behavioural network hidden layer and the output layer iterated to the moment j+1, W _a2 (j) represents a weight matrix between the behavioural network hidden layer and the output layer iterated to the moment j, and alpha _a > 0 and j are respectively the learning rate and the internal circulation times of the behavioural network;

thus, a final control strategy may be obtained as:

Wherein, The result of (2) is:

In view of the above-mentioned, it is desirable,

。