CN114819093A - Method and Apparatus for Policy Optimization Using Memristor Array-Based Environment Models - Google Patents

Abstract

A strategy optimization method and strategy optimization device using a dynamic environment model based on a memristor array. The method includes: acquiring a dynamic environment model based on a memristor array; performing multiple predictions at multiple times according to the dynamic environment model and an object strategy, and obtaining a data sample set including the optimization cost of the object strategy corresponding to multiple times; A collection of samples to perform policy search using the policy gradient optimization algorithm to optimize the object policy. The method uses the dynamic environment model based on the memristor array to generate a data sample set, realizes the long-term dynamic programming based on the dynamic environment model, and then uses a more stable algorithm such as the policy gradient optimization algorithm for policy search, which can effectively optimize the object policy.

Description

Method and Apparatus for Policy Optimization Using Memristor Array-Based Environment Models

technical field

Embodiments of the present disclosure relate to a policy optimization method and policy optimization apparatus using a memristor array-based dynamic environment model.

Background technique

Artificial Neural Network (ANN) has a wide range of applications in the modeling of dynamic systems. However, long-term task planning using traditional artificial neural networks remains a challenge due to the lack of ability to model uncertainty. The inherent randomness (uncertainty) of real systems - process noise and approximation errors introduced by data-driven modeling can cause long-term estimates of artificial neural networks to deviate from the actual behavior of the system. Probabilistic models provide a way to address uncertainty, and these models enable people to use the model's predictions to make informed decisions while being cautious about the uncertainty of those predictions.

SUMMARY OF THE INVENTION

At least one embodiment of the present disclosure provides a strategy optimization method using a memristor array-based dynamic environment model, including: acquiring a memristor array-based dynamic environment model; The second prediction is performed to obtain a data sample set including the optimization cost of the target strategy corresponding to multiple moments; based on the data sample set, a policy gradient optimization algorithm is used to perform a strategy search to optimize the target strategy.

For example, in the strategy optimization method provided by an embodiment of the present disclosure, acquiring a dynamic environment model includes: acquiring a Bayesian neural network, where the Bayesian neural network has a weight matrix obtained by training; The weight matrix obtains the corresponding multiple target conductance values, and maps the multiple target conductance values to the memristor array; the state and the latent input variables corresponding to the time t of the dynamic system are input to the weight-mapped memristor as the input signal. Through the memristor array, the state and hidden input variables at time t are processed according to the Bayesian neural network, and the output signal corresponding to the processing result is obtained from the memristor array, and the output signal is used to obtain the time t of the dynamic system. +1 for the predicted results.

For example, in the strategy optimization method provided by an embodiment of the present disclosure, the expression of the dynamic environment model is s _{t +1} =f(s _t , at ; W, ε), where s _t is the state of the dynamic system at time t, a _t is the action of the object policy at time t, W is the weight matrix of the Bayesian neural network, ε is the additive noise corresponding to the memristor array, and s _t+1 is the prediction result of the dynamic system at time t+1 ; the action of the target strategy at time t at _t = π(s _t ; Wπ), π represents the function of the target strategy, Wπ represents the strategy parameter, the weight matrix W of the Bayesian neural network satisfies the distribution W～q(W), plus The additive noise ε is the additive Gaussian noise ε～N(0,σ ² ).

For example, in the strategy optimization method provided by an embodiment of the present disclosure, the multiple times include time 1 to time T arranged in order from early to late, and multiple predictions are performed at multiple times according to the dynamic environment model and the object strategy to obtain The data sample set including the optimization cost of the object strategy corresponding to multiple times, including: for any time t-1 from time 1 to time T, the execution action a _t-1 obtained by the object strategy, by a _t-1 = π(s _t-1 ; Wπ) obtains the action a _{t-1 of the object strategy at time t-1} ; according to the dynamic environment model s _t =f(s _t-1 , at _-1 ; W, ε), the time is calculated state s _t at the next time t after t-1 and obtain the cost ct corresponding to the state s _t at time _t , thereby obtaining the cost sequence {c ₁ ,c ₂ ,...,c from time 1 to time t _t }, obtain the optimization cost J _t-1 at time t based on the cost sequence, where 1≤t≤T; obtain the data sample set from time 1 to time T {[a ₀ ,J ₀ ],…,[a _{T- 1} , J _T-1 ]}.

来获得。For example, in the policy optimization method provided by an embodiment of the present disclosure, the expected value of the cost c _t at time t is E[c _t ], then the optimization cost at time t can be obtained by

to obtain.

For example, in the strategy optimization method provided by an embodiment of the present disclosure, the cost also includes the cost change caused by accidental uncertainty and the cost change caused by cognitive uncertainty, and the accidental uncertainty is caused by hidden input variables , the epistemic uncertainty is caused by the intrinsic noise of the memristor array.

来获得，σ(η,θ)为偶然不确定性和认知不确定性的函数，η表示偶然不确定性，θ表示认知不确定性。For example, in the policy optimization method provided by an embodiment of the present disclosure, the optimization cost at time t passes through

to obtain, σ(η, θ) is a function of contingent uncertainty and epistemic uncertainty, where η represents contingent uncertainty and θ represents epistemic uncertainty.

For example, in the strategy optimization method provided by an embodiment of the present disclosure, the state s _t at time t is calculated and obtained according to the dynamic environment model s _t =f(s _t-1 , at _-1 ; W, ε) and the corresponding The cost c _t of the state s _t at time t includes: sampling the latent input variable z from the p(z) distribution; inputting the samples and the state s _{t-1 at time t-1} into the memristor after weight mapping For the predicted state s _t , the cost _ct = c(s _t ) _is obtained.

For example, in the policy optimization method provided by an embodiment of the present disclosure, the policy gradient optimization algorithm includes the REINFORCE algorithm, the PRO algorithm, or the TPRO algorithm.

At least one embodiment of the present disclosure also provides a strategy optimization device using a memristor array-based dynamic environment model, including: an acquisition unit configured to acquire the memristor array-based dynamic environment model; The environment model and the object strategy perform multiple predictions at multiple times, and obtain a data sample set including the optimization cost of the object strategy corresponding to multiple times; the strategy search unit is configured to use the strategy gradient optimization algorithm to search for the strategy based on the data sample set. to optimize the object strategy.

Description of drawings

In order to explain the technical solutions of the embodiments of the present disclosure more clearly, the accompanying drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, rather than limit the present disclosure. .

1A shows a schematic flowchart of a strategy optimization method based on a dynamic environment model of a memristor array provided by at least one embodiment of the present disclosure;

Fig. 1B shows a schematic flowchart of step S101 in Fig. 1A;

FIG. 2A shows a schematic structure of a memristor array;

2B is a schematic diagram of a memristor device;

2C is a schematic diagram of another memristor device;

Figure 2D shows a schematic diagram of mapping the weight matrix of a Bayesian neural network to a memristor array;

Fig. 3 shows a schematic flowchart of step S102 in Fig. 1A;

FIG. 4 shows a schematic diagram of an example of a policy optimization method provided by at least one embodiment of the present disclosure;

FIG. 5 shows a schematic block diagram of a strategy optimization apparatus using a dynamic environment model based on a memristor array provided by at least one embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present disclosure. Obviously, the described embodiments are some, but not all, embodiments of the present disclosure. Based on the described embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present disclosure.

Unless otherwise defined, technical or scientific terms used in this disclosure shall have the ordinary meaning as understood by one of ordinary skill in the art to which this disclosure belongs. As used in this disclosure, "first," "second," and similar terms do not denote any order, quantity, or importance, but are merely used to distinguish the various components. Likewise, words such as "a," "an," or "the" do not denote a limitation of quantity, but rather denote the presence of at least one. "Comprises" or "comprising" and similar words mean that the elements or things appearing before the word encompass the elements or things recited after the word and their equivalents, but do not exclude other elements or things. Words like "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "Down", "Left", "Right", etc. are only used to represent the relative positional relationship, and when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

In model-free deep reinforcement learning (Deep Reinforcement Learning), the agent usually needs to perform a lot of interactive trial and error with the real environment, and the data efficiency is not high, so it cannot be applied to real tasks with high trial and error costs. Model-based deep reinforcement learning can use data more efficiently. In model-based deep reinforcement learning, the agent first learns the dynamic environment model from the historical experience of interacting with the real dynamic environment (such as state transition data collected in advance), and then interacts with the dynamic environment model to obtain suboptimal policies. .

The model-based reinforcement learning method learns an accurate dynamic environment model. Using this model when training the agent does not need to interact with the real environment too many times. The agent can "imagine" the feeling of interacting in the real environment. It can greatly improve data efficiency and is suitable for practical physical scenarios where the cost of data acquisition is high; at the same time, the dynamic environment model can predict the unknown state of the environment, generalize the cognition of the agent, and can also be used as a new data source to provide Contextual information to aid decision-making, which in turn can alleviate the exploration-exploitation dilemma. When modeling a real environment, the inherent randomness (uncertainty) of the environment - process noise and approximation errors introduced by data-driven modeling can cause the long-term estimates of artificial neural networks to deviate from the actual behavior of the system. Probabilistic models provide a way to address uncertainty, and these models enable making informed decisions using the model's predictions while being cautious about the uncertainty of those predictions.

The inventor found that: Bayesian Neural Network (BNN) is a probabilistic model that puts neural network in a Bayesian framework, which can describe complex random patterns; and, Bayesian with hidden input variables Neural network (BNN with latent input variables, BNN+LV) can describe complex random patterns through the distribution on latent input variables (accidental uncertainty), while considering the model through the distribution on weights (cognitive uncertainty) of uncertainty. A latent input variable is a variable that cannot be directly observed, but has an impact on the state and output of a probability model. The inventor described a method and device for implementing a Bayesian neural network using memristor intrinsic noise in Chinese Patent Application Publication CN110956256A, which is incorporated herein by reference in its entirety as a part of this application.

The structure of the Bayesian neural network includes but is not limited to a fully connected structure, a convolutional neural network (Convolutional Neural Network, CNN) structure, etc., and its network weight W is a random variable (W~q(W)) based on a certain distribution.

The inventors further found that, assuming that there is a data set D={X, Y} of the dynamic system for the Bayesian neural network, where X is the state feature vector of the dynamic system, and Y is the next state of the dynamic system. The input of the Bayesian neural network is the state feature vector X of the dynamic system and the hidden input variable z (z～p(z)); the parameters of the Bayesian neural network can be trained; The independent additive Gaussian noise ε(ε～N(0,σ ² )) superimposed on the output is to predict y for the next state of the dynamic system, that is, y=f(X, z, W, ε). Thus, each weight of the Bayesian neural network is a distribution after training. For example, each weight is a distribution independent of each other.

In the long-term planning task, the gradient will go through multi-step backpropagation, and there are problems of gradient disappearance and explosion; at the same time, when the neural network directly based on the memristor array performs policy search, due to the inherent randomness of the memristor However, the back-propagation of gradients through the memristor array introduces additional noise, and these noisy gradients cannot effectively optimize policy search.

At least one embodiment of the present disclosure provides a strategy optimization method using a memristor array-based dynamic environment model, including: acquiring a memristor array-based dynamic environment model; The second prediction is performed to obtain a data sample set including the optimization cost of the target strategy corresponding to multiple moments; based on the data sample set, a policy gradient optimization algorithm is used to perform a strategy search to optimize the target strategy.

The strategy optimization method provided by the above embodiments of the present disclosure utilizes a dynamic environment model based on a memristor array to generate a data sample set, implements long-term dynamic programming based on the dynamic environment model, and then uses a more stable algorithm such as a policy gradient optimization algorithm to perform policy search , without the vanishing and exploding gradient problem, and can effectively optimize the object policy.

At least one embodiment of the present disclosure further provides a strategy optimization apparatus corresponding to the above strategy optimization method.

The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

FIG. 1A shows a schematic flowchart of a strategy optimization method based on a dynamic environment model of a memristor array provided by at least one embodiment of the present disclosure.

As shown in FIG. 1A, the strategy optimization method includes the following steps S101-S103.

Step S101: Obtain a dynamic environment model based on the memristor array.

In the embodiment of the present disclosure, for example, the dynamic environment model can be obtained by modeling the dynamic system by using the BNN+LV based on the memristor array. The specific steps will be shown in FIG. 1B , and will not be repeated here .

Step S102: Perform multiple predictions at multiple times according to the dynamic environment model and the object strategy, and obtain a data sample set including the optimization cost of the object strategy corresponding to the multiple times.

For example, the object policy involved is used in deep reinforcement learning, and can be, for example, the agent's policy to maximize rewards or achieve specific goals during interactions with the environment.

Step S103: Based on the data sample set, use a policy gradient optimization algorithm to perform policy search to optimize the target policy.

For example, in different examples of embodiments of the present disclosure, the policy gradient optimization algorithm may include the REINFORCE algorithm, the PRO (Proximal Policy Optimization) algorithm, or the TPRO (Trust Region Policy Optimization) algorithm. In the embodiments of the present disclosure, these policy gradient optimization methods are more stable and can effectively optimize the target policy.

FIG. 1B shows a schematic flowchart of an example of step S101 in FIG. 1A .

As shown in FIG. 1B , an example of step S101 may include steps S111 to S113 as follows.

Step S111: Obtain a Bayesian neural network, wherein the Bayesian neural network has a weight matrix obtained by training.

For example, the structure of Bayesian neural network includes fully connected structure or convolutional neural network structure. Each network weight of this Bayesian neural network is a random variable. For example, after the Bayesian neural network is trained, each weight is a distribution, such as a Gaussian distribution or a Laplacian distribution.

For example, the weight matrix can be obtained by offline training of the Bayesian neural network, and the method of training the Bayesian neural network can refer to conventional methods, such as central processing unit (CPU), image processing unit (GPU) , a neural network processing unit (NPU), etc. for training, which will not be repeated here.

Step S112 : obtaining corresponding multiple target conductance values according to the weight matrix of the Bayesian neural network, and mapping the multiple target conductance values to the memristor array.

After the training of the Bayesian neural network is completed to obtain the weight matrix, the weight matrix is processed to obtain corresponding multiple target conductance values. For example, in this process, the weight matrix can be biased and scaled until the weight matrix meets an appropriate conductance window for the memristor array used. After biasing and scaling the weight matrix, the target conductance value is calculated according to the processed weight matrix and the conductance value of the memristor. For the specific process of calculating the target conductance value, reference may be made to the relevant description of the memristor-based Bayesian neural network, which will not be repeated here.

FIG. 2A shows a schematic structure of a memristor array, for example, the memristor array is composed of a plurality of memristor cells, the plurality of memristor cells form an array of M rows and N columns, M and N All are positive integers. Each memristor cell includes a switching element and one or more memristors. In FIG. 2A, WL<1>, WL<2>...WL<M> represent the word lines of the first row, the second row...the Mth row, respectively, and the switching elements in the memristor cell circuit of each row The control electrode (such as the gate of the transistor) is connected to the corresponding word line of the row; BL<1>, BL<2>...BL<N> represent the bits of the first column, the second column...the Nth column, respectively Line, the memristor in the memristor unit circuit of each column is connected to the corresponding bit line of the column; SL<1>, SL<2>...SL<M> represent the first row, the second row... For the source lines of the Mth row, the sources of the transistors in the memristor unit circuits of each row are connected to the corresponding source lines of the row. According to Kirchhoff's law, by setting the state of the memristor cell (such as resistance value) and applying the corresponding word line signal and bit line signal to the word line and bit line, the above-mentioned memristor array can complete the multiply-accumulate calculation in parallel .

FIG. 2B is a schematic diagram of a memristor device including a memristor array and a peripheral driving circuit thereof. For example, as shown in FIG. 2B, the memristor device includes a signal acquisition device, a word line driver circuit, a bit line driver circuit, a source line driver circuit, a memristor array, and a data output circuit.

For example, the signal acquisition device is configured to convert a digital signal into a plurality of analog signals through a digital to analog converter (DAC) for inputting to a plurality of column signal input terminals of the memristor array.

For example, a memristor array includes M source lines, M word lines, and N bit lines, and a plurality of memristor cells arranged in M rows and N columns.

For example, operation of the memristor array is accomplished by word line driver circuits, bit line driver circuits, and source line driver circuits.

For example, the word line driver circuit includes a plurality of multiplexers (Multiplexer, Mux for short) for switching the word line input voltage; the bit line driver circuit includes a plurality of multiplexers for switching the bit line input voltage; the source line The driver circuit also includes a plurality of multiplexers (Mux) for switching the source line input voltage. For example, the source line driver circuit also includes a plurality of ADCs for converting analog signals into digital signals. In addition, a Trans-Impedance Amplifier (TIA) (not shown in the figure) may be further set between the Mux and the ADC in the source line driver circuit to complete the current-to-voltage conversion for the ADC to process.

For example, a memristor array includes an operating mode and a computing mode. When the memristor array is in the operating mode, the memristor cell is in an initialization state, and the values of the parameter elements in the parameter matrix can be written into the memristor array. For example, the source line input voltage, bit line input voltage and word line input voltage of the memristor are switched to corresponding preset voltage ranges through multiplexers.

For example, the word line input voltage is switched to the corresponding voltage range through the control signals WL_sw[1:M] of the multiplexer in the word line driving circuit in FIG. 2B . For example, when the memristor is set, the word line input voltage is set to 2V (volts), for example, when the memristor is reset, the word line input voltage is set to 5V, for example, the word line input voltage It can be obtained by the voltage signals V_WL[1:M] in FIG. 2B.

For example, the source line input voltage is switched to a corresponding voltage range through the control signals SL_sw[1:M] of the multiplexer in the source line driving circuit in FIG. 2B . For example, when the memristor is set, the source line input voltage is set to 0V. For example, when the memristor is reset, the source line input voltage is set to 2V. For example, the source line input voltage can be obtained by Fig. The voltage signals V_SL[1:M] in 2B are obtained.

For example, the bit line input voltage is switched to the corresponding voltage range through the control signals BL_sw[1:N] of the multiplexer in the bit line driving circuit in FIG. 2B . For example, when the memristor is set, the bit line input voltage is set to 2V. For example, when the memristor is reset, the bit line input voltage is set to 0V. For example, the bit line input voltage can be obtained by Fig. DAC obtained in 2B.

For example, when the memristor array is in calculation mode, the memristors in the memristor array are in a conductive state that can be used for calculation, and the bit line input voltage input at the column signal input terminal does not change the conductance value of the memristor, e.g. , the calculation can be done by performing a multiply-add operation with an array of memristors. For example, the word line input voltage is switched to the corresponding voltage range by the control signal WL_sw[1:M] of the multiplexer in the word line driving circuit in FIG. 2B, for example, when the turn-on signal is applied, the word line input voltage of the corresponding row is The voltage is set to 5V, for example, when the turn-on signal is not applied, the input voltage of the word line of the corresponding row is set to 0V, for example, the GND signal is turned on; through the control signal SL_sw[1 of the multiplexer in the source line driver circuit in FIG. 2B :M] Switch the input voltage of the source line to the corresponding voltage range, for example, set the input voltage of the source line to 0V, so that the current signals of the multiple row signal output terminals can flow into the data output circuit, through the bit line driving circuit in FIG. 2B The control signal BL_sw[1:N] of the multiplexer in the bit line switches the input voltage of the bit line to the corresponding voltage range, for example, the input voltage of the bit line is set to 0.1V-0.3V, so as to use the memristor array to multiply and add operation.

For example, the data output circuit may include multiple transimpedance amplifiers (TIAs), ADCs, and may convert current signals at multiple row signal output terminals into voltage signals, and then into digital signals for subsequent processing.

2C is a schematic diagram of another memristor device. The structure of the memristor device shown in FIG. 2C is basically the same as that of the memristor device shown in FIG. 2B , and also includes a memristor array and its peripheral driving circuit. For example, as shown in FIG. 2C, the memristor device includes a signal acquisition device, a word line driver circuit, a bit line driver circuit, a source line driver circuit, a memristor array, and a data output circuit.

For example, a memristor array includes M source lines, 2M word lines, and 2N bit lines, and a plurality of memristor cells arranged in M rows and N columns. For example, each memristor unit has a 2T2R structure, and the operation of mapping the parameter matrix used for transformation processing to different multiple memristor units in the memristor array will not be repeated here. It should be noted that the memristor array may also include M source lines, M word lines and 2N bit lines, and a plurality of memristor cells arranged in M rows and N columns.

For the description of the signal acquisition device, the control driving circuit, and the data output circuit, reference may be made to the previous description, which will not be repeated here.

Figure 2D shows the process of mapping the weight matrix of a Bayesian neural network to a memristor array. Use the memristor array to realize the weight matrix between layers in the Bayesian neural network, and use N memristors for each weight to realize the distribution corresponding to the weight, where N is an integer greater than or equal to 2. The corresponding random probability distribution is calculated to obtain N conductance values, and the N conductance value distributions are mapped to the N memristors. In this way, the weight matrices in the Bayesian neural network are transformed into target conductance values mapped into the crossover sequence of the memristor array.

As shown in Figure 2D, the left side of the figure is a three-layer Bayesian neural network, which includes 3 layers of neurons connected one by one. For example, the input layer includes the first layer of neurons, the hidden layer includes the second layer of neurons, and the output layer includes the third layer of neurons. For example, the input layer transmits the received input data to the hidden layer, the hidden layer calculates and transforms the input data and sends it to the output layer, and the output layer outputs the output structure of the Bayesian neural network.

As shown in FIG. 2D , the input layer, the hidden layer and the output layer all include multiple neuron nodes, and the number of neuron nodes in each layer can be set according to different application conditions. For example, the number of neurons in the input layer is 2 (including N ₁ and N ₂ ), the number of neurons in the middle hidden layer is 3 (including N ₃ , N ₄ and N ₅ ), and the number of neurons in the output layer is 1 (including N ₆ ).

As shown in Figure 2D, two adjacent neuron layers of the Bayesian neural network are connected by a weight matrix. For example, the weight matrix is implemented by a memristor array as shown on the right in Figure 2D. For example, the weight parameter can be programmed directly to the conductance of the memristor array. For example, the weight parameter can also be mapped to the conductance of the memristor array according to a certain rule. For example, the difference in conductance of two memristors can also be used to represent a weight parameter. Although the present disclosure describes the technical solutions of the present disclosure by directly programming the weight parameters into the conductance of the memristor array or by mapping the weight parameters to the conductance of the memristor array according to a certain rule, it is only exemplary , not a limitation of the present disclosure.

The structure of the memristor array on the right side in FIG. 2D is, for example, as shown in FIG. 2A , the memristor array may include a plurality of memristors arranged in an array. In the example shown in Figure 2D, the weights connecting the input N1 and the output N3 are implemented by 3 memristors (G ₁₁ , G ₁₂ , G ₁₃ ), other weights in the weight matrix can be implemented the same. More specifically, source line SL ₁ corresponds to neuron N ₃ , source line SL ₂ corresponds to neuron N ₄ , source line SL ₅ corresponds to neuron N ₅ , bit lines BL ₁ , BL ₂ and BL ₃ correspond to neuron N1 , A weight between the input layer and the hidden layer (the weight between neuron N ₁ and neuron N ₃ ) is transformed into three target conductance values according to the distribution, and the distribution is mapped into the cross sequence of the memristor array, here The target conductance values are G ₁₁ , G ₁₂ , and G ₁₃ , respectively, framed by dashed lines in the memristor array.

Returning to FIG. 1B , step S113 : input the state and latent input variables corresponding to the time t of the dynamic system as input signals to the weight-mapped memristor array, and the state and latent input variables of time t are analyzed by the memristor array. The processing is performed according to the Bayesian neural network, and the output signal corresponding to the processing result is obtained from the memristor array, and the output signal is used to obtain the prediction result of the dynamic system at time t+1.

For example, in some embodiments of the present disclosure, the dynamic environment model is expressed as s _{t +1} = f(s _t , at ; W, ε), where s _t is the state of the dynamic system at time _t , and at is the object The action of the policy at time t, W is the weight matrix of the Bayesian neural network, ε is the additive noise corresponding to the memristor array, and s _t+1 is the prediction result of the dynamic system at time t+1; the object policy is at The action at time _t = π(s _t ; Wπ), π represents the function of the target strategy, Wπ represents the strategy parameter, the weight matrix W of the Bayesian neural network satisfies the distribution W～q(W), and the additive noise ε is Additive Gaussian noise ε～N(0,σ ² ).

For a memristor array, the input signal is a voltage signal and the output signal is a current signal, and the output signal is read and analog-to-digital converted for subsequent processing. For example, the input sequence is applied to BL (Bit-line, bit line) in the form of voltage pulses, and then the output current flowing out from SL (Source-line, source line) is collected for further calculation processing. For example, for a memristor device as shown in Figure 2B or 2C, the input sequence can be converted by a DAC into an analog voltage signal, which is applied to BL through a multiplexer. Correspondingly, an output current is obtained from the SL, which can be converted into a voltage signal by a transimpedance amplifier, and converted into a digital signal by an ADC, and the digital signal can be used for subsequent processing. When N memristors read current and N is relatively large, the total output current exhibits a certain distribution, for example, a distribution similar to a Gaussian distribution or a Laplace distribution. The total output current of all voltage pulses is the result of multiplying the input vector by the weight matrix. In the memristor interleaved array, such a parallel read operation is equivalent to two operations of sampling and vector-matrix multiplication.

The following describes how to perform multiple predictions at multiple times according to the dynamic environment model and the object strategy, and obtain a data sample set including the optimization costs of the object strategy corresponding to multiple times.

For example, the plurality of times include time 1 to time T arranged in order from early to late.

FIG. 3 shows a schematic flowchart of an example of step S102 in FIG. 1A .

As shown in FIG. 3 , step S102 may include the following steps S301 to S303.

Step S301: For any time t-1 from time 1 to time T, the execution action a _t-1 is obtained from the target strategy, and the target strategy at time t is obtained from a _t-1 =π(s _t-1 ; Wπ) -1 action a _t-1 .

For example, action a _t-1 is the optimal action selected by the target policy in the state of time t-1.

Step S302: According to the dynamic environment model s _t =f(s _t-1 , at _-1 ; W, ε), the state s _t at the next time t after the time t-1 is calculated and the state corresponding to the time t is obtained cost c _t of s _t , thus obtain the cost sequence {c ₁ ,c ₂ ,...,c _t } from time 1 to time t, and obtain the optimization cost J _t-1 at time t based on the cost sequence, where 1≤ t≤T.

For example, in some embodiments of the present disclosure, an example of step S302 may include: sampling the latent input variable z from the p(z) distribution to obtain a sample; inputting the sample and the state s _{t-1 at time t-1} into the weight The mapped memristor array yields the predicted state s _t ; for the predicted state s _t , the cost _{ct =c(s t )} is obtained.

For example, the latent input variable z is first sampled from the p(z) distribution, then the state s _{t-1 at time t-1} and the sample of the latent input variable are applied with the read (READ) voltage pulse of the memristor array on BL, and then collect the output current flowing from SL for further calculation processing to obtain the cost ct corresponding to time _t . The above operations are performed on the state at any time from time 1 to time t, and the cost sequence {c ₁ ,c ₂ ,...,c _t } can be obtained.

Step S303: Obtain a data sample set {[a ₀ , J ₀ ], . . . , [a _T-1 , J _T-1 ]} from time 1 to time T.

来获得。For example, the expected value of the cost c _t at time t is E[c _t ], then the optimization cost at time t can be obtained by

to obtain.

For example, in some embodiments of the present disclosure, the cost also includes the cost change caused by accidental uncertainty and the cost change caused by cognitive uncertainty. The accidental uncertainty is caused by latent input variables, and the cognitive uncertainty The determinism is caused by the intrinsic noise of the memristor array.

来获得，σ(η,θ)为偶然不确定性和认知不确定性的函数，η表示偶然不确定性，θ表示认知不确定性。For example, if the cost changes caused by accidental uncertainty and cognitive uncertainty are further considered, the optimization cost at time t can be obtained by

to obtain, σ(η, θ) is a function of contingent uncertainty and epistemic uncertainty, where η represents contingent uncertainty and θ represents epistemic uncertainty.

For any time between time 1 and time T, the data sample corresponding to this time can be obtained, and the data sample set from time 1 to time T is {[a ₀ ,J ₀ ],...,[a _T-1 ,J _T-1 ]}.

For example, the exemplary flow of the above-mentioned policy optimization method based on the dynamic environment model of the memristor array is as follows:

Input: Memristor array based dynamic environment model and initial object policy

n=1 to N, loop

Initialize state s ₀

t=1 to T, loop

The execution action a _t-1 is obtained from the object policy π

Using the dynamic environment model based on the memristor array s _t =f(s _t-1 ,at _-1 ; W,ε) to predict the state s _t at time t, and obtain the data sample [s _t ]

Calculate the cost and optimization cost corresponding to the object strategy at this moment

c _t =c(s _t )

record[a _t-1 ,J _t-1 ]

Obtain a set of data samples {[a ₀ ,J ₀ ],…,[a _T-1 ,J _T-1 ]}, and use the policy gradient optimization algorithm to perform policy search on the data sample set

end the loop of n

Output: Optimized policy π

FIG. 4 shows a schematic diagram of an example of a policy optimization method provided by at least one embodiment of the present disclosure.

As shown in Fig. 4, in one exemplary application, a boat is driven and thereby combats the waves to get as close as possible to a target location on the shoreline, thereby requiring training of a control model for driving the boat. A boat at position (x,y) can choose an action (a _x ,a _y ) that represents the direction and magnitude of the drive. However, due to the dynamic environment of the sea surface with waves, the subsequent position of the ship exhibits drift and disturbance. Also, the closer the location is to the coast, the greater the disturbance. Ships are only given a limited batch dataset of spatial position transformations, and there is no way to optimize action policies for safety by directly interacting with the marine environment. At this time, it is necessary to rely on empirical data to learn a marine environment model (dynamic environment model) that can predict the next state. Epistemic uncertainty and contingent uncertainty would arise from the missing information of unvisited locations and the randomness of the marine environment, respectively.

In this embodiment, for the control model, the sea surface is a dynamic environment, and the object strategy refers to the method used in the process of solving the ship from the current position to the target position. First, a dynamic environment model and an initial object policy for the dynamic environment are obtained. The initial state of the ship is the current position, the execution action at the current moment is obtained from the object strategy, the state of the next moment (the position of the ship) is predicted by the dynamic environment model, the cost and optimization cost corresponding to the object strategy are calculated, and the action and optimization cost are recorded. composed data samples. Assuming that the current time is time 1, from time 1 to the subsequent time T, a data sample set is obtained, and a policy gradient optimization algorithm is used to perform policy search on the data sample set, thereby obtaining an optimized object policy.

FIG. 5 shows a schematic block diagram of a strategy optimization apparatus 500 using a dynamic environment model based on a memristor array provided by at least one embodiment of the present disclosure. The strategy optimization apparatus can be used to execute the data processing method shown in FIG. 1A . .

As shown in FIG. 5 , the strategy optimization apparatus 500 includes an acquisition unit 501 , a calculation unit 502 and a strategy search unit 503 .

The acquisition unit 501 is configured to acquire a dynamic environment model based on the memristor array.

The computing unit 502 is configured to perform multiple predictions at multiple times according to the dynamic environment model and the object strategy, and obtain a data sample set including the optimization cost of the object strategy corresponding to the multiple times.

The policy search unit 503 is configured to, based on the data sample set, perform a policy search using a policy gradient optimization algorithm to optimize the target policy.

For example, the policy optimization apparatus 500 may be implemented by using hardware, software, firmware and any feasible combination thereof, which is not limited in the present disclosure.

The technical effect of the above strategy optimization device is the same as that of the strategy optimization method shown in FIG. 1A , and details are not repeated here.

The following points need to be noted:

(1) The drawings of the embodiments of the present disclosure only relate to the structures involved in the embodiments of the present disclosure, and other structures may refer to general designs.

(2) The embodiments of the present disclosure and features in the embodiments may be combined with each other to obtain new embodiments without conflict.

The above descriptions are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims (10)

1. A strategy optimization method utilizing a memristor array-based dynamic environment model, comprising: obtaining the dynamic environment model based on the memristor array; Perform multiple predictions at multiple times according to the dynamic environment model and the object strategy, and obtain a data sample set including the optimization cost of the object strategy corresponding to the multiple times; Based on the set of data samples, a policy search is performed using a policy gradient optimization algorithm to optimize the target policy. 2. The strategy optimization method according to claim 1, wherein obtaining the dynamic environment model comprises: obtaining a Bayesian neural network, wherein the Bayesian neural network has a weight matrix obtained by training; Obtaining a plurality of corresponding target conductance values according to the weight matrix of the Bayesian neural network, and mapping the plurality of target conductance values to the memristor array; Input the state and hidden input variables corresponding to the time t of the dynamic system as input signals to the weight-mapped memristor array, and the state of the time t and the hidden input variables are analyzed by the memristor array. Processing is performed according to the Bayesian neural network, and an output signal corresponding to the processing result is obtained from the memristor array, wherein the output signal is used to obtain the prediction result of the dynamic system at time t+1. 3. The strategy optimization method according to claim 2, wherein the expression of the dynamic environment model is s _{t +1} =f(s _t , at ; W, ε), Among them, s _t is the state of the dynamic system at the time t, a _t is the action of the object strategy at the time t, W is the weight matrix of the Bayesian neural network, ε is the memory corresponding to the memory the additive noise of the resistor array, s _t+1 is the predicted result of the dynamic system at the time t+1; Wherein, the action of the target strategy at time t at _t =π(s _t ; Wπ), π represents the function of the target strategy, Wπ represents the strategy parameter, and the weight matrix W of the Bayesian neural network satisfies the distribution W ~q(W), the additive noise ε is an additive Gaussian noise ε∼N(0,σ ² ). 4. The strategy optimization method according to claim 3, wherein the plurality of moments comprise time 1 to time T arranged in order from early to late, Perform multiple predictions on the multiple moments according to the dynamic environment model and the object strategy, and obtain the data sample set including the optimization cost of the object strategy corresponding to the multiple moments, including: For any time t-1 from the time 1 to the time T, the execution action a _t-1 is obtained from the object strategy, Obtain the action at-1 of the object strategy at the time t-1 from at _-1 =π(s _{t-1 ;} Wπ); According to the dynamic environment model s _t =f(s _t-1 , at _-1 ; W, ε), the state s _t at the next time t after the time t-1 is calculated and obtained corresponding to the time the cost c t of the state s _t of _t , thus obtaining the cost sequence {c ₁ ,c ₂ ,...,c _t } from the time 1 to the time t, Obtain the optimization cost J _t-1 at the time t based on the cost sequence, where 1≤t≤T; Obtain the data sample set {[a ₀ , J ₀ ],...,[a _T-1 ,J _T-1 ]} from the time 1 to the time T. 5. The strategy optimization method according to claim 4, wherein the expected value of the cost c _t at the time t is E[c _t ], then the optimization cost of the time t can be obtained by

to obtain. 6. The strategy optimization method according to claim 4, wherein the cost further comprises a cost change caused by accidental uncertainty and a cost change caused by cognitive uncertainty, Wherein the contingent uncertainty is caused by the latent input variable and the epistemic uncertainty is caused by the intrinsic noise of the memristor array. 7. The strategy optimization method according to claim 6, wherein the optimization cost at the time t is obtained by

to obtain, where σ(η, θ) is a function of the contingent uncertainty and the epistemic uncertainty, η represents the contingent uncertainty, and θ represents the epistemic uncertainty. 8. The strategy optimization method according to claim 4, wherein the state s _t at the time t is obtained by calculating the dynamic environment model s _t =f(s _t-1 , at _-1 ; W, ε) and obtain the cost ct corresponding to the state s _t at the time _t , including: sampling the hidden input variable z from the p(z) distribution to obtain a sample; Inputting the sample and the state s _{t-1 at the time t-1} into the weight-mapped memristor array to obtain the predicted state s _t ; For the predicted state st, the cost _ct = c(s _t ) _is obtained. 9. The policy optimization method according to claim 1, wherein the policy gradient optimization algorithm comprises a REINFORCE algorithm, a PRO algorithm or a TPRO algorithm. 10. A strategy optimization device utilizing a memristor array-based dynamic environment model, comprising: an acquisition unit, configured to acquire the dynamic environment model based on the memristor array; a computing unit, configured to perform multiple predictions at multiple times according to the dynamic environment model and the object strategy, and obtain a data sample set including the optimization cost of the object strategy corresponding to the multiple times; A policy search unit, configured to use a policy gradient optimization algorithm to perform policy search based on the data sample set to optimize the target policy.

Images