CN112199637B - Regression modeling method for generating contrast network data enhancement based on regression attention - Google Patents

Abstract

The invention discloses a regression modeling method for generating data enhancement of an countermeasure network based on regression attention, wherein the regression attention generation countermeasure network is added with an attention mechanism in a generator and a discriminator; the attention module 1 in the generator constructs regression loss by using independent variables and dependent variables of the output generated data of the generator; meanwhile, the real data also fine-tunes the attention module 1; the attention module 2 in the discriminator constructs a new loss by using the difference between the true data and the generated data regression loss; the feature containing the maximum regression information is extracted by minimizing the loss, and the feature contains the regression difference information between the maximized real data and the generated data, so that the consideration of the regression information by the discriminator is facilitated. According to the regression model, the original data is enhanced by using the generated data based on the regression attention generation countermeasure network, and then the regression modeling is performed by using the data driving method, so that the performance and the prediction accuracy of the regression model are effectively improved.

Description

Regression modeling method based on regression attention generative adversarial network data enhancement

Technical Field

The present invention belongs to the field of industrial process soft measurement, and in particular relates to a regression modeling method based on regression attention generation adversarial network data enhancement.

Background technique

In today's big data era, data-driven models play an important role. Among them, regression models are widely used in many scenarios as a practical tool, such as stock trend prediction in the financial industry, soft sensors in the process industry, etc. The quality of data is crucial for data-driven models. In application scenarios such as limited data accumulation, difficult data acquisition, and data privacy protection, the lack of data affects the prediction accuracy of the regression model. Therefore, how to improve the performance of the regression model under limited data is an important topic.

Generative adversarial network (GAN) is a generative model proposed by Goodfellow in 2014. The idea of data augmentation is to add the generated data of the trained GAN to the real data set and participate in modeling together. It can help the original data to obtain information expansion to improve the effect of the data-driven model. However, there is currently no GAN model for regression problems to perform data augmentation regression modeling. When using the existing GAN model for data generation, the independent variable and the dependent variable are simply spliced and reconstructed, ignoring the regression relationship between the independent variable and the dependent variable in the data. In addition, since the dependent variable is the variable to be predicted, the acquisition method is often more rigorous and the accuracy will be higher. However, the dependent variable is not given enough attention in the training of GAN, which will propagate the error of the independent variable to the dependent variable. Therefore, these factors restrict the effect of data augmentation regression modeling.

Summary of the invention

The purpose of the present invention is to provide a regression modeling method based on regression attention generative adversarial network data enhancement to address the deficiencies of the prior art.

The objective of the present invention is to achieve the following technical solution: a regression modeling method based on regression attention generation adversarial network data enhancement, comprising the following steps:

(1) Collect the original data, remove outliers and normalize the data to obtain independent variables _xi and _yi , where i = 1 to M.

(2) Concatenate _xi and yi and reconstruct _them to obtain a new data set D = [X, Y] = {[x ₁ , y ₁ ], [x ₂ , y ₂ ], …, [x _M , y _M ]}, M∈Z.

(3) Train a RA-GAN unsupervisedly using D until it converges.

(4) Use the trained RA-GAN to generate virtual data D′=[X′,Y′]={[x′ ₁ ,y′ ₁ ],[x′ ₂ ,y′ ₂ ],…,[x′ _N ,y′ _N ]}, N∈Z.

(5) Add the generated data to the real dataset D for data augmentation to obtain an enhanced dataset.

(6) The data of the enhanced dataset D is divided into independent variables {X, X′} and dependent variables {Y, Y′} for training the regression model.

(7) Input the independent variables of the test data into the trained regression model to obtain the predicted dependent variables.

Furthermore, the regression attention generative adversarial network introduces two attention modules, namely attention module 1 in the generator and attention module 2 in the discriminator.

The generator is a multi-layer perceptron. The input layer is random noise z, and the output layer outputs the generated data D′. The generated data is divided into independent variables x′ _j and dependent variables y _j ′, j∈[1,2,…,N]. The attention module 1 is a multi-layer perceptron connected to the generator. The input is the independent variable x′ _j of the generated data, and the output layer is the predicted value. In order to make/> To make it as equal to y _j ′ as possible, the generator’s loss function adds a new loss term _LA1 and is updated as follows:

Where D(·) refers to the output of the discriminator, x~ _Pg refers to the sample x coming from the generator, and β is the regression coefficient of the attention module 1.

The discriminator is also a multi-layer perceptron, which is used to distinguish whether the input is real data or generated data. The attention module 2 is set at the front end of the discriminator network and is also a multi-layer perceptron. The input data D or D' is divided into independent variables and dependent variables. The independent variables are input into the attention module 2 and output after network mapping. or/> The loss function of attention module 2 is:

Among them, γ is the regression coefficient of attention module 2. Then, or/> It is concatenated with the corresponding input independent variable _xi or _x′j and input into the subsequent discriminator. The loss function of the discriminator with attention module 2 is:

Where D(·) refers to the output of the discriminator, x~P _r means that the sample x comes from the real data; λ represents the gradient penalty factor, Indicates that the data comes from the sample space between the real data and the generated data, /> is the discriminator gradient/> The second norm of .

Furthermore, the network parameters of the attention module 1 can also be fine-tuned by real data D, and the fine-tuning loss is as follows:

Where M is the number of real data, and α is the fine-tuning coefficient. Then the loss function of the generator can also be _LG ^* '= _LG ^* + _LA1 '.

The beneficial effects of the present invention are as follows: the present invention designs a regression attention generative adversarial network (RA-GAN), introduces the attention mechanism into the generator and discriminator of the generative adversarial network, and considers the internal relationship of the generated data variables when training the network parameters. The attention module 1 in the generator constructs the regression loss using the independent variables and dependent variables of the generated data output by the generator; at the same time, the real data also fine-tunes the attention module 1; the attention module 2 in the discriminator constructs a new loss using the difference between the regression loss of the real data and the generated data; it extracts the feature containing the maximum regression information by minimizing this loss, and this feature contains the regression difference information between the maximized real data and the generated data, which is beneficial to the discriminator's consideration of the regression information. The present invention uses the generated data to enhance the original data based on the regression attention generative adversarial network, and then uses the data-driven method for regression modeling, which effectively improves the performance and prediction accuracy of the regression model.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a flowchart based on the regression attention generation adversarial network (RA-GAN);

Figure 2 is a flow chart of the attention module 1 in the generator of RA-GAN;

Figure 3 is a flow chart of attention module 2 in the discriminator of RA-GAN;

FIG4 is a flowchart of data enhancement regression modeling based on RA-GAN;

Figure 5 is a comparison of the effects of WGAN-GP and RA-GAN generated data on support vector regression model data enhancement.

Detailed ways

The following is a further detailed description of the regression modeling method based on regression attention generative adversarial network data enhancement of the present invention in conjunction with a specific implementation method.

The Regression Attention Generative Adversarial Network (RA-GAN) proposed in the present invention draws on the basic structure of the Wasserstein network with gradient penalty terms, and introduces two attention modules, namely, attention module 1 in the generator and attention module 2 in the discriminator. The independent variables in the present invention are process variables in the industrial process, and the dependent variables are the corresponding quality variables.

As shown in Figure 1, the generator of the Regression Attention Generative Adversarial Network (RA-GAN) is a multi-layer perceptron. The input layer is random noise z, the hidden layer is set to [32,32], and the output layer outputs the generated data D′=[X′,Y′]={[x′ ₁ ,y′ ₁ ],[x′ ₂ ,y′ ₂ ],…,[x′ _N ,y′ _N ]}, N∈Z, N represents the sample size in D', and Z represents an integer; the generated data D' is divided into independent variables x′ _j and dependent variables y _j ′, i∈[1,2,…,N].

As shown in Figure 2, attention module 1 is a multi-layer perceptron connected to the generator, and its hidden layer is set as [32]; its input is the independent variable x′ _i of the generated data, and the output layer is the predicted value of the dependent variable In order to make/> _As much as possible, the generator’s loss function _LG ^* adds a new loss term _LA1 related to attention module 1 and is updated as follows:

Where D(·) refers to the output of the discriminator, E represents the expectation, x~ _Pg refers to the sample x coming from the generated data; β is the regression coefficient of the attention module 1, which is used to adjust the proportion of the regression loss _LA1 in the generator loss _LG *; Indicates taking the two-norm. At the same time, the network parameters of the attention module 1 can also be fine-tuned by the real data D = [X, Y] = { _xi , _yi } = {[ _x1 , _y1 ], [ _x2 , _y2 ], ..., [ _xM , _yM ]}, M∈Z, i = 1, 2, ..., M. At this time, the independent variable _xi of the real data is used as input to control the regression relationship between the generated data variables. The fine-tuning loss is as follows:

Among them, M is the number of real data, α is the fine-tuning coefficient; then the loss function of the generator can also be _LG ^* '= _LG ^* + _LA1 '.

As shown in Figure 3, the attention module 2 is set at the front end of the discriminator network. The discriminator of RA-GAN is also a multi-layer perceptron, which is used to determine whether the input data is real data or generated data. Its hidden layer setting is [32, 64, 32].

Attention module 2 is also a multi-layer perceptron, and its hidden layer setting is [32]. The input real data or generated data is divided into independent variables and dependent variables. The independent variable _xi of the real data or the independent variable _xj of the generated data is input into attention module 2 and output after network mapping. or/> In order to reflect the difference in the regression relationship between the real data and the generated data in the discriminator's results, the loss function _LA2 of the attention module 2 is designed as:

Among them, γ is the regression coefficient of attention module 2.

Minimizing _LA2 makes the last layer of the hidden layer of attention module 2 extract features with maximum regression information; then, the output of attention module 2 is or/> The independent variables corresponding to the input (the independent variables _xi of the real data or the independent variables _xj ' of the generated data) are concatenated and input into the subsequent discriminator multi-layer perceptron. Finally, the loss function _LD * of the discriminator with attention module 2 is:

Among them, the first term is the expected value of the output after the real data is input into the discriminator, D(·) refers to the output of the discriminator, and x~P _r means that the sample x comes from the real data; the second term is the expected value of the output after the generated data is input into the discriminator; the third term is the gradient penalty term, λ represents the gradient penalty factor, Indicates that the data comes from the sample space between the real data and the generated data, /> is the discriminator gradient/> The second norm of .

As shown in Figure 4, based on the above RA-GAN, a regression modeling method for data enhancement in industrial processes is proposed:

1. Collect the original data, and obtain the independent variables x _i and y _i after removing outliers and normalizing the data, where i = 1 to M.

2. _{Concatenate xi} and yi to reconstruct the new data D = [X, Y] = {[x ₁ , y ₁ ], [x ₂ , y ₂ ], …, [x _M , y _M ]}, M∈Z.

3. Use D to unsupervisedly train the above RA-GAN until it converges.

4. Use the trained RA-GAN to generate virtual data D′=[X′,Y′]={[x′ ₁ ,y′ ₁ ],[x′ ₂ ,y′ ₂ ],…,[x′ _N ,y′ _N ]}, N∈Z.

5. Add the generated data D’ to the real data set D for data enhancement to obtain the enhanced data set {D, D’}.

6. Split the data of the augmented dataset into independent variables {X, X′} and dependent variables {Y, Y′} for training the regression model.

7. Input the independent variables of the test data into the trained regression model to obtain the predicted dependent variables.

The following example uses a specific carbon dioxide absorption process to illustrate the performance of data augmented regression modeling based on RA-GAN. The carbon dioxide absorption tower is a key equipment in the urea synthesis industry. It is used to remove carbon dioxide from the mixed gas to prevent it from affecting the quality of the final product. However, the carbon dioxide content is a variable that is difficult to detect in real time and requires offline analysis by a mass spectrometer. Therefore, we need to establish a soft measurement model for carbon dioxide in the case of a small sample.

Table 1: List of independent variables for carbon dioxide absorption process

There are a total of 11 independent variables in the carbon dioxide absorption process, as shown in Table 1. We collected 300 samples including qualitative variables and dependent variables, 100 of which were used for training and the remaining 200 for testing. Based on the use of the support vector regression model to establish the regression model, the Wasserstein generative adversarial network with gradient penalty (WGAN-GP) and RA-GAN were used for data enhancement. WAGAN-GP and RA-GAN were iterated for 12,000 cycles under the same learning rate and optimization algorithm, respectively. Figure 5 shows the final results. We used the root mean square error (RMSE) between the regression model prediction result and the true value as the comparison indicator.

From Figure 5, we can see that WGAN-GP does not take into account the regression relationship between generated data variables. The generated data it generates adds new regression information, so it cannot improve the performance of the original regression model. However, RA-GAN introduces an attention mechanism for the reference variables when generating data, which effectively retains the regression relationship between data variables. Therefore, its data enhancement effect is obvious. In addition, with the increase of generated data, the performance of the data enhancement regression model is also improved accordingly.

Claims (2)

1. A regression modeling method based on regression attention generative adversarial network data enhancement, characterized by comprising the following steps: (1) Collect the original data, remove outliers and normalize the data to obtain independent variables x _i and y _i , i = 1 to M; (2) _xi and _yi can be concatenated and reconstructed to obtain a new data set D = [X, Y] = {[x ₁ , y ₁ ], [x ₂ , y ₂ ], …, [x _M , y _M ]}, M ∈ Z; (3) Use D to train a RA-GAN unsupervised until it converges; (4) Use the trained RA-GAN to generate virtual data D′=[X′,Y′]={[x′ ₁ ,y′ ₁ ],[x′ ₂ ,y′ ₂ ],…,[x′ _N ,y′ _N ]}, N∈Z; (5) The generated data can be added to the real data set D for data enhancement to obtain an enhanced data set; (6) Split the data of the enhanced dataset D into independent variables {X, X′} and dependent variables {Y, Y′} for training the regression model; (7) Input the independent variables of the test data into the trained regression model to obtain the predicted dependent variables; The regression attention generative adversarial network introduces two attention modules, namely attention module 1 in the generator and attention module 2 in the discriminator; The generator is a multi-layer perceptron, the input layer is random noise z, and the output layer outputs the generated data D′; the generated data is divided into independent variables x′ _j and dependent variables y _j ′, j∈[1,2,…,N]; the attention module 1 is a multi-layer perceptron connected to the generator, the input is the independent variable x′ _j of the generated data, and the output layer is the predicted value In order to make/> To make it as equal to y _j ′ as possible, the generator’s loss function adds a new loss term _LA1 and is updated as follows: Where D(·) refers to the output of the discriminator, x~P _g refers to the sample x coming from the generator, and β is the regression coefficient of attention module 1; The discriminator is also a multi-layer perceptron, which is used to distinguish whether the input is real data or generated data; the attention module 2 is set at the front end of the discriminator network, which is also a multi-layer perceptron; the input data D or D' is divided into independent variables and dependent variables, and the independent variables are input into the attention module 2 and output after network mapping. or/> The loss function of attention module 2 is: Among them, γ is the regression coefficient of attention module 2; then, or/> It is concatenated with the corresponding input independent variable _xi or _x′j and input into the subsequent discriminator; the loss function of the discriminator with attention module 2 is: Where D(·) refers to the output of the discriminator, x~P _r means that the sample x comes from the real data; λ represents the gradient penalty factor, Indicates that the data comes from the sample space between the real data and the generated data, /> is the discriminator gradient/> The second norm of . 2. According to the regression modeling method based on regression attention generative adversarial network data enhancement according to claim 1, it is characterized in that the network parameters of the attention module 1 can also be fine-tuned by real data D, and the fine-tuning loss is as follows: Among them, M is the number of real data, α is the fine-tuning coefficient; then the loss function of the generator can also be _LG ^* '= _LG ^* + _LA1 '.