CN116467930A - A General Modeling Method for Structured Data Based on Transformer - Google Patents

Abstract

The invention relates to a general modeling method of structured data based on a transducer, which comprises the steps of firstly removing irrelevant features from original data, then using different embedding methods for category features and numerical features, then splicing feature vectors after embedding the category features and the numerical features, and inputting the spliced feature vectors into a transducer+neural network (improved transducer) and an MLP+neural network, wherein the transducer+neural network is formed by adding a leakage Gate before the original transducer and an MLP+neural network after the original transducer, and finally distributing different weights for output values of two modules. The invention is applicable to both classification problems and multi-classification problems.

Description

A General Modeling Method for Structured Data Based on Transformer

technical field

The invention belongs to the field of structured data processing, in particular to a Transformer-based general modeling method for structured data

Background technique

Tabular data is the most commonly used form of data, and it is ubiquitous in various applications such as medical diagnosis based on medical records, predictive analytics in the financial sector, cybersecurity, etc. At present, tree-based integration methods are generally used, such as the gradient boosting decision tree GBDT, which has a good effect in processing tabular data, mainly because it has better learning ability for continuous numerical features, and can automatically select and combine useful numerical features, and effectively construct decision trees by calculating information gain. However, since category features are generally transformed into high-dimensional sparse one-hot encoding, the gradient boosting decision tree GBDT will obtain little information gain when processing such data, and cannot effectively learn such features.

In recent years, methods based on the Transformer framework have achieved great success in the fields of computer vision and natural language processing. In the field of computer vision, the setting of the convolution kernel limits the size of the receptive field, resulting in the network often requiring multi-layer stacking to pay attention to the entire feature map; in the field of natural language processing, RNN or LSTM needs to accumulate information in several time steps to connect the two. The farther the distance is, the less likely it is to be effectively captured. The self-attention self-attention in Transformer can capture global attention information. In addition, it also directly helps to increase the parallelism of calculations, which is the main reason why Transformer is widely used.

Multilayer perceptron (MLP) is probably the simplest and most general neural network. Multilayer perceptron MLP usually learns parameter embeddings to encode categorical data features, but due to their relatively shallow architecture and use of context-free embeddings, they are not robust to missing and noisy data. Most importantly, in most cases, multilayer perceptron MLP does not perform as well as tree-based models.

To sum up, effectively learning tabular data and overcoming the above problems are urgent problems to be solved in the application of deep learning in the tabular field.

Contents of the invention

In order to solve the shortcomings of existing tree-based integration methods in table prediction, the present invention provides a Transformer-based general modeling method for structured data.

In order to solve the above-mentioned technical problems, the present invention adopts the following technical solutions to realize:

A Transformer-based general modeling method for structured data, comprising the following steps:

(1) Input the public data set for feature processing: After obtaining the original data, it is necessary to eliminate irrelevant features, encode the category features in the data into recognizable digital forms, and scale the numerical features according to standardized operations;

(2) Word embedding of the feature vector after feature processing: before passing through the Transformer+encoder, the high-dimensional discrete data of numerical features and category features are projected into a low-dimensional dense d-dimensional space through word embedding Embedding;

(3) Input the word embedding embedding vector obtained in the previous step into two branches of the model: the model is divided into two branches of Transformer+neural network and MLP+neural network, and the feature vector of the training data after word embedding Embedding is input into the Transformer+neural network for learning, and the original output of the neural network is obtained, and the same input is input into the MLP+neural network for modeling learning, and a trained MLP+neural network is obtained. Transformer+neural network and MLP+neural network are fused into a classification model, so the two parts of the original output The weighted sum forms the overall output value of the model, and then the overall prediction result given by the classification model is obtained through the activation function;

(4) Use the focal loss Focal Loss as the objective function to guide the training: use the preprocessed training data to train the classification model, use the focal loss Focal Loss as the objective function to guide the training process, search for the best parameters, and obtain the trained classification model;

(5) Accepting other tabular data for prediction: the preprocessing will be performed on the tabular data to be classified, and then input into the trained classification model for classification prediction.

Further, in said step (1), the method for input feature processing includes the following steps:

(1-1) Eliminate useless features: perform feature recognition on each data set based on prior knowledge, and remove useless features;

(1-2) Processing continuous features: continuous features are standardized with the standard scaler StandardScaler, and the numerical features are scaled;

(1-3) Processing category features: Category features use the label encoder LabelEncoder to encode the features into digital form. In order to avoid encoding sparseness and increase the computational cost, one-hot encoding is not performed.

Further, in the step (2), word embedding Embedding is a technology for mapping feature vectors to low-dimensional space vectors, which can convert discrete feature vectors into continuous vector representations, perform general word embedding Embedding processing for category features, and use a separate fully connected layer for numerical features. Each numerical feature has a ReLU nonlinearity, thereby projecting 1-dimensional input to d-dimensional space, and then connecting the embedding of category features and numerical features in the first dimension.

Further, in the step (3), the neural network model includes the following parts:

(3-1) The neural network Transformer+ has the following improvements relative to the original transformer Transformer. A leaky gate Leaky Gate is added before the encoder encoder of the original transformer Transformer, and an MLP+ neural network is added afterwards. The leaky gate Leaky Gate is a combination of two simple elements, that is, an element-level linear transformation and a LeakyRelu activation function;

(3-2) The neural network MLP+ has the following improvements relative to the multi-layer perceptron MLP. Starting from the sub-block of the multi-layer perceptron MLP, the ordinary batch normalization Batch Norm is replaced by Ghost normalized Ghost Batch Norm (GBN), and a linear skip layer is added on the right side of the sub-block. Ghost normalized Ghost Batch Norm (GBN) allows training with large batches of data, and a great motivation for using Ghost normalized GhostBatch Norm (GBN) in the present invention is to speed up training.

The present invention provides a Transformer-based general modeling method for structured data, which is characterized in that the Transformer is used to process category features and numerical features at the same time, and on the premise of fully retaining the performance of the Transformer model, it is fused with a multi-layer perceptron MLP into a model, instead of giving category predictions and then weighted voting, so it can be optimized by introducing a loss function in end-to-end training, effectively enhancing the recognition ability of the model. Compared with prior art, positive effect of the present invention:

1. The present invention proposes a data processing method that enters both categorical features and numerical features into Transformer, which means that any information about the correlation between categorical features and numerical features will not be lost.

2. The present invention proposes a Transformer-based general modeling method for structured data, which can effectively integrate the simpler MLP neural network and the more complex attention-based Transformer neural network, thereby learning category features and numerical features.

3. The present invention evaluates the proposed new model using seven public datasets such as adult, blastchar, and shrutime, and the experimental results show that the method of the present invention is superior to other advanced methods in the binary classification scenario.

Description of drawings

Fig. 1 is the overall frame diagram of the method of the present invention.

Fig. 2 is the processing flow of the MLP+ of the present invention.

specific implementation plan

The technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. The specific implementations described here are only used to explain the present invention, not to limit the present invention.

Example 1

Fig. 1 is the overall structure diagram of the present invention, a kind of general modeling method of structured data based on Transformer, concrete steps are as follows:

Step (1) input feature processing;

At the data level, public data sets such as adult, blastchar, and spambase are used. Some data sets only have numerical features, and some data sets contain both numerical features and categorical features. At the same time, the data is divided into two parts: training set and test set. For different data sets, we use prior knowledge to remove some useless features. Since most of the categorical features are in the form of strings, they are encoded as numbers (1, 2, 3...) that the model can recognize; for numerical features, standardization operations are used for scaling.

For the original data (including training set and test set), unnecessary features are removed, categorical features are numerically coded, and numerical features are standardized to obtain a data set D={( _xi , y _{i )} , y _i ∈ [0, classnum), i=1, 2, 3,..., N} (where xi is the feature vector of each sample, y is the label corresponding _to xi, classnum is the number of categories, and N is the number of samples). To distinguish different types of features, divide the data into categorical features x _cat and Numeric feature x _cont .

Step (2) category feature and numerical feature embedding;

The embedding layer E embeds each feature into a d-dimensional space. In order to effectively process tabular data, the present invention treats discrete categorical features and continuous numerical features differently. The present invention obtains a new embedded representation of category features through word embedding Embedding technology, and obtains a new embedded representation of numerical features by using a fully connected layer. is a single sample with categorical or numerical features, the embedding layer e uses different embedding functions for different types of features, for a given /> get /> Then stitching is performed in the feature dimension, and E _Φ (X) is the result of embedding representation of all features.

E _Φ (x) = {e _Φ1 (x ₁ ),..., e _ΦN (x _N )} (1)

Step (3) inputting the feature vector of the embedding output into the model;

(3-1) The feature vector output from the previous step first enters the Leaky Gate, which is a combination of two simple elements, an element-level linear transformation, and then the LeakyRelu activation function. The LeakyRelu activation function will let any positive value pass through without changing, and compress any negative value to almost zero. In other words, if w _i and b _i are the linear layer parameters of the i-th column, the leaky gate of the i-th column is:

A Leaky Gate is intended to act as a simple filter, with a different behavior for each column, where whether to block or pass depends on each individual value.

The Transformer layer of the converter takes the output of the Leaky Gate as input, and passes the output to the second Transformer layer, and so on. As shown in Figure 1, the output of the last Transformer layer will be directly input into the MLP+ neural network (improved multi-layer perceptron MLP). The MLP+ neural network is shown in Figure 2, and the output value y _Transformer+ of the model is obtained. Among them, θ ₁ , θ ₂ and θ ₃ are the model parameters of Leaky Gate, Transformer, and MLP+neural network, respectively.

y _Transformer+ (x)=M(f _transformer (G _Θ (E _Φ (x); θ ₁ ); θ ₂ ); θ ₃ ) (3)

(3-2) Similarly, the feature vector output in the previous step enters the MLP+ neural network (right branch in Figure 1), and the output value y _mlp+ of the model is obtained.

y _mlp+ = M(E _Φ (X); θ ₁ ) (4)

Step (4) fusion of left and right branches;

Specifically, in order to combine the improved converter Transformer and the improved multi-layer perceptron MLP to obtain the prediction of the entire model and perform end-to-end training, the present invention assigns different weights w ₁ and w ₂ to the output values of the two modules (the two weights can be learned through backpropagation training), and the predicted probability of the final model output As in formula (5), σ represents the activation function (sigmoid for binary classification and softmax for multi-classification).

Step (5) trains the classification model based on focal loss Focal Loss;

The preprocessed data is used for model training, and the focal loss Focal Loss is used as the loss function to guide the training process. The focal loss Focal Loss can make the model pay more attention to the minority class samples that are difficult to classify, and reduce the deviation caused by the majority class.

According to Equation (5), the loss of the model can be expressed as Equation (6), Represents the loss function, and y is the real label of the sample x.

In order to deal with the problem of unbalanced data categories, the present invention adopts the idea of cost sensitivity, and introduces Focal Loss as the loss function of the model. Focus loss Focal Loss was originally used to solve the category imbalance problem in target detection tasks, and is an improvement on the traditional cross-entropy loss. The present invention introduces it into the field of table classification. For the binary classification problem, the focal loss Focal Loss can be expressed in the form of formula (7), where is the probability prediction defined in Equation (5), _yi is the label of the input sample, α is the balance factor, and γ≥0 is called the focusing parameter.

For multi-classification problems, one-to-many thinking can be adopted to extend formula (7) to formula (8), where y is the one-hot encoded one-hot representation of the category label, is a probability output of the form (m, n) (m is the number of samples, n is the number of categories).

Based on the loss function defined by Equation (7) and Equation (8), end-to-end model training can be performed, and the gradient off-shelf method is used to select the model with the smallest loss.

Example 2

Application of the present invention provides a commodity recommendation method based on Transformer's general modeling method for structured data.

Fig. 1 is an overall architecture diagram of the present invention, and the concrete steps of described method are as follows:

Step (1) Input feature processing.

In the application scenario of a recommendation system, taking the product recommendation system as an example, a general modeling method for structured data based on Transformer is used to classify the product recommendation system according to the user's behavior, thereby recommending corresponding types of products. At the data level, use online_shoppers to expose the data set, which contains both numerical features and categorical features. At the same time, the data is divided into two parts: training set and test set. For this dataset, we use prior knowledge to remove some useless features. Since most of the categorical features are in the form of strings, they are encoded as numbers (1, 2, 3...) that the model can recognize; for numerical features, standardization operations are used for scaling.

For the original data (including training set and test set), unnecessary features are removed, categorical features are numerically coded, and numerical features are standardized to obtain a data set D={( _xi , y _{i )} , y _i ∈ [0, classnum), i=1, 2, 3,..., N} (where xi is the feature vector of each sample, y is the label corresponding _to xi, classnum is the number of categories, and N is the number of samples). To distinguish different types of features, divide the data into categorical features x _cat and Numeric feature x _cont .

Step (2) categorical features and numerical feature embeddings.

The embedding layer E embeds each feature into a d-dimensional space. In order to effectively process tabular data, the present invention treats discrete categorical features and continuous numerical features differently. The present invention obtains a new embedded representation of category features through word embedding Embedding technology, and obtains a new embedded representation of numerical features by using a fully connected layer. is a single sample with categorical or numerical features, the embedding layer e uses different embedding functions for different types of features, for a given /> get /> Then splice in the feature dimension, E _Φ (X) is the result of embedding representation of all features:

E _Φ (x) = {e _Φ1 (x ₁ ),..., e _ΦN (x _N )} (1)

Step (3) feeds the feature vector of the embedding output into the model.

(3-1) The feature vector output from the previous step first enters the Leaky Gate, which is a combination of two simple elements, an element-level linear transformation, and then the LeakyRelu activation function. The LeakyRelu activation function will let any positive value pass through without changing, and compress any negative value to almost zero. In other words, if w _i and b _i are the linear layer parameters of the i-th column, the leaky gate of the i-th column is:

A Leaky Gate is intended to act as a simple filter, with a different behavior for each column, where whether to block or pass depends on each individual value.

The Transformer layer of the converter takes the output of the Leaky Gate as input, and passes the output to the second Transformer layer, and so on. As shown in Figure 1, the output of the last Transformer layer will be directly input into the MLP+ neural network (improved multi-layer perceptron MLP). The MLP+ neural network is shown in Figure 2, and the output value y _Transformer+ of the neural network is obtained. Among them, θ ₁ , θ ₂ and θ ₃ are the model parameters of Leaky Gate, Transformer, and MLP+neural network, respectively.

y _Transformer+ (x)=M(f _transformer (G _Θ (E _Φ (x); θ ₁ ); θ ₂ ); θ ₃ ) (3)

(3-2) Similarly, the feature vector output in the previous step enters the MLP+ neural network (right branch in Figure 1), and the output value y _mlp+ of the neural network is obtained:

y _mlp+ = M(E _Φ (X); θ ₁ ) (4)

Step (4) fuses the left and right branches.

Specifically, in order to combine the improved Transformer and the improved multi-layer perceptron MLP to obtain the prediction of the entire model and perform end-to-end training, the present invention assigns different weights w ₁ and w ₂ to the output values of the two modules (the two weights can be learned through backpropagation training), and the predicted probability of the final model output As in formula (5), σ represents the activation function (sigmoid for binary classification and softmax for multi-classification).

Step (5) trains the classification model based on Focal Loss.

The preprocessed data is used for model training, and the focal loss Focal Loss is used as the loss function to guide the training process. The focal loss Focal Loss can make the model pay more attention to the minority class samples that are difficult to classify, and reduce the deviation caused by the majority class.

According to Equation (5), the loss of the model can be expressed as Equation (6), Represents the loss function, and y is the real label of the sample x.

In order to deal with the problem of unbalanced data categories, the present invention adopts the idea of cost sensitivity, and introduces Focal Loss as the loss function of the model. Focus loss Focal Loss was originally used to solve the category imbalance problem in target detection tasks, and is an improvement on the traditional cross-entropy loss. The present invention introduces it into the field of table classification. For the binary classification problem, the focal loss Focal Loss can be expressed in the form of formula (7), where is the probability prediction defined in Equation (5), _yi is the label of the input sample, α is the balance factor, and γ≥0 is called the focusing parameter.

For multi-classification problems, one-to-many thinking can be adopted to extend formula (7) to formula (8), where y is the one-hot encoded one-hot representation of the category label, is a probability output of the form (m, n) (m is the number of samples, n is the number of categories).

Based on the loss function defined by Equation (7) and Equation (8), end-to-end model training can be performed, and the gradient off-shelf method is used to select the model with the smallest loss.

Step (6) Input user characteristics into the model to realize product recommendation.

When the product recommendation system acquires user behavior or adds or modifies behavior on the basis of the original behavior, the product recommendation system inputs the newly constructed user behavior into the model, obtains new classification results, and then recommends corresponding products.

The numerical feature and category feature embedding module is used to collect feature vectors embedded in new user behaviors for subsequent model input.

Said inputting the embedding output feature vector into the model module is used to adjust the parameters by inputting the new feature vector into the model.

The Focal Loss-based training classification model module is used to train a new model after parameters are changed.

Those of ordinary skill in the art can understand that the above descriptions are only preferred examples of the invention, and are not intended to limit the invention. Although the invention has been described in detail with reference to the aforementioned examples, those skilled in the art can still modify the technical solutions described in the aforementioned examples, or perform equivalent replacements for some of the technical features. All modifications, equivalent replacements, etc. within the spirit and principles of the invention shall be included in the scope of protection of the invention.

Claims (7)

1. The general modeling method for the structured data based on the Transformer is characterized by comprising the following steps of:

(1) The input public data set is subjected to characteristic processing: after the original data is obtained, irrelevant features are required to be removed, category features in the data are encoded into identifiable digital forms, and the digital features are scaled according to standardized operation;

(2) Word Embedding is carried out on the feature vectors after feature processing, and the feature vectors are embedded into the Embedding: projecting the high-dimensional discrete data of the numerical features and the class features into a low-dimensional dense d-dimensional space by word Embedding before passing through an encoder of a transfomer+neural network;

(3) Inputting the word embedded coding vector obtained in the step (2) into two branches of a model: the model is divided into two branches of a transducer+neural network and an MLP+neural network, the feature vector of training data after word Embedding is input into the transducer+neural network for learning, the original output of the neural network is obtained, the same input is input into the MLP+neural network for modeling learning, a trained MLP+neural network is obtained, the transducer+neural network and the MLP+neural network are fused into a classification model, so that the two parts of original outputs are weighted and summed to form the integral output value of the model, and then the integral prediction result given by the classification model is obtained through an activation function;

(4) Training is guided by adopting Focal Loss as an objective function: training the classification model by utilizing the preprocessed training data, guiding the training process by adopting Focal Loss Focal Loss as an objective function, and searching the optimal parameters to obtain a trained classification model;

(5) Receive other tabular data for prediction: and preprocessing the form data to be classified, and inputting the form data to the trained classification model for classification prediction.

2. The method of claim 1, wherein the method of input feature processing of step (1) comprises the steps of:

(1-1) culling the useless features: carrying out feature recognition on each data set according to priori knowledge, and eliminating useless features;

(1-2) processing the sequential features: the continuous features are standardized by a standard scaler, and the numerical features are scaled;

(1-3) processing category characteristics: the class features are coded into digital form by a label coder LabelEncoder, so that the calculation cost is increased in order to avoid sparse coding, and single-hot one-hot coding is not carried out.

3. The method of claim 1, wherein the word Embedding method of step (2) is a technique of mapping feature vectors to low dimensional space vectors, wherein discrete feature vectors are converted into continuous vector representations, wherein general word Embedding method processing is performed for category features, wherein a separate full-connected layer is used for numerical features, wherein each numerical feature has ReLU nonlinearity, thereby projecting 1-dimensional input into d-dimensional space, and wherein Embedding of category features and numerical features is performed in a first dimension.

4. A method as claimed in claim 3, wherein: the embedding of the category characteristics and the numerical characteristics in the first dimension is connected, and the method specifically comprises the following steps: the embedding layer E embeds each feature into d-dimensional space, and in order to effectively process table data, the invention discriminates discrete type features and continuous numerical features. The invention obtains new embedded representation of category characteristics through word Embedding technology, obtains new embedded representation of numerical characteristics through using a full connection layer, and obtains x _i ＝[f _i ^{1} ,f _i ^{2} ,...,f _i ^{n} ]Is a single sample with class or numerical features, the embedding layer e uses different embedding functions for different types of features, for a givenObtain->Then splice in the feature dimension, E _Φ (X) is the result of all features being represented by the embedding:

E _Φ (x)＝{e _Φ1 (x ₁ ),...,e _ΦN (x _N )} (1)

5. the method of claim 1, wherein the model of step (3) comprises the following parts:

(3-1) the neural network fransformer+ is improved over the converter fransformer by adding a Leaky Gate before the encoder of the converter fransformer and adding an mlp+ neural network after, the Leaky Gate being a combination of two simple elements, namely a linear transformation based on element level and a Leaky relu activation function;

(3-2) the neural network mlp+ is improved relative to the multi-layer perceptron MLP by starting with a sub-block of the multi-layer perceptron MLP, replacing the normal Batch normalization Batch Norm with a Ghost normalization Ghost Batch Norm (GBN), adding a linear jump layer on the right side of the sub-block, the jump layer being just a fully connected linear layer, then a LeakyRelu activation function, and finally adding a Leaky Gate before the multi-layer perceptron MLP sub-block and the linear jump layer.

6. The method of claim 5, wherein: the steps (3-1) and (3-2) specifically comprise: (3-1) the feature vector outputted in the previous step is first entered into a leakage Gate, which is a combination of two simple elements, one linear transformation at element level, followed by a leakage Relu activation function that will let any positive value pass without change and compress any negative value to almost zero, in other words, if w _i And b _i Is the linear layer parameter of the ith column, then the leakage Gate leak Gate of the ith column is:

the Leaky Gate is intended to act as a simple filter, with different behavior for each column, with or without masking or passing depending on each individual value;

the converter layer takes the output of the Leaky Gate as input and passes the output to the second converter layer, and so on, as shown in FIG. 1, the output of the last converter layer will be directly input to the MLP+ neural network (modified multi-layer perceptron MLP), which is shown in FIG. 2, to obtain the model output value y _Transformer+ The method comprises the steps of carrying out a first treatment on the surface of the Wherein θ is ₁ ，θ ₂ And theta ₃ The model parameters of the Leaky Gate, transducer, mlp+ neural network are:

y _Transformer+ (x)＝M(f _transformer (G _Θ (E _Φ (x)；θ ₁ )；θ ₂ )；θ ₃ ) (3)

(3-2) similarly, the feature vector outputted in the previous stepInto MLP+ neural network (right branch of FIG. 1) to obtain model output value y _mlp+ ：

y _mlp+ ＝M(E _Φ (X)；θ ₁ ) (4)

7. The method of claim 1, wherein: the step (4) specifically comprises: to combine the improved converter and the improved multi-layer perceptron MLP to obtain predictions of the whole model and perform end-to-end training, the output values of the two modules are assigned different weights w ₁ And w ₂ (the two weights can be obtained by back propagation training learning), and the prediction probability of the final model outputAs in equation (5), σ represents an activation function, two is classified as sigmoid, and multiple is classified as softmax: