CN111353040A - GRU-based attribute level emotion analysis method - Google Patents

Abstract

The invention discloses an attribute-level sentiment analysis method. Sentiment analysis is a basic task in natural language processing, and attribute-level sentiment analysis is an important topic of sentiment analysis. Different words in a sentence have different effects on the sentiment polarity of an aspect in the sentence. How to model the relationship between the attribute and the words in the sentence and the meaning of the whole sentence is the key to solving this problem. In this paper, we will use two recurrent networks to model sentence information and introduce an attention mechanism to fuse attribute information to achieve better results. In this paper, experiments on public datasets show that the algorithm proposed in this paper achieves better results without the need for complicated feature engineering.

Description

Attribute-level sentiment analysis method based on GRU

technical field

The present invention relates to the field of Internet, and more particularly, to an attribute-level sentiment analysis method based on GRU.

Background technique

With the rapid development of the Internet and more and more text information, how to obtain useful information from massive text information has become more and more important. These massive text information also objectively promote the development of natural language processing, and deep learning is a natural language processing technology. brought a new direction. Sentiment analysis (i.e. opinion mining) is a fundamental but important task in natural language processing. Enterprises can use customer comments on products to obtain timely feedback to provide reference for decision-making. Therefore, how to extract emotional information from massive text data has become an important research topic in natural language processing in recent years.

At present, the main text sentiment analysis research is mainly based on sentiment dictionary and machine learning. The method based on sentiment dictionary relies on sentiment dictionary, which has a great influence on sentiment analysis. Yang Ding et al. processed and represented text based on sentiment dictionary to construct a classifier based on Naive Bayes theory. Another approach is a machine learning based approach. The machine learning method obtains a sentiment analysis classifier by training the manually calibrated data, and the excellent classification performance of the support vector machine is proved by experiments. Both methods require manual labeling of data for sentiment dictionary construction and feature engineering, which are tedious and complex tasks that deep learning algorithms can handle well. In recent years, deep learning has achieved great success in natural language processing, such as machine translation, question answering systems. It also has applications in the field of sentiment analysis. Socher et al. proposed a deep learning method based on semi-supervised recurrent auto-encoder (RAE) to achieve text sentiment classification; Jurgovsky et al. used convolutional neural network (CNN) to achieve text sentiment classification. Text sentiment analysis can be divided into chapter level, sentence level and word level. This paper mainly studies the sentiment analysis based on aspect. This is because the emotional polarity of different aspects may be different in the same sentence. For example, in the sentence "The voice quality of this phone is not good, but the battery life is long." The evaluation of the sentence is negative, but it is positive for battery life. Wang et al. proposed AE-LSTM, AT-LSTM, and AEAT-LSTM recurrent network algorithms for aspect-granular sentiment analysis, which integrated aspect information into long short-term memory network LSTM to improve classification accuracy. The SVM-dep algorithm divides the features into the features related to the attribute aspect and the features not related to the aspect, and extracts them respectively to complete the sentiment analysis based on the attribute level.

The attention mechanism is a mechanism that selectively focuses on some important information during information processing, while ignoring and paying attention to an information processing mechanism that is less relevant to the meaning of the target. Essentially information, it has achieved great success by concentrating limited resources on the processing of important information. Attention mechanism has achieved great success in image recognition, automatic translation and other fields. Combined with the theme of this paper, when dealing with attribute-based sentiment analysis, we can pay more attention to attribute-related information to improve the accuracy of sentiment classification.

Recurrent network (RNN) is widely used in natural language processing because of its network memory ability to process contextual information. Typical recurrent networks include long short-term memory network (LSTM), gated recurrent unit (GRU) and MUT network. This paper proposes an algorithm for attribute granular sentiment analysis based on GRU network, and then integrates attribute information into the model through the attention mechanism, so that the algorithm model can pay more attention to the influence of attributes on sentiment classification, thereby improving the accuracy of sentiment classification.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide an attribute-level sentiment analysis model and method based on GRU network. The present invention is based on the attribute-level sentiment classification algorithm of Att-CGRU to improve sentiment classification accuracy.

In order to achieve the above purpose, the attribute-level sentiment analysis model based on the GRU network designed by the present invention includes the following:

In the Att-CGRU model, the introduction of the attention mechanism to reflect the attribute has a very important impact on the emotional polarity of the entire sentence. In the processing of sequence problems, the encoder-decoder is a very commonly used model. By assigning different weights to the hidden state vector of the encoding output according to different algorithms and task goals in the encoder-decoder model, extracting To improve the performance of the algorithm model by characterizing the vector representation of the input data as much as possible, this process essentially concentrates limited resources on information that is more relevant to the target task to improve the performance of the algorithm. The specific structure of the Att-CGRU model is shown in Figure 1 of the description. The model consists of five parts namely input layer, embedding layer, GRU layer, attention layer and output layer. The input layer inputs short texts or sentences into the model; the embedding layer maps each word in the sentence into a vector; the GRU layer uses the word embedding to obtain feature information; the attention layer implements the attention mechanism, which will pass weights Calculate and fuse word-level feature information into sentence-level feature information to generate a sentence feature vector; finally classify the sentence feature vector.

1.1 Input layer

In the input layer, input each sentence that needs to be classified by sentiment polarity. Assuming that the sentence length is T, the sentence can be expressed as s={x ₁ ,x ₂ ,...,x _T }, and x _i represents the first sentence in the sentence. i words.

1.2 Embedding layer

In a sentence s={x ₁ ,x ₂ ,...,x _T } obtained from the input layer containing T words, each word gets its corresponding word vector e _i in the embedding layer

中获得每一个词的词向量，这里V是词表的长度，d^w是可以指定的词向量维数，则有First from the word embedding matrix

The word vector of each word is obtained in , where V is the length of the vocabulary, and d ^w is the dimension of the word vector that can be specified, then there are

emb _i =W ^wrd v ⁱ (1)

where v ⁱ is a vector of length |V|, which is 1 at i and 0 elsewhere. The word vector emb _asp of aspect can also be obtained. When there are multiple words in the sentence, the word vector of aspect is obtained by adding up the values of the same dimension of the word vector of each word. Then concatenate emb _i and emb _asp to get the final word vector e _i :

e _i = [emb _i : emb _asp ] (2)

Finally, e={e ₁ ,e ₂ ,...,e _T } is input to the next layer.

1.3 GRU layer

In the GRU layer, the attribute is used as the dividing point, and the sentence is divided into left and right parts to model the attribute context. Its structure is shown in Figure 1, where {x _l+1 ,x _l+2 ,...,x _{r -1} } represents aspect, {x ₁ ,x ₂ ,...,x _l } represents the word before the attribute in the sentence, {x _r-1 ,x _r-2 ,...,x _T } represents the word after the attribute word. After inputting the left and right sequences into the left and right networks, the hidden layers get {h ₁ ,h ₂ ,...,h _r-1 } and {h _l+1 ,h _l+2 ,...,h _T respectively }.

1.4 attention layer

The attention mechanism is introduced into this model to obtain better classification results, because different words and attributes in the front and back parts of the sentence have different connections, and more attention will be paid to the information that is closely related to the attributes. The attention mechanism is implemented as follows:

a _t =softmax(w ^T M) (4)

r=Ha _t (5)

表示重复e_asp多次至和H的维度保持一致,H 是模型中隐藏层输出组成的矩阵，r表示的是加权后的表示句子含义的向量,W_h、 W_v、w是参数矩阵，然后得到能够最终表针句子信息的向量oHere a _t represents the attention weight coefficient,

Represents repeating e _asp multiple times to be consistent with the dimension of H, H is the matrix composed of the output of the hidden layer in the model, r represents the weighted vector representing the meaning of the sentence, W _h , W _v , w are the parameter matrix, and then Get the vector o that can finally pin the sentence information

o ₌ tanh(Wpr+ _Wxh ) (6)

h represents the sum of h _r-1 and h _l+1 vectors.

1.5 Output layer

Finally, the output o of the attention layer is input to the classifier

Implements sentiment polarity classification, where _Wo and _bo are the parameter matrices to be trained.

The specific experimental steps of this method are as follows:

Step S1, first input the data set collected on Twitter used in the present invention into the input layer of the Att-CGRU model,

Step S2, input the data obtained in S1 into the embedding layer to obtain the word vector of each word in the input sentence,

Step S3: After obtaining the word vector of each word in the sentence in the GRU layer by means of S2, take the attribute word {x _l+1 ,x _l+2 ,...,x _r-1 } as the dividing point to divide the left The word vector of the word {x ₁ ,x ₂ ,...,x _l } and the word vector of the word on the right {x _r-1 ,x _r-2 ,...,x _T } are input to two left and right GRUs The network models the context of the attribute word separately, and obtains the outputs {h ₁ ,h ₂ ,...,h _r-1 } and {h _l+1 ,h _l+2 ,...,h _T from the hidden layer, respectively }.

Step S4, according to the output of S4, according to the following formula to calculate the vector o that can represent the sentence information, the specific formula is as follows:

a _t =softmax(w ^T M)

_r =Hat

表示重复属性词词向量e_asp多次至和H的维度保持一致,H是模型中隐藏层输出组成的矩阵,tanh代表tanh函数，W_h、W_v、 w是参数矩阵。最终得到能够最终表针句子信息的向量oHere r represents the weighted vector that can represent the meaning of the sentence, and a _t represents the attention weight coefficient, which is obtained by inputting w ^T M into the softmax function, and M represents a hidden layer in the GRU layer of the model. The vector obtained from the matrix H composed of the output,

Indicates that the word vector e _asp is repeated several times until the dimension of H is consistent. H is the matrix composed of the output of the hidden layer in the model, tanh represents the tanh function, and W _h , W _v , and w are the parameter matrices. Finally, the vector o that can finally pin the sentence information is obtained

o ₌ tanh(Wpr+ _Wxh )

h represents the sum of h _r-1 and h _l+1 vectors, h _r-1 represents the output of the hidden layer corresponding to the r-1th word in the left GRU network, and h _l+1 represents the l+th word in the right GRU network The output of the hidden layer corresponding to 1 word, W _p and W _x represent the parameter matrix.

具体由

得出，W_o和b_o都是参数矩阵。Step S5, the output layer is to input the vector o that can indicate sentence information into the softmax function to obtain the predicted sentiment polarity.

Specifically by

It is concluded that both W _o and b _o are parameter matrices.

Step 6. Calculate the loss function value loss according to the output of S5 and the actual classification y corresponding to each sentence

Among them, λ is the regularization coefficient, and the error back-propagation algorithm is trained to iterate until the Accuracy gets the maximum value. The optimization algorithm in the error back-propagation algorithm is the AdaGrad algorithm with an initialization coefficient of 0.01.

Compared with the prior art, the present invention has the following technical effects.

This method is compared with traditional machine learning methods (including support vector machine algorithm, SVM-dep algorithm) and deep learning methods (AdaRNN-w/E, AdaRNNcomb, TC-LSTM). Accuracy rate (Accuracy) to evaluate various models, the results are shown in the following table:

Table 1 Experimental results

Description of drawings

Figure 1 is a specific structural diagram of the Att-CGRU model.

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. In the structural diagram of the Att-CGRU model in FIG. The five parts are input layer, embedding layer, GRU layer, attention layer and output layer. The input layer inputs short texts or sentences into the model; the embedding layer maps each word in the sentence into a vector; the GRU layer uses the word embedding to obtain feature information; the attention layer implements the attention mechanism, which will pass weights Calculate and fuse word-level feature information into sentence-level feature information to generate a sentence feature vector; finally classify the sentence feature vector.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

When implementing the present invention, a data set must be collected first, and the data set used in the present invention is a basic data set collected on Twitter.

The specific experimental steps of this algorithm are as follows:

Step S1, the data set used in this paper is a basic data set collected on Twitter. Each training and testing data has been manually calibrated. The training dataset is used to train the model, and the test dataset is used to test the model performance. The training dataset has 6248 sentences and the test dataset has 692 sentences. 25%, 25%, and 50% of positive, negative, and neutral data in test and training datasets, respectively

Step S2, the model in this paper includes five parts, namely the input layer, the embedding layer, the GRU layer, the attention layer and the output layer. The input layer inputs short texts, namely sentences, into the model. The sentences can be represented as s={x ₁ ,x ₂ ,...,x _T } where x _i represents the ith word that composes the sentence, and T represents the length of the sentence. The embedding layer maps each word x _i in the sentence into a word vector e _i =[emb _i : emb _asp ] according to the word vector dictionary, where emb _i represents the word vector corresponding to the i-th word in the dictionary, and emb _asp represents the word vector is the word vector of the attribute word. When the attribute word consists of multiple words, take the mean of the word vectors of these words. The GRU layer will use the attribute as the demarcation point to obtain the semantic feature information from the embedding layer, and divide the sentence into left and right parts to model the attribute context. Its structure is shown in Figure 1, where {x _l+1 , x _{l +2} ,...,x _r-1 } represents the attribute aspect, {x ₁ ,x ₂ ,...,x _l } represents the word before the attribute in the sentence, {x _r-1 ,x _r-2 ,. ..,x _T } represents the word after the attribute. After inputting the left and right sequences into the left and right GRU networks, the hidden layers get {h ₁ ,h ₂ ,...,h _r-1 } and {h _l+1 ,h _l+2 ,...,h respectively _T }. The attention layer implements the attention mechanism. It fuses the word-level feature information into sentence-level feature information through weight calculation to generate a sentence feature vector and finally classifies the sentence feature vector. The specific implementation is as follows

a _t =softmax(w ^T M)

_r =Hat

表示重复属性词词向量e_asp多次至和H的维度保持一致,H是模型中隐藏层输出组成的矩阵,tanh代表tanh函数，W_h、W_v、w是参数矩阵。最终得到能够最终表针句子信息的向量o；Here r represents the weighted vector that can represent the meaning of the sentence, and a _t represents the attention weight coefficient, which is obtained by inputting w ^T M into the softmax function, and M represents a hidden layer in the GRU layer of the model. The vector obtained from the matrix H composed of the output,

Indicates that the word vector e _asp is repeated several times until the dimension of H is consistent. H is the matrix composed of the output of the hidden layer in the model, tanh represents the tanh function, and W _h , W _v , and w are the parameter matrices. Finally, the vector o that can finally indicate the sentence information is obtained;

o ₌ tanh(Wpr+ _Wxh )

具体由

得出，W_o和b_o都是参数矩阵。h represents the sum of h _r-1 and h _l+1 vectors, h _r-1 represents the output of the hidden layer corresponding to the r-1th word in the left GRU network, and h _l+1 represents the l+th word in the right GRU network The output of the hidden layer corresponding to 1 word, W _p and W _x represent the parameter matrix. The output layer is to input the vector o that can indicate sentence information to the softmax function to obtain the predicted sentiment polarity

Specifically by

It is concluded that both W _o and b _o are parameter matrices.

表示预测结果。训练的过程是最小化所有句子真实极性y和预测

间的交叉熵损失值：Step S3, use cross entropy as the loss function when training the model, use

represents the prediction result. The training process is to minimize the true polarity y of all sentences and predict

The cross-entropy loss value between:

Here j represents the type of sentiment polarity, which is positive, negative and neutral in this paper; i represents the index number of the sentence, λ is the second-order norm regularization coefficient, and θ is the parameter to be solved; at the same time, set the dropout probability to 0.5 to Prevent overfitting. In this method, a 200-dimensional word vector is used to initialize each word in the sentence, the dimension of the hidden layer is also 100, and the other parameter matrices are initialized as uniformly distributed sampling. The batch training method is adopted when training the model, and each batch contains 20 sentences. The L ₂ regularization coefficient λ is 0.001, the optimization algorithm adopts AdaGrad, and its initialization coefficient is 0.01.

Step S4, in the experiment, the experiments were compared with traditional machine learning methods (including support vector machine algorithm, SVM-dep algorithm) and deep learning methods (AdaRNN-w/E, AdaRNNcomb, TC-LSTM) respectively, and the evaluation Models are evaluated by accuracy (Accuracy), and the results are shown in the following table:

Table 1 Experimental results

From the experimental data, it can be seen that the method of introducing an attention mechanism based on attribute words while modeling sentences with the left and right networks has certain advantages compared with other models in terms of accuracy.

Claims (2)

1. An attribute level emotion analysis model based on a GRU network is characterized in that: the model comprises five parts, namely an input layer, an embedded layer, a GRU layer, an attention layer and an output layer; the input layer inputs short texts, namely sentences, into the model; the embedding layer maps each word in the sentence into a vector; the GRU layer acquires characteristic information by embedding words; the attention layer realizes an attention mechanism, and the attention mechanism fuses the word-level characteristic information into sentence-level characteristic information through weight calculation to generate a sentence characteristic vector; finally, classifying the sentence characteristic vectors;

1.1 input layer

Inputting each sentence needing emotion polarity classification at an input layer, and assuming that the sentence length is T, the sentence is expressed as s ═ x₁,x₂,...,x_T}，x_iRepresenting the ith word in the sentence;

1.2 embedding layer

Obtaining a sentence s containing T words from an input layer₁,x₂,...,x_TAnd fourthly, obtaining a corresponding word vector e of each word in the embedding layer_i

First embedding a matrix from words

A word vector for each word is obtained, where V is the length of the vocabulary, d^wIs a word vector dimension that can be specified, then there is

emb_i＝W^wrdvⁱ(1)

Wherein v isⁱIs a vector of length | V | where i is 1 and others are 0; likewise, the word vector emb of aspect is obtained_aspWhen the aspect in the sentence is a plurality of words, adding the values of the same dimensionality of the word vector of each word to obtain the word vector of the aspect; then will emb_iAnd emb_aspSpliced to obtain a final word vector e_i：

e_i＝[emb_i:emb_asp](2)

Finally, e is ═ e₁,e₂,...,e_TThe input is to the next layer;

1.3 GRU layer

In the GRU layer, the sentence is divided into left and right parts by taking the attribute as a demarcation point to model the context of the attribute, wherein { x_l+1,x_l+2,...,x_r-1Denotes aspect, { x₁,x₂,...,x_lDenotes the word before the attribute in the sentence, { x_r-1,x_r-2,...,x_TRepresents the words after the attribute; the left and right sequences are input into the left and right networks, and then the hidden layer respectively obtains { h₁,h₂,...,h_r-1H and_l+1,h_l+2,...,h_T}；

1.4 attention layer

An attention mechanism is introduced into the model to obtain a better classification effect, because different words and attributes of the front part and the rear part in the sentence are in different relations, more information which is closely related to the attributes is concerned; the attention mechanism is implemented as follows:

a_t＝softmax(w^TM) (4)

r＝Ha_t(5)

where a is_tIt is indicated that the attention weight coefficient,

denotes repetition e_aspMultiple times until the dimension of the model is consistent with that of H, H is a matrix formed by hidden layer outputs in the model, r represents a weighted vector representing the meaning of a sentence, W_h、W_vW is a parameter matrix, and then a vector o capable of finally representing sentence information is obtained

o＝tanh(W_pr+W_xh) (6)

h represents h_r-1And h_l+1The sum of the vectors;

1.5 output layer

Finally, the output o of the attention layer is input into the classifier

Implementing a polarity classification of the emotion, where W_oAnd b_oIs a ginseng to be trainedA matrix of numbers.

2. The attribute level emotion analysis method based on GRU is characterized by comprising the following steps: the method comprises the following specific steps:

step S1, firstly, inputting the collected twitter data set to an input layer of the Att-CGRU model;

step S2, inputting the data obtained in step S1 into an embedding layer to obtain a word vector of each word in the input sentence,

step S3, obtaining the word vector of each word in the sentence through the mode of S2 in the GRU layer, and then using the attribute words { x_l+1,x_l+2,...,x_r-1As the demarcation point, the left side { x }₁,x₂,...,x_lWord vector and the right side of the word { x }_r-1,x_r-2,...,x_TInputting the word vector of the word into two left and right GRU networks to model the context of the attribute word respectively, and obtaining output { h } from the hidden layer respectively₁,h₂,...,h_r-1H and_l+1,h_l+2,...,h_T}；

step S4, according to the output of S4, calculating a vector o capable of representing sentence information according to the following formula:

a_t＝softmax(w^TM)

r＝Ha_t

where r denotes a weighted vector, a, which characterizes the meaning of the sentence_tDenoted is an attention weight coefficient, which is formed by dividing w^TM is obtained after inputting the softmax function, M represents a vector obtained by a matrix H formed by the output of the hidden layer in the model GRU layer,

word vector e representing repetitive attributes_aspMultiple times until the dimension of the model is consistent with that of H, H is a matrix formed by hidden layer outputs in the model, tanh represents a tanh function, W_h、W_vW is a parameter matrix; finally, the vector o capable of finally pointing the sentence information is obtained

o＝tanh(W_pr+W_xh)

h represents h_r-1And h_l+1Sum of vectors, h_r-1Denotes the hidden layer output, h, corresponding to the r-1 th word in the left GRU network_l+1Denotes the hidden layer output, W, corresponding to the l +1 th word in the right GRU network_pAnd W_xRepresenting a parameter matrix;

in step S5, the output layer inputs the vector o capable of representing sentence information into the softmax function to obtain the predicted emotion polarity

In particular by

To obtain W_oAnd b_oAre all parameter matrices; step 6, calculating the loss function value loss according to the output of S5 and the actual classification y corresponding to each sentence

Wherein lambda is a regularization coefficient, and training iteration is carried out to Accuracy through an error back propagation algorithm to obtain a maximum value, and an optimization algorithm in the error back propagation algorithm is an AdaGrad algorithm with an initialization coefficient of 0.01.

Images