CN114429132A - Named entity identification method and device based on mixed lattice self-attention network - Google Patents

Abstract

The invention discloses a named entity identification method based on a mixed lattice self-attention network, which comprises the following steps: s1, coding the sentence characteristic vector expressed by the word pair into a matrix with fixed dimensionality to obtain the word vector expression with a mixed lattice structure; constructing a self-attention network to capture the influence of word vectors in the vector on the word vectors and enhance the feature representation of each word vector; word features are fused in an Embedding layer of BERT, and better word vector representation is obtained through fine tuning of a learning process; and realizing an entity sequence labeling task and a decoding process in entity identification according to a BilSTM-CRF network, completing modeling of the character characteristics after fusion through the network, and constructing and completing an entity identification model based on a mixed lattice self-attention network. The invention can capture global vocabulary information, generate word vector representation with rich semantics and improve the recognition precision of the Chinese named entity on a plurality of data sets.

Description

A Named Entity Recognition Method and Device Based on Hybrid Lattice Self-Attention Network

technical field

The invention relates to the technical field of natural language processing in artificial intelligence, in particular to a named entity recognition method and device based on a hybrid lattice self-attention network.

Background technique

Named Entity Recognition (NER), also called entity extraction, was first proposed at the MUC-6 conference. It is a technology for extracting entities from text in information extraction technology. Early entity recognition used rule-based and statistics-based methods. Since these traditional methods rely too much on manual design, the recognition coverage is small, and the recognition accuracy is low, they have been replaced by deep learning methods. In the method based on deep learning, entity recognition models are divided into character-based and word-based models. Some other languages such as English usually use word-based models, because each word is There is a clear meaning; the meaning of words in Chinese is vague, while the meaning of words is specific, so word-based models are used in Chinese NER methods. In order to better represent each word vector in Chinese, some scholars later proposed a method based on representation learning, which is a learning method that converts human language information into features that can be recognized by machines, which can improve the semantic expression in machine learning. accuracy.

In named entity recognition methods, external lexical information can effectively improve the recognition accuracy, but these methods rely on the performance of fusion algorithms. For example, the invention with the patent number CN113836930A proposes a Chinese named entity recognition method for hazardous chemicals. On the basis of the BiLSTM-CRF model, the pre-trained language model BERT is used to obtain the text character-level encoding in the field of hazardous chemicals. Information word vectors, and then introduce an attention mechanism to enhance the model’s ability to mine the global and local features of the text. The invention with the patent number of CN113128232A proposes a named entity recognition method based on ALBERT and multi-word information embedding, which can effectively characterize the ambiguity of words and improve the efficiency of entity recognition. The invention with the patent number CN111310470A discloses a Chinese named entity recognition method integrating word features. The result data obtained after comprehensive analysis strengthens the model's understanding of the text and improves the F1 value in the model recognition task.

Although the existing methods have achieved good results in the fusion of word feature vectors, the existing technical means have the following problems: 1) The word feature fusion method does not take into account the semantic expression of word vectors trained by different models. There are differences in terms of learning, and the direct and empty fusion of the two cannot effectively enhance the word-level features of word vectors; 2) In the vocabulary enhancement method based on learned word weights, only the matching word pairs of each word feature are considered. The impact of semantic representation ignores the role of global lexical information.

SUMMARY OF THE INVENTION

Aiming at the deficiencies in the prior art, the present invention provides a named entity recognition method and device based on a hybrid lattice self-attention network. Based on the idea of representation learning, the proposed model can fuse lexical information, so as to enhance the performance of word vectors. The feature representation makes the generated word vector contain more entity boundary information, which can improve the accuracy of the NER task.

To achieve the above object, the present invention adopts the following technical solutions:

A named entity recognition method based on a hybrid lattice self-attention network, the named entity recognition method comprises the following steps:

S1: Find words composed of consecutive words in the input sentence in the dictionary, merge them into a single multi-dimensional vector through positional alternate mapping, and encode the sentence feature vector represented by the word pair into a single dimension by means of mixed word lattice encoding. A fixed matrix to obtain the word vector representation of the corresponding mixed lattice structure;

S2, based on the word vector of the mixed lattice structure generated in step S1, construct a corresponding self-attention network to capture the influence of the word vector in the vector on the word vector, so as to enhance the feature representation of each word vector;

S3, fuse word features in the Embedding layer of BERT, and learn to obtain better word vector representation by fine-tuning the learning process; realize the entity sequence labeling task and decoding process in entity recognition based on the BiLSTM-CRF network, and complete the fusion process through the network. Modeling of word features, constructing an entity recognition model based on hybrid lattice self-attention network;

S4, train the entity recognition model based on the hybrid lattice self-attention network on the dataset.

In order to optimize the above technical solutions, the specific measures taken also include:

Further, in step S1, the sentence feature vector represented by the word pair is encoded into a matrix with a fixed dimension by means of mixed word lattice encoding, and the process of obtaining the word vector representation of the corresponding mixed lattice structure includes the following steps:

其中

c_i表示s_c中的第i个字，n表示s的字数长度，e^B表示BERT预训练字向量的查找表；S11, given a sentence s _c ={c ₁ ,c ₂ ,...,c _n }, by loading the pre-trained BERT weights, the word feature vector representation of the sentence s _c is obtained

in

c _i represents the i-th word in s _c , n represents the word length of s, and e ^B represents the look-up table of the BERT pre-training word vector;

S12, given a Chinese dictionary L, construct a Trie dictionary tree, traverse the nodes of the tree, and obtain the vocabulary matched by each word;

S13, group all the matched words according to BMES tags, that is, for the character c _i , the word set B(ci ) consists of the matching words starting with it, and the set M ₍ ci _{) consists of the matching words of c i as} its internal words The set E(ci ) consists of matching words ending in c _i , and the set S(ci ₎ consists of single-character words of c _i ; the word set _wi of each word c _i in the sentence _{s c} is expressed as :

w _i = {e ^w (B( _ci )), e ^w (M( _ci )), e ^w (E( _ci )), e ^w (S( _ci ))};

where e ^w represents the pre-trained word vector lookup table;

一致，BERT在微调的时候，对这两层权重进行学习，使预训练的词特征向量映射到BERT的语义特征空间；处理后的词特征向量表示如下：S14, set up a two-layer learnable nonlinear fully connected layer to increase the dimension of w _i to the sum word vector

Consistently, when BERT is fine-tuning, it learns the weights of these two layers, so that the pre-trained word feature vector is mapped to the semantic feature space of BERT; the processed word feature vector is represented as follows:

where W ₁ ∈(d _c ×d _c ), W ₂ ∈(d _c ×d _w ) are the learnable weight matrices, b ₁ and b ₂ are the corresponding biases, d _c represents the dimension of the BERT word vector, and d _w represents the dimension of the pre-trained word vector;

作为特征融合模型的输入，按照字和词集的对应关系，将每个字-词对特征表示为：S15, convert the transformed word feature vector

As the input of the feature fusion model, according to the corresponding relationship between the word and the word set, the feature of each word-word pair is expressed as:

S16, the features of word-word pairs are represented as follows:

表示向量拼接符。in

Represents a vector splice.

Further, in step S2, based on the word vector of the mixed lattice structure generated in step S1, a corresponding self-attention network is constructed to capture the influence of the word vector in the vector on the word vector, so as to enhance the effect of each word vector. The process of feature representation includes the following steps:

S21, Design a Mixed-lattice self-attention network to capture the association between word features. The self-attention network uses the mixed word encoding vector V ^ME and the word position masking matrix M as the input of the enhancement network. The global word vector and word vector modeling enable the model to learn the weight of the word sense correlation between words and words. The Q, K, and V matrices are calculated as follows:

[Q, K, V]=[W _q V ^ME , W _k V ^ME , W _v V ^ME ];

是可学习的权重矩阵，且d_e＝d_c+d_w；Q、K、V矩阵分别为查询项矩阵、查询项对应的键项矩阵和待加权平均的值项矩阵；d_e表示mixed-lattice向量的维度、d_c表示字向量的维度、d_w词向量的维度；in

is a learnable weight matrix, and d _e =d _c +d _w ; Q, K, V matrices are respectively the query item matrix, the key item matrix corresponding to the query item, and the value item matrix to be weighted average; d _e represents mixed- The dimension of lattice vector, d _c represents the dimension of word vector, and the dimension of d _w word vector;

S22, use the dot product operation as the calculation formula of the similarity score:

F _Att =Softmax(S _Att +εM)V;

是自注意力网络的输出；其中S_Att表示归一化后的注意力得分、K^T表示矩阵K的转置；where M is a static word position masking matrix, ε is an infinitesimal value matrix,

is the output of the self-attention network; where S _Att represents the normalized attention score, and K ^T represents the transpose of the matrix K;

S23, the word feature information is added to the BERT pre-training word vector as a residual, and the obtained word feature vector for lexical enhancement is:

C'=C+g(F _Att );

表示BERT的预训练字向量特征，函数g(*)用于移除self-attention网络中的词向量通道来保证C和F_Att向量维度的一致性，得到词汇增强后的字嵌入向量C′。in

Represents the pre-training word vector feature of BERT. The function g(*) is used to remove the word vector channel in the self-attention network to ensure the consistency of the C and F _Att vector dimensions, and obtain the word embedding vector C' after vocabulary enhancement.

Further, in step S3, the process of constructing and completing the entity recognition model based on the hybrid lattice self-attention network includes the following steps:

S31, given a sentence sequence s _c ={c ₁ ,c ₂ ,...,c _n } of length n, the word vector after lexical enhancement is expressed as C'={c' ₁ ,c' ₂ ,..., c' _n }, the word vector C' is fine-tuned in the BERT model, and the BERT word embedding vector after vocabulary enhancement is expressed as:

E′ _i =C′ _i +E _s (i)+E _p (i);

where E _s and E _p represent the separation vector and position vector lookup table respectively; i represents the ith character in the character sequence s _c of length n;

S32, input the obtained E' into BERT, and the calculation formula of each transformer block is as follows:

D=LN(H _k-1 +MHA(H _k-1 );

H _k =LN(FFN(D)+D);

where H _k represents the hidden state output of the kth layer, H ₀ =E′ represents the word vector of the bottom layer; LN is the layer normalization function; MHA is the multi-head self-attention module; FFN represents the two-layer feedforward neural network; D Represents the normalized output vector of the multi-head attention module;

将

输入到一个双向的LSTM网络中，分别从句子的左到右和右到左捕捉语义信息；前向的LSTM网络的隐状态输出表示为

后向的LSTM网络的输出为

BI-LSTM网络的输出是sequence-labeling层的输出，表示为：S33, obtain the hidden state output vector of the last layer of transformer

Will

The input is fed into a bidirectional LSTM network that captures semantic information from left-to-right and right-to-left sentences, respectively; the hidden state output of the forward LSTM network is expressed as

The output of the backward LSTM network is

The output of the BI-LSTM network is the output of the sequence-labeling layer, which is expressed as:

where hi represents the cascaded hidden state output of the _i -th Bi- _LSTMs neuron, which is used to represent the character-level contextual semantic representation of ci;

S34, use the standard CRF layer to predict the NER label, given the hidden state output vector H={h ₁ , h ₂ ,..., h _n } of the last layer of the network, if y={y ₁ , y ₂ ,..., y _n } represents the label sequence. For a sentence s={s ₁ ,s ₂ ,...,s _n }, the probability of the corresponding label sequence is defined as follows:

表示对应于y_i的网络中可学习的权重参数；

表示对应于y_i-1和y_i之间的偏置量；同样地，

分别表示在任意可能的标签y′下的模型权重参数和偏置量；where y' represents any tag sequence in all tag sequences;

represents the learnable weight parameters in the network corresponding to _yi ;

represents the amount corresponding to the offset between y _i-1 and y _i ; similarly,

represent the model weight parameters and biases under any possible label y', respectively;

S35, taking the negative log-likelihood loss as the loss function of the model, expressed as:

Based on the aforementioned named entity recognition method, the present invention proposes a named entity recognition device based on a hybrid lattice self-attention network. The named entity recognition device includes a hybrid lattice structure encoding module, a vocabulary enhancement module, a sequence labeling and decoding module, and a model training module;

The mixed lattice structure coding module is used to find words composed of consecutive words in the input sentence in the dictionary, merge them into a single multi-dimensional vector through alternate position mapping, and use the mixed word lattice encoding to represent the word pairs. The sentence feature vector is encoded as a matrix with a fixed dimension, and the corresponding word vector representation of the mixed lattice structure is obtained;

The vocabulary enhancement module is used to construct a corresponding self-attention network based on the generated word vector of the mixed lattice structure to capture the influence of the word vector in the vector on the word vector, so as to enhance the feature representation of each word vector;

The sequence labeling and decoding module is used to fuse word features in the Embedding layer of BERT, and learn to obtain better word vector representation by fine-tuning the learning process; realize the entity sequence labeling task and decoding process in entity recognition according to the BiLSTM-CRF network, Through this network, the model of the fused word features is completed, and the entity recognition model based on the hybrid lattice self-attention network is constructed;

The model training module is used to train the entity recognition model based on the hybrid lattice self-attention network on the dataset.

The beneficial effects of the present invention are:

The named entity recognition method and device based on the hybrid lattice self-attention network of the present invention captures global vocabulary information by constructing a fusion network, generates a word vector representation with rich semantics, and improves Chinese named entities on multiple data sets. recognition accuracy. Compared with the BERT benchmark model without the vocabulary enhancement network, the invention has 4.55%, 0.54%, 1.82% and 0.91% performance improvements on the four datasets respectively, which shows that the feature representation of word vectors is improved by the vocabulary enhancement technology. It is an effective method to improve the performance of medium NER. At the same time, compared with other vocabulary enhancement methods, the feature fusion framework (MELSN) proposed in the present invention can effectively integrate richer vocabulary semantic features. With the fine-tuning mechanism of the pre-training model BERT, the word feature representation after vocabulary enhancement contains more Lexical semantics. The comparative experimental results show that the present invention can better utilize the word-word case structure to achieve vocabulary enhancement, which also shows the high efficiency of the method proposed by the present invention in the task of Chinese named entity recognition.

Description of drawings

FIG. 1 is a schematic structural diagram of a named entity recognition device based on a hybrid lattice self-attention network of the present invention.

Detailed ways

The present invention will now be described in further detail with reference to the accompanying drawings.

It should be noted that the terms such as "up", "down", "left", "right", "front", "rear", etc. quoted in the invention are only for the convenience of description and clarity, and are not used for Limiting the applicable scope of the present invention, the change or adjustment of the relative relationship shall be regarded as the applicable scope of the present invention without substantially changing the technical content.

The present invention refers to a named entity recognition method based on a hybrid lattice self-attention network, and the named entity recognition method comprises the following steps:

S1: Find words composed of consecutive words in the input sentence in the dictionary, merge them into a single multi-dimensional vector through positional alternate mapping, and encode the sentence feature vector represented by the word pair into a single dimension by means of mixed word lattice encoding. A fixed matrix to obtain the word vector representation of the corresponding mixed lattice structure.

S2, based on the word vector of the mixed lattice structure generated in step S1, construct a corresponding self-attention network to capture the influence of the word vector in the vector on the word vector, so as to enhance the feature representation of each word vector.

S3, fuse word features in the Embedding layer of BERT, and learn to obtain better word vector representation by fine-tuning the learning process; realize the entity sequence labeling task and decoding process in entity recognition based on the BiLSTM-CRF network, and complete the fusion process through the network. Modeling of word features, and constructing an entity recognition model based on hybrid lattice self-attention network.

S4, train the entity recognition model based on the hybrid lattice self-attention network on the dataset.

Based on the aforementioned method, the present embodiment also mentions a named entity recognition device based on a hybrid lattice self-attention network. FIG. 1 is a schematic structural diagram of a named entity recognition device based on a hybrid lattice self-attention network of the present invention. The entire architecture is divided into three parts: Mixed-lattice Encoding Module, Lexicon Enhancement Module, and Sequence-labeling and Decoding. In the first module, the encoding of the word-word pair vector is completed, that is, all words in the dictionary tree are searched, and the pre-trained words and word vectors are loaded, and then the word-word pair vector is encoded into a mixed lattice embedding vector (Mixed-lattice embedding), and the Words mask generated at this stage is sent to the next module; the second module implements the vocabulary enhancement process through the proposed self-attention model, and the enhanced word vector representation is passed to the BERT model Fine-tuning is performed in the last module; the last module is modeled according to the word vector after the enhancement and fine-tuning, and the label prediction and decoding process of each word vector is completed.

The invention proposes a named entity recognition method based on vocabulary enhancement. The model is improved based on the BERT-BiLSTM-CRF network. The method adds a vocabulary enhancement module based on Attention network to the Embedding layer of BERT; After encoding, a cross-arranged word vector encoding method is designed. After the vocabulary enhancement is completed through the Attention network, the fusion features are normalized and then added to the original BERT word vector in the form of residual. The specific implementation process of the present invention is as follows:

Step 1: Word vector construction with mixed lattice structure

The function of the mixed word lattice (Mixed-Lattice) encoding is to encode the sentence feature vector represented by the word pair into a matrix with a fixed dimension. Words consisting of consecutive words in the input sentence are first looked up in the dictionary, and then merged into a single multidimensional vector through a positional alternation map.

其中

c_i表示s_c中的第i个字，n表示s的字数长度，e^B表示BERT预训练字向量的查找表。给定一个中文词典L，首先构造Trie字典树，遍历该树的节点，可以得到每个字所匹配到的词汇。将所有匹配到的词汇按照“BMES”标记分组，即对于字符c_i，词集B(c_i)由以它开头的匹配词组成。类似地，集合M(c_i)由c_i为其内部字的匹配词组成，集合E(c_i)由以c_i结尾的匹配词组成，集合S(c_i)由c_i的单字符词组成，所以句子s_c中每个字c_i的词集w_i可以表示为：Specifically, given a sentence s _c ={c ₁ ,c ₂ ,...,c _n }, by loading the pre-trained BERT weights, the word feature vector representation of the sentence s _c can be directly obtained

in

c _i represents the ith word in s _c , n represents the word length of s, and e ^B represents the lookup table of BERT pre-training word vectors. Given a Chinese dictionary L, first construct a Trie dictionary tree, traverse the nodes of the tree, and get the vocabulary matched by each word. All matched words are grouped by "BMES" token, ie for character c _i , word set B(ci ₎ consists of matched words starting with it. Similarly, set M( _ci ) consists of matching words with _ci as its internal word, set E( _ci ) consists of matching words ending in _ci , and set S( _ci ) _consists of single-character words of ci composition, so the word set wi for each word _{c i} in sentence _sc can be expressed as:

w _i = {e ^w (B( _ci )), e ^w (M( _ci )), e ^w (E( _ci )), e ^w (S( _ci ))};

一致，BERT在微调的时候，对这两层权重进行学习，能够使预训练的词特征向量映射到BERT的语义特征空间。处理后的词特征向量表示如下：where e ^w represents the pre-trained word vector lookup table. In order to ensure the consistency of the vector dimensions, two learnable non-linear fully connected layers are set up to increase the dimension of _wi to the sum word vector

Consistently, when BERT is fine-tuning, it learns the weights of these two layers, so that the pre-trained word feature vector can be mapped to the semantic feature space of BERT. The processed word feature vector is represented as follows:

作为特征融合模型的输入。按照字和词集的对应关系，可以将每个字-词对特征表示为：where W ₁ ∈(d _c ×d _c ), W ₂ ∈(d _c ×d _w ) are learnable weight matrices, and b ₁ and b ₂ are the corresponding biases. d _c represents the dimension of the BERT word vector, and d _w represents the dimension of the pre-trained word vector. In our feature fusion method, the transformed word feature vector

as the input to the feature fusion model. According to the correspondence between words and word sets, each word-word pair feature can be expressed as:

At present, many NER methods based on vocabulary fusion directly use the word-word pair feature I _i as the input of the vocabulary enhancement network. This method can only integrate local vocabulary information. In the experiment, we found that in the sentence _sc , the matching of other words Words also contain rich lexical semantics. In order to enable the model to capture global word-level features, this embodiment proposes a new word-word pair encoding method, namely Mixed-Lattice Embedding. The features of word-word pairs are represented as follows:

表示向量拼接符，基于该编码方法，字词向量交叉编码成一个固定维度的特征向量，在该方法中，词向量排列越近，与字的关联性越高，这也与实际相符合。在下一阶段，本实施例构造了基于Attention机制的融合网络，来对字-词对特征表示V^ME进行建模。in

Represents a vector splicer. Based on this encoding method, word vectors are cross-encoded into a fixed-dimensional feature vector. In this method, the closer the word vectors are arranged, the higher the correlation with the word, which is also in line with reality. In the next stage, this embodiment constructs a fusion network based on the Attention mechanism to model the word-word pair feature representation ^VME .

Step 2: Calculation process of word feature fusion network

The role of the vocabulary enhancement network is to model the word-word feature representation vector V ^ME , and the word features are used to enhance the feature representation of the word vectors in the sequence. Based on the design idea of attention mechanism proposed by Vaswani et al., a Mixed-latticeself-attention network is designed to capture the association between word features. In the previous stage, we have obtained the mixed lattice vector representation V ^ME , which encodes word pairs. This Lattice structure tightly organizes words and word vectors into a multi-dimensional vector. Words and word features are cross-distributed in the embedding. The self-attention network uses V ^ME and word position masking matrix M as the input of the enhanced network. Through the modeling of the self-attention network, the model can learn the meaning of words and words. Relevance weights. Given the mixed word encoding vector V ^ME and M matrix of a sentence, the Q, K, and V matrices are calculated as follows:

[Q, K, V]=[W _q V ^ME , W _k V ^ME , W _v V ^ME ];

是可学习的权重矩阵，且d_e＝d_c+d_w，然后将点积运算作为相似性分数的计算公式：in

is a learnable weight matrix, and d _e =d _c +d _w , and then the dot product operation is used as the calculation formula of the similarity score:

F _Att =Softmax(S _Att +εM)V;

是self-attention网络的输出。将M点乘一个无穷小的矩阵来屏蔽所以词位置上得到的注意力得分。经过Softmax函数的计算，词位置的概率全为零，而字符所在的位置包含了每个词的权重分数。在self-attention网络中利用词位置mask的方法完成字词特征的融合。where M is a static word position masking matrix, ε is an infinitesimal value matrix,

is the output of the self-attention network. Multiply M points by an infinitesimal matrix to mask the attention scores obtained at all word positions. After the calculation of the Softmax function, the probability of the word position is all zero, and the position of the character contains the weight score of each word. In the self-attention network, the word position mask method is used to complete the fusion of word features.

After the above process, the integration of word features can be realized. The word feature information is added to the BERT pre-training word vector as a residual, and the final word feature vector for vocabulary enhancement is:

C'=C+g(F _Att );

表示BERT的预训练字向量特征，使用函数g(*)来移除self-attention网络中的词向量通道来保证C和F_Att向量维度的一致性，从而得到了词汇增强后的字嵌入向量C′。in

Represents the pre-trained word vector feature of BERT, and uses the function g(*) to remove the word vector channel in the self-attention network to ensure the consistency of the C and F _Att vector dimensions, thus obtaining the word embedding vector C after vocabulary enhancement. '.

Based on the above process, the word-word pair vector is re-encoded, and the global word vector and word vector are modeled by a special self-attention network, and the fused word feature vector is obtained, which realizes the global word feature information. fusion.

Step 3: Construct Named Entity Recognition Model

1) The calculation process of the BERT structure

This embodiment is based on the improvement of the BERT-BiLSTM network, enhances the features of the embedding layer of BERT, and proposes a mechanism for the embedding layer to incorporate word features. In step 2, the word vector representation C′ after lexical enhancement is obtained, given a sentence sequence of length n s _c ={c ₁ ,c ₂ ,...,c _n }, and then fine-tune in the BERT model The word vector C', the BERT word embedding vector after vocabulary enhancement can be expressed as:

E′ _i =C′ _i +E _s (i)+E _p (i)

where E _s and E _p represent the separation vector and position vector lookup table, respectively, and then the obtained E′ is input into BERT. The calculation formula of each transformer block is as follows:

D=LN(H _k-1 +MHA(H _k-1 )

H _k =LN(FFN(D)+D)

将该向量输出到后续的序列标注和解码任务。Where H _k represents the hidden state output of the kth layer (H ₀ =E' represents the word vector of the bottom layer); LN is the layer normalization function; MHA is a multi-head self-attention module; FFN represents a two-layer feedforward neural network. Finally, the hidden state output vector of the last layer of transformer is obtained

This vector is output to subsequent sequence labeling and decoding tasks.

2) The calculation process of the LSTM network

将

输入到一个双向的LSTM网络中，这种网络能够分别从句子的左到右和右到左捕捉语义信息。After fine-tuning, the mixed-case embedding vector already contains the semantic information at the word level. Since in the process of word fusion, more attention is paid to the word meaning information association between words, in order to capture the global semantic information between words in a sentence, more To improve the performance of NER, the common practice of most NER models is adopted, that is, BiLSTM is used as the sequence labeling layer of the model in this example. Given a sentence s _c ={c ₁ ,c ₂ ,...,c _n }, in the previous steps, the character feature representation after lexical enhancement has been obtained C'={c' ₁ ,c' ₂ ,...,c' _n }, and the hidden state output matrix after fine-tuning by BERT

Will

The input is fed into a bidirectional LSTM network that captures semantic information from left-to-right and right-to-left sentences, respectively.

同样的后向的LSTM网络的输出为

因此BI-LSTM网络的输出的就是sequence-labeling层的输出，它可以表示为：The hidden state output of the forward LSTM network can be expressed as

The output of the same backward LSTM network is

Therefore, the output of the BI-LSTM network is the output of the sequence-labeling layer, which can be expressed as:

where hi represents the cascaded hidden state output of the _ith Bi- _LSTMs neuron, which we use to represent the character-level contextual semantic representation of ci.

3) Decoding process of CRF network

After the model goes through the computation of the sequence labeling layer, a standard CRF layer is used to predict NER labels. Given the hidden state output vector H={h ₁ ,h ₂ ,…,h _n } of the last layer of the network, if y={y ₁ ,y ₂ ,…,y _n } represents the label sequence, for a sentence s= {s ₁ ,s ₂ ,…,s _n }, the probability of the corresponding label sequence is defined as follows:

表示对应于y_i的网络中可学习的权重参数；

表示对应于y_i-1和y_i之间的偏置量。将负对数似然损失作为模型的损失函数，其可以表示为：where y' represents any tag sequence in all tag sequences;

represents the learnable weight parameters in the network corresponding to _yi ;

represents the offset between y _i-1 and y _i . Taking the negative log-likelihood loss as the loss function of the model, it can be expressed as:

Step 4: Model Learning Process

The method proposed in this embodiment is trained separately on four public data, and the model converges by optimizing the above-mentioned negative log-likelihood loss. The model uses the learning rate update strategy of warmup, the BERT model parameters use a learning rate of 1e-5, and a small learning rate in the fine-tuning process can make the model converge quickly; for the parameters of the LSTM model, a learning rate of 1e-3 is used , and set a learning rate of 1e-4 for all other parameters. On the four datasets, the model converges within 20 epochs. The V100 graphics card was used for experiments on the MSRA dataset, and the other datasets were performed on the 1080Ti graphics card. In the experiment, it is found that there are certain differences in the results on different machines, so the experimental results are the average of multiple experimental results.

In this embodiment, the evaluation is performed on four public Chinese named entity recognition data sets, and the effects of other models are compared, and the data is displayed in a table to illustrate the effects of the invention.

Dataset introduction:

The Weibo dataset is a social media public dataset collected from the Sina Weibo website and contains four entity types: place names, person names, organization and politically related entity names. The Resume dataset also comes from Sina social media data, which is about financial resume data. The MSRA and OntoNotes4 datasets are derived from the public news domain and contain ground-truth labels for training data. The statistics of these datasets are shown in Table 1.

Table 1 Statistics table of datasets

Metrics:

Here, the commonly used measurement index of classification models, that is, the F1 value, is used to compare the recognition accuracy of other models and the present invention. First, some preconditions for calculating the F1 value are introduced: TP, that is, True Positives, means that the sample is divided into positive samples and the allocation is correct; TN, that is, True Negatives, means that the sample is divided into samples and that the allocation is correct; FP, that is, FalsePositives, means that The samples are classified as positive but wrongly assigned; FN, False Negatives, means that the samples are classified as negative but wrongly assigned. Precision, that is, precision, indicates the proportion of the number of positive samples that are correctly assigned to the total number of positive samples assigned, and the formula is:

Recall, the recall rate, indicates the proportion of the number of positive samples that are correctly assigned to the total number of positive samples. The formula is:

F1-Score, also known as F1 score, is a measure of classification problems and is often used as the final indicator of multi-classification problems. It is the harmonic mean of precision and recall. For a single category F1 score, it can be calculated using the following formula

Effect description:

The F1 values on the four data are shown in the table below. This embodiment not only compares the method of using BERT pre-training word vector, but also compares other classic vocabulary enhancement SOTA methods. On the basis of ensuring the use of the same pre-trained word vectors and word vectors, the F1 values of the method proposed in this example on the four public datasets Weibo, Resume, MSRA, and Ontonote4 reach 71.88%, 96.22%, and 95.72 respectively. % and 81.75% performance. Compared with the performance on the classic Chinese NER models Lattice-LSTM, LR-CNN and FLAT, the method proposed in this example achieves 8.46%, 0.69%, 5% and 1.37% improvement on the four datasets, respectively. Compared with NER models that use BERT pre-trained word vectors, such as Softlexicon-BERT, FLAT-BERT, LEBERT and other models, on the Resume, MSRA and OnNote4 datasets, the method proposed in this example achieves 0.28%, 0.12% and a slight improvement of 0.41%, but the F1 score on the Weibo dataset significantly outperforms the Softlexicon BERT model by 1.64%. This embodiment is improved based on BERT-BiLSTM. Compared with the BERT benchmark model without vocabulary enhancement network, this embodiment has 4.55%, 0.54%, 1.82% and 0.91% performance improvements on the four datasets, respectively. This shows that improving the feature representation of word vectors through lexical enhancement techniques is an effective method to improve the performance of mid-NER. At the same time, compared with other vocabulary enhancement methods, the feature fusion framework (MELSN) proposed in this embodiment can effectively integrate richer vocabulary semantic features. With the fine-tuning mechanism of the pre-training model BERT, the word feature representation after vocabulary enhancement contains more lexical semantics. Referring to Table 2, the comparative experimental results show that this embodiment can better utilize the word-word lattice structure to achieve vocabulary enhancement, which also illustrates the efficiency of the method proposed in this embodiment in the task of Chinese named entity recognition.

Table 2 Comparative experiment result table

The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above-mentioned embodiments, and all technical solutions that belong to the idea of the present invention belong to the protection scope of the present invention. It should be pointed out that for those skilled in the art, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims (5)

1. A named entity recognition method based on a mixed lattice self-attention network is characterized by comprising the following steps:

s1, searching words composed of continuous words in the input sentence in the dictionary, combining into a single multidimensional vector through position alternate mapping, and coding the sentence characteristic vector represented by the word pair into a matrix with fixed dimensionality by adopting a mixed word lattice coding mode to obtain the word vector representation of a corresponding mixed lattice structure;

s2, constructing a corresponding self-attention network based on the word vectors of the mixed lattice structure generated in the step S1 to capture the influence of the word vectors in the vectors on the word vectors so as to enhance the feature representation of each word vector;

s3, fusing word features at the Embedding layer of BERT, and learning to obtain better word vector representation through a fine tuning learning process; according to a BilSTM-CRF network, realizing an entity sequence labeling task and a decoding process in entity identification, completing modeling of character features after fusion through the network, and constructing and completing an entity identification model based on a mixed lattice self-attention network;

and S4, training the entity recognition model based on the mixed lattice self-attention network on the data set.

2. The method for recognizing named entities based on mixed-lattice self-attention network as claimed in claim 1, wherein in step S1, the process of encoding sentence feature vectors represented by word pairs into a matrix with fixed dimension by using mixed-word lattice encoding to obtain word vector representations with corresponding mixed-lattice structure comprises the following steps:

s11, a sentence S is given_c＝{c₁,c₂,…,c_nBy loading with prestageThe trained BERT weights to obtain a sentence s_cWord feature vector representation of

Wherein

c_iDenotes s_cOf (a), n represents the word length of s, e^BA lookup table representing a BERT pre-training word vector;

s12, giving a Chinese dictionary L, constructing a Trie dictionary tree, traversing the nodes of the tree, and obtaining the vocabulary matched with each word;

s13, all matched words are grouped according to BMES marks, namely for the character c_iWord set B (c)_i) Consisting of matching words starting with it, set M (c)_i) From c_iSet E (c) for matching word composition of its internal characters_i) By c_iMatching word composition at the end, set S (c)_i) From c_iThe single character word composition of (1); sentence s_cIn each word c_iWord set w of_iExpressed as:

w_i＝{e^w(B(c_i)),e^w(M(c_i)),e^w(E(c_i)),e^w(S(c_i))}；

wherein e^wA lookup table of word vectors representing pre-training;

s14, setting two learnable nonlinear full-connection layers to connect w_iIs raised to a sum word vector

When BERT is in fine adjustment, learning the two layers of weights to enable pre-trained word feature vectors to be mapped to a semantic feature space of the BERT; the processed word feature vector is represented as follows:

wherein W₁∈(d_c×d_c),W₂∈(d_c×d_w) Is a learnable weight matrix, b₁And b₂Is a corresponding offset, d_cDimension, d, representing the BERT word vector_wA dimension representing a pre-training word vector;

s15, converting the word feature vector v_i ^wAs the input of the feature fusion model, according to the corresponding relation between the words and the word sets, the feature of each word-word pair is expressed as:

s16, representing the characteristics of the word-word pairs as follows:

wherein

Representing a vector concatenator.

3. The method for named entity recognition based on mixed-lattice self-attention network as claimed in claim 2, wherein the step S2 of constructing a corresponding self-attention network based on the word vectors of the mixed-lattice structure generated in step S1 to capture the influence of the word vectors in the vectors on the word vectors, so as to enhance the feature representation of each word vector comprises the following steps:

s21, designing Mixed-lattice self-attention network to capture the association between word features, the self-attention network will mix word code vector V^MEAnd a word position shielding matrix M is used as an input of the enhancement network, global word vectors and word vectors are modeled through the self-attention network, so that the model learns word meaning correlation weights among words, and the Q, K, V matrix is calculated as follows:

[Q,K,V]＝[W_qV^ME,W_kV^ME,W_vV^ME]；

wherein

Is a learnable weight matrix, and d_e＝d_c+d_w(ii) a Q, K, V matrix is query item matrix, key item matrix corresponding to query item and value item matrix to be weighted average; d_eDimension, d, representing mixed-lattice vector_cDimension, d, representing word vector_wThe dimension of the word vector;

s22, using the dot product operation as a formula for calculating the similarity score:

F_Att＝Softmax(S_Att+εM)V；

where M is a static word-position mask matrix, ε is a matrix whose value is infinitesimal,

is an output from the attention network; wherein S_AttDenotes the normalized attention score, K^TRepresents the transpose of matrix K;

s23, adding the word feature information as a residual into the BERT pre-training word vector, and obtaining the word enhanced word feature vector as follows:

C′＝C+g(F_Att)；

wherein

Representing the pre-training word vector features of BERT, the function g (. + -) is used to remove the word vector path in self-entry network to ensure C and F_AttAnd (5) obtaining word embedded vector C' after vocabulary enhancement by the consistency of vector dimensions.

4. The method for named entity recognition based on hybride self-attention network of claim 3, wherein the step S3 of constructing the entity recognition model based on hybride self-attention network comprises the following steps:

s31, a sentence sequence S with the length of n is given_c＝{c₁,c₂,…,c_nThe vocabulary enhanced word vector is denoted as C '═ C'₁,c′₂,…,c′_nFine-tuning a word vector C' in the BERT model, the vocabulary-enhanced BERT word-embedding vector is represented as:

E′_i＝C′_i+E_s(i)+E_p(i)；

wherein E_sAnd E_pRespectively representing a separation vector and a position vector lookup table; i denotes a character sequence s of length n_cThe ith character in (a);

s32, inputting the obtained E' into BERT, wherein the calculation formula of each transform block is as follows:

D＝LN(H_k-1+MHA(H_k-1)；

H_k＝LN(FFN(D)+D)；

wherein H_kRepresents the hidden state output of the k-th layer, H₀E' represents the underlying word vector; LN is the layer normalization function; MHA is a multi-headed self-attention module; FFN represents a two-layer feedforward neural network; d represents an output vector after the multi-head attention module is normalized;

s33, obtaining the hidden state output vector of the last layer of transformer

Will be provided with

Inputting the semantic information into a bidirectional LSTM network, and capturing semantic information from left to right and from right to left of a sentence respectively; the hidden state output of the forward LSTM network is represented as

The backward LSTM network outputs

The output of the BI-LSTM network is the output of the sequence-labeling layer, expressed as:

wherein h is_iRepresents the cascade hidden state output of the ith Bi-LSTMs neuron and is used for representing c_iThe character-level context semantic representation of;

s34, predicting the NER label by using standard CRF layer, and giving hidden state output vector H ═ H of last layer of network₁,h₂,…,h_nIf y is equal to { y }₁,y₂,…,y_nDenotes a sequence of labels, s ═ s for a sentence₁,s₂,…,s_nThe probability of its corresponding tag sequence is defined as follows:

wherein y' represents any one of all tag sequences;

the representation corresponds to y_iA learnable weight parameter in the network of (a);

representation corresponds to y_i-1And y_iAn offset between;

respectively representing model weight parameters and offset under any possible label y';

s35, the negative log-likelihood loss as a loss function of the model is expressed as:

5. a named entity recognition device based on a mixed lattice self-attention network is characterized by comprising a mixed lattice structure coding module, a vocabulary enhancing module, a sequence marking and decoding module and a model training module;

the mixed lattice structure coding module is used for searching words consisting of continuous words in input sentences in a dictionary, alternately mapping and merging the words into a single multidimensional vector through positions, and coding sentence characteristic vectors represented by word pairs into a matrix with fixed dimensionality by adopting a mixed word lattice coding mode to obtain word vector representation of a corresponding mixed lattice structure;

the vocabulary enhancement module is used for constructing a corresponding self-attention network based on the generated word vectors of the mixed lattice structure so as to capture the influence of the word vectors in the vectors on the word vectors, thereby enhancing the feature representation of each word vector;

the sequence labeling and decoding module is used for fusing word features at an Embedding layer of BERT and learning to obtain better word vector representation through a fine tuning learning process; according to a BilSTM-CRF network, realizing an entity sequence labeling task and a decoding process in entity identification, completing modeling of character features after fusion through the network, and constructing and completing an entity identification model based on a mixed lattice self-attention network;

the model training module is used for training the entity recognition model based on the hybrid grid self-attention network on the data set.

Images