CN114818717A - Chinese named entity recognition method and system fusing vocabulary and syntax information - Google Patents

Abstract

The invention discloses a Chinese named entity recognition method and system integrating lexical and syntactic information. In the input representation of each word; step 2, according to the input representation of the word, use bidirectional LSTM to extract context information; step 3, use NLP tools to obtain part-of-speech tags and syntactic components from the original input text, and use the key value memory network to construct Syntax vector, and then weighted fusion of context vector and syntax vector through the gating mechanism to obtain feature vector; Step 4, input the feature vector into the CRF of the label prediction layer to realize Chinese named entity recognition. The invention can solve the problem of insufficient entity boundary information in Chinese named entities and fuse the syntactic information of the input text.

Description

Chinese named entity recognition method and system integrating lexical and syntactic information

technical field

The invention relates to the field of information extraction of natural language processing, in particular to a Chinese named entity recognition method and system integrating lexical and syntactic information.

Background technique

Named entity recognition (NER) aims to identify entities in text and classify them into different categories such as person names, place names, institution names, etc. NER is an important task in NLP and has been widely used in relation extraction, question answering, machine translation, knowledge base construction and other fields. Therefore, the research and breakthrough of NER is of great significance.

The named entity recognition in Chinese and English is different. Every word in English can express complete semantic information; while in Chinese, in most cases, only one word or phrase can express the complete meaning, and there is no obvious meaning in Chinese. The characteristics of lexical boundary characters and capitalization of the first letter make it difficult to identify the entity boundary. But word-entity boundaries are usually the same as entity boundaries, so word-boundary information plays an important role in Chinese named entity recognition (CNER).

Facing the difficulty of word boundary recognition, it can be solved by introducing external features. Among these features, lexical information and syntactic information have important meanings, which can help the CNER model to find the corresponding entities. When external features are used in existing CNER models, they are rarely differentiated and processed, and noise in the features may affect the performance of the model. Therefore, it is still a difficult problem to find a suitable method to integrate the external feature information into the CNER model. Most of the time, it is expected that the CNER model can contain multiple additional features. Therefore, it is necessary to design an effective mechanism to weight the combination of these features to limit the noise information.

At the same time, the existing SoftLexion word set matching method relies on static word frequency statistics in the data set, and uses word frequency to measure the effect of different words on the Chinese named entity recognition task. Considering the different scales of different data sets, there is a problem of too low word frequency in small-scale data sets, and word frequency sometimes cannot reflect the importance of words well. Therefore, a more reasonable method can be found to reasonably measure the weight of the words in the vocabulary.

SUMMARY OF THE INVENTION

The main purpose of the present invention is to overcome the above-mentioned defects in the prior art, and to propose a method and system for Chinese named entity recognition that fuses lexical and syntactic information, and fuses lexical and syntactic information (abbreviated as “fusion”). It is the LSF-CNER model), which is to fuse the external vocabulary information and the syntactic information of the input text using the gating unit, and introduce an attention mechanism into the model to build a Chinese named entity recognition model, which is expected to improve the accuracy of Chinese named entity recognition. Rate. The problems mainly solved by the present invention are embodied in the following two aspects: on the one hand, each word of the input text sequence is subjected to an improved word set matching algorithm, and the static word set vector and dynamic word set vector after the matching is completed and the initial The word vector is spliced, so that the external vocabulary information is integrated into the word vector, which can solve the problem of insufficient boundary features of Chinese text words. On the other hand, NLP tools are used to extract the syntactic information of the input text, and the context vector extracted by the gating mechanism and the bidirectional LSTM is integrated, which enriches the representation of the feature vector and integrates deeper syntactic information.

The present invention adopts following technical scheme:

On the one hand, a Chinese named entity recognition method integrating lexical and syntactic information, including:

Step 1. Map the original input text into word vectors, use the improved word set matching algorithm to introduce external vocabulary information, and integrate it into the input representation of each word;

Step 2. According to the input representation of the word, use bidirectional LSTM to extract context information;

Step 3. Use NLP tools to obtain part-of-speech tags and syntactic components from the original input text, and use a key-value memory network to construct a syntactic vector, and then perform a weighted fusion of the context vector and the syntactic vector through a gating mechanism to obtain a feature vector;

Step 4. Input the feature vector into the CRF of the label prediction layer to realize Chinese named entity recognition.

Preferably, the step 1 specifically includes:

Step 1.1, treat the input text as a sentence, which is represented by a sequence as x=(x ₁ , x ₂ ,..,x _n ); where x _i represents the i-th word in the sentence x of length n; In order to make better use of vocabulary information, the results of each word matching dictionary are divided into the following four word sets "BIES":

(1) Word set B( _xi ) contains all words starting with x _i on x;

(2) The word set I(x _i ) contains all words with _xi being the middle on x;

(3) The word set E(x _i ) contains all words ending in x _i in x;

(4) The word set S(x _i ) contains all words whose _xi is a single character;

Step 1.2, after obtaining the "BIES" word set corresponding to each word, compress each word set into a fixed-dimensional vector; the improved word set matching algorithm includes static word set algorithm and dynamic word set algorithm, static word set In order to ensure the calculation efficiency, the algorithm uses the frequency of word occurrence to represent the corresponding weight. The calculation method of the static word set vector for a single word set is as follows:

表示词

在语料库中出现的次数；

表示词集T出现词语的总次数；

表示将词语

映射为词向量；T表示“BIES”四个词集中的一个；

表示字x_i对应词集T的向量表示；in,

express word

the number of occurrences in the corpus;

Represents the total number of occurrences of words in the word set T;

to express words

Mapped to word vector; T represents one of the four word sets of "BIES";

Represents the vector representation of the word set T corresponding to the word _xi ;

In order to better retain the information, the four static word sets are represented as a whole and integrated into a fixed-dimensional vector by splicing:

Among them, τ _i represents the static word set vector corresponding to the word _xi ;

The dynamic word set algorithm uses the attention mechanism to measure the information between characters and matching words, calculates the attention weight of different matching words, enhances useful words and suppresses insignificant words, as follows:

表示将词语

映射为词向量；q是与

维度相同的训练向量；

为通过注意力机制得到的词语注意力分数；

为归一化后的词语注意力权重；

表示单个词集的动态词集向量；m表示字x_i对应词集T匹配到的词语个数；in,

to express words

maps to word vectors; q is the same as

training vectors with the same dimensions;

is the word attention score obtained by the attention mechanism;

is the normalized word attention weight;

Represents the dynamic word set vector of a single word set; m represents the number of words matched by the word _xi corresponding to the word set T;

The dynamic word set vector is obtained by the weighted summation of the attention weight, and the four dynamic word sets are represented as a whole and compressed into a fixed-dimensional vector:

Among them, Aτ _i is represented as the dynamic word set vector corresponding to the word x _i ;

和静态词集向量

动态加权组合；使用评估函数

来衡量静态词集向量

和动态词集向量

对实体识别任务的作用:Step 1.3, in order to fully consider the importance of each word in the two word sets, pair the dynamic word set vector

and static word set vector

Dynamically weighted combinations; use evaluation functions

to measure the static word set vector

and dynamic word set vector

Effects on entity recognition tasks:

是可训练矩阵；

是偏置项；in,

is a trainable matrix;

is the bias term;

The word vector, the static word set vector τ _i and the dynamic word set vector Aτ _i are combined together as the final input representation containing the external vocabulary information:

表示字x_i的最终向量表示；l是向量维度与

匹配的1向量；e^x表示将字x_i转为对应的字向量；*表示点乘计算；

表示向量拼接；in,

represents the final vector representation of word x _i ; l is the vector dimension and

Matching 1 vector; e ^x means to convert word _xi to corresponding word vector; * means dot product calculation;

Represents vector concatenation;

Preferably, the step 2 specifically includes:

表示i时刻前向LSTM的隐藏层状态，使用

表示i时刻反向LSTM的隐藏层状态；通过拼接相对应的前向和反向LSTM状态，获得最终的上下文向量

The sequence encoding layer uses a bidirectional LSTM to obtain the context vector of each word, and the bidirectional LSTM is a combination of forward LSTM and reverse LSTM; use

Represents the hidden layer state of the forward LSTM at time i, using

Represents the hidden layer state of the reverse LSTM at time i; by splicing the corresponding forward and reverse LSTM states, the final context vector is obtained

Preferably, the step 3 specifically includes:

Step 3.1, use the Stanford CoreNLP tool to segment the original text, and use the BerkelyNeural Parse tool to extract two kinds of syntactic information: the part-of-speech tag and the syntactic component information of the input text; the part-of-speech tag represents the label information of a single word, and the syntactic component represents the structure of the text span group information;

和

Step 3.2, for each _xi in the input sequence, map its contextual features and syntactic information to the keys and values in the key-value memory network KVMN, respectively expressed as

and

和

上，对于每个x_i关联的上下文特征K_i和句法信息V_i，分配给句法信息的权重γ_i,j为：Step 3.3, use two embedding matrices to map k _i,j and v _i,j to

and

Above, for each _xi associated context feature K _i and syntactic information V _i , the weight γ _i,j assigned to the syntactic information is:

是h_i的转置形式；where _{hi is the hidden vector of xi obtained} from the sequence encoding layer;

is the transposed form of _hi ;

Apply the weight γ _i,j to the corresponding syntactic information v _i,j as follows:

Among them, α _i is the weighted syntactic information of the KVMN model corresponding to _xi ; therefore, KVMN can ensure that the syntactic information is weighted according to its corresponding context features, thereby distinguishing and using important information;

Step 3.4, in order to make better use of the syntactic information encoded by KVMN, dynamically weight the syntactic vector α _i and the context vector _hi , and use the evaluation function λ _i to measure the contribution of the context vector _{hi and the syntactic vector α i to} the sentence:

λ _i =σ(W _λ1 .h _i +W _λ2 .α _i +b _λ )

Among them, W _λ1 and W _λ2 are trainable matrices; b _λ is the bias term; σ represents the sigmoid activation function;

Then combine the syntax vector α _i and the context vector _hi together:

Among them, l is a 1 vector whose dimension matches h _i , and the final O _i is a feature vector that combines contextual information and syntactic information.

Preferably, the step 4 specifically includes:

For the input sequence x=(x ₁ ,x ₂ ,..,x _n ), given the predicted sequence y=(y ₁ ,y ₂ ,y ₃ ,...,y _n ), calculate the score for the predicted sequence ,as follows:

表示从标签y_i到标签y_i+1的转移分数；

表示句子中第i个字的第y_i'个标签的概率得分；y₀和y_n+1分别表示开始和结束标签；n表示输入序列的长度；where M is the transition matrix,

represents the transition score from label y _i to label y _i+1 ;

represents the probability score of the _yi 'th label of the ith word in the sentence; y ₀ and y _n+1 represent the start and end labels, respectively; n represents the length of the input sequence;

Use the softmax function to calculate the probability p(y|x) generated by the predicted sequence y as follows:

是整个序列标注解空间的一个实例，

表示该实例的得分；Among them, Y _x is the solution space, which represents the set of all possible prediction sequences of y;

is an instance of the entire sequence annotation space,

Indicates the score of the instance;

During the training process, the likelihood probability log(p(x,y)) of the correctly predicted sequence is maximized, as follows:

After continuous iterative training and backpropagation, the obtained y ^* is the result of using the CRF sequence annotation, that is, the output of the final model:

The training process continuously maximizes the objective function

On the other hand, a Chinese named entity recognition system that fuses lexical and syntactic information, including:

The input representation acquisition module is used to map the original input text into word vectors, and use the improved word set matching algorithm to introduce external vocabulary information and integrate it into the input representation of each word;

The context vector acquisition module is used to extract context information using bidirectional LSTM according to the input representation of the word;

The context information and syntactic information fusion module is used to obtain part-of-speech tags and syntactic components from the original input text using NLP tools, and use the key-value memory network to construct syntactic vectors, and then perform weighted fusion of the context vectors and syntactic vectors through the gating mechanism. get the feature vector;

The sequence labeling module is used to input the feature vector into the CRF of the label prediction layer to realize Chinese named entity recognition.

As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following beneficial effects:

The invention introduces external vocabulary information into the word vector through an improved word set matching algorithm, and integrates the context vector and the syntax vector in the syntactic information layer, so as to solve the problem of insufficient entity boundary information in Chinese named entities and fuse the syntactic information of the input text. .

Description of drawings

Fig. 1 is the flow chart of the Chinese named entity recognition method of the fusion vocabulary and syntactic information of the embodiment of the present invention;

Fig. 2 is the model diagram of the Chinese named entity recognition method of fusing vocabulary and syntactic information according to the embodiment of the present invention;

3 is a dictionary matching method corresponding to "Chinese linguistics" according to an embodiment of the present invention;

FIG. 4 is a diagram for obtaining syntax information according to an embodiment of the present invention;

FIG. 5 is a structural block diagram of a Chinese named entity recognition system integrating lexical and syntactic information according to an embodiment of the present invention.

Detailed ways

The present invention will be further described below in conjunction with specific embodiments. It should be understood that these examples are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

Referring to Fig. 1, a Chinese named entity recognition method integrating lexical and syntactic information of the present invention includes:

Step 1. Map the original input text into word vectors, use the improved word set matching algorithm to introduce external vocabulary information, and integrate it into the input representation of each word;

Step 2. According to the input representation of the word, use bidirectional LSTM to extract context information;

Step 3. Use NLP tools to obtain part-of-speech tags and syntactic components from the original input text, and use a key-value memory network to construct a syntactic vector, and then perform a weighted fusion of the context vector and the syntactic vector through a gating mechanism to obtain a feature vector;

Step 4. Input the feature vector into the CRF of the label prediction layer to realize Chinese named entity recognition.

Specifically, the model diagram of the method of the present invention is shown in FIG. 2 , and the model is divided into four parts: input representation layer, sequence coding layer, syntax information layer and label prediction layer.

The specific implementation steps of a Chinese named entity recognition method integrating vocabulary and syntactic information of the present invention are as follows:

(1) Acquisition of input representations containing external lexical information

The present invention proposes a word set matching method, which introduces an attention mechanism on the basis of SoftLexicon, combines the matched static and dynamic word set information, and then uses a gate control mechanism to match the static word set information with the dynamic word set information. word set information fusion.

1.1) Static word set algorithm

In the Chinese NER model, the sentence of the original input text is regarded as a sequence x=(x ₁ ,x ₂ ,..,x _n ). where x _i represents the ith word in a sentence x of length n. In order to make better use of vocabulary information, the results of each word matching dictionary are divided into four word sets:

a. Word set B( _xi ) contains all words starting with x _i on x;

b. The word set I( _xi ) contains all the words with _xi being the middle on x;

c. The word set E(x _i ) contains all the words ending in x _i in x;

d. The word set S( _xi ) contains all words with _xi being a single character.

These four word sets are collectively referred to as "BIES" and are represented as:

Among them, L represents the external dictionary; w _{j, k} represents a word that exists in the external dictionary and starts with w _j and ends with w _k in the sequence. When the input of the input layer is "Chinese linguistics", the corresponding dictionary matching method is shown in Figure 3.

其中，

表示字x_i对应的长度为m的词集T中的第j个词语。根据该思想，更新其权重得到静态词集向量:After obtaining the "BIES" word set corresponding to each word, each word set is compressed into a fixed-dimensional vector. The processing speed of the static word set algorithm is better than that of the dynamic word set algorithm using the attention mechanism. In order to ensure the calculation efficiency, the frequency of word occurrence is used to represent the corresponding weight. Since the frequency of a word is a static value, the frequency of the word can reflect the importance of the word, which can speed up the calculation of the word weight. For word x _i , the corresponding word set T is

in,

Indicates the jth word in the word set T of length m corresponding to word x _i . According to this idea, update its weight to get the static word set vector:

表示词

在语料库中出现的次数；

表示词集T出现词语的总次数；

表示将词语

映射为词向量，T表示“BIES”四个词集中的一个；上标ws表示词集(word set)，

表示字x_i对应词集T的向量表示。为了更好的保留信息，将这四个静态词集表示为一个整体并压缩为一个固定维度的向量。in,

express word

the number of occurrences in the corpus;

Represents the total number of occurrences of words in the word set T;

to express words

Mapped to a word vector, T represents one of the four word sets of "BIES"; the superscript ws represents a word set,

A vector representation representing the word set T corresponding to word _xi . In order to preserve the information better, the four static word sets are represented as a whole and compressed into a fixed-dimensional vector.

1.2) Dynamic word set algorithm

其中，

表示字x_i对应的长度为m的词集T中的第j个词语。接下来将

作为输入，采用一个三层的神经网络，得到维度为1×m，取值范围为(0,1)的词语重要度分数：The attention mechanism can assign different weights to different words, assigning more weight to a few key words and less weight to irrelevant words. The dynamic vocabulary algorithm uses an attention mechanism to measure the information between characters and matching words, calculate the importance of different matching words, enhance useful words and suppress insignificant words. For word x _i , the corresponding word set T is

in,

Indicates the jth word in the word set T of length m corresponding to word x _i . Next will be

As input, a three-layer neural network is used to obtain a word importance score with a dimension of 1×m and a value range of (0,1):

维度相同的训练向量，

为通过注意力机制得到的词语重要度分数，

为归一化后的词语重要度分数。接下来，再根据词语重要度分数得到动态词集向量，将四个动态词集表示为一个整体并压缩为一个固定维度的向量：where q is the same as

training vectors with the same dimensions,

is the word importance score obtained by the attention mechanism,

is the normalized word importance score. Next, the dynamic word set vector is obtained according to the word importance score, and the four dynamic word sets are represented as a whole and compressed into a fixed-dimensional vector:

表示字x_i对应词集T的动态词集向量，Aτ_i表示为字x_i对应的动态词集向量，m为字x_i对应词集T匹配到的词语个数。in,

Represents the dynamic word set vector of the word set T corresponding to the word _xi , Aτ _i is the dynamic word set vector corresponding to the word _xi , and m is the number of words matched by the word set T corresponding to the word _xi .

1.3) Gating mechanism

和静态词集向量

动态加权组合。使用评估函数

来衡量静态词集向量

和动态词集向量α_i对实体识别任务的作用:In order to fully consider the importance of each word in each word set, the dynamic word set vector

and static word set vector

Dynamically weighted combination. Use an evaluation function

to measure the static word set vector

and the effect of dynamic word set vector α _i on entity recognition task:

是可训练矩阵，

是偏置项。然后，将静态词集向量τ_i和动态词集向量Aτ_i组合在一起:in,

is the trainable matrix,

is the bias term. Then, the static word set vector τ _i and the dynamic word set vector Aτ _i are combined together:

表示字的字向量，

表示字x_i的最终向量表示，l是向量维度与

匹配的1向量，e^x表示将字x_i转为对应的字向量，*是点乘计算，

是向量拼接。通过这种方法，将预先训练好的词向量嵌入到字符表示中，能够较合理的使用外部词典信息。in,

word vector representing the word,

represents the final vector representation of word x _i , where l is the vector dimension with

The matching 1 vector, e ^x means to convert the word _xi to the corresponding word vector, * is the dot product calculation,

is vector concatenation. Through this method, the pre-trained word vector is embedded into the character representation, and the external dictionary information can be used reasonably.

(2) Acquisition of context vector

Long Short-Term Memory Network (LSTM) is a variant of Recurrent Neural Network and is widely used in NLP tasks such as NER, text classification, sentiment analysis. LSTM introduces cell state and uses input gate, forget gate and output gate to maintain and control information, which effectively overcomes the gradient explosion and gradient loss caused by long-distance dependence of RNN model. The mathematical expression of the LSTM model is as follows:

i _t =σ(W _i [h _t-1 ,τ _t ]+b _i ) (15)

f _t =σ(W _f [h _t-1 ,τ _t ]+b _f ) (16)

o _t =σ(W _o [h _t-1 ,τ _t ]+b _o ) (17)

h _t =o _t *tanh(c _t ) (20)

表示当前输入状态；c_t表示当前状态的输出。Among them, σ represents sigmoid activation function, tanh represents hyperbolic tangent function; τ _t represents unit input; i _t , f _t , _{o t} represent input gate, forget gate and output gate respectively; Wi , W _f , W _o represent respectively The weights of the input gate, the forgetting gate and the output gate; b _i , b _f , and b _o represent the biases of the input gate, the forgetting gate and the output gate, respectively;

Represents the current input state; _ct represents the output of the current state.

表示i时刻前向LSTM的隐藏层状态，使用

表示i时刻反向LSTM的隐藏层状态。通过连接相对应的前向和反向LSTM状态，获得最终的上下文向量

In order to use the contextual information of words at the same time, the model in this paper uses a bidirectional LSTM to obtain the context vector of each word, which is a combination of forward LSTM and reverse LSTM. use

Represents the hidden layer state of the forward LSTM at time i, using

Represents the hidden layer state of the reverse LSTM at time i. The final context vector is obtained by concatenating the corresponding forward and reverse LSTM states

(3) Fusion of contextual information and syntactic information

The context information is output by the bidirectional LSTM of the sequence encoding layer, and the syntactic information is first extracted from the input sequence by NLP tools, then mapped by the key-value memory network, and finally integrated by the gating mechanism.

3.1) Syntax information acquisition

The present invention uses the Stanford CoreNLP tool to firstly segment the text, and then uses the Berkely NeuralParse tool to extract two kinds of syntactic information: part-of-speech tags and syntactic components. The part-of-speech tag represents the tag information of a single word, and the syntactic component represents the structural grouping information of the text span. Taking the example of "liberating water in the area of the road", Figure 4 shows the part-of-speech tags and syntactic components in the example sentence. For each x _i in the input sequence x = (x ₁ , x ₂ , .., x _n ), the following steps are followed to extract contextual features and syntactic information.

3.1.1) Part-of-speech tags: take each _xi as the central word, and use a ±1-word window to extract the context words and their part-of-speech tags on both sides of _xi . As shown in Figure 4(a), for the word "dao", the obtained context features are (liberation, avenue, road), and the combination of these words and the corresponding part-of-speech tags is used as the part-of-speech information of the NER task, that is, the corresponding syntactic information is (liberation_NN, avenue_NN, pavement_NN).

3.1.2) Syntactic components: starting from the leaf node of the syntax tree of x _i , search up the tree to find the first syntax node, then use all words under the node as context features, and combine the words and the corresponding syntax labels. as syntactic component information for NER tasks. As shown in Figure 4(b), for the word "Dao", the node is searched upward, and the syntactic node "NP" contains two words "Jiefang" and "Dao". Therefore, the context feature is (Jiefang, Dao), and the context feature is combined with the "NP" label as the syntactic component information of the NER task, that is, the corresponding syntactic information is (Jiefang_NP, Dao_NP).

3.2) KVMN build

Since there is a certain amount of noise in the syntactic information extracted by the tool, if these noises are not used properly, the performance of the model may be affected. Variants of KVMN have been shown to be effective in incorporating contextual features and provide an appropriate way to exploit contextual features and their corresponding dependency types.

和

其中，m_i表示x_i的上下文特征数，K和V表示上下文特征和句法信息。接下来，使用两个嵌入矩阵将k_i,j和v_i,j分别映射到

和

上。对于每个x_i关联的上下文特征K_i和句法信息V_i，分配给句法信息的权重为：In the process of constructing the KVMN, the parsing result corresponding to the input sequence x=(x ₁ , x ₂ , .., x _n ) is obtained first. For each _xi in the input sequence, map its contextual features and syntactic information to the keys and values in KVMN, denoted as

and

Among them, m _i represents the number of contextual features of _xi , and K and V represent contextual features and syntactic information. Next, use two embedding matrices to map k _i,j and v _i,j to

and

superior. For each _xi associated context feature K _i and syntactic information V _i , the weights assigned to the syntactic information are:

where _{hi is the hidden vector of xi obtained} from the sequence encoding layer. Apply the weight γ _i,j to the corresponding syntactic information v _i,j :

Among them, α _i is the weighted syntactic information of the KVMN model corresponding to _xi . Therefore, KVMN can ensure that grammatical information is weighted according to its corresponding contextual features, thereby distinguishing and using important information.

3.3) Gating mechanism

In order to make better use of the syntactic information encoded by KVMN, the syntactic vector α _i and the context vector h _i are dynamically weighted and combined. The evaluation function λ _i is used to measure the contribution of the context vector h _i and the syntactic vector α _i to the sentence:

λ _i =σ(W _λ1 .h _i +W _λ2 .α _i +b _λ ) (23)

Among them, W _λ1 and W _λ2 are trainable matrices, and b _λ is the bias term. Then, the syntactic vector α _i and the context vector h _i are combined together:

Among them, O _i is the output of KVMN corresponding to x _i , l is a 1 vector whose vector dimension matches hi _i , * is the matrix dot product calculation, and ⊕ is the vector splicing.

(4) Sequence annotation using CRF

Compared with HMM, CRF does not have strict requirements on the independence assumption of HMM, and can effectively use sequence and external observation information, avoiding label bias and differentiation caused by only directly assuming labels. CRF can capture more dependencies: for example, "I-ORG" tag cannot come after "B-LOC". In CNER, the input of CRF is the context feature vector O _i learned from the syntactic information layer. For the input sequence x=(x ₁ , x ₂ , .., x _n ), let P _i,j denote the probability score of the j th label of the ith word in the sentence. For the predicted sequence y=(y ₁ , y ₂ , y ₃ ,...,y _n ), calculate the score for the predicted sequence:

where M is the transition matrix, M _i,j represents the transition score from label i to label j; y ₀ and yn ₊₁ represent the start and end labels, respectively. Use the softmax function to calculate the probability that the predicted sequence y will be produced:

where Y _x represents all possible prediction sequences for y. During training, maximize the likelihood of correctly predicting the sequence:

After training, the predicted sequence gets a score of:

(5) Effect evaluation

In order to verify the effect of the method of the present invention and compare with other models, this paper uses three Chinese named entity datasets for experiments, including: Weibo dataset, Resume dataset and MSRA dataset. The experimental metrics are precision, recall and F1 value.

The Weibo dataset is information from social network Weibo, including four types of entities: person names, place names, institution names, and geopolitical entities. The MSRA dataset is a dataset from the news domain, with only a training set and a test set, which contain three entity types: person name, place name and institution. The Resume dataset is resume information from Sina Finance, including seven entity types: city, educational institution, place name, person name, institution name, proper noun, professional background and job title. The conditions of each dataset are shown in Table 1.

Table 1 Dataset Details

In order to verify the effectiveness of the method in this chapter, the present invention uses the following three models as baselines for comparison:

(1) Lattice-LSTM: A word cell structure is introduced, and for the current character, all word information ending with this character is fused;

(2) FLAT: Based on the improvement of the transformer structure, a clever positional encoding is designed to integrate the lattice structure, and the absolute positional encoding is improved to make it more suitable for NER tasks;

(3) SoftLexicon: The method of simply using vocabulary in the input presentation layer has high portability.

The evaluation criteria are as follows:

The evaluation indicators of the experiment use the precision rate (Precision), recall rate (Recall) and F1 value (F1 score). Among them, Precision is the correct proportion of all recognized entity words; Recall represents the proportion of correctly recognized entities in all entities in the dataset. Since accuracy and accuracy are inversely proportional, F1 is their harmonic mean. The evaluation function is:

The hyperparameters used by the model are shown in Table 2:

Table 2 Hyperparameter configuration

The experimental results of the three baseline models and the model of the present invention on each data set are shown in Table 3, and the best results under each data set are shown in bold.

Table 3 Experimental results on different datasets

As shown in Table 3, on the Weibo and Resume datasets, the LSF-CNER proposed in this paper outperforms other methods.

On the Weibo dataset with more noise and indeterminate format, LSF-CNER achieves better results. Compared with SoftLexicon, the F1 value is improved by 0.71%.

Compared with SoftLexicon, the F1 value of LSF-CNER is improved by 0.11% on a dataset with a relatively fixed format and less noise such as Resume.

On the MSRA dataset with formal text and more data, compared with SoftLexicon, the F1 value of LSF-CNER dropped by 0.01%, indicating that the model has been overfitted. The reason is that SoftLexicon has been able to fit the larger-scale MSRA and Resume datasets well, and adding feature information cannot significantly improve the model effect. Due to the lack of feature information on small-scale datasets, syntactic and lexical information can be used to improve model performance.

Further, in order to verify the generality of the LSF-CNER model. The lexical information of the word is combined with the BERT word vector, and is sent to the sequence encoding layer as the output of the input representation layer. Table 4 shows the experimental results in combination with BERT, in which the F1 value is given. Compared with BERT-Tagger, the F1 average result of the method proposed in this paper is improved by 4.68%; compared with BERT+BiLSTM+CRF, it is improved by 2.48%; compared with SoftLexicon+BERT, it is improved by 0.89%. Especially in the Weibo dataset, the improvement of the method in this paper is more significant.

Table 4 Experimental results (%) combined with pre-trained language models

It can be seen from the experimental results that combining with the pre-training model can better improve the effect of the input representation layer. The method in this paper outperforms traditional CNER methods in Resume dataset, Weibo dataset and MSRA dataset. This verifies that fusing lexical information and grammatical information is effective.

In order to study the contribution of the dynamic word set and syntactic information to the entity recognition task respectively, this paper conducts ablation experiments on the Resume dataset, and sets the model to five cases for ablation experiments.

(1) BERT+LSTM+CRF: initial model, excluding external lexical information and syntactic information;

(2) SoftLexicon+BERT: Contains external vocabulary information;

(3) SoftLexicon+Attention+BERT: An attention mechanism is introduced on the basis of SoftLexicon to dynamically adjust the weights of different words in the word set;

(4) Syntactic+BERT: contains syntactic information, but does not contain external lexical information;

(5) LSF+BERT: Contains syntactic information and improved word set information.

Table 5 Ablation experiment results (%)

Model P R F1 BERT+BiLSTM+CRF 95.75 95.28 95.51 SoftLexicon+BERT 96.08 96.13 96.11 SoftLexicon+Attention+BERT 96.19 96.35 96.27 Syntactic+BERT 95.86 96.08 95.97 LSF+BERT 96.73 96.45 96.59

As shown in Table 5, when the attention mechanism is introduced into the SoftLexicon+BERT model to adjust the weight of the word set vector, its average F1 value is improved by 0.16%; when syntactic information is introduced into the BERT+BiLSTM+CRF model, its average F1 value The value is improved by 0.46%; when both syntactic information and static dynamic vocabulary information are included, the average F1 value is improved by 0.48% compared with SoftLexicon+BERT. It can be seen from the above experimental results that both the static and dynamic vocabulary information and the syntactic information are helpful for the performance improvement of the model; the coexistence of the two can further improve the recognition accuracy of Chinese named entities. This shows that dynamically adjusting the weight of words in the vocabulary and introducing syntactic information can help sentences identify the importance of different words, thereby improving the accuracy of Chinese named entities.

To sum up, the present invention proposes a Chinese named entity recognition method integrating lexical and syntactic information. The new word set matching method integrates static dynamic word set information on the input representation layer, and then uses a gating mechanism to dynamically weight the output and syntactic information of the sequence encoding layer, not only considering the potential boundaries of Chinese named entities, but also considering The potential syntactic information in the sentence is taken into account, and the two kinds of information are fused in a more balanced way to improve the expression effect of the model. The experimental results on three CNER datasets show that the new method has good performance compared with the mainstream methods.

Referring to Fig. 5, the present invention also includes a Chinese named entity recognition system integrating lexical and syntactic information, including:

The input representation acquisition module 501 is used to map the original input text into a word vector, use the improved word set matching algorithm to introduce external vocabulary information, and integrate it into the input representation of each word;

a context vector acquisition module 502, used for extracting context information by using bidirectional LSTM according to the input representation of the word;

The context information and syntactic information fusion module 503 is used to obtain part-of-speech tags and syntactic components from the original input text by using NLP tools, and use the key value memory network to construct the syntactic vector, and then perform weighted fusion of the context vector and the syntactic vector through the gating mechanism , get the feature vector;

The sequence labeling module 504 is used to input the feature vector into the CRF of the label prediction layer to realize Chinese named entity recognition

The specific implementation of a Chinese named entity recognition system integrating lexical and syntactic information is the same as a Chinese named entity recognition method integrating lexical and syntactic information, which will not be repeated in this embodiment.

The above are only specific embodiments of the present invention, but the design concept of the present invention is not limited to this, and any non-substantial modification of the present invention by using this concept should be regarded as an act of infringing the protection scope of the present invention.

Claims (6)

1. A Chinese named entity recognition method fusing vocabulary and syntax information is characterized by comprising the following steps:

step 1, mapping an original input text into a word vector, introducing external vocabulary information by using an improved word set matching algorithm, and integrating the external vocabulary information into the input representation of each word;

step 2, extracting context information by using a bidirectional LSTM according to the input representation of the character;

step 3, obtaining part-of-speech tags and syntax components from an original input text by using an NLP tool, constructing a syntax vector by using a healthy value memory network, and performing weighted fusion on a context vector and the syntax vector by using a gate control mechanism to obtain a feature vector;

and 4, inputting the feature vector into a CRF (conditional random access memory) of a label prediction layer to realize Chinese named entity identification.

2. The method for recognizing named entities in chinese that fuses lexical and syntactic information according to claim 1, wherein said step 1 specifically comprises:

step 1.1, regarding the input text as a sentence, and expressing x ═ x (x) in a sequence ₁ ,x ₂ ,..,x _n ) (ii) a Wherein x is _i Represents the ith word in a sentence x of length n; to better utilize the lexical information, the results of each word matching dictionary are divided into the following four word sets of "BIES":

(1) word set B (x) _i ) Including all in x with x _i The first word;

(2) word set I (x) _i ) Including all x on x _i The words in the middle;

(3) word set E (x) _i ) Including all in x and x _i A word ending;

(4) word set S (x) _i ) Containing all x _i Is the word of a single character;

step 1.2, after a BIES word set corresponding to each word is obtained, compressing each word set into a vector with fixed dimension; the improved word set matching algorithm comprises a static word set algorithm and a dynamic word set algorithm, wherein the static word set algorithm uses the occurrence frequency of words to represent corresponding weight in order to ensure the calculation efficiency, and the static word set vector calculation method of a single word set comprises the following steps:

wherein,

expression word

Number of occurrences in the corpus;

representing the total times of occurrence of words in the word set T;

meaning the word

Mapping into a word vector; t represents one of four word sets of "BIES";

representing a word x _i Vector representation of the corresponding word set T;

for better information retention, four static word sets are represented as a whole and integrated into a vector with a fixed dimension by splicing:

wherein, tau _i Representing a word x _i A corresponding static word set vector;

the dynamic vocabulary set algorithm uses an attention mechanism to measure information between characters and matching words, calculates attention weights of different matching words, enhances useful vocabularies and suppresses vocabularies with insignificant effects as follows:

wherein,

meaning the word

Mapping into a word vector; q is and

training vectors with the same dimension;

is the word attention score obtained by the attention mechanism;

the normalized word attention weight;

a dynamic word set vector representing a single word set; m represents a word x _i The number of words matched with the corresponding word set T;

carrying out weighted summation through attention weight to obtain a dynamic word set vector, representing four dynamic word sets as a whole and compressing the four dynamic word sets into a fixed-dimension vector:

wherein, A tau _i Expressed as a word x _i A corresponding dynamic word set vector;

step 1.3, to fully consider the importance of each word in the two word sets, the pair dynamic word set vector

And static word set vectors

Dynamic weighted combination; using an evaluation function theta _i To measure static word set vector

And dynamic word set vector

Role on entity recognition task:

θ _i ＝σ(W _θ1 .τ _i +W _θ2 .Aτ _i +b _θ )

wherein, W _θ1 、W _θ2 Is a trainable matrix; b _θ Is a bias term;

combining the word vector and the static word set vector tau _i And a dynamic word set vector A tau _i Taken together, as the input representation ultimately containing the external vocabulary information:

wherein,

representing a word x _i The final vector representation of (a); l is the vector dimension and

a matched 1 vector; e.g. of the type ^x Representing the word x _i Converting into corresponding word vectors; denotes dot product calculation;

representing vector stitching.

3. The method for recognizing named entities in chinese that fuses lexical and syntactic information according to claim 2, wherein said step 2 specifically comprises:

the sequence coding layer adopts a bidirectional LSTM to obtain a context vector of each word, wherein the bidirectional LSTM is a combination of a forward LSTM and a backward LSTM; use of

Representing hidden layer states of the i-time forward LSTM, using

A hidden layer state representing the inverted LSTM at time i; by concatenating the corresponding forward and backward LSTM states, a final context vector is obtained

4. The method for recognizing named entities in chinese with fused lexical and syntactic information according to claim 3, wherein said step 3 specifically comprises:

step 3.1, segmenting words of an original text by using a Stanford CoreNLP tool, and extracting two syntactic information, namely part-of-speech labels and syntactic component information of an input text by using a Berkely Neural Parse tool; wherein, the part-of-speech tag represents the tag information of a single word, and the syntactic component represents the structure grouping information of the text span;

step 3.2, for each x in the input sequence _i Mapping its context characteristics and syntax information to keys and values in a key-value memory network KVMN, denoted as K respectively _i ＝[k _i,1 ,...k _i,j ,...,k _i,mi ]And V _i ＝[v _i,1 ,...v _i,j ,...v _i,mi ]；

Step 3.3, k is embedded using two embedding matrices _i,j And v _i,j Respectively map to

And

for each x _i Associated context feature K _i And syntax information V _i Weight γ assigned to syntax information _ij Comprises the following steps:

wherein h is _i Is x _i A concealment vector obtained from a sequence coding layer; h is _i ^T Is h _i Transposed form of (1);

weight gamma _i,j Applied to corresponding syntax information v _i,j Above, as follows:

wherein alpha is _i For KVMN model corresponds to x _i Weighted syntax information of (1); therefore, the KVMN can ensure that the grammar information is weighted according to the corresponding context characteristics, so that the important information is distinguished and used;

step 3.4, sentence-wise Normal vector α for better utilization of syntax information encoded by KVMN _i And a context vector h _i Dynamic weighted combination using an evaluation function lambda _i To measure the context vector h _i And syntax vector alpha _i Contribution to the sentence:

λ _i ＝σ(W _λ1 .h _i +W _λ2 .α _i +b _λ )

wherein, W _λ1 And W _λ2 Is a trainable matrix; b _λ Is a bias term; sigma represents a sigmoid activation function;

and then the syntactic vector alpha is expressed _i And a context vector h _i Combining together:

where l is the vector dimension and h _i Matched 1 vector, and the resulting O _i Namely, the feature vector fusing the context information and the syntax information.

5. The method for recognizing named entities in chinese with fused lexical and syntactic information as claimed in claim 4, wherein said step 4 specifically comprises:

for an input sequence x ═ x ₁ ,x ₂ ,..,x _n ) Given a prediction sequence y ═ y (y) ₁ ,y ₂ ,y ₃ ,...,y _n ) The score for the predicted sequence is calculated as follows:

where, M is the transition matrix,

indicating slave label y _i To the label y _i+1 The transfer fraction of (a);

y for i word in sentence _i ' probability scores of tags; y is ₀ And y _n+1 Respectively representing a start and end tag; n represents the length of the input sequence;

the probability p (y | x) of the predicted sequence y generation is calculated using the softmax function as follows:

wherein, Y _x For solution space, represent the set of all possible predicted sequences for y;

is an example of the entire sequence tagged solution space,

a score representing the instance;

during the training process, the likelihood probability log (p (x, y)) of a correct predicted sequence is maximized as follows:

after continuous iterative training and back propagation, the obtained y ^* The result of labeling with the CRF sequence, i.e. the output of the final model:

training process continuously maximizes objective function

6. A Chinese named entity recognition system fusing lexical and syntactic information, comprising:

the input representation acquisition module is used for mapping an original input text into a word vector, introducing external vocabulary information by using an improved word set matching algorithm and integrating the external vocabulary information into the input representation of each word;

a context vector acquisition module for extracting context information using a bidirectional LSTM according to the input representation of the word;

the context information and syntax information fusion module is used for acquiring part-of-speech tags and syntax components from an original input text by using an NLP tool, constructing a syntax vector by using a healthy value memory network, and performing weighted fusion on the context vector and the syntax vector by using a gate control mechanism to acquire a feature vector;

and the sequence marking module is used for inputting the feature vectors into a CRF (conditional random access memory) of the label prediction layer to realize Chinese named entity identification.

Images