CN107203511B - A Network Text Named Entity Recognition Method Based on Neural Network Probabilistic Disambiguation - Google Patents

Abstract

The invention discloses a network text named entity recognition method based on neural network probability disambiguation, which comprises the steps of segmenting words of a non-tag corpus, extracting Word vectors by using Word2Vec, converting a sample corpus into Word feature matrixes and windowing, constructing a deep neural network for training, adding a softmax function into an output layer of the neural network for normalization processing, and obtaining a probability matrix of each Word corresponding to a named entity category; and (4) re-windowing the probability matrix, and disambiguating by using a conditional random field model to obtain the final named entity label. The invention provides a word vector increment learning method without changing a neural network structure according to the characteristics of network words and new words, and adopts a probability disambiguation method for solving the problems of non-standard grammatical structure and many wrongly written words in a network text. Therefore, the method of the invention can generate higher accuracy in the task of identifying the network text naming entity.

Description

A Network Text Named Entity Recognition Method Based on Neural Network Probabilistic Disambiguation

technical field

The invention relates to the processing and analysis of network texts, in particular to a method for network text named entity recognition based on neural network probability disambiguation.

Background technique

The network makes the speed and scale of information collection and dissemination reach an unprecedented level, and realizes the global information sharing and interaction. It has become an indispensable infrastructure of the information society. Modern communication and dissemination technologies have greatly improved the speed and breadth of information dissemination. But the accompanying problems and "side effects" are: the surging information sometimes makes people at a loss, and it becomes very difficult to quickly and accurately obtain the information they need most from the vast sea of information. How to analyze the people, places, institutions and other named entities that Internet users pay attention to from the massive network texts provides important support information for various upper-level applications such as online marketing and group sentiment analysis. This makes named entity recognition for web text an important core technology in web data processing and analysis.

The research on the methods that people deal with named entity recognition is mainly divided into two categories, rule-based methods and statistical-based methods. With the continuous improvement of machine learning theory and the great improvement of computing performance, methods based on statistics are more favored by people.

At present, statistical model methods for named entity recognition applications mainly include: Hidden Markov Model, Decision Tree, Maximum Entropy Model, Support Vector Machine, Conditional Random Field and Artificial Neural Network. Artificial neural network can achieve better results than conditional random field and maximum entropy model in terms of named entity recognition, but it is still mainly based on conditional random field and maximum entropy model, such as patent number CN201310182978.X. Combined with the named entity library, the airport proposed a named entity recognition method and device for microblog text, and the patent number CN200710098635.X proposed a named entity recognition method using word features and maximum entropy model modeling. The reason why artificial neural networks are difficult to use is that artificial neural networks often need to convert words into vectors in the word vector space in the field of named entity recognition, so the corresponding vectors cannot be obtained for new words, so they cannot be applied on a large scale.

Based on the above situation, the named entity recognition for network text mainly has the following problems: First, because there are a large number of network words, new words, and typos in network text, it is impossible to train a word vector space containing all words to train neural networks. Second, the existence of arbitrary language forms, irregular grammatical structures, and many typos in network texts lead to a decline in the accuracy of named entity recognition.

SUMMARY OF THE INVENTION

Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a network text named entity recognition method based on neural network probability disambiguation, which does not require retraining of neural network and simultaneous probability disambiguation recognition by incrementally extracting word features , the method obtains the predicted probability matrix of the named entity type to which the word belongs by training the neural network, and then disambiguates the predicted matrix output by the neural network with the probability model, which improves the accuracy and accuracy of the named entity recognition in the network text. .

Technical scheme: In order to realize the above-mentioned purpose, the technical scheme adopted in the present invention is:

A network text named entity recognition method based on neural network probability disambiguation, segmenting unlabeled corpus, using Word2Vec to extract word vectors, converting the sample corpus into word feature matrix and windowing, and constructing a deep neural network for training. The softmax function is added to the output layer for normalization, and the probability matrix of each word corresponding to the named entity category is obtained. Rewindow the probability matrix and disambiguate using the conditional random field model to obtain the final named entity annotation.

Specifically include the following steps:

Step 1: Obtain unlabeled corpus through a web crawler, obtain sample corpus marked with named entities from the corpus, and use natural language tools to segment the unlabeled corpus.

Step 2: Train the word vector space on the segmented unlabeled corpus and sample corpus through the Word2Vec tool.

Step 3: Convert the text in the sample corpus into a word vector representing word features according to the trained Word2Vec model, window the word vector, and use the window w multiplied by the two-dimensional matrix of the word vector length d as the input of the neural network. Convert the labels in the sample corpus into one-hot form as the output of the neural network. The output layer of the neural network is normalized by the softmax function, so that the classification result of the neural network is the probability that the vocabulary belongs to non-named entities and various named entities, and the structure, depth, number of nodes, step size, and activation function in the neural network are adjusted. , the initial value parameters, and the selected activation function to train the neural network.

Step 4: Re-window the prediction matrix output by the neural network, take the context prediction information of the word to be labeled as the correlation point of the actual classification of the word to be labeled in the conditional random field model, and use the EM algorithm according to the training corpus to calculate the value of each side. Expected value, train the corresponding conditional random field model.

Step 5: During recognition, first convert the text to be recognized into word vectors representing word features according to the trained Word2Vec model. If the Word2Vec model does not contain the corresponding training vocabulary, then incremental learning, obtaining word vectors, and backtracking word vectors are used. The spatial method converts the word into a word vector and window the word vector, and takes the two-dimensional matrix of the window w multiplied by the word vector length d as the input of the neural network. Then, the prediction matrix obtained by the neural network is re-windowed into the trained conditional random field model for disambiguation, and the final named entity annotation in the text to be recognized is obtained.

Preferably: the parameters of the Word2Vec tool are as follows: the length of the word vector is 200, the number of iterations is 25, the initial step size is 0.025, the minimum step size is 0.0001, and the CBOW model is selected.

Preferably, the parameters of the neural network are as follows: 2 hidden layers, 150 hidden nodes, a step size of 0.01, a batchSize of 40, and the activation function using a sigmoid function.

Preferred: the method of converting the tags in the sample corpus into one-hot form: correspondingly convert the "/o", "/n", and "/p" tags in the sample corpus into named entity tags "/Org-B" ", "/Org-I", "/Per-B", "/Per-I", "/Loc-B", "/Loc-I" are converted into one-hot form.

Preferred: The window size for word vector windowing is 5.

Preferably: during the training of the neural network, one tenth of the vocabulary is extracted from the sample data and does not participate in the training of the neural network, which is used as the measurement standard of the neural network.

Compared with the prior art, the present invention has the following beneficial effects:

The word vector can be incrementally extracted without retraining the neural network, and the neural network can be used to predict and disambiguate with a probability model, which makes the method more practical, accurate and accurate in the named entity recognition of network text. In the named entity recognition task of network text, the present invention provides a word vector incremental learning method that does not change the neural network structure according to the characteristics of the existence of network words and new words. For the problem of many typos, the method of probability disambiguation is adopted. Therefore, the method of the present invention can produce a higher accuracy rate in the network text named entity recognition task.

Description of drawings

FIG. 1 is a flow chart of training a network text named entity recognition device based on neural network probability disambiguation according to the present invention.

FIG. 2 is a flow chart of converting words into word features according to the present invention.

FIG. 3 is a schematic diagram of text processing and neural network structure according to the present invention.

Detailed ways

Below in conjunction with the accompanying drawings and specific embodiments, the present invention will be further clarified. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. Modifications in the form of valence all fall within the scope defined by the appended claims of the present application.

A network text named entity recognition method based on neural network probability disambiguation, segmenting unlabeled corpus, using Word2Vec to extract word vectors, converting the sample corpus into word feature matrix and windowing, and constructing a deep neural network for training. The softmax function is added to the output layer for normalization, and the probability matrix of each word corresponding to the named entity category is obtained. Rewindow the probability matrix and disambiguate using the conditional random field model to obtain the final named entity annotation.

Specifically include the following steps:

Step 1, through the web crawler unlabeled network text, and download the corpus marked with named entities from each corpus as the sample corpus, and use natural language tools to segment the unlabeled corpus.

Step 2: Train the word vector space on the segmented unlabeled corpus and sample corpus through the Word2Vec tool.

Step 3: Convert the text in the sample corpus into a word vector representing word features according to the trained Word2Vec model, and use it as the input of the neural network. Convert the labels in the sample corpus into one-hot form as the output of the neural network, because in the text processing task, a named entity may be divided into multiple words, so in order to ensure the integrity of the identified named entity, the labeling form adopts IOB mode is marked.

What kind of named entity a word is can not be determined only by the word itself, but also by the context information in which the word is located. Therefore, when building a neural network, we introduce the concept of a window, that is, when judging a word, the word and its fixed-length context are combined. The feature information is used as the input of the neural network. The input of the neural network is no longer the length d of the word feature vector, but a two-dimensional matrix of the window w multiplied by the word feature length d.

The output layer of the neural network is normalized by the softmax function, so that the classification result of the neural network is the probability that the vocabulary belongs to non-named entities and various named entities. Adjust the structure, depth, number of nodes, step size, activation function, initial value parameters in the neural network and select the activation function to train the neural network.

Step 4: Re-window the prediction matrix output by the neural network, take the context prediction information of the word to be labeled as the correlation point of the actual classification of the word to be labeled in the conditional random field model, and use the EM algorithm according to the training corpus to calculate the value of each side. Expected value, train the corresponding conditional random field model.

Step 5: During recognition, first convert the text to be recognized into word vectors representing word features according to the trained Word2Vec model. If the Word2Vec model does not contain the corresponding training vocabulary, then incremental learning, obtaining word vectors, and backtracking word vectors are used. The spatial method converts the word into a word vector.

(1) Match the words to be converted in the trained word vector space.

(2) If the words to be converted can be matched in the word vector space, the words are directly converted into corresponding word vectors.

(3) If the Word2Vec model does not contain the corresponding vocabulary, back up the word vector space to prevent the word space offset caused by incremental learning from reducing the accuracy of the neural network model, load the Word2Vec model, and obtain the sentence where the mismatched vocabulary is located. The sentence where the vocabulary is located, put it into the Word2Vec model for incremental training, and obtain the word vector of the vocabulary, and use the backed up word vector space to backtrack the model.

Windowing the word vector, the two-dimensional matrix of the window w multiplied by the word vector length d is used as the input of the neural network. Then, the prediction matrix obtained by the neural network is re-windowed into the trained conditional random field model for disambiguation, and the final named entity annotation in the text to be recognized is obtained.

example

Crawler web text from Sogou News website, download named entity corpus from Datatang corpus as sample corpus, use natural language tools to segment the crawler web text, and use the gensim package in python to pass Word2Vec model. The training of the word vector space is carried out. The specific parameters are as follows. The length of the word vector is 200, the number of iterations is 25, the initial step size is 0.025, the minimum step size is 0.0001, and the CBOW model is selected.

Convert the text of the sample corpus into the word vector representing the word feature according to the trained Word2Vec model. If the Word2Vec model does not contain the corresponding training vocabulary, the method of incremental learning, obtaining the word vector, and backtracking the word vector space is used for the word. Convert to word vectors. as a feature of each word. Convert the "/o", "/n", "/p" and other tags in the sample corpus provided by Datatang into named entity tags "/Org-B", "/Org-I", "/Per-B accordingly. ", "/Per-I", "/Loc-B", "/Loc-I", etc., and converted into one-hot form as the output of the neural network.

The window size is set to 5, that is, when considering the named entity category of the current word, the word features of itself and the two words before and after are used as the input of the neural network. The input of the neural network is a vector of batchSize*1000, from the sample data. One-tenth of the vocabulary is extracted from the neural network and does not participate in the training of the neural network. As the measurement standard of the neural network, the output layer of the neural network is normalized by the softmax function, so that the classification result of the neural network is that the vocabulary belongs to non-named entities and various types. The probability of the named entity, temporarily taking the maximum probability as the final classification result. Adjust the structure, depth, number of nodes, step size, activation function, initial value and other parameters in the neural network, so that the neural network can achieve better accuracy, the final specific parameters are as follows, the hidden layer is 2, the number of hidden nodes is 150, step The length is 0.01, and the batchSize is 40. When the activation function uses sigmoid, it can produce a good classification effect, and the accuracy can reach 99.83%. The F value of the most representative person, place, and institution name can reach 93.4%, 84.2%, 80.4% .

Remove the step of taking the prediction matrix output by the neural network to take the maximum probability as the final classification result, directly rewindow the probability matrix, and use the contextual prediction information of the word to be labeled as the association of the actual classification of the word to be labeled in the conditional random field model According to the training corpus, the EM algorithm is used to calculate the expected value of each side of the conditional random field, and the corresponding conditional random field model is trained. After using the conditional random field for disambiguation, the F value of the names of people, places, and institutions can be increased to 94.8 %, 85.0%, 82.0%.

It can be seen from the above specific embodiments that, compared with the traditional supervised named entity recognition method, the text named entity recognition method based on neural network probability disambiguation provided by the present invention uses an incrementally extractable word The word vector transformation method that does not generate the spatial offset of the word vector, enables the neural network to be applied to the network text with many new words and typos. Moreover, the present invention re-windows the probability matrix output by the neural network, adopts the conditional random field model to disambiguate the context, and can better solve the phenomenon of many typos and irregular grammar in the network text.

The above is only the preferred embodiment of the present invention, it should be pointed out that: for those skilled in the art, without departing from the principle of the present invention, several improvements and modifications can also be made, and these improvements and modifications are also It should be regarded as the protection scope of the present invention.

Claims (6)

1. a network text named entity recognition method based on neural network probability disambiguation, it is characterized in that: with unlabeled corpus word segmentation, utilize Word2Vec to extract word vector, sample corpus is converted into word feature matrix and windowed, constructs deep neural network After training, the softmax function is added to the output layer of the neural network for normalization to obtain the probability matrix of each word corresponding to the named entity category; the probability matrix is re-windowed, and the conditional random field model is used for disambiguation to obtain the final name. Entity annotation, including the following steps: Step 1, obtain unlabeled corpus through web crawler, obtain sample corpus marked with named entities from the corpus, and use natural language tools to segment the unlabeled corpus; Step 2: Train the word vector space on the segmented unlabeled corpus and sample corpus through the Word2Vec tool; Step 3: Convert the text in the sample corpus into a word vector representing word features according to the trained Word2Vec model, window the word vector, and use the window w multiplied by the two-dimensional matrix of the word vector length d as the input of the neural network; The labels in the sample corpus are converted into one-hot form as the output of the neural network; the output layer of the neural network is normalized by the softmax function, so that the classification result of the neural network is the probability that the vocabulary belongs to non-named entities and various named entities, Adjust the structure, depth, number of nodes, step size, activation function, initial value parameters in the neural network and select the activation function to train the neural network; Step 4: Re-window the prediction matrix output by the neural network, take the context prediction information of the word to be labeled as the correlation point of the actual classification of the word to be labeled in the conditional random field model, and use the EM algorithm according to the training corpus to calculate the value of each side. Expected value, train the corresponding conditional random field model; Step 5: During recognition, first convert the text to be recognized into word vectors representing word features according to the trained Word2Vec model. If the Word2Vec model does not contain the corresponding vocabulary, then incremental learning, acquisition of word vectors, and backtracking of the word vector space are used. The method of converting the word into a word vector, and windowing the word vector, takes the two-dimensional matrix of the window w multiplied by the word vector length d as the input of the neural network; then re-window the prediction matrix obtained by the neural network into the trained Disambiguation is carried out in the conditional random field model of , and the final named entity annotation in the text to be recognized is obtained. 2. the network text named entity recognition method based on neural network probability disambiguation according to claim 1, is characterized in that: the parameter of described Word2Vec tool is as follows: word vector length is selected 200, iteration times 25 times, initial step size 0.025, The minimum step size is 0.0001, and the CBOW model is selected. 3. the network text named entity recognition method based on neural network probability disambiguation according to claim 1, is characterized in that: the parameter of described neural network is as follows: hidden layer 2 layers, 150 hidden nodes, step size 0.01, batchSize Choose 40, the activation function uses the sigmoid function. 4. the network text named entity recognition method based on neural network probability disambiguation according to claim 1, is characterized in that: the label in the sample corpus is converted into the method of one-hot form: "/o" in the sample corpus , "/n", "/p" tags are correspondingly converted to named entity tags "/Org-B", "/Org-I", "/Per-B", "/Per-I", "/Loc- B", "/Loc-I", and then converted to one-hot form. 5 . The network text named entity recognition method based on neural network probability disambiguation according to claim 1 , wherein the window size of the word vector windowing is 5. 6 . 6. the network text named entity recognition method based on neural network probability disambiguation according to claim 1, it is characterized in that: during neural network training, from the sample data, extract one tenth of the vocabulary not to participate in the training of neural network, as Metrics for Neural Networks.

Images