CN106569998A - Text named entity recognition method based on Bi-LSTM, CNN and CRF - Google Patents

Abstract

The invention discloses a text named entity recognition method based on Bi-LSTM, CNN and CRF. The method includes the following steps: (1) using a convolutional neural network to encode information at the character level of text words into character vectors; (2) combining character vectors and word vectors and passing them as input to a bidirectional LSTM neural network to The context information of each word is modeled; (3) at the output of the LSTM neural network, the continuous conditional random field is used to decode the label of the entire sentence and mark the entities in the sentence. The present invention is an end-to-end model that does not require data preprocessing in the unlabeled corpus except for pre-trained word vectors, so the present invention can be widely used in sentence labeling in different languages and fields.

Description

Bi-LSTM, CNN and CRF-based text named entity identification method

Technical Field

The invention relates to natural language processing, in particular to a text named entity recognition method based on a bidirectional LSTM neural network, a convolutional neural network and a Conditional Random Field (CRF).

Background

Natural Language Processing (NLP) is a cross discipline integrating linguistics and computer disciplines. Named Entity Recognition (NER) is a basic task in natural language processing, and aims to recognize proper nouns and meaningful quantitative phrases in natural language texts and classify the proper nouns and meaningful quantitative phrases. With the rise of information extraction and big data concepts, named entity recognition tasks are increasingly emphasized by people and become important components of natural language processing such as public opinion analysis, information retrieval, automatic question answering and machine translation. How to automatically, accurately and quickly identify named entities from massive internet text information gradually becomes a hot problem concerned by academia and industry.

High performance named entity recognition methods typically employ Conditional Random Fields (CRFs), SVMs, and perceptrons to model manually processed features. Some researchers have proposed an efficient neural network model that requires only a small amount of feature engineering, and the model does not need important word vector features learned from a large amount of unlabeled data, but rather extracts vector features from a large amount of data through an unsupervised learning method.

However, this model has many limitations. First, the model utilizes a simple feedforward neural network that limits the context information for each word to a fixed length window, which gives up information between long-distance words; secondly, because only the word vector information is relied on, the method can not explore the character hierarchy characteristics of the words, such as prefix information and suffix information, and the information is often useful, especially in the new words expressed by word vectors with poor training effect. We seek a more efficient neural network model that addresses the above problems.

A Recurrent Neural Network (RNN) is capable of handling variable length input vectors and of long term memory in the process. In recent years, RNNs have had great success in the processing of diverse NLP tasks such as speech recognition, machine translation, and language modeling. Long-short term memory artificial neural network (LSTM) with forget gate allows important dependencies of Long-distance dependencies to be easily learned.

For sequence labeling tasks such as named entity recognition and speech recognition, the Bi-directional LSTM (Bi-LSTM) neural network is efficient for word indeterminate length context problems, or context constraint problems: when LSTMs are learned from past information in the named entity recognition task, limitations in computing power and quality of word vectors can limit their efficiency.

Convolutional Neural Network (CNN) is also used in NLP task to model information at character level, and there is an example of successfully applying CNN to extract character features and apply the features to named entity recognition task.

Disclosure of Invention

The invention aims to detect and label entities in natural texts to obtain effective text information, and provides a text named entity identification method based on a bidirectional LSTM neural network, a convolutional neural network and a Conditional Random Field (CRF).

The text named entity recognition method based on Bi-LSTM, CNN and CRF comprises the following steps:

(1) performing data preprocessing according to text data, and performing sentence segmentation and word segmentation on the text to obtain character feature vectors;

(2) extracting character feature vectors of each word by using a convolutional neural network;

(3) combining the character feature vectors and the word vectors, transmitting the character feature vectors and the word vectors into an LSTM neural network, and training by using a bidirectional LSTM neural network to obtain input sentence information features;

(4) and (4) aiming at the semantic features obtained in the step (3), carrying out entity labeling on each word by using a conditional random field, and marking entity information in the sentence sequence.

The steps can be realized in the following way:

the steps of preprocessing the text data, and performing sentence segmentation and word segmentation on the text to obtain the character feature vector can be specifically realized as follows:

(1) sentence and word segmentation is carried out on the document by adopting a natural language processing tool, so that the document is presented in sentences and one token per sentence;

(2) counting the sentences, words and labels obtained in the step (1) to form a sentence table, a vocabulary table and a label table;

(3) counting the characters in the word list in the last step (1) to form a character list;

(4) and forming a character feature vector by using the existing pre-trained word vector and the character table.

The step of extracting the character feature vector of each word by using the convolutional neural network can be specifically realized as follows:

(1) let C be the character table, d be the dimension of each character vector, and the character vector matrix be Q ∈ R^d×|C|；

(2) The predetermined word k ∈ V is composed of consecutive characters c₁,c₂,...,c_l]Where l is the length of the word k, the character vector matrix for k is composed of C^k∈R^d×lGiven, where the ith column is the character c_iThe vector of (a);

(3) at C^kAnd a kernel H ∈ R of width w^d×wAfter that, adding bias value bias to the convolution layer, and carrying out nonlinear regression on the whole convolution result to realize the feature mapping map f^k∈R^l-w+1Wherein the function f is mapped^kThe ith element f^k[i]Is given by formula (1);

f^k[i]＝tanh(<C^k[*,i:i+w-1],H>+b) (1)

wherein is all rows, C^k[*,i:i+w-1]Is C^kFrom the ith column to the (i + w-1) th column,<,>is Frobenius inner product, b is offset vector;

(4) finally, toExpressed as the corresponding characteristic of the kernel H.

Combining the character feature vectors and the word vectors, transmitting the character feature vectors and the word vectors into an LSTM neural network, and training by using a bidirectional LSTM neural network to obtain the information features of the input sentences by the following steps:

(1) definition of x_tIs the input character feature vector at time t, h_tFor the hidden layer state vector to store all useful information at time t, σ is sigmoid regression layer, and x is inner product, U_i,U_f,U_c,U_oFor input x in different states_tWeight matrix of W_i,W_f,W_c,W_oIs a hidden layer state h_tWeight matrix of b_i,b_f,b_c,b_oIs a bias vector;

(2) the calculation of the forget gate at time t is shown in equation (2):

f_t＝σ(W_fh_t-1+U_fx_t+b_f) (2)

(3) update h at time t_t-1The calculation formula of all the information stored in (1) is shown as (3) and (4):

i_t＝σ(W_ih_t-1+U_ix_t+b_i) (3)

whereinA vector that can be added to the cell state at time t;

(4) and updating the information stored at the time t-1 to the stored information at the time t, wherein the calculation formula is shown as an expression (5):

whereinA vector of a cell state at the time t;

(5) the output at time t is as shown in equation (6), and h is updated_tThe calculation formula is shown in formula (7):

o_t＝σ(W_oh_t-1+U_ox_t+b_o) (6)

h_t＝o_t*tanh(C_t) (7)

wherein o is_tIs the output at time t; h is_tA vector of a hidden layer at time t;

(6) h in the above step_tStoring all the past time information, and setting a g again by the same method_tFor storing future information, the last two hidden layer information are concatenated to form the final output vector.

The method for marking out the entity information in the sentence sequence by using the conditional random field to mark the entity of each word comprises the following specific steps:

(1) with z ═ z₁,z₂,...,z_nDenotes the input sequence, where n is the length of the input sequence, z_iIs the input vector of the ith word, y ═ y₁,y₂,...,y_nY (z) ═ y'₁,y'₂,...,y'_nA possible output tag sequence of z;

(2) the conditional random field probability model defines the conditional probability p (yz; W, b) over all possible tag sequences y' for a given input sequence z:

wherein,w is a weight vector and b is a bias vector for the potential gain function;and b_y',yWeight vectors and bias vectors corresponding to the tag pair (y', y), respectively;

(3) a Conditional Random Field (CRF) training phase, using a maximum conditional likelihood estimation method to select the parameters that maximize likelihood L (W, b), for a training set { (z)_i,y_i) The logarithm of likelihood is:

(4) finding the y with the highest conditional probability among the tag sequences to tag the sequences:

compared with the prior art, the invention has the beneficial effects that:

the invention uses a method of combining a convolution neural network and a bidirectional LSTM neural network to extract character features and word features, and compares the character features and the word features with the traditional method such as a hidden Markov model; the method combines the neural network and the conditional random field, has wide application scenes, can solve the problem of labeling the named entities in multiple fields only by training different data corresponding to different problems once, and has the advantage of end-to-end.

Drawings

FIG. 1 is a key step workflow diagram for Bi-LSTM, CNN and CRF based textual named entity recognition;

FIG. 2 is a schematic diagram of a convolutional neural network architecture;

FIG. 3 is a schematic diagram of a bi-directional LSTM neural network.

FIG. 4 is a named entity recognition model training process algorithm based on bidirectional LSTM and CRF.

Detailed Description

The invention is further described with reference to the following figures and detailed description.

The invention mainly combines the neural networks aiming at the named entity recognition task, presents a bidirectional LSTM and CNN-based hybrid model, and the model can simultaneously learn character feature vectors and word vectors, and labels the learned feature vectors through a conditional random field on the basis. Therefore, efficient and accurate named entity recognition results are obtained. As shown in fig. 1, which is a work flow diagram of key steps of text named entity recognition based on Bi-LSTM, CNN and CRF, wherein the character expression of each word is completed by the convolutional neural network structure of fig. 2; the result of cascading the character feature vector and the word vector is input into a bidirectional LSTM neural network, wherein the LSTM neural network is shown in FIG. 3, and a dotted line in FIG. 1 indicates that a dropout layer is arranged on both the input vector and the output vector (to prevent the neural network from generating overfitting).

The text named entity recognition method based on Bi-LSTM, CNN and CRF comprises the following specific steps (1) - (4):

(1) and performing data preprocessing according to the text data, and performing sentence segmentation and word segmentation on the text to obtain character feature vectors. The method comprises the following specific steps:

(1.1) performing sentence and word segmentation on the document by adopting a natural language processing tool, so that the document is presented in sentences and one word per sentence (token);

(1.2) carrying out statistics on sentences, words and labels to form a sentence table, a vocabulary table and a label table;

(1.3) counting the characters in the word list to form a character list;

and (1.4) forming a character feature vector by using the existing pre-trained word vector and the character table.

(2) And extracting the character feature vector of each word by using a convolutional neural network. The method comprises the following specific steps:

(2.1) let C be the character table, d be the dimension of each character vector, and the character vector matrix be Q ∈ R^d×C；

(2.2) Preset word k ∈ V consisting of consecutive characters [ c ]₁,c₂,...,c_l]Where l is the length of the word k, the character vector matrix for k is composed of C^k∈R^d×lGiven, where the ith column is the character c_iThe vector of (a);

(2.3) at C^kAnd a kernel H ∈ R of width w^d×wAfter that, adding bias value bias to the convolution layer, and carrying out nonlinear regression on the whole convolution result to realize the feature mapping map f^k∈R^l-w+1Wherein the function f is mapped^kThe ith element f^k[i]Is given by formula (1);

f^k[i]＝tanh(<C^k[*,i:i+w-1],H>+b) (1)

wherein is all rows, C^k[*,i:i+w-1]Is C^kFrom the ith column to the (i + w-1) th column,<,>is Frobenius inner product, b is offset vector;

(2.4) finally, byExpressed as the corresponding characteristic of the kernel H.

(3) And combining the character feature vectors and the word vectors, and transmitting the character feature vectors and the word vectors into an LSTM neural network. As shown in FIG. 4, the input sentence information features are trained using a bi-directional LSTM neural network. The method comprises the following specific steps:

(3.1) definition of x_tIs the input character feature vector at time t, h_tFor the hidden layer state vector to store all useful information at time t, σ is sigmoid regression layer, and x is inner product, U_i,U_f,U_c,U_oFor input x in different states_tWeight matrix of W_i,W_f,W_c,W_oIs a hidden layer state h_tWeight matrix of b_i,b_f,b_c,b_oIs a bias vector;

(3.2) the calculation of the forget gate at time t is shown in equation (2):

f_t＝σ(W_fh_t-1+U_fx_t+b_f) (2)

(3.3) update h at time t_t-1The calculation formula of all the information stored in (1) is shown as (3) and (4):

i_t＝σ(W_ih_t-1+U_ix_t+b_i) (3)

whereinA vector that can be added to the cell state at time t;

(3.4) updating the information stored at the time t-1 to the stored information at the time t, wherein the calculation formula is shown as an expression (5):

whereinA vector of a cell state at the time t;

(3.5) the output at time t is as shown in equation (6), and h is updated_tThe calculation formula is shown in formula (7):

o_t＝σ(W_oh_t-1+U_ox_t+b_o) (6)

h_t＝o_t*tanh(C_t) (7)

wherein o is_tIs the output at time t; h is_tA vector of a hidden layer at time t;

(3.6) h in the above step_tStoring all the past time information, and setting a g again by the same method_tFor storing future information, the last two hidden layer information are concatenated to form the final output vector.

(4) And (4) aiming at the semantic features obtained in the step (3), carrying out entity labeling on each word by using a conditional random field, and marking entity information in the sentence sequence. The method comprises the following specific steps:

(4.1) with z ═ z₁,z₂,...,z_nDenotes the input sequence, where n is the length of the input sequence, z_iIs the input vector of the ith word, y ═ y₁,y₂,...,y_nY (z) ═ y'₁,y'₂,...,y'_nA possible output tag sequence of z;

(4.2) the conditional random field probability model defines the conditional probability p (yz; W, b) over all possible tag sequences y' for a given input sequence z:

wherein,w is a weight vector and b is a bias vector for the potential gain function;and b_y',yWeight vectors and bias vectors corresponding to the tag pair (y', y), respectively;

(4.3) Conditional Random Field (CRF) training orderA section for selecting a parameter for maximizing the likelihood L (W, b) using a maximum conditional likelihood estimation method, for the training set { (z)_i,y_i) The logarithm of likelihood is:

(4.4) finding the highest conditional probability y in the tag sequence to tag the sequence:

examples

Taking documents of English news of New York Times as an example, the method is applied to text named entity recognition, and specific parameters and methods in each step are as follows:

1. sentence and word segmentation are carried out on the document by adopting a natural language processing tool, so that each word of the document is a line, and the sentences are spaced by spaces;

2. respectively counting sentences, words and labels in 1 to form a sentence table, a word table and a label table, wherein five types of labels including ' PER (person name) ' LOC (place name) ' ORG (organization) ' FAC (organization) ' and ' GPE (geopolitical name) ' are marked in a training document, and all the labels in a test document are ' marked ', and the 1 Chinese document has 17 sentences and 466 words through counting;

3. carrying out character statistics on the word list in the step 1 to form a character list C;

4. combining the trained GloVe 100-dimensional vector disclosed by 6 hundred million Stanford with the word list in 2 to form a 100-dimensional word vector;

extracting character feature vectors of each word by using a convolutional neural network:

5. let C be the character table, d be the dimension of each character vector, and the character vector matrix be Q ∈ R^d×|C|；

6. Suppose the word k ∈ V consists of consecutive characters c₁,c₂,...,c_l]Where l is the length of the word k, the character vector matrix for k is composed of C^k∈R^d×lGiven, where the ith column is the character c_iThe vector of (a);

7. at C^kAnd a kernel H ∈ R of width w^d×wAfter that, adding bias value bias to the convolution layer, and carrying out nonlinear regression on the whole convolution result to realize the feature mapping map f^k∈R^l-w+1In particular, f^kIs given by the formula (1):

f^k[i]＝tanh(<C^k[*,i:i+w-1],H>+b) (1)

wherein is all rows, C^k[*,i:i+w-1]Is C^kFrom the ith column to the (i + w-1) th column,<A,B>＝Tr(AB^T) Is Frobenius inner product.

8. To be provided withExpressed as a corresponding characteristic of kernel H;

and inputting the character vector features obtained in the step 8 and the word vectors obtained in the step 4 into the bidirectional LSTM neural network:

9. the two-way LSTM neural network parameter updating mode is that a stochastic gradient descent algorithm with 30 being batch size and 0.9 being momentum term is used for updating, the initial learning rate is 0.015, and after each iteration, the learning rate updating formula is

η_t＝η₀(1+ρt) (1)

Wherein, the delay rate rho is 0.05, and t is the number of times of completing iteration;

10. definition of x_tIs the input character feature vector at time t, h_tFor the hidden layer state vector to store all useful information at time t, σ is sigmoid regression layer, and x is inner product, U_i,U_f,U_c,U_oFor input x in different states_tWeight matrix of W_i,W_f,W_c,W_oIs a hidden layer state h_tWeight matrix of b_i,b_f,b_c,b_oIs a bias vector;

11. the calculation of the forget gate at time t is shown in equation (2):

f_t＝σ(W_fh_t-1+U_fx_t+b_f) (2)

12. update h at time t_t-1The calculation formula of all the information stored in (1) is shown as (3) and (4):

i_t＝σ(W_ih_t-1+U_ix_t+b_i) (3)

13. and updating the information stored at the time t-1 to the stored information at the time t, wherein the calculation formula is shown as an expression (5):

14. the output at time t is as shown in equation (6), and h is updated_tThe calculation formula is shown in formula (7):

o_t＝σ(W_oh_t-1+U_ox_t+b_o) (6)

h_t＝o_t*tanh(C_t) (7)

15. in the above stepH of_tStoring all the past time information, and setting a g again by the same method_tFor storing future information, the last two hidden layer information are concatenated to form the final output vector.

Entity labeling is performed on each word using Conditional Random Fields (CRFs):

16. with z ═ z₁,z₂,...,z_nDenotes an input sequence, where z_iIs the input vector of the ith word, y ═ y₁,y₂,...,y_nY (z) ═ y'₁,y'₂,...,y'_nA possible output tag sequence of z;

17. a Conditional Random Field (CRF) probabilistic model defines the conditional probability p (y | z; W, b) over all possible tag sequences y for a given input sequence z:

wherein,w is a weight vector and b is a bias vector for the potential gain function;and b_y',yWeight vectors and bias vectors corresponding to the tag pair (y', y), respectively;

18. conditional Random Field (CRF) training phase, we use maximum conditional likelihood estimation for the training set { (z)_i,y_i) The logarithm of likelihood is:

the maximum likelihood training process selects the parameters that maximize the likelihood L (W, b).

19. Finding the y with the highest conditional probability among the tag sequences to tag the sequences:

20. and identifying the position of the marked words in the original file, and neatly feeding back the marked result to the user.

The following table is the partial final annotation result for the selected news document.

Claims (5)

1. a text named entity recognition method based on Bi-LSTM, CNN and CRF, is characterized in that comprising the steps: (1) Carry out data preprocessing according to the text data, divide the text into sentences and words, and obtain character feature vectors; (2) utilize convolutional neural network to carry out character feature vector extraction to each word; (3) Combining character feature vectors and word vectors, importing them into the LSTM neural network, and using the two-way LSTM neural network to train to obtain the input sentence information features; (4) For the semantic features obtained in (3), use the conditional random field to carry out entity labeling for each word, and mark the entity information in the sentence sequence. 2. a kind of text named entity recognition method based on Bi-LSTM, CNN and CRF according to claim 1, it is characterized in that text data is carried out data preprocessing, text is carried out sentence, participle, obtains the character feature vector Proceed as follows: (1) Use natural language processing tools to segment the document into sentences and words, so that the document is presented as sentences and a token for each sentence; (2) Sentences, words and tags are counted to form sentence tables, vocabulary lists and tag tables; (3) counting characters in the word table to form a character table; (4) Use the existing pre-trained word vectors and character tables to form character feature vectors. 3. a kind of text named entity recognition method based on Bi-LSTM, CNN and CRF according to claim 1, it is characterized in that described utilize convolutional neural network to carry out the step of character feature vector extraction to each word as follows: (1) Let C be the character list, d be the dimension of each character vector, and the character vector matrix is: Q∈R ^d×|C| ; (2) The preset word k∈V consists of continuous characters [c ₁ ,c ₂ ,...,c _l ], where l is the length of word k, then the character vector matrix of k is composed of C ^k ∈ R ^{d× l} is given, where the i-th column is a vector of characters c _i ; (3) Realize the convolution layer between C ^k and a kernel H∈R ^d×w with a width of w, after which add a bias value bias to the convolution layer, and perform nonlinear regression on the entire convolution result to Realize the feature map f ^k ∈ R ^l-w+1 , where the i-th element f ^k [i] of the mapping function f ^k is given by (1); f ^k [i]＝tanh(<C ^k [*,i:i+w-1],H>+b) (1) Where * is the number of all rows, C ^k [*,i:i+w-1] is the i-th column to i+w-1 column in C ^k , <,> is the Frobenius inner product, and b is the bias vector; (4) Finally, with as the corresponding feature expression of the kernel H. 4. a kind of text named entity recognition method based on Bi-LSTM, CNN and CRF according to claim 1, it is characterized in that character feature vector and word vector are combined, import into LSTM neural network, utilize two-way LSTM neural network The steps to train and obtain the information features of the input sentence are as follows: (1) Define x _t as the input character feature vector at time t, h _t is the hidden layer state vector storing all useful information at time t, σ is the sigmoid regression layer, * is the inner product, U _i , U _f , U _c , U _o is the weight matrix for the input x _t in different states, W _i , W _f , W _c , W _o are the weight matrices of the hidden layer state h _t , b _i , b _f , b _c , b _o are the bias vector; (2) The calculation of the forget gate at time t is shown in formula (2): f _t ＝σ(W _f h _t-1 +U _f x _t +b _f ) (2) (3) Update all information stored in h _t-1 at time t, and the calculation formulas are shown in (3) and (4): i _t =σ(W _i h _t-1 +U _i x _t +b _i ) (3) ${\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; C</span>}}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; tanh</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; b</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$ in is a vector that can be added to the cell state at time t; (4) Update the information stored at time t-1 to the stored information at time t at time t, and the calculation formula is formula (5): ${\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; *</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; i</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; *</span>{\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; C</span>}}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$ in is the vector of the cell state at time t; (5) The output at time t is shown in formula (6), and h _t is updated, and the calculation formula is shown in formula (7): o _t ＝σ(W _o h _t-1 +U _o x _t +b _o ) (6) h _t ＝o _t *tanh(C _t ) (7) Where o _t is the output at time t; h _t is the vector of the hidden layer at time t; (6) h _t in the above steps stores all the information of the past time, and another g _t is set in the same way to store future information, and the last two hidden layer information are concatenated to form the final output vector. 5. a kind of text named entity recognition method based on Bi-LSTM, CNN and CRF according to claim 1, it is characterized in that utilize conditional random field to carry out entity label to each word, mark out the entity information in the sentence sequence The steps are as follows: (1) Use z={z ₁ ,z ₂ ,...,z _n } to represent the input sequence, where n is the length of the input sequence, z _i is the input vector of the i-th word, y={y ₁ ,y ₂ ,...,y _n } is the actual output label sequence of z, Y(z)={y' ₁ ,y' ₂ ,...,y' _n } is the possible output label sequence of z; (2) The conditional random field probability model defines the conditional probability p(y|z; W, b) for a given input sequence z on all possible label sequences y': $\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; ;</span>\mathrm{<span\; class="notranslate">\; W</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Pi;</span>}}}{\mathrm{<span\; class="notranslate">\; \&psi;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; )</span>}{\underset{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; Y</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Pi;</span>}}}{\mathrm{<span\; class="notranslate">\; \&psi;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 8</span>}<span\; class="notranslate">\; )</span>$ in, is the potential gain function, W is the weight vector, and b is the bias vector; and b _{y', y} are the weight vector and bias vector corresponding to the label pair (y', y), respectively; (3) In the conditional random field (CRF) training stage, the maximum conditional likelihood estimation method is used to select the parameter that maximizes the likelihood rate L(W,b). For the training set {(z _i ,y _i )}, the likelihood rate The logarithm of is: $\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; W</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}\mathrm{<span\; class="notranslate">\; log</span>}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; ;</span>\mathrm{<span\; class="notranslate">\; W</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 9</span>}<span\; class="notranslate">\; )</span>$ (4) Find the y with the highest conditional probability in the label sequence to label the sequence: $\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; Y</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; argmax</span>}}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; ;</span>\mathrm{<span\; class="notranslate">\; W</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 10</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; .</span>$