CN110705293A - Electronic medical record text named entity recognition method based on pre-training language model - Google Patents

Abstract

The invention belongs to the technical field of medical information data processing, and in particular relates to a method for recognizing a named entity of electronic medical record text based on a pre-trained language model, comprising: collecting electronic medical record text from a public data set as original text, and preprocessing; Set to label the preprocessed original text entities, and get the labeling text; input the labeling text into the pre-training language model to obtain the training text represented by word vectors; build a BiLSTM-CRF sequence labeling model, learn the training text, and obtain the training labeling Model; using the trained annotation model as the entity recognition model, input the test text to output the labeled category label sequence. Using the text features and semantic information in the deep language model obtained by training in the super large-scale Chinese corpus can provide better semantic compression effect, avoid the tedious and complicated problem of manual annotation, and does not rely on dictionaries and rules, which improves the performance of named entity recognition. recall and accuracy.

Description

Named Entity Recognition Method for Electronic Medical Record Text Based on Pre-trained Language Model

technical field

The invention belongs to the technical field of medical information data processing, and in particular relates to a named entity recognition method for electronic medical record text based on a pre-trained language model.

Background technique

Case history is a record of medical staff's medical activities such as inspection, diagnosis, and treatment of the occurrence, development, and outcome of a patient's disease. A written patient medical health record is required. With the development of computer and Internet technology, most hospitals have realized the electronicization of clinical medical records. Electronic medical records are digital medical records that use electronic equipment to record, save, manage, transmit and reproduce digital medical records. , sharing, etc. The application of electronic medical records can not only provide the most practical and rich data for health management, medical diagnosis and treatment and scientific research, but also an important basis for determining responsibility for evaluating medical quality, management level and handling medical disputes. At present, electronic medical records are mostly entered in natural language, and then converted into structured data by computer for future data analysis and search.

Natural language processing is a cross direction in the field of computer and artificial intelligence. Named Entity Recognition (NER) is a basic task of natural language processing, which aims to identify entities with specific meanings in natural language texts, such as names, Place names, institution names, proper nouns, etc. As an important part of natural language processing such as public opinion analysis, information retrieval, query classification, automatic question answering, and machine translation, the quality of named entity recognition results directly affects the effect of natural language processing.

Existing named entity recognition methods generally include three types: dictionary-based methods, heuristic rule-based methods, and machine learning-based methods. The dictionary-based method first constructs a large-scale entity dictionary, and then realizes entity recognition by matching sentences and dictionaries, but this method relies heavily on the dictionary, cannot identify unregistered words, and cannot identify entity matching situations, and the accuracy is not high; based on The method of heuristic rules is to construct rules according to the specific context features of entities, and then match the text with the rules to realize entity recognition. However, this method requires linguistic background knowledge when constructing rules. Due to the diversity of Chinese expressions, rules are difficult to achieve. Enumeration and easy conflict, resulting in limited recall and accuracy; machine learning-based methods formalize the task of named entity recognition as a sequence labeling task, and jointly predict entity boundaries and labels by predicting each word or each word. entity type, but this method requires a lot of manual annotation.

SUMMARY OF THE INVENTION

Therefore, the technical problem to be solved by the present invention is to overcome the defects of limited accuracy and recall and cumbersome and complicated existing named entity recognition methods, thereby providing a fast and simple pre-training language-based language with high accuracy and recall. Model named entity recognition method for electronic medical record text.

In order to solve the above-mentioned technical problems, the technical scheme adopted in the present invention is:

The present invention provides an electronic medical record text named entity recognition method based on a pre-trained language model, comprising the following steps:

Step 1, collect the electronic medical record text from the public data set as the original text, and perform data preprocessing on the original text;

Step 2, performing entity labeling on the original text preprocessed by the data in Step 1 based on the standard medical term set to obtain the labeling text;

Step 3, inputting the marked text into the pre-training language model to obtain the training text represented by the word vector;

Step 4, build a BiLSTM-CRF sequence labeling model, train the training text, and obtain a trained labeling model;

Step 5, using the trained labeling model as an entity recognition model, and inputting the test text to output the labelled category label sequence.

Preferably, in the pre-trained language model-based named entity recognition method for electronic medical record text, in step 1, the words in all the original texts in the data set are counted, stop words and useless symbols are removed, and a dictionary file is generated.

Preferably, in the named entity recognition method for electronic medical record text based on a pre-trained language model, in step 2, based on the SNOMED CT medical term set and using the BIO labeling mode to mark the original text that has undergone data preprocessing in step 1. Diseases and six entities of diagnosis, examination, examination, surgery, medicine, anatomy.

Further preferably, in the named entity recognition method for electronic medical record text based on a pre-trained language model, in step 3, an ERNIE pre-trained language model is used.

Further preferably, in the named entity recognition method for electronic medical record text based on a pre-trained language model, in step 3, the ERNIE pre-trained language model is trained in a semi-supervised parallel manner.

Further preferably, in the named entity recognition method for electronic medical record text based on a pre-trained language model, in step 4, the BiLSTM-CRF sequence labeling model includes a look-up layer, a bidirectional LSTM layer and a CRF layer.

Further preferably, in the named entity recognition method for electronic medical record text based on a pre-trained language model, in step 4, the word vector of the training text is input into the BiLSTM-CRF sequence annotation model and iteratively trained.

Further preferably, in the named entity recognition method for electronic medical record text based on a pre-trained language model, in step 4, a loss value is calculated by constructing a loss function to determine the number of iterative training times, until the loss value is less than a set threshold.

Further preferably, in the named entity recognition method for electronic medical record text based on a pre-trained language model, in step 4, the category label sequence and labeled data of the training text are input into the loss function to calculate the loss value.

The technical scheme of the present invention has the following advantages:

1. The electronic medical record text named entity recognition method based on the pre-trained language model provided by the present invention comprises: Step 1, collect electronic medical record text from a public data set as the original text, and perform data preprocessing on the original text; Step 2, based on The standard medical term set performs entity labeling on the original text preprocessed by data in step 1, and obtains the labeling text; step 3, inputs the labeling text into the pre-training language model, and obtains the training text represented by the word vector; step 4, constructs BiLSTM -CRF sequence labeling model, learn the training text to obtain the labeling model; step 5, use the labeling model as the entity recognition model, and input the test text to output the labelled category label sequence.

The electronic medical record text named entity recognition method based on the pre-trained language model provided by the present invention utilizes the text features and semantic information in the deep language model obtained by training in the super large-scale Chinese corpus, provides better semantic compression effect, and avoids manual annotation. Cumbersome and complex problems, and do not rely on dictionaries and rules, improve the recall rate and accuracy of named entity recognition, and improve the recognition effect of named entities.

2. The electronic medical record text named entity recognition method based on the pre-trained language model provided by the present invention can ensure that the key information learned from the public data set is more comprehensive, improves the recall rate, and is more convenient to check the electronic medical record text. mistake.

3. The electronic medical record text named entity recognition method based on the pre-training language model provided by the present invention adopts the ERNIE pre-training language model to replace the conventional Bert pre-training language model, and learns the complete entity semantic representation by modeling the entity concept in the massive data , which can better represent the semantic information of entities.

Description of drawings

In order to illustrate the specific embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the specific embodiments or the prior art. Obviously, the accompanying drawings in the following description The drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

1 is a schematic flowchart of a method for identifying named entities in electronic medical record text based on a pre-trained language model provided in Embodiment 1 of the present invention;

Figure 2 is a schematic diagram of the BiLSTM-CRF sequence annotation model.

Detailed ways

In order to facilitate understanding of the purpose, technical solutions and gist of the present invention, the embodiments of the present invention will be described in further detail below. The present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, this embodiment is provided so that this disclosure will be thorough and complete, and will fully convey the concept of the present invention to those skilled in the art, and the present invention will only be defined by the appended claims.

Example 1

This embodiment provides a method for identifying named entities in electronic medical record text based on a pre-trained language model, as shown in FIG. 1 , including the following steps:

Step 1, collect the electronic medical record text from the public data set as the original text, and perform data preprocessing on the original text;

Specifically, in this embodiment, the corpus used in the original text comes from the electronic medical record text collected by the public data set, and the words appearing in all the original texts in the data set are counted, and stop words, irrelevant symbols, etc. are removed to generate a dictionary document.

Step 2: Perform entity labeling on the original text preprocessed by the data in Step 1 based on the standardized medical term set to obtain the labelled text;

Specifically, the corpus in the original text preprocessed by the data in step 1 is annotated based on the SNOMED CT medical term set and the BIO annotation mode is used to obtain the annotated text. In this annotated text, corpus entities are divided into six categories:

Diseases and diagnosis: diseases defined in medicine and judgments made by doctors on etiology, pathophysiology, classification and staging, etc. in clinical work;

Inspection: imaging inspection (such as X-ray, CT, MR, PETCT, etc.), angiography, ultrasound, electrocardiogram, etc. In order to avoid excessive conflict between inspection operations and surgical operations, other diagnostic operations (such as gastroscope, colonoscopy, etc. are not included) );

Test: physical or chemical tests performed in the laboratory, this embodiment specifically refers to tests performed by the laboratory in clinical work, excluding generalized laboratory tests such as immunohistochemistry;

Surgery: Resection, suture and other treatments performed by doctors on the part of the patient's body are the main treatment methods of surgery;

Drugs: specific chemicals used in the treatment of diseases;

Anatomical Site: Refers to the anatomical site of the human body where disease, symptoms and signs occur.

1000 items were extracted from the annotated texts marked by entities as the annotation set, and the statistics of the annotation set are shown in Table 1:

Table 1 Statistics of the number of entities

original text Diseases and Diagnoses an examination test Operation drug anatomical site total callout set 1000 2116 222 318 765 456 1486 5363

.

Step 3: Input the marked text into the pre-training language model to obtain the training text represented by the word vector;

Specifically, obtain Baidu's ERNIE pre-trained language model and several electronic medical record text corpora, train the ERNIE model in a semi-supervised parallel two-way manner according to the electronic medical record text corpus, obtain a new language model, and input the annotation set of the annotated text in step 2 To the new training model, the training set of the training text represented by the word vector is obtained.

Step 4, build a BiLSTM-CRF sequence labeling model, train the training text, and obtain the trained labeling model;

The BiLSTM-CRF sequence labeling model is a bidirectional recurrent neural network, which can predict the probability of the label for the input word according to the context information. The model is divided into three layers: the first layer is the look-up layer, the second layer is the bidirectional LSTM layer (ie BiLSTM layer), and the third layer is the CRF layer.

As shown in Figure 2, input the one-hot vector of the words of the training text in step 3 into the model, that is, take the sentence as the unit, and denote a sentence (sequence of words) containing n words as x=(x ₁ ,x ₂ ,...,x _n ), where x _i represents the id of the i-th word of the sentence in the dictionary file, and then the one-hot vector of the word is obtained, and the dimension of the vector is the size of the dictionary file.

As the first layer of the model, the look-up layer uses a pre-trained or randomly initialized embedding matrix to map each word x _i in the sentence from a one-hot vector to a low-dimensional dense word vector (character embedding), where, x _i ∈ R ^d , d is the dimension of embedding. Set dropout to alleviate overfitting before feeding into the next layer.

A bidirectional LSTM layer is used as the second layer of this model to automatically extract sentence features. Use the character embedding sequence (x ₁ , x ₂ ,..., x _n ) of each word of a sentence as the input of each time step of the bidirectional LSTM model, and then put the hidden state sequence output by the forward LSTM and the reverse LSTM sequence in the The hidden states output by each position are spliced by position to obtain a complete hidden state sequence (h ₁ , h ₂ ,...,h _n )∈R ^n×m . After setting dropout, a linear layer is connected to map the hidden state vector from m dimension to k dimension, where k is the number of labels in the training set, so as to obtain automatically extracted sentence features, denoted as matrix P=(p ₁ ,p ₂ ,...,p _n )∈R ^n×k . Each dimension p _ij of p _i ∈ R ^k is regarded as a scoring value for classifying the word x _i to the jth label. If Softmax is performed on P, it is equivalent to classifying each position independently of k classes, but In this way, the marked information cannot be used when marking each position, so it is necessary to access the CRF layer for marking.

As the third layer of the model, the CRF layer is used for sentence-level sequence annotation. The parameter of the CRF layer is a (k+2)×(k+2) transition matrix A, and _Aij represents the transition score from the i-th label to the j-th label, and then it can be used to label a position. Utilize the tags that have been marked before. Since we need to add a start state at the beginning of the sentence and a stop state at the end of the sentence, add 2 to k. If we remember a label sequence y=(y ₁ , y ₂ ,..., y _n ) whose length is equal to the length of the sentence, then the model's score for the label of sentence x is equal to y's score:

According to formula (1), the score of the entire sequence is equal to the sum of the scores of each position, and the score of each position is obtained by two parts, one part is determined by the pi of the _LSTM , and the other part is determined by the transition matrix A of the CRF layer, and then The normalized probability can be obtained using Softmax:

By maximizing the log-likelihood function during model training, equation (3) gives the log-likelihood of a training sample (x, y ^x ):

The model uses the Viterbi algorithm of dynamic programming to solve the optimal path during the prediction process (decoding):

Input the word vector of the training set of the training text in step 3 into the above-mentioned BiLSTM-CRF sequence labeling model to obtain the category label sequence. Each word vector in the training set of the input training sample corresponds to a label, which is marked with the BIO ternary notation method. For the above six types of entities, there are 'DISEASE-I': 1, 'DISEASE-B': 2, 'CHECK-B': 3, 'CHECK-I': 4, 'EXAMINE-I': 5, 'EXAMINE- B': 6, 'OPERATION-B': 7, 'OPERATION-I': 8, 'MEDICINE-I': 9, 'MEDICINE-B': 10, 'BODY-B': 11, 'BODY-I' :12 and O have a total of 13 categories, and there are no labels in the test set.

In order to ensure that the class label sequence output in each round of the training process is more correct, a loss function is constructed, the class label sequence and label data of the training set of the training text are input into the loss function, the loss value is calculated, and a standard value is set as Set the threshold. By comparing the loss value with the set threshold each time, if the loss value is greater than or equal to the set threshold, the next round of training is iteratively trained until the loss value is less than the set threshold, that is, the sequence output by the CRF layer of the labeling model is considered to be the optimal solution.

Step 5: Use the trained labeling model as the entity recognition model, and input the test text to output the labelled category label sequence;

Specifically, save the learned labeling model as the entity recognition model, take the new electronic medical record text as the test text, select 300 records as the test set, and input it into the entity recognition model to output the labeled category label sequence, training set and test set. The number of texts and accuracy statistics are shown in Table 2 respectively:

Table 2 Statistics of the number of texts and the accuracy of the training set and test set

Training set test set number of text 6292 1573 Accuracy 0.9487 0.8196

It can be seen from Table 2 that the labeling model trained by the training text has a higher recognition accuracy rate, and thus the labeling model as an entity labeling model has a higher accuracy rate for entity recognition in the test set.

Obviously, the above-mentioned embodiments are only examples for clear description, and are not intended to limit the implementation manner. For those of ordinary skill in the art, changes or modifications in other different forms can also be made on the basis of the above description. There is no need and cannot be exhaustive of all implementations here. And the obvious changes or changes derived from this are still within the protection scope of the present invention.

Claims (9)

1. A method for recognizing a text named entity of an electronic medical record based on a pre-training language model is characterized by comprising the following steps of:

step 1, collecting an electronic medical record text from a public data set as an original text, and performing data preprocessing on the original text;

step 2, carrying out entity labeling on the original text subjected to data preprocessing in the step 1 based on a normative medical term set to obtain a labeled text;

step 3, inputting the marked text into a pre-training language model to obtain a training text represented by a word vector;

step 4, constructing a BilSTM-CRF sequence labeling model, and training the training text to obtain a trained labeling model;

and 5, inputting a test text by taking the trained labeling model as an entity recognition model, and outputting a labeled class label sequence.

2. The method for recognizing the text-named entities in the electronic medical record based on the pre-trained language model as claimed in claim 1, wherein in step 1, all words in the original text in the data set are counted, deactivated words and useless symbols are removed, and a dictionary file is generated.

3. The method for recognizing named entities in electronic medical record texts based on a pre-training language model as claimed in claim 1 or 2, wherein in step 2, six entities of diseases, diagnosis, examination, surgery, drugs and anatomical region appearing in the original texts subjected to data preprocessing in step 1 are labeled based on SNOMED CT medical term set and using BIO labeling mode.

4. The method for recognizing the named entity of the text of the electronic medical record based on the pre-trained language model as claimed in claim 3, wherein the ERNIE pre-trained language model is used in the step 3.

5. The method for recognizing the named entities in the texts of the electronic medical records based on the pre-trained language model as claimed in claim 4, wherein in the step 3, the ERNIE pre-trained language model is trained in a semi-supervised parallel manner.

6. The method for recognizing the text-based named entity of the electronic medical record based on the pre-trained language model as claimed in claim 5, wherein in the step 4, the BilSTM-CRF sequence labeling model comprises a look-up layer, a bidirectional LSTM layer and a CRF layer.

7. The method of claim 6, wherein in step 4, the word vector of the training text is input into the BilSTM-CRF sequence labeling model and iteratively trained.

8. The method for recognizing the text-named entities in the electronic medical records based on the pre-trained language model as claimed in claim 7, wherein in step 4, the number of iterative training is determined by constructing a loss function to calculate a loss value until the loss value is smaller than a set threshold.

9. The method as claimed in claim 8, wherein in step 4, the category label sequence and labeled data of the training text are input into the loss function to calculate the loss value.

Images