CN118395987A - BERT-based landslide hazard assessment named entity identification method of multi-neural network - Google Patents

Abstract

The invention provides a method for identifying landslide hazard assessment named entities based on a BERT (binary-weighted round robin) multi-neural network, which mainly comprises the following steps: (1) English literature containing landslide hazard assessment is obtained, a Stanford CoreNLP natural language toolkit and a BIO marking method are adopted to mark the literature abstract, and the mark contains landslide occurrence positions, landslide influence factors and landslide hazard assessment method information; (2) Dividing landslide hazard assessment corpus into a training set, a testing set and a verification set according to a certain proportion, adopting a BERT model and a variant model (such as ALBERT and RoBERTa) as a pre-training language model to encode the landslide hazard assessment dataset, comparing named entity recognition effects of different models, and acquiring context semantic dependency information to generate word vectors with more representation and expression capability; (3) Inputting the obtained word vector into BiLSTM layers of learning text long-distance semantic information, and then inputting the captured semantic vector into a CRF decoding layer for decoding to obtain an optimal named entity recognition result in the landslide risk assessment field. The invention has strong effect on natural language processing through BERT and variant models thereof, and simultaneously utilizes BiLSTM and CRF fusion feature vectors to be used for landslide hazard assessment naming entity recognition tasks, thereby having stronger feature extraction capability.

Description

A named entity recognition method for landslide hazard assessment based on BERT multi-neural network

Technical Field

The present invention relates to the field of landslide risk assessment, and in particular to a named entity recognition method for landslide hazard assessment based on a BERT multi-neural network.

Background technique

Timely and accurate assessment of landslide hazard is of great significance for disaster prevention and mitigation. However, information related to landslide hazard assessment is usually scattered in a large amount of unstructured text data, such as geological exploration reports, academic papers, news reports, etc., making it difficult to efficiently use these valuable information resources.

The traditional method is to manually identify and extract key entity information related to landslides from these texts, such as location, disaster factors, etc. However, this process has problems such as low efficiency, high labor cost, and high error rate, which makes it difficult to meet actual needs. Therefore, an automated method is urgently needed to efficiently and accurately identify and extract landslide-related entity information from unstructured texts.

Named entity recognition is an important technology in the field of natural language processing. It lays the foundation for further information processing by automatically identifying and classifying entity words in text (such as names of people, places, and organizations). With the development of deep learning technology, neural network-based named entity recognition models have shown excellent performance. Among them, BERT is a language model that obtains general semantic representation capabilities through large-scale unsupervised pre-training and has been widely used in natural language processing tasks. Using BERT and its variants as pre-training models, combined with neural networks such as BiLSTM and CRF, can greatly improve the accuracy and robustness of named entity recognition.

Summary of the invention

The present invention provides a method for named entity recognition of landslide hazard assessment based on a BERT multi-neural network, aiming to solve the problem that the traditional landslide hazard assessment method relies on manual analysis of literature and data, which is time-consuming and labor-intensive, and the existing named entity recognition method is limited by the corpus and model expression capabilities when processing complex natural language texts. By combining the advantages of BERT and its variant models, the automatic extraction and annotation of key information in landslide hazard assessment literature is achieved, thereby improving the accuracy and efficiency of landslide hazard assessment, reducing the manual burden, and providing more effective technical support for landslide disaster prevention and mitigation.

In order to achieve the above object, the present invention adopts the following technical scheme:

A named entity recognition method for landslide hazard assessment based on a BERT-based multi-neural network includes the following steps:

Step 1: Corpus construction: Collect English literature abstracts containing landslide hazard assessment information, preprocess them to generate standardized structured data, and build a landslide hazard assessment field prediction database;

Step 2: Data annotation: Stanford CoreNLP natural language toolkit was used to perform word segmentation, part-of-speech tagging and syntactic analysis on the data, and then the BIO (Beginning-Inside-Outside) tagging method was used to manually annotate the named entities of landslide hazard assessment, including the location of landslides, landslide influencing factors and landslide hazard assessment methods;

Step 3: Divide the labeled data into training set, test set and validation set according to a certain ratio, and use pre-trained language models such as BERT and its variants ALBERT and RoBERTa to encode the labeled data and obtain word vector representation containing contextual semantic information;

Step 4: Input the word vector obtained in step 3 into the BiLSTM (Bi-directional Long Short-Term Memory) model to capture the long-range dependency semantic features of the input sequence;

Step 5: Considering the relationship and constraints between labels in the sequence, the annotation path of the entire input sequence is decoded based on the CRF decoding layer;

Step 6: Fine-tune the target domain corpus dataset based on the BERT multi-neural network to obtain the best named entity recognition results.

Further in the step 1, the corpus is constructed by setting the literature search keyword: Landslide Hazard Assessment based on Google Scholar, and then using Python to crawl and collect English literature abstracts in the field of landslide hazard assessment. The preprocessing includes removing repeated characters, empty characters and special symbols in the corpus, and outputting it as standardized text data to maintain high data accuracy;

Furthermore, in step 2, the preprocessed landslide hazard assessment corpus is annotated, and the specific annotation process includes:

Step 2.1: Use the Python interactive interface to annotate the data using the Stanford CoreNLP natural language toolkit: First, use sentence splitting to split the text into sentences, use tokenization to split the text into words, punctuation marks and other tokens, use part-of-speech tagging to assign the corresponding part of speech (such as noun, verb, adjective, etc.) to each token, and finally use parsing to build the syntactic analysis tree and part-of-speech structure of the text;

Step 2.2: Use the BIO (Beginning-Inside-Outside) tagging method to manually annotate the named entities of landslide hazard assessment, where B-Beginning represents the initial label of the entity, I-Inside represents the internal label of the entity, and O-Outside represents the non-entity. The domain corpus is annotated into three types of entities, including landslide location information, landslide influencing factors, and landslide hazard assessment methods;

Furthermore, the specific contents in step 3 include:

Step 3.1: Divide the labeled dataset into training set, test set and validation set in a ratio of 7:1:2;

Step 3.2: Based on the training set, test set and validation set, train the BERT-based fully supervised scenario pre-trained language model on the laboratory server and output the word vector x = (x ₁ , x ₂ , …, x _N );

Step 3.3: Based on the training set, test set and validation set, train the ALBERT-based fully supervised scenario pre-trained language model on the laboratory server and output the word vector y = (y ₁ ,y ₂ ,…,y _N );

Step 3.4: Based on the training set, test set and validation set, train a fully supervised scenario pre-trained language model based on RoBERTa on the laboratory server, and output the word vector z = (z ₁ , z ₂ , …, z _N );

Furthermore, in step 4, the word vectors x, y and z obtained in step 3 are respectively input into the BiLSTM layer, and the word vectors are feature learned from the text sequence context, and the output contains forward and backward information corresponding to each word in the text sequence, capturing the semantic features in the text sequence.

Further in step 5, the CRF layer decodes based on the output of the BiLSTM layer to solve the dependency between the label sequences and obtain the optimal label sequence, so that the output label sequence satisfies the overall optimal probability;

Furthermore, in step 6, based on the named entity recognition result obtained in step 5, the training hyperparameters (such as learning rate, batch size, number of training rounds, etc.) are adjusted to maximize the comprehensive performance of the three evaluation indicators of accuracy, recall rate and F1 value.

The beneficial effects of the technical method provided by this application are:

1. Collect unstructured data in a specific field and preprocess it to generate a structured domain corpus;

2. The present invention uses BERT, ALBERT and RoBERTa pre-trained language models to train data sets in specific fields, and obtains word vectors containing contextual semantic information by using different models to fully mine the deep semantic features of the text;

3. Based on the pre-trained model, the BiLSTM+CRF model is integrated to enhance the generalization ability of the model through powerful cross-domain migration capabilities and multi-neural network integration. Finally, the hyperparameters of the named entity recognition model are fine-tuned based on the domain corpus to achieve optimal performance in terms of accuracy, recall rate and F1 value, and obtain the best results for named entity recognition in landslide hazard assessment.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of the overall steps of the method of the present invention;

FIG2 is a flowchart of corpus construction in an embodiment of the present invention;

FIG3 is a schematic diagram of the structure of a BERT-based multi-neural network model in an embodiment of the present invention.

Detailed ways

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

This embodiment provides a method for named entity recognition of landslide hazard assessment based on a BERT-based multi-neural network, as shown in FIG1 , which mainly includes the following steps:

Step 1: Corpus construction: Collect English literature abstracts containing landslide hazard assessment information, preprocess them to generate standardized structured data, and build a landslide hazard assessment field prediction library. As shown in Figure 2, the specific detailed sub-steps are as follows:

Step 1.1: Based on Google Scholar, set the literature search keyword Landslide Hazard Assessment, and then use Python to crawl and collect English literature abstracts in the field of landslide hazard assessment;

Step 1.2: Process the repeated characters, empty characters and special symbols in the crawled target domain data, and output them as standardized text data to build a landslide hazard assessment corpus. The corpus mainly covers three types of information: landslide hazard assessment method (Method), landslide location (Location) and landslide influencing factor (Factor).

Step 2: Data annotation: First, the Stanford CoreNLP natural language toolkit is used to perform word segmentation, part-of-speech tagging, and syntactic analysis on the data, and then the BIO (Beginning-Inside-Outside) tagging method is used to manually annotate the landslide hazard assessment named entities.

Step 2.1: Use the Python interactive interface to annotate the data using the Stanford CoreNLP natural language toolkit: First, use sentence splitting to split the text into sentences, use tokenization to split the text into words, punctuation marks and other tokens, use part-of-speech tagging to assign the corresponding part of speech (such as noun, verb, adjective, etc.) to each token, and finally use parsing to build the syntactic analysis tree and part-of-speech structure of the text;

Step 2.2: Use the BIO (Beginning-Inside-Outside) tagging method to manually label the landslide hazard assessment named entities, where B-Beginning represents the initial label of the entity, I-Inside represents the internal label of the entity, and O-Outside represents the non-entity. The three types of information in the domain corpus, including landslide location information, landslide influencing factors, and landslide hazard assessment methods, are annotated as entity types:

① Landslide hazard assessment method: M

②Location of landslide: L

③ Landslide influence factor: F

After combining with the BIO labeling method, there are seven labels in the dataset: B-M, I-M, B-F, I-F, B-L, I-L, and O.

Step 3: Divide the labeled data into training set, test set and validation set according to a certain ratio, and use pre-trained language models such as BERT and its variants ALBERT and RoBERTa to encode the labeled data and obtain word vector representation containing contextual semantic information;

Step 3.1: Divide the labeled dataset into training set, test set and validation set in a ratio of 7:1:2 for model training;

Step 3.2: Input the data into the BERT model. During the text input process, BERT stores the text information of the training samples in the object InputFeatures. BERT dynamically generates word vectors x = (x ₁ , x ₂ , …, x _N ) according to the context based on two pre-training methods: Masked Language Model (MLM) and Next Sentence Prediction (NSP);

Step 3.3: ALBERT is a lightweight BERT model that is more efficient in terms of memory and computing resources. In this embodiment, the ALBERT model is used to perform natural language processing tasks. The original text sequence a = (a ₁ , a ₂ , ..., a _N ) is input into the ALBERT model, where a _N represents the Nth word in the text sequence, and the output word vector y = (y ₁ , y ₂ , ..., y _N );

Step 3.4: The RoBERTa model is an improved version of the BERT model. In this embodiment, the RoBERTa model is used to improve the training task and data generation method. The original text sequence a = (a ₁ , a ₂ , ..., a _N ) is input into the RoBERTa model for training, and the word vector z = (z ₁ , z ₂ , ..., z _N ) is output;

Step 4: Input the word vectors x, y and z obtained in step 3 into the BiLSTM (Bi-directional Long Short-Term Memory) model to capture the long-range dependency semantic features of the input sequence;

The calculation process of the internal structure of BiLSTM neurons is as follows:

Forward calculation:

① Forget Gate

f _t =σ(W _f ·[h _t-1 ,x _t ]+b _f )

②Output gate

i _t =σ(W _i ·[h _t-1 ,x _t ]+b _i )

③New memory candidates

④Update current memory

⑤Output gate

o _t =σ(W _o ·[h _t-1 ,x _t ]+b _o )

⑥Update the current hidden state

h _t = o _t *tanh(C _t )

Backward calculation:

① Forget Gate

f _t ′=σ(W _f ′·[h _t-1 ,x _t ′]+b _f ′)

②Output gate

i _t ′=σ(W _i ′·[h _t-1 ,x _t ′]+b _i ′)

③New memory candidates

④Update current memory

⑤Output gate

o _t ′＝σ(W _o ′·[h _t-1 ,x _t ′]+b _o ′)

⑥Update the current hidden state

h _t ′＝o _t ′*tanh(C _t ′)

Among them, x _t , f _t , i _t , C _t , o _t , They represent the input, forget gate, memory gate, cell state, output gate, and temporary cell state at time t respectively; W is the model parameter, b is the bias vector; σ is the sigmoid function, and tanh is the hyperbolic function.

LSTM is a forward propagation network, while BiLSTM consists of two directional LSTM networks. BiLSTM can make full use of the information before and after the text and better capture the semantic relevance of the text before and after. In contrast, in LSTM, the state information is always transmitted from front to back, and backward information encoding is not possible.

In the BiLSTM layer, the word vector sequences x, y, and z are respectively learned from the context to obtain the previous feature vectors _hx , _hy , _hz and the following feature vectors _hx ', _hy ', _hz ' for integration.

Step 5: Considering the relationship and constraints between labels in the sequence, the annotation path of the entire input sequence is decoded based on the CRF decoding layer, and the loss function is calculated;

CRF is a statistical model used to label sequence data. In the NER task, it can model the entire labeled sequence and consider the relationship between entity labels. CRF further improves the performance of NER by modeling the probability of entity sequences. Specifically, there are two types of feature functions in CRF, namely state features and transition features. The state feature is represented by the state score of the current node (a state in the possible states of a certain output position is called a node), and the transition feature is represented by the transition score from the previous node to the current node. Its loss function is defined as follows:

LossFunction＝/P _RealPath /(P ₁ +P ₂ +P ₃ +…+P _N )

The calculation of the CRF loss function requires the true path score (including state score and transfer score) and the scores of all other possible paths (including state score and transfer score).

Step 6: Fine-tune the target domain corpus dataset based on the BERT multi-neural network to obtain the best named entity recognition results.

Model integration: As shown in Figure 3, BERT, ALBERT and RoBERTa are combined with BiLSTM and CRF to form an end-to-end model. First, BERT and its variant models are used to fully extract the contextual semantic features of the landslide hazard assessment annotation text to obtain dynamic word vectors. Then, the forward LSTM network layer of the BiLSTM feature extraction layer is combined to capture the forward text information of the current word, and the backward LSTM layer captures the backward text semantic information. The CRF layer performs a normalized test on the probability score obtained by the BiLSTM network based on the transfer matrix between labels obtained by previous training. Then, Viterbi is used to calculate the global tag sequence with the highest probability of occurrence, and finally the recognition result of the entity in the input text sequence is input.

Model training and evaluation: The model is trained using labeled landslide hazard assessment training data, and the model is tuned using the validation set to avoid overfitting. In the evaluation phase, the model is evaluated using test data, and the performance of the model is evaluated using indicators such as precision (P), recall (R), and F1 value.

Among them, TP: the number of entities correctly identified when the predicted result is a positive sample and the actual situation is also a positive sample; FP: the number of entities recognized but incorrectly identified when the predicted result is a positive sample and the actual situation is a negative sample; FN: the number of entities that should be recognized but are not recognized when the predicted result is a negative sample but the actual situation is a positive sample.

In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims (7)

1. The identifying method of the landslide hazard assessment named entity based on the BERT multi-neural network is characterized by comprising the following steps:

Step 1, corpus construction: collecting English literature abstracts containing landslide hazard assessment information, preprocessing the English literature abstracts to generate normalized structured data, and constructing a prediction library in the landslide hazard assessment field;

Step 2, data marking: performing word segmentation, part-of-speech labeling and syntactic analysis on the data by adopting Stanford CoreNLP natural language toolkits, and manually labeling landslide hazard assessment named entities by using a BIO (Beginning-Inside-Outside) labeling method, wherein the manual labeling method comprises landslide occurrence positions, landslide influence factors and landslide hazard assessment methods;

Step 3, dividing the annotation data into a training set, a testing set and a verification set according to a certain proportion, and coding the annotation data by utilizing a BERT and a pre-training language model such as variants ALBERT and RoBERTa thereof to obtain word vector representation containing context semantic information;

Step 4, inputting the word vector obtained in the step 3 into BiLSTM (Bi-directional Long Short-Term Memory) model for capturing the long-range dependent semantic features of the input sequence;

Step 5, considering the relation and constraint among the labels in the sequence, decoding the labeling path of the whole input sequence based on the CRF decoding layer;

and 6, performing fine adjustment on the target domain corpus data set by using the BERT-based multi-neural network to obtain an optimal named entity recognition result.

2. The BERT-based multi-neural network landslide hazard assessment named entity recognition method of claim 1, wherein the method comprises the steps of: in step 1, corpus construction is based on google academic set-up document retrieval topics: LANDSLIDE HAZARDASSESSMENT, crawling by using Python, and collecting English literature abstracts in the landslide hazard assessment field, wherein preprocessing comprises removing repeated characters, blank characters, special symbols and the like in corpus, and outputting text data with high accuracy.

3. The BERT-based multi-neural network landslide hazard assessment named entity recognition method of claim 1, wherein the method comprises the steps of: in the step 2, labeling is performed on the preprocessed landslide hazard assessment corpus, and the specific labeling process comprises the following steps:

Step 2.1: labeling the data with Stanford CoreNLP natural language toolkits using a Python interactive interface: firstly, segmenting a text into sentences by using sentence segmentation (SENTENCE SPLITTING), segmenting the text into token(s) such as words, punctuations and the like by using segmentation words (Tokenization), distributing corresponding parts of speech (such as nouns, verbs, adjectives and the like) for each token (Part-of-SPEECH TAGGING), and finally constructing a syntactic analysis tree and a Part of speech structure of the text by using grammar analysis (Parsing);

Step 2.2: the landslide hazard assessment named entities were manually labeled using BIO (Beginning-Inside-Outside) labeling, where B-Beginning (beginning) represents the initial label of the entity, I-Inside represents the internal label of the entity, and O-Outside (outside) represents the non-entity. The domain corpus is marked as 3 types of entities, and the 3 types of entities respectively comprise landslide occurrence position information, landslide influence factors and landslide risk assessment methods.

4. The BERT-based multi-neural network landslide hazard assessment named entity recognition method of claim 1, wherein the method comprises the steps of: the specific content in the step 3 comprises:

Step 3.1: dividing the labeling data set into a training set, a testing set and a verification set according to a mode of 7:1:2;

Step 3.2: training a BERT-based full-supervision scene pre-training language model on a laboratory server based on the training set, the testing set and the verification set, and outputting a word vector x= (x ₁,x₂,…,x_N);

Step 3.3: training a ALBERT-based full-supervision scene pre-training language model on a laboratory server based on the training set, the testing set and the verification set, and outputting a word vector y= (y ₁,y₂,…,y_N);

Step 3.4: based on the training set, the testing set and the verification set, training a RoBERTa-based full-supervision scene pre-training language model on a laboratory server, and outputting a word vector z= (z ₁,z₂,…,z_N).

5. The BERT-based multi-neural network landslide hazard assessment named entity recognition method of claim 1, wherein the method comprises the steps of: in step 4, the word vectors x, y and z obtained in step 3 are respectively input into BiLSTM layers, feature learning is carried out on the word vectors from the context of the text sequence, forward and backward information corresponding to each word in the text sequence is output, and semantic features in the text sequence are captured.

6. The BERT-based multi-neural network landslide hazard assessment named entity recognition method of claim 1, wherein the method comprises the steps of: in step5, the CRF layer decodes based on the output of BiLSTM layers, solves the dependency relationship between the tag sequences, and obtains an optimal tag sequence, so that the output tag sequence meets the overall optimal probability.

7. The BERT-based multi-neural network landslide hazard assessment named entity recognition method of claim 1, wherein the method comprises the steps of: in step 6, based on the named entity recognition result obtained in step 5, training super parameters (such as learning rate, batch size, training round number, etc.) are adjusted for the target domain data set to maximize the comprehensive performance of three evaluation indexes, namely accuracy, recall and F1 value.