CN114398482A - A dictionary construction method, device, electronic device and storage medium - Google Patents

Abstract

The present application provides a dictionary construction method, device, electronic device and storage medium, and relates to the technical field of natural language processing. After obtaining the text to be processed and a basic dictionary containing multiple word categories, the method selects at least one target word whose semantic information belongs to a named entity from the text to be processed based on the trained category prediction model, and determines at least one target word. The target words respectively belong to the probability values of multiple word categories, so that at least one target word is respectively assigned to the word category whose corresponding probability value meets the set probability condition. Since the target words belonging to named entities in the text and the probability values of the target words belonging to each named entity category in the basic dictionary can be accurately determined according to the semantic information, the target words can be accurately assigned to the corresponding named entity categories. , to improve the accuracy of dictionary construction.

Description

A dictionary construction method, device, electronic device and storage medium

technical field

The embodiments of the present application relate to the technical field of natural language processing, and in particular, to a dictionary construction method, an apparatus, an electronic device, and a storage medium.

Background technique

When recognizing text information, a dictionary is usually constructed first, various categories of words are added to the dictionary, and then the constructed dictionary is used to recognize the text, so that the text contained in the text can be quickly and accurately identified. The category to which the word belongs.

In the related art, the method based on N-Gram is often used for dictionary construction. The dictionary construction method based on N-Gram is to first use the N-Gram algorithm to segment the text to obtain N words, then use multiple existing dictionaries for comparison, and select new words from the N words to add to the dictionary middle.

Since the new words extracted by this method are related to the N words obtained by the N-Gram algorithm, when the words obtained by the N-Gram algorithm are inaccurate, the new words extracted from the N words are not accurate enough. Therefore, the accuracy of the constructed dictionary is not high.

SUMMARY OF THE INVENTION

In order to solve the technical problems existing in the related art, the embodiments of the present application provide a dictionary construction method, an apparatus, an electronic device and a storage medium, which can improve the accuracy of dictionary construction.

The specific technical solutions provided by the embodiments of the present application are as follows:

A dictionary construction method that includes:

Obtain the text to be processed and a basic dictionary; wherein, the basic dictionary contains multiple word categories;

Based on the trained category prediction model, determine at least one candidate word included in the text to be processed, and the respective semantic information of the at least one candidate word;

Through the category prediction model, according to the respective semantic information of the at least one candidate word, at least one target word that meets the set semantic conditions is selected, and it is determined that the at least one target word belongs to each of the multiple word categories. probability value;

The at least one target word is respectively assigned to word categories whose corresponding probability values meet the set probability conditions.

A dictionary construction device, comprising:

an acquisition module for acquiring the text to be processed and a basic dictionary; wherein the basic dictionary contains multiple word categories;

A word recognition module, configured to determine at least one candidate word contained in the text to be processed based on the trained category prediction model, and the respective semantic information of the at least one candidate word;

The category recognition module is used to select at least one target word that meets the set semantic conditions according to the respective semantic information of the at least one candidate word through the category prediction model, and determine that the at least one target word belongs to the The probability value of describing multiple word categories;

The dictionary construction module is used for attributing the at least one target word to word categories whose corresponding probability values meet the set probability conditions, respectively.

Optionally, the category prediction model includes a pre-trained language sub-model and a named entity recognition sub-model; the word recognition module is specifically used for:

Based on the text to be processed, through the pre-trained language sub-model, a word vector corresponding to at least one word contained in the text to be processed is obtained; wherein, each word vector represents the semantic information of the corresponding word;

Based on the respective word vectors corresponding to the at least one word, the at least one word is combined by the named entity recognition sub-model to obtain at least one candidate word and the respective semantic information of the at least one candidate word; wherein, Each candidate word contains at least one word.

Optionally, the category identification module is specifically used for:

Through the named entity recognition sub-model, from the at least one candidate word, at least one target word whose semantic information belongs to a named entity is selected; the named entity is an entity name with specific semantics;

For the at least one target word, the following operations are respectively performed: using the named entity recognition sub-model, according to the semantic information of the one target word, determine a probability value that the one target word belongs to the multiple word categories respectively.

Optionally, it also includes a model training module, and the model training module is used for:

Obtaining a training data set; the training data set includes a plurality of text data samples, and the text data samples are marked with set categories;

Based on the training data set, the category prediction model is iteratively trained until a set convergence condition is met, wherein an iterative training process includes:

Based on the text data samples extracted from the training data set, through the category prediction model, at least one target word in the text data sample is determined, and the target word category corresponding to each of the at least one target word is determined;

According to the target word category and the set category, a corresponding loss value is determined, and according to the loss value, the parameters of the category prediction model are adjusted.

Optionally, the category prediction model includes a pre-trained language sub-model and a named entity recognition sub-model; the training data set includes encyclopedia text data samples, domain text data samples and named entity recognition text data samples; the model training module Also used for:

Based on the text data samples extracted from the encyclopedia text data samples and the domain text data samples, the corresponding embedding vector samples are determined through the pre-trained language sub-model; the encyclopedia text data samples are texts in untargeted fields Data, each domain text data sample is text data that includes multiple named entities of set categories, and is marked with the corresponding domain category;

Based on the vector samples extracted from the embedding vector samples and the word vector samples, the named entity recognition sub-model determines at least one corresponding target word and its corresponding target word category; the word vector samples are based on the The named entity identification text data sample is obtained by describing the named entity identification text data sample; the named entity identification text data sample is text data including at least one named entity, and each named entity word is marked with a corresponding named entity category.

Optionally, the model training module is also used for:

Determine a first loss value according to the target word category and the domain category, and adjust the parameters of the pre-trained language sub-model according to the first loss value;

According to the target word category, the domain category, and the named entity category, a second loss value is determined, and parameters of the named entity recognition sub-model are adjusted according to the second loss value.

Optionally, the model training module is also used for:

Preprocessing operations are respectively performed on the encyclopedia text data samples, the domain text data samples and the named entity recognition text data samples in the training data set; the preprocessing operations include at least one of data screening and format conversion.

Optionally, the dictionary construction module is specifically used for:

Based on the probability values that a target word belongs to the multiple word categories respectively, determine the word category corresponding to the maximum probability value, and use the word category as the target category;

If the probability value of the one target word belonging to the target category is greater than the first set threshold, the one target word is assigned to the target category.

Optionally, the dictionary construction module is further used for:

If the probability value of the target word belonging to the multiple word categories is not greater than the second set threshold, select at least one word category whose probability value is greater than the third set threshold as a candidate category; the third The set threshold is smaller than the second set threshold;

Based on the similarity between the one target word and the words contained in the at least one candidate category, a candidate category that meets the set similarity condition is selected as the target category, and the one target word is assigned to the target in the category.

An electronic device provided by an embodiment of the present application includes a processor and a memory, wherein the memory stores a program code, and when the program code is executed by the processor, the processor is made to execute any one of the above steps of a dictionary construction method.

A computer-readable storage medium provided by an embodiment of the present application includes program code, and when the program code runs on an electronic device, the program code is used to make the electronic device execute any of the above dictionary construction methods A step of.

A computer program product provided by an embodiment of the present application includes a computer program/instruction, which, when running on a computer, causes the computer to execute the above dictionary construction method.

The beneficial effects of this application are as follows:

The embodiments of the present application provide a dictionary construction method, device, electronic device, and storage medium. After the text to be processed and a basic dictionary containing multiple word categories are acquired, based on the trained category prediction model, the text to be processed can be extracted from the text to be processed. Selecting at least one target word whose semantic information belongs to a named entity, and determining the probability values of the at least one target word respectively belonging to multiple word categories, and assigning the at least one target word to the corresponding probability value that conforms to the set probability condition in the word category. Compared with the N-Gram-based dictionary construction method in the related art, because this scheme can accurately determine the target words belonging to the named entities in the text to be processed based on the category prediction model and the semantic information of the words, and according to the target words belonging to the named entity. The probability value of the named entity category in the basic dictionary determines the target named entity category to which the target word belongs, so the accuracy of the dictionary construction can be improved.

Other features and advantages of the present application will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the present application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description, claims, and drawings.

Description of drawings

FIG. 1a is a schematic diagram of an application scenario in an embodiment of the present application;

FIG. 1b is a schematic diagram of another application scenario in the embodiment of the present application;

2a is a schematic flowchart of training a category prediction model in an embodiment of the present application;

2b is a schematic diagram of annotating a named entity recognition text data sample in an embodiment of the present application;

FIG. 2c is a schematic diagram of the output of embedded vector samples in an embodiment of the present application;

2d is a schematic diagram of training a pre-trained language sub-model according to an embodiment of the application;

2e is another schematic diagram of training the pre-trained language sub-model in the embodiment of the application;

2f is a schematic diagram of training a named entity recognition sub-model in an embodiment of the present application;

3 is a schematic flowchart of analyzing and processing a training data set in the embodiment of the present application;

4a is a schematic flowchart of a dictionary construction method in an embodiment of the application;

4b is a schematic diagram of determining candidate words in an embodiment of the application;

FIG. 4c is a schematic diagram of determining the output result of the category prediction model in the embodiment of the present application;

FIG. 4d is a schematic flowchart of determining a target category in an embodiment of the present application;

FIG. 5 is another schematic flowchart of determining a target category in an embodiment of the present application;

6 is a schematic diagram of a dictionary construction method in an embodiment of the application;

FIG. 7 is a schematic structural diagram of a dictionary construction device in an embodiment of the application;

8 is a schematic structural diagram of another dictionary construction device in an embodiment of the application;

9 is a schematic structural diagram of a hardware composition of an electronic device according to an embodiment of the application;

FIG. 10 is a schematic structural diagram of a computing device in an embodiment of the present application.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are of the present application. Some embodiments of the technical solution, but not all embodiments. All other embodiments obtained by persons of ordinary skill in the art without creative work based on the embodiments recorded in the present application documents fall within the protection scope of the technical solutions of the present application.

The terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein.

Some terms in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

Dictionary: Extract predefined categories of words from the text to construct the required dictionary. When identifying a text, the constructed dictionary can be used to identify the text, which can quickly and accurately identify all the words in the text. The category to which the included word belongs.

Named entities: entity names with specific semantics in the text, mainly including person names, place names, institution names, proper nouns, etc.

As used hereinafter, the word "exemplary" means "serving as an example, embodiment, or illustration." Any embodiment described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The terms "first" and "second" in the text are only used for the purpose of description, and should not be construed as expressing or implying relative importance or implying the number of indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, the "multiple" The meaning is two or more.

The embodiments of the present application relate to artificial intelligence (Artificial Intelligence, AI) and machine learning (Machine Learning, ML) technology and natural language processing (Nature Language processing, NLP), and are designed based on the machine learning technology and natural language processing technology in artificial intelligence.

Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning/deep learning, autonomous driving, and smart transportation.

Machine Learning (ML) is a multi-field interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or realize human learning behaviors to acquire new knowledge or skills, and to reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications are in all fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, teaching learning and other technologies.

Natural Language Processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between humans and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, the language that people use on a daily basis, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

The embodiment of the present application adopts a category prediction model based on machine learning to determine at least one target word in the text, and determine the probability values that each of the at least one target word belongs to multiple word categories in the basic dictionary.

The design ideas of the embodiments of the present application are briefly introduced below:

Dictionaries can be used for informational identification of text to quickly and accurately determine the predefined categories to which words in the text belong. In the related art, the method based on N-Gram is often used for dictionary construction. This method firstly divides the text based on the N-Gram algorithm to obtain N words, and then uses multiple existing dictionaries to compare, from the N words. to select new words to add to the dictionary. However, the N-Gram algorithm used in this method cannot accurately segment the words in the text, resulting in a low accuracy of the final constructed dictionary.

In view of this, the embodiments of the present application provide a dictionary construction method, device, electronic device, and storage medium, which can first select semantic information from the text to be processed based on the semantic information of the words in the text based on the trained category prediction model. The target word belonging to the named entity, and then according to the probability value of the target word belonging to each named entity category in the dictionary, the target word is assigned to the target named entity category in the dictionary to complete the construction of the dictionary, which can improve the accuracy of dictionary construction. Rate.

The preferred embodiments of the present application will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not intended to limit the present application. The embodiments and features of the embodiments can be combined with each other.

Referring to FIG. 1a, it is a schematic diagram of an application scenario in the embodiment of the present application. The application scenario diagram includes the terminal device 100 and the server 200 . Communication between the terminal device 100 and the server 200 may be performed through a communication network. Alternatively, the communication network may be a wired network or a wireless network. The terminal device 100 and the server 200 may be directly or indirectly connected through wired or wireless communication, which is not limited in this application.

In this embodiment of the present application, the terminal device 100 is an electronic device used by a user, and the electronic device may be a personal computer, a mobile phone, a tablet computer, a notebook, an e-book reader, a smart home, a vehicle terminal, and other devices. The server 200 may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or may provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, Cloud servers for basic cloud computing services such as middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

Exemplarily, when the user browses text on the terminal device 100, the terminal device 100 may send the browsed text to the server 200, and the server 200 may determine at least one candidate word contained in the text based on the trained category prediction model. , and the respective semantic information of at least one candidate word, and through the category prediction model, according to the respective semantic information of at least one candidate word, select at least one target word that meets the set semantic conditions, and determine that at least one target word belongs to the basic Probability values for multiple word categories in the dictionary. After the probability value is determined, for at least one target word, the following operations may be performed respectively: based on the probability values of a target word belonging to multiple word categories, select a word category that meets the set probability conditions as the corresponding target category, And a target word is assigned to the target category, so as to complete the construction and expansion of the dictionary.

In addition, after determining the target word contained in the text and the word category to which the target word belongs, the server 200 may also send the result to the terminal device 100, so that the terminal 100 can show the user that the word contained in the text belongs to category, to achieve the purpose of text information recognition.

It should be noted that FIG. 1a is an example introduction of the application scenario of the dictionary construction method of the present application, and the actual application scenario to which the method in the embodiment of the present application can be applied is not limited to this.

In some embodiments, the application scenarios in the embodiments of the present application may also be shown in FIG. 1 b , including the terminal device 100 and the server 200 . The server 200 includes a dictionary, and the dictionary is a dictionary constructed by the dictionary construction method in this application.

Specifically, the terminal device 100 can send the text to be recognized to the server 200 , and after receiving the text to be recognized, the dictionary in the server 200 can recognize the named entity in the text to be recognized, and send the obtained recognition result to the terminal device 100 .

For example, if the text to be recognized is "Xiaohua climbs Mount Tai at 5:00 in the morning to watch the sunrise", after the terminal device 100 sends the text to be recognized to the server 200, it can recognize the text to be recognized through the dictionary in the server 200. The name of the person in the text to be recognized is "Xiaohua", the festival is "Mid-Autumn Festival", the time is "5 am", and the place name is "Taishan", that is, "person name: Xiaohua, festival: Mid-Autumn Festival, time: 5 am, place name : Mount Tai". After obtaining the identification result, the server 200 may send the identification result to the terminal device 100 .

Further, after receiving the recognition result, the terminal device 100 can analyze and manage the text to be recognized according to the named entity in the recognition result. For example, assuming that the text to be identified is a news message, the named entity in the news message can be identified through the dictionary, so as to extract the important information in the news message, and analyze the news message as a fake according to the extracted important information. news, or a news message that contains sensitive words, etc.

First, the training process of the category prediction model in the embodiment of the present application is described in detail, and the process may be executed by a server, such as the server 200 in FIG. 1a. Referring to Fig. 2a, which is a schematic diagram of the training process of the category prediction model in the embodiment of the present application, the following describes the training process of the category prediction model in the embodiment of the present application in detail with reference to Fig. 2a.

Step S201, acquiring a training data set.

The obtained training data set may include multiple text data samples. Specifically, the training data set may include encyclopedia text data samples, domain text data samples and named entity recognition text data samples.

Among them, the encyclopedia text data samples are text data in untargeted fields. For example, the text "Shippingcompany Evergreen Marine leased container ships longer grant round in the Egyptian Suez Canal stranded re-float rescue 6days, the canal restoretraffic." and the text "Hong Kong veteran actor Liao Qizhi, who won the Hong Kong FilmAwards for Best Supporting Actor, died in Prince of Wales Hospital at the age of 66.” etc. can be encyclopedia text data. When using the encyclopedia text data as sample data to train the category prediction model, it is not necessary to label each encyclopedia text data sample.

Domain text data samples include multiple named entities of a given category, that is, text data that are highly related to a specific named entity, or contain many named entities. For example, the text "After the Danish government sold the Caribbean colony of the West Indies to the United States, the United Statesbegan to control the U.S.Virgin Islands." can be a domain text data, and the text is a location domain text data. When using domain text data as sample data to train the category prediction model, it is necessary to label the corresponding domain category for each domain text data sample. For example, for the above-mentioned domain text data, since the text data contains many location named entities, the text data can be marked as "Location".

Named entity recognition text data samples include at least one named entity, that is, text data that needs to be labeled with secondary named entities. For example, the text "A cruise ship carrying tourists on Qiandao Lake inZhejiang Province in southeastern China." can identify text data for a named entity. When using named entity recognition text data as sample data to train the category prediction model, it is necessary to label each named entity word included in each named entity recognition text data with the corresponding named entity category. For each named entity recognition text data sample, the "BIOES labeling method" can be used to mark the named entity category and relative position of each word included in the named entity recognition text data sample. For example, as shown in Figure 2b, for the named entity recognition text data "A ship carrying tourists on Qiandao Lake in Zhejiang Province", the named entity recognition text data is marked with the "BIOES labeling method", and the "O O O S-LOC" can be obtained. O O B-LOC E-LOCO". Among them, B represents the start position, I represents the middle position, E represents the end position, S represents a single named entity, followed by a label to represent the entity category, for example, "-LOC" represents the named entity belongs to the location named entity, O represents the non-named entity .

Step S202, extracting text data samples from the encyclopedia text data samples and the domain text data samples.

The category prediction model to be trained includes a pre-trained language sub-model and a named entity recognition sub-model. When training the pre-trained language sub-model, text can be extracted from the encyclopedia text data samples and domain text data samples in the training dataset. The data samples are used as training sample data.

Step S203: Input the extracted text data samples into the pre-trained language sub-model to be trained to obtain corresponding embedding vector samples.

Input the extracted text data samples into the pre-trained language sub-model to be trained, and perform feature extraction on the text data samples based on the pre-trained language sub-model to obtain corresponding embedding vector samples, which may include corresponding text data samples. The word vector corresponding to each word in . For example, for a text data sample "After the Danish government sold the Caribbean colony", after pre-training the language sub-model, the embedding vector samples shown in Figure 2c can be obtained, namely E1, E2, E3,...E8.

Specifically, as shown in Figure 2d, before inputting the extracted text data samples into the pre-trained language sub-model to be trained, the text data samples also need to be processed into [CLS] + word segmentation + [SEP] Structure, that is, processing text data samples into "[CLS], I1, I2, I3, ..., [SEP]" structures. Among them, the special label [CLS] can be used for the recognition task of domain categories.

After the text data samples are processed into "[CLS], I1, I2, I3, ..., [SEP]" structures, random Mask processing can also be performed on the text data samples, and the text data samples can be processed into "[CLS], I1 , [MASK], I3, …, [SEP]” structure. The random Mask mechanism is used to process the text data samples input to the pre-trained language sub-model, so that the pre-trained language sub-model can complete the task of Mask prediction. Among them, the random mask mechanism is to randomly cover or replace any word or word in a sentence, and then let the pre-trained language sub-model predict the covered or replaced part through contextual understanding.

After performing random Mask processing on the text data samples, the pre-trained language sub-model can be trained by using the processed text data samples. First, embed the text data sample, that is, vectorize, to obtain "E[CLS], E1, E[MASK], E3, ..., E[SEP]", and then let the vectorized text data sample undergo pre-processing Train multiple hidden layers in the language sub-model to get the output "C, T1, T[MASK], T3, ..., T[SEP]". Since the text data samples input into the pre-trained language sub-model also include domain text data samples marked with domain categories, therefore, through the training of the pre-trained language sub-model, the pre-trained language sub-model can be used for the input text data. The domain category of the sample is predicted. At the same time, the two tasks of random mask mechanism and domain category recognition are used to train the pre-trained language sub-model, which can allow the pre-trained language sub-model to learn more semantic information.

For example, a sample of extracted text data could be "Shipping company Green Marine leasedcontainer ships longer grant round in the Egyptian Suez Canal stranded re-float rescue 6days, the canal restore traffic.". As shown in Figure 2e, this text data sample can be processed into "[CLS] Shipping companyGreen Marine leased container ships longer grant round in the Egyptian SuezCanal stranded re-float before feeding it into the pretrained language submodel. rescue 6days, the canal restore traffic.[SEP]", and perform random mask processing on the text data sample to get "[CLS]Shipping[MASK]Green[MASK] leasedcontainer...[SEP]".

After inputting the text data sample after random mask processing into the pre-training language sub-model, the pre-training sub-model can vectorize the text data sample to obtain "E[CLS], E1, E2, E[MASK], E4 , E[MASK], E6, E[…], E[SEP]”, and then get the output result “C, T1, T2, T[MASK], T4, T[MASK], T6, T[…], T [SEP]".

The extracted text data samples are subjected to random mask processing, and then input into the pre-trained language sub-model for training, so that the pre-trained language sub-model can predict the covered or replaced part through contextual understanding, that is, for the input "[[ CLS]Shipping[MASK]Green[MASK]leased container...[SEP]", the pre-trained language sub-model can predict that the first "[MASK]" is "company" and the second "[MASK] " for "Marine".

Optionally, the pre-trained language sub-model in this embodiment of the present application may be implemented by using a Transformer-based bidirectional encoder representation (Bidirectional Encoder Representations from Transformers, BERT).

In the embodiment of the present application, the encyclopedia text data and the domain text data marked with the domain category are used as training data to train the pre-trained language sub-model, so that the pre-trained language sub-model can learn more semantic information, so that it can learn more semantic information. Therefore, when performing named entity recognition on text data, the semantic information in the text data can be used to recognize the target words belonging to the named entities in the text data faster and better.

Step S204, extracting vector samples from the embedding vector samples and the word vector samples obtained based on the named entity recognition text data samples.

In this embodiment, the named entity recognition text data samples of the training data set may be vectorized to obtain corresponding word vector samples. When vectorizing the named entity recognition text data samples, the WordEmbedding method can be used to obtain the word vector samples corresponding to the named entity recognition text data samples. The embodiment of the present application does not make any method for obtaining the word vector samples from the named entity recognition text data samples. limited.

After obtaining the word vector samples corresponding to the named entity recognition text data samples, the vector samples can be extracted from the embedding vector samples and the word vector samples obtained based on the pre-trained language sub-model.

Step S205: Input the extracted vector samples into the named entity recognition sub-model to obtain at least one corresponding target word and its corresponding target word category.

Input the extracted vector samples into the named entity recognition sub-model. Based on the named entity recognition sub-model, at least one target word belonging to the named entity in the text data sample can be identified, and the named entity category corresponding to each target word.

Specifically, as shown in Figure 2f, it is assumed that the extracted text data sample is "[CLS] Shipping company Green Marine leased container ships longer...[SEP]", and the text data sample has the label "[CLS] O O B-ORG E -ORG O O O O…[SEP]”. Among them, O represents the non-named entity, B represents the start position of the named entity, E represents the end position of the named entity, S represents the single named entity, the following label represents the entity category, "-LOC" represents the location entity category, "-ORG" ” indicates the organization entity category.

Input the labeled text data samples into the pre-training language sub-model, and the corresponding embedding vector samples "E[CLS], E1, E2, E3, E4, E5, E6, E7, E[CLS], E1, E2, E3, E4, E5, E6, E7, E8, E…, E[SEP]”.

The obtained embedding vector samples are input into the named entity recognition sub-model to train the model. The named entity recognition sub-model includes two sub-models, one is the border recognition sub-model (Border Segment Model), which is used to identify the named entity boundary in the text data, and the other is the category recognition sub-model (Named EntityRecognition Model, NER Model) , which is used to identify the named entity category to which named entity words in text data belong. After training the two sub-models included in the named entity recognition sub-model, the prediction results of the two sub-models are combined to obtain the final prediction result.

After inputting the embedded vector samples "E[CLS], E1, E2, E3, E4, E5, E6, E7, E8, E..., E[SEP]" into the named entity recognition sub-model, based on the named entity recognition sub-model The boundary recognition sub-model of , can identify the named entity boundary in the text data sample, that is, through the boundary recognition sub-model, the text “[CLS] Shipping company GreenMarine leased container ships longer…[SEP]” can be recognized, and the result is “O O B E O O OO...", based on the identified named entity boundaries and embedded vector samples, through the category identification submodel in the named entity identification submodel, the named entity category to which each named entity word belongs can be obtained, that is, through the category identification submodel, the The text "[CLS]Shipping company Green Marine leased container ships longer...[SEP]" can be identified, and one can get "O O ORG ORG O O O O...O". Then, combining the recognition results of the boundary recognition sub-model with the recognition results of the category recognition sub-model, the prediction result of the text "[CLS] Shipping company GreenMarine leased container ships longer...[SEP]" can be obtained, that is, the text "[ CLS]Shipping company Green Marine leased container ships longer…[SEP]” and the named entity class to which the named entity term belongs.

Based on the named entity recognition sub-model, the text "[CLS]Shipping company Green Marineleased container ships longer…[SEP]" is recognized, and the predicted result is "[CLS]O O B-ORGE-ORG O O O O...[SEP]", According to the prediction results, we can know that "Shipping", "company", "leased", "container", "ships" and "longer" in the text are all non-named entities, "Green Marine" is a named entity, and "Green Marine" is a named entity. ” names the entity for the organization.

In the embodiment of the present application, the embedding vector output by the pre-trained language sub-model and the word vector obtained based on the named entity recognition text sample with the named entity category label are used as training data to train the named entity recognition sub-model. The overall semantic information of multiple words in the text data learned in the pre-training language sub-model stage can more accurately identify the target words belonging to named entities in the text data.

Encyclopedia text data and domain text data with domain category annotations are used as training data to train the pre-trained language sub-model, so that the pre-trained language sub-model can learn more semantic information, which can make it possible to name the text data. During entity recognition, the semantic information in the text data can be used to identify the target words belonging to the named entities in the text data faster and better.

Step S206: Determine the first loss value according to the target word category and the field category.

The first loss value can be determined according to the target word category corresponding to the target word identified by the named entity recognition sub-model and the domain category marked by the text data sample where the target word is located. Typically, the loss value is a measure of how close the actual output is to the desired output. The smaller the loss value, the closer the actual output is to the desired output.

In step S207, it is determined whether the first loss value converges to the preset target value; if not, step S208 is performed; if yes, step S209 is performed.

Judging whether the first loss value converges to the preset target value, if the first loss value is less than or equal to the preset target value, or the variation range of the first loss value obtained from N consecutive trainings is less than or equal to the preset target value value, it is considered that the first loss value has converged to the preset target value, indicating that the first loss value has converged; otherwise, it indicates that the first loss value has not converged.

Step S208: Adjust the parameters of the pre-trained language sub-model to be trained according to the determined first loss value.

If the first loss value does not converge, then adjust the model parameters of the pre-trained language sub-model, and after adjusting the model parameters, return to step S202 to continue the next round of training.

Step S209: Determine the second loss value according to the target word category, domain category, and named entity category.

The second loss value can be determined according to the target word category corresponding to the target word identified by the named entity recognition sub-model, and the domain category or named entity category marked by the text data sample where the target word is located. Typically, the loss value is a measure of how close the actual output is to the desired output. The smaller the loss value, the closer the actual output is to the desired output.

In the embodiment of the present application, the target word category corresponding to the target word output by the named entity recognition sub-model and the domain category marked in the extracted text data sample are used, and the determined first loss value is used for the model parameters of the pre-trained language sub-model. Make adjustments, and the target word category corresponding to the target word output by the named entity recognition sub-model, and the domain category or named entity category marked in the extracted vector samples, and the determined second loss value is the model parameter of the named entity recognition sub-model. For adjustment, the training process of the pre-trained language sub-model can be completed first, and after the pre-trained language sub-model finishes training, the weight of the pre-trained sub-model will no longer participate in the training of the named entity recognition sub-model, so as to reduce the training of the model. time, and can make the named entity recognition sub-model better combine the two sub-tasks of boundary recognition and category prediction, more accurately identify the target words belonging to named entities in the text, and accurately attribute the target words to the corresponding named entity category. middle.

In step S210, it is determined whether the second loss value converges to the preset target value; if not, step S211 is performed; if yes, step S212 is performed.

Judging whether the second loss value converges to the preset target value, if the second loss value is less than or equal to the preset target value, or the variation range of the second loss value obtained from N consecutive trainings is less than or equal to the preset target value value, it is considered that the second loss value has converged to the preset target value, indicating that the second loss value has converged; otherwise, it indicates that the second loss value has not converged.

Step S211: Adjust the parameters of the named entity recognition sub-model to be trained according to the determined second loss value.

If the second loss value does not converge, then adjust the model parameters of the named entity recognition sub-model, and after adjusting the model parameters, return to step S204 to continue the next round of training process.

Step S212, end the training to obtain the trained category prediction model.

If both the first loss value and the second loss value converge, the currently obtained category prediction model is used as the trained category prediction model.

In the embodiment of the present application, a training data set including encyclopedia text data samples, domain text data samples with domain category annotations, and named entity recognition text data samples with named entity category annotations is used to analyze the predictions in the category prediction model. Training the language sub-model and the named entity recognition sub-model allows the model to learn more semantic information in the text data, and based on the semantic information in the text data, more accurately identify the named entities in the text data, and Named entities are divided into corresponding named entity categories. In addition, after the training of the category prediction model is completed, a basic dictionary can be constructed according to the existing training data set, thereby completing the construction of the dictionary.

After obtaining the trained category prediction model, based on the training data set including encyclopedia text data samples, domain text data samples and named entity recognition text data samples, a basic dictionary can be constructed, which contains multiple named entity categories , each named entity category can contain multiple named entity terms.

Specifically, after obtaining the embedding vector samples corresponding to the text data samples through the pre-training language sub-model, the embedding vector samples are input into the named entity recognition sub-model, and in the process of training the model, the naming of the named entity recognition sub-model The entity recognition task mainly includes two sub-tasks, one is the identification of named entity boundaries, and the other is the identification of named entity categories. Using the BIO/BIOES tagging method can simultaneously tag the boundary and category for a word. Based on this tagging method, the previous NER model treats the NER problem as a multi-label classification task and directly predicts the named entity location and category of each word. This processing method combines the two subtasks of boundary recognition and category prediction well, but there are still two main problems. First, this type of model performs well when there are few named entity categories, but it is difficult to accurately predict the entity category of each word when there are many categories; second, only the semantics of a single word can be used in the model recognition process. Information, for named entities composed of multiple words, the semantic information of the whole of multiple words cannot be used.

The named entity recognition sub-model in the embodiment of the present application is also trained on two sub-tasks, but the combination method is different from the previous model. First, use the boundary annotation information to train the model to identify the boundaries of named entities, and divide the text into different semantic units according to the prediction results. Words in the same unit are regarded as belonging to the same named entity, and their word vectors are averaged and then assigned to each words, so as to utilize the semantic information of the named entity as a whole.

The training of the pre-trained language sub-model and the named entity recognition sub-model can be implemented in two ways. One is the fine-tune method, which unifies all the parameters of the pre-trained language sub-model and the named entity recognition sub-model. Adjustment; the other is the feature-base method, that is, the output of the pre-trained language sub-model is directly used as the embedding vector, as the input of the named entity recognition sub-model, and the parameters of the pre-trained language sub-model are fixed, and only the training name is required. Entity Recognition Submodel. The training method adopted in the embodiment of the present application is the feature-base method. After the training process of the pre-trained language sub-model is completed, the weight of the pre-trained language sub-model will no longer participate in the training, which reduces the model training time and enables the naming of The entity recognition sub-model better combines the two sub-tasks of boundary recognition and class prediction.

In one embodiment, after the training data set is acquired, multiple text data samples in the training data set may also be analyzed and processed. FIG. 3 shows a specific process of analyzing and processing training data in the training data set in the embodiment of the present application. As shown in FIG. 3 , the process may include the following steps:

Step S301, acquiring a training data set including encyclopedia text data, domain text data and named entity recognition text data.

Among them, the encyclopedia text data is the text data of untargeted fields, the field text data is the selected text data that is highly related to a specific named entity category, or contains more named entities of a certain category, and the named entity recognition text data is the word that needs to be identified. Text data for level named entity annotations.

Step S302, preprocessing operations are respectively performed on the encyclopedia text data, the domain text data and the named entity recognition text data.

After the encyclopedia text data, the domain text data and the named entity recognition text data are obtained, the encyclopedia text data, the domain text data and the named entity recognition text data can be preprocessed respectively. Wherein, the preprocessing operation may include at least one of data filtering and format conversion. Specifically, after obtaining the encyclopedia text data samples, the domain text data samples and the named entity recognition text data samples, data cleaning, denoising and serialization operations can be performed on each text data sample respectively. Among them, data cleaning can include missing value cleaning, format content cleaning, logic error cleaning, non-required data cleaning and correlation verification; data denoising can include outlier filling, distance-based detection, etc.; data serialization is to convert data into standard Format.

Step S303, obtain the corresponding encyclopedia text sequence, domain text sequence and named entity recognition text sequence.

After preprocessing the encyclopedia text data, the domain text data and the named entity recognition text data respectively, the encyclopedia text sequence, the domain text sequence and the named entity recognition text sequence can be obtained accordingly.

In the embodiment of the present application, by performing a data cleaning operation on the text data in the training data set, data that meets the data quality requirements can be obtained, and then the text data can be de-noised and serialized to convert the text data into a standard format , so that the category prediction model can be trained based on the standard format text data obtained after processing, which can improve the training efficiency of the model and the training results.

After the trained category prediction model is obtained, dictionary construction can be performed based on the category prediction model. Fig. 4a shows a flowchart of a dictionary construction method provided by an embodiment of the present application, and the method can be executed by a server, for example, the server 200 in Fig. 1a. The dictionary construction process in the embodiment of the present application will be described in detail below with reference to FIG. 4a.

Step S40, acquiring the text to be processed and the basic dictionary.

The basic dictionary is a dictionary including a plurality of named entity categories obtained during the training phase of training the category prediction model. Moreover, each named entity category of the basic dictionary contains a plurality of corresponding named entity words. For example, the basic dictionary may include multiple named entity categories such as person names, place names, and organization names. A person name may include multiple names, a place name may include multiple locations, and an organization name may include multiple organizations.

In an embodiment, after acquiring the text to be processed, data cleaning, denoising and serialization operations may be performed on the text to be processed to obtain a text sequence corresponding to the text to be processed.

Step S41 , based on the trained category prediction model, determine at least one candidate word included in the text to be processed, and the respective semantic information of the at least one candidate word.

The category prediction model may include a pre-trained language sub-model and a named entity recognition sub-model.

Inputting the text to be processed into the category prediction model, and based on the pre-trained language sub-model in the category prediction model, word vectors corresponding to at least one word contained in the text to be processed can be obtained. Among them, each word vector can represent the semantic information of the corresponding word.

After obtaining the word vector corresponding to the at least one word, based on the named entity recognition sub-model in the category prediction model, the at least one word can be combined to obtain at least one candidate word and the respective semantic information of the at least one candidate word. Wherein, each candidate word may contain at least one word.

For example, as shown in Figure 4b, the to-be-processed text is "Xiao Ming goes to Hangzhou to play". Before inputting the to-be-processed text into the category prediction model, the to-be-processed text needs to be processed into "[CLS] Xiao Ming goes to Hangzhou to play [SEP] ]" structure, and then input "[CLS] Xiaoming to Hangzhou to play [SEP]" into the category prediction model, based on the pre-trained language sub-model, we can get "E[CLS]E1 E2 E3 E4 E5 E6 E7 E[SEP] ". Based on the obtained "E[CLS]E1 E2 E3 E4 E5 E6 E7 E[SEP]", the submodel can be identified by the named entity, and the obtained "E[CLS]E1+E2 E1+E2 E3 E4+E5 E4+E5 E6 E7E [SEP]".

In the embodiment of the present application, based on the pre-trained language sub-model and the named entity recognition sub-model, at least one candidate word contained in the text to be processed and the respective semantic information of the at least one candidate word can be obtained, so that the semantic information of the text to be processed can be obtained according to the semantics of the text to be processed. The information divides the text to be processed into multiple candidate words, and then when performing named entity recognition, the semantic information of the multiple words in the text as a whole can be used to accurately identify the target words belonging to the named entity in the text to be processed.

Step S42 , select at least one target word that meets the set semantic condition according to the respective semantic information of the at least one candidate word through the category prediction model, and determine the probability values that each of the at least one target word belongs to multiple word categories.

After determining at least one candidate word contained in the text to be processed and the respective semantic information of the at least one candidate word, the named entity recognition sub-model can select at least one candidate word whose semantic information belongs to the named entity from the at least one candidate word target word. Among them, named entities are entity names with specific semantics.

For at least one target word, the following operations may be respectively performed: by using the named entity recognition sub-model, according to the semantic information of a target word, determine a probability value that a target word belongs to multiple word categories respectively.

For example, as shown in Figure 4c, it is assumed that the basic dictionary includes three named entity categories: person name, organization name and place name. According to the to-be-processed text "Xiao Ming went to Hangzhou to play", the corresponding "E[CLS] E1+E2 E1 After +E2 E3 E4+E5E4+E5 E6 E7 E[SEP]”, it can be determined from the semantic information that the words corresponding to “E1+E2” and “E4+E5” are the target words whose semantic information belongs to named entities. Predict the probability that "E1+E2" and "E4+E5" belong to person's name, organization name and place name respectively, and get the probability values P1, P2, P3 of "E1+E2" belonging to person's name, organization name and place name respectively, " E4+E5” belong to the probability values P4, P5, and P6 of the names of persons, organizations and places, respectively.

In the embodiment of the present application, based on the named entity recognition sub-model, the target words belonging to the named entities in the text to be processed can be identified, and the probability values of the target words belonging to each named entity category in the basic dictionary can be obtained, so that the target words can be accurately identified. Named entities in the text to be processed and get the probabilities of the named entities relative to each named entity class.

Step S43, at least one target word is assigned to word categories whose corresponding probability values meet the set probability conditions.

After the probability values of at least one target word belonging to multiple word categories are determined, for each target word, based on the probability values of the target word belonging to multiple word categories, a word category that meets the set probability conditions can be selected, As the target category to which the target word belongs, and add the target word to the target category of the basic dictionary. For example, after it is determined that the named entity "Xiao Ming" in the pending text "Xiao Ming goes to Hangzhou to play" belongs to the probability values P1, P2 and P3 of the person's name, the organization name and the place name, respectively, and "Hangzhou" belongs to the person's name, the organization name and the place name respectively. After the probability values P4, P5, and P6, assuming that the probability value P1 and the probability value P6 meet the set probability conditions, the person name can be used as the named entity category of the named entity "Xiao Ming", and the place name can be used as the named entity category of the named entity "Hangzhou". , and add "Xiao Ming" to the names of people in the basic dictionary, and add "Hangzhou" to the place names of the basic dictionary.

In the above step S43, for a target word in the at least one target word, the process of determining the target category to which the target word belongs, and attributing the target word to the target category of the basic dictionary can be as shown in Figure 4d, including the following steps: :

Step S431 , based on the probability values that the target words belong to multiple word categories respectively, determine the word category corresponding to the maximum probability value, and use the word category as the target category.

The target words are words belonging to named entities contained in the text to be processed output by the category prediction model, and the word categories are named entity categories contained in the basic dictionary. After inputting the text to be processed into the category prediction model, the category prediction model can output the prediction result of at least one target word that belongs to the named entity contained in the text to be processed, and the prediction result is the probability value of each target word belonging to each named entity category. , forming (category, probability) outcome pairs.

After determining the probability values of each target word belonging to each named entity category, take one target word in each target word as an example to describe in detail how to assign each target word to the corresponding named entity category in the basic dictionary middle.

Arrange the probability values of the target words belonging to each named entity category in descending order, that is, sort the probability values of the target words belonging to each named entity category in descending order, determine the maximum probability value among them, and assign the maximum probability value. The named entity category corresponding to the value is used as the target category.

Step S432, if the probability value of the target word belonging to the target category is greater than the first set threshold, the target word is assigned to the target category.

If the maximum probability value is greater than the first set threshold, the target word can be assigned to the target category, that is, the target word can be assigned to the target category in the basic dictionary, thereby completing the dictionary expansion.

For example, if the target word is "Hangzhou", the named entity categories included in the basic dictionary are human names, organization names, and place names. The probability value of the target word "Hangzhou" belonging to a person's name is 0.08, and the probability value of the target word "Hangzhou" belonging to an organization name is 0.08. is 0.12, the probability value of the target word "Hangzhou" belonging to a place name is 0.98, and the first set threshold is 0.8. The probability values of the target words belonging to the person's name, the organization name and the place name are sorted in descending order, and the maximum probability value can be determined to be 0.98, and the named entity category corresponding to the maximum probability value is the place name. Since the maximum probability value of 0.98 is greater than the first set threshold 0.8, the target word "Hangzhou" can be attributed to the place names of the basic dictionary.

In this embodiment of the present application, after obtaining the probability values that the target words belong to each named entity category, based on the fact that the maximum probability value exceeds the set threshold, the target word can be assigned to the named entity category corresponding to the maximum probability value, and the completion of Dictionary extension. Therefore, the target words belonging to the named entities determined in the text can be classified into the named entity category with the highest probability value, and the accuracy of the dictionary construction can be improved.

Step S433 , if the probability value of the target word belonging to multiple word categories is not greater than the second preset threshold, select at least one word category whose probability value is greater than the third preset threshold as a candidate category.

Wherein, the third set threshold is smaller than the second set threshold, and the second set threshold may be equal to or not equal to the first set threshold.

And when the second set threshold is equal to the first set threshold, it is equivalent to selecting a probability value greater than the first set threshold if the maximum probability value among the probability values of the target words belonging to multiple word categories is not greater than the first set threshold. 3. Set at least one word category of the threshold as a candidate category.

When the probability values of the target words belonging to multiple word categories are not greater than the second set threshold, the probability values of the target words belonging to each named entity category may be sorted in descending order, and at least one of the probability values greater than the third set threshold is selected. A named entity category, as a candidate category.

Step S434 , based on the similarity between the target word and the words contained in at least one candidate category, select a candidate category that meets the set similarity condition as the target category, and assign the target word to the target category.

After selecting at least one candidate category, the similarity between the target word and the words contained in at least the candidate category can be determined, and from the similarity, the candidate category with the highest similarity is selected as the target category, and the target Words are assigned to the target category of the base dictionary.

In the embodiment of the present application, after determining the probability values that the target words belong to each named entity category, a plurality of candidate categories can also be selected from the named entity categories according to the probability values, and the target words and each candidate category The included words are compared for similarity, and then the target category is determined according to the similarity, and then the target words are assigned to the target category to complete the dictionary expansion. Therefore, based on the similarity judgment, the determined target words belonging to the named entities in the text can be classified into the target categories corresponding to the words with the highest similarity to the target words, so as to improve the accuracy of dictionary construction.

In one embodiment, for each candidate category, the WordNet dictionary tool can be used to determine the similarity between the target word and the existing word set contained in the candidate category, and after determining that the target word is respectively associated with each candidate category After the similarity of the word set has been obtained, the candidate category with the highest similarity is selected from each similarity as the target category, and the target word is assigned to the target category of the basic dictionary to complete the dictionary expansion.

For example, the target word is "New York", and the named entity categories included in the basic dictionary are person's name, organization name, place name, holiday, product name and time. The target word "New York" belongs to the person's name with a probability value of 0.45. The probability value of belonging to the organization name is 0.67, the probability value of the target word "New York" belonging to the place name is 0.8, the probability value of the target word "New York" belonging to the festival is 0.32, the probability value of the target word "New York" belonging to the product name is 0.76, The probability value of the target word "New York" belonging to a person's name is 0.06, the second set threshold is 0.9, and the third set threshold is 0.5. Since the probability value of the target word "New York" belonging to the person's name, organization name, place name, festival, product name and time respectively is not greater than the second set threshold of 0.9, it is possible to select a plurality of probability values greater than the third set threshold of 0.5. Named entity category, as a candidate category, that is, the selected candidate categories are organization name, place name and product name. Determine the similarity between the target word "New York" and the existing word set contained in the organization name, place name and product name, assuming that the similarity between the target word "New York" and the existing word set contained in the organization name is 0.06, The similarity with the existing word set contained in the place name is 0.96, and the similarity with the existing word set contained in the product name is 0.21, then the place name can be used as the target category corresponding to the target word "New York", and the target word "New York" belongs to the place-names of the base dictionary.

To sum up, according to the probability values of a target word belonging to each named entity category, the detailed process of attributing a target word to the corresponding target category in the basic dictionary can also be shown in Figure 5, including the following steps:

Step S501: Determine the probability values that the target words belong to each named entity category respectively.

Step S502: Arrange the probability values in descending order, and determine the maximum probability value.

Step S503, determine whether the maximum probability value is greater than the first set threshold; if not, go to step S504; if so, go to step S506.

Step S504, at least one named entity category with a probability value greater than the third preset threshold is selected as a candidate category.

Step S505, according to the similarity between the target word and the words contained in at least one candidate category, select the candidate category with the highest similarity as the target category, and assign the target word to the target category.

Step S506, take the named entity category corresponding to the maximum probability value as the target category, and assign the target word to the target category.

The dictionary construction method provided by the embodiment of the present application can perform a class prediction model based on a training data set including encyclopedia text data, domain text data with domain category annotations, and named entity recognition text samples with named entity category annotations. Train, get the trained category prediction model, and build a basic dictionary including multiple named entity categories based on the training data set. After the trained category prediction model is obtained, based on the category prediction model, the target words belonging to the named entities in the text to be processed are identified, and the probability values that the target words belong to each named entity category in the basic dictionary respectively. After obtaining the probability value, based on the maximum probability value greater than the set threshold, the target word is assigned to the named entity category corresponding to the maximum probability value to complete the expansion of the dictionary, or if the maximum probability value is not greater than the set threshold value, according to the probability Select multiple named entity categories from the value as candidate categories, and determine the similarity between the target word and the words contained in each candidate category. Based on the similarity, the target word is assigned to the named entity category corresponding to the highest similarity, and the dictionary is completed. extension. Therefore, it effectively avoids the complex rules designed by domain experts or a large number of manual annotation work, effectively utilizes massive natural language texts and deep models to learn text semantic representation, and finally realizes the construction and expansion of dictionaries with high efficiency and high quality. At the same time, the judgment of new corpus information can be more flexible and efficient, and the speed of dictionary matching is fast, which greatly improves the accuracy of dictionary construction and alleviates performance problems to a certain extent.

Referring to Fig. 6, a specific application scenario is adopted below to further describe the above embodiment in detail:

Assuming that the acquired text to be processed is "Visit Huangshan Mingming", the basic dictionary contains 6 named entity categories: person's name, organization name, place name, festival, product name and time.

After processing the to-be-processed text "Mingming to visit Huangshan" into "[CLS] Mingming to visit Huangshan [SEP]", input it into the pre-trained language sub-model of the category prediction model, and get the corresponding embedding vector "E[CLS] E1 , E2, E3, E4, E5, E6, E7, E[SEP]".

Input the embedding vector "E[CLS]E1, E2, E3, E4, E5, E6, E7, E[SEP]" into the named entity recognition sub-model of the category prediction model to get "E1+E2" and "E6+ E7" is the target word belonging to the named entity. At the same time, it can be obtained that the probability value of "E1+E2" belonging to a person's name is 0.98, the probability value of belonging to an organization name is 0.31, the probability value of belonging to a place name is 0.12, the probability value of belonging to a festival is 0.06, and the probability value of belonging to a product name is 0.06. 0.45, the probability value of belonging to time is 0.03; the probability value of "E6+E7" belonging to a person's name is 0.68, the probability value of belonging to an organization name is 0.78, the probability value of belonging to a place name is 0.89, the probability value of belonging to a festival is 0.07, and the probability value of belonging to a holiday is 0.07. The probability value of product name is 0.72, and the probability value of belonging to time is 0.04.

Assuming that the first set threshold is 0.95, the second set threshold is 0.9, and the third set threshold is 0.6, it can be determined that the named entity category to which the target word "Mingming" corresponding to "E1+E2" belongs is a person's name, and the The target word "Mingming" is added to the person's name in the basic dictionary.

Because the probability values of the target word "Huangshan" for "E6+E7" belong to the name of the person, the name of the organization, the name of the place, the festival, the name of the product and the time are all less than the first set threshold of 0.95 and the second set threshold of 0.9, and greater than the first set threshold of 0.95 and the second set threshold of 0.9. 3. If the named entity category with a threshold of 0.6 is set as human name, organization name, place name and product name, the similarity between the target word "Huangshan" and the words contained in the name of the person, organization name, place name and product name can be determined respectively. , and the similarity between the target word "Huangshan" and the words contained in the person's name is 0.06, the similarity with the words contained in the organization name is 0.14, the similarity with the words contained in the place name is 0.76, and the similarity with the words contained in the product name is 0.06. The similarity is 0.04. It can be determined that the named entity category to which the target word "Huangshan" belongs is a place name, and the target word "Huangshan" is added to the place name of the basic dictionary.

In some embodiments, the dictionary construction method proposed in this application can be combined with the N-Gram-based dictionary construction method, word2Vec-based sentiment dictionary construction method, textrank-based dictionary construction method, and product evaluation-oriented object mining in the related art. The domain dictionary construction methods are compared, and the comparison results can be shown in Table 1 below:

Table 1

Model Precision Recall F1-score Based on N-Gram 40.0 — — Based on word2Vec 60.3 37.5 46.2 based on textrank 59.8 58.6 59.2 Product Evaluation Object Mining 64.5 69.2 66.8 Our Method 85.6 86.2 85.9

Among them, Precision represents the accuracy rate, Recall represents the recall rate, and F1-score is a comprehensive evaluation index. The higher the Precision, Recall, and F1-score, respectively, the better the performance of the method.

As can be seen from the above table, compared with the N-Gram-based dictionary construction method, the word2Vec-based sentiment dictionary construction method, the textrank-based dictionary construction method, and the product evaluation object mining-oriented domain dictionary construction method, this application proposes The Precision, Recall, and F1-score of the dictionary construction method are the highest, indicating that the dictionary construction method proposed in this application has the best performance.

Based on the same inventive concept as the dictionary construction method shown in FIG. 4a, an embodiment of the present application also provides a dictionary construction apparatus, and the dictionary construction apparatus may be arranged in a server or a terminal device. Since the device is a device corresponding to the dictionary construction method of the present application, and the principle of solving the problem of the device is similar to that of the method, the implementation of the device can refer to the implementation of the above method, and the repetition will not be repeated.

FIG. 7 shows a schematic structural diagram of a dictionary construction device provided by an embodiment of the present application. As shown in FIG. 7 , the dictionary construction detection device includes an acquisition module 701 , a word recognition module 702 , a category recognition module 703 and a dictionary construction module 704 .

Wherein, the obtaining module 701 is used for obtaining the text to be processed and a basic dictionary; wherein, the basic dictionary includes a plurality of word categories;

A word recognition module 702, configured to determine, based on the trained category prediction model, at least one candidate word contained in the text to be processed, and the respective semantic information of the at least one candidate word;

The category identification module 703 is configured to select at least one target word that meets the set semantic conditions according to the respective semantic information of at least one candidate word through the category prediction model, and determine the probability that each of the at least one target word belongs to multiple word categories. value;

The dictionary construction module 704 is configured to assign at least one target word to word categories whose corresponding probability values meet the set probability conditions, respectively.

Optionally, the category prediction model includes a pre-trained language sub-model and a named entity recognition sub-model; the word recognition module 702 is specifically used for:

Based on the text to be processed, by pre-training the language sub-model, a word vector corresponding to at least one word contained in the text to be processed is obtained; wherein, each word vector represents the semantic information of the corresponding word;

Based on the word vector corresponding to at least one word, at least one word is combined through a named entity recognition sub-model to obtain at least one candidate word and the respective semantic information of at least one candidate word; wherein, each candidate word contains at least one word .

Optionally, the category identification module 703 is specifically used for:

Through the named entity recognition sub-model, from at least one candidate word, at least one target word whose semantic information belongs to the named entity is selected; the named entity is an entity name with specific semantics;

For at least one target word, the following operations are respectively performed: through the named entity recognition sub-model, according to the semantic information of a target word, determine a probability value that a target word belongs to multiple word categories respectively.

Optionally, as shown in FIG. 8 , the above-mentioned apparatus may further include a model training module 801 for:

Obtain a training data set; the training data set includes multiple text data samples, and the text data samples are marked with set categories;

Based on the training data set, the category prediction model is iteratively trained until the set convergence conditions are met, wherein an iterative training process includes:

Based on the text data samples extracted from the training data set, through the category prediction model, at least one target word in the text data sample is determined, and the target word category corresponding to each of the at least one target word is determined;

According to the target word category and the set category, determine the corresponding loss value, and adjust the parameters of the category prediction model according to the loss value.

Optionally, the category prediction model includes a pre-trained language sub-model and a named entity recognition sub-model; the training data set includes encyclopedia text data samples, domain text data samples and named entity recognition text data samples; the model training module 801 is also used for:

Based on the text data samples extracted from the encyclopedia text data samples and domain text data samples, the corresponding embedding vector samples are determined by pre-training the language sub-model; the encyclopedia text data samples are text data in untargeted fields, and each domain text The data sample is text data that includes multiple named entities of set categories, and is marked with the corresponding field category;

Based on the vector samples extracted from the embedding vector samples and the word vector samples, the named entity recognition sub-model is used to determine the corresponding at least one target word and its corresponding target word category; the word vector samples are obtained based on the named entity recognition text data samples The named entity recognition text data sample is text data including at least one named entity, and each named entity word is marked with a corresponding named entity category.

Optionally, the model training module 801 is also used for:

Determine the first loss value according to the target word category and domain category, and adjust the parameters of the pre-trained language sub-model according to the first loss value;

The second loss value is determined according to the target word category, the field category, and the named entity category, and the named entity recognition sub-model is parameter adjusted according to the second loss value.

Optionally, the model training module 801 is also used for:

Preprocessing operations are respectively performed on the encyclopedia text data samples, the domain text data samples and the named entity recognition text data samples in the training data set; the preprocessing operations include at least one of data screening and format conversion.

Optionally, the dictionary construction module 704 is specifically used for:

Based on the probability values of a target word belonging to multiple word categories, determine the word category corresponding to the maximum probability value, and use the word category as the target category;

If the probability value of a target word belonging to the target category is greater than the first set threshold, then a target word is assigned to the target category.

Optionally, the dictionary construction module 704 is further used for:

If the probability value of a target word belonging to multiple word categories is not greater than the second set threshold, at least one word category whose probability value is greater than the third set threshold is selected as a candidate category; the third set threshold is less than the second set threshold set threshold;

Based on the similarity between a target word and words contained in at least one candidate category, a candidate category that meets the set similarity condition is selected as a target category, and a target word is assigned to the target category.

After introducing the dictionary construction method and apparatus of the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

As will be appreciated by one skilled in the art, various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application can be embodied in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, which may be collectively referred to herein as implementations "circuit", "module" or "system".

Based on the same inventive concept as the above method embodiments, an electronic device is also provided in the embodiments of the present application. Referring to FIG. 9, it is a schematic diagram of a hardware structure of an electronic device applying the embodiments of the present application. 900 may include at least a processor 901 and a memory 902 . Wherein, the memory 902 stores program codes, and when the program codes are executed by the processor 901, the processor 901 executes the steps of any one of the above dictionary construction methods.

In some possible implementations, a computing device according to the present application may include at least one processor, and at least one memory. The memory stores program codes, which, when executed by the processor, cause the processor to execute the steps of the dictionary construction method according to various exemplary embodiments of the present application described above in this specification. For example, the processor may perform the steps shown in Figure 4a.

A computing device 1000 according to such an embodiment of the present application is described below with reference to FIG. 10 . As shown in Figure 10, computing device 1000 takes the form of a general-purpose computing device. Components of the computing device 1000 may include, but are not limited to, the above-mentioned at least one processing unit 1001 , the above-mentioned at least one storage unit 1002 , and a bus 1003 connecting different system components (including the storage unit 1002 and the processing unit 1001 ).

Bus 1003 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus structures.

Storage unit 1002 may include readable media in the form of volatile memory, such as random access memory (RAM) 10021 and/or cache storage unit 10022 , and may further include read only memory (ROM) 10023 .

The storage unit 1002 may also include a program/utility 10025 having a set (at least one) of program modules 10024 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, An implementation of a network environment may be included in each or some combination of these examples.

Computing device 1000 may also communicate with one or more external devices 1004 (eg, keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with computing device 1000, and/or communicate with the computing device 1000 can communicate with any device (eg, router, modem, etc.) that communicates with one or more other computing devices. Such communication may take place through input/output (I/O) interface 1005 . Also, the computing device 1000 may communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and/or a public network such as the Internet) through a network adapter 1006 . As shown, network adapter 1006 communicates with other modules for computing device 1000 via bus 1003 . It should be understood that, although not shown, other hardware and/or software modules may be used in conjunction with computing device 1000, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives and data backup storage systems.

Based on the same inventive concept as the above-mentioned method embodiments, various aspects of the dictionary construction method provided by the present application can also be implemented in the form of a program product, which includes program code, and when the program product runs on an electronic device, the program code is used for The electronic device is caused to perform the steps in the dictionary construction method according to various exemplary embodiments of the present application described above in this specification, for example, the electronic device may perform the steps shown in FIG. 4a.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

While the preferred embodiments of the present application have been described, additional changes and modifications to these embodiments may occur to those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be construed to include the preferred embodiment and all changes and modifications that fall within the scope of this application.

Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims (15)

1. a dictionary construction method, is characterized in that, comprises: Obtain the text to be processed and a basic dictionary; wherein, the basic dictionary contains multiple word categories; Based on the trained category prediction model, determine at least one candidate word included in the text to be processed, and the respective semantic information of the at least one candidate word; Through the category prediction model, according to the respective semantic information of the at least one candidate word, at least one target word that meets the set semantic conditions is selected, and it is determined that the at least one target word belongs to each of the multiple word categories. probability value; The at least one target word is respectively assigned to word categories whose corresponding probability values meet the set probability conditions. 2. The method of claim 1, wherein the category prediction model comprises a pre-trained language sub-model and a named entity recognition sub-model; and the category prediction model based on the training determines the to-be-processed text At least one candidate word included, and the respective semantic information of the at least one candidate word, including: Based on the text to be processed, through the pre-trained language sub-model, a word vector corresponding to at least one word contained in the text to be processed is obtained; wherein, each word vector represents the semantic information of the corresponding word; Based on the respective word vectors corresponding to the at least one word, the at least one word is combined by the named entity recognition sub-model to obtain at least one candidate word and the respective semantic information of the at least one candidate word; wherein, Each candidate word contains at least one word. 3. The method according to claim 2, wherein, according to the respective semantic information of the at least one candidate word through the category prediction model, at least one target word that meets the set semantic conditions is selected, and Determining the probability values that the at least one target word respectively belongs to the plurality of word categories includes: Through the named entity recognition sub-model, from the at least one candidate word, at least one target word whose semantic information belongs to a named entity is selected; the named entity is an entity name with specific semantics; For the at least one target word, the following operations are respectively performed: using the named entity recognition sub-model, according to the semantic information of the one target word, determine a probability value that the one target word belongs to the multiple word categories respectively. 4. The method according to any one of claims 1 to 3, wherein the training process of the category prediction model comprises: Obtaining a training data set; the training data set includes a plurality of text data samples, and the text data samples are marked with set categories; Based on the training data set, the category prediction model is iteratively trained until a set convergence condition is met, wherein an iterative training process includes: Based on the text data samples extracted from the training data set, through the category prediction model, at least one target word in the text data sample is determined, and the target word category corresponding to each of the at least one target word is determined; According to the target word category and the set category, a corresponding loss value is determined, and according to the loss value, the parameters of the category prediction model are adjusted. 5. The method of claim 4, wherein the category prediction model comprises a pre-trained language sub-model and a named entity recognition sub-model; the training data set comprises encyclopedia text data samples, domain text data samples and naming Entity recognition text data samples; determining at least one target word in the text data sample through the category prediction model based on the text data samples extracted from the training data set, and determining the target word category corresponding to each of the at least one target word, include: Based on the text data samples extracted from the encyclopedia text data samples and the domain text data samples, the corresponding embedding vector samples are determined through the pre-trained language sub-model; the encyclopedia text data samples are texts in untargeted fields Data, each domain text data sample is text data that includes multiple named entities of set categories, and is marked with the corresponding domain category; Based on the vector samples extracted from the embedding vector samples and the word vector samples, the named entity recognition sub-model determines at least one corresponding target word and its corresponding target word category; the word vector samples are based on the The named entity identification text data sample is obtained by describing the named entity identification text data sample; the named entity identification text data sample is text data including at least one named entity, and each named entity word is marked with a corresponding named entity category. 6. The method according to claim 5, wherein the corresponding loss value is determined according to the target word category and the set category, and the category prediction model is performed according to the loss value. Parameter adjustment, including: Determine a first loss value according to the target word category and the domain category, and adjust the parameters of the pre-trained language sub-model according to the first loss value; According to the target word category, the domain category, and the named entity category, a second loss value is determined, and parameters of the named entity recognition sub-model are adjusted according to the second loss value. 7. The method according to claim 5, wherein after acquiring the training data set, the method further comprises: Preprocessing operations are respectively performed on the encyclopedia text data samples, the domain text data samples and the named entity recognition text data samples in the training data set; the preprocessing operations include at least one of data screening and format conversion. 8. The method according to any one of claims 1 to 3, wherein the attributing the at least one target word to word categories whose corresponding probability values meet the set probability conditions, respectively, comprises: For the at least one target word, respectively perform the following operations: Based on the probability values that a target word belongs to the multiple word categories respectively, determine the word category corresponding to the maximum probability value, and use the word category as the target category; If the probability value of the one target word belonging to the target category is greater than the first set threshold, the one target word is assigned to the target category. 9. The method according to any one of claims 1 to 3, wherein the attributing the at least one target word to word categories whose corresponding probability values meet the set probability conditions, respectively, comprises: For the at least one target word, respectively perform the following operations: If the probability value of a target word belonging to the multiple word categories is not greater than the second set threshold, at least one word category whose probability value is greater than the third set threshold is selected as a candidate category; the third set The threshold is less than the second set threshold; Based on the similarity between the one target word and the words contained in the at least one candidate category, a candidate category that meets the set similarity condition is selected as the target category, and the one target word is assigned to the target in the category. 10. A dictionary construction device, comprising: an acquisition module for acquiring the text to be processed and a basic dictionary; wherein the basic dictionary contains multiple word categories; A word recognition module, configured to determine at least one candidate word contained in the text to be processed based on the trained category prediction model, and the respective semantic information of the at least one candidate word; The category recognition module is used to select at least one target word that meets the set semantic conditions according to the respective semantic information of the at least one candidate word through the category prediction model, and determine that the at least one target word belongs to the The probability value of describing multiple word categories; The dictionary construction module is used for attributing the at least one target word to word categories whose corresponding probability values meet the set probability conditions, respectively. 11. The apparatus according to claim 10, wherein the category prediction model comprises a pre-trained language sub-model and a named entity recognition sub-model; the word recognition module is specifically used for: Based on the text to be processed, through the pre-trained language sub-model, a word vector corresponding to at least one word contained in the text to be processed is obtained; wherein, each word vector represents the semantic information of the corresponding word; Based on the respective word vectors corresponding to the at least one word, the at least one word is combined by the named entity recognition sub-model to obtain at least one candidate word and the respective semantic information of the at least one candidate word; wherein, Each candidate word contains at least one word. 12. The apparatus of claim 11, wherein the category identification module is specifically configured to: Through the named entity recognition sub-model, from the at least one candidate word, at least one target word whose semantic information belongs to a named entity is selected; the named entity is an entity name with specific semantics; For the at least one target word, the following operations are respectively performed: using the named entity recognition sub-model, according to the semantic information of the one target word, determine a probability value that the one target word belongs to the multiple word categories respectively. 13. An electronic device, characterized in that it comprises a processor and a memory, wherein the memory stores a program code that, when executed by the processor, causes the processor to execute claim 1 The steps of any one of to 9. 14. A computer-readable storage medium, characterized in that it comprises program code, when the program code is run on an electronic device, the program code is used to cause the electronic device to execute any one of claims 1 to 9 a step of the method. 15. A computer program product, comprising a computer program/instruction, characterized in that, when the computer program/instruction is executed by a processor, the steps of the method of claims 1-9 are implemented.

Images