RU2750852C1 - Method for attribution of partially structured texts for formation of normative-reference information - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: invention relates to computer technology. The method for attributing partially structured texts for generating normative-reference information includes selecting a training set of texts in the natural language of partially structured texts, extracting the appropriate set of features for each category of named entities, training a classification model using the training set of texts and sets of features for each category of named entities, performing training using attributes, obtaining a model for each named entity and checking attributes, extracting tokens from unmarked text by the processor, generating a marked-up representation by the processor of at least a part of the text based on at least one of the tokens classified by categories.

EFFECT: increased speed of data attribution processes.

1 cl, 2 dwg, 1 tbl

Description

Technology area

The invention relates to the field of extracting named entities from natural language texts using computer systems, in particular to the processes of normative reference information (NSI) normalization, which include finding and differentiating contextual attributes of entities from partially structured text sources. Attributes are used to characterize a named entity, its validation, differentiation, and the final decision on their inclusion in the NSI.

Prior art

Master data are general business objects distributed between operational and analytical systems, including information about customers, suppliers, organizational units, as well as reference data necessary for the operation of the enterprise. The NSI includes dictionaries, reference books and classifiers, the data from which are used in the formation of new documents. Dictionaries, reference books and classifiers in the process of functioning can be replenished, consolidated. The replenishment of the reference data is often carried out from partially structured sources - csv tables, database tables, price lists from Internet sources, etc. The main challenge in managing master data is ensuring that it is complete, consistent and consistent. In the NSI this is achieved by normalizing, "cleaning" records of reference books and dictionaries of the NSI, as a result of which:

- errors, incompleteness and incorrectness of data are eliminated;

- the key attributes are specified, and the unification of the names of the elements of the directories is carried out.

Well-known master data management systems are 1C MDM, SAP MDM, Oracle MDM and IBM InfoSphere MDM, which are based on a classifier, which is a hierarchy of classes that describe the main business objects of the industry. By comparing the descriptions of the reference data objects and the classifier fields, the expert can understand what characteristics a particular object has, find duplicates and fill in the values of the missing characteristics. The introduction of automated subsystems that use algorithms that process text in natural language and perform two main operations - classification of reference data objects with respect to a given classifier and attribution (highlighting the values of characteristics of reference data objects), can significantly reduce the qualification requirements of experts involved in normalization.

When classifying, the system must receive from the expert the identifiers of the classifier, relative to which the classification is performed, and the reference data object, and return several most likely classes indicating the degree of confidence of the system in each answer.

When identifying the values of characteristics (attribution), the system must receive from the expert the identifiers of the class, the values of the characteristics of which are to be extracted, and the reference data object, and return the most probable possible values of each of the characteristics, indicating the degree of confidence of the system.

Particular subproblems of normalization were solved in different areas of text information processing. Thus, the classification problem can be viewed as the problem of extracting named entities from unstructured natural language text. The task of extracting the values of the characteristics of objects also involves the use of machine learning methods for text analysis.

A number of approaches are currently used, each with its own advantages and disadvantages.

A system and process are known for analyzing, qualifying, and ingesting unstructured data sources through empirical attribution. The application describes the attribution of the data source in accordance with the rules. Analyze the data to identify characteristics of entanglement in the data. The qualitative measure of the attribute is computed and thus a weighted attribute is produced. A qualitative measure of the entanglement characteristic is calculated and thus a weighted entanglement characteristic is provided. The weighted attribute and the weighted entanglement characterization are analyzed to create a disposition. Filter the data in accordance with the disposition and, thus, issue the extracted data (RU2674331, IPC G06F 17/22, G06F 17/30, publ. 06.12.2018).

The known method and system of natural language processing, in terms of entity recognition (ERDS). The application describes the automatic determination of which objects the text belongs to, using natural language processing methods and analysis of information obtained from contextual data in the text (US20090144609, IPC G06F 17/00, G06F 17/30, publ. 04.06.2009).

A known method of processing text in natural language and, more specifically, to methods of automatic identification of significant objects in documents. This method describes the identification of a set of references to entities in each abstract and each corresponding document based on their respective parts of speech (POS) and dependency analysis (US9619457, IPC G06F 17/27, G06K 9/00, publ. 11.04.2017).

A known method of automatically extracting and organizing information by a processing device from a variety of data sources. For data sources, a pipeline is used to retrieve natural language processing information, which enables automatic object detection. The information about the detected objects is identified by analyzing the products of the natural language processing pipeline. The identified information is grouped into equivalence classes containing equivalent information. At least one displayable representation of the equivalence classes is created. The order in which at least one rendered view is displayed is calculated. A combined representation of equivalence classes is produced, which corresponds to the display order of the displayed representation (US20140195884, IPC G06F 17/21, G06F 17/30, publ. 10.07.2014).

There is a known method for recognizing named entities in natural language texts, based on the use of morphological and semantic features, including lexical and morphological analysis using a computing system of a natural language text containing many tokens, determining one or more lexical meanings based on lexical and morphological analysis and grammatical meanings associated with each token in the set of tokens; calculating, for each token in the plurality of tokens, one or more classifier functions using lexical and grammatical meanings associated with the token, characterized in that the value of each classifier function indicates an estimate of the degree of association of the token with a category of named entities; interpretation of semantic structures using a set of production rules to determine, for one or more tokens included in a selected part of a natural language text, an assessment of the degree of association of a token with a category of named entities (RU2619193, IPC G06F 17/28, publ. 05/12/2017).

Among the shortcomings of the above analogs, it can be noted that the mechanism for searching and determining tokens associated with a category of named entities is made either based on the analysis of the corresponding parts of speech (POS), or based on the lexico-morphological analysis of the text in natural language, which excludes the markup and search for numerical characteristics , units of measurement, abbreviations typical for the attributes of the reference data.

The closest technical solution to the claimed invention is a means of recognizing named entities from an unlabeled text corpus. The technical result consists in increasing the efficiency of recognition and markup of named entities in texts. Choose a natural language training set. The processor retrieves the appropriate set of features for each category of named entities. The classification model is trained by the processor using a training set of texts and sets of features for each category of named entities. The processor retrieves tokens from the unmarked text. The processor generates a set of attributes for each unlabeled text token based on at least deep semantic-syntactic analysis. Possible syntactic links are determined in at least one sentence of unmarked text, including obtaining a plurality of syntactic attributes. A language-independent semantic structure is formed, including the definition of semantic links and the corresponding semantic attributes of each token. Each token is classified by the processor into at least one of the categories based on the classifier model and a set of token attributes. The processor forms a marked-up representation of at least a part of the text based on at least one of the tokens classified by categories (RU 2665239, IPC G06F 17/27, publ. 28.08.2018).

The disadvantage of the prototype is the procedure for identifying attributes of named entities based on semantic relationships and parsing, which is reasonable for marking up unstructured tests with a high semantic load and is not suitable for semi-structured tests - sources of replenishment and consolidation of reference data, where the attributes of named entities are very often a combination words with numerical characteristics and units of measurement.

The essence of the invention

The technical result achieved when using the claimed invention is to increase the speed, provide scalability, flexibility and consistency of data attribution processes.

The essence of the invention lies in the fact that the method of attributing partially structured texts for the formation of normative and reference information, including the selection of a training set of texts in natural language, the processor extracting the corresponding set of features for each category of named entities, training the processor of the classification model using the training set of texts and sets features for each category of named entities, the processor extracts tokens from the unlabeled text, the processor generates a marked-up representation of at least a portion of the text based on at least one of the tokens classified by categories. To train the classification model for named entities, partially structured texts were selected and preprocessed - lemmatization, stemming, removal of stop words, punctuation marks, as well as vectorization of words with weight assignment to each word of the entity name in accordance with the assessment of the importance of the word in the context of the document. To extract tokens, the unlabeled text is marked up on the trained model of the name classifier, and the names are divided into tokens according to a given set of delimiters, after which all possible combinations of 1-5 consecutive tokens are compiled. To extract the attributes, the obtained combinations of tokens are processed by the attribute classifier model trained on a given set of class attributes, which returns for each combination the probability that this combination is one or another attribute from the set of attributes of a given class or is not an attribute at all, for each attribute as the main one. value, the combination with the highest probability and exceeding the specified threshold is selected.

BRIEF DESCRIPTION OF THE DRAWINGS OF THE INVENTION

FIG. 1 shows a diagram of the process of attributing partially structured texts; FIG. 2 shows a price list in CSV format, as an example - a partially structured text. Table 1 shows an example of the probability distribution of assigning combinations of tokens to given attributes.

Implementation of the invention

Partially structured text (Fig. 1), after lemmatization and stemming, is classified by a classifier trained on marked-up entity names. The name of an entity means the actual name of the entity (for example, "floor board") and the context, which may contain a set of attributes. For each entity name, the classifier gives the probability of belonging to a certain class. Each class is assigned a specific set of attributes.

Each entity name is split into tokens. A token is a part of the entity name between some separator characters. Combinations are made from the received tokens. Combinations of tokens are submitted to the LightGBM classifier trained on the marked attributes of the class [1]. The classifier produces an array of probabilities for each combination. Each probability value determines the degree to which the combination belongs to the validated attributes. Based on the obtained array of probabilities and the value of the combination of tokens, the expert sets the threshold and decides to assign the combination to one or another attribute of a given class.

In more detail, the process of attributing partially structured texts for the formation of the reference data is as follows.

1. Assignment of the names of entities from the text in natural language to classes according to a given classifier.

1.1 Performing text preprocessing: lemmatization, stemming, removal of stop words and punctuation marks, vectorization.

1.1.1 When performing the lemmatization stage, the words in the text are reduced to their original vocabulary form (for example, "fir" - "fir"). This stage is implemented using dictionaries. This removes stop words (prepositions and auxiliary parts of speech, as well as stop words from special bases of stop words).

1.1.2 In the process of stamming, words are brought to the stem, namely, the ending and the formative suffix are cut off.

1.1.3 Vectorization of words with a similar semantic meaning is carried out using the word2vec method. Word2Vec, trained on search queries, performs word clustering based on context and the SKIP gram method. The TF-IDF function is used to assign weight to each word. TF-IDF (TF - Word Frequency, IDF - Inverse Document Frequency) is a statistical measure used to assess the importance of a word in the context of a document that is part of a document collection or corpus. The weight of a word is proportional to the frequency of use of this word in a document and is inversely proportional to the frequency of use of a word in all documents of the collection.

1.2 The assignment of entities to a specific category (class) is based on machine learning algorithms using open source libraries (Keras). In particular, a convolutional artificial neural network (Conovolution Neural Network, CNN) with one Embedding and two Convolution layers is used. The neural network is trained on a set of texts obtained from a validated NSI. For training, only the names of entities are used.

2. Highlighting the values of the attributes of the names of entities

The second stage of attribution is shown in the following example:

There is a certain price list in CSV format (Fig. 2), in which the columns have the following meanings:

- name_mtr - product name (entity)

- id_class - class identifier

- name_atr - attribute name

- val_atr - attribute value

Further processing is carried out as follows.

2.1 Based on the value of the class obtained in the previous step, it is determined which attributes are found within this class, and then only these attributes are selected.

2.2 For each possible attribute, there is a part of the name that is most likely the value of the attribute in question. For this, the name is divided into tokens. A set of delimiters that delimit tokens can be specified. For example, the following separators have been selected: "", ",", ";".

2.3 Next, combinations of consecutive tokens are made - all combinations that consist of 1-5 consecutive tokens combined through specified separators. For example, consider the name “Glued planed timber, spruce 100 * 200mm; 11.985 m; humidity 12 +/- 2%; class GL 24h ".

This name will be split into the following tokens: "Timber", "glued", "planed", "spruce", "100 * 200mm", "11", etc. The following combinations will be made of them: "Timber", "Timber" glued "," Glued beams "," Beams; glued "," Glued planed timber "," Glued, planed timber ", etc.

2.4 Each of the resulting combinations is processed by a trained attribute classifier. This classifier returns for each combination the probability that the given combination is one or another attribute from the set of attributes of a given class or is not an attribute at all. In other words, the probability distribution between the attributes for the considered combination is obtained.

Since each class of names has its own set of attributes, it will be necessary to train a separate model for each class. For training the classifier requires pairs of the form {attribute value, attribute name}, which are available in the price list. For each model, pairs are selected from the corresponding class of names. The attribute value must be vectorized before being fed to the input to the classifier. For vectorization, a method based on the N-gram CountVectorizer from the scikit-learn library was chosen, since the semantic load is carried by combinations of words, taking into account their sequence.

Example. Let's take the name from paragraph 2.3. For the class of this name ("01.01.10.01 Non-profiled glued laminated timber") we have two attributes: section and length.

The probability distribution for some combinations is shown in Table. one.

For each attribute, the combination that has the highest probability and exceeds the specified threshold is selected as the main value. For the considered example, with a threshold value of 0.6, the following will be selected as the main values: for the “Length” attribute - “11.985”, for the “Section” attribute - “100 * 200mm”.

Information sources

1. LightGBM - gradient boosting framework. Access mode: https://lightgbm.readthedocs.io/en/latest/. Access date: 15.10.

Table 1 Combination The probability of belonging to an attribute "Section" "Length" "Not an attribute" humidity 12 +/- 2% 0.03 0.015 0.955 100 * 200mm 0.84 0.11 0.05 Glued timber 0.012 0.0114 0.9766 11.985 0.21 0.73 0.06

Claims (1)

A method for attributing semi-structured texts to form normative and reference information, including the selection of a training set of texts in natural language, the processor extracting the corresponding set of features for each category of named entities, training the processor of the classification model using the training set of texts and sets of features for each category of named entities, extraction by the processor of tokens from the unlabeled text, the formation by the processor of a marked-up representation of at least part of the text based on at least one of the tokens classified by categories, characterized in that semi-structured texts are selected for training the named entity classification model and executed preprocessing - lemmatization, stemming, removal of stop words, punctuation marks, as well as vectorization of words with weight assignment to each word of the entity name in accordance with the assessment of the importance of the word in the context of the document nta, to extract tokens, the unlabeled text is marked on the trained model of the classifier of names, and the names are divided into tokens according to a given set of delimiters, after which all possible combinations of 1-5 consecutive tokens are compiled, to extract attributes, the resulting combinations of tokens are processed by the trained one on a given set of class attributes the attribute classifier model, which returns for each combination the probability that a given combination is one or another attribute from a set of attributes of a given class or is not an attribute at all, for each attribute the combination with the highest probability and exceeding a given threshold is selected as the main value.

Images