CN115687939A - Mask text matching method and medium based on multi-task learning - Google Patents

Abstract

The invention discloses a Mask text matching method and medium based on multi-task learning. The method steps include: 1) obtaining at least two texts to be matched; 2) performing feature extraction on the texts to be matched to obtain the text word features of each text to be matched; 3) establishing a text matching model based on BERT; 4) Input the text word features of all texts to be matched into the text matching model to obtain matching results of different texts to be matched. The media includes a computer program. The present invention proposes the idea of constructing a simplified model of the Mask matrix in combination with data characteristics, which can enlarge the difference between texts to be matched while simplifying the model, and enhance the generalization ability of the final model training.

Description

A mask text matching method and medium based on multi-task learning

technical field

The invention relates to the field of natural language processing, in particular to a mask text matching method and medium based on multi-task learning.

Background technique

Text matching method aims to judge whether the semantic equivalence between two natural sentences is an important research direction in the field of natural language processing. Text matching research also has high commercial value and plays an important role in information retrieval, intelligent customer service and other fields.

In recent years, although neural network models have achieved similar or even better accuracy than humans in some standard question matching evaluations, these models are less robust when dealing with real application scenarios. It is impossible to make a correct judgment on the issue of easy judgment), resulting in extremely poor product experience and economic losses.

Most of the current text matching tasks are tested on the test set with the same distribution as the training set, and the effect is good, but the model ability is actually exaggerated, and the real evaluation of the fine-grained advantages and disadvantages of the model is lacking, because some information retrieval from the actual industry , In terms of intelligent customer service scenarios, it is difficult to guarantee the same distribution of application scenarios and model development. Therefore, the present invention focuses on the robustness of the text matching model in real application scenarios, and discovers the shortcomings of the current text matching algorithm model from multiple dimensions such as vocabulary, syntax, and pragmatics, and promotes the application of semantic matching technology in intelligent interaction, etc. The development of the industrial field.

Traditional text matching methods include algorithms such as BoW, VSM, TF-IDF, BM25, Jaccord, and SimHash. For example, the BM25 algorithm calculates the matching score between the two through the coverage of the network field on the query field. The matching degree is better, and it mainly solves the matching problem at the vocabulary level, or the similarity problem at the vocabulary level. In fact, the matching algorithm based on the coincidence degree of vocabulary has great limitations. The reasons include: the limitation of word meaning, "taxi" Although it is not similar to "taxi" in words, it is actually the same means of transportation; "apple" means different things in different contexts, either for fruit or for a company; due to structural limitations, "machine learning" and "learning Although the vocabulary of "machine" completely overlaps, the meanings expressed are different; knowledge is limited, "Qin Shihuang played Dota", although this sentence has no problem in terms of grammar and syntax, it is wrong to look at this sentence in combination with knowledge. The above shows that, For text matching tasks, we should not only stay at the level of literal matching, but also need to match at the semantic level.

Deep semantic matching method, with the successful application of deep learning in the fields of computer vision, speech recognition, and recommendation systems, in recent years, many studies have devoted themselves to applying deep neural network models to natural language processing tasks to reduce the cost of feature engineering. The Word Embedding trained based on the neural network is used for text matching calculation. The training method is simple, and the semantic computability of the obtained word vector representation is further enhanced. However, the Word Embedding obtained by only using unlabeled data training has a practical effect on the text matching degree calculation. It is not much different from the topic model technology. They are essentially based on the training of co-occurrence information. In addition, Word Embedding itself does not solve the semantic representation of phrases and sentences, nor does it solve the asymmetry problem of matching.

The current text matching algorithm is mainly based on the pre-trained language model of BERT, which improves the semantic information of the text vector as much as possible. However, the text vector obtained by the pre-trained model cannot recognize the differences in the text well in some scenarios. For example, the difference in content between the sentences "How to exchange RMB for Hong Kong dollars" and "How to exchange Hong Kong dollars for RMB" is relatively small. But the meanings are very different. Therefore, if you only rely on the pre-trained model to get the text vector, it is difficult to capture the differences in the text from the dimensions of vocabulary, syntax, and pragmatics.

It can be seen that the current text matching algorithm has the following defects:

1) The language model based on statistics cannot express rich semantic information, and it is difficult to capture the differences between texts in some short text matching scenarios with relatively small differences.

2) Algorithmic models based on word vectors and attention require a lot of labeled data, and the model structure is complex, and there is no further mining and utilization of the structural features of the text itself, such as syntactic structure and parts of speech.

3) The text matching model based on pre-training pays more attention to the output results of pre-training, and designs a more complex network structure for classification according to the output results of the pre-training model, without combining the structural features of the text itself with the pre-training model. Some useful prior information is lost.

Contents of the invention

The object of the present invention is to provide a kind of Mask text matching method based on multi-task learning, comprising the following steps:

1) Obtain at least two texts to be matched;

2) feature extraction is carried out to described text to be matched, obtain the text word feature of each text to be matched;

3) Establish a text matching model based on BERT;

4) Input the text word features of all texts to be matched into the text matching model to obtain matching results of different texts to be matched.

Further, the step of extracting features from the target text to be matched includes: word segmentation processing, part-of-speech tagging, named entity recognition, semantic role tagging, and dependency syntax analysis.

Further, the word features of the target text include one or more of part-of-speech features, named entity features, semantic role features, and dependency syntactic relationship features.

Further, the BERT-based text matching model includes an embedding input layer, a multi-head attention layer, a forward propagation layer and an output layer.

Further, the steps of obtaining matching results of different texts to be matched include:

a) Utilize the embedding input layer to convert the text word features to obtain the embedding input X, and convert the embedding input X to feature components Q=XW ^Q , feature components K=XW ^K , feature components V=XW ^V ; W ^Q , W ^K , W ^V are the weights corresponding to different feature components;

b) Use the multi-head attention layer to process the feature components of the embedding input X, and obtain the multi-head attention layer processing result MultiHead(Q, K, V), namely:

MultiHead (Q, K, V) = Concat (head ₁ , . . . , head _h ) W ^O (1)

In the formula, W ^O is the weight;

Among them, the parameter head _i is as follows:

head _i = Attention(QW _i ^Q , KW _i ^k , VW _i ^v ), i=1, 2, . . . , h (2)

Attention(QW _i ^Q , KW _i ^k , VW _i ^v )=Mask*Attention(Q, K, V) (3)

In the formula, softmax is the activation function; d _k represents the dimension of the word vector. In order to prevent the input value of softmax from being too large, the derivative is close to 0. Mask represents a mask; Attention(QW _i ^Q , KW _i ^k , VW _i ^v ), Attention(Q, K, V) are intermediate parameters; h is an integer greater than 0;

c) Use the forward propagation layer to process the processing result MultiHead (Q, K, V) of the multi-head attention layer, and obtain the processing result x of the forward propagation layer, namely:

x=norm(X+MultiHead(Q, K, V)) (5)

d) process the result x of the forward propagation layer by using the output layer, and obtain the output of the BERT-based text matching model as the matching result of different texts to be matched;

The output of the BERT-based text matching model is as follows:

FFN(x)=max(0, xW ₁ +b ₁ )W ₂ +b ₂ (6)

In the formula, W ₁ and W ₂ are weights; b ₁ and b ₂ are biases; FFN(x) is the output.

Further, the embedding input X=x ₁ +x ₂ ;

Among them, the input component _x1 and the input component _x2 are as follows:

X ₁ =E _tok +E _seg +E _pos (7)

x ₂ =embedding1(pos)+embedding2(ner)+embedding3(seg) (8)

In the formula, E _tok , E _seg , and E _pos respectively represent Token Embedding encoding, PositionEmbeddings encoding and SegmentEmbeddings encoding of text word features; embedding1, embedding2, and embedding3 represent the embedding layer of part of speech, named entity, and semantic role; pos, ner, and seg Represents the part-of-speech, nomenclature, and semantic role encoding of the input text.

Further, the mask Mask is a 0-1 variable, and Mask=1 for the same position of words in different texts to be matched, and Mask=0 for different positions of words in different texts to be matched.

Further, the output of the BERT-based text matching model includes a sequence output and a vector output; the vector output is a classification vector, and the sequence output is a part-of-speech tagging vector. The classification vectors include semantic sameness and semantic difference.

Further, the BERT-based text matching model has been pre-trained;

The criterion for the end of pre-training is the convergence of the loss function Loss;

The loss function Loss is as follows:

Loss＝Loss _nll + Loss _pos-tag (9)

In the formula, Loss _nll is the classification vector loss function; Loss _pos-tag is the loss function of part-of-speech tagging;

Among them, the classification vector loss function Loss _nll is as follows:

In the formula, n is the number of training samples; j is the jth sample; Z is the number of categories; c is the cth category; h _{j, c} is the probability that the jth sample belongs to the cth category; y _{j, c} indicates whether the j-th sample belongs to the c-th category; y _{j, c} = 1 indicates that the j-th sample belongs to the c-th category; y _{j, c} = 0 indicates that the j-th sample does not belong to the c-th category;

The part-of-speech tagging loss function Loss _pos-tag is as follows:

In the formula, P ₁ , P ₂ , P ₃ , P _n are the scores of the first, second, and nth possible part-of-speech tagging sequences corresponding to a sample; P _real-path is the real part-of-speech tagging sequence corresponding to a sample score.

A computer-readable storage medium storing a computer program;

When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 9 are implemented.

The technical effect of the present invention is unquestionable, and the present invention solves the problem that it is difficult to capture the differences between texts in scenarios such as intelligent interaction, natural language understanding, and similar sentence extraction.

Aiming at the problem of complex short text matching models and large parameters, the present invention proposes the idea of constructing a simplified model of the Mask matrix in combination with data characteristics. While simplifying the model, it can also enlarge the difference between the texts to be matched, so that the final model training can be generalized Enhanced capabilities.

Considering that the difference before text matching is relatively small, this difference can be reflected in the language features of part of speech and syntactic structure. At the input end, the present invention uses syntax, entity and other features as embedding to increase the semantic information of the model input.

General text matching mainly uses sentence vectors for matching. The present invention introduces multi-task learning to learn the difference between parts of speech of text to be matched from word granularity, thereby enhancing the generalization ability of the model.

Description of drawings

Fig. 1 is a flow chart of text matching;

Fig. 2 is a flow chart of text feature mining;

Figure 3 is a schematic diagram of the Mask mask.

Detailed ways

The present invention will be further described below in conjunction with the examples, but it should not be understood that the scope of the subject of the present invention is limited to the following examples. Without departing from the above-mentioned technical ideas of the present invention, various replacements and changes made according to common technical knowledge and conventional means in this field shall be included in the protection scope of the present invention.

Example 1:

Referring to Figures 1 to 3, a mask text matching method based on multi-task learning includes the following steps:

1) Obtain at least two texts to be matched;

2) feature extraction is carried out to described text to be matched, obtain the text word feature of each text to be matched;

3) Establish a text matching model based on BERT;

4) Input the text word features of all texts to be matched into the text matching model to obtain matching results of different texts to be matched, and the matching results include semantic similarity and semantic dissimilarity.

The step of feature extraction for the target text to be matched includes: word segmentation processing, part-of-speech tagging, named entity recognition, semantic role tagging, dependency syntax analysis and a series of natural language processing operations. The method proposed in the present invention relies on the social computing and The natural language processing technology provided by the "Language Technology Platform (LTP)" developed by the Information Retrieval Research Center performs the above operations.

This method uses the named entity recognition technology provided by the language technology platform of Harbin Institute of Technology and the iterative heuristic method for named entity recognition. The latter is to obtain the maximum noun phrase by merging connected nouns, in which the part of speech of nouns can only be {ni, nh, ns, nz, j}, which respectively represent organization names, personal names, geographical names, other proper nouns and acronyms .

The word features of the target text include one or more of part-of-speech features, named entity features, semantic role features and dependent syntactic relationship features.

The BERT-based text matching model includes an embedding input layer, a multi-head attention layer, a forward propagation layer and an output layer.

The steps for obtaining matching results of different texts to be matched include:

1) Use the embedding input layer to convert the text word features to obtain the embedding input X, and convert the embedding input X to feature components Q=XW ^Q , feature components K=XW ^K , feature components V=XW ^V ; W ^Q , W ^K , W ^V are the weights corresponding to different feature components;

2) Use the multi-head attention layer to process the feature components of the embedding input X, and obtain the multi-head attention layer processing result MultiHead(Q, K, V), namely:

MultiHead (Q, K, V) = Concat (head ₁ , . . . , head _h ) W ^O (1)

In the formula, W ^O is the weight;

Among them, the parameter head _i is as follows:

head _i = Attention(QW _i ^Q , KW _i ^k , VW _i ^v ), i=1, 2, . . . , h (2)

Attention(QW _i ^Q , KW _i ^k , VW _i ^v )=Mask*Attention(Q, K, V) (3)

In the formula, softmax is the activation function; d _k represents the dimension of the word vector. In order to prevent the input value of softmax from being too large, the derivative is close to 0. Mask represents a mask; Attention(QW _i ^Q , KW _i ^k , VW _i ^v ), Attention(Q, K, V) are intermediate parameters; h is an integer greater than 0;

3) Use the forward propagation layer to process the processing result MultiHead (Q, K, V) of the multi-head attention layer, and obtain the processing result x of the forward propagation layer, namely:

x=norm(X+MultiHead(Q, K, V)) (5)

4) Utilize the output layer to process the result x of the forward propagation layer to obtain the output of the BERT-based text matching model as the matching result of different texts to be matched;

The output of the BERT-based text matching model is as follows:

FFN(x)=max(0, xW ₁ +b ₁ )W ₂ +b ₂ (6)

In the formula, W ₁ and W ₂ are weights; b ₁ and b ₂ are biases. FFN(x) is the output.

The embedding input input X=x ₁ +x ₂ ;

Among them, the input component _x1 and the input component _x2 are as follows:

X ₁ =E _tok +E _seg +E _pos (7)

x ₂ =embedding1(pos)+embedding2(ner)+embedding3(seg) (8)

In the formula, E _tok , E _seg , and E _pos represent the Token Embedding encoding, PositionEmbeddings encoding, and SegmentEmbeddings encoding of the text word feature respectively; embedding1, embedding2, and embedding3 represent the embedding layer of part of speech, named entity, and semantic role; pos, ner, and seg Represents the part-of-speech, nomenclature, and semantic role encoding of the input text.

The mask Mask is a 0-1 variable, and Mask=1 for the same position of words in different texts to be matched, and Mask=0 for different positions of words in different texts to be matched.

The output of the BERT-based text matching model includes a sequence output and a vector output; the vector output is a classification vector, and the sequence output is a part-of-speech tagging vector. The classification vectors include semantic sameness and semantic difference.

The BERT-based text matching model has been pre-trained;

The criterion for the end of pre-training is the convergence of the loss function Loss;

The loss function Loss is as follows:

Loss＝Loss _nll + Loss _pos-tag (9)

In the formula, Loss _nll is the classification vector loss function; Loss _pos-tag is the loss function of part-of-speech tagging;

Among them, the classification vector loss function Loss _nll is as follows:

In the formula, n is the number of training samples; j is the jth sample; Z is the number of categories; c is the cth category; h _{j, c} is the probability that the jth sample belongs to the cth category; y _{j, c} indicates whether the j-th sample belongs to the c-th category; y _{j, c} = 1 indicates that the j-th sample belongs to the c-th category; y _{j, c} = 0 indicates that the j-th sample does not belong to the c-th category;

The calculation principle of the part-of-speech tagging loss function Loss _pos-tag is as follows:

For any sample, the class sequence for part-of-speech tagging might be:

Part-of-speech tagging sequence 1: START N-B N-I N-E O O

Part-of-speech tagging sequence 2: START N-B N-E O O O

Part-of-speech tagging sequence 3: START O N-B N-E O O

Part-of-speech tagging sequence 4: START V-B V-I V-E O O

…

Part-of-speech tagging sequence n: START N-B N-E V-B V-E O

Each possible path has a score P _i , and there are N paths in total, then the total score is:

P _total ＝P ₁ +P ₂ +P ₃ +...+P _n

During the training process, the parameter values of the model will be continuously updated with the iterations of the training process, so that the proportion of the real path will become larger and larger.

A computer-readable storage medium storing a computer program;

When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 9 are realized.

Example 2:

See Figures 1 to 3, a mask text matching method based on multi-task learning, including the following:

1) The text is processed by language analysis tools, and features such as parts of speech, named entities, semantic roles, and dependent syntactic relationships can be obtained.

2) According to the input text to be matched, compare the differences between the text pairs, and mark the positions with the same word position and the positions with different words. The same word position is set to 0 in the Mask matrix, otherwise it is set to 1.

The embedding input of BERT can be expressed as Token Embedding, Segmentation Embedding and Position Embedding synthesis x ₁ :

X ₁ ＝E _word ＝E _tok +E _seg +E _pos

Other language features can be expressed as x ₂ through the embedding layer:

x ₂ ＝embedding1(pos)+embedding2(ner)+embedding3(seg)

Final input x=x ₁ +x ₂

Convert input X to Q, K, V:

Q=XW ^Q , K=XW ^K , V=XW ^V

Attention calculation formula:

Here, in order to let the model pay attention to the inconsistency of the text to be matched, we can obtain the Mask matrix through simple processing in the data processing stage, so the model does not need to pay attention to the full amount of attention, but only needs to pay attention to the attention of the characters with inconsistent text:

Attention=Mask*Attention(Q, K, V)

Multi-head attention layer:

MultiHead (Q, K, V) = Concat (head ₁ , . . . , head _h ) W ^O

in:

head _i = Attention(QW _i ^Q , KW _i ^k , VW _i ^v ), i=1, 2, . . . , h

Forward propagation layer:

FFN(x)=max(0, xW ₁ +b ₁ )W ₂ +b ₂

Then the output is:

x=norm(X+MultiHead(Q,K,V))

The output of the final encoder:

Y=FFN(x)

The output includes a sequence output and a vector output. In this embodiment, the vector output is used as a classification vector, and the sequence output is used as the second task part-of-speech tagging vector. For the classification vector, we can obtain the first loss function Loss _KL as:

The loss function of part-of-speech tagging is: Loss _pos-tag , and the final loss function will be:

Loss＝Loss _nll + Loss _pos-tag

Example 3:

A kind of Mask text matching method based on multi-task learning, comprises the following steps:

1) Obtain at least two texts to be matched;

2) feature extraction is carried out to described text to be matched, obtain the text word feature of each text to be matched;

3) Establish a text matching model based on BERT;

4) Input the text word features of all texts to be matched into the text matching model to obtain matching results of different texts to be matched.

Example 4:

A mask text matching method based on multi-task learning, the main content is shown in embodiment 3, wherein the step of feature extraction of the target text to be matched includes: word segmentation processing, part-of-speech tagging, named entity recognition, semantic role tagging, dependency Syntax analysis.

Example 5:

A Mask text matching method based on multi-task learning, the main content is shown in embodiment 3, wherein the target text word features include one or more of part-of-speech features, named entity features, semantic role features and dependent syntactic relationship features .

Embodiment 6:

A mask text matching method based on multi-task learning, the main content is shown in embodiment 3, wherein the BERT-based text matching model includes an embedding input layer, a multi-head attention layer, a forward propagation layer and an output layer.

Embodiment 7:

A kind of Mask text matching method based on multi-task learning, see embodiment 3 for main content, wherein, the step of obtaining the matching result of different text to be matched comprises:

1) Use the embedding input layer to convert the text word features to obtain the embedding input X, and convert the embedding input X to feature components Q=XW ^Q , feature components K=XW ^K , feature components V=XW ^V ; W ^Q , W ^K , W ^V are the weights corresponding to different feature components;

2) Use the multi-head attention layer to process the feature components of the embedding input X, and obtain the multi-head attention layer processing result MultiHead(Q, K, V), namely:

MultiHead (Q, K, V) = Concat (head ₁ , . . . , head _h ) W ^O (1)

In the formula, W ^O is the weight;

Among them, the parameter head _i is as follows:

head _i = Attention(QW _i ^Q , KW _i ^k , VW _i ^v ), i=1, 2, . . . , h (2)

Attention(QW _i ^Q , KW _i ^k , VW _i ^v )=Mask*Attention(Q, K, V) (3)

In the formula, softmax is the activation function; d _k represents the dimension of the word vector; Mask represents the mask; Attention(QW _i ^Q , KW _i ^k , VW _i ^v ), Attention(Q, K, V) are intermediate parameters; h is an integer greater than 0;

3) Use the forward propagation layer to process the processing result MultiHead (Q, K, V) of the multi-head attention layer, and obtain the processing result x of the forward propagation layer, namely:

x=norm(X+MultiHead(Q, K, V)) (5)

4) Utilize the output layer to process the result x of the forward propagation layer to obtain the output of the BERT-based text matching model as the matching result of different texts to be matched;

The output of the BERT-based text matching model is as follows:

FFN(x)=max(0, xW ₁ +b ₁ )W ₂ +b ₂ (6)

In the formula, W ₁ and W ₂ are weights; b ₁ and b ₂ are biases; FFN(x) is the output.

Embodiment 8:

A kind of Mask text matching method based on multi-task learning, main content see embodiment 3, wherein, described embedding input X=x ₁ +x ₂ ;

Among them, the input component _x1 and the input component _x2 are as follows:

X ₁ =E _tok +E _seg +E _pos (7)

x ₂ =embedding1(pos)+embedding2(ner)+embedding3(seg) (8)

In the formula, E _tok , E _seg , and E _pos respectively represent Token Embedding encoding, PositionEmbeddings encoding and SegmentEmbeddings encoding of text word features; embedding1, embedding2, and embedding3 represent the embedding layer of part of speech, named entity, and semantic role; pos, ner, and seg Represents the part-of-speech, nomenclature, and semantic role encoding of the input text.

Embodiment 9:

A kind of Mask text matching method based on multi-task learning, see embodiment 3 for main content, wherein, described mask Mask is 0-1 variable, the mask Mask=1 of the word same position of different text to be matched, different to be matched Mask=0 at different positions of words in the text.

Example 10:

A kind of Mask text matching method based on multi-task learning, see embodiment 3 for main content, wherein, the output of the text matching model based on BERT comprises sequence output and vector output; The vector output is a classification vector, and the sequence output is a part-of-speech tagging vector; the classification vector includes the same semantics and different semantics.

Example 11:

A kind of Mask text matching method based on multi-task learning, main content sees embodiment 3, wherein, described text matching model based on BERT has been through pre-training;

The criterion for the end of pre-training is the convergence of the loss function Loss;

The loss function Loss is as follows:

Loss＝Loss _nll + Loss _pos-tag (9)

In the formula, Loss _nll is the classification vector loss function; Loss _pos-tag is the loss function of part-of-speech tagging;

Among them, the classification vector loss function Loss _nll is as follows:

In the formula, n is the number of training samples; j is the jth sample; Z is the number of categories; c is the cth category; h _{j, c} is the probability that the jth sample belongs to the cth category; y _{j, c} indicates whether the j-th sample belongs to the c-th category; y _{j, c} = 1 indicates that the j-th sample belongs to the c-th category; y _{j, c} = 0 indicates that the j-th sample does not belong to the c-th category;

The part-of-speech tagging loss function Loss _pos-tag is as follows:

In the formula, P ₁ , P ₂ , P ₃ , P _n are the scores of the first, second, and nth possible part-of-speech tagging sequences corresponding to a sample; P _real-path is the real part-of-speech tagging sequence corresponding to a sample score.

Example 12:

A computer-readable storage medium, where the computer-readable medium stores a computer program; when the computer program is executed by a processor, the steps of the method described in Embodiments 1-11 are implemented.

Claims (10)

1. A Mask text matching method based on multitask learning is characterized by comprising the following steps:

1) And acquiring at least two texts to be matched.

2) And extracting the characteristics of the texts to be matched to obtain the text word characteristics of each text to be matched.

3) Establishing a text matching model based on BERT;

4) And inputting the text word characteristics of all the texts to be matched into the text matching model to obtain matching results of different texts to be matched.

2. The Mask text matching method based on multitask learning according to claim 1, wherein the step of extracting the features of the target text to be matched comprises the following steps: word segmentation processing, part of speech tagging, named entity recognition, semantic role tagging and dependency syntactic analysis.

3. The Mask text matching method based on multitask learning according to claim 1, characterized in that said target text word characteristics include one or more of part-of-speech characteristics, named entity characteristics, semantic role characteristics and dependency syntactic relation characteristics.

4. The Mask text matching method based on multitask learning as claimed in claim 1, wherein said BERT based text matching model includes embedding input layer, multi-head attention layer, forward propagation layer and output layer.

5. The Mask text matching method based on multitask learning according to claim 1, wherein the step of obtaining matching results of different texts to be matched comprises:

1) Converting text word features by using an embedding input layer to obtain an embedding input X, and converting the embedding input X into a feature component Q = XW ^Q Characteristic component K = XW ^K Characteristic component V = XW ^V ；W ^Q 、W ^K 、W ^V Weights corresponding to different feature components;

2) Processing the feature component of the embedding input X by using a multi-head attention layer to obtain a multi-head attention layer processing result MultiHead (Q, K, V), namely:

MultiHead(Q,K,V)＝Concat(head ₁ ,...,head _h )W ^O (1)

in the formula, W ^O Is a weight;

wherein, the parameter head _i As follows:

head _i ＝Attention(QW _i ^Q ,KW _i ^k ,VW _i ^v ),i＝1,2,...,h (2)

Attention(QW _i ^Q ,KW _i ^k ,VW _i ^v )＝Mask*Attention(Q,K,V) (3)

in the formula, softmax is an activation function; d is a radical of _k A dimension representing a word vector; mask denotes a Mask; attention (QW) _i ^Q ,KW _i ^k ,VW _i ^v ) Attention (Q, K, V) is an intermediate parameter; h is an integer greater than 0;

3) Processing the multi-head attention layer processing result MultiHead (Q, K, V) by using a forward propagation layer to obtain a forward propagation layer processing result x, that is:

x＝norm(X+MultiHead(Q,K,V)) (5)

4) Processing the processing result x of the forward propagation layer by utilizing an output layer to obtain a text matching model output based on BERT, and taking the text matching model output as matching results of different texts to be matched;

the BERT-based text matching model output is as follows:

FFN(x)＝max(0,xW ₁ +b ₁ )W ₂ +b ₂ (6)

in the formula, W ₁ 、W ₂ Is a weight; b ₁ 、b ₂ Is an offset; FFN (x) is the output.

6. The Mask text matching method based on multitask learning as claimed in claim 5, characterized in that said embedding input X = X ₁ +x ₂ ；

Wherein a component x is input ₁ And an input component x ₂ Respectively as follows:

X ₁ ＝E _tok +E _seg +E _pos (7)

x ₂ ＝embedding1(pos)+embedding2(ner)+embedding3(seg) (8)

in the formula, E _tok 、E _seg 、E _pos Token Embedding codes, positionembedding codes and segmentembedding codes which respectively represent the characteristics of text words; the embedding layers of parts of speech, named entities and semantic roles are represented by embedding1, embedding2 and embedding 3; pos, ner, seg represent part of speech, named body, semantic role encoding of the input text.

7. The Mask text matching method based on multitask learning according to claim 5, wherein the Mask is a variable 0-1, mask masks =1 at the same positions of words of different texts to be matched, and Mask masks =0 at different positions of words of different texts to be matched.

8. The Mask text matching method based on multitask learning according to claim 1, characterized in that the output of said text matching model based on BERT includes sequence output and vector output; the vector output is a classification vector, and the sequence output is a part-of-speech tagging vector; the classification vectors include semantically identical and semantically different.

9. The Mask text matching method based on multitask learning according to claim 1, characterized in that said text matching model based on BERT is pre-trained;

the standard of the pre-training end is Loss function Loss convergence;

the Loss function Loss is as follows:

Loss＝Loss _nll +Loss _pos-tag (9)

in the formula, loss _nll Is a classification vector loss function; loss _pos-tag A loss function labeled for part of speech;

wherein, the Loss function Loss of classification vector _nll As follows:

in the formula, n is the number of training samples; j represents the jth sample; z represents the number of categories classified; c represents the c-th category; h is a total of _j,c Represents the probability that the jth sample belongs to the c-th class; y is _j,c Indicating whether the jth sample belongs to the c category; y is _j,c =1 indicates that the jth sample belongs to the c-th class; y is _j,c =0 indicates that the jth sample does not belong to the c-th class;

loss function Loss of part of speech tagging _pos-tag As follows:

in the formula, P ₁ 、P ₂ 、P ₃ 、P _n The 1 st, 2 nd and n th possible part-of-speech tags corresponding to one sampleScoring of the annotation sequence; p is _real-path And marking the score of the sequence for the real part of speech corresponding to one sample.

10. A computer-readable storage medium, characterized in that the computer-readable medium stores a computer program;

the computer program, when executed by a processor, performs the steps of the method of any one of claims 1 to 9.

Images