CN115658881A - Sequence-to-sequence text summary generation method and system based on causality - Google Patents

Abstract

The invention provides a sequence-to-sequence text abstract generation method and system based on causal relation, and belongs to the field of natural language processing and automatic text abstract generation. The method is inspired by the causal theory, and the causal relationship of each element in the abstract task is researched from the data generation perspective. Firstly, introducing two non-observable variables to obtain a structural cause-and-effect model of an abstract task; and then obtaining a corresponding sequence to a sequence generation framework according to the structure cause and effect model, wherein the corresponding sequence is used for modeling the generation process of the original text and the abstract. The framework comprises three core modules: a double hidden variable variation coder, a textual reconstruction decoder and a digest prediction decoder. Compared with the existing end-to-end deep text summarization method, the method has stronger interpretability, better summarization performance and stronger generalization capability. The method is a sequence-to-sequence framework with strong applicability, and therefore can be migrated to more model bodies, generation tasks and different data sets.

Description

Sequence-to-sequence text summary generation method and system based on causality

technical field

The invention belongs to the technical field of sequence-to-sequence text summarization in the text summarization technology of natural language processing, and particularly relates to a sequence-to-sequence text summarization method and system based on causality.

Background technique

Automatic text summarization technology aims to automatically identify important topics and key information from input documents, and generate accurate, concise and fluent text as a summary. Early summarization techniques were mainly explored around artificially designed heuristic rules and templates; today, deep text summarization techniques based on neural networks have become mainstream. It learns matching patterns of source text and summarization end-to-end through supervised training, which can be divided into extractive and generative summarization methods.

The deep extractive summarization method extracts a subset of sentences in the original text as the summary text. It usually converts the summarization task into a sequence labeling or sorting task, and selects key sentences to form a summary by classifying each candidate sentence in the document, or sorting the candidate sentences according to their importance. The deep generative summarization method regards the summarization task as a sequence-to-sequence generation task, and uses words or phrases as the basic generation unit to generate summaries from scratch. Since the abstract text is not limited to the expression of the original text, the abstract generated by this method has higher flexibility and diversity, and stronger applicability.

In recent years, the pre-training model with Transformer as the main structure has further brought performance gains to deep text summarization methods. Generative methods inherit excellent language expressiveness from pre-trained models. Combined with the advantages of its own strong applicability, the generative summarization method based on the pre-trained model has gradually become a mainstream research topic.

The generative summarization methods based on pre-trained models have the following disadvantages.

(1) First, the data-driven methods in the prior art lack interpretability. This is because in the end-to-end training method, the user only needs to feed the model samples and their labels, and the model automatically learns the matching mode of the original text and the abstract. This "black box mode" is incomprehensible to the user.

(2) However, summaries generated by prior art often contain redundant information. These redundant information are not the core information in the original text. This is due to the fact that existing techniques tend to exploit all correlations in the dataset when learning the matching patterns of the original text and the abstract, which contain a large number of pseudo-correlations, inheriting the statistical bias in the training corpus. At this time, the model is easily misled by the easy-to-learn surface features in the data set, such as text pairs that co-occur frequently but do not have necessary connections. For example, "red maple leaf" is a high-frequency co-occurrence text pair. "Red" and "maple leaf" have a high statistical correlation, but do not have an inevitable causal relationship. "Maple leaf" can also be "green".

Contents of the invention

The purpose of the present invention is to solve the problem that the above-mentioned prior art relies too much on correlation and is easily misled by false correlation, and proposes a sequence-to-sequence framework inspired by causality theory.

The present invention also proposes a sequence-to-sequence text summary generation method based on causality, which includes:

Step 1, inputting the original document into a neural network-based double hidden variable variational encoder, the double hidden variable variational encoder performs multiple sampling of the original document to extract abstract-related features and abstract-independent features in the original document;

Step 2. Concatenate the abstract-related features and abstract-independent features to obtain abstract comprehensive features. Based on the abstract comprehensive features, the original document is reconstructed to obtain a reconstructed document, and the original document is used as the training target. Based on the reconstructed document and The training objective constructs a loss function and trains a double hidden variable variational encoder;

Step 3: Using the trained double hidden variable variational encoder to extract the summary-related features of the original document as target features, and obtain the text summary of the original document based on the target features.

The sequence-to-sequence text summary generation method based on causality, wherein the step 1 includes:

Obtain the encoded representation vector of the document through the document encoder, and the document encoder in the double hidden variable variational encoder encodes the original document X to obtain the encoded representation vector h _doc of the document;

The variational encoder in the double hidden variable variational encoder module encodes and samples the coded representation vector h _doc of the document to obtain hidden variables h _c and h _nc respectively;

Input h _c and h _nc into the original text reconstruction decoder to obtain the output o _i at each position; superimpose h _c and h _nc and o _i , input the output layer of the language model to generate the reconstructed original document X′; according to X′ and X to calculate the reconstruction loss L _R ; store the hidden variable representations h _c and h _nc corresponding to the minimum L _R as the summary-related features and the summary-independent features, respectively.

The sequence-to-sequence text summary generation method based on causality, wherein the step 3 includes:

The abstract predictive decoder calculates the output r _j at each position according to the target feature; superimposes the target feature and r _j , and inputs it into the output layer of the language model to obtain the vocabulary distribution probability to generate the corresponding words to form the text summary.

The causality-based sequence-to-sequence text summary generation method, wherein the training methods of the double hidden variable variation encoder, the summary prediction decoder and the original text reconstruction decoder are:

和

Obtain the training document and the corresponding reference abstract, input the training document into the double hidden variable variational encoder based on the neural network, and obtain the hidden variable representations h _c and h _nc ; at the same time, obtain the Gaussian distribution of h _c and h _nc respectively

and

Use hidden variables to represent h _c and h _nc to generate reconstructed original text; and use hidden variables to represent h _c to generate summary results;

和该高斯分布

计算KL散度；Construct the original text reconstruction loss L _R based on the training document and reconstructed original text; construct the summary prediction loss L _P based on the reference summary and the summary result; based on the standard normal distribution

and the Gaussian distribution

Calculate the KL divergence;

The final training loss function is L=L _R +L _P +λL _KL , where λ is used to adjust the degree of distribution normalization constraints. Based on the training loss function, the dual hidden variable variational encoder and the summary prediction decoder are trained for L and reconstruct the decoder from the original text.

The present invention also proposes a sequence-to-sequence text summary generation system based on causality, which includes:

The sampling module is used to input the original document into a neural network-based double hidden variable variation encoder, and the double hidden variable variation encoder performs multiple sampling on the original document to extract abstract-related features and abstract-independent features in the original document;

The splicing module is used to splicing the abstract-related features and the abstract-independent features to obtain the abstract comprehensive features, reconstruct the original document based on the abstract comprehensive features, and obtain the reconstructed document, and take the original document as the training target, based on the reconstructed The document and the training target construct a loss function to train a double hidden variable variational encoder;

The extraction module is used to use the trained double hidden variable variational encoder to extract the summary-related features of the original document as target features, and obtain the text summary of the original document based on the target features.

The sequence-to-sequence text summarization system based on causality, wherein the sampling module is used for:

Obtain the encoded representation vector of the document through the document encoder, and the document encoder in the double hidden variable variational encoder encodes the original document X to obtain the encoded representation vector h _doc of the document;

The variational encoder in the double hidden variable variational encoder module encodes and samples the coded representation vector h _doc of the document to obtain hidden variables h _c and h _nc respectively;

Input h _c and h _nc into the original text reconstruction decoder to obtain the output o _i at each position; superimpose h _c and h _nc and o _i , input the output layer of the language model to generate the reconstructed original document X′; according to X′ and X to calculate the reconstruction loss L _R ; store the hidden variable representations h _c and h _nc corresponding to the minimum L _R as the summary-related features and the summary-independent features, respectively.

The sequence-to-sequence text summarization system based on causality, wherein the extraction module is used for:

The abstract predictive decoder calculates the output r _j at each position according to the target feature; superimposes the target feature and r _j , and inputs it into the output layer of the language model to obtain the vocabulary distribution probability to generate the corresponding words to form the text summary.

The causality-based sequence-to-sequence text summary generation system, wherein the training methods of the double hidden variable variation encoder, the summary prediction decoder and the original text reconstruction decoder are:

和

Obtain the training document and the corresponding reference abstract, input the training document into the double hidden variable variational encoder based on the neural network, and obtain the hidden variable representations h _c and h _nc ; at the same time, obtain the Gaussian distribution of h _c and h _nc respectively

and

Use hidden variables to represent h _c and h _nc to generate reconstructed original text; and use hidden variables to represent h _c to generate summary results;

和该高斯分布

计算KL散度；Construct the original text reconstruction loss L _R based on the training document and reconstructed original text; construct the summary prediction loss L _P based on the reference summary and the summary result; based on the standard normal distribution

and the Gaussian distribution

Calculate the KL divergence;

The final training loss function is L=L _R +L _P +λL _KL , where λ is used to adjust the degree of distribution normalization constraints. Based on the training loss function, the dual hidden variable variational encoder and the summary prediction decoder are trained for L and reconstruct the decoder from the original text.

The present invention also proposes a storage medium for storing a program for executing any one of the sequence-to-sequence text summarization methods based on causality.

The present invention also proposes a client, which is used for any of the sequence-to-sequence text summary generation systems based on causality.

It can be seen from the above scheme that the advantages of the CI-Seq2Seq framework of the present invention compared with the prior art are:

(1) Stronger interpretability. Compared with the prior art which generally learns the representation of the original text, the method proposed in this patent can effectively distinguish and extract the latent variable representations h _c and h _nc of SC and SNC in the original text. We use t-SNE visual analysis to verify the distribution of the learned hidden variables, as shown in Figure 1. From Figure 1, it can be seen that the distribution spaces represented by the learned h _c and h _nc are significantly different: the distribution space of h _c is more concentrated, while the distribution space of h _nc is more dispersed. This proves that SC and SNC have learned core information and information representing diversity, respectively.

(2) Better summarization performance. Compared with the prior art, the method proposed in this patent can effectively improve the quality of summary generation. We conduct experiments on publicly available summarization datasets CNN/DM and XSUM to compare CI-Seq2Seq with other baseline models. The baseline models for comparison include two Unified VAE-PGN and VHTM, which are also based on variational autoencoders, three classic general-purpose pre-training models T5, BART, and GLM, and methods such as contrastive learning, confrontational learning, and length control Implemented CLIFF, Debiased-Ext and PtLAAM. The evaluation metrics are Rouge-1, Rouge-2, and Rouge-L, which measure the recall of words, bigrams, and longest common subsequences, respectively. The comparison results are shown in Table 1. According to Table 1, it can be seen that not only compared with other models based on variational autoencoders, general and powerful pre-training models, but also compared with models using other training techniques, our proposed method has better performance on the two data sets. All indicators can be significantly improved.

Table 1: Summary performance comparison on CNN/DM and XSUM datasets

(3) Stronger generalization ability. Compared with the prior art, the method proposed in this patent can effectively improve the generalization ability of the model. We use the model trained on CNN/DM for the XSUM test set, and the model trained on XSUM for the CNN/DM test set to test the performance capabilities of different models on unseen datasets. Experimental results See Table 2. By comparing with the original framework BART, our method can get a certain improvement, which proves that our model has stronger generalization ability.

Table 2: Comparison of generalization performance of crossover experiments on CNN/DM and XSUM datasets

Description of drawings

Figure 1 is a schematic diagram of t-SNE visual analysis;

Figure 2 is a general flowchart of generating summary text using CI-Seq2Seq;

Figure 3 is a flow chart of obtaining the initial hidden variable representation sum through n sampling when using CI-Seq2Seq to generate summary text;

Figure 4 is a flowchart of obtaining the optimal hidden variable representation sum through k rounds of iterative updates when CI-Seq2Seq is used to generate summary text;

Figure 5 is a flow chart of generating a summary using the optimal hidden variable representation when using CI-Seq2Seq to generate a summary text;

Fig. 6 is a diagram of the training process of the method of the present invention.

Detailed ways

When the inventor combined the causal theory to study the text summarization task, he found that the defect of generating redundant summaries in the prior art is caused by the over-reliance on the correlation relationship in the data set. Existing techniques rely too much on the correlation of components in the summarization task, without considering the more essential causal relationship. The causal relationship describes the underlying data generation process, and it is difficult for existing technologies to describe the natural causal relationship behind the data only by relying on observable variables.

The present invention introduces two unobservable variables and uses causal perception to model the corresponding data generation process, so as to solve the problem of excessive dependence on correlation relationship and easy misleading by false correlation in the prior art. Specifically, the present invention designs a structural causal model for the summarization task, in which the variables involved include two unobservable variables in addition to the observable original text and abstract, respectively representing the information that determines the content of the abstract (summary-causal factors , SC), such as the core information in the original text, and other information (summary-non-causal factors, SNC) in the corpus that do not determine the content of the summary, such as marginal information that leads to the diversity of the original text. The fundamental difference between the two unobservable variables is that only the former SC is responsible for the summary. By explicitly distinguishing the two kinds of information, the present invention designs a sequence-to-sequence framework (Causality Inspired Sequence-to-Sequence model, CI-Seq2Seq) inspired by causality theory to describe the generation process of abstracts and texts. When generating abstracts Only SC is utilized, while both are used when generating source text. In terms of model structure, the present invention makes the following improvements to the encoder-decoder architecture in the traditional sequence-to-sequence framework: the single encoder and single decoder are improved to double hidden variable variational encoders, original text reconstruction decoders and Digest predictive decoder. Specifically, the present invention includes the following key technical points:

Key point 1, the text summarization structural causal model designed from the perspective of the data generation process; technical effect: the text summarization method designed according to the structural causal model can explicitly distinguish two kinds of information, learn the respective representations of SC and SNC, and have Stronger interpretability;

Key point 2, CI-Seq2Seq, a sequence-to-sequence framework inspired by causality theory; technical effect: the framework can effectively distinguish SC and SNC and obtain their respective representations, where the representation of SC is used to generate summaries, and both SC and SNC are used to generate Original text; The framework is implemented based on a supervised variational autoencoder, which can effectively improve the quality of generated summaries. The core modules include: double hidden variable variational encoder: use the neural network to encode the original text into the hidden variable representation of SC and SNC; original text reconstruction decoder: use the neural network to decode the hidden variable representation of SC and SNC, Used to reconstruct the original text; abstract prediction decoder: use the neural network to decode the latent variable representation of SC to predict the abstract.

In order to make the above-mentioned features and effects of the present invention more clear and understandable, the following specific examples are given together with the accompanying drawings for detailed description as follows.

As shown in Figure 2, it is the corresponding overall flow chart. The overall flow of the present invention to generate abstract text includes the following steps:

Step S1: Input the original document X, and input the original document X into the neural network-based double hidden variable variational encoder module.

Step S2: Obtain initial hidden variable representations h _c0 and h _nc0 through n times of sampling. The specific process is shown in Figure 3, including the following sub-steps:

Step S201: Obtain the encoded representation vector h _doc of the document through the document encoder. The document encoder in the double hidden variable variational encoder module encodes the input original document X to obtain the encoded representation vector h _doc of the document.

μ_nc和

)。Step S202: Gaussian distributions of latent variables SC and SNC are obtained through a variational encoder. The variational encoder in the double hidden variable variational encoder module further encodes the coded representation vector h _doc of the document, and obtains the Gaussian distribution of hidden variables SC and SNC (the distribution parameters are μ _c ,

μ _nc and

).

再根据如下公式分别得到所需的隐变量表示h_c和h_nc：Step S203: Obtain hidden variable representations h _c and h _nc through the hidden variable sampler. The latent variable sampler performs sampling according to the Gaussian distribution obtained in the previous step, and obtains the representations h _c and h _nc of latent variables SC and SNC respectively. The sampling process is completed through a classic reparametrization technique. Specifically, the variables ε _c and ε _nc are first sampled from the standard normal distribution, namely

Then obtain the required latent variable representations h _c and h _nc according to the following formulas:

h _c =μ _c +σ _c *ε _c ,

h _nc =μ _nc +σ _nc *ε _nc .

Step S204: Input the h _cnc obtained by splicing the latent variable representation h _c and h _nc into the original text reconstruction decoder. Concatenate the hidden variable representations h _c and h _nc obtained in the previous step to obtain h _cnc , which is used as the initial input of the original text reconstruction decoder.

Step S205: Obtain o _i through the original text reconstruction decoder. The original text reconstruction decoder calculates the output o _i at each position according to the initial input. The position is the original text word sequence, in order from left to right, each position corresponds to a word.

Step S206: superimpose h _cnc and o _i , and input them into the output layer of the language model. Superimpose the h _cnc obtained in step S204 on the o _i obtained in the previous step as the input of the language model output layer. The language model output layer is an existing technology, which is implemented based on linear transformation and beam search.

Step S207: Generate the reconstructed original document X' through the output layer of the language model. The language model output layer maps the results obtained in the previous step to the vocabulary through linear transformation to obtain the distribution probability of the vocabulary, and generates the corresponding words to form the original document X′ according to the beam search method.

Step S208: Calculate the reconstruction loss _LR according to X' and X. Compare the reconstructed original document X′ with the input original document X, and calculate the reconstruction loss L _R .

Generating the original text is to obtain the best SC representation h _c and SNC representation h _nc for the current original text, where h _c will be used for abstract generation. Because the original text is given during training and testing, the present invention uses the original text as a supervisory signal: after reconstructing the original text, compare the generated original text X′ with the given original text X, and determine the optimal Hidden variables represent h _c and h _nc , where h _c will be used for summary generation. That is to say, the accurate SC representation can not only work with SNC to reconstruct the original text information well, but also can work alone to generate a summary. When an SC representation can well reconstruct the original text, it is also suitable for generating summaries.

Step S209: Store hidden variable representations h _c and h _nc corresponding to the current minimum _LR . Compare the size of _LR in this cycle with the current minimum _LR , and store the hidden variable representations h _c and h _nc corresponding to the minimum _LR after this cycle.

Step S210: Obtain initial hidden variable representations h _c0 and h _nc0 . Repeat the above steps (S203-S209) n times, and return the hidden variable representations h _c and h _nc that minimize the reconstruction loss _LR through multiple samplings as initial hidden variable representations h _c0 and h _nc0 .

Step S3: Obtain the optimal hidden variable representations h _c and h _nc through k rounds of iterative updating. As shown in Figure 4, the specific process includes the following sub-steps:

Step S301: Use initial hidden variable representations h _c0 and h _nc0 to initialize hidden variable representations h _c and h _nc . According to the initial hidden variable representations h _c0 and h _nc0 obtained in step S2, the representations h _c and h _nc of the hidden variables SC and SNC are initialized.

Step S302: Input the h _cnc obtained by splicing the latent variable representation h _c and h _nc into the original text reconstruction decoder. Concatenate the hidden variable representations h _c and h _nc obtained in the previous step to obtain h _cnc , which is used as the initial input of the original text reconstruction decoder.

Step S303: Obtain o _i through the original text reconstruction decoder. The original text reconstruction decoder calculates the output o _i at each position according to the initial input.

Step S304: superimpose h _cnc and o _i , and input them into the output layer of the language model. Superimpose the h _cnc obtained in step S302 on the o _i obtained in the previous step as the input of the output layer of the language model.

Step S305: Generate a reconstructed original document X' through the language model output layer. The language model output layer maps the results obtained in the previous step to the vocabulary, obtains the distribution probability of the vocabulary, and generates the corresponding words to form the original document X′.

Step S306: Calculate the reconstruction loss _LR according to X' and X. Compare the reconstructed original document X′ with the input original document X, and calculate the reconstruction loss L _R .

Step S307: Optimizing to obtain new hidden variable representations h _c and h _nc . The hidden variable representations h _c and h _nc are optimized according to the reconstruction loss _LR and the Adam optimization algorithm. The specific optimization process is as follows: first, set the hidden variable representation as a learnable parameter, and then set a special Adam optimizer for it; for the reconstruction loss L _R calculated in the previous step, the gradient is returned through the backpropagation algorithm, and finally The learnable latent variable representations h _c and h _nc are directly updated according to the Adam algorithm.

Step S308: Obtain optimal hidden variable representations h _c and h _nc . The above steps (S302-S307) are repeated k times, and through multiple optimizations, the optimal hidden variable representations h _c and h _nc are returned.

In summary, the purpose of steps S2 and S3 is to obtain accurate hidden variable representation, and the evaluation basis is the reconstruction loss. S2 is used as the preceding step of S3 for the initialization of the latter. Technically, the way it obtains hidden variables is multiple sampling. S3 performs subsequent optimization operations based on the optimal initialization vector obtained by S2. The specific optimization process is as follows: first, set the hidden variable representation as a learnable parameter, and then set a special Adam optimizer for it; for the reconstruction loss L _R calculated in the previous step, the gradient is returned through the backpropagation algorithm, and finally The learnable latent variable representations h _c and h _nc are directly updated according to the Adam algorithm. Good initialization is crucial for the optimization process, and optimization operations take longer. Therefore, in S2, it is faster to select the current best representation by means of multiple sampling.

Step S4: Use the optimal latent variable representation h _c to generate a summary Y'. As shown in Figure 5, the specific process includes the following sub-steps:

Step S401: Input the optimal latent variable representation _hc into the digest prediction decoder. The optimal hidden variable representation h _c obtained in step S3 is used as the initial input of the digest prediction decoder.

Step S402: Obtain r _j through the digest prediction decoder. The summary predictive decoder computes the output r _j at each position based on the initial input.

Step S403: superimpose h _c and r _j , and input them into the output layer of the language model. Superimpose the h _c of step S401 on the r _j obtained in the previous step, and use it as the input of the output layer of the language model.

Step S404: Generate a predicted summary Y' through the output layer of the language model. The language model output layer maps the results obtained in the previous step to the vocabulary, obtains the distribution probability of the vocabulary, and generates the corresponding word composition summary Y′.

The above is the use process of the present invention, and the training process of the method of the present invention is described as follows. The corresponding training flow chart is shown in Figure 6.

Step S1: Input original document X and reference abstract Y. Input the original document X into the neural network-based double hidden variable variational encoder module.

和

(分布参数μ_c、μ_nc为均值，

为方差)。Step S2: Obtain hidden variable representations h _c and h _nc through sampling. For specific steps, refer to S201-S203 of the usage process. This step will get the Gaussian distribution of latent variables SC and SNC

and

(distribution parameters μ _c and μ _nc are mean values,

is the variance).

Step S3: Use hidden variables to represent h _c and h _nc to generate the original text X'. For specific steps, refer to S204-S207 of the usage process.

Step S4: Generate summary Y' using hidden variable representation _hc . For specific steps, refer to S4 of the usage process.

Step S5: Calculate the training loss and optimize the model.

The training loss consists of three parts: original text reconstruction loss _LR , summary prediction loss _LP , and distribution constraint loss L _KL .

The original text reconstruction loss _LR is the cross-entropy calculated based on the input original document X and the reconstructed X′, the purpose is to train the double hidden variable variational encoder and the original text reconstruction decoder.

The digest prediction loss _LP is the cross-entropy computed based on the input reference digest Y and the predicted Y′ for the purpose of training a dual hidden variable variational encoder and a digest predictive decoder.

(0，I)和隐变量SC/SNC的高斯分布

(由步骤S2得到)计算的KL散度，目的是为了训练双隐变量变分编码器，使预测的变量分布更加规范化。The distribution constrained loss L _KL is based on the standard normal distribution

Gaussian distribution of (0, I) and latent variable SC/SNC

The calculated KL divergence (obtained by step S2) is aimed at training a double hidden variable variational encoder to make the predicted variable distribution more normalized.

The final training loss function is L=L _R +L _P +λL _KL , where λ is used to adjust the degree of distribution normalization constraints.

According to the above training loss function, the Adam optimizer is used to train the model.

The following are system embodiments corresponding to the foregoing method embodiments, and this implementation manner may be implemented in cooperation with the foregoing implementation manners. The relevant technical details mentioned in the foregoing implementation manners are still valid in this implementation manner, and will not be repeated here in order to reduce repetition. Correspondingly, the relevant technical details mentioned in this implementation manner may also be applied in the foregoing implementation manners.

The present invention also proposes a sequence-to-sequence text summary generation system based on causality, which includes:

The sampling module is used to input the original document into a neural network-based double hidden variable variation encoder, and the double hidden variable variation encoder performs multiple sampling on the original document to extract abstract-related features and abstract-independent features in the original document;

The splicing module is used to splicing the abstract-related features and the abstract-independent features to obtain the abstract comprehensive features, reconstruct the original document based on the abstract comprehensive features, and obtain the reconstructed document, and take the original document as the training target, based on the reconstructed The document and the training target construct a loss function to train a double hidden variable variational encoder;

The extraction module is used to use the trained double hidden variable variational encoder to extract the summary-related features of the original document as target features, and obtain the text summary of the original document based on the target features.

The sequence-to-sequence text summarization system based on causality, wherein the sampling module is used for:

Obtain the encoded representation vector of the document through the document encoder, and the document encoder in the double hidden variable variational encoder encodes the original document X to obtain the encoded representation vector h _doc of the document;

The variational encoder in the double hidden variable variational encoder module encodes and samples the coded representation vector h _doc of the document to obtain hidden variables h _c and h _nc respectively;

Input h _c and h _nc into the original text reconstruction decoder to obtain the output o _i at each position; superimpose h _c and h _nc and o _i , input the output layer of the language model to generate the reconstructed original document X′; according to X′ and X to calculate the reconstruction loss L _R ; store the hidden variable representations h _c and h _nc corresponding to the minimum L _R as the summary-related features and the summary-independent features, respectively.

The sequence-to-sequence text summarization system based on causality, wherein the extraction module is used for:

The abstract predictive decoder calculates the output r _j at each position according to the target feature; superimposes the target feature and r _j , and inputs it into the output layer of the language model to obtain the vocabulary distribution probability to generate the corresponding words to form the text summary.

The causality-based sequence-to-sequence text summary generation system, wherein the training methods of the double hidden variable variation encoder, the summary prediction decoder and the original text reconstruction decoder are:

和

Obtain the training document and the corresponding reference abstract, input the training document into the double hidden variable variational encoder based on the neural network, and obtain the hidden variable representations h _c and h _nc ; at the same time, obtain the Gaussian distribution of h _c and h _nc respectively

and

Use hidden variables to represent h _c and L _nc to generate reconstructed original text; and use hidden variables to represent h _c to generate summary results;

和该高斯分布

计算KL散度；Construct the original text reconstruction loss L _R based on the training document and reconstructed original text; construct the summary prediction loss L _P based on the reference summary and the summary result; based on the standard normal distribution

and the Gaussian distribution

Calculate the KL divergence;

The final training loss function is L=L _R +L _P +λL _KL , where λ is used to adjust the degree of distribution normalization constraints. Based on the training loss function, the double hidden variable variational encoder and the summary prediction decoder are trained for L and reconstruct the decoder from the original text.

The present invention also proposes a storage medium for storing a program for executing any one of the sequence-to-sequence text summarization methods based on causality.

The present invention also proposes a client, which is used for any of the sequence-to-sequence text summary generation systems based on causality.

Claims (10)

1. A sequence-to-sequence text summary generation method based on causal relationship is characterized by comprising the following steps:

step 1, inputting an original document into a dual-hidden variable variation coder based on a neural network, wherein the dual-hidden variable variation coder samples the original document for multiple times to extract abstract related features and abstract irrelevant features in the original document;

step 2, splicing the abstract relevant features and the abstract irrelevant features to obtain abstract comprehensive features, reconstructing an original document based on the abstract comprehensive features to obtain a reconstructed document, constructing a loss function based on the reconstructed document and the training target by taking the original document as a training target, and training a double-hidden-variable variational encoder;

and 3, extracting the abstract relevant characteristics of the original document by adopting the trained double-hidden variable variational encoder to serve as target characteristics, and obtaining the text abstract of the original document based on the target characteristics.

2. The method for generating a sequence-to-sequence text summary based on causal relationship of claim 1, wherein the step 1 comprises:

obtaining the coding expression vector of the document through a document coder, coding the original document X through the document coder in the double hidden variable variation coder to obtain the coding expression vector h of the document _doc ；

Variational encoding in dual latent variational encoder modulesEncoder-to-document encoded representation vector h _doc Coding and sampling are carried out to respectively obtain hidden variables h _c And h _nc ；

H is to be _c And h _nc Input the original text reconstruction decoder to obtain an output o at each position _i (ii) a Superposition h _c And h _nc And o _i Inputting a language model output layer to generate a reconstructed original document X'; calculating the reconstruction loss L from X' and X _R (ii) a Store L _R Minimum corresponding hidden variable representation h _c And h _nc As the abstract-related feature and the abstract-independent feature, respectively.

3. The method for generating a sequence-to-sequence text summary based on causal relationship of claim 2, wherein the step 3 comprises:

the abstract prediction decoder calculates the output r at each position according to the target characteristics _j (ii) a Superimposing the target feature and r _j And inputting the language model output layer to obtain the word list distribution probability so as to generate corresponding words to form the text abstract.

4. The method as claimed in claim 3, wherein the dual hidden variable variant coder, the digest predictive decoder and the original text reconstruction decoder are trained by:

obtaining a training document and a corresponding reference abstract, inputting the training document into a neural network-based double-hidden-variable variation encoder to obtain a hidden-variable representation h _c And h _nc (ii) a Simultaneously obtain h respectively _c And h _nc Gaussian distribution of

And

using hidden variables to represent h _c And h _nc Generating a reconstructed original text; and using hidden variable representationsh _c Generating a summary result;

constructing an original reconstruction loss L based on a training document and a reconstructed original _R (ii) a Constructing a summary prediction loss L based on the reference summary and the summary result _P (ii) a Based on a standard normal distribution

And the Gaussian distribution

Calculating KL divergence;

final training loss function L = L _R +L _P +λL _KL Wherein λ is used to adjust the degree of distribution normalization constraint, training the bilatent variable variational encoder, the digest prediction decoder, and the textual reconstruction decoder based on the training loss function L.

5. A causal relationship-based sequence-to-sequence text summary generation system, comprising:

the sampling module is used for inputting the original document into a double-hidden variable variational encoder based on a neural network, and the double-hidden variable variational encoder samples the original document for multiple times to extract abstract related features and abstract irrelevant features in the original document;

the splicing module is used for splicing the abstract related features and the abstract irrelevant features to obtain abstract comprehensive features, reconstructing an original document based on the abstract comprehensive features to obtain a reconstructed document, taking the original document as a training target, constructing a loss function based on the reconstructed document and the training target, and training a double-latent variable variational encoder;

and the extraction module is used for extracting the related features of the abstract of the original document by using the trained double-hidden variable variational encoder as target features and obtaining the text abstract of the original document based on the target features.

6. The causal relationship-based sequence-to-sequence text summary generation system of claim 5, wherein the sampling module is configured to:

obtaining the coding expression vector of the document through a document coder, coding the original document X through the document coder in the double hidden variable variation coder to obtain the coding expression vector h of the document _doc ；

Vector h for representing coding of document by variational coder in dual hidden variational coder module _doc Coding and sampling are carried out to respectively obtain hidden variables h _c And h _nc ；

H is to be _c And h _nc Input the original text reconstruction decoder to obtain an output o at each position _i (ii) a Superposition h _c And h _nc And o _i Inputting a language model output layer to generate a reconstructed original document X'; calculating the reconstruction loss L from X' and X _R (ii) a Store L _R Minimum corresponding hidden variable representation h _c And h _nc As the abstract-related feature and the abstract-independent feature, respectively.

7. The causal relationship-based sequence-to-sequence text summary generation system of claim 6, wherein the extraction module is configured to:

the abstract prediction decoder calculates the output r at each position according to the target characteristics _j (ii) a Superimposing the target feature and r _j And inputting the language model output layer to obtain the word list distribution probability so as to generate corresponding words to form the text abstract.

8. The system of claim 7, wherein the dual hidden variable variant coder, the digest predictive decoder and the textual reconstruction decoder are trained by:

obtaining a training document and a corresponding reference abstract, inputting the training document into a neural network-based dual-hidden variable variation coder to obtain a hidden variable representation h _c And h _nc (ii) a Simultaneously obtain h respectively _c And h _nc Gaussian distribution of

And

using hidden variables to represent h _c And h _nc Generating a reconstructed original text; and using hidden variables to represent h _c Generating a summary result;

constructing an original reconstruction loss L based on a training document and a reconstructed original _R (ii) a Constructing a summary prediction loss L based on the reference summary and the summary result _P (ii) a Based on a standard normal distribution

And the Gaussian distribution

Calculating KL divergence;

the final training loss function is L = L _R +L _P +λL _KL Wherein λ is used to adjust the degree of distribution normalization constraint, training the bilatent variable variational encoder, the digest prediction decoder, and the plaintext reconstruction decoder for L based on the training loss function.

9. A storage medium storing a program for executing the cause-and-effect based sequence-to-sequence text digest generation method according to any one of claims 1 to 4.

10. A client for use in any one of the causality-based sequence-to-sequence text summary generation systems of claims 5 to 8.

Images