CN118152559A - News element extraction method based on large model - Google Patents

Abstract

The invention discloses a news element extraction method based on a large model. By labeling the related information of 5W1H in news, a high-quality data set for news element extraction is constructed and used for a news element extraction model. Extracting elements in news by a labeling person, and then constructing a training format according to the extracting problem and the original text and the sequence of extracting the elements; models LLaMa, vicuna and Guanaco are selected for training respectively, and the models are trained on the data marked in different fields so as to learn the extraction capacity of each news field. The scoring selection module calculates ROUGE and BLEU scores between the elements extracted from the news and the elements marked by the marking personnel by the model to judge the extraction effect. The experimental effect performed on the CNN/DAILYMAIL dataset gave a higher score than ChatGPT. Aiming at the problem of lacking high-quality manually marked 5W1H data sets, the learning capacity of a large model is further mined by constructing high-quality data, so that the level of news extraction is improved.

Description

A method for extracting news elements based on large models

Technical Field

The invention relates to a method for extracting news elements based on a large model, and belongs to the field of natural language processing in the computer field.

Background technique

The extraction of news elements (5W1H) is very important for event extraction and text summarization tasks. These questions cover the most basic information that people care about about a thing or event. By answering these questions, we can fully understand the process of what happened. How to ensure the accuracy of the extracted content on the basis of understanding the news content has always been a very challenging problem. Most of the previous studies extracted the original text based on semantic and grammatical rules. This extraction method not only takes a lot of time but also misses the information of the article. With the continuous development of deep learning, the performance of the model has been greatly improved. In particular, the emergence of ChatGPT has also brought new development directions to the extraction task. Although ChatGPT has shown strong capabilities in semantic understanding and information extraction tasks, for long texts such as news, the content extracted by large models often has problems such as inconsistent facts and missing important information. Therefore, it becomes particularly difficult to extract news elements relying solely on the performance of the large model itself.

Recent studies have shown that fine-tuning large models with manually annotated datasets in specific fields can improve performance. Therefore, the quality and quantity of datasets become particularly important for fine-tuning tasks. However, there is currently no high-quality dataset for news feature extraction. Therefore, we selected 3,500 news datasets from four different fields, manually annotated the 5W1H elements in the news to build a dataset suitable for news feature extraction, and used a large model to efficiently fine-tune the annotated dataset. The final model effect can be applied to news feature extraction tasks in different fields.

Summary of the invention

The present invention aims to solve the following technical problems:

The present invention is dedicated to solving the following difficulties in news element extraction and extracting comprehensive and accurate event elements from news:

-High-quality datasets: The existing news element extraction datasets are missing important elements or are too small in number. There is no high-quality dataset that covers the 5W1H news element extraction.

-Consistency: The content of news element extraction must ensure consistency with the content of the original news text, and there must be no incorrect statements of news facts.

- Comprehensiveness: The extracted news elements can include all the contents of the news event. For example, the original text of an event describes multiple reasons for the occurrence of an event, and there should be no omissions.

The present invention adopts the following technical solutions to solve the technical problems:

A method for extracting news elements based on a large model comprises the following steps:

1) Construct a data set including multiple news items. Labelers classify each news item and extract all sentences or phrases related to 5W1H from each news item as extracted elements. 5W1H refers to Why, What, Where, When, Who, and How.

2) The elements extracted from step 1) are used as labels for model training to construct a fine-tuning instruction dataset format;

3) Put the constructed instruction dataset format into the large model for training to learn the ability to extract news elements.

Preferably, the step 2) of constructing the fine-tuned instruction data set format is:

First, build the required question template:

Question: "Below is an instruction that describes a task."

"Write a response that appropriately completes the request."

"Instruction: please extract what when where why who and how from the news."

Then keep the news in the format of text: news content; finally save the extracted elements in json format and use Response:"output":{element} as the format of the model output result. The final instruction dataset format is:

{“input”:<Question>,”text”:<News>,”Response”:<”output”:{element}>}.

Preferably, in step 3), the constructed data set format is put into the large model for training. Here, we select three large models for training labeled data, namely LLaMA, Vicuna and Guanaco; then the data set including multiple news in step 1) is divided into training set, test set and validation set in a ratio of 8:1:1; QLoRA is used to fine-tune the large model during the training process;

In order to speed up the inference process, the CT2 framework is used. Although int 8 quantization slightly reduces the model effect, it further reduces the time consumption.

In the inference stage, the Top-p sampling algorithm is used to generate sentences. In the decoding process, random sampling is only performed from the minimum set of words whose cumulative probability exceeds the threshold p, where the threshold p is 0.9. At the same time, max_token is set to 2000, so that the large model can output as many elements extracted from the news as possible.

Preferably, ROUGE and BLEU are calculated based on the elements extracted by the LLaMA, Vicuna and Guanaco large models and the elements annotated by the annotators. The higher the ROUGE and BLEU, the closer they are to the sentences annotated by the annotators and the better the effect.

Compared with the prior art, the present invention adopts the above technical solution and has the following beneficial effects:

-The present invention aims to solve the problem of lack of high-quality data sets for news element extraction. By manually labeling news elements in different fields and combining the learning ability of efficient fine-tuning of large models, the model is trained to be able to extract news elements in various fields, so as to better complete the task of news element extraction.

-The present invention verifies that the large model fine-tuned with manually annotated data is stronger than ChatGPT in the ability to extract news elements.

- The present invention tests the situation between four different domain data sets. Even in news from different domains, the extraction capability is also reflected. Therefore, we believe that the extraction capability can be expanded to different news domains.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a schematic diagram of the overall process of the proposed news element extraction framework.

Detailed ways

The present invention is further described in detail below in conjunction with the accompanying drawings.

The news element extraction method proposed in the present invention consists of three modules, and the overall architecture of the method is shown in FIG1 .

(1) Building a dataset module

We used four public news datasets (CNN/DailyMail, XSum, NYT, and RA-MADS) as our annotation objects. We selected 1,000 items from each of the first three datasets and 450 items from the last dataset, for a total of 3,500 items. To facilitate annotation by the annotators, the selected news items were about 550 words.

First, the labelers determine which category the news belongs to. We roughly divide the news categories into the following six categories: ① Accidents and natural disasters, ② Attacks (criminal/terrorist), ③ New technologies, ④ Health and safety, ⑤ Endangered resources, and ⑥ Investigations and trials (criminal/legal/other).

According to the content of the article, the annotators extract all the sentences or phrases related to 5W1H from the original text. In order to ensure the accuracy of the extraction, all the extracted elements are extracted directly from the original text without any semantic generalization. Secondly, for each sentence (if it belongs to the element to be extracted), there must be a unique element corresponding to it. For example, if the sentence is defined as What, it cannot be defined as other cases. This can avoid the ambiguity of the sentence. At the same time, in order to avoid the redundancy of information in each sentence, we select a fragment of the sentence for extraction. For example, the extraction of time and place only needs to find the believed information from the sentence, without having to extract the whole sentence. There are also some types of news in the extraction process that are difficult to identify the elements to which they belong. In order to identify certain elements, we have broadly defined them according to the meaning of the original text.

(2) Model training and inference module

The extracted features obtained by the dataset building module are used as labels for model training. In order to follow the large model fine-tuning method, it is necessary to build a fine-tuning instruction dataset format.

First, we build the required question template. We designed the following template:

-Question: "Below is an instruction that describes a task."

"Write a response that appropriately completes the request."

"Instruction: please extract what when where why who and how from the news."

In order to facilitate the output of the model, we did not ask questions to each element to be extracted in a dialogue manner, but directly let the model extract all the elements. After subsequent experiments, we found that the model can also achieve the extraction effect with this way of asking questions.

Next is the original text input. Due to the limitations of computing power and time, we truncated the news and retained 750 words for each news article. The retained news was retained in the format of text: news content.

Finally, the extracted elements are saved in json format and the format of the output result learned by the model is Response:"output":{element}. The format of the instruction dataset finally constructed is as follows:

-{“input”:<Question>,”text”:<News>,”Response”:<”output”:{element}>}

1) Fine-tuning of large models

The constructed dataset format is put into the large model for training to learn the ability to extract news elements. We selected three large language models: LLaMA, Vicuna and Guanaco.

The LLaMA model is an open and efficient large-scale basic language model released by MetaAI. The entire training dataset contains approximately 1.4T tokens after tokenization. The training datasets used are all public. The Vicuna model is a model fine-tuned using supervised data based on LLaMa-13B. The dataset comes from user conversation data generated by ShareGPT.com, totaling 70K. The Guanaco model is an advanced instruction-following language model built on Meta's LLaMa7B model.

We divide the labeled dataset into training set, test set and validation set in a ratio of 8:1:1. The RA-MADS dataset has only 450 records. We did not divide it but added it to the other three datasets to enhance the training effect of the model. We use QLoRA to fine-tune the model. QLoRA uses a new high-precision technology to quantize the pre-trained model to int 4. It greatly reduces the memory requirements while ensuring the predictive performance of the model. It has become the preferred method for fine-tuning large models.

In order to speed up the model reasoning during the model reasoning stage, we use the CT2 framework to fuse the LORA weights of the fine-tuned model and the weights of the base model into the CT2 format, and further reduce the time consumption by slightly losing the model effect through int 8 quantization, but without affecting the performance of model reasoning.

The generation of large models is uncertain. In the inference stage, we use the Top-p sampling algorithm to generate them. During the decoding process, random sampling is only performed from the minimum set of words whose cumulative probability exceeds a certain threshold p, without considering other words with low probabilities. Because it only focuses on the core part of the probability distribution and ignores the tail part. Here we set p to 0.9, then we only select a word from the minimum set of words with a cumulative probability of 0.9, without considering other words with a cumulative probability less than 0.9. This can avoid sampling some inappropriate or irrelevant words, and also retain some creative words. At the same time, max_token is set to 2000, so that the model outputs as much feature information extracted from the original text as possible.

Even if the Top-p sampling algorithm is used, the generated answers may be inconsistent with the original training format. The strategy we adopt is to search the generated results, and the division of each element of 5W1H can be based on the generated prompt words. For example, What will appear in front of the What part, and there will be a colon after it to indicate that the content extracted from the original text belongs to What. In this way, we can divide each element. If none of the answers given have such prompt words, the results of the model output are considered invalid and regenerated. Finally, we save the generated results in a json file.

2) Zero-shot and few-shot

We tested the ability of ChatGPT and GPT-4 to extract news elements in zero-shot and few-shot scenarios. The experimental results show that the effect after fine-tuning exceeds that of ChatGPT in zero-shot and few-shot scenarios and is close to that of GPT-4 in 5-shot scenarios.

(3) Rating selection module

ROUGE and BLEU are calculated based on the news elements extracted by the model and the elements annotated by the annotators. The higher the result, the closer it is to the sentences annotated by the annotators and the better the effect.

ROUGE is a type of evaluation metric used to automatically evaluate text generation tasks. It focuses on the overlapping information between the generated text and the reference text, and measures the similarity between the two by calculating the recall rate. Specifically, the ROUGE score calculation formula calculates the similarity based on the shared words, shared phrases, and shared sequences of the generated text and the reference text. The ROUGE calculation formula mainly includes three indicators: ROUGE-1, ROUGE-2, and ROUGE-L.

Rouge-N is used to measure the overlap between the automatically generated N-grams (N adjacent words) and the N-grams in the reference summary. The formula is as follows:

Where S is the automatically generated summary set, N is the N-tuple set, Count _match(n,s) is the number of times N-tuple n in the automatically generated summary set appears in the reference summary set, and Count(n) is the number of times N-tuple n appears in the reference summary set.

ROUGE-L is used to measure the longest common subsequence (LCS) between the automatically generated summary and the reference summary. The formula is as follows:

Where S is the set of automatically generated summaries, r is the reference summary, LCS(s,r) is the length of the longest common subsequence between the automatically generated summary s and the reference summary r. |s| is the length of the automatically generated summary s.

BLEU was first used in machine translation tasks to evaluate the rationality of machine translated sentences. Specifically, BLEU is calculated by measuring the overlap between the generated sequence and the reference sequence. The formula is as follows:

Where l _c represents the length of the machine translation text, and l _s represents the effective length of the reference answer. When there are multiple reference texts, the length closest to the translation text is selected. When the translation text length is greater than the reference text length, the penalty coefficient is 1.

The prototype system of BLEU adopts uniform weighting, and the upper limit of N is 4, which means that at most 4-gram accuracy is counted.

For the results of BLEU and ROUGE indicators, the model with higher score is selected as the benchmark model for extracting elements of news in other fields.

The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims (4)

1. A method for extracting news elements based on a large model, which is characterized by comprising the following steps:

1) Constructing a data set comprising a plurality of news, classifying each news by a labeling person, and extracting all sentences or phrases related to 5W1H from each news as extracted elements, wherein the 5W1H refers to a reason Why, an object What, a place while, a time while, a person wha and a method How;

2) The elements extracted in the step 1) are used as labels for model training, and a fine-tuning instruction data set format is constructed;

3) And putting the constructed instruction data set format into a large model for training to learn the capability of extracting news elements.

2. The method of claim 1, wherein the step of 2) constructing a fine-tuned instruction data set format is:

Firstly, constructing a required problem template:

Question: "Below is an instruction that describes a task:" i.e. problem: "following are instructions describing tasks; "

"Write a response that appropriately completes the request" i.e. "write a response to the appropriate completion request; "

"Construction: simple extract WHAT WHEN WHERE WHY who and how from the news," i.e. "description: please extract objects, time, place, reason, personnel and method from news; "

Then reserving news according to the text, namely reserving the format of news content; finally, the extracted elements are stored in json format, response: "output": { elements } is used as the format of the model output result, and the format of the finally obtained instruction data set is as follows:

{ "input": < Question >, "text": < news >, "Response": < "output": { element }.

3. A method of large model based news element extraction as claimed in claim 2, wherein in step 3), the constructed dataset format is put into a large model for training, where we select three large models for training annotation data, LLaMA, vicuna and Guanaco respectively; the dataset comprising a plurality of news in step 1) is then processed according to 8:1:1, dividing a training set, a test set and a verification set in proportion; QLoRA is used in the training process to fine tune the large model;

In order to accelerate the reasoning speed in the reasoning stage, a CT2 framework is used, and the time consumption is further reduced through the int 8 quantization;

In the reasoning stage, a Top-P SAMPLING algorithm is adopted to generate sentences, and in the decoding process, random sampling is only carried out from the minimum word set with the cumulative probability exceeding a threshold p, wherein the value of the threshold p is 0.9; at the same time, max_token is set to 2000.

4. A method of large model news element extraction according to claim 3, wherein the elements extracted from the large models LLaMA, vicuna and Guanaco are calculated ROUGE and BLEU with the elements marked by the marking person, and the higher the results ROUGE and BLEU indicate the closer to the sentence or phrase marked by the marking person, the better the effect.