CN108491512A - The method of abstracting and device of headline - Google Patents

Abstract

The invention provides a method and device for summarizing news headlines. The method includes: obtaining the original headline of the news, performing lexical and syntactic analysis on the original headline of the news, and obtaining the analysis result; based on the analysis result, extracting the main content of the sentence in the original headline of the news, and using the main content of the sentence as the news Candidate headlines: using the abstract quality evaluation strategy of news headlines to evaluate the quality of the news candidate headlines, and then determine the news abstract headlines according to the evaluation results. The embodiment of the present invention utilizes lexical and syntactic analysis to compress and summarize news headlines, so that the main content of the news headlines is extracted while retaining key information in the original news headlines as much as possible, and more accurate and rigorous news headlines can be obtained.

Description

Summary method and device for news headlines

technical field

The invention relates to the technical field of Internet applications, in particular to a method and device for summarizing news headlines.

Background technique

In today's Internet with a huge amount of information, when network users use search engines to search for news, they generally filter the content they need based on the content and description of the news title, and then generate click behavior. Accuracy and key information coverage largely determine the user experience of the search engine.

Most of the current search engine products, especially news searches, directly use the original title of the news as the title of the search results. However, the original title of the news is often filled with a lot of redundant information, or even too Emphasizing more on a certain aspect in order to overgeneralize, resulting in imprecise and inaccurate titles, and may also mislead users. Such titles in active news push products will directly cause users to be unable to quickly obtain key news information, affect user experience, reduce users' desire to obtain information about push content, and reduce stickiness to push products.

Therefore, for the original news headlines, removing redundant information to obtain more accurate and rigorous news headlines has become an urgent technical problem to be solved.

Contents of the invention

In view of the above problems, the present invention is proposed to provide a method and device for summarizing news headlines which overcome the above problems or at least partly solve the above problems.

According to one aspect of the present invention, a method for summarizing news headlines is provided, including:

Obtain the original title of the news, perform lexical and syntactic analysis on the original title of the news, and obtain the analysis result;

Based on the analysis result, extract the sentence main content in the original title of the news, and use the extracted sentence main content as the news candidate title;

The quality evaluation strategy of news headline abstracts is used to evaluate the quality of the news candidate headlines, and then the news abstract headlines are determined according to the evaluation results.

Optionally, said acquiring the original headline of the news includes:

Obtain crawling logs about news resources crawled by web crawlers;

Extract the original headlines of the news from scraped logs.

Optionally, the extracting the original headlines of the news from the crawl log includes:

For each record about the news resource in the crawl log, extract the field value of the specified field of the record as the original title of the news.

Optionally, the lexical and syntactic analysis of the original title of the news is carried out to obtain the analysis results, including:

Perform word segmentation processing on the original headline of the news to obtain multiple word segmentation;

Carry out part-of-speech tagging and entity category tagging respectively to each participle in the plurality of participles;

Based on the part-of-speech tagging and entity category tagging of each participle, the dependency syntax analysis of the original headline of the news is carried out, and the subscript and dependency type of each participle are identified.

Optionally, the method for word segmentation of the original headline of the news includes at least one of the following:

Word segmentation method based on string matching;

Word segmentation method based on semantic understanding;

Statistical word segmentation method.

Optionally, performing entity category labeling on each of the plurality of word segmentations, including:

A sequence labeling model is used to identify entity words in each of the multiple word segmentations, and label entity categories.

Optionally, the entity category includes any one of the following:

Names of people, places, organizations, brands, and software.

Optionally, based on the part-of-speech tagging and entity category tagging of each participle, the original headline of the news is subjected to dependency syntactic analysis, and the dependent node subscript and dependency type of each participle are identified, including:

Through the part-of-speech tagging and entity category tagging of each participle, the grammatical components of the original headline of the news are identified;

Analyze the identified dependency relationship between each grammatical component, and obtain the dependency node subscript and dependency type of each participle.

Optionally, based on the analysis result, the main content of the sentence in the original headline of the news is extracted, including:

According to the part-of-speech tagging, entity category tagging, dependent node subscripts, and dependency types of each participle, a syntax tree is generated, and then the sentence backbone content of the original headline of the news is generated by screening and pruning the syntax tree.

Optionally, generating a syntax tree according to the part-of-speech tagging, entity category tagging, dependent node subscript, and dependency type of each participle, and then generating the main sentence content of the original headline of the news by screening and pruning the syntax tree, include:

Select the head master node corresponding to the core relationship in the dependency type as the main predicate;

If the part of speech of the main node is a noun part of speech after the word segmentation, then merge and update the predicate for all shallow dependent nouns of a specific analogy;

If the part of speech after the main node participle is a verb part of speech, then set the main node as a predicate verb;

Negative attributives are identified and incorporated into predicates.

Optionally, the method also includes:

Identify the subject-predicate relationship nodes, merge the surrounding nodes of the subject, keep the part of speech of the noun for the parallel relationship nodes according to the subject rules, and prune the rest of the nodes, and set the subject node.

Optionally, the method also includes:

According to the object type, if it is a noun to identify the object, all the parallel relationship nodes are removed, and the object node is set.

Optionally, the quality of the candidate news headlines is evaluated by using an abstract quality evaluation strategy for news headlines, including:

Use the neural machine translation model to compress the original headlines of the news to get the headlines for news measurement;

For the news measurement headline and the news candidate headline, use a language model to calculate the quality score of the sentence under the language model;

The calculated quality score result is used as an evaluation result for evaluating the quality of the news candidate headline.

Optionally, the determining the headline of the news summary according to the evaluation result includes:

Among the news measurement headlines and the news candidate headlines, according to the calculated quality score result, determine the headline with the highest quality score as the headline to be selected;

If the quality score corresponding to the candidate title is greater than the quality score threshold, it is judged whether the candidate title satisfies the preset review condition, and if so, the candidate title is determined as a news summary title.

Optionally, whether the title to be selected meets the preset review conditions includes at least one of the following:

Whether the title to be selected is a subject-predicate structure grammar;

Whether the title to be selected is a subject-predicate structure grammar, and the predicate verb contains a verb component;

Whether the edit distance between the candidate title and the original news title is less than the edit distance threshold;

Whether the semantic distance between the candidate title and the original news title is smaller than the semantic distance threshold.

Optionally, after determining the title of the news summary according to the evaluation result, the method further includes:

The news summary title is provided to the real-time hotspot product module, so that the real-time hotspot product module displays the news summary title as a real-time hotspot.

According to another aspect of the present invention, a device for summarizing news headlines is also provided, including:

Obtaining module, suitable for obtaining the original headline of the news;

The analysis module is suitable for performing lexical and syntactic analysis on the original headline of the news to obtain the analysis result;

The extraction module is adapted to extract the main sentence content in the original headline of the news based on the analysis result, and use the main sentence content extracted as the news candidate title;

The determining module is adapted to evaluate the quality of the news candidate headlines by using the abstract quality evaluation strategy of the news headlines, and then determine the news abstract headlines according to the evaluation results.

Optionally, the acquisition module is also suitable for:

Obtain crawling logs about news resources crawled by web crawlers;

Extract the original headlines of the news from scraped logs.

Optionally, the acquisition module is also suitable for:

For each record about the news resource in the crawl log, extract the field value of the specified field of the record as the original title of the news.

Optionally, the analysis module includes:

The word segmentation unit is suitable for performing word segmentation processing on the original headline of the news to obtain multiple word segmentations;

A tagging unit, adapted to perform part-of-speech tagging and entity category tagging on each of the plurality of word segments;

The recognition unit is adapted to perform dependency syntactic analysis on the original headline of the news based on the part-of-speech tagging and entity category tagging of each participle, and identify the dependent node subscript and dependency type of each participle.

Optionally, the method for word segmentation of the original headline of the news includes at least one of the following:

Word segmentation method based on string matching;

Word segmentation method based on semantic understanding;

Statistical word segmentation method.

Optionally, the labeling unit is also suitable for:

A sequence labeling model is used to identify entity words in each of the multiple word segmentations, and label entity categories.

Optionally, the entity category includes any one of the following:

Names of people, places, organizations, brands, and software.

Optionally, the identification unit is also suitable for:

Through the part-of-speech tagging and entity category tagging of each participle, the grammatical components of the original headline of the news are identified;

Analyze the identified dependency relationship between each grammatical component, and obtain the dependency node subscript and dependency type of each participle.

Optionally, the extraction module is also suitable for:

According to the part-of-speech tagging, entity category tagging, dependent node subscripts, and dependency types of each participle, a syntax tree is generated, and then the sentence backbone content of the original headline of the news is generated by screening and pruning the syntax tree.

Optionally, the extraction module is also suitable for:

Select the head master node corresponding to the core relationship in the dependency type as the main predicate;

If the part of speech of the main node is a noun part of speech after the word segmentation, then merge and update the predicate for all shallow dependent nouns of a specific analogy;

If the part of speech after the main node participle is a verb part of speech, then set the main node as a predicate verb;

Negative attributives are identified and incorporated into predicates.

Optionally, the extraction module is also suitable for:

Identify the subject-predicate relationship nodes, merge the surrounding nodes of the subject, keep the part of speech of the noun for the parallel relationship nodes according to the subject rules, and prune the rest of the nodes, and set the subject node.

Optionally, the extraction module is also suitable for:

According to the object type, if it is a noun to identify the object, all the parallel relationship nodes are removed, and the object node is set.

Optionally, the determination module is also suitable for:

Use the neural machine translation model to compress the original headlines of the news to get the headlines for news measurement;

For the news measurement headline and the news candidate headline, use a language model to calculate the quality score of the sentence under the language model;

The calculated quality score result is used as an evaluation result for evaluating the quality of the news candidate headline.

Optionally, the determination module is also suitable for:

Among the news measurement headlines and the news candidate headlines, according to the calculated quality score result, determine the headline with the highest quality score as the headline to be selected;

If the quality score corresponding to the candidate title is greater than the quality score threshold, it is judged whether the candidate title satisfies the preset review condition, and if so, the candidate title is determined as a news summary title.

Optionally, whether the title to be selected meets the preset review conditions includes at least one of the following:

Whether the title to be selected is a subject-predicate structure grammar;

Whether the title to be selected is a subject-predicate structure grammar, and the predicate verb contains a verb component;

Whether the edit distance between the candidate title and the original news title is less than the edit distance threshold;

Whether the semantic distance between the candidate title and the original news title is smaller than the semantic distance threshold.

Optionally, the device further includes: a providing module, adapted to provide the news summary title to the real-time hot product module after the determination module determines the news summary title according to the evaluation result, so that the real-time hot product module sends the news summary title The above news summary headlines are displayed as real-time hotspots.

According to yet another aspect of the present invention, a computer storage medium is also provided, the computer storage medium stores computer program codes, and when the computer program codes are run on a computing device, the computing device is caused to execute the above-mentioned Summary method for news headlines.

According to still another aspect of the present invention, there is also provided a computing device, including: a processor; and a memory storing computer program code; when the computer program code is executed by the processor, it causes the computing device to execute According to the summary method of news headlines above.

The embodiment of the present invention provides a method for summarizing news headlines. First, obtain the original headlines of the news, and then perform lexical and syntactic analysis on the original headlines of the news to obtain the analysis results; then, based on the analysis results, extract the sentences in the original headlines of the news. Then, use the abstract quality evaluation strategy of news titles to evaluate the quality of news candidate titles, and then determine the news summary titles according to the evaluation results. It can be seen that the embodiment of the present invention utilizes lexical and syntactic analysis to compress and summarize the news headlines, so that the main content in the news headlines is extracted while retaining the key information in the original news headlines as much as possible, and more accurate and rigorous news headlines can be obtained. At the same time, the summary quality evaluation strategy is introduced to evaluate the quality of news candidate titles, and automatically review the results with better abstract quality, so as to reduce the cost of manual operation review and greatly reduce the summaries caused by manual review. Push delay.

The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable , the specific embodiments of the present invention are enumerated below.

Those skilled in the art will be more aware of the above and other objects, advantages and features of the present invention according to the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiment. The drawings are only for the purpose of illustrating a preferred embodiment and are not to be considered as limiting the invention. Also throughout the drawings, the same reference numerals are used to designate the same components. In the attached picture:

Fig. 1 illustrates a flow chart of a method for summarizing news headlines according to an embodiment of the present invention;

FIG. 2 illustrates a flow chart of a method for lexical and syntactic analysis of an original news headline according to an embodiment of the present invention;

FIG. 3 illustrates a flow chart of a method for extracting sentence stem content in an original headline of news according to an embodiment of the present invention;

FIG. 4 illustrates a flowchart of a method for evaluating the quality of news candidate titles according to an embodiment of the present invention;

FIG. 5 illustrates a flow chart of a method for determining a headline of a news abstract according to an evaluation result according to an embodiment of the present invention;

FIG. 6 illustrates displaying news summary titles on a search result page according to an embodiment of the present invention;

FIG. 7 illustrates a flow chart of a method for summarizing news headlines according to another embodiment of the present invention;

FIG. 8 illustrates a structural diagram of an apparatus for summarizing news headlines according to an embodiment of the present invention; and

Fig. 9 shows a structural diagram of an apparatus for summarizing news headlines according to another embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

In related technologies, the main methods used in sentence compression include: deletion of words in a sentence, replacement, rearrangement or insertion of words in a sentence. Among them, the word deletion method in the sentence has become the mainstream method because of its low complexity. The techniques used mainly include noise channel model, structured discrimination model, tree-to-tree conversion, integer linear programming, etc. As far as the overall effect is concerned, the current mainstream method technology has a limited amount of deleted words in the sentence, and the compression effect is not obvious, as in the following example:

Original sentence: But they are still continuing to search the area try and see if there were, in fact, any further shooting incidents.

Compressed sentence: They are continuing to search the area to see if there were any further incidents.

In the related technologies mentioned above, based on the way of deleting words in sentences, replacing words in sentences, rearranging or inserting, on the one hand, it is difficult to capture the content and information in all titles, on the other hand, after rewriting based on this technical solution The titles are generally too long. Therefore, it is difficult to meet the user's needs and experience for the product in terms of accuracy and the length of the rewritten title. In addition, due to the effect and status quo of related technical solutions, it is necessary to manually review the summary results, and after the review is passed, it will be pushed online to meet the high accuracy needs of user products. Therefore, this technical solution still cannot get rid of the large manual operating cost overhead, and the low coverage and poor timeliness of the summary results caused by the manual process.

In order to solve the above technical problem, an embodiment of the present invention provides a method for summarizing news headlines. As shown in Fig. 1, the method may include the following steps S102 to S106.

Step S102, obtaining the original headline of the news, performing lexical and syntactic analysis on the original headline of the news, and obtaining the analysis result.

Step S104, based on the analysis result, extract the sentence main content in the original headline of the news, and use the extracted sentence main content as a candidate news headline.

Step S106, using the news headline abstract quality evaluation strategy to evaluate the quality of the news candidate headlines, and then determine the news abstract headlines according to the evaluation results.

The embodiment of the present invention provides a method for summarizing news headlines. First, obtain the original headlines of the news, and then perform lexical and syntactic analysis on the original headlines of the news to obtain the analysis results; then, based on the analysis results, extract the sentences in the original headlines of the news. Then, use the abstract quality evaluation strategy of news titles to evaluate the quality of news candidate titles, and then determine the news summary titles according to the evaluation results. It can be seen that the embodiment of the present invention utilizes lexical and syntactic analysis to compress and summarize the news headlines, so that the main content in the news headlines is extracted while retaining the key information in the original news headlines as much as possible, and more accurate and rigorous news headlines can be obtained. At the same time, the summary quality evaluation strategy is introduced to evaluate the quality of news candidate titles, and automatically review the results with better abstract quality, so as to reduce the cost of manual operation review and greatly reduce the summaries caused by manual review. Push delay.

The original title of the news is obtained in step S102 above. The embodiment of the present invention provides an optional solution. In this solution, the crawling log about the news resource captured by the web crawler can be obtained, and then from the crawling log Extract the original headline of the news.

The web crawler (Web Crawlers) here is a program or script that automatically grabs information on the World Wide Web according to certain rules. When a web crawler downloads Internet resources, for example, starting from the homepage of a portal website, it first downloads the webpage on the homepage of the portal website, and then by analyzing this webpage, it can find all the hyperlinks in the page, which means it knows the portal website All web pages directly linked from the home page, such as mail, finance, news, etc. Next, visit, download and analyze the e-mail and other web pages of this portal, and find other connected web pages. Let the computer go on and on, and you can download the entire Internet. Of course, it is also necessary to record which web page has been downloaded to avoid duplication. In the web crawler, use a list called "hash table" (Hash Table) instead of a notepad to record whether the web page has been downloaded.

In the above scheme of extracting the original title of the news from the crawling log, specifically, for each record about the news resource in the crawling log, the field value of the specified field of the record can be extracted as the original title of the news. For example, if the record format of a news resource in a crawl log of a web crawler is url_id+\t+url_title+\t+crawl_time, then the field value of url_title is extracted as the original title of the news. It should be noted that the enumeration here is only illustrative and does not limit the embodiment of the present invention.

Further, in step S102 above, the original headline of the news is analyzed lexically and syntactically, and the analysis result is obtained. The embodiment of the present invention provides an optional solution. FIG. 2 shows the original headline of the news according to an embodiment of the present invention. A flowchart of a method for performing lexical and syntactic analysis on a title. As shown in Fig. 2, the method may include the following steps S202 to S206.

Step S202, perform word segmentation processing on the original headline of the news to obtain multiple word segments.

Step S204, perform part-of-speech tagging and entity category tagging on each of the plurality of word segments.

Step S206, based on the part-of-speech tagging and entity category tagging of each participle, perform dependency syntactic analysis on the original headline of the news, and identify the dependent node subscript and dependency type of each participle.

In step S202, the word segmentation method for the original headline of the news may include a word segmentation method based on character string matching, a word segmentation method based on semantic understanding, or a word segmentation method based on statistics, etc., which is not limited in this embodiment of the present invention.

The word segmentation method based on string matching, also known as the mechanical word segmentation method, matches the Chinese character string to be analyzed with an entry in a "sufficiently large" machine dictionary according to a certain strategy. If a character is found in the dictionary string, the match is successful (a word is recognized). According to different scanning directions, string matching word segmentation methods can be divided into forward matching and reverse matching; according to different length priority matching, they can be divided into maximum (longest) matching and minimum (shortest) matching. Several commonly used mechanical word segmentation methods are as follows:

1) forward maximum matching method (direction from left to right);

2) reverse maximum matching method (direction from right to left);

3) Minimal segmentation (making the minimum number of words cut out in each sentence);

4) Two-way maximum matching method (carry out two scans from left to right and from right to left).

In the actual word segmentation process, the above various methods can also be combined with each other, for example, the forward maximum matching method and the reverse maximum matching method can be combined to form a two-way matching method. Due to the characteristics of Chinese characters into words, forward minimum matching and reverse minimum matching are generally seldom used. Generally speaking, the segmentation accuracy of reverse matching is slightly higher than that of forward matching, and there are fewer ambiguities encountered. The statistical results show that the error rate of purely using forward maximum matching is 1/169, and the error rate of purely using reverse maximum matching is 1/245. But this precision is far from meeting the actual needs. The word segmentation systems actually used all use mechanical word segmentation as a means of initial segmentation, and it is necessary to use various other language information to further improve the accuracy of segmentation. One method is to improve the scanning method, which is called feature scanning or flag segmentation, and firstly identify and segment some words with obvious characteristics in the string to be analyzed. Using these words as breakpoints, the original string can be divided into Perform mechanical word segmentation for smaller strings, thereby reducing the error rate of matching. Another method is to combine word segmentation and part-of-speech tagging, use rich part-of-speech information to help word segmentation decisions, and in turn check and adjust the word segmentation results during the tagging process, thereby greatly improving the accuracy of segmentation.

The word segmentation method based on semantic understanding is to achieve the effect of recognizing words by allowing computers to simulate human understanding of sentences. Its basic idea is to perform syntactic and semantic analysis at the same time of word segmentation, and use syntactic information and semantic information to deal with ambiguity. It usually includes three parts: the word segmentation subsystem, the syntax and semantics subsystem, and the general control part. Under the coordination of the general control part, the word segmentation subsystem can obtain syntactic and semantic information about words, sentences, etc. to judge the ambiguity of word segmentation, that is, it simulates the process of human understanding of sentences. This method of word segmentation requires the use of a large number of languages knowledge and information.

Based on the statistical word segmentation method, from a formal point of view, a word is a combination of stable words, so in the context, the more adjacent words appear at the same time, the more likely it is to form a word. Therefore, the frequency or probability of adjacent co-occurrence of words can better reflect the credibility of words. The frequency of combinations of adjacent co-occurring characters in the corpus can be counted, and their mutual occurrence information can be calculated. Define the mutual occurrence information of two characters, and calculate the adjacent co-occurrence probability of two Chinese characters X and Y. Mutual appearance information reflects the closeness of the combination relationship between Chinese characters. When the degree of closeness is higher than a certain threshold, it can be considered that this word group may form a word. This method only needs to count the frequency of words in the corpus, and does not need to segment the dictionary, so it is also called the dictionary-free word segmentation method or the statistical word extraction method. However, this method also has certain limitations. It will often extract some common word groups that have a high co-occurrence frequency but are not words, such as "this", "one", "some", "my", "Many", etc., and the recognition accuracy of common words is poor, and the time and space overhead is large. The statistical word segmentation system used in practice must use a basic word segmentation dictionary (common word dictionary) for string matching word segmentation, and at the same time use statistical methods to identify some new words, that is, to combine string frequency statistics and string matching. It has the characteristics of fast speed and high efficiency, and also utilizes the advantages of no-dictionary word segmentation combined with contextual recognition of new words and automatic disambiguation.

Another category is based on statistical machine learning methods. First, a large number of texts that have been segmented are given, and the statistical machine learning model is used to learn the rules of word segmentation (called training), so as to realize the segmentation of unknown texts. The ability of each character in Chinese to be used as a word is different. In addition, some characters often appear as prefixes, and some characters often appear as suffixes. Combining the information of whether two characters form words when they meet together, we can get a lot of words related to word segmentation. This method is to make full use of the rules of Chinese word formation to segment words.

In step S204 above, the part-of-speech tagging is performed on each participle in the multiple participle words. The specific part-of-speech category of the tagging can be nouns, verbs, adjectives, adverbs, conjunctions, interjections, or quantifiers, etc., which are not limited in the embodiment of the present invention .

In step S204, the entity category labeling is performed on each of the multiple word segmentations. The embodiment of the present invention provides an optional solution, that is, a sequence labeling model can be used to perform entity classification on each of the multiple word segmentations. Identify and label entity categories. The entity category here may be a person name, a place name, an organization name, a brand name, or a software name, etc., and the embodiment of the present invention is not limited thereto.

In practical applications, the sequence labeling model can be HMM (Hidden Markov Model, hidden Markov model), MEMM (Maximum Entropy Markov Model, maximum entropy hidden Markov model) and CRF (Conditional Random Field Algorithm, conditional random field model )and many more. Unlike general classification problems, the output of a sequence labeling model is a sequence of labels. Generally speaking, tags are related to each other, constituting the structural information between tags. Using these structural information, sequence labeling models can often achieve higher performance than traditional classification methods on sequence labeling problems.

The dependency types mentioned in step S206 above can be shown in Table 1 as an example. It should be noted that the dependency types and examples shown in Table 1 are only illustrative and do not limit the embodiment of the present invention.

Table 1

Dependency type Tag Description (descriptive information) example subject-verb relationship SBV subject-verb I send her a bouquet of flowers (I <-- send) Verb-object relationship VOB direct object, verb-object I send her a bouquet of flowers (send --> flowers) guest relationship IOB indirect object, indirect-object I send her a bouquet of flowers (send --> her) prepositional object FOB Fronting object, fronting-object He reads everything (book <-- read) fixed relationship ATT attribute redapple(red <-- apple) State structure ADV adverbial very beautiful (very <-- beautiful) Verb structure CMP complement Finished homework (done --> finished) Constellation COO coordinate Mountain and Sea (Mountain --> Sea) Guest Relations POB preposition-object In trade zone (in -->inside) independent structure IS independent structure Two single sentences are structurally independent of each other core relationship HED the head Refers to the core of the entire sentence Concurrent language DBL double He invites me to dinner (please --> me)

In step S206 above, based on the part-of-speech tagging and entity category tagging of each participle, the original headline of the news is analyzed by dependency syntax, and the subscripts and dependency types of each participle are identified. The embodiment of the present invention provides an optional solution , in this alternative, the grammatical components of the original headline of the news can be identified through the part-of-speech tagging and entity category tagging of each participle, and then the dependency relationship between the identified grammatical components can be analyzed to obtain the dependency of each participle Node index and dependency type.

Based on the above dependency syntactic analysis, when the above step S104 extracts the main content of the sentence in the original headline of the news based on the analysis result, specifically, it can be based on the part-of-speech tagging, entity category tagging, dependent node subscript and dependency type of each participle, Generate a syntax tree, and then generate the main sentence content of the original headline of the news by screening and pruning the syntax tree.

Fig. 3 shows a flowchart of a method for extracting sentence stem content in an original headline of news according to an embodiment of the present invention. As shown in Fig. 3, the method may include the following steps S302 to S306.

Step S302, selecting the head node corresponding to the core relationship in the dependency type as the main predicate.

Step S304, if the part of speech of the main node after word segmentation is the part of speech of a noun, then merge and update the predicates for all shallowly dependent nouns of a specific analogy; if the part of speech of the main node after word segmentation is a part of speech of a verb, set the main node as a predicate verb.

Step S306, identifying and merging negative attributives into predicates.

In an optional embodiment of the present invention, it is also possible to identify the subject-predicate relationship nodes, merge the surrounding nodes of the subject, keep the part of speech of the noun for the parallel relationship nodes according to the subject rules, and perform node pruning on the rest, and set the subject node. In addition, according to the type of the object, if the object is a noun, the object can be identified, all the parallel relationship nodes are removed, and the object node is set.

The embodiment of the present invention utilizes lexical and syntactic analysis to compress news headlines, so that the main content of the news headlines is extracted while retaining key information in the original news headlines as much as possible, and more accurate and rigorous news headlines can be obtained.

In step S106 above, the summary quality evaluation strategy of news titles is used to evaluate the quality of news candidate titles. The embodiment of the present invention provides an optional solution. FIG. Flowchart of the method for assessing the quality of titles. As shown in Fig. 4, the method may include the following steps S402 to S406.

In step S402, the neural machine translation model is used to compress the original headlines of the news to obtain news measurement headlines.

In this step, the neural machine translation model can be trained in advance, for example, the neural machine translation model can be trained using Seq2Seq combined with the Attention mechanism using historically reviewed data pairs and manually labeled data sets.

Step S404, for the news measurement headline and the news candidate headline, use the language model to calculate the quality score of the sentence under the language model.

In step S406, the calculated quality score is used as an evaluation result for evaluating the quality of news candidate headlines.

After using the calculated quality score results according to steps S402 to S406 as evaluation results for evaluating the quality of news candidate titles, the news abstract titles may be further determined according to the evaluation results.

Fig. 5 shows a flow chart of a method for determining a headline of a news abstract according to an evaluation result according to an embodiment of the present invention. As shown in Fig. 5, the method may include the following steps S502 to S504.

Step S502, among the measured news headlines and news candidate headlines, according to the calculated quality score result, determine the headline with the highest quality score as the headline to be selected.

Step S504, if the quality score corresponding to the candidate title is greater than the quality score threshold, judge whether the candidate title satisfies the preset review condition, and if so, determine the candidate title as a news summary title.

Here, whether the title to be selected satisfies the preset review conditions may include at least one of the following:

Whether the title to be selected is a subject-predicate structure grammar;

Whether the title to be selected is a subject-predicate structure grammar, and the predicate verb contains a verb component;

Whether the edit distance between the candidate title and the original news title is less than the edit distance threshold;

Whether the semantic distance between the candidate title and the original news title is smaller than the semantic distance threshold.

In practical applications, if only one of the preset review conditions is met, the title to be selected will be determined as the title of the news summary; it can also be a combination of any two or more of the preset review conditions, and the The title to be selected is determined as the title of the news abstract; or if all preset review conditions are met, the title to be selected is determined as the title of the news abstract. For example, it is first possible to judge whether the title to be selected is a subject-predicate structure grammar, and if so, continue to judge whether the predicate verb contains a verb component. If the predicate verb contains a verb component, continue to judge whether the edit distance between the candidate title and the original news title is less than the edit distance threshold. If the edit distance between the candidate title and the original news title is less than the edit distance threshold, continue to judge whether the semantic distance between the candidate title and the original news title is less than the semantic distance threshold. If the semantic distance between the candidate title and the original news title is smaller than the semantic distance threshold, the candidate title is determined as a news summary title.

In an optional embodiment of the present invention, after the headline of the news abstract is determined according to the evaluation result, the headline of the news abstract can also be provided to the real-time hotspot product module, so that the real-time hotspot product module can display the headline of the news abstract as a real-time hotspot. In practical applications, the real-time hotspot product module can display news summary titles as real-time hotspots on the search result page, which can improve the user's search experience and increase the click-through rate of the search result items generated by the search engine. As shown in Figure 6, on the search result page corresponding to the search term "rural revitalization", the headlines of news abstracts are displayed in the form of real-time hotspots.

A variety of implementation modes of each link of the embodiment shown in FIG. 1 have been introduced above, and the implementation process of the method for summarizing news headlines of the present invention will be described in detail below through specific embodiments.

Fig. 7 shows a flow chart of a method for summarizing news headlines according to another embodiment of the present invention. As shown in Fig. 7, the method may include the following steps S702 to S708.

Step S702, crawling news resources on the Internet, and extracting original headlines corresponding to the news.

Step S704, using word segmentation technology, lexical analysis technology, syntactic analysis technology, and entity recognition technology on the original headline of the news to extract the main content of the sentence in the original headline of the news.

Step S706, using the neural machine translation model to generate corresponding rewriting candidate results.

Step S708, using the language model and semantic features to evaluate the rewriting quality, and automatically review the high-quality rewriting results.

The embodiment of the present invention uses syntactic analysis to compress the original headline of the news, so that the main content of the original headline of the news is extracted and the key information in the original news is retained as much as possible. The effect of rewriting the abstract is evaluated, and the results with better abstract quality are automatically reviewed to reduce the cost of manual operation review and greatly reduce the delay in pushing the summary caused by manual review.

The following will use a specific example, that is, the original headline of the news is "13 people have been rescued from the vegetable market crushed by sudden heavy snowfall in Anlu, Hubei" to introduce the specific implementation process of each part in detail.

(1) Model pre-training and acquisition of existing models

Use historically reviewed data pairs and manually labeled data sets to train the neural machine translation model using Seq2Seq combined with the Attention mechanism. The model training tool is 360's existing neural machine translation toolkit.

The training data is a parallel corpus format as follows:

Ori: Bank account managers illegally lent 1.6 million of which 1.38 million were not recovered

Sum: bank account manager illegally lent money

Obtain 360 existing language models as the rewriting quality score evaluation.

(2) Title acquisition and lexical analysis of the title

Obtaining news original headlines from crawl logs in a web crawler.

The format is as follows: url_id+\t+url_title+\t+crawl_time.

As a basic step in natural language processing technology, lexical analysis produces part-of-speech tagging, dependency relationships, and entity tag types, which are the basic features that subsequent sentence stem extraction, compressed summarization, and other technologies rely on. Output after calling the existing 360 word segmentation module:

Example: 13 people have been rescued after a sudden heavy snowfall crushed the vegetable market in Anlu, Hubei

After word segmentation: Hubei/ns Anlu/ns sudden/d drop/v heavy snow/n crushed/v vegetable market/n has/d rescued/v13 people/mq

Among them, /before is the result after coarse-grained word segmentation, and /post is the part-of-speech tag after word segmentation.

Based on the results of word segmentation, sequence-based tagging is used to identify proper names and entity words.

The format of the original data to be labeled is shown in the first column of Table 2, and the labeling results produced by using the sequence labeling model are shown in the second and third columns of Table 2. In Table 2, B indicates the beginning byte, E indicates the last byte, and LOC indicates the location. It should be noted that the enumeration here is only illustrative and does not limit the embodiment of the present invention.

Table 2

lake B LOC north E. LOC install B LOC land E. LOC sudden 0 drop 0 Big 0 Snow 0 to press 0 collapse 0 vegetable 0 city 0 field 0 already 0 save 0 out 0 13 0 people 0

Merge the results in Table 2 above with the results after word segmentation.

After word segmentation and entity recognition:

Hubei/ns/LOC Anlu/ns/LOC sudden/d/drop/v/heavy snow/n/crushed/v/vegetable market/n/already/d/rescued/v/13 people/mq/

Among them, the first column after segmentation is the result after coarse-grained word segmentation, the second column is the part-of-speech tag after word segmentation, and the third column is the entity category tag.

Based on the results of word segmentation and recognition, the 360 basic syntax analysis module is called to complete the syntax analysis. The final lexical analysis result is:

Hubei/ns/LOC/2/ATT

Anlu/ns/LOC/4/SBV

sudden /d//4/ADV

drop /v//0/HEAD

heavy snow/n//4/VOB

crush /v//4/COO

Vegetable market/n//6/VOB

Already /d//9/ADV

Rescue /v//6/COO

13 people/mq//9/VOB

Among them, the first column after segmentation is the result of coarse-grained word segmentation, the second column is the part-of-speech tag after word segmentation, the third column is the entity category tag, the fourth column is the subscript of the dependent node in the dependency syntax analysis, and the fifth column is the dependency type.

(3) Extraction of the main content of the sentence

According to the lexical analysis features produced in (2) above, a syntax tree is generated, and the sentence trunk is generated by screening and pruning the syntax tree. The specific rules and algorithms are as follows:

Select the dependency syntax head node as the main predicate;

If the part of speech after the main node is a noun part of speech:

Merge and update predicates for all shallowly dependent nouns of a specific analogy;

If the part of speech after the main node is a verb part of speech:

Set the main node as the predicate verb;

Identify and incorporate negative word attributes into predicates;

Identify its subject-predicate logical relationship nodes:

Merge the peripheral nodes of the subject, and prune the remaining part of the part of speech of the noun according to the subject rule for the parallel relationship nodes, and set the subject node;

According to the line of the object, if it is a noun to identify the object, all the nodes of the parallel relationship are removed, and the object node is set.

Original sentence: Sudden heavy snowfall in Anlu, Hubei crushed the vegetable market and rescued 13 people

Sentence stem: Heavy snowfall in Anlu, Hubei crushes the vegetable market

(4) Rewrite and generalize using the neural machine translation model

For each original news headline, after word segmentation, use a pre-trained neural machine translation model to perform a compressed summary, generate candidates, and add the sentence stem to the candidate set at the same time. Neural machine translation can perform end-to-end compressed summarization for sentence articles.

Input example: Sudden heavy snowfall in Anlu, Hubei crushed the vegetable market and rescued 13 people

Output candidate set:

Main sentence of the original sentence: Heavy snowfall crushed the vegetable market in Anlu, Hubei

Neural machine translation results: Heavy snow in Hubei crushes vegetable market

(5) Title rewriting review based on language model

Use the language model to calculate the score of the sentence under the model for the candidates produced by each title, and name it quality_score.

(6) Screen high-quality titles based on rules for automatic online launch

Initialize the following parameters:

quality_threshold,

jaccard_semantic_gap_threshold,

ed_semantic_gap_threshold;

For each candidate for reelection under the original title:

final_candidate=The result with the highest quality score after sorting all candidates by quality score.

For final_candidate, if its quality score is greater than quality_threshold:

if it satisfies the subject-predicate structure grammar, and the predicate verb is a Chinese verb component:

And the edit distance and jaccard semantic distance from the original title are both smaller than the corresponding semantic_gap_threshold:

Then the final_candidate is the automatic review compression summary result of the corresponding title.

The embodiment of the present invention greatly reduces the large amount of manpower input required in traditional title rewriting, and solves the problem of inconsistent rewriting effects caused by inconsistent subjective standards of operating personnel. After getting the title of the news summary, it can also be provided to 360 to search for real-time hot products, which can be displayed on the search homepage, the right side of the search result page, the homepage of the browser or 360 navigation, etc. After the product uses this method to rewrite the original news headline and automatically go online, the click-through rate of the product has increased significantly compared with the original manual editing method.

It should be noted that, in practical applications, all the above optional implementation manners may be combined in any way to form optional embodiments of the present invention, which will not be repeated here.

Based on the methods for summarizing news headlines provided in the above embodiments and based on the same inventive concept, an embodiment of the present invention also provides a device for summarizing news headlines.

Fig. 8 shows a structural diagram of an apparatus for summarizing news headlines according to an embodiment of the present invention. As shown in FIG. 8 , the apparatus may include an acquisition module 810 , an analysis module 820 , an extraction module 830 and a determination module 840 .

Now introduce the functions of each composition or device of the abstract device of the news headline of the embodiment of the present invention and the connection relationship between each part:

An acquisition module 810, adapted to acquire the original headline of the news;

The analysis module 820, coupled with the acquisition module 810, is suitable for performing lexical and syntactic analysis on the original headline of the news to obtain the analysis result;

The extraction module 830, coupled with the analysis module 820, is suitable for extracting the sentence main content in the original headline of the news based on the analysis result, and using the extracted sentence main content as the news candidate title;

The determining module 840, coupled with the extracting module 830, is adapted to evaluate the quality of the candidate news headlines by using the abstract quality evaluation strategy of the news headlines, and then determine the news abstract headlines according to the evaluation results.

In an optional embodiment of the present invention, the acquiring module 810 is further adapted to:

Obtain crawling logs about news resources crawled by web crawlers;

Extract the original headlines of the news from scraped logs.

In an optional embodiment of the present invention, the acquiring module 810 is further adapted to:

For each record about the news resource in the crawl log, extract the field value of the specified field of the record as the original title of the news.

In an optional embodiment of the present invention, as shown in FIG. 9, the analysis module 820 shown in FIG. 8 above may include:

The word segmentation unit 821 is adapted to perform word segmentation processing on the original headline of the news to obtain multiple word segments;

The labeling unit 822, coupled with the word segmentation unit 821, is adapted to perform part-of-speech labeling and entity category labeling on each of the multiple word segmentations;

The identification unit 823, coupled with the labeling unit 822, is adapted to perform dependency syntax analysis on the original headline of the news based on the part-of-speech tagging and entity category tagging of each participle, and identify the dependent node subscript and dependency type of each participle.

In an optional embodiment of the present invention, the method for segmenting the original headline of the news includes at least one of the following:

Word segmentation method based on string matching;

Word segmentation method based on semantic understanding;

Statistical word segmentation method.

In an optional embodiment of the present invention, the labeling unit 822 is also adapted to:

A sequence labeling model is used to identify entity words in each of the multiple word segmentations, and label entity categories.

In an optional embodiment of the present invention, the entity category includes any one of the following:

Names of people, places, organizations, brands, and software.

In an optional embodiment of the present invention, the identification unit 823 is further adapted to:

Through the part-of-speech tagging and entity category tagging of each participle, the grammatical components of the original headline of the news are identified;

Analyze the identified dependency relationship between each grammatical component, and obtain the dependency node subscript and dependency type of each participle.

In an optional embodiment of the present invention, the extraction module 830 is also adapted to:

According to the part-of-speech tagging, entity category tagging, dependent node subscripts, and dependency types of each participle, a syntax tree is generated, and then the sentence backbone content of the original headline of the news is generated by screening and pruning the syntax tree.

In an optional embodiment of the present invention, the extraction module 830 is also adapted to:

Select the head master node corresponding to the core relationship in the dependency type as the main predicate;

If the part of speech of the main node is a noun part of speech after the word segmentation, then merge and update the predicate for all shallow dependent nouns of a specific analogy;

If the part of speech after the main node participle is the verb part of speech, then set the main node as the predicate verb;

Negative attributives are identified and incorporated into predicates.

In an optional embodiment of the present invention, the extraction module 830 is also adapted to:

Identify the subject-predicate relationship nodes, merge the surrounding nodes of the subject, keep the part of speech of the noun for the parallel relationship nodes according to the subject rules, and prune the rest of the nodes, and set the subject node.

In an optional embodiment of the present invention, the extraction module 830 is also adapted to:

According to the object type, if it is a noun to identify the object, all the parallel relationship nodes are removed, and the object node is set.

In an optional embodiment of the present invention, the determining module 840 is further adapted to:

Use the neural machine translation model to compress the original headlines of the news to get the headlines for news measurement;

For the news measurement headline and the news candidate headline, use a language model to calculate the quality score of the sentence under the language model;

The calculated quality score result is used as an evaluation result for evaluating the quality of the news candidate headline.

In an optional embodiment of the present invention, the determining module 840 is further adapted to:

Among the news measurement headlines and the news candidate headlines, according to the calculated quality score result, determine the headline with the highest quality score as the headline to be selected;

If the quality score corresponding to the candidate title is greater than the quality score threshold, it is judged whether the candidate title satisfies the preset review condition, and if so, the candidate title is determined as a news summary title.

In an optional embodiment of the present invention, whether the title to be selected satisfies a preset review condition includes at least one of the following:

Whether the title to be selected is a subject-predicate structure grammar;

Whether the title to be selected is a subject-predicate structure grammar, and the predicate verb contains a verb component;

Whether the edit distance between the candidate title and the original news title is less than the edit distance threshold;

Whether the semantic distance between the candidate title and the original news title is smaller than the semantic distance threshold.

In an optional embodiment of the present invention, as shown in FIG. 9, the device shown in FIG. 8 above may also include:

The providing module 910 is adapted to provide the news summary title to the real-time hotspot product module after the determination module 840 determines the news summary title according to the evaluation result, so that the real-time hotspot product module uses the news summary title as a real-time hotspot exhibit.

Based on the same inventive concept, an embodiment of the present invention also provides a computer storage medium, the computer storage medium stores computer program code, and when the computer program code runs on the computing device, it causes the computing device to execute the above-mentioned A summary method for news headlines.

Based on the same inventive concept, an embodiment of the present invention also provides a computing device, including: a processor; and a memory storing computer program code; when the computer program code is run by the processor, the computing device Execute the summary method based on the news headlines described above.

According to any one of the above optional embodiments or a combination of multiple optional embodiments, the embodiments of the present invention can achieve the following beneficial effects:

The embodiment of the present invention provides a method for summarizing news headlines. First, obtain the original headlines of the news, and then perform lexical and syntactic analysis on the original headlines of the news to obtain the analysis results; then, based on the analysis results, extract the sentences in the original headlines of the news. Then, use the abstract quality evaluation strategy of news titles to evaluate the quality of news candidate titles, and then determine the news summary titles according to the evaluation results. It can be seen that the embodiment of the present invention utilizes lexical and syntactic analysis to compress and summarize the news headlines, so that the main content in the news headlines is extracted while retaining the key information in the original news headlines as much as possible, and more accurate and rigorous news headlines can be obtained. At the same time, the summary quality evaluation strategy is introduced to evaluate the quality of news candidate titles, and automatically review the results with better abstract quality, so as to reduce the cost of manual operation review and greatly reduce the summaries caused by manual review. Push delay.

Those skilled in the art can clearly understand that for the specific working processes of the above-described systems, devices, and units, reference can be made to the corresponding processes in the foregoing method embodiments, and for the sake of brevity, details are not described here.

In addition, each functional unit in each embodiment of the present invention may be physically independent of each other, or two or more functional units may be integrated together, or all functional units may be integrated into one processing unit. The above-mentioned integrated functional units can be implemented not only in the form of hardware, but also in the form of software or firmware.

Those skilled in the art can understand that: if the integrated functional unit is implemented in the form of software and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present invention or all or part of the technical solution can be embodied in the form of software products, the computer software products are stored in a storage medium, which includes a number of instructions to make a A computing device (such as a personal computer, a server, or a network device, etc.) executes all or part of the steps of the methods described in the various embodiments of the present invention when executing the instructions. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various media capable of storing program codes.

Alternatively, all or part of the steps for realizing the aforementioned method embodiments may be implemented by program instruction-related hardware (such as a personal computer, server, or computing device such as a network device), and the program instructions may be stored in a computer-readable memory In the medium, when the program instructions are executed by the processor of the computing device, the computing device executes all or part of the steps of the methods described in the various embodiments of the present invention.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: Within the spirit and principles of the present invention, it is still possible to modify the technical solutions described in the foregoing embodiments, or to perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from protection scope of the present invention.

Claims (10)

1. a kind of method of abstracting of headline, including：

The original header for obtaining news carries out morphology syntactic analysis to the original header of news, obtains analysis result；

Based on the analysis result, the sentence trunk content in the original header of news is extracted, and will be in the sentence trunk of extraction Hold and is used as news candidate's title；

Using the abstract quality evaluation strategy of headline, the quality of the news candidate title is assessed, and then basis Assessment result determines news in brief title.

2. according to the method described in claim 1, wherein, the original header for obtaining news, including：

Obtain the crawl log about News Resources of web crawlers crawl；

The original header of news is extracted from crawl log.

3. method according to claim 1 or 2, wherein the original header for extracting news from crawl log, packet It includes：

For being recorded about each item of News Resources in crawl log, the field value of the specific field of this record is extracted as new The original header of news.

4. method according to any one of claim 1-3, wherein the original header to news carries out morphology syntax Analysis, obtains analysis result, including：

Word segmentation processing is carried out to the original header of news, obtains multiple participles；

Part-of-speech tagging and entity class mark are carried out respectively to each participle in the multiple participle；

Part-of-speech tagging based on each participle and entity class mark carry out interdependent syntactic analysis, identification to the original header of news The interdependent node subscript and dependency type respectively segmented.

5. according to the described method of any one of claim 1-4, wherein the original header to news carries out word segmentation processing Method include at least one following：

Segmenting method based on string matching；

Segmenting method based on semantic understanding；

Segmenting method based on statistics.

6. method according to any one of claims 1-5, wherein carry out entity to each participle in the multiple participle Classification marks, including：

Using sequence labelling model, the entity word respectively segmented in the multiple participle is identified, marks entity class.

7. according to the method described in any one of claim 1-6, wherein the entity class includes following one of arbitrary：

Name, place name, mechanism name, brand name, software name.

8. a kind of summarization device of headline, including：

Acquisition module is suitable for obtaining the original header of news；

Analysis module is suitable for carrying out morphology syntactic analysis to the original header of news, obtains analysis result；

Extraction module is suitable for being based on the analysis result, extracts the sentence trunk content in the original header of news, and will extraction Sentence trunk content as news candidate's title；

Determining module is suitable for the abstract quality evaluation strategy using headline, is carried out to the quality of the news candidate title Assessment, and then news in brief title is determined according to assessment result.

9. a kind of computer storage media, the computer storage media is stored with computer program code, when the computer When program code is run on the computing device, the computing device is caused to execute according to described in any one of claim 1-7 The method of abstracting of headline.

10. a kind of computing device, including：

Processor；And

It is stored with the memory of computer program code；

When the computer program code is run by the processor, the computing device is caused to execute according to claim 1- The method of abstracting of headline described in any one of 7.