CN103544255B - Text semantic relativity based network public opinion information analysis method - Google Patents

Abstract

The present invention relates to a network public opinion information analysis system based on text semantic correlation, including the following modules: a network public opinion information collection module, which collects various public opinion information rich in content from web pages; a public opinion information extraction module and a public opinion information preprocessing module collect Initially filter and segment the public opinion information, extract the meta information of the text, establish the feature semantic network graph of the text, and perform weighted calculation and feature extraction to provide services for public opinion information mining. The public opinion information mining module uses an improved text clustering analysis method based on semantic similarity to classify texts; the public opinion information analysis module performs OLAP multi-dimensional statistics on the mined data of public opinion information, analyzes public opinion evaluation indicators, and generates relevant public opinion information Decision support. The invention solves the problem of incomplete semantic information of words in the text, and efficiently realizes cluster analysis and hot topic discovery of dynamic data in a large-scale network environment.

Description

Analysis method of network public opinion information based on text semantic correlation

technical field

The invention relates to the technical field of network information, in particular to a method for analyzing network public opinion information based on text semantic correlation.

Background technique

In today's society, the Internet has penetrated into people's daily life, and instant communication tools such as Weibo, forums, and blogs have become important channels for people to obtain information, express opinions, and disseminate information. With the help of network platforms, public opinion information spreads rapidly and attracts widespread attention. Its speed of transmission, wide range, and influence are far from comparable to traditional media. Internet public opinion, a powerful force of social public opinion, has a certain impact and influence on social development and stability. Positive Internet public opinion is like "positive energy", which promotes and promotes social development; negative Internet public opinion has a negative effect on social stability and triggers a public opinion crisis. Therefore, strengthening the monitoring, analysis and management of network public opinion information has important practical significance for stabilizing social order and building a harmonious society. Timely monitoring of network public opinion information, correct judgment and decision-making, rapid and timely response, and active and effective measures to resolve public opinion crises have become the key and difficult issues of network public opinion management.

Contents of the invention

In view of the characteristics of Internet public opinion information in the above background technology and the problems to be solved in the management of Internet public opinion information, the present invention provides a method for analyzing Internet public opinion information based on text semantic correlation.

The technical solution adopted by the present invention to solve the technical problem is a method for analyzing network public opinion information based on text semantic correlation. Adopting a network public opinion information analysis system including a network public opinion information collection module, a public opinion information extraction module, a public opinion information preprocessing module, a public opinion information mining module, a public opinion information analysis module and a public opinion information database, and including the following steps:

a. The network public opinion information collection module collects various public opinion information from web pages and stores them in the public opinion information database;

b. The public opinion information extraction module and the public opinion information preprocessing module preliminarily filter and segment the public opinion information collected in step a, extract the content information contained in the text, and provide data services for public opinion information mining;

c. On the basis of step b, the public opinion information mining module adopts an improved text clustering analysis method based on semantic similarity to generate category description information, and screen out the text information contained in the clustering analysis results; use TFIDF word frequency features based on feature statistics The calculation method counts category features, obtains category feature words, selects nouns as candidate category feature words, sorts according to the weight of candidate feature words, uses the candidate feature words with larger weight values as category keywords, and utilizes the semantic relationship between category keywords, Form classification results; identify and establish new online public opinion topics, detect and track relevant content of existing public opinion topics;

d. Finally, the public opinion information analysis module conducts OLAP multi-dimensional statistical analysis on the data mined by the public opinion information through step c, and analyzes the public opinion evaluation indicators such as the degree of attention to the content of the public opinion topic and the emotional tendency of the public opinion topic.

In step a, the public opinion information collection module is to collect network public opinion information sources, which is different from general web crawlers in that it not only completes the crawling of web pages, but also formats the content of web pages to extract The theme and content of public opinion, the obtained data are stored in txt or html format files, and stored in the public opinion information database; the network public opinion information collection module adopts a combination of time-sharing access, regular IP address replacement and simulated browser for single sign-on Perform anti-shielding. The network public opinion information collection module adopts a combination of time-sharing access, regular IP address replacement and simulated browser single sign-on for anti-shielding. The specific steps that the network public opinion information collection module executes are: the specific steps that the public opinion information collection module executes are, start from the URL of the pre-defined topic-related webpage, obtain the text information in the webpage, and extract new URL from the current webpage Put it into the queue until the public opinion information that meets the conditions is collected and the URL queue is empty; the collected webpage text information is stored in the public opinion information database according to the field classification, and the public opinion information extraction module is provided for calling.

The public opinion information extraction module is to remove irrelevant content in the webpage, such as noise data such as advertisements in the webpage, navigation information, pictures, copyright descriptions, extract the meta information of the useful text part for public opinion analysis, and reconstruct the text, Gather together representative information of the theme; the public opinion information preprocessing module is to perform Chinese word segmentation processing, filter stop words, named entity recognition, Part-of-speech tagging, grammatical analysis and feature word extraction, establishment of forward index and inverted index; establishment of text feature semantic network graph, with the entity E contained in the text as the node of the graph, and the semantic relationship between the two entities as the effective For the edge, the semantic relationship between entities combined with the word frequency information is used as the weight of the node. The weight of the directed edge indicates the importance of the entity relationship in the text. The entity E includes a thing entity NE, an event entity VE, and an event relationship entity RE; The word frequency and text frequency information of the text are counted, and then the feature words are extracted, and the words that reflect the text characteristics are selected to represent the text.

In step b, the public opinion information extraction module is to clear the irrelevant content in the webpage, extract the meta information of the text part useful for public opinion analysis, reconstruct the text, and gather together the representative information of the theme; The public opinion information preprocessing module is to extract the collected public opinion information sources through the public opinion information extraction module, perform Chinese word segmentation, filter stop words, named entity recognition, part-of-speech tagging, grammar analysis and feature word extraction, and establish positive Sequence index and inverted index; establish a semantic network graph of text features, take the entity E contained in the text as the node of the graph, and the semantic relationship between two entities as the directed edge of the graph, and the semantic relationship between entities combined with word frequency information As the weight of the node, the weight of the directed edge represents the importance of the entity relationship in the text, and the entity E includes the thing entity NE, the event entity VE, and the event relationship entity RE; the word frequency and text frequency information of the text are counted, and then the feature Word extraction, selecting words that reflect the characteristics of the text to represent the text.

In order to realize text analysis such as network public opinion information text mining and natural language processing, word segmentation processing must first be carried out, drawing on domestic research results in the field of Chinese word segmentation, and using the word segmentation of the ICTCLAS Chinese lexical analysis system developed by the Institute of Computing Technology, Chinese Academy of Sciences , part-of-speech tagging, named entity recognition and other functions, through word segmentation of public opinion information text content, to extract words with a length greater than two. After the text is segmented, stop words that are useless for the computer to understand the text are filtered, and words with parts of speech such as nouns, verbs, nouns, and verbs are reserved to obtain a set of candidate feature words, which effectively reduces the size of the index and increases retrieval efficiency. Improve accuracy. For the text document processed by word segmentation, positive sequence index and inverted index are established to realize user query interaction. After the text is segmented, part-of-speech tagged, and stop words removed, a feature semantic network map of the text is established, word frequency and text frequency and other information of the text are counted, and then weighted calculations and feature extraction are performed.

In step c, the public opinion information mining module is to preprocess the text set, including Chinese word segmentation processing, stop word filtering and structured label information analysis, and then extract the text data set generated by the information extraction module according to the text The text semantic feature description structure constructed by the feature semantic network graph uses the similarity evaluation method to calculate the semantic similarity between texts, constructs a similarity matrix, and uses the improved text clustering analysis method based on semantic similarity to generate clustering results; clustering The analysis results generate category description information, and screen out the text information contained in the cluster analysis results; use the TFIDF word frequency feature calculation method based on feature statistics to count category features, obtain candidate category feature words, select nouns as candidate category feature words, and select nouns as candidate category feature words. Word weight sorting, the weight value is used to determine the candidate feature words as category keywords, and the semantic relationship between category keywords is used to form classification results; the mining results are constructed into a knowledge base, and the knowledge base can also be set to support public opinion topic discovery, Text mining functions such as public opinion tendency analysis.

In step d, the public opinion information analysis module is to perform OLAP multi-dimensional statistical analysis on the data mined in step c in the public opinion information database, and analyze the public opinion topic attention, public opinion content sensitivity, public opinion diffusion degree, Public opinion evaluation indicators such as the influence of public opinion releases provide support for relevant departments to grasp public opinion trends in a timely manner, release public opinion information in a timely manner, and make correct decisions.

Compared with the prior art, the present invention has the following beneficial effects:

1. The current network public opinion information reflects the characteristics of mass, dynamic, incomplete, and diverse forms of expression, but the existing public opinion information analysis methods often ignore the correlation between the content of the public opinion information text, resulting in the results of public opinion information analysis. Inaccurate; the present invention adopts the text feature semantic network graph model of constructing public opinion information text, and introduces the connection between the semantic association of words and the context context in the text description structure; combined with the improved text clustering algorithm based on semantic similarity, mining analysis Content related to the context and semantics in the public opinion information text.

2. By establishing the text feature semantic network graph of the public opinion information text, the contextual relationship between words in the public opinion information text is formed into a directed graph structure composed of feature items and weights. The connotation of word context semantics can better describe the hidden semantic information and topic features in the text, and solve the problem of lack of word semantic information in the text.

3. The improved text clustering algorithm based on semantic similarity is suitable for the cluster analysis of dynamic data and the discovery of public opinion topic hotspots in a large-scale network environment. By calculating the semantic similarity of text, a text semantic similarity matrix is constructed to dig out in depth Contextual semantics-related content in public opinion information texts, timely detection and tracking of new topic events; using the topic representation method of multiple centers in the class, selecting the maximum similarity between the text and each center in the class as the similarity of the text in this class , which effectively improves the operating efficiency of the system. As the number of texts increases, the effect of cluster analysis will be more obvious.

Description of drawings

Fig. 1 is a working flow chart of a method for analyzing network public opinion information based on text semantic correlation according to an embodiment of the present invention.

detailed description

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. However, the embodiments of the present invention are not limited thereto.

As shown in Figure 1, in the method of the present invention, comprise network public opinion information collection module, public opinion information extraction module, public opinion information preprocessing module, public opinion information digging module, public opinion information analysis module and the network public opinion information analysis system that comprises public opinion information database . Its processing flow is:

(1) Collection of public opinion information

Collecting network public opinion information sources is different from general web crawlers in that it not only crawls web pages, but also formats the content of web pages to extract useful public opinion information, such as the topic and content of public opinion. The data is stored in txt or html format files and written into the original public opinion information database. The specific steps are: according to the preset network public opinion information collection strategy, starting from the URLs of multiple seed webpages, sending instructions following the http protocol through various ports (using the GET method); the remote server returns an HTML-type URL according to the content of the application instruction document. The public opinion information collection module collects all the information in the returned document and saves it to the cache first, then transfers it to the database for storage, and obtains the text information in the web page; in the process of obtaining the text information of the web page, it continuously extracts new hyperlinks from the current web page URL access, and remove hyperlink URLs that have been visited, and repeat the cycle until the text information of web pages that meet the search strategy is collected and the queue of unvisited URLs is empty. Store the collected webpage text information in the database according to the field classification, and provide the call of the public opinion information extraction module.

The network public opinion information collection module usually adopts an anti-shielding strategy combining time-sharing access, regularly changing IP addresses, and simulating a browser for single sign-on. For many websites such as forums, blogs, microblogs, etc. can only be accessed through user login, it is easier to implement the strategy of simulating a browser here. Using the Web Browser control provided by Microsoft .NET development tool VisualStudio2008 to call the API of Microsoft IE browser, Use SSO single sign-on to simulate submitting user name and password to log in. After the user login information is loaded, the page jumps to the corresponding URL address, and searches by submitting keywords to obtain the source file of the required web page.

The collected webpage text information includes two parts: web content information, web structure and usage record information. Web content information includes text content information such as news headlines, text content, and comment information, and Web structure and Web usage record information includes statistical information such as clicks, page views, and comments.

(2) Public opinion information extraction

The collected webpage information contains noise data such as advertisements, navigation information, pictures, copyright descriptions, etc. What is really needed for the analysis of public opinion information is the meta information of the body part, clearing these irrelevant content, and extracting the meta information of the body part that is useful for the analysis of public opinion information. Meta information provides services for the subsequent mining and analysis of the text. The specific process is as follows:

(2-1) First use the Tidy tool to normalize the HTML tags on the text webpage, then use the html parser tool to build an HTML tree, and use the HTML tags as the nodes of the tree, which means that it is convenient to manage and operate the HTML code, and can better understand Code for structured mining.

(2-2) Extract relevant information such as title, keyword, text, length, update time and URL from the collected public opinion information sources. The title can intercept the information between the tags <TITLE> and </TITLE>; the keywords include The META tag in the head of the HTML file can be extracted from the META tag information; the time information can be extracted through pattern matching analysis and web page analysis.

(2-3) The concrete steps that text extracts are: select appropriate keyword, obtain the URL address of relevant webpage, obtain the HTML source code of webpage by visiting the server where URL address is in; Delete the useless tag row in the webpage source code, Retain the main content of the webpage; replace paragraph symbols (such as </p>, <br>, etc.) in the HTML code with special symbols (such as *[/p]*, *[/br]*, etc.), carriage return and Newline characters are replaced by line separators, and the row structure storage method is used to retain the content format of the web page; the text between the HTML tags "<" and ">" of each line is extracted; special symbols (such as *[/p] are replaced by carriage returns *, *[/br]*, etc.), keep the original paragraph of the text; remove HTML special escape characters (such as &quot, &lt, etc.) from the result string, combine regular expressions, match and extract the final text result .

After extracting relevant information such as title, keywords, text, length, update time and URL from the collected public opinion information sources, the public opinion information extraction module also needs to realize the reconstruction of text information.

Text reconstruction analyzes the existing forms of public opinion information such as network news, forum posts, Weibo blog posts, and the structural characteristics of the text, and organizes information on representative topics into "subject blocks" and the rest of the information into "content blocks" to improve cluster analysis effect.

For the text reconstruction of webpage news, the title and the first paragraph of information of the webpage news are composed of "subject block", and the remaining news description information and comment content are composed of "content block".

For the text reconstruction of forum posts, the title and main post of the post are composed of the "subject block", and the replies and follow-up information are purified, and the posts without Chinese characters and the posts using common evaluation words are removed, and several posts are selected to form "Content Block".

(3) Preprocessing of public opinion information

After the public opinion information is extracted, preprocessing such as Chinese word segmentation, named entity recognition, part-of-speech tagging, grammar analysis, and feature word extraction is performed, and the results are saved in the database. In order to realize text analysis such as text mining and natural language processing of network public opinion information, word segmentation processing must be carried out first, drawing on domestic research results in the field of Chinese word segmentation, and using the Chinese lexical analysis system ICTCLAS developed by the Institute of Computing Technology, Chinese Academy of Sciences to perform text segmentation and parts of speech Annotation, through Chinese word segmentation processing, extracts words with a length greater than two. The functions of ICTCLAS include Chinese text word segmentation, part-of-speech tagging, new word recognition, etc.; use the method of role model (role model) for named entity recognition; at the same time, it supports users to define personalized dictionaries according to their needs, which not only has high word segmentation accuracy, but also The effect is also better. Its implementation code is as follows:

After the text is segmented, stop words that are useless for the computer to understand the text are filtered, and words of parts of speech such as nouns, verbs, nouns, and verbs are reserved to obtain a set of alternative feature words to avoid redundant text and effectively reduce indexing. size, increase retrieval efficiency, and improve retrieval accuracy.

For the text processed by word segmentation, a positive sequence index and an inverted index are established to realize user query interaction. For the positive sequence index, according to the sorting of the word frequency, select the first N words to represent the text, and use the hash table to express it as: <file name, keyword phrase>; after the positive sequence index is established, search for the keywords in the text to find out the For all file names of this keyword, establish a file noun group to obtain an inverted index, which is represented by a hash table as: <keyword, file noun group>.

The establishment of the index and the retrieval service of the index are implemented based on the Apache open source project Lucene, which provides a complete query engine, index engine, and text analysis engine; Hadoop is used to store and manage massive index files.

The index creation process is as follows:

1. Create an index write object IndexWriter. A lexical parser must be provided when creating this object, and different lexical parsers use different lexicons. Use ThesaurusAnalyzer to extract content summaries;

2. Create a Document object for each result set from the database;

3. Create a Field object for each data element in the result set and add it to the Document object;

4. Write to the Document object.

The process of index retrieval is as follows: first create a query parser, which needs parameters such as the Field object name and the corresponding vocabulary parser; then obtain the query object from the query parser and keywords; obtain the retrieved result set through the query object, The result set consists of Document objects.

After the text is segmented, part-of-speech tagged, and stop words removed, a feature semantic network map of the text is established, word frequency and text frequency and other information of the text are counted, and then weighted calculations and feature extraction are performed.

The text feature semantic network graph is a directed graph that uses entities and their semantic relationships to express public opinion information. The entities E contained in the text (including thing entities NE, event entities VE, and event relationship entities RE) are used as graph nodes. The semantic relationship between two entities is used as the directed edge of the graph, and the semantic relationship between entities combined with word frequency information is used as the weight of the node. The weight of the directed edge indicates the importance of the entity relationship in the text. Through the introduction of network node weights and concept-based merging and simplification, a text feature semantic network graph is constructed to extract the core semantics of the text. That is, the words represented by the network nodes are merged, and the node weights are added; then the directed edges are merged, and the weights of the directed edges are added to construct a text feature semantic network graph to describe the semantic information and topic features in the text. The specific concepts are described as follows:

C1: The thing entity NE is defined as NE (id, concept, property, power). id stands for entity identifier, concept stands for entity concept, property stands for entity attribute, and power stands for weight.

C2: The event entity VE is defined as VE (id, concept, property, power, isN, subT, objT1, objT2). In addition to several data items including NE, isN represents whether it is negated, subT represents the header of the subject entity, and objT1 and objT2 represent the headers of object entities 1 and 2.

C3: Event relation entity RE is defined as RE (id, concept, property, power, isN, subT, objT). RE can be fully described by a pair of subject and object entities.

The analysis steps of text feature semantic network graph model are as follows:

S1: When analyzing the text, firstly, the sentence is used as the unit to construct the feature semantic network graph corresponding to each sentence. Analyze which NEs are generated in each sentence sentence by sentence, and record the NEs and their attribute information into the entity information table.

S2: After the NE analysis is completed, analyze the VE, and register the concepts, attributes, subjects and objects of the VE. VE entities with the same subject and object are represented as the same VE, otherwise different ids are set.

S3: Next analyze the RE. When analyzing RE, it should be distinguished from NE and VE, and the concept, attribute, subject, and object of RE should be registered in the entity information table.

S4: After the analysis ends, the entity information table of the statement is obtained. The entity information table describes the relationship between entities and is used to construct the entity relationship diagram, between NE and VE, between RE and NE, VE, and between entity E and attribute T to visualize the entity relationship through different connections.

S5: Based on the analysis and construction of the feature semantic network graph of the first sentence, merge the feature semantic network graphs of subsequent sentences, first merge nodes, and then merge directed edges.

S6: When merging nodes, merge nodes with the same word or semantic similarity meeting the threshold condition, and add node weights; otherwise, keep the node.

S7: Directed edge merging is to merge the directed edges existing between the merged nodes, and add the weights of the directed edges.

S8: Update the weight of the adjacent edge of the newly merged node to the weight of the node to strengthen the semantic relationship between nodes.

S9: After outputting the feature semantic network graph of all merged sentences, complete the construction of the feature semantic network graph of the entire text.

The next step is to assign weights to the part-of-speech features to accurately mark the text. According to Chinese part-of-speech characteristics and complete event description elements (time, place, person, and event content), combined with the Chinese Academy of Sciences Chinese part-of-speech tag set, the weight assignment of text features is divided into: title weight value 3, subtitle and keyword weight value 2 , the weight value of the abstract is 1.5, and the weight value of the first sentence and the last sentence of the paragraph is 1.3.

After the public opinion information is preprocessed, different tags are set for the title, body and reply of the text. When calculating the weight, the tag information of the keyword is read to complete the assignment of the position weight of the word.

(4) Public opinion information mining

The public opinion information mining module is a text data set generated by the information extraction module after preprocessing the text set, including Chinese word segmentation processing, stop word filtering, and structured label information analysis, and constructing the text according to the text feature semantic network graph. Semantic feature description structure, use the similarity evaluation method to calculate the semantic similarity between texts, construct a similarity matrix, and use the improved text clustering analysis method based on semantic similarity to generate clustering results; the clustering analysis results generate category description information, Filter out the text information contained in the cluster analysis results; use the TFIDF word frequency feature calculation method based on feature statistics to count category features, obtain candidate category feature words, select nouns as candidate category feature words, sort according to the weight of candidate feature words, and use the weight value Determine the candidate feature words as category keywords, and use the semantic relationship between category keywords to form classification results; build a knowledge base from the mining results, and the knowledge base can also be set to support text mining such as public opinion topic discovery and public opinion tendency analysis. Features.

Firstly, the similarity between texts is defined and calculated, that is, the degree of relevance between the topics discussed between texts, and Sim(D ₁ , D ₂ ) is used to represent the similarity between text D ₁ and text D ₂ . The value range of similarity is between ₀ and ₁ , which is proportional to the similarity between texts D1 and D2. The greater the similarity between texts, the greater the topic correlation between texts. The semantic similarity evaluation method between texts is as follows:

Let the text after public opinion information extraction and preprocessing in step b be D ₁ (t ₁₁ ,t ₁₂ ,t ₁₃ ,…,t _1m ), D ₂ (t ₂₁ ,t ₂₂ ,t ₂₃ ,…,t _2m ) , calculate the similarity between all keywords t _1i in text D ₁ and all keywords t _2i in text D ₂ , and form a similarity matrix as follows:

$\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left(\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; Sim</span>}}_{\mathrm{<span\; class="notranslate">\; 11</span>}}& {\mathrm{<span\; class="notranslate">\; Sim</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}\mathrm{<span\; class="notranslate">\; m</span>}}\\ {\mathrm{<span\; class="notranslate">\; Sim</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}& {\mathrm{<span\; class="notranslate">\; Sim</span>}}_{\mathrm{<span\; class="notranslate">\; mm</span>}}\end{array}\right)$

Sim _ij (1=i, j=m) represents the similarity between text D ₁ keyword t _1i and text D ₂ keyword t _2j ; M(D ₁ , D ₂ ) represents the similarity between text D ₁ and text D ₂ Similarity matrix; i is the number of keywords in text D1; _m is the number of keywords in text D2 _;

The formula for calculating word similarity is: S(T ₁ ,T ₂ )=Max(i=1,2,…,n;j=1,2,…,m)S(y _1i ,y _2j ), that is, word similarity The degree is the maximum value of the similarity between all senses of two words (multiple senses contained in a word).

Traverse the similarity matrix M in turn, find the corresponding combination of keywords with the largest similarity Sim value, and delete the corresponding row and column. Then continue to traverse the similarity matrix M to find the keyword combination with the largest similarity value, and repeat the cycle until the matrix M is a zero-valued matrix. Finally, the semantic similarity of texts D ₁ and D ₂ is obtained by using the keyword combination sequence with the largest similarity obtained, and the calculation formula is as follows:

$\mathrm{<span\; class="notranslate">\; Sim</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}}\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; Sim</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>$

Among them, max is the maximum value of similarity Sim; i is the number _of keywords in text D1 _; j is the number of keywords in text D2.

An improved text clustering analysis method based on semantic similarity is described as follows:

1. First, after preprocessing all the collected texts, use the TFIDF weighting method to perform feature weighting on all category keywords, and extract m optimal feature keywords to form the original keyword-based feature vector Di*.

2. Preprocessing the keywords in the original keyword-based feature vector Di* according to the knowledge base: find and replace the words matching the keywords in the knowledge base to form new feature vectors D _i , D _i =(T ₁ ,T ₂ ,...,T _i ), i=1,2,3,...,m.

3. Form m feature vectors D _i of n texts, use the text semantic similarity calculation formula to calculate the semantic similarity between the collected texts, form the similarity matrix M of the text set, and calculate the average similarity of all feature vectors Degree MA. Calculated as follows:

$m=\left(\begin{array}{cc}{S}_{11}& S_{1\mathrm{no}}\\ S_{\mathrm{no}1}& S_n\end{array}\right),\mathrm{MA}=\frac{\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}\underset{j=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}\mathrm{Sij}-\mathrm{no}\underset{i=1}{\overset{\mathrm{no}}{\mathrm{\&Sigma;}}}\mathrm{Sii}}{\mathrm{no}*(\mathrm{no}-1)};$ Among them, n is the number of texts;

4. Set three similarity thresholds, one repetition threshold is 0.9, one topic center threshold is 0.5, and one new topic threshold is 0.3;

5. Compare the text with the central topic. If the initial central similarity between the text and the central topic is greater than the repetition threshold of 0.9, the text is considered to belong to the same topic and the same content text; if the similarity is less than the new topic threshold of 0.3, the text needs to create a new one. class; if the similarity is in the range of 0 to 0.5, the text belongs to the core content text discussed on different sides of the same topic, and is marked as the second center, and so on, forming a hierarchical clustering result of multiple centers.

6. For the topic representation method of multiple centers, select the maximum value of the similarity between the text and each center in the class as the similarity of the text in this class.

The improved text clustering algorithm based on semantic similarity is suitable for cluster analysis of dynamic data and discovery of public opinion topic hotspots in a large-scale network environment. It can detect new events in time, detect and track new public opinion topics; The public opinion theme representation method in the center has effectively improved the operating efficiency of the system, and the effect will be more obvious as the number of texts increases.

5) Analysis of public opinion information

The public opinion information analysis module performs OLAP multidimensional statistical analysis on the data that has been stored in the public opinion information database and has been mined in step c, and analyzes the public opinion evaluation indicators such as the degree of attention to the content of the public opinion theme and the emotional tendency of the public opinion theme, so as to grasp the dynamics of public opinion in time for relevant departments , Release public opinion information in a timely manner, and provide support for making correct decisions.

The public opinion topic generated through collection, processing and mining analysis is expressed as: T=(T ₁ ,T ₂ ,…,T _n ), where T _i represents the text of the public opinion topic. The attention degree of public opinion topic text is expressed as: T _i =αN _p +βN _r , and the attention degree measurement formula of public opinion topic is: Among them, α and β represent the weight, N _p represents the number of clicks on the public opinion topic text, and N _r represents the number of comments; Np_i represents the number of hits on the i-th public opinion topic text, and Nr_i represents the number of comments on the i-th public opinion topic text. Since N _p >N _r , after statistics, the value of α is 0.02, and the value of β is 0.98.

The emotional tendency of public opinion topics is based on the cluster analysis data description of public opinion topic texts. First, a threshold is set, and only when the tendency measure of the text is greater than the threshold, the text shows polarity (positiveness, negativity). If the propensity measure value of the text is positive, then the text is a positive comment, otherwise it is a negative comment.

After public opinion information is collected, preprocessed, information extracted, mined and analyzed, detailed data on public opinion topics can be obtained, processed according to the established public opinion index evaluation system, and the processed results provide decision-making assistance.

Claims (7)

1. A method for analyzing network public opinion information based on text semantic correlation, characterized in that: the method comprises a network public opinion information collection module, a public opinion information extraction module, a public opinion information preprocessing module, a public opinion information mining module, a public opinion information analysis module and a public opinion information database The network public opinion information analysis system includes the following steps: a. The network public opinion information collection module collects various public opinion information from web pages and stores them in the public opinion information database; b. The public opinion information extraction module and the public opinion information preprocessing module preliminarily filter and segment the public opinion information collected in step a, extract the content information contained in the text, and provide data services for public opinion information mining; c. On the basis of step b, the public opinion information mining module adopts an improved text clustering analysis method based on semantic similarity to generate category description information, and screen out the text information contained in the clustering analysis results; use TFIDF word frequency features based on feature statistics The calculation method counts category features, obtains category feature words, selects nouns as candidate category feature words, sorts according to the weight of candidate feature words, uses the candidate feature words with larger weight values as category keywords, and utilizes the semantic relationship between category keywords, Form classification results; identify and establish new online public opinion topics, detect and track relevant content of existing public opinion topics; d. Finally, the public opinion information analysis module performs OLAP multi-dimensional statistical analysis on the data mined by the public opinion information through step c, and analyzes the public opinion evaluation indicators such as the degree of attention to the content of the public opinion topic, the emotional tendency of the public opinion topic; In step a, the public opinion information collection module is to collect network public opinion information sources, not only to complete the crawling of web pages, but also to format the content of the web pages, extract the theme and content of public opinion, and store the obtained data in txt or html format files, and store them in the public opinion information database; the network public opinion information collection module adopts time-sharing access, regular IP address replacement and simulated browser for single sign-on to prevent shielding. 2. the network public opinion information analysis method based on text semantics correlation according to claim 1, it is characterized in that, the concrete step that described public opinion information collection module executes is, starts from the URL of the relevant webpage of pre-defined topic, obtains in the webpage text information, and extract new URLs from the current webpage and put them into the queue until the collection of public opinion information that satisfies the conditions is completed and the URL queue is empty; the collected webpage text information is stored in the public opinion information database according to the field classification, Provide public opinion information extraction module calls. 3. the network public opinion information analysis method based on text semantic correlation according to claim 1, it is characterized in that, in step b, described public opinion information extraction module, is to remove the irrelevant content in the webpage, extracts useful to public opinion analysis The meta-information in the text part reconstructs the text and gathers the representative information of the theme together; the public opinion information preprocessing module extracts the collected public opinion information sources through the public opinion information extraction module, and performs Chinese Word segmentation processing, filtering stop words, named entity recognition, part-of-speech tagging, grammatical analysis and feature word extraction, establishment of forward index and inverted index; establishment of text feature semantic network graph, with entity E contained in the text as the node of the graph, The semantic relationship between two entities is used as the directed edge of the graph, and the semantic relationship between the entities is combined with the word frequency information as the weight of the node. The weight of the directed edge indicates the importance of the entity relationship in the text. The entity E includes things Entity NE, event entity VE, and event relationship entity RE; count the word frequency and text frequency information of the text, and then perform feature word extraction, and select words that reflect text features to represent the text. 4. the network public opinion information analysis method based on text semantic correlation according to claim 3, it is characterized in that, in step c, described public opinion information mining module is carrying out preprocessing to text collection, comprises Chinese word segmentation process, After stop word filtering and structured label information analysis, the text dataset generated by the information extraction module is used to calculate the semantic similarity between texts using the similarity evaluation method based on the text semantic feature description structure constructed by the text feature semantic network graph. Construct a similarity matrix, use the improved text clustering analysis method based on semantic similarity to generate clustering results; generate category description information from the clustering analysis results, and filter out the text information contained in the clustering analysis results; use TFIDF word frequency based on feature statistics The feature calculation method counts category features, obtains candidate category feature words, selects nouns as candidate category feature words, sorts candidate feature words according to their weight, determines candidate feature words as category keywords with weight values, and utilizes the semantic relationship between category keywords, Form classification results; build knowledge base with mining results. 5. according to claim 3 or 4 described network public opinion information analysis methods based on text semantic correlation, it is characterized in that, text feature semantic network graph is the directed graph that utilizes entity and its semantic relation to express public opinion information, through network node The expressed words are merged, and the node weights are added; then the directed edges are merged, and the weights of the directed edges are added to construct a text feature semantic network graph to describe the semantic information and topic features in the text. 6. the network public opinion information analysis method based on text semantic correlation according to claim 4, it is characterized in that, the semantic similarity evaluation method between texts is: Let the text after public opinion information extraction and preprocessing in step b be D ₁ (t ₁₁ ,t ₁₂ ,t ₁₃ ,…,t _1m ), D ₂ (t ₂₁ ,t ₂₂ ,t ₂₃ ,…,t _2m ) , calculate the similarity between all keywords t _1i in text D ₁ and all keywords t _2i in text D ₂ , and form a similarity matrix as follows: $\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\left(\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; Sim</span>}}_{\mathrm{<span\; class="notranslate">\; 11</span>}}& {\mathrm{<span\; class="notranslate">\; Sim</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}\mathrm{<span\; class="notranslate">\; m</span>}}\\ {\mathrm{<span\; class="notranslate">\; Sim</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; 1</span>}}& {\mathrm{<span\; class="notranslate">\; Sim</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; m</span>}}\end{array}\right)$ Sim _ij (1=i, j=m) represents the similarity between text D ₁ keyword t _1i and text D ₂ keyword t _2j ; M(D ₁ , D ₂ ) represents the similarity between text D ₁ and text D ₂ Similarity matrix; i is the number of keywords in text D1; _m is the number of keywords in text D2 _; Word similarity calculation formula S(T ₁ ,T ₂ )=Max _{(i=1,2,…,n; j=1,2,…,m)} S(y _1i ,y _2j ), that is, the word similarity is The maximum value in the similarity of all meaning items of two words, and described meaning item refers to a plurality of meanings contained in a word; Traverse the similarity matrix M in turn, find the keyword combination with the largest similarity Sim value, and delete the corresponding row and column; then continue to traverse the similarity matrix M to find the keyword combination with the largest Sim value, and repeat the cycle until the matrix M is zero value matrix; finally, use the keyword combination sequence with the largest similarity obtained to obtain the semantic similarity of texts D ₁ and D ₂ , and the calculation formula is as follows: $\mathrm{<span\; class="notranslate">\; S</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}}\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; Sim</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>$ Among them, max is the maximum value of similarity Sim; i is the number _of keywords in text D1 _; j is the number of keywords in text D2. 7. the network public opinion information analysis method based on text semantic correlation according to claim 6, is characterized in that, the improved text cluster analysis method based on semantic similarity is: 1) First, after preprocessing all the collected texts, use the TFIDF weighting method to carry out feature weighting on all category keywords, and extract m optimal feature keywords to form the original keyword-based feature vector Di*; 2) Preprocessing the keywords in the original keyword-based feature vector Di* according to the knowledge base: find the words matching the keywords in the knowledge base and replace them to form new feature vectors D _i , D _i =(T ₁ ,T ₂ ,...,T _i ), i=1,2,3,...,m; 3) Form m eigenvectors D _i of n texts, use the text semantic similarity calculation formula to calculate the semantic similarity between the collected texts, form the similarity matrix M of the text set, and calculate the average similarity of all eigenvectors degree MA; the calculation formula is as follows: Among them, n is the number of texts; 4) Set three similarity thresholds, one repetition threshold is 0.9, one topic center threshold is 0.5, and one new topic threshold is 0.3; 5) Compare the text with the central topic, if the initial central similarity between the text and the central topic is greater than the repetition threshold 0.9, the text is considered to belong to the same topic and the same content text; if the similarity is less than the new topic threshold 0.3, the text needs to create a new one class; if the similarity is in the range of 0 to 0.5, the text belongs to the core content text discussed on different sides of the same topic, and is marked as the second center, and so on, forming a hierarchical clustering result of multiple centers; 6) For the topic representation method of multiple centers, select the maximum value of the similarity between the text and each center in the class as the similarity of the text in this class.