CN101149739A - An internet-oriented meaningful string mining method and system - Google Patents

Abstract

The invention discloses an Internet-oriented meaningful string mining method and system. The method includes the following steps: step A, repeat character string discovery; step B, filter the character string through context adjacency analysis; step C, filter the character string through language model analysis. It can effectively extract meaningful strings in web pages or large-scale text data.

Description

An internet-oriented meaningful string mining method and system

technical field

The invention relates to an information retrieval field and an operating system field, in particular to an Internet-oriented meaningful string mining method and system.

Background technique

There is a vast amount of information on the Internet, but its huge amount makes it difficult for Web users to effectively obtain useful information from it. Users often feel overwhelmed by the vast ocean of information that is updated day and night, and do not know how to learn from the massive amount of information. Looking for the information you really want, let alone how to obtain or grasp the key information in the massive information, and grasp the current important information in time. At the same time, in the face of new information emerging all the time, no one can "see six directions and listen to all directions". At this time, people urgently need the strong support of natural language processing technology to deal with the increasingly serious problem of information overload.

Extracting useful key information from massive network information has become a major problem, and it has also become an urgent need to be solved in the era of network information explosion. The solution to this problem will also have broad application prospects: for individuals, it can be used to more conveniently discover and organize current important information, and it can become an entry point for people to control massive amounts of information. For enterprises, it can be used to keep abreast of the latest developments in related fields, the development direction of strategic partners, and the latest actions of competitors, and provide information assistance for enterprises to formulate strategic guidelines. For the country, it can be used to understand current important social events, fashion trends, public opinion directions, etc., and become an information window to understand and grasp social conditions, and provide assistance for relevant decision-making.

In this context, how to extract useful information from web texts has highlighted its own importance and become a direction worthy of further study.

Contents of the invention

The purpose of the present invention is to provide an Internet-oriented meaningful string mining method and system, which can effectively extract meaningful strings in web pages or large-scale text data.

A kind of digging method of Internet-oriented meaningful string provided for realizing the purpose of the present invention, comprises the following steps:

Step A, repeat string discovery;

Step B, filtering the character string through contextual adjacency analysis;

Step C, analyzing and filtering the character string through a language model.

Described step A comprises the following steps:

Step A1, process the webpage corpus to obtain formatted plain text files, classify the text files, record the repeated strings and their frequency of occurrence in the text, and filter out the strings whose occurrences are less than a certain threshold.

Said step B comprises the following steps:

Step B1, calculate the context adjacency feature value of each repeated string, and judge whether these feature values reach the set threshold, and filter out the text strings that do not reach the threshold according to the judgment result.

Described step C comprises the following steps:

Step C1, scan the adjacent word pairs of the text string one by one, find the coupling degree of the adjacent word pairs, filter the text string according to the coupling degree, and then further filter according to the position word probability of the text string to obtain a meaningful string.

Said step A1 comprises the following steps:

Step A11, processing the webpage corpus to obtain a formatted plain text file, and then converting Chinese characters into corresponding IDs;

Step A12, build an index on the processed ID sequence, expand from the information of each single-word index to obtain all repeated strings, write the newly generated repeated strings into the file, continue to expand to obtain long strings, and iterate repeatedly until an interval symbol or When the length reaches the specified threshold, stop the extension;

Step A13, record the adjacent word information and document information of each string, each type of information is independently stored in a file.

Said step B1 comprises the following steps:

Step B11, calculating the contextual adjacency feature quantity of each repeated string, and judging whether these feature quantities reach the set threshold;

Step B12, if the threshold is reached, go to step C;

Step B13, if the feature quantity does not reach the threshold, it is filtered out.

Said step C1 comprises the following steps:

Step C11, mark a part of the training corpus, and generate a dictionary of coupling degrees of adjacent words and a dictionary of probability of forming words by word positions;

Step C12, scanning adjacent word pairs word by word, looking for the coupling degree of adjacent word pairs;

Step C13, when the coupling degree of the adjacent word pair is less than the set threshold, it does not constitute a part of the word, and is filtered out as a garbage string;

Step C14, for the character strings that are not filtered out by the adjacent word pairs, search for the word probability of the single word position, and judge whether the string head and the string tail include commonly used function words;

Step C15, if it is a function word, it is filtered out;

In step C16, the characters that have not been filtered out are determined as meaningful strings.

Also provide a kind of Internet-oriented meaningful string mining system for realizing the object of the present invention, comprising:

The repeated string discovery module is used to process the webpage corpus to obtain formatted plain text files, classify the text files, record the repeated strings and their frequency of occurrence in the text, and filter out the strings whose occurrences are less than a certain threshold ;

The context adjacency analysis module is used to calculate the context adjacency feature quantity of each repeated string, and judge whether these feature quantities reach a set threshold, and filter out text strings that do not reach the threshold according to the judgment result;

The statistical language model analysis module is used to scan the adjacent word pairs of the text string one by one, find the coupling degree of the adjacent word pairs, and filter the text string according to the coupling degree to obtain a meaningful string.

The statistical language model analysis module is also used to further filter character strings to obtain meaningful strings according to the probability of forming words at positions of text strings after scanning adjacent word pairs.

The context adjacency feature quantity is one or a combination of adjacency set, adjacency seed type, adjacency entropy, adjacency pair set, adjacency pair type, and adjacency pair entropy.

The character strings that appear repeatedly in the record text and their frequency of occurrence are obtained by discovering repeated strings through the suffix tree algorithm, sequitur algorithm, n-ary incremental distribution algorithm or improved n-ary incremental distribution algorithm.

The beneficial effects of the present invention are: the Internet-oriented meaningful string mining method and system of the present invention realizes the purpose of mining meaningful strings through three stages of repeated string discovery, context adjacency analysis, and statistical language model analysis in the text to be recognized. The present invention has done participle in the preprocessing, further reduces the time complexity that repeating string finds, also improves the accuracy rate and the recall rate of extraction result greatly simultaneously; The space complexity that repeating string finds is O(N) (N is corpus scale), it can analyze plain text data comparable to the size of memory, which is about 10 times larger than the traditional suffix tree method; different feature quantities can be used in adjacency analysis according to application needs, and adjacency entropy tends to discover various Strings with relatively uniform linguistic environment distribution, these strings have a wide spatial distribution and are often versatile; finally, the double word coupling degree is used to measure the tightness of the combination of two words, combined with the judgment of stop words, it is more flexible and intelligent .

Description of drawings

Fig. 1 is the schematic diagram of the mining method process of the Internet-oriented meaningful string of the present invention;

Fig. 2 extracts meaningful string process flow chart from repeated string among Fig. 1;

Fig. 3 is the flow chart of the judging process of the beginning and end of a string of meaningful strings facing the Internet in the present invention;

Fig. 4 is a schematic diagram of the Internet-oriented meaningful string mining system of the present invention.

Detailed ways

In order to make the purpose, technical solution and advantages of the present invention clearer, the method and system for mining meaningful strings oriented to the Internet of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The present invention defines a character string that has useful information on the Internet and is used in various environments as a meaningful string. The most important feature of meaningful strings is semantic integrity. The present invention analyzes statistics, structure, pragmatics and semantics, and proposes a universal mining method and system for meaningful strings.

The present invention divides meaningful string mining method process into repeated string discovery, context adjacency analysis, language model analysis three stages, and whole process as shown in Figure 1, comprises the following steps:

Step S100, in the stage of discovering repeated strings, process the webpage corpus to obtain formatted plain text files, classify the text files, record the strings that appear repeatedly in the text and their frequency of occurrence, and list the strings that appear less than a certain threshold filter out.

Step S200, in the context adjacency analysis stage, calculate the context adjacency feature value of each repeated string, judge whether these feature values reach the set threshold, and filter out text strings that do not reach the threshold according to the judgment result.

Step S300, in the statistical language model analysis stage, scan the text string for adjacent word pairs one by one, find the coupling degree of adjacent word pairs, filter the text string according to the coupling degree, and then further filter according to the position word probability of the text string And get a meaningful string.

The present invention mainly uses two standards to measure. First, the present invention calculates the closeness of the combination of two adjacent words in a character string, and if the degree of closeness is less than a certain threshold, the character string is deleted.

Secondly, the present invention also will test the word in a word, the probability that appears in its current position (the position refers to the beginning or the end of a word), if the probability is lower than a certain threshold, just delete this word.

In step S100, process the webpage corpus to obtain a formatted plain text file, classify the text file, record the strings that appear repeatedly in the text and their frequency of occurrence, and filter out the strings that appear less than a certain threshold the process of.

Process the webpage corpus to obtain a formatted plain text file, and then perform preprocessing, including word segmentation, to convert Chinese characters into corresponding IDs. The word segmentation part adopts the faster maximum matching word segmentation method. Experiments show that the word segmentation dictionary contains 6-step multi-core vocabulary, and the word segmentation process does not recognize unregistered words but performs word segmentation. The effect of the step of maximum matching word segmentation is significantly better than the result without word segmentation.

Build an index on the processed ID sequence, expand from the information of each single-word index to get all the repeated strings, after the newly generated repeated strings are written into the file, continue to expand to get a long string, and iterate repeatedly until the interval symbol appears or the length reaches the specified When the threshold is exceeded, the expansion is stopped. At the same time, it is also necessary to record the adjacent word information and document information of each string, and each type of information is stored in a file independently.

At present, the more mature repeated string discovery algorithms applied to Chinese text include suffix tree algorithm, sequitur algorithm and n-ary incremental distribution algorithm. Applying any one of these algorithms can achieve the purpose of counting repeated strings. The embodiment of the present invention adopts an improved n-element incremental distribution algorithm. The specific method is as follows.

The time complexity of the method of the present invention is lower than that of the n-element incremental algorithm, because the index records the address information of each string, and when expanding, the next extended character is directly located according to the address information and the string length, and the range of statistical frequency information is only The current extension string does not need to traverse the entire corpus for global comparison statistics.

At the same time, it is also necessary to record the adjacent word information and document information of each string, and each type of information is stored in a file independently. In the subsequent analysis of meaningful strings, the document information and adjacency pair information of the strings need to be used. If repeated strings are found and then the above statistics are performed, the entire corpus needs to be traversed multiple times, which increases time overhead. However, when repeated strings are found, the address information of each string is known, and the above information can be obtained almost without increasing the time complexity.

It is verified by experiments that if the text is segmented before searching for repeated strings, the effect of meaningful string mining will be better.

The following describes in detail the process of calculating the contextual adjacency feature values of each repeated string in step S200, judging whether these feature values reach a set threshold, and filtering out text strings that do not reach the threshold according to the judgment result.

In order to describe the flexibility of the context environment of the character string S, the present invention proposes a series of context adjacency feature quantity concepts, namely adjacency set, adjacency seed type, adjacency entropy, and adjacency pair set, adjacency pair type, adjacency pair entropy.

Adjacency set: divided into left adjacency set L _NB and right adjacency set R _NB , which respectively refer to the set of words or word elements adjacent to the left or right of the string S in the real text.

Adjacency category: It is divided into left adjacency category V _L and right adjacency category V _R , respectively referring to the number of words or word elements in the left adjacency set and the right adjacency set, which reflect the number of context types above and below the string S .

Adjacency entropy: Indicates the information entropy of the adjacency set of string S, and string S has left adjacency entropy and right adjacency entropy.

Correspondingly, the concepts of contextual adjacency feature values such as adjacency pair set, adjacency pair type, and adjacency pair entropy are also proposed.

Adjacent pair set: the left and right adjacent elements of each occurrence of the string S form an adjacent pair <Li, Ri>, and all adjacent pairs of the string S form the adjacent pair set PNB.

Adjacency pair type: The number of elements in the adjacency pair set PNB is called the adjacency pair type VP.

Adjacency pair entropy: Indicates the information entropy of the adjacency pair set.

These context adjacency features can be used to measure a character string context.

As shown in Figure 2, contextual adjacency analysis mainly calculates the contextual adjacency feature quantity of each repeated string, including adjacency set, adjacency seeding type, adjacency entropy, and adjacency pair set, adjacency pair type, adjacency pair entropy, etc., to judge whether these feature quantities Reach the set threshold, if reached, it means that the string is more flexible in language use, and enter the statistical language model analysis stage.

The calculation of context adjacency features of repeated strings, including adjacency sets, adjacency seed types, adjacency pair sets, and adjacency pair types, is obtained through statistics on the corpus of repeated strings.

Entropy (including adjacency entropy and adjacency pair entropy) is obtained by calculation.

The formula for calculating entropy is as follows:

For example, each element l _i in the adjacency set (such as the left adjacency set) L _NB corresponds to an occurrence frequency n _i in the real text, and the sum of the frequencies is recorded as N, then the entropy calculation formula is:

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; L</span>}}<span\; class="notranslate">\; |</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\frac{{\mathrm{<span\; class="notranslate">\; no</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; no</span>}}\mathrm{<span\; class="notranslate">\; log</span>}<span\; class="notranslate">\; (</span>\frac{{\mathrm{<span\; class="notranslate">\; no</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; )</span>$

For example: The new word "bird flu" has been used frequently since 2000, appearing in the following sentences:

Zhong Nanshan revealed that the bird flu virus has not mutated significantly.

The situation of bird flu prevention and control in Guangdong has slowed down.

Seven people were infected with bird flu.

A suspected case of avian influenza has been detected.

Promulgated five bans to prevent and control bird flu.

If words are used as the granularity of adjacency analysis, the calculation results of the context adjacency feature quantities in the strings of "bird flu" are:

Left-adjacent set: L _NB = {disclosure, prevention and control, infection, one case}

Right adjacency set: R _NB = {virus, situation, event, suspected, EOS}

Left Neighbor Class: V _L = 4

Right Neighbor Class: _VR = 5

Left adjacency entropy: ${\mathrm{E.}}_L=-(\frac{1}{5}\log \frac{1}{5}+\frac{2}{5}\log \frac{2}{5}+\frac{1}{5}\log \frac{1}{5}+\frac{1}{5}\log \frac{1}{5})=-0.718$

Right adjacency entropy: ${\mathrm{E.}}_R=-(\frac{1}{5}\log \frac{1}{5}+\frac{1}{5}\log \frac{1}{5}+\frac{1}{5}\log \frac{1}{5}+\frac{1}{5}\log \frac{1}{5}+\frac{1}{5}\log \frac{1}{5})=-0.699$

Adjacent pair set PNB={<disclosure, virus>, <control, situation>, <infection, event>, <one case, suspected>, <control, EOS>}

Adjacency pair type: PNB=5

adjacency pair entropy ${\mathrm{E.}}_P=-(\frac{1}{5}\log \frac{1}{5}+\frac{1}{5}\log \frac{1}{5}+\frac{1}{5}\log \frac{1}{5}+\frac{1}{5}\log \frac{1}{5}+\frac{1}{5}\log \frac{1}{5})=-0.699$

If the feature quantity does not reach the threshold, it means that the string is a garbage string, and it will be filtered out. Among them, the threshold is obtained from the training corpus.

The corpus is the language material that has actually appeared in the actual use of the language; the basic resource of language knowledge carried by the computer as the carrier. Real corpus needs to be processed (analyzed and processed) before it can become a useful resource.

The corpus training method is a prior art, such as the method of training the training corpus by Hidden Markov Model (HMM). It is not the invention point of the present invention, therefore, it will not be described in detail one by one in the present invention.

It is verified by experiments that the accuracy rate of the unit of adjacent elements is a word is higher than that of a word.

The following describes step S300 in detail. The adjacent word pairs are scanned word by word for the text string, the coupling degree of the adjacent word pair is found, the text string is filtered according to the coupling degree, and then the word formation probability of the text string is further filtered to obtain a meaningful string process.

In order to describe the tightness of combination of two consecutive words in a word, the present invention defines the concept of coupling degree of adjacent word pairs. Its definition is: scan all the consecutive word pairs that have appeared in the segmented training corpus, count the total number of occurrences of each word pair and the total number of times the word pair is used as a certain word string, the difference between the latter and the former The ratio is called the coupling degree of adjacent word pairs, represented by the symbol Coup. For example, the two-character pair of "Guomu" appears 16 times in the statistics of this article, among which it appears 12 times in words like "Knowing the memory" and "One by one", and it appears in the context of "beyond the present". 4 times, so Coup(<over, mesh>)=12/(12+4)=0.75.

The higher the Coup value, the higher the combination degree of the word pair, and vice versa, the less likely the word pair appears in a word. The degree of coupling is obtained from the training corpus.

In addition, the present invention introduces the positional word probability to represent the probability that a certain Chinese character appears at a certain position (word initial or lexeme, etc.). For example, the probability of the beginning of the word "A" is very high, but the probability of the end of the word is very small. If "A" appears at the end of a word, it can basically be considered that the word is a string of garbage. The word-forming probability of the position is also obtained from the training corpus.

Before language module analysis, a part of the training corpus should be manually annotated to generate a dictionary of coupling degrees of adjacent words (such as a dictionary of double-word coupling degrees) and a dictionary of word-forming probability of single-word positions.

As shown in Figure 3, first scan the adjacent word pairs word by word to find the coupling degree of the adjacent word pairs. For example, when the double word coupling degree is less than the set threshold, it does not constitute a part of a word and should be regarded as a garbage string. delete.

The text strings that are not deleted by the double-word pair scanning will enter the next step of filtering to find the probability of word formation in the single-word position. First, find the word probability of the position of the first character. If the probability is lower than a certain threshold, it means that the word should not appear at the beginning of the word, and then filter it.

For the character strings that have not been deleted, look up the word probability of the position of the tail character to determine whether the string head and the string tail contain common function words, and if it is a function word, filter it. That is, if the word-forming probability of the position is lower than the set threshold, it means that the string should not appear at the end of the word, and it will be filtered.

Preferably, the first word pair in the string is also taken out to judge its double-word coupling degree. If it is greater than a certain threshold, then the word pair is considered to be closely combined, enough to form the head of a certain word, and no longer form a word at the single word position of the first word. Probability is used to judge, which can avoid the problem of absolutization of garbage dictionary. For example, the double word pair of "taxi" constitutes a word. If you only judge the word probability of the position of the first word "的", it may need to be filtered, but first judge the double word coupling degree of the word pair, and find that its coupling degree is high, you should reserve.

After this step, the strings that have not been filtered out are determined to be meaningful strings. Output these meaningful strings, and the process ends.

Wherein, all the thresholds in this process are obtained from the training corpus.

The original webpages from 9 domestic news websites such as Sina and Netease were used as part of the original webpages of the test data. The collection time was between April 19, 2006 and June 14, 2006, with a total of more than 310,000 webpages It is test data with a size of 12G. After extracting the text, the final text size is 470MB. The correct rate of extracting meaningful strings from these news web pages by the method for mining meaningful strings of the present invention can reach 70.55%.

Corresponding to the method for mining meaningful Internet-oriented strings, the present invention also provides a mining system 400 for Internet-oriented meaningful strings, as shown in FIG. 4 , which includes:

The repeated string discovery module 410 is used to process the webpage corpus to obtain formatted plain text files, classify the text files, record the strings that appear repeatedly in the text and their frequency of occurrence, and filter the strings that appear less than a certain threshold. Lose.

The context adjacency analysis module 420 is used to calculate the context adjacency feature value of each repeated string, judge whether these feature values reach a set threshold, and filter out text strings that do not reach the threshold according to the judgment result.

The statistical language model analysis module 430 is used to scan the adjacent word pairs word by word for the text string, find the coupling degree of the adjacent word pair, filter the text string according to the coupling degree, and then further filter and form words according to the position word probability of the text string get a meaningful string.

The Internet-oriented meaningful string mining system 400 of the present invention uses the same process as the Internet-oriented meaningful string mining method. Therefore, in the embodiment of the present invention, the system will not be described repeatedly.

The specific embodiments of the present invention have been described and illustrated above, and these embodiments should be considered as exemplary only, and are not used to limit the present invention, and the present invention should be interpreted according to the appended claims.

Claims (12)

1. An Internet-oriented meaningful string mining method is characterized by comprising the following steps:

step A, repeating character string discovery;

step B, filtering the character string through context adjacency analysis;

and C, analyzing and filtering the character strings through a language model.

2. The method for mining internet-oriented meaningful strings according to claim 1, wherein the step a comprises the following steps:

step A1, processing the webpage linguistic data to obtain formatted plain text files, classifying the text files, recording character strings which repeatedly appear in the text and the occurrence frequency of the character strings, and filtering the character strings of which the occurrence frequency is less than a certain threshold value.

3. The method for mining internet-oriented meaningful strings according to claim 2, wherein the step B comprises the following steps:

and B1, calculating context adjacent characteristic quantities of each repeated string, judging whether the characteristic quantities reach a set threshold value, and filtering out text strings which do not reach the threshold value according to a judgment result.

4. The Internet-oriented meaningful string mining method according to claim 3, wherein the step C comprises the steps of:

and step C1, scanning adjacent character pairs of the text string character by character, searching the coupling degree of the adjacent character pairs, filtering the text string according to the coupling degree, and further filtering according to the position word forming probability of the text string to obtain the meaningful string.

5. The method for mining internet-oriented meaningful strings according to claim 2, wherein the step A1 comprises the following steps:

step A11, processing the webpage corpus to obtain a formatted plain text file, and then converting the Chinese characters into corresponding IDs;

step A12, establishing indexes for the processed ID sequences, starting to expand from the information of each single character index to obtain all repeated strings, continuously expanding to obtain long strings after the newly generated repeated strings are written into a file, repeatedly iterating until interval symbols appear or the length reaches a specified threshold value, and stopping expansion;

and step A13, recording the adjacent word information and the document information of each string, and independently storing each type of information in a file.

6. The method for mining Internet-oriented meaningful strings according to claim 3, wherein the step B1 comprises the following steps:

step B11, calculating context adjacent characteristic quantities of each repeated string, and judging whether the characteristic quantities reach a set threshold value;

step B12, if the threshold value is reached, the step C is carried out;

and step B13, if the characteristic quantity does not reach the threshold value, filtering the characteristic quantity.

7. The Internet-oriented meaningful string mining method according to claim 4, wherein the step C1 comprises the following steps:

step C11, labeling a part of the training corpus to generate a coupling degree dictionary of adjacent words and a word position word forming probability dictionary;

step C12, scanning adjacent word pairs word by word, and searching the coupling degree of the adjacent word pairs;

step C13, when the coupling degree of the adjacent character pairs is smaller than a set threshold value, the adjacent character pairs do not form a part of the characters and are used as garbage strings for filtering;

step C14, searching the character string of adjacent character pairs which is not filtered, searching the position word forming probability of the single character, and judging whether the head and the tail of the character string contain common functional characters;

step C15, if the character is a functional character, filtering the functional character;

in step C16, characters that have not been filtered are determined to be meaningful strings.

8. The Internet-oriented meaningful string mining method according to claim 4, wherein the step C1 comprises the following steps:

step C11', labeling a part of the training corpus to generate a coupling degree dictionary of adjacent words and a word forming probability dictionary of single word positions;

and step C12', taking out the first character pair in the character string, judging the coupling degree of the adjacent characters, if the coupling degree is more than a threshold value, considering that the character pair is tightly combined to form the head of the character, and not judging the word forming probability of the single character position of the first character.

9. An internet-oriented meaningful string mining system, comprising:

the repeated string finding module is used for processing the webpage linguistic data to obtain formatted plain text files, classifying the text files, recording character strings which repeatedly appear in the text and the appearance frequency of the character strings, and filtering out the character strings of which the appearance frequency is less than a certain threshold value;

the context adjacency analysis module is used for calculating the context adjacency characteristic quantity of each repeated string, judging whether the characteristic quantity reaches a set threshold value or not, and filtering out the text strings which do not reach the threshold value according to the judgment result;

and the statistical language model analysis module is used for scanning adjacent character pairs word by word of the text string, searching the coupling degree of the adjacent character pairs, and filtering the text string according to the coupling degree to obtain the meaningful string.

10. The system of claim 9, wherein the statistical language model analysis module is further configured to filter the strings to obtain the meaningful strings according to the position word-forming probability of the text strings after scanning the adjacent word pairs.

11. The system for mining internet-oriented meaningful strings according to claim 9 or 10, wherein the contextual adjacency feature quantity is one or more of an adjacency set, an adjacency category, an adjacency entropy, an adjacency pair set, an adjacency pair category, and an adjacency pair entropy.

12. The system for mining meaningful internet-oriented strings as claimed in claim 9 or 10, wherein the repeated strings and the occurrence frequency thereof in the recorded text are obtained by repeated string discovery through a suffix tree algorithm, a sequitur algorithm, an n-ary incremental distribution algorithm or a modified n-ary incremental distribution algorithm.

Images