CN111104801B - Text segmentation method, system, equipment and media based on website domain name - Google Patents

Abstract

The invention discloses a text segmentation method, system, equipment and medium based on website domain names, including: data collection, collecting several website domain names; performing word segmentation processing on each website domain name; and text formatting processing of words after word segmentation processing; Analyze the text formatting process to obtain the part-of-speech of the word; perform lemmatization according to the word's part-of-speech; store the result after lemmatization in the word library; use the two-way maximum matching algorithm to match the URL domain name to be segmented with the word library , if the match is successful, the text vectorization result will be obtained; if the match fails, the URL domain name to be segmented will be cleaned, and the cleaned result will be matched with the word library using the two-way maximum matching algorithm again.

Description

Text segmentation method, system, equipment and media based on website domain name

Technical field

The present disclosure relates to the technical field of natural language processing, and in particular to text segmentation methods, systems, equipment and media based on website domain names.

Background technique

The statements in this section merely mention background technology related to the present disclosure and do not necessarily constitute prior art. This disclosure is based on the premise of not tracking user behavior and not obtaining user privacy.

In recent years, the Internet has become one of the most important infrastructures of human society, and is having an increasingly broad and profound impact on people's economic and social activities. For users, jumps between different URLs can be regarded as the user's behavioral trajectory. Among the huge amounts of online behavior data generated, the URL domain name is the most representative. It includes the name and nature of the web pages the user browses. , which can fully reflect the user's preference between URLs and the correlation between corresponding URLs.

The website domain name is mainly composed of English letters, Arabic numerals and some special characters "_", "@", "/", etc. The purpose is to facilitate the memory and communication of a group of server addresses (website, email, FTP, etc.).

In the process of realizing the present disclosure, the inventor discovered that the following technical problems exist in the prior art:

First: The URL domain name is extremely short, and existing word segmentation technology cannot effectively extract keywords.

Second: URL domain names are irregular unstructured texts, which makes it more difficult to extract the required refined and understandable knowledge and to vectorize the text later.

Third: When each company, organization or individual sets up their own website domain name, they will name it according to their personal habits. Domain name abbreviations, misspellings, language inconsistencies, etc. often occur.

Fourth: The time and space complexity of web mining for existing website domain names is too high, which can easily cause dimensionality disaster.

These problems will cause data analysts to be unable to quickly obtain the nature information of the web page from the URL domain name, thus affecting the accuracy and efficiency of analyzing user online behavior.

Contents of the invention

In order to solve the deficiencies of the existing technology, the present disclosure provides a text segmentation method, system, device and medium based on a website domain name; it can perform text parsing on any existing website domain name, and can extract keywords with high accuracy Technology.

In the first aspect, this disclosure provides a text segmentation method based on website domain names;

Text segmentation methods based on URL domain names include:

Data collection: collect several website domain names; perform word segmentation processing on each website domain name;

Perform text formatting on the words after word segmentation; analyze the text formatting to obtain the part-of-speech of the word;

Perform lemmatization according to the part of speech of the word; store the result of lemmatization in the word library;

Use the two-way maximum matching algorithm to match the URL domain name to be segmented with the word library. If the match is successful, the text vectorization result will be obtained; if the match fails, the URL domain name to be segmented will be cleaned, and the cleaned results will be used again using the two-way The maximum matching algorithm matches the word library.

In the second aspect, the present disclosure also provides a text word segmentation system based on website domain names;

Text segmentation system based on URL domain name, including:

The data collection module is configured to: collect several URL domain names; perform word segmentation processing on each URL domain name;

A text formatting module, which is configured to: format the words after word segmentation processing; analyze the text formatting processing to obtain the part-of-speech of the word;

The lemmatization module is configured to: perform lemmatization according to the part of speech of the word; store the result of lemmatization into the word library;

Matching output module, which is configured as follows: use the two-way maximum matching algorithm to match the URL domain name to be segmented with the word library. If the match is successful, the text vectorization result will be obtained; if the match fails, the URL domain name to be segmented will be cleaned. , the cleaned results are matched with the word library again using the two-way maximum matching algorithm.

In a third aspect, the present disclosure also provides an electronic device, including a memory, a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the computer instructions in the first aspect are completed. Method steps.

In a fourth aspect, the present disclosure also provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the method described in the first aspect are completed.

Compared with the prior art, the beneficial effects of the present disclosure are:

This method can more quickly eliminate domain name redundancy, meaningless identification and other information that appears when companies, organizations or individuals name their own websites; modify domain name misspellings with higher accuracy; and combine personalized thesaurus and official dictionaries It can segment the main information in the domain name more efficiently and more targetedly. It provides reliable preparation for the vectorization of website domain names in the next step of online behavior analysis. For situations where it is necessary to analyze patterns from the behavior trajectories of a large number of users, this disclosure improves the traditional method of analyzing users' online behavior by loading URL records one by one and then artificially classifying them according to the nature of the web pages. This disclosure provides a time-consuming method It is a very small method that consumes very little space. It does not require loading web pages and is not affected by network bandwidth. It performs text analysis through URL domain names and obtains the properties of web pages in real time. It enhances the timeliness of user online behavior analysis and reduces the research cost of analysis.

Description of the drawings

The description and drawings that constitute a part of this application are used to provide a further understanding of this application. The illustrative embodiments and their descriptions of this application are used to explain this application and do not constitute an improper limitation of this application.

Figure 1 is a method flow chart of the first embodiment;

Figure 2 shows a random piece of raw data after data collection in the first embodiment;

Figure 3 is a piece of data processed by the word segmentation technology based on the very small text of the website domain name in the first embodiment.

Detailed ways

It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meanings commonly understood by one of ordinary skill in the art to which this application belongs.

It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular forms are also intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, it will be understood that when the terms "comprises" and/or "includes" are used in this specification, they indicate There are features, steps, operations, means, components and/or combinations thereof.

Embodiment 1: This embodiment provides a text segmentation method based on website domain names;

As shown in Figure 1, the text segmentation method based on the URL domain name includes:

S1: Data collection, collect several website domain names; perform word segmentation processing on each website domain name;

S2: Perform text formatting on the words after word segmentation; analyze the text formatting to obtain the part-of-speech of the word;

S3: Perform lemmatization according to the part of speech of the word; store the result of lemmatization in the word library;

S4: Use the two-way maximum matching algorithm to match the URL domain name to be segmented with the word library. If the match is successful, the text vectorization result will be obtained; if the match fails, the URL domain name to be segmented will be cleaned, and the cleaned results will be processed again. A two-way maximum matching algorithm is used to match the word library.

As one or more embodiments, in S1, data collection includes collecting several website domain names; specific steps include:

Collect several website domain names, remove the set sensitive words from each website domain name, store the website domain name after removing the sensitive words in units of time, and store it in the data set S.

As one or more embodiments, after the step of collecting several website domain names and before the step of word segmentation processing for each website domain name, the step further includes: a data preprocessing step; the data preprocessing step includes:

S101: Delete missing values or complete missing values for each URL domain name in data set S;

S102: Taking the user as the unit, extract the URL domain name into a column vector.

It should be understood that after the step of collecting several website domain names and before the step of word segmentation processing of each website domain name, a data preprocessing step is also included; the data preprocessing step includes:

Perform data preprocessing and denoising on the data set S. For missing values in the data, if the attribute contains only a very small amount of missing values, you can delete the missing values; if the attribute contains some missing values, you can use Completion is performed using the method of similar mean interpolation.

When performing text segmentation operations on this data, the original data is shown in Figure 2, which contains server, user terminal and other information. For analysis of user online behavior, we need to distinguish between some tags between texts, and classify each Taking the user as a unit, extract the domain name of the browsing website to the column vector L ₁ .

As one or more embodiments, in S1, word segmentation processing is performed on each website domain name; specific steps include:

For each website domain name, use jieba word segmentation tool for word segmentation processing.

It should be understood that in S1, word segmentation processing is performed on each website domain name; specific steps include:

Realize efficient word graph scanning based on Trie tree structure, generate a directed acyclic graph (DAG) composed of all possible word formation situations in English and Chinese sentences, use dynamic programming to find the maximum probability path, and find the maximum segmentation combination based on word frequency. Enter the website domain name column vector L ₁ into the jieba word segmentation full-mode model, remove symbols, scan out all the strings contained in each record that can be regarded as words, and store them in the column vector L ₂ .

As one or more embodiments, in S2, text formatting is performed on the words after word segmentation processing; specific steps include:

Format text after word segmentation, and delete glyphs and set useless characters.

It should be understood that in S2, text formatting is performed on the words after word segmentation; specific steps include:

Perform text formatting operations on the column vector L ₂ , completely delete symbols and useless characters, and record them in units of a URL domain name, with several word strings contained in it as sub-records, and store them in the data set S ₁ .

As one or more embodiments, in S2, the part-of-speech of the word is obtained after analyzing the text formatting process; specific steps include:

Get the part of speech of the current word based on the suffix information in the word.

It should be understood that in S2, the part of speech of the word is obtained after analyzing the text formatting process; the specific steps include:

Use a regular expression annotator to convert the tagset into a unified symbol, use the suffix and other information in English words to infer the part of speech of a word, and match the sub-records in the data set _S1 in order. When none of them match, will be marked as the part of speech with the highest probability, and finally recorded in units of a website domain name, with each English word and its corresponding part of speech as sub-records, and stored in the data set S ₂ .

As one or more embodiments, in S3, lemmatization is performed according to the part of speech of the word; specific steps include:

According to the part-of-speech of the word, the WordNet function is called to perform the lemmatization operation, and then the deformations of various words are restored to the same form, and a dictionary D ₁ is generated.

It should be understood that in S3, lemmatization is performed according to the part of speech of the word; specific steps include:

Extract the English words and their corresponding parts of speech in each sub-record of the data set _S2 , call the WordNet function, perform lemmatization operations, and normalize the deformations of various types of words into one form, and use a URL domain name as the unit Record and store to data set S ₃ .

As one or more embodiments, in S3, the result after lemmatization is stored in the word library; specific steps include:

The user builds a personalized vocabulary D ₂ and uses the StandfordNLP toolkit in NLTK to complete the operation of the vocabulary D ₂ ; take the union of the personalized vocabulary D ₂ and the dictionary D ₁ to generate the vocabulary D3, D3 = D1 ∪ D2 .

As one or more embodiments, in S4, the website domain name to be segmented is matched with the word library using a two-way maximum matching algorithm; specific steps include:

Use the forward maximum matching algorithm to match the URL domain name to be segmented with the vocabulary database D3, and record the matching result R ₁ ;

Use the reverse maximum matching algorithm to match the URL domain name to be segmented with the vocabulary database D3, and record the matching result R ₂ ;

If the matching result R ₁ is equal to the matching result R ₂ , then the matching result R ₁ is selected as the final word segmentation result of the URL domain name to be segmented.

Further, if the matching result R ₁ is not equal to the matching result R ₂ , then the result R ₁ of the forward maximum matching algorithm of the website domain name and the result R ₂ of the reverse maximum matching algorithm of the website domain name is selected, which has a larger number of single English words, as The final result R ₃ of the two-way maximum matching algorithm for the URL and domain name to be matched.

It should be understood that in S4, the website domain name to be segmented is matched with the word library using a two-way maximum matching algorithm; the specific steps include:

First use the forward maximum matching algorithm of the website domain name, and then compare it with the vocabulary D ₃ :

If it is an English word, record it. Otherwise, by adding a word, continue the comparison from left to right until there is one word left, then terminate.

If the string cannot be segmented, it will be treated as not logged in, and the processed URL domain name will be used as a unit, and the dictionary _D3 will be matched again. If the record matches correctly, the result of the forward maximum matching algorithm of this URL domain name will be recorded. _R1 ;

Then S ₃ is compared with the vocabulary D ₃ using the URL domain name reverse maximum matching algorithm:

If it is an English word, record it. Otherwise, by reducing one word, continue the comparison from right to left until there is one word left, then terminate.

If the string cannot be segmented, it will be treated as not logged in. The processed URL domain name will be used as a unit and the vocabulary database D ₃ will be matched again. If the record matches correctly, the result R of the reverse maximum matching algorithm of this URL domain name will be recorded. ₂ .

If R ₁ is equal to R ₂ , the result R ₁ of the forward maximum matching algorithm for the URL and domain name can be selected as the final result R ₃ of the bidirectional maximum matching algorithm for the URL and domain name of this record;

If the matching result R ₁ is not equal to the matching result R ₂ , then the result R ₁ of the forward maximum matching algorithm of the URL domain name and the result R ₂ of the reverse maximum matching algorithm of the URL domain name is selected as the result with a larger number of single English words as the URL to be matched. The final result of the domain name two-way maximum matching algorithm is R ₃ ;

The final result R ₃ is stored in the data set S ₄ .

As one or more embodiments, in S4, if the matching fails, the URL domain name to be segmented is cleaned, and the cleaned result is matched with the word library again using the two-way maximum matching algorithm. The specific steps include:

If the URL domain name to be segmented cannot be correctly matched, the excess strings will be cleaned and the two-way maximum matching algorithm will be returned again until all strings of the URL domain name to be segmented correctly match the vocabulary D ₃ and the operation of storing it in the data set S ₄ is completed. then terminate; the final data set S ₄ is the word segmentation result of the domain name of the website to be segmented.

As can be seen from Figure 2, problems that may arise in domain name URLs include several interference items, such as: dldir1, which have no actual meaning for this type of sample and need to be cleaned; there are word combinations and splicing, such as: checkresupdate, for which several words are concatenated. For samples containing abbreviations and misspellings, it is necessary to select useful words, eliminate meaningless words, and restore abbreviations and misspelled words with the greatest probability;

There are mixed names of character identifiers, such as: 80002486_fa55fa1d3a4b43bab792c6a8ff463f72.zip, wrd_template_HEAD_06281609. For this type of sample, it is necessary to delete the identifier and extract meaningful words in the sample, restore the tense, passive and other transformations of the word, and the file suffix needs to be set relatively A high weight because it has a high degree of discrimination in terms of discriminative properties.

Figure 3 shows a piece of data processed by word segmentation technology based on the very small text of the website domain name.

Table 1 Case 1

Table 2 Case 2

Table 3 Case 3

Table 4 Case 4

Table 5 Case 5

Embodiment 2: This embodiment also provides a text segmentation system based on website domain names;

Text segmentation system based on URL domain name, including:

The data collection module is configured to: collect several URL domain names; perform word segmentation processing on each URL domain name;

A text formatting module, which is configured to: format the words after word segmentation processing; analyze the text formatting processing to obtain the part-of-speech of the word;

The lemmatization module is configured to: perform lemmatization according to the part of speech of the word; store the result of lemmatization into the word library;

Matching output module, which is configured as follows: use the two-way maximum matching algorithm to match the URL domain name to be segmented with the word library. If the match is successful, the text vectorization result will be obtained; if the match fails, the URL domain name to be segmented will be cleaned. , the cleaned results are matched with the word library again using the two-way maximum matching algorithm.

Embodiment 3: This embodiment also provides an electronic device, including a memory, a processor, and computer instructions stored in the memory and run on the processor. When the computer instructions are run by the processor, the steps of Embodiment 1 are completed. Describe the steps of the method.

Embodiment 4: This embodiment also provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the method described in Embodiment 1 are completed.

The above descriptions are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included in the protection scope of this application.

Claims (7)

1. The text word segmentation method based on the website domain name is characterized by comprising the following steps:

data acquisition, namely acquiring a plurality of website domain names; word segmentation processing is carried out on each website domain name, wherein the website domain name is extracted and browsed to a column vector L1 according to each user as a unit; based on the Trie structure, realizing efficient word graph scanning, generating a directed acyclic graph formed by all word forming conditions of Chinese and English in sentences, adopting dynamic programming to search a maximum probability path, finding out a maximum segmentation combination based on word frequency, inputting a website domain name column vector L1 into a jieba word segmentation full-mode model, eliminating symbols, scanning out all character strings which are regarded as words and are contained in each record, and storing the character strings into a column vector L2;

carrying out text formatting processing on the word subjected to word segmentation processing; analyzing word parts of speech of the word obtained after the text formatting process, wherein text formatting operation is carried out on the column vector L2, sign symbols and useless characters are thoroughly deleted, a website domain name is used as a unit for recording, and a plurality of word character strings contained in the character strings are used as sub-records and stored in a data set S1;

adopting a regular expression labeler, converting tagset into unified symbols by formulating, utilizing suffix information in English words to infer the part of speech of a word, and collecting data S ₁ The sub-records in the table are matched in sequence, and when all the sub-records are not matched, the sub-records are marked as probabilityThe largest part of speech is recorded according to a website domain name as a unit, each English word and the part of speech corresponding to the English word are recorded as sub-records and stored in a data set S ₂ ；

Performing morphological reduction according to word parts of speech; storing the word shape restored result into a word library, specifically: according to word part of speech, call WordNet function, carry on the morphological reduction operation, and then reduce the deformation of various words into the same form, produce dictionary D ₁ ；

User-built personalized word stock D ₂ Word library D is completed in NLTK by using StanfordNLP toolkit ₂ Is performed according to the operation of (1); taking personalized word stock D ₂ And dictionary D ₁ Generates word stock by union of (1)，/>；

Matching the website domain name to be segmented with a word library by adopting a bidirectional maximum matching algorithm, and if the matching is successful, obtaining a text vectorization result; if the matching fails, cleaning the website domain name of the word to be segmented, and matching the cleaned result with the word library again by adopting a bidirectional maximum matching algorithm, wherein the method specifically comprises the following steps: the website domain name to be segmented adopts a forward maximum matching algorithm and a word stockMatching is carried out, and matching results are recorded>The method comprises the steps of carrying out a first treatment on the surface of the The website domain name to be segmented adopts a reverse maximum matching algorithm and a word stock +.>Matching is carried out, and matching results are recorded>；

If the result is matchedEqual to match result->Then select the matching result +.>As a final word segmentation result of the website domain name to be segmented;

if the matching result is thatUnequal match results->Selecting the result R of the forward maximum matching algorithm of the website domain name ₁ Result R of reverse maximum matching algorithm with website domain name ₂ The result with more Chinese and single English words is used as the final result R of the bidirectional maximum matching algorithm of the website domain name to be matched ₃ 。

2. The method of claim 1, wherein data collection is performed to collect a plurality of web site names; the method comprises the following specific steps:

and collecting a plurality of website domain names, removing set sensitive words from each website domain name, and storing the website domain names with the sensitive words removed according to time units into a data set S.

3. The method of claim 1, wherein after the step of collecting a plurality of web site domain names, before the step of word segmentation for each web site domain name, further comprises: a data preprocessing step; the data preprocessing step comprises the following steps:

s101: deleting the missing value or complementing the missing value of each website domain name in the data set S;

s102: and extracting the website domain name to a column vector by taking the user as a unit.

4. The method of claim 1, wherein each web site domain name is subjected to word segmentation; the method comprises the following specific steps: and performing word segmentation processing on each website domain name by utilizing a jieba word segmentation tool.

5. The text word segmentation system based on the website domain name is characterized by comprising the following components:

a data acquisition module configured to: collecting a plurality of website domain names; word segmentation processing is carried out on each website domain name, wherein the website domain name is extracted and browsed to a column vector L1 according to each user as a unit; based on the Trie structure, realizing efficient word graph scanning, generating a directed acyclic graph formed by all word forming conditions of Chinese and English in sentences, adopting dynamic programming to search a maximum probability path, finding out a maximum segmentation combination based on word frequency, inputting a website domain name column vector L1 into a jieba word segmentation full-mode model, eliminating symbols, scanning out all character strings which are regarded as words and are contained in each record, and storing the character strings into a column vector L2;

a text formatting module configured to: carrying out text formatting processing on the word subjected to word segmentation processing; analyzing word parts of speech of the word obtained after the text formatting process, wherein text formatting operation is carried out on the column vector L2, sign symbols and useless characters are thoroughly deleted, a website domain name is used as a unit for recording, and a plurality of word character strings contained in the character strings are used as sub-records and stored in a data set S1;

adopting a regular expression labeler, converting tagset into unified symbols by formulating, utilizing suffix information in English words to infer the part of speech of a word, and collecting data S ₁ The sub-records in the database are matched in sequence, when all the sub-records are not matched, the sub-records are marked as part of speech with the highest probability, finally the sub-records are recorded according to a website domain name as a unit, each English word and the part of speech corresponding to each English word are used as sub-records, and the sub-records are stored in a data set S ₂ ；

A lexical reduction module configured to: performing morphological reduction according to word parts of speech; knot for restoring word shapeThe results are stored in a word stock, specifically: according to word part of speech, call WordNet function, carry on the morphological reduction operation, and then reduce the deformation of various words into the same form, produce dictionary D ₁ ；

User-built personalized word stock D ₂ Word library D is completed in NLTK by using StanfordNLP toolkit ₂ Is performed according to the operation of (1); taking personalized word stock D ₂ And dictionary D ₁ Generates word stock by union of (1)，/>；

A match output module configured to: matching the website domain name to be segmented with a word library by adopting a bidirectional maximum matching algorithm, and if the matching is successful, obtaining a text vectorization result; if the matching fails, cleaning the website domain name of the word to be segmented, and matching the cleaned result with the word library again by adopting a bidirectional maximum matching algorithm, wherein the method specifically comprises the following steps: the website domain name to be segmented adopts a forward maximum matching algorithm and a word stockMatching is carried out, and matching results are recorded>The method comprises the steps of carrying out a first treatment on the surface of the The website domain name to be segmented adopts a reverse maximum matching algorithm and a word stock +.>Matching is carried out, and matching results are recorded>；

If the result is matchedEqual to match result->Then select the matching result +.>As a final word segmentation result of the website domain name to be segmented;

if the matching result is thatUnequal match results->Selecting the result R of the forward maximum matching algorithm of the website domain name ₁ Result R of reverse maximum matching algorithm with website domain name ₂ The result with more Chinese and single English words is used as the final result R of the bidirectional maximum matching algorithm of the website domain name to be matched ₃ 。

6. An electronic device comprising a memory and a processor and computer instructions stored on the memory and running on the processor, which when executed by the processor, perform the steps of the method of any of claims 1-4.

7. A computer readable storage medium storing computer instructions which, when executed by a processor, cause the steps of the method of any one of claims 1-4.