CN109993216A - A text classification method based on K nearest neighbors KNN and its equipment - Google Patents

Abstract

The invention discloses a text classification method and equipment based on K nearest neighbor KNN. It is used to reduce the computational complexity of text classification, more effectively represent text feature information, and improve the accuracy of text classification. The method includes: decomposing text into words, and extracting words representing feature information of the text from the words; encoding the text into a string vector by using the extracted words; calculating the character by using the KNN model The similarity between the string vector and the sample character string vector in the KNN model, according to the similarity and the classification label corresponding to the sample character string vector, determine the classification label of the character string vector and output.

Description

A text classification method based on K nearest neighbors KNN and its equipment

technical field

The invention relates to the technical field of artificial intelligence, in particular to a text classification method based on K-Nearest Neighbor (k-Nearest Neighbor) and a device thereof.

Background technique

At present, text classification is to automatically classify and mark a text collection according to a certain classification system or standard, which belongs to an automatic classification based on a classification system. The text classification process can be understood as the process of matching the data to be classified with the sample data according to some characteristics of the data to be classified. Generally, there are two ways to extract and classify text data, as shown below:

One is to encode the feature information in the text data into a digital vector, calculate the similarity between the digital vector and the sample digital vector, and determine the classification result of the text data corresponding to the digital vector according to the obtained similarity result.

However, in this method, the dimension of the digital vector is relatively large, with at least several hundreds of dimensions. Therefore, when using the digital vector to calculate the similarity, it is easy to cause an excessive amount of computation; in addition, due to the large number of dimensions, the The sparse distribution of digital vectors and poor transparency reduce the accuracy of text classification.

The other is to encode the feature information in the text data into a structured form, such as encoding into a feature information table, and use the table matching algorithm to calculate the similarity between the feature information table and the sample feature information table. The similarity result determines the classification result of the text data corresponding to the feature information table.

However, in this method, the calculation performance of the table matching algorithm is not stable enough due to the influence of noise.

SUMMARY OF THE INVENTION

The invention provides a text classification method and equipment based on KNN, which can extract text feature information by encoding text into the form of string vectors, calculate the similarity between the string vectors, reduce the amount of computation, and improve the extracted feature information The problem of sparse distribution, and the string vector is more symbolic and transparent, which can more effectively represent the text feature information, which is convenient to improve the accuracy of text classification.

In a first aspect, the present invention provides a KNN-based text classification method, the method comprising:

Decomposing the text into words, and extracting words representing the feature information of the text from the words;

Using the extracted words, encoding the text into a string vector;

Calculate the similarity between the string vector and the sample string vector in the KNN model by using the KNN model, and determine the similarity of the string vector according to the similarity and the classification label corresponding to the sample string vector. Classify labels and output.

As an optional implementation manner, using the KNN model to calculate the similarity between the character string vector and the sample character string vector in the KNN model, including:

Using the KNN model, the cosine similarity algorithm between vectors is used to calculate the similarity between the character string vector and the sample character string vector in the KNN model; or

Using the KNN model, a similarity matrix algorithm for obtaining the similarity between string vectors by calculating the similarity between words is used to calculate the similarity between the string vector and the sample string vector in the KNN model.

As an optional implementation, it also includes:

Obtain training samples including multiple string vectors, and the classification labels corresponding to the string vectors in the training samples;

Initialize the model parameters of the KNN model, and input the training sample into the KNN model;

Utilize the KNN model, adopt the cosine similarity algorithm between vectors, calculate the similarity between the character string vector and the sample character string vector in the KNN model;

According to the classification label corresponding to the string vector in the output training sample and the classification label corresponding to the character string vector in the training, the model parameters of the current KNN model are adjusted until the preset conditions are met.

As an optional implementation, including:

decompose the text into long strings based on the words in the corpus in the server;

The long character string is divided into paragraphs by a text segmentation method, the characters belonging to the stem in each paragraph are extracted, and the characters with no extraction meaning are deleted.

As an optional implementation, extracting words representing the feature information of the text from the words, including:

According to the occurrence frequency, grammatical attribute and position distribution of the words in the words, words representing the feature information of the text are respectively extracted from the words.

As an optional implementation manner, according to the occurrence frequency, grammatical attributes and position distribution of the words in the words, words representing the feature information of the text are respectively extracted from the words, including:

Extract at least one word according to the order of occurrence frequency of the words from high to low;

According to the grammatical attribute of the word, according to the order of the word frequency-inverse text frequency index TF-IDF weighted value in the word from high to low, extract at least one word;

According to the position distribution of the words, the words distributed in the set paragraph are extracted.

As an optional implementation manner, the set paragraph is the first paragraph and/or the last paragraph.

As an optional implementation manner, according to the position distribution of the words, the words distributed in the set paragraphs are extracted, including:

Extract the last word distributed in the last paragraph, the first word of the first paragraph, the last word of the first paragraph, and the first word of the last paragraph.

As an optional implementation manner, determining the classification label of the character string vector according to the similarity and the classification label corresponding to the sample character string vector, including:

Selecting a preset number of sample string vectors corresponding to the string vector similarity from high to low;

calculating the sampling probability of the preset number of sample string vectors;

The classification label of the character string vector is determined according to the classification label corresponding to the sample character string vector selected by the sampling probability.

In a second aspect, the present invention provides a KNN-based text classification device, the device includes: a processor and a memory, wherein the memory stores a program code, when the program code is executed by the processor , so that the processor is used to:

Decomposing the text into words, and extracting words representing the feature information of the text from the words;

Using the extracted words, encoding the text into a string vector;

Calculate the similarity between the string vector and the sample string vector in the KNN model by using the KNN model, and determine the similarity of the string vector according to the similarity and the classification label corresponding to the sample string vector. Classify labels and output.

As an optional implementation manner, the processor is specifically used for:

Using the KNN model, the cosine similarity algorithm between vectors is used to calculate the similarity between the character string vector and the sample character string vector in the KNN model; or

Using the KNN model, a similarity matrix algorithm for obtaining the similarity between string vectors by calculating the similarity between words is used to calculate the similarity between the string vector and the sample string vector in the KNN model.

As an optional implementation manner, the processor is further used for:

Obtain training samples including multiple string vectors, and the classification labels corresponding to the string vectors in the training samples;

Initialize the model parameters of the KNN model, and input the training sample into the KNN model;

Utilize the KNN model, adopt the cosine similarity algorithm between vectors, calculate the similarity between the character string vector and the sample character string vector in the KNN model;

According to the classification label corresponding to the string vector in the output training sample and the classification label corresponding to the character string vector in the training, the model parameters of the current KNN model are adjusted until the preset conditions are met.

As an optional implementation manner, the processor is specifically used for:

decompose the text into long strings based on the words in the corpus in the server;

The long character string is divided into paragraphs by a text segmentation method, the characters belonging to the stem in each paragraph are extracted, and the characters with no extraction meaning are deleted.

As an optional implementation manner, the processor is specifically used for:

According to the occurrence frequency, grammatical attribute and position distribution of the words in the words, words representing the feature information of the text are respectively extracted from the words.

As an optional implementation manner, the processor is specifically used for:

Extract at least one word according to the order of occurrence frequency of the words from high to low;

According to the grammatical attribute of the word, according to the order of the word frequency-inverse text frequency index TF-IDF weighted value in the word from high to low, extract at least one word;

According to the position distribution of the words, the words distributed in the set paragraph are extracted.

As an optional implementation manner, the set paragraph is the first paragraph and/or the last paragraph.

As an optional implementation manner, the processor is specifically used for:

Extract the last word distributed in the last paragraph, the first word of the first paragraph, the last word of the first paragraph, and the first word of the last paragraph.

As an optional implementation manner, the processor is specifically used for:

Selecting a preset number of sample string vectors corresponding to the string vector similarity from high to low;

calculating the sampling probability of the preset number of sample string vectors;

The classification label of the character string vector is determined according to the classification label corresponding to the sample character string vector selected by the sampling probability.

In a third aspect, the present invention provides another KNN-based text classification device, the device includes: a decomposition module, an encoding module and a classification module, wherein:

A decomposition module for decomposing text into words, and extracting words representing the feature information of the text from the words;

an encoding module for encoding the text into a string vector using the extracted words;

The classification module is used to calculate the similarity between the character string vector and the sample character string vector in the KNN model by using the KNN model, and determine the similarity according to the similarity and the classification label corresponding to the sample character string vector. Describe the categorical label of the string vector and output it.

As an optional implementation manner, the classification module is specifically used for:

Using the KNN model, the cosine similarity algorithm between vectors is used to calculate the similarity between the character string vector and the sample character string vector in the KNN model; or

Using the KNN model, a similarity matrix algorithm for obtaining the similarity between string vectors by calculating the similarity between words is used to calculate the similarity between the string vector and the sample string vector in the KNN model.

As an optional implementation manner, the device is also used for:

Obtain training samples including multiple string vectors, and the classification labels corresponding to the string vectors in the training samples;

Initialize the model parameters of the KNN model, and input the training sample into the KNN model;

Utilize the KNN model, adopt the cosine similarity algorithm between vectors, calculate the similarity between the character string vector and the sample character string vector in the KNN model;

According to the classification label corresponding to the string vector in the output training sample and the classification label corresponding to the character string vector in the training, the model parameters of the current KNN model are adjusted until the preset conditions are met.

As an optional implementation manner, the decomposition module is specifically used for:

decompose the text into long strings based on the words in the corpus in the server;

The long character string is divided into paragraphs by a text segmentation method, the characters belonging to the stem in each paragraph are extracted, and the characters with no extraction meaning are deleted.

As an optional implementation manner, the decomposition module is specifically used for:

According to the occurrence frequency, grammatical attribute and position distribution of the words in the words, words representing the feature information of the text are respectively extracted from the words.

As an optional implementation manner, the decomposition module is specifically used for:

Extract at least one word according to the order of occurrence frequency of the words from high to low;

According to the grammatical attribute of the word, according to the order of the word frequency-inverse text frequency index TF-IDF weighted value in the word from high to low, extract at least one word;

According to the position distribution of the words, the words distributed in the set paragraph are extracted.

As an optional implementation manner, the set paragraph is the first paragraph and/or the last paragraph.

As an optional implementation manner, the decomposition module is specifically used for:

Extract the last word distributed in the last paragraph, the first word of the first paragraph, the last word of the first paragraph, and the first word of the last paragraph.

As an optional implementation manner, the classification module is specifically used for:

Selecting a preset number of sample string vectors corresponding to the string vector similarity from high to low;

calculating the sampling probability of the preset number of sample string vectors;

The classification label of the character string vector is determined according to the classification label corresponding to the sample character string vector selected by the sampling probability.

In a fourth aspect, the present invention provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, implements the steps of the method described in the first aspect.

A KNN-based text classification method and device thereof provided by the present invention have the following beneficial effects:

It is used to encode the text into a string vector to extract text feature information, and calculate the similarity between the text string vector and the sample string vector. Since the string vector has less dimensions in the present invention, the similarity between the strings can be improved. In addition, due to the small dimension of the string vector, the problem of sparse distribution of the extracted feature information can be improved. At the same time, the string vector is more symbolic and more transparent, and can more effectively represent the text feature information, which is convenient for improving the The accuracy of text classification.

Description of drawings

1 is a flowchart of a KNN-based text classification method provided by an embodiment of the present invention;

2 is a specific flowchart of a KNN-based text classification method provided by an embodiment of the present invention;

3 is a diagram of a KNN-based text classification device provided by an embodiment of the present invention;

FIG. 4 is a diagram of another KNN-based text classification device provided by an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Example 1

Because the traditional KNN text classification method needs to encode the text into a numerical vector and then input it into the KNN model for text classification, but the encoding of the text into a numerical vector has the problems of many vector dimensions and sparse distribution of text feature information, resulting in KNN-based text encoding. When the numerical vector is used for text classification, the classification result obtained has a low accuracy rate. Therefore, the embodiment of the present invention provides a text classification method based on KNN, which can quickly and effectively obtain the result of text classification by encoding text into a string vector and then inputting it into the KNN model.

As shown in Figure 1, the specific implementation steps are as follows:

Step 10: decompose the text into words, and extract words representing the feature information of the text from the words;

The above-mentioned words can be English words, Chinese words or any single character that can be processed by a computer. The above-mentioned specific methods for decomposing text into words are as follows:

decompose the text into long strings based on the words in the corpus in the server;

The long character string is divided into paragraphs by a text segmentation method, the characters belonging to the stem in each paragraph are extracted, and the characters without the meaning of feature extraction are deleted.

A large number of words and characters corresponding to the words are stored in the above-mentioned corpus, and the words in the text are matched with the words in the corpus to obtain the words that appear in the corpus in the text, and the characters corresponding to the words obtained are as in the text. The sequence of appearance is connected into a long string, and the long string is divided into paragraphs by the text segmentation method, and the characters belonging to the stem in each paragraph are extracted according to the stem extraction rules, in order to improve the extraction efficiency of the text feature information. , removes characters with no extractive meaning, such as propositions, conjunctions, pronouns, etc. The above-mentioned word stems may include, but are not limited to, verbs, nouns, and formative words, etc. After text decomposition, the text can be decomposed into character strings consisting of verb characters, noun characters, and adjective characters. The text segmentation method is in the prior art and will not be described in detail here.

Words representing feature information of the text are extracted from the words, and the feature information may include, but is not limited to, the occurrence frequency, grammatical attributes, and location distribution of the words.

According to the occurrence frequency, grammatical attributes and position distribution of the words in the words, words representing the feature information of the text are extracted from the words, including:

Extract at least one word according to the order of occurrence frequency of the words from high to low;

Specifically, the word with the highest frequency among the words, the word with the second highest frequency among the words, and the word with the third highest frequency among the words can be extracted, and the word with the highest frequency/the word with the highest frequency The word with the second highest frequency/the word with the third highest frequency can be one or more;

According to the grammatical attribute of the word, according to the order of the word frequency-inverse text frequency index TF-IDF weighted value in the word from high to low, extract at least one word;

The TF-IDF is a commonly used weighted statistical method, and the higher the weighted value of the TF-IDF, the better the classification ability of the word, which is suitable for text classification. Specifically, the word with the highest TF-IDF weight value among the words, the word with the second highest TF-IDF weight value among the words, and the word with the third highest TF-IDF weight value among the words can be extracted. In addition, the word with the highest TF-IDF weight value/the word with the second highest TF-IDF weight value/the word with the third highest TF-IDF weight value may be one or more than one word.

According to the position distribution of the words, the words distributed in the set paragraph are extracted. The set paragraph may be the first paragraph, or the last paragraph, or the first paragraph and the last paragraph.

Specifically, according to the position distribution of the words, extracting the words distributed in the set paragraph includes: extracting the last word distributed in the last paragraph, the first word of the first paragraph, the last word of the first paragraph, and the last word of the last paragraph. first word.

Step 11: using the extracted words to encode the text into a string vector;

Specifically, using the extracted words, the encoded string vector consists of the following characters:

The word with the highest frequency in the word, the word with the second highest frequency in the word, the word with the third highest frequency in the word, the word with the highest TF-IDF weighting value in the word, the word in the word The word with the second highest TF-IDF weighting value in the word, the word with the third highest TF-IDF weighting value in the word, the last word distributed in the last paragraph, the first word of the first paragraph, the last word of the first paragraph and the first word of the last paragraph, form a ten-dimensional string vector based on the above 10 characters.

Step 12: Calculate the similarity between the character string vector and the sample character string vector in the KNN model by using the KNN model, and determine the character according to the similarity and the classification label corresponding to the sample character string vector. String vector of categorical labels and output.

Specifically, using the KNN model to calculate the similarity between the character string vector and the sample character string vector in the KNN model, including any of the following methods:

Manner 1: Using the KNN model, a cosine similarity algorithm between vectors is used to calculate the similarity between the character string vector and the sample character string vector in the KNN model.

Specifically, the character string vector in the embodiment of the present invention is a limited character string, and the lengths of different character strings can be defined according to user requirements, and the cosine similarity between the vectors can be calculated as the similarity between the character string vectors. algorithm, the calculation formula of the cosine similarity algorithm is as follows:

Among them, str _i in the above formula is a string vector, d represents the number of characters in the string vector, d _1i represents the ith character in the string vector str ₁ whose text identifier is d ₁ , and d _2i represents the text identifier is d The ith character in the string vector str ₂ of ₂ . sim(str ₁ , str ₂ ) is the cosine similarity between the string vector str ₁ and the string vector str ₂ . The above text identifier d ₁ can identify the text to be classified, and the above text identifier d ₂ can identify the text corresponding to the sample character string vector in the KNN model.

Method 2: Using the KNN model, a similarity matrix algorithm that obtains the similarity between string vectors by calculating the similarity between words is used to calculate the similarity between the string vector and the sample string vector in the KNN model.

Specifically, first, a similarity matrix can be constructed according to the corpus obtained by the server, and each item in each row and each column of the similarity matrix corresponds to two words in the corpus, wherein each item in the similarity matrix Item content represents the similarity between any two words in the corpus.

Assuming that the corpus contains N texts, two texts are arbitrarily selected, a word is selected from the two texts, the similarity between the two words selected from the two texts is calculated, and the two The similarity between words is quantified as a quantified value between 0 and 1. If the two texts are exactly the same, the similarity of each item in the calculated similarity matrix is 1. If the two documents are completely different, then The similarity of each item in the calculated similarity matrix is 0.

Among them, the formula for calculating the similarity between the above two words is as follows:

Wherein, T _i and T _j in the above formula represent two different texts in the corpus, t _i is a word in the text T _i , t _j is a word in the text T _j , and in the T _i and T _j If the number of words is N, then 0<i, j≤N, and i, j are integers.

After obtaining the similarity between any two words in the above two texts, the similarity between the character string vector and the sample character string vector in the KNN model can be calculated based on the obtained similarity.

Further, a cosine similarity operation may be used to calculate the similarity between the character string vector and the sample character string vector in the KNN model based on the obtained similarity. Therefore, the embodiment of the present invention provides the following methods in combination with the first and second modes:

First, using the KNN model, a similarity matrix algorithm that obtains the similarity between string vectors by calculating the similarity between words is used to obtain a similarity matrix. Based on the similarity of any item in the similarity matrix, the cosine similarity between vectors is used. The algorithm calculates the similarity between the character string vector and the sample character string vector in the KNN model.

The embodiment of the present invention not only uses the KNN model to classify the string vector, but also uses a large number of string vector training samples to train the KNN model, as follows:

1) obtaining training samples including multiple string vectors, and the classification labels corresponding to the string vectors in the training samples;

Specifically, the classification labels of string vectors are divided into two categories, positive and negative. Wherein, the classification categories of the text corresponding to the positive class and the negative class are different, and the user can determine the classification class of the text corresponding to the positive class label and the classification class of the text corresponding to the negative class label according to requirements. In the embodiment of the present invention, the number of specific classification categories and the specific categories of the text are not specifically limited, and may be set according to actual needs.

2) initialize the model parameters of the KNN model, and input the training sample into the KNN model;

Specifically, when the KNN model is established, multiple sample string vectors and corresponding classification labels are stored in the KNN model for training the model parameters of the KNN model. At the same time, the KNN model can also store the above corpus, that is The text is broken down into long strings based on the words in the corpus stored in the KNN model.

3) utilize the KNN model, adopt the cosine similarity algorithm between the vectors, calculate the similarity between the character string vector and the sample character string vector in the KNN model;

As an optional embodiment, the KNN model is used, and the similarity matrix algorithm for obtaining the similarity between string vectors by calculating the similarity between words is used to calculate the difference between the string vector and the sample string vector in the KNN model. similarity between.

4) According to the classification label corresponding to the character string vector in the output training sample and the classification label corresponding to the character string vector in the training, the model parameters of the current KNN model are adjusted until the preset conditions are met.

The preset condition that is satisfied may be adjusting the model parameters of the current KNN model until the classification label of the character string vector in the training sample output by the KNN model and the classification label corresponding to the character string vector pre-obtained in the training sample. The accuracy rate meets the preset conditions.

As an optional implementation manner, determining the classification label of the character string vector according to the similarity and the classification label corresponding to the sample character string vector includes the following steps:

1) select a preset number of sample string vectors corresponding to the string vector similarity from high to low;

2) calculating the sampling probability of the preset number of sample character string vectors;

Specifically, when initializing the model parameters of the KNN model, the sampling probability of each sample string vector can be preset, and when the KNN model is trained by using the training samples, the sampling probability of each sample string vector can be determined according to the sampling probability. , the preset sampling probability is adjusted, and when the KNN completes training, the sampling probability adjustment is completed.

3) Determine the classification label of the character string vector according to the classification label corresponding to the sample character string vector selected according to the sampling probability.

The KNN in the embodiment of the present invention performs text classification by calculating the similarity between the character string vector and the sample character string vector in the KNN model, and selects samples according to the sampling probability of the preset number of sample character string vectors. The classification label corresponding to the string vector. Specifically, when KNN is used to classify the text of the string vector, if a string vector belongs to a certain category within the preset number of the k most similar sample string vectors, the The string vector also belongs to this category, otherwise the classification label corresponding to the sample string vector with the highest similarity among the k most similar sample string vectors is selected.

However, in the embodiment of the present invention, by using the cosine similarity algorithm between vectors and the similarity matrix algorithm, on the one hand, the computational complexity of the similarity between strings can be reduced, and because the string vector has less dimensions , which can improve the problem of sparse distribution of extracted feature information; on the other hand, string vectors are more symbolic and transparent, and can more effectively represent text feature information, which is convenient to improve the accuracy of text classification.

As shown in Figure 2, the specific steps of KNN-based text classification are as follows:

Step 20: Encode the text into a string vector and input it into the KNN model;

Step 21: utilize the KNN model to calculate the similarity between the character string vector and the sample character string vector in the KNN model;

Step 22: Sort the corresponding sample string vectors according to the increasing relationship with the similarity of the string vectors;

Step 23: According to the sorting result, select K sample character string vectors in the sample character string vectors according to the sequence of the similarity of character string vectors from high to low, wherein, the K is a positive integer;

Step 24: Calculate the sampling probability of the K sample string vectors;

Step 25: select a sample string vector according to the sampling probability;

Step 26: Use the classification label corresponding to the selected sample character string vector as the classification label of the character string vector.

Embodiment 2

Based on the same inventive concept, an embodiment of the present invention also provides a text classification device based on K nearest neighbors KNN. As shown in FIG. 3 , the device includes: a processor 30 and a memory 31, wherein the memory stores program codes, The program code, when executed by the processor, causes the processor 30 to:

Decomposing the text into words, and extracting words representing the feature information of the text from the words;

Using the extracted words, encoding the text into a string vector;

Calculate the similarity between the string vector and the sample string vector in the KNN model by using the KNN model, and determine the similarity of the string vector according to the similarity and the classification label corresponding to the sample string vector. Classify labels and output.

As an optional implementation manner, the processor 30 is specifically configured to:

Using the KNN model, the cosine similarity algorithm between vectors is used to calculate the similarity between the character string vector and the sample character string vector in the KNN model; or

Using the KNN model, a similarity matrix algorithm for obtaining the similarity between string vectors by calculating the similarity between words is used to calculate the similarity between the string vector and the sample string vector in the KNN model.

As an optional implementation manner, the processor 30 is further configured to:

Obtain training samples including multiple string vectors, and the classification labels corresponding to the string vectors in the training samples;

Initialize the model parameters of the KNN model, and input the training sample into the KNN model;

Utilize the KNN model, adopt the cosine similarity algorithm between vectors, calculate the similarity between the character string vector and the sample character string vector in the KNN model;

According to the classification label corresponding to the string vector in the output training sample and the classification label corresponding to the character string vector in the training, the model parameters of the current KNN model are adjusted until the preset conditions are met.

As an optional implementation manner, the processor 30 is specifically configured to:

decompose the text into long strings based on the words in the corpus in the server;

The long character string is divided into paragraphs by a text segmentation method, the characters belonging to the stem in each paragraph are extracted, and the characters with no extraction meaning are deleted.

As an optional implementation manner, the processor 30 is specifically configured to:

According to the occurrence frequency, grammatical attribute and position distribution of the words in the words, words representing the feature information of the text are respectively extracted from the words.

As an optional implementation manner, the processor 30 is specifically configured to:

Extract at least one word according to the order of occurrence frequency of the words from high to low;

According to the grammatical attribute of the word, according to the order of the word frequency-inverse text frequency index TF-IDF weighted value in the word from high to low, extract at least one word;

According to the position distribution of the words, the words distributed in the set paragraph are extracted.

As an optional implementation manner, the set paragraph is the first paragraph and/or the last paragraph.

As an optional implementation manner, the processor 30 is specifically configured to:

Extract the last word distributed in the last paragraph, the first word of the first paragraph, the last word of the first paragraph, and the first word of the last paragraph.

As an optional implementation manner, the processor 30 is specifically configured to:

Selecting a preset number of sample string vectors corresponding to the string vector similarity from high to low;

calculating the sampling probability of the preset number of sample string vectors;

The classification label of the character string vector is determined according to the classification label corresponding to the sample character string vector selected by the sampling probability.

Embodiment 3

The present invention provides another text classification device based on K nearest neighbors KNN. As shown in FIG. 4, the device includes: a decomposition module 40, an encoding module 41 and a classification module 42, wherein:

The decomposition module 40 is used for decomposing the text into words, and extracting the words representing the feature information of the text from the words;

an encoding module 41, for using the extracted word to encode the text into a string vector;

The classification module 42 is used to calculate the similarity between the character string vector and the sample character string vector in the KNN model by using the KNN model, and determine the similarity according to the similarity and the classification label corresponding to the sample character string vector. The categorical label of the string vector and output.

As an optional implementation manner, the classification module 42 is specifically used for:

Using the KNN model, the cosine similarity algorithm between vectors is used to calculate the similarity between the character string vector and the sample character string vector in the KNN model; or

Using the KNN model, a similarity matrix algorithm for obtaining the similarity between string vectors by calculating the similarity between words is used to calculate the similarity between the string vector and the sample string vector in the KNN model.

As an optional implementation manner, the device is also used for:

Obtain training samples including multiple string vectors, and the classification labels corresponding to the string vectors in the training samples;

Initialize the model parameters of the KNN model, and input the training sample into the KNN model;

Utilize the KNN model, adopt the cosine similarity algorithm between vectors, calculate the similarity between the character string vector and the sample character string vector in the KNN model;

According to the classification label corresponding to the string vector in the output training sample and the classification label corresponding to the character string vector in the training, the model parameters of the current KNN model are adjusted until the preset conditions are met.

As an optional implementation manner, the decomposition module 40 is specifically used for:

decompose the text into long strings based on the words in the corpus in the server;

The long character string is divided into paragraphs by a text segmentation method, the characters belonging to the stem in each paragraph are extracted, and the characters with no extraction meaning are deleted.

As an optional implementation manner, the decomposition module 40 is specifically used for:

According to the occurrence frequency, grammatical attribute and position distribution of the words in the words, words representing the feature information of the text are respectively extracted from the words.

As an optional implementation manner, the decomposition module 40 is specifically used for:

Extract at least one word according to the order of occurrence frequency of the words from high to low;

According to the grammatical attribute of the word, according to the order of the word frequency-inverse text frequency index TF-IDF weighted value in the word from high to low, extract at least one word;

According to the position distribution of the words, the words distributed in the set paragraph are extracted.

As an optional implementation manner, the set paragraph is the first paragraph and/or the last paragraph.

As an optional implementation manner, the decomposition module 40 is specifically used for:

Extract the last word distributed in the last paragraph, the first word of the first paragraph, the last word of the first paragraph, and the first word of the last paragraph.

As an optional implementation manner, the classification module 42 is specifically used for:

Selecting a preset number of sample string vectors corresponding to the string vector similarity from high to low;

calculating the sampling probability of the preset number of sample string vectors;

The classification label of the character string vector is determined according to the classification label corresponding to the sample character string vector selected by the sampling probability.

Embodiment 4

The present invention provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the following steps are implemented:

Decomposing the text into words, and extracting words representing the feature information of the text from the words;

Using the extracted words, encoding the text into a string vector;

Calculate the similarity between the string vector and the sample string vector in the KNN model by using the KNN model, and determine the similarity of the string vector according to the similarity and the classification label corresponding to the sample string vector. Classify labels and output.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein, including but not limited to disk storage, optical storage, and the like.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce A device that implements the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory result in an article of manufacture comprising the instruction apparatus, the instructions The device implements the functions specified in the flow or flows of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention. Thus, provided that these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims (11)

1. a kind of file classification method based on K arest neighbors KNN, which is characterized in that this method comprises:

Text is decomposed into word, the word for indicating the characteristic information of text is extracted from the word；

It is character string vector by the text code using the word of the extraction；

The similarity between the sample character string vector in the character string vector and KNN model is calculated using the KNN model, According to the similarity and the corresponding tag along sort of the sample character string vector, the contingency table of the character string vector is determined It signs and exports.

2. the method according to claim 1, wherein using the KNN model calculate the character string vector with The similarity between sample character string vector in KNN model, comprising:

The character string vector and KNN model are calculated using the cosine similarity algorithm between vector using the KNN model In sample character string vector between similarity；Or

Using the KNN model, using the similar matrix for obtaining similarity between character string vector by similarity between calculating word Algorithm calculates the similarity between the sample character string vector in the character string vector and KNN model.

3. method according to claim 1 or 2, which is characterized in that further include:

Obtaining includes the corresponding contingency table of character string vector in multiple character string vector training samples and the training sample Label；

The training sample is inputted the KNN model by the model parameter for initializing KNN model；

The character string vector and KNN model are calculated using the cosine similarity algorithm between vector using the KNN model In sample character string vector between similarity；

According to corresponding point of character string vector in exporting the corresponding tag along sort of character string vector in the training sample and training Class label adjusts the model parameter of current KNN model, until meeting preset condition.

4. the method according to claim 1, wherein text is decomposed into word, comprising:

Based on the word in corpus in server, the text is decomposed into long character string；

The long character string is marked off into paragraph by text segmenting method, extracts the character for belonging to stem in each paragraph, and delete Except without the character for extracting meaning.

5. the method according to claim 1, wherein extracting the characteristic information for indicating text from the word Word, comprising:

According to the frequency of occurrences of word, grammatical attribute and position distribution in the word, table is extracted from the word respectively Show the word of the characteristic information of text.

6. according to the method described in claim 5, it is characterized in that, according to the frequency of occurrences of word, grammer category in the word Property and position distribution, respectively from the word extract indicate text characteristic information word, comprising:

According to the sequence of the word frequency of occurrences from high to low, at least one word is extracted；

According to the grammatical attribute of the word, according to word frequency in the word-inverse document frequency TF-IDF weighted value by height To low sequence, at least one word is extracted；

According to the position distribution of the word, the word for being distributed in setting paragraph is extracted.

7. according to the method described in claim 6, it is characterized in that, the paragraph that sets is the first paragraph and/or last paragraph.

8. the method according to the description of claim 7 is characterized in that extraction, which is distributed in, to be set according to the position distribution of the word Determine the word of paragraph, comprising:

Extract the last one for being distributed in the last one word of last paragraph, the first word of the first paragraph, the first paragraph The first word of word and last paragraph.

9. the method according to claim 1, wherein according to the similarity and the sample character string vector Corresponding tag along sort determines the tag along sort of the character string vector, comprising:

It chooses and the sample character string vector of the character string vector similarity corresponding preset quantity from high to low；

Calculate the sampling probability of the sample character string vector of the preset quantity；

According to the corresponding tag along sort of sample character string vector that the sampling probability is chosen, point of the character string vector is determined Class label.

10. a kind of text classification equipment based on K arest neighbors KNN, which is characterized in that the equipment includes: processor and storage Device, wherein the memory is stored with program code, when said program code is executed by the processor, so that the place Manage the step of device perform claim requires 1~9 any the method.

11. a kind of computer storage medium, is stored thereon with computer program, which is characterized in that the program is executed by processor The step of Shi Shixian such as claim 1~9 any the method.