CN103744928B - A kind of network video classification method based on history access record - Google Patents

Abstract

The invention relates to a network video classification method based on historical access records, belonging to the technical field of computer network data mining. First, by automatically analyzing the historical access record data set of the video, extracting meaningful features and generating a data file for use, through the data file, the historical access record is converted into a structured document that can be used for training, and then used Logistic regression performs machine learning on structured documents to obtain a predictive model. Using the prediction model, according to the completeness of the historical access record information of the video to be predicted, select the corresponding method to classify and predict it. Compared with the prior art, the present invention reduces the labor costs and at the same time makes the parameters involved in the calculation more streamlined, the prediction effect is more accurate, and the time spent is less. At the same time, because the operation of clustering or not can be selected according to the completeness of the historical access record information of the video to be predicted, the application of the model is more extensive.

Description

A Network Video Classification Method Based on Historical Access Records

technical field

The invention relates to a network video classification method, which belongs to the technical field of computer network data mining.

Background technique

With the rapid development of database technology, the wide application of database management systems and the rapid popularization of the Internet, the amount of historical access records of video (hereinafter referred to as video) on the Internet has increased dramatically. Behind the surge of data lies a large number of "treasures", that is, previously unknown and potentially useful information. In the face of large-scale massive data, data mining technology emerges as the times require, and extracts hidden but unknown but hidden information from a large number of incomplete, noisy, fuzzy, random, and practically applied data. And useful information and knowledge of the process.

The tasks of data mining mainly include classification, prediction, association analysis, time series pattern, clustering, deviation detection and so on. Each problem has many specific data mining or statistical models to solve it.

Among them, classification is a technique of constructing a classifier according to the characteristics of the data set, and using the classifier to assign categories to samples of unknown categories. The process of constructing a classifier is generally divided into two steps: model training and classification using the model. In the model training phase, the characteristics of the training data set are analyzed to generate an accurate description or model of the corresponding data set for each category. In the model use stage, according to the data description information of the objects to be classified, the model is used to classify them.

Classification algorithms mainly include neural network methods, decision tree classification methods, statistical methods, etc. Among them, statistical methods mainly include regression and naive Bayesian classification algorithms. Regression classification includes general linear regression and Logist regression (or called logistic regression), both of which divide data into two categories. Ordinary Logist regression is classified by the ratio of the probability of event occurrence to the probability of non-occurrence of the event. For multi-classification problems, Logit regression, a natural extension of Logist regression, is used. At present, the most widely used prediction method is based on logistic regression: by analyzing and modeling the data set, the object to be classified is predicted by two categories. However, the knowledge (attributes) in the dataset are not equally important and there is redundancy, which is not conducive to making correct and concise decisions. A better data set has the following indicators: fewer numbers; fewer rules for attributes; fewer final normalization rules, etc. However, the existing predictive methods based on logistic regression have some limitations in the streamlining of data sets, such as only sorting the importance of attributes while ignoring the discrete distribution of values, not considering the correlation between attributes, etc. .

Contents of the invention

The purpose of the present invention is to propose a network video classification method based on historical access records in order to overcome the deficiencies in data set simplification in the current predictive method based on logistic regression.

Under the condition of keeping the classification and decision-making ability of the knowledge base unchanged, the method of the present invention optimizes the feature extraction process of the data set, deletes irrelevant or unimportant attributes, and avoids the overlap of information reflected between variables , so that the data set is the most streamlined and the labor cost is reduced. Since the parameters involved in the calculation are more streamlined, the prediction effect is more accurate and the time effect is improved. The method is simple and easy to implement, and is suitable for widely popular distributed computing applications at present.

The method of the present invention comprises the following steps:

Step 1. Build a predictive model

First, feature extraction is performed. By automatically analyzing the historical access record data set of the video, the most streamlined attribute features are extracted to generate a data file to be used, and the historical access record data set is converted into a structured document that can be used for training through the data file.

Then, model training is performed. A logistic regression method is used to perform machine learning on the structured document to obtain a predictive model.

Step 2. Use the prediction model to predict the popularity of the video

First, judge the information integrity of the video historical access records. If the video is a new video, that is, the historical access record information is incomplete, a clustering method is used to find the video with the highest similarity, and its historical access record information is set as the historical access record information of the new video. If the video is not a new video, that is, the historical access record information is complete, proceed directly to the following operations.

Then, feature extraction is performed on the historical access record information of the video to be predicted, and the prediction model is used to classify whether it is welcome or not.

Beneficial effect

The invention adopts a network video classification method based on historical access records to predict whether the video is welcome or not. By performing attribute reduction such as feature extraction on the video historical access record data set, a corresponding prediction model is established. The complete historical access record analysis method, while reducing labor costs, makes the parameters involved in the calculation more streamlined, the prediction effect is more accurate, and the time spent is less. At the same time, because the operation of clustering or not can be selected according to the completeness of the historical access record information of the video to be predicted, the application of the model is more extensive.

Description of drawings

Fig. 1 is the flowchart of the method of the present invention.

detailed description

The specific implementation manners of the present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments.

As shown in Figure 1, a kind of network video classification method based on historical access records comprises the following steps:

Step 1: Analyze the historical video access record data set, extract the most streamlined attribute features, and generate a data file to be used. The historical video access record is converted into a structured document to be trained through the data file to be used. The specific process is as follows:

First, for the video historical access record data set, use the value analysis method to remove data and attributes with abnormal values, including attributes with no change in value, missing or noisy data, and remove video records with playback times less than a certain threshold , to get the data set U.

Then, using the heuristic attribute reduction algorithm based on the mutual information gain rate, the training reduces the attribute set of the data set U. The reduction starts from the kernel, gradually selects Z(c, R, D) to reach the maximum attribute addition, and ends when the classification ability of the selected attribute subset is the same as that of the entire attribute set. Specific steps are as follows:

In the first step, the prediction system S is defined as a quadruple: S=(U,A,V,f), where U={u ₁ ,u ₂ ,…,u _n } is the set of video objects, that is, the domain of discourse ;A is the attribute set of the video; is a collection of attribute values, V _a is the value range of attribute a; f is the mapping of U×A→V _a , which specifies a unique value for the attribute of each video object in U.

For the prediction system S, the attribute set A is divided into a conditional attribute set C and a decision attribute set D, A=C∪D, and C∩D=φ, where the elements contained in the attribute set C include video IDc ₁ , title c ₂ , type c ₃ , duration level c ₄ , URLc ₅ , URL reputation c ₆ , play count c ₁₀ , comment count c ₁₂ , share count c ₁₅ , favorite count c ₁₆ , download count c ₁₇ , share rate c ₁₈ , favorite rate c ₁₉ , download rate c ₂₀ , like rate c ₂₁ , growth rate of playback times c ₂₂ , praise rate c ₂₃ , time stamp c ₂₄ , watched duration c ₂₅ , proportion of watched duration c ₂₆ ; decision attribute set D includes popular or not d. The prediction system S with the above changes is named decision system L. Since in S, for the attribute set The corresponding binary equivalence relation can be constructed, when I _G is called an indistinguishable relation constructed by G. Then for the decision system L=(U,C∪D,V,f), set The divisions derived from I _R and ID are respectively X={X ₁ ,X ₂ ,…X _n } and Y={Y ₁ ,Y ₂ ,…Y _n }, then the entropy of _R is defined as where p(X _i )=card(X _i )/card(U). The conditional entropy of R with respect to D is defined as where p(Y _j /X _i )=card(Y _j ∩X _i )/card(X _i ). The mutual information of the decision attribute set D and the conditional attribute subset R is defined as: W(R; D) = H(D)-H(D/R), and the measurement method of attribute importance is defined as: Z(c,R, D)=(W(R∪{c};D)-W(R;D))/H(c), where p _i is the ratio of the number of objects whose attribute values are x _i to the total number of objects N. Let the attribute c have m types of values x ₁ , x ₂ . . . , x _m , and N is the total number of objects.

The second step is to calculate the mutual information W(C; D)=H(D)-H(D/C) of the conditional attribute set C and the decision attribute set D;

In the third step, calculate the core R=CORE _D (C), and calculate W(R; D). The calculation process of the kernel is:

1. Let CORE _D (C) = φ;

2. For all attributes r in conditional attribute set C, if H({d}/C)<H({d}/C-{r}), then

CORE _D (C)＝CORE _D (C)∪{r}.

3. End.

The fourth step, let C _candidate = CR, calculate the attributes in C _candidate according to Z(c,R,D)=(W(R∪{c};D)-W(R;D))/H(c) The importance of , and choose Z(c,R,D) to achieve the largest attribute c _i ;

The fifth step, let R=R∪{c _i }, if W(C; D)=W(R; D), then terminate, and denote the data set corresponding to the reduced attribute set by U′; Otherwise, go to step 4 to continue.

Afterwards, principal component analysis is performed on the data set U' to obtain several principal components that are not correlated with each other. Specific steps are as follows:

The first step is to perform Z standardization on the data set U′ to obtain the data set U″;

The second step is to conduct principal component analysis on the data set U″ to obtain the eigenvalues, variance contribution rates and cumulative variance contribution rates of each principal component, where the eigenvalues of each principal component are sorted from large to small The number k of principal components is determined according to the number of principal components whose cumulative variance contribution rate is greater than 85%, and the relationship between the k principal components and each attribute in the data set U″ is written according to the factor loading table obtained during principal component analysis. The relationship between is as follows, where Z _k represents the k-th principal component, β _km represents the m-th factor loading of Z _k , c _m is the value of the m-th attribute in the data set U″, c _m ∈ {video IDc ₁ , title c ₂ , type c ₃ , duration level c ₄ , URLc ₅ , URL reputation c ₆ , play count c ₁₀ , comment count c ₁₂ , like rate c ₂₁ , share rate c ₁₈ , Collection rate c ₁₉ , growth rate of playback times c ₂₂ , ratio of watched time to c ₂₆ }:

Step 2, using a logistic regression method to perform machine learning on the structured document to obtain a predictive model. The specific process is as follows:

Perform binary logistic regression analysis on the principal component values obtained in step 2 to obtain the logistic regression model:

Among them, α ₁ , α ₂ ,…, α _k are the parameters obtained after the prediction model is trained. The closer the value of P is to 1, the more popular the video to be classified is, and the closer P is to 0, the less the video to be classified is. Popular, if p≥0.5, the video to be classified is a popular video; if p<0.5, the video to be classified is an unpopular video;

Step 3. Use the prediction model to test whether the video is welcome or not. The specific process is as follows:

First, judge the information integrity of the video historical access records. If the video to be predicted is a new video, that is, the historical access record of the video does not exist, but some characteristic information of itself does exist, such as video ID, query ID, video title, description, keywords, etc., according to the video The feature information calculates the tf-idf value, and uses the tf-idf matrix as the input of the clustering model. Using tf-idf can cluster the content of the text to get the most similar video of the new video, and set its historical access record information as the historical access record information of the new video; if the video to be predicted is not a new video, directly perform Next step.

Then, corresponding conversion is performed on the historical access record data of the video to be predicted, that is, feature extraction is performed.

Finally, a predictive model is used to classify it as welcome or not.

Example

The method of the present invention includes three stages, the first stage is the feature extraction stage of the historical access records of the video, the second stage is the training stage of the prediction model, and the third stage is the prediction stage of whether the video to be classified is welcome or not.

Referring to Fig. 1, the specific process of the first stage of the present embodiment is described in detail below:

Step 1: According to the data size of historical access records of videos, remove video access records whose playback times are less than a certain threshold. Specifically, according to the analysis of some data sets, these historical access records are subject to the long-tail effect to a certain extent, that is, they contain many video records with insufficient clicks, so the first step of processing should be set as the threshold to remove video records with clicks below this threshold. Then remove some attribute columns with no change in value, so as to obtain the preliminary input data set U;

Step 2: Reduce the attribute set of the data set U. The reduction starts from the core, and gradually selects important attributes to add until the classification ability of the selected attribute subset is the same as that of the entire attribute set U. Specifically, after the initial screening in step 1, the initially obtained set of condition attributes in the input data set = {video IDc ₁ , title c ₂ , type c ₃ , duration level c ₄ , URLc ₅ , URL reputation c ₆ , video upload User ID c ₇ , uploader fan level c ₈ , upload time c ₉ , playback times c ₁₀ , number of viewers from different IP addresses c ₁₁ , number of comments c ₁₂ , number of good reviews c ₁₃ , video quality c ₁₄ , number of shares c ₁₅ , favorites c ₁₆ , downloads c ₁₇ , share rate c ₁₈ , collection rate c ₁₉ , download rate c ₂₀ , like rate c ₂₁ , growth rate of play times c ₂₂ , praise rate c ₂₃ , time stamp c ₂₄ , watched Duration c ₂₅ , ratio c ₂₆ } of the watched duration. First calculate the mutual information W(C,D)=0.283 of the conditional attribute C and the decision attribute D, and the relative core attribute K _D (C)={video ID, URL reputation, uploader fan level, number of playbacks, number of comments} , and then calculate the importance of the remaining attributes respectively as

Z(c ₂₁ ,R,D)=(W(R∪{c ₂₁ };D)-W(R;D))/H(c ₂₁ )=0.2182,

Z(c ₉ ,R,D)=(W(R∪{c ₉ };D)-W(R;D))/H(c ₉ )=0.2180,

Z(c ₄ ,R,D)=(W(R∪{c ₄ };D)-W(R;D))/H(c ₄ )=0.2160,

...,

Z(c ₁₄ ,R,D)＝(W(R∪{c ₁₄ };D)-W(R;D))/H(c ₁₄ )＝0.0110, according to the order of importance, add attributes to the condition Attribute set-get C'={video IDc ₁ , title c ₂ , type c ₃ , duration level c ₄ , URLc ₅ , URL reputation c ₆ , video uploader IDc ₇ , uploader fan level c ₈ , upload time c _9. The number of plays c ₁₀ , the number of comments c ₁₂ , the number of shares c ₁₅ , the number of favorites c ₁₆ , the share rate c ₁₈ , the rate of favorites c ₁₉ , the rate of likes c ₂₁ , the growth rate of the number of plays c ₂₂ , and the percentage of viewing time ratio c ₂₆ };

Step 3: Perform principal component analysis on the conditional attribute set C′ to obtain several principal components that are not correlated with each other. Specific steps are as follows:

i) Perform Z standardization on the data set U' corresponding to the condition attribute set C' to obtain the data set U";

ii) Conduct principal component analysis on the data set U″, and find out the eigenvalues of each principal component (sorted from large to small), variance contribution rate and cumulative variance contribution rate, and the cumulative variance contribution rate of the principal components is greater than 85% The number k of principal components is determined by the number of principal components. According to the factor loading table obtained during principal component analysis, the relationship between k principal components and each attribute in the conditional attribute set C′ is written, such as:

The above steps 1-3 are the specific process of the feature extraction stage of the first stage of this embodiment, and the obtained structured document is used as input for subsequent model training.

After the first stage, enter the second stage, that is, the model training stage. In this stage, logistic regression is used to perform machine learning on the structured documents obtained in the first stage to obtain a predictive model.

Among many machine learning algorithms, logistic regression is an efficient and ideal algorithm. Logistic regression will make full use of all the features to train the prediction model, such as the resulting logistic regression model:

The third stage is the prediction stage of whether the video is popular or not, including the following stages:

Step 1: Judging the integrity of the description information of the video to be predicted;

Step 2: If the video to be predicted is not a new video, that is, there is a certain amount of historical access record data, the feature value of the data is extracted, converted into a structured document form, and then substituted into the prediction model to predict whether it is welcome or not;

Step 3: If the video to be predicted is a new video, use clustering to find the video with the highest similarity, and set the new description information as the description information of the video to be predicted, and then perform corresponding prediction operations on it;

Specifically, the problem of how to predict whether a new video is welcome or not is transformed into finding the most similar set to this video, that is, into a clustering problem.

The present invention calculates the tf-idf value for the conditional attribute of the video to be predicted, uses the tf-idf matrix as the input of the clustering model, uses the tf-idf to perform clustering from the content of the data set, and the similarity calculated by this method more accurate.

Thus, in this embodiment, through three steps of processing, the prediction of whether the new video is welcome or not is obtained, so that the video can be predicted more accurately and placed more accurately.

The specific examples described above are further explanations of the present invention, and are not used to limit the protection scope of the present invention. All changes and equivalent replacements made within the principles and spirit of the present invention should be within the protection scope of the present invention Inside.

Claims (1)

1. a kind of network video classification method based on historical access record, it is characterized in that, may further comprise the steps: Step 1. Analyzing the historical video access record data set, extracting attribute features and generating a data file to be used, and converting the historical video access record into a structured document to be trained through the data file to be used; the specific process is as follows: First, for the video historical access record data set, use the value analysis method to remove data and attributes with abnormal values, including attributes with no change in value, missing or noisy data, and remove video records with playback times less than a certain threshold , get the data set U; Then, use the heuristic attribute reduction algorithm based on the mutual information gain rate to train and reduce the attribute set of the data set U; the reduction starts from the core, and gradually selects Z(c,R,D) to reach the maximum attribute to join, Until the classification ability of the selected attribute subset is the same as that of the entire attribute set, the specific steps are as follows: In the first step, the prediction system S is defined as a quadruple: S=(U,A,V,f), where U={u ₁ ,u ₂ ,…,u _n } is the set of video objects, that is, the domain of discourse ;A is the attribute set of the video; is the set of attribute values, V _a is the value range of attribute a; f is the mapping of U×A→V _a , it specifies a unique value for the attribute of each video object in U; For the prediction system S, attribute set A is divided into conditional attribute set C and decision attribute set D, A=C∪D, and C∩D=φ, where the elements contained in attribute set C include video ID c ₁ , title c _2. Type c ₃ , duration level c ₄ , URL c ₅ , URL reputation c ₆ , play count c ₁₀ , comment count c ₁₂ , share count c ₁₅ , favorite count c ₁₆ , download count c ₁₇ , share rate c ₁₈ , collection rate c ₁₉ , download rate c ₂₀ , like rate c ₂₁ , growth rate of playback times c ₂₂ , praise rate c ₂₃ , time stamp c ₂₄ , watched duration c ₂₅ , proportion of watched duration c ₂₆ ; decision-making The attribute set D includes whether it is popular or not d; the prediction system S that has made the above changes is named decision system L; because in S, for the attribute set Construct the corresponding binary equivalence relation, when I _G ={(x,y)∈U×U; There is a(x)=a(y)}, I _G is said to be an indistinguishable relationship constructed by G, then for the decision system L=(U,C∪D,V,f), let The divisions derived from I _R and ID are respectively X={X ₁ ,X ₂ ,…X _n } and Y={Y ₁ ,Y ₂ ,…Y _n }, then the entropy of _R is defined as where p(X _i )=card(X _i )/card(U); the conditional entropy of R relative to D is defined as Where p(Y _j /X _i )=card(Y _j ∩X _i )/card(X _i ); the mutual information of decision attribute set D and conditional attribute subset R is defined as: W(R; D)=H( D)-H(D/R), the measurement method of attribute importance is defined as: Z(c,R,D)=(W(R∪{c};D)-W(R;D))/H( c), where p _i is the ratio of the number of objects whose attribute value is x _i to the total number of objects N, let the attribute c have m kinds of values x ₁ , x ₂ . . . , x _m , and N is the total number of objects; The second step is to calculate the mutual information W(C; D)=H(D)-H(D/C) of the conditional attribute set C and the decision attribute set D; The third step is to calculate the core R=CORE _D (C), and calculate W(R; D), where the calculation process of the core is: Let CORE _D (C)=φ, for all attributes r in the conditional attribute set C, if H({d}/C)<H({d}/C-{r}), then CORE _D (C)= CORE _D (C)∪{r}; The fourth step, let C _candidate = CR, calculate the attributes in C _candidate according to Z(c,R,D)=(W(R∪{c};D)-W(R;D))/H(c) The importance of , and choose Z(c,R,D) to achieve the largest attribute c _i ; The fifth step, let R=R∪{c _i }, if W(C; D)=W(R; D), then terminate, and denote the data set corresponding to the reduced attribute set by U′; Otherwise, go to the fourth step to continue; Afterwards, principal component analysis is performed on the data set U′ to obtain several principal components that are not related to each other. The specific steps are as follows: The first step is to perform Z standardization on the data set U′ to obtain the data set U″; The second step is to conduct principal component analysis on the data set U″ to obtain the eigenvalues, variance contribution rates and cumulative variance contribution rates of each principal component, where the eigenvalues of each principal component are sorted from large to small ; Determine the number k of principal components according to the number of principal components whose cumulative variance contribution rate is greater than 85%, and write the relationship between the k principal components and each attribute in the data set U″ according to the factor loading table obtained during principal component analysis The relationship between is as follows, where Z _k represents the k-th principal component, β _km represents the m-th factor loading of Z _k , c _m is the value of the m-th attribute in the data set U″, c _m ∈ {video ID c ₁ , title c ₂ , type c ₃ , duration level c ₄ , URLc ₅ , URL reputation c ₆ , play count c ₁₀ , comment count c ₁₂ , like rate c ₂₁ , share rate c ₁₈ , collection rate c ₁₉ , growth rate of playback times c ₂₂ , ratio of watched time to c ₂₆ }: Step 2, using the logistic regression method to perform machine learning on the structured document to obtain a predictive model, the specific process is as follows: Perform binary logistic regression analysis on the principal component values obtained in step 2 to obtain the logistic regression model: Among them, α ₁ , α ₂ ,…, α _k are the parameters obtained after the prediction model is trained. The closer the value of P is to 1, the more popular the video to be classified is, and the closer P is to 0, the less the video to be classified is. Popular, if p≥0.5, the video to be classified is a popular video; if p<0.5, the video to be classified is an unpopular video; Step 3. Use the above prediction model to test whether the video is welcome or not. The specific process is as follows: First, judge the information integrity of the video historical access records. If the video to be predicted is a new video, that is, the historical access records of the video do not exist, calculate the tf-idf value according to the feature information of the video, and use the tf-idf matrix as the clustering model Input, get the most similar video of the new video, and set its historical access record information as the historical access record information of the new video; if the video to be predicted is not a new video, proceed to the next step directly; Then, the historical access record data of the video to be predicted is converted accordingly, that is, feature extraction is performed; Finally, use a predictive model to classify it as welcome or not.