CN112232374B - Irrelevant label filtering method based on depth feature clustering and semantic measurement - Google Patents

Abstract

The invention discloses a method for filtering irrelevant tags based on deep feature clustering and semantic measurement, comprising the following steps: step 1, a sensor acquires an image set; step 2, establishing a tag set corresponding to the image set; step 3, extracting an image Collect the depth features of the image; step 4, cluster the depth features to obtain clusters; step 5, construct the relevant semantic label set of the cluster; step 6, construct the set of labels to be measured for the cluster; step 7, generate semantics vector; Step 8, calculating the relevance of the semantic vector; Step 9, filtering irrelevant tags according to the relevance. The present invention clusters huge sample image data to obtain cluster clusters for pre-classification of sample image data. By analyzing the clustered sample image data, it has higher validity and correctness, and at the same time labels Semantics is used to measure the relevance, so as to realize the automatic filtering of irrelevant labels, which can improve the generalization and robustness of the deep network.

Description

Irrelevant Label Filtering Method Based on Deep Feature Clustering and Semantic Measure

technical field

The invention belongs to the technical field of deep learning, and in particular relates to an irrelevant label filtering method based on deep feature clustering and semantic measurement.

Background technique

With the development of artificial intelligence technology, deep learning technology has been widely used and has become an indispensable part of people's work and life, especially in the fields of computer vision and artificial intelligence. Deep learning technology is a branch of machine learning. It is an algorithm that uses artificial neural networks as the framework to perform representation learning on data.

The convolutional neural network proposed by YannLecun et al. has been widely and successfully applied to various image fields such as detection, segmentation, and object recognition. These applications all use large amounts of labeled data. The prerequisite for deep learning technology to achieve good results is to have massive training data. The acquisition of massive training data requires a large number of people to label these training data. This process requires expensive manpower and material costs. Even if the pre-training model is obtained by using unlabeled data combined with unsupervised technology to train the network, there is still a correlation between the semantic distribution of the training data and the data to be predicted in order to obtain a model with relatively strong generalization ability.

The process of manual labeling is tedious and complicated. For different deep learning tasks, such as target detection, semantic segmentation, etc., due to the diversity of data sources, some sample information and labels in the sample data are irrelevant, and the keyword labels of sample information play an important role in the review, retrieval and organization of samples. The key role, so the irrelevance of annotations will easily lead to the fact that the annotation information cannot accurately reflect the characteristics of the sample data, and will lead to a longer time for the deep learning model to fit the parameters, and the efficiency is low. For the deep learning neural network with complex structure, the number of layers This is especially true for many deep learning neural networks. The problem of data mislabeling has always been a key research area of computer vision and artificial intelligence. Therefore, in order to improve the efficiency of deep learning models, it is necessary to study irrelevant label filtering technology for data sets.

The current existing technology cannot meet the requirements of filtering irrelevant labels in the data set, so there is an urgent need to filter irrelevant labels in the data set to facilitate subsequent deep learning tasks and improve the generalization and robustness of deep networks. A method for filtering irrelevant labels in datasets.

Contents of the invention

The technical problem to be solved by the present invention is to provide a method for filtering irrelevant labels based on deep feature clustering and semantic measurement, which has a simple structure and reasonable design, and can cluster huge sample image data. Obtain clusters for pre-classification of sample image data. By analyzing the clustered sample image data, it has higher validity and correctness. At the same time, it measures the relevance of label semantics, thus realizing different Automatic filtering of relevant tags can improve the generalization and robustness of deep networks.

In order to solve the above-mentioned technical problems, the technical solution adopted in the present invention is: an irrelevant label filtering method based on deep feature clustering and semantic measurement, characterized in that: comprising the following steps:

Step 1: The sensor acquires the image set X, and stores the image set X in the storage unit, X={x ₁ ,... _xi ,...x _n }, where x _i represents the i-th sample image data, 1≤i≤n, n is a positive integer;

Step 2: Establish a label set corresponding to the image set X in the storage unit;

Step 3, extract the depth features of the images in the image set: extract the depth features from the sample image data x _i in the image set X, and obtain the depth features φ(xi ₎ ;

Step 4. Acquire clusters by clustering the deep features: cluster the deep features φ( _xi ) with the preset number k as the number of clusters, and obtain the cluster set A, A={A ₁ ,... ,A _f ,...A _k }, where 1≤f≤k, k is a positive integer;

Step 5. Construct the relevant semantic label set of clusters: According to the original category label set U in step 2, the semantic labels of the cluster centers of each cluster A _f are obtained, and the semantic labels of the cluster centers of the cluster A _f are used The label is used as the relevant semantic label y _f , and the relevant semantic label set Y corresponding to the cluster set A is obtained, Y={y ₁ ,...,y _f ,...y _k };

t为正整数；Step 6. Construct the label set to be measured for the cluster: obtain the label set P to be measured, P={P ₁ ,...,P _f ,...P _k }, P _f represents the corresponding value of the cluster A _f For the set of labels to be measured, according to the original category label set U in step 2, other category labels except the cluster center under each cluster A _f are obtained, and the other category labels under the cluster A _f except the cluster center Add the label set P _f to be measured,

t is a positive integer;

Step 7. Generate semantic vectors: take the relevant semantic label set Y and the label set P to be measured as input, and obtain all the semantic vectors H _f of the relevant semantic label set Y and all the semantic vectors K _fg of P _f in the label set P to be measured ;

计算相关语义标签集合Y和第f个聚类簇的待度量标签集合P_f中每个标签的相关度Sim_fg，其中H_f表示相关语义标签集合Y中相关语义标签y_f的语义向量，K_fg表示待度量标签集合P中P_f中第g个标签的语义向量；Step 8. Calculate the relevance of the semantic vector: the computer according to the formula

Calculate the correlation Sim _fg of each tag in the set of related semantic tags Y and the tag set P _f to be measured in the fth cluster, where H _f represents the semantic vector of the related semantic tag y _f in the set Y of related semantic tags, K _fg represents the semantic vector of the gth label in P _f in the label set P to be measured;

Step 9: Filter irrelevant tags according to the correlation: delete the tags whose correlation Sim _fg is lower than the threshold η in the cluster A _f .

The above-mentioned unrelated label filtering method based on deep feature clustering and semantic measurement is characterized in that: in step 3, using the deep convolutional residual neural network model pre-trained in the large image data set Imagenet to image set X The sample image data _xi in extracts deep features, and the network model consists of convolutional layers, residual layers and fully connected layers.

The aforementioned method for filtering irrelevant labels based on deep feature clustering and semantic measurement is characterized in that: in step 4, the spectral clustering algorithm is used to cluster the deep feature φ(xi ₎ , and the specific steps include:

Step 401: Construct the similarity matrix W of the depth feature φ(xi ₎ , W is a similarity matrix composed of _sij ,

其中w_ij进行表示相似度矩阵W中第i行第j列的元素；Step 402: Calculate the diagonal matrix D, where

Among them, w _ij represents the element in row i and column j in the similarity matrix W;

Step 403: Obtain the Laplacian matrix L of the depth feature φ( _xi ) according to L=DW;

Step 404: Perform eigenvalue decomposition on the Laplacian matrix L, construct an eigenvector space, and cluster the eigenvectors in the eigenvector space through a clustering algorithm to obtain a cluster set A, A={A ₁ ,. ..,A _f ,...A _k }.

The above-mentioned irrelevant tag filtering method based on deep feature clustering and semantic measurement is characterized in that: in step 7, the semantic vector is generated using the Synonyms network model Synonyms.

The above-mentioned unrelated label filtering method based on deep feature clustering and semantic measurement is characterized in that: in step 9, the cosine distance between the semantic vector of the original category label set U and the semantic vector of the related semantic label set Y is used, and The difference between the cosine distance between the semantic vector of the mislabeled label set V and the semantic vector of the related semantic label set Y sets the correlation threshold η.

Compared with the prior art, the present invention has the following advantages:

1. The structure of the present invention is simple, the design is reasonable, and the realization, use and operation are convenient.

2. The present invention uses the deep convolutional residual neural network model pre-trained in the large-scale image data set Imagenet to carry out deep feature extraction, which combines the characteristics of strong learning ability of the deep convolutional neural network and good convergence of residual learning, feature extraction and Selection is more robust, protecting the integrity of image information and improving result performance.

3. The present invention clusters huge sample image data to obtain clusters for pre-classification of sample image data, reduces the time required for manual classification, and avoids different classification results caused by subjective differences. The clustered sample image data is analyzed, which can better screen the image set and have higher validity and correctness.

4. The present invention obtains the semantic vector of the relevant semantic label y _f in the related semantic label set Y and the semantic vector of the g-th label in the set of labels to be measured P _f respectively, and calculates the correlation between the g-th label in the set of labels to be measured P _f The average value of the cosine distance of each related semantic label in the semantic label set Y is used as the correlation degree between the gth label in the label set P _f to be measured and the image set X, so as to perform correlation screening. A correlation measure is carried out, which enables automatic filtering of irrelevant tags.

To sum up, the present invention has a simple structure and a reasonable design. It clusters huge sample image data to obtain clusters for pre-classification of sample image data. By analyzing the clustered sample image data, it has more High effectiveness and correctness, and at the same time measure the relevance of tag semantics, thereby realizing the automatic filtering of irrelevant tags, which can improve the generalization and robustness of deep networks.

The technical solutions of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

Description of drawings

Fig. 1 is a flow chart of the method of the present invention.

Detailed ways

The method of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments of the present invention.

It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and examples.

It should be noted that the terminology used here is only for describing specific implementations, and is not intended to limit the exemplary implementations according to the present application. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural, and it should also be understood that when the terms "comprising" and/or "comprising" are used in this specification, they mean There are features, steps, operations, means, components and/or combinations thereof.

It should be noted that the terms "first" and "second" in the description and claims of the present application and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein, for example, can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

For the convenience of description, spatially relative terms may be used here, such as "on ...", "over ...", "on the surface of ...", "above", etc., to describe the The spatial positional relationship between one device or feature shown and other devices or features. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, devices described as "above" or "above" other devices or configurations would then be oriented "beneath" or "above" the other devices or configurations. under other devices or configurations”. Thus, the exemplary term "above" can encompass both an orientation of "above" and "beneath". The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptions used herein interpreted accordingly.

As shown in Figure 1, the present invention comprises the following steps:

Step 1: The sensor acquires the image set X, and stores the image set X in the storage unit, X={x ₁ ,... _xi ,...x _n }, where x _i represents the i-th sample image data, 1≤i≤n, n is a positive integer.

In actual use, different types of sample image data are collected by the sensor, and the sample image data collected by the sensor is also different for different objects.

Step 2: Establish a label set corresponding to the image set X in the storage unit. During specific implementation, the data set includes an image set X and a label set, and the label set includes the original category label set U and the mislabeled label set V, U={u ₁ ,...,up _p ,...u _h }, V ={v ₁ ,...,v _q ,...v _l }, wherein, 1≤p≤h, 1≤q≤l, h and l are both positive integers, and l+h=n. Each sample image data _xi in the image set X corresponds to a label in the original category label set U. The mislabeled label set V is used to store the last screened out irrelevant labels.

Step 3: Obtain image depth features: the computer extracts depth features from the sample image data x _i in image set X to obtain depth features φ(xi ₎ .

During specific implementation, in order to improve the clustering effect, it is necessary to convert the sample image data _xi in the image set X into an appropriate feature representation. In the specific implementation, the computer uses the large-scale image data set Imagenet to pre-train the deep convolutional residual neural network, and extracts the depth feature φ( _xi ) of the sample image data x _i according to the collected sensor data, and integrates the deep convolutional neural network With strong network learning ability and good convergence of residual learning, feature extraction and selection are more robust, which solves the problem of lack of image feature details, protects the integrity of image information, and improves the performance of results.

Specifically: the deep convolutional neural network model uses the convolution kernel of each layer to perform convolution operations on the sample image data _xi , and extracts the initial features of each sample image data _xi , and uses the residual network model in the deep convolutional neural network Adding a skip connection can directly transfer the initial features of the sample image data _xi to the back layer of the deep convolutional neural network to improve the performance of the result, and then stitch the features into the fully connected layer of the model to obtain the depth of the sample image data _xi Feature φ( _xi ), depth feature φ(xi ₎ contains features related to sample image data _xi , so depth feature φ( _xi ) affects the final classification and filtering effect.

Step 4. Obtain clusters: cluster the depth features φ( _xi ) with the preset number k as the number of clusters, and obtain a cluster set A, A={A ₁ ,...,A _f ,. ..A _k }, where 1≤f≤k, k is a positive integer;

The spectral clustering algorithm is based on the similarity matrix, and transforms the ordinary clustering problem into a graph division problem. The spectral clustering algorithm is based on the spectral graph theory, and it is not limited by the shape of the sample space when clustering, so it is better than Traditional clustering algorithms. The spectral clustering algorithm starts from the whole world when solving, and has the advantage of converging on the global optimal solution without falling into the local optimal solution. At the same time, it can ensure the minimum similarity between different classes and the largest similarity within the same class. Its performance And applicable scenarios are better than traditional clustering algorithms. Therefore, the present application preferably uses a spectral clustering algorithm to cluster the depth features φ(xi ₎ .

σ表示标准差。相似度矩阵W表示为W＝(w_ij)_{i，j＝1,...n}，计算机根据公式

计算对角矩阵D，D＝{d₁,...d_i,...d_n}。根据L＝D-W获取深度特征φ(x_i)的拉普拉斯矩阵L，对拉普拉斯矩阵L进行特征值分解，构建特征向量空间，通过聚类算法对特征向量空间中的特征向量进行聚类，得到聚类簇集合A，A＝{A₁,...,A_f,...A_k}。The specific process of clustering using the spectral clustering algorithm is: constructing the similarity matrix W of the image set X, W is a similarity matrix composed of _sij ,

σ represents the standard deviation. The similarity matrix W is expressed as W=(w _ij ) _{i, j=1,...n} , and the computer according to the formula

Calculate the diagonal matrix D, D={d ₁ ,...d _i ,...d _n }. Obtain the Laplacian matrix L of the depth feature φ( _xi ) according to L=DW, perform eigenvalue decomposition on the Laplacian matrix L, construct the eigenvector space, and perform clustering algorithm on the eigenvectors in the eigenvector space Clustering to obtain a cluster set A, A={A ₁ ,...,A _f ,...A _k }.

This application clusters huge sample image data to obtain cluster clusters for pre-classification of sample image data, reduces the time required for manual classification, and avoids different classification results caused by subjective differences. Analyzing the generated sample image data can better screen the image set, have higher validity and correctness, and provide a more reliable method for filtering irrelevant labels.

Step 5. Construct the relevant semantic label set: According to the original category label set U in step 2, obtain the semantic label of the cluster center of each cluster A _f , and use the semantic label of the cluster center of the cluster A _f as the relevant semantic label y _f , to obtain a set of related semantic labels Y, Y={y ₁ ,...,y _f ,...y _k }.

Since each sample image data _xi in the image set X corresponds to a label in the original category label set U, the cluster center of the cluster A _f also corresponds to a semantic label in the original category label set U.

t为正整数。Step 6. Build a set of labels to be measured: obtain a set of labels to be measured P, P={P ₁ ,...,P _f ,...P _k }, P _f represents the set of labels to be measured corresponding to the cluster A _f , combine the labels corresponding to the cluster elements under the cluster A _f except the cluster center to form the label set P _f to be measured,

t is a positive integer.

Similarly, since each sample image data _xi in the image set X corresponds to a label in the original category label set U, the cluster elements under the cluster A _f except the cluster center also correspond to the original category label set A semantic label in U.

Step 7. Generate semantic vectors: take the relevant semantic label set Y and the label set P to be measured as input, and obtain all the semantic vectors H _f of the relevant semantic label set Y and all the semantic vectors K _fg of P _f in the label set P to be measured .

It should be noted that the present application preferably adopts the Synonyms network model. The Synonyms network model is a well-trained word2vec model. word2vec uses a large amount of data, uses contextual information for training, and maps vocabulary to low-dimensional space. At the algorithm level, retrieval is based on "distance" rather than "match". , based on "semantics" rather than "form". The synonym network model Synonyms, as a trained word2vec model, can map each word to a vector, which can be used to represent the relationship between words and words, and has the ability to measure the correlation between words and words.

Input words to the Synonyms network model for prediction, and the Synonyms network model Synonyms outputs hidden layer variables, and the parameters calculated according to the hidden layer variables are the semantic vectors corresponding to the words. In other words, the synonym network model Synonyms can output the mathematical expression of the word according to the input word, which is the semantic vector.

计算相关语义标签集合Y和第f个聚类簇的待度量标签集合P_f中每个标签的相关度Sim_fg，其中H_f表示相关语义标签集合Y中相关语义标签y_f的语义向量，K_fg表示待度量标签集合P中P_f中第g个标签的语义向量；Step 8. Calculate the relevance of the semantic vector: the computer according to the formula

Calculate the correlation Sim _fg of each tag in the set of related semantic tags Y and the tag set P _f to be measured in the fth cluster, where H _f represents the semantic vector of the related semantic tag y _f in the set Y of related semantic tags, K _fg represents the semantic vector of the gth label in P _f in the label set P to be measured;

Sim _fg represents the average value of the cosine distance between the gth tag in the tag set P _f to be measured and each related semantic tag in the related semantic tag set Y, so Sim _fg is used as the gth tag in the tag set P _f to be measured and The correlation degree of the image set X is used as an indicator of whether the gth label in the label set P _f to be measured should be filtered.

Step 9. Filter irrelevant labels according to the correlation: In the original category label set U, delete the labels whose correlation Sim _fg is lower than the threshold η in the cluster A _f ; in the image set X, delete the cluster cluster A f The sample image data _xi corresponding to the label whose correlation Sim _fg is lower than the threshold η in A _f is deleted, thereby obtaining a trainable data set, reducing the fitting time of the deep learning model parameters, and improving the fitting efficiency. The effect is good.

During specific implementation, the cosine distance between the semantic vector of the original category label set U and the semantic vector of the related semantic label set Y, and the cosine distance between the semantic vector of the mislabeled label set V and the semantic vector of the related semantic label set Y are used, Set the correlation threshold η.

The above is only an embodiment of the present invention, and does not limit the present invention in any way. Any simple modifications, changes and equivalent structural changes made to the above embodiments according to the technical essence of the present invention still belong to the technical solution of the present invention. within the scope of protection.

Claims (5)

1. The irrelevant label filtering method based on depth feature clustering and semantic measurement, is characterized in that: comprise the following steps: Step 1: The sensor acquires the image set X, and stores the image set X in the storage unit, X={x ₁ ,... _xi ,...x _n }, where x _i represents the i-th sample image data, 1≤i≤n, n is a positive integer; Step 2: Establish a label set corresponding to the image set X in the storage unit, the label set includes the original category label set U and the mislabeled label set V; Step 3, extract the depth features of the images in the image set: extract the depth features from the sample image data x _i in the image set X, and obtain the depth features φ(xi ₎ ; Step 4. Acquire clusters by clustering the deep features: cluster the deep features φ( _xi ) with the preset number k as the number of clusters, and obtain the cluster set A, A={A ₁ ,... ,A _f ,...A _k }, where 1≤f≤k, k is a positive integer; Step 5. Construct the relevant semantic label set of clusters: According to the original category label set U in step 2, the semantic labels of the cluster centers of each cluster A _f are obtained, and the semantic labels of the cluster centers of the cluster A _f are used The label is used as the relevant semantic label y _f , and the relevant semantic label set Y corresponding to the cluster set A is obtained, Y={y ₁ ,...,y _f ,...y _k }; Step 6. Construct the label set to be measured for the cluster: obtain the label set P to be measured, P={P ₁ ,...,P _f ,...P _k }, P _f represents the corresponding value of the cluster A _f For the set of labels to be measured, according to the original category label set U in step 2, other category labels except the cluster center under each cluster A _f are obtained, and the other category labels under the cluster A _f except the cluster center Add the label set P _f corresponding to the cluster A _f to be measured,

1≤g≤t, t is a positive integer; Step 7. Generate semantic vectors: take the relevant semantic label set Y and the label set P to be measured as input, and obtain all the semantic vectors H _f of the relevant semantic label set Y and all the semantic vectors K _fg of P _f in the label set P to be measured ; Step 8. Calculate the relevance of the semantic vector: the computer according to the formula

Calculate the correlation degree Sim _fg of each label in the related semantic label set Y and the f-th cluster to be measured label set P _f , where ||H _f || represents the relevant semantic label y _f in the related semantic label set Y Semantic vector, ||K _fg || represents the semantic vector of the gth label in P _f in the label set P to be measured; Step 9: Filter irrelevant tags according to the correlation: delete the tags whose correlation Sim _fg is lower than the threshold η in the cluster A _f . 2. according to claim 1, a kind of irrelevant label filtering method based on depth feature clustering and semantic measure, it is characterized in that: in step 3, utilize the deep convolution residual of pre-training in large-scale image data set Imagenet The neural network model extracts deep features from the sample image data x _i in the image set X, and the network model consists of a convolutional layer, a residual layer, and a fully connected layer. 3. according to a kind of irrelevant label filtering method based on depth feature clustering and semantic measure according to claim 1, it is characterized in that: in step 4, adopt spectral clustering algorithm to carry out clustering to depth feature φ (xi ₎ , the specific steps include: Step 401: Construct the similarity matrix W of the depth feature φ(xi ₎ , W is a similarity matrix composed of _sij ,

σ represents the standard deviation; Step 402: Calculate the diagonal matrix D, D={d ₁ ,...d _i ,...d _n }, where

Step 403: Obtain the Laplacian matrix L of the depth feature φ( _xi ) according to L=DW; Step 404: Perform eigenvalue decomposition on the Laplacian matrix L, construct an eigenvector space, and cluster the eigenvectors in the eigenvector space through a clustering algorithm to obtain a cluster set A, A={A ₁ ,. ..,A _f ,...A _k }. 4. A method for filtering irrelevant labels based on deep feature clustering and semantic measurement according to claim 1, characterized in that: in step 7, the semantic vector is generated using the synonym network model Synonyms. 5. according to claim 1, a kind of irrelevant label filtering method based on depth feature clustering and semantic measure is characterized in that: in step 9, utilize the semantic vector of original category label set U and the related semantic label set Y The cosine distance of the semantic vector, and the difference between the semantic vector of the mislabeled label set V and the cosine distance of the semantic vector of the related semantic label set Y, set the correlation threshold η.

Images