CN111914110A - Example retrieval method based on deep activation salient region - Google Patents

Abstract

An instance retrieval method based on depth-activated salient regions, involving visual instance retrieval. 1) Model design: The model includes a forward propagation module, a pattern localization module and a feature extraction module; 2) For a given image database, each image in the database is used as the input of the model, and the output instance localization results and corresponding instances are extracted. 3) For each query picture, use it as the input of the forward propagation part of the deep pattern mining module, extract regional instance features, and use the instance features with all instance-level features extracted by the model on the database picture. The features are compared for similarity. The area with the highest similarity in each database image is the result of instance retrieval on the image, and the similarity corresponding to this area is the similarity of the image. All images in the database are based on similarity from high to low. Arrange in order to obtain the results of instance retrieval of the entire database. It can be applied to intelligent retrieval and video editing of video media.

Description

A Instance Retrieval Method Based on Deeply Activated Salient Regions

technical field

The invention relates to visual instance retrieval, in particular to an instance retrieval method based on depth activation salient regions, which can be applied to Internet companies, intelligent retrieval of video media, video editing and other fields.

Background technique

Visual instance retrieval (hereinafter referred to as "instance retrieval") is generally regarded as a subtask of image retrieval. The retrieved target is a visual instance, usually given in the query image in the form of box annotations. For example, to retrieve a specific person or a specific car. Instance retrieval requires finding the image containing the query instance in the image database and giving the location of the instance in the image. Instance retrieval can be widely used in intelligent commodity retrieval, video editing and other fields. It supports tasks related to a specific instance by retrieving and locating it. It is a common and basic technology in the field of current image data processing.

At present, the mainstream technologies in the industry cannot locate the instance while retrieving the instance. They inherit from the traditional image retrieval technology, that is, extract a global feature from the image for retrieval. Some methods directly extract global features on the basis of the global image; some methods first extract a large number of local features from the image, and then aggregate the local features to generate global features to reduce the direct comparison of local features in subsequent retrieval. a lot of computational cost to come. When global features are used for retrieval, the features of multiple instances in the entire image are mixed together, which makes it impossible to locate a single instance. At the same time, due to the mixing of features from other instances or backgrounds, the discriminative and expressive power of global features for each instance is seriously reduced. There are few technologies in the industry that use the framework of target detection or instance segmentation based on deep learning to solve the problem of visual instance localization. These techniques use supervised training methods to train target detection or instance segmentation frameworks. After obtaining object localization through detection or segmentation, instance-level features are obtained in the localization area for retrieval. Instance-level retrieval makes it possible to locate the retrieved instances, but supervised training inevitably makes the localization of objects only sensitive to the categories in the training dataset, and cannot locate and locate the instance objects that are not in the training category. retrieve.

In practical application scenarios, the instance classes of the query cannot be assumed in advance; in addition, it is unrealistic to collect all possible visual instance classes for training. Therefore, how to train on a data set with only a few thousand object categories and find instances of categories (unknown categories) that do not appear in the training category is a key technical problem to be solved by the present invention.

At present, there is no instance retrieval method in the industry that can both perform instance localization and be robust to instance categories.

SUMMARY OF THE INVENTION

The purpose of the present invention is to solve the problem of localization of unknown category instances by using deeply activated salient regions, to improve the robustness of instance object types in instance retrieval tasks, and to provide solutions that can be applied to intelligent commodity retrieval, video editing and other fields. An instance retrieval method based on deeply activated salient regions.

The present invention includes the following steps:

1) Model design: The model consists of three parts. The forward propagation module mines the depth pattern, which is the first part of the model; the pattern localization module is mainly responsible for activating salient regions and performing region shape estimation to locate instances, which is the second part of the model. Part; the feature extraction module extracts instance-level features through region pooling, which is the third part of the model. Take the picture as input, output the location information and corresponding features of the instance detected in the picture, and participate in the subsequent retrieval part;

2) Data preprocessing: For a given image database, each image in the database is used as the input of the model, and the output instance location results and corresponding instance-level features are extracted and saved for future use;

3) Instance retrieval: For each query image, take it as the input of the forward propagation part of the deep pattern mining module, use the given query instance region to extract regional instance features, and then combine the instance features with the model in the model. All instance-level features extracted from the database image are compared for similarity. The area with the highest similarity in each database image is the result of instance retrieval on the image, and the similarity corresponding to this area is the similarity of this image. All images in the database are arranged in descending order of similarity, and the result of instance retrieval of the entire database is obtained.

In step 1), the model design adopts the popular residual network (ResNet-50) in deep learning technology, and the fully convolutional structure before the fully connected layer of the network is used as the skeleton of the convolutional neural network, without additional training, directly Use pretrained residual network weights; specific steps in model design include:

来挖掘输入图片上激活的位置。平均响应图

的值反应了输入图片各处对深度激活模式的响应得分；

上的峰点处即为响应最高处；这些局部极值点所对映的输入图像的区域是最有可能存在实例的区域；First, the network accepts an input image, and through the above network structure, the output tensor of the last convolutional layer is obtained, denoted as X∈R ^W*H*C , where W and H are the width and height of the output tensor, respectively, C is the number of channels of the output tensor; the average response map is obtained by averaging over the channel dimension

to mine the activation locations on the input image. Average Response Plot

The value of reflects the response score of the input image to the deep activation pattern;

The peak point on the upper part is the highest response point; the region of the input image corresponding to these local extreme points is the region where the instance is most likely to exist;

In the pattern localization module, the corresponding C-dimensional vector is extracted at each peak point and backpropagated to activate the salient region, and the region corresponding to the peak point on the input image is obtained, and then the region is Perform shape estimation to complete positioning. Here, backpropagation can be approximately expressed as a process of solving conditional probability layer by layer. The specific operation of backpropagating a certain convolutional layer in the convolutional structure is given below, and the propagation of other layers is carried out in this way. With the input tensor, output tensor, convolution kernel of the convolutional layer, and the known result P(B) when backpropagating to the layer, the process of backpropagation is expressed as:

where the transit probability is:

上的某一个点，经迭代上述过程传播到输入层时得到的激活概率图记为。反映了输入图像上与该峰点有关的深度激活显著区域。将激活概率图的值正则化至[0,1]，对值大于0.1的所有像素点r(x,y)，通过计算图像二阶矩进行区域的形状估计。图像二阶矩的计算如下：Among them, is the regularization term, so that the sum of the transit probabilities to any point on A is 1. F _xi,yj is the value related to A _x,y and B _i,j on the convolution kernel. For the average response graph

At a certain point on , the activation probability map obtained when iteratively propagates the above process to the input layer is denoted as . It reflects the depth activation salient region related to this peak on the input image. The value of the activation probability map is normalized to [0, 1], and the shape of the region is estimated by calculating the second-order moment of the image for all pixel points r(x, y) whose value is greater than 0.1. The second moment of the image is calculated as follows:

The parameters of the approximate ellipse of the deep activation salient region are obtained through the above second-order moment calculation, and the circumscribing rectangle of the ellipse is the final positioning result.

上除峰点外的其他位置。在所有对深度激活模式响应大于

的均值的位置上，通过反向传播得到的定位结果和每个位置所对应的深度激活模式响应得分一起，进行一个非极大值抑制过程来筛选得到最终的定位结果。在每个定位结果区域上提取特征，即可得到实例级特征用以进行后续检索。The above positioning process is also applied to the average response map

other than the peak point. active mode responses at all depths greater than

At the position of the mean value of , the positioning result obtained by backpropagation and the response score of the deep activation mode corresponding to each position are combined with a non-maximum suppression process to filter to obtain the final positioning result. By extracting features on each location result area, instance-level features can be obtained for subsequent retrieval.

Compared with the prior art, the present invention has the following outstanding advantages:

The present invention proposes a simple and effective instance retrieval model, which utilizes deep activation of salient regions to improve the robustness to instance categories without the need for a training process, and solves the problem that the previous methods have both instance localization and instance category robustness. problems on the . It can be applied to Internet companies, intelligent retrieval of video media, video editing and other fields. It is an instance retrieval method based on deep activation of salient regions. In the model proposed by the present invention, on the one hand, the feature extraction part adopts the method of instance localization and instance-level feature extraction, which avoids the problem that global features will mix noise information, resulting in insufficient feature discrimination and inability to locate a single instance. On the other hand, the localization process adopts the method of deep activation of salient regions, avoiding the constraints of the training process and training categories in the industry method, and uses the response of the deep activation mode to locate possible foreground objects, thereby It includes the localization of instances of unknown categories, improves the robustness of instance retrieval to retrieved instance categories, avoids the time-consuming training process, and improves the portability and portability of the model.

Description of drawings

FIG. 1 is a schematic diagram of a model structure according to an embodiment of the present invention.

Detailed ways

The following embodiments will further illustrate the present invention in conjunction with the accompanying drawings.

The embodiment of the present invention includes the following steps:

1) Model design: The model of the present invention is shown in Figure 1. In Figure 1, there are three main modules, the forward propagation module 1, the pattern localization module 2 and the feature extraction module 3, which form a sequential structure in turn. The forward propagation module mines deep patterns, which is the first part of the model; the pattern localization module is mainly responsible for activating salient regions and performing region shape estimation to locate instances, which is the second part of the model; the feature extraction module extracts instance-level through region pooling features for the third part of the model. The model takes database images as input, and obtains all instance locations and corresponding instance-level features as database features. After that, regional features are extracted from the query instances, and the database features are queried to retrieve similar instances.

Specifically, the popular residual network (ResNet-50) in deep learning technology is adopted. The fully convolutional structure before the fully connected layer of the network serves as the skeleton of the convolutional neural network of the present invention. Without additional training, the pre-trained residual network weights are directly used.

来挖掘输入图片上激活的位置。该图的值反应了输入图片各处对深度激活模式的响应得分。该图的峰点处即为响应最高处。这些局部极值点所对映的输入图像的区域是最有可能存在实例的区域。First, the network accepts an input image, and through the above network structure, the output tensor of the last convolutional layer is obtained, denoted as X∈R ^W*H*C , where W and H are the width and height of the output tensor, respectively, C is the number of channels of the output tensor. Average response plot by averaging over the channel dimension

to mine the activation locations on the input image. The values in this map reflect the response scores to deep activation patterns across the input image. The peak of the graph is the highest response. The regions of the input image to which these local extreme points correspond are the regions where instances are most likely to exist.

In the pattern localization module, the corresponding C-dimensional vector is extracted at each peak point and backpropagated to activate the salient region, and the region corresponding to the peak point on the input image is obtained, and then the region is Perform shape estimation to complete positioning. Here, backpropagation can be approximately expressed as a process of solving conditional probability layer by layer.

The specific operation of backpropagating a certain convolutional layer in the convolutional structure is given below, and the propagation of other layers is carried out in this way. With the input tensor, output tensor, convolution kernel of the convolutional layer, and the known result P(B) when backpropagating to the layer, the process of backpropagation can be expressed as:

where the transit probability is:

上的某一个点，经迭代上述过程传播到输入层时得到的激活概率图记为。反映了输入图像上与该峰点有关的深度激活显著区域。将激活概率图的值正则化至[0,1]，对值大于0.1的所有像素点r(x,y)，通过计算图像二阶矩进行区域的形状估计。图像二阶矩的计算如下：Among them, is the regularization term, so that the sum of the transit probabilities to any point on A is 1. F _xi,yj is the value related to A _x,y and B _i,j on the convolution kernel. For the average response graph

At a certain point on , the activation probability map obtained when iteratively propagates the above process to the input layer is denoted as . It reflects the depth activation salient region related to this peak on the input image. The value of the activation probability map is normalized to [0, 1], and the shape of the region is estimated by calculating the second-order moment of the image for all pixel points r(x, y) whose value is greater than 0.1. The second moment of the image is calculated as follows:

The parameters of the approximate ellipse of the deep activation salient region are obtained through the above second-order moment calculation, and the circumscribing rectangle of the ellipse is the final positioning result.

上除峰点外的其他位置。在所有对深度激活模式响应大于

的均值的位置上，通过反向传播得到的定位结果和每个位置所对应的深度激活模式响应得分一起，进行一个非极大值抑制过程来筛选得到最终的定位结果。在每个定位结果区域上提取特征，即可得到实例级特征用以进行后续检索。In the present invention, the above positioning process is also applied to the average response map

other than the peak point. active mode responses at all depths greater than

At the position of the mean value of , the positioning result obtained by backpropagation and the response score of the deep activation mode corresponding to each position are combined with a non-maximum suppression process to filter to obtain the final positioning result. By extracting features on each location result area, instance-level features can be obtained for subsequent retrieval.

2) Data preprocessing: For a given image database, each image in the database is used as the input of the model of the present invention, and the output instance location results and corresponding instance-level features are extracted and saved for future use.

3) Instance retrieval: For each query image, it is used as the input of the forward propagation module, and the region of the given query instance is used to extract regional instance features. Then compare the similarity between the instance feature and all instance-level features extracted from the database image by the model of the present invention. The area with the highest similarity in each database image is the result of instance retrieval on the image. This area corresponds to The similarity is the similarity of the graph. All images in the database are arranged in descending order of similarity to obtain the results of instance retrieval on the entire database.

In the model proposed by the present invention, on the one hand, the feature extraction part adopts the method of instance localization and instance-level feature extraction, which avoids the problem that global features will mix noise information, resulting in insufficient feature discrimination and inability to locate a single instance. On the other hand, the localization process adopts the method of deep activation of salient regions, avoiding the constraints of the training process and training categories in the industry method, and uses the response of the deep activation mode to locate possible foreground objects, thereby It includes the localization of instances of unknown categories, improves the robustness of instance retrieval to retrieved instance categories, avoids the time-consuming training process, and improves the portability and portability of the model.

The present invention proposes a simple and effective instance retrieval method, which utilizes deep activation of salient regions to improve the robustness to instance categories without the need for a training process, and solves the problem that the previous method takes both instance localization and instance category robustness into consideration. existing problems.

In instance retrieval, the retrieval evaluation of the technology of the present invention and the prior art R-MAC, CroW, CAM, BLCF, BLCF-SalGAN, RegionalAttention, DeepVision, FCIS+XD and PCL*+SPN on Instance-335 and INSTRE datasets The comparison and comparison of the indicator mAP are shown in Table 1.

Table 1

Among all the contrasting methods, only the FCIS+XD and PCL*+SPN methods can localize all the retrieved instances, and they all use supervised training. On the Instance-335 dataset, the evaluation index adopts the setting of FCIS+XD, including the top 10, top 20, top 50, top 100 and all the results of the retrieval results evaluation comparison. It can be seen from Table 1 that the final mAP of the method of the present invention is higher than that of all localizable methods under the two datasets, and shows stable performance between the two datasets. The INSTRE data set contains many class instances that are not in the training class, and the stability of the method of the present invention compared with other localization methods on this data set proves the robustness to instance classes. Although BLCF-SalGAN performs better on the INSTRE dataset, it requires additional annotation of the generated saliency map information and its inability to locate instances makes it impractical in real scenes. Therefore, the model of the present invention obtains better results than all localization methods and guarantees the practicality of the actual scene.

The method corresponding to R-MAC is the method proposed by GiorgosTolias et al. (Tolias G, Sicre R, JégouH.Particular object retrieval with integral max-pooling of CNN activations[J].arXivpreprint arXiv:1511.05879,2015.); the method corresponding to CroW For the method proposed by Yannis Kalantidis et al. (Kalantidis Y, Mellina C, Osindero S. Cross-dimensional weighting for aggregated deep convolutional features[C]//European conference on computer vision. Springer, Cham, 2016:685-701.); corresponding to CAM The method is proposed by Albert Jimenez et al. (Jimenez A, Alvarez J M, Giro-i-Nieto X. Class-weighted convolutional features for visual instance search [J]. arXiv preprint arXiv:1707.02581, 2017.); BLCF and BLCF-SalGAN The corresponding methods are the bag-of-words model encoding convolutional feature method proposed by Eva Mohedano et al. .Saliency weighted convolutional features for instance search[C]//2018international conference on content-basedmultimedia indexing(CBMI).IEEE, 2018:1-6.); The corresponding method of Regional Attention is the method proposed by Jaeyoon Kim et al. (Kim J, Yoon S E.Regional Attention Based DeepFeature for Image Retrieval[C]//BMVC.2018:209.); the method corresponding to DeepVision is the method proposed by AmaiaSalvador et al. (Salva dor A, Giró-i-Nieto X, Marqués F, et al. Faster r-cnnfeatures for instance search[C]//Proceedings of the IEEE conference oncomputer vision and pattern recognition workshops.2016:9-16.); FCIS+ The method corresponding to XD is the method proposed by Zhan Yu et al. to extract instance-level features for instance retrieval using an instance segmentation framework (Zhan Y, Zhao W L. Instance Search via Instance Level Segmentation and FeatureRepresentation [J].arXiv preprint arXiv:1806.03576 , 2018.); the method corresponding to PCL*+SPN is the method proposed by Lin Jie et al. to extract features using a weakly supervised target detection framework for instance retrieval (Lin J, ZhanY, Zhao W L.Instance search based on weakly supervised feature learning[J].Neurocomputing, 2019.).

The invention discloses a visual instance retrieval method based on depth activation map. In this technology, the input image is calculated by the pre-training network to obtain the deep activation map. Local peaks in the graph can map to known classes in the pre-training data or unknown classes outside the pre-training data. These peak points are back-propagated layer by layer, which approximates the conditional probability, until the input image layer, and the potential visual object regions corresponding to these peak points on the input image can be obtained, thereby realizing the localization of visual instances. Based on localization, the present technology further extracts features in the region for feature representation in visual instance retrieval. The present invention is different from the existing technologies in that the present technology only relies on a pre-trained deep convolutional neural network; in addition, it can effectively detect the known visual object category and the unknown visual object category in the image, so that all the visual object categories in the image can be effectively detected. Latent objects generate instance-level feature representations for retrieval.

Claims (2)

1. an instance retrieval method based on depth activation significant region, is characterized in that comprising the following steps: 1) Model design: The model consists of three parts. The forward propagation module mines the depth pattern, which is the first part of the model; the pattern localization module is mainly responsible for activating salient regions and performing region shape estimation to locate instances, which is the second part of the model. Part; the feature extraction module extracts instance-level features through regional pooling, which is the third part of the model; takes the picture as input, outputs the location information and corresponding features of the instance detected in the picture, and participates in the subsequent retrieval part; 2) Data preprocessing: For a given image database, each image in the database is used as the input of the model, and the output instance location results and corresponding instance-level features are extracted and saved for future use; 3) Instance retrieval: For each query image, take it as the input of the forward propagation part of the deep pattern mining module, use the given query instance region to extract regional instance features, and then combine the instance features with the model in the model. All instance-level features extracted from the database image are compared for similarity. The area with the highest similarity in each database image is the result of instance retrieval on the image, and the similarity corresponding to this area is the similarity of this image. All images in the database are arranged in descending order of similarity, and the result of instance retrieval of the entire database is obtained. 2. a kind of instance retrieval method based on deep activation significant area as claimed in claim 1 is characterized in that in step 1), described model design adopts the popular residual network (ResNet-50) in deep learning technology, this The fully convolutional structure before the fully connected layer of the network is used as the skeleton of the convolutional neural network, and the pre-trained residual network weights are directly used without additional training; the specific steps of model design include: First, the network accepts an input image, and through the above network structure, the output tensor of the last convolutional layer is obtained, denoted as X∈R ^W*H*C , where W and H are the width and height of the output tensor, respectively, C is the number of channels of the output tensor; the average response map is obtained by averaging over the channel dimension

to mine the locations of activations on the input image; the average response map

The value of reflects the response score of the input image to the deep activation pattern;

The peak point on the upper part is the highest response point; the region of the input image corresponding to these local extreme points is the region where the instance is most likely to exist; In the pattern localization module, the corresponding C-dimensional vector is extracted at each peak point and backpropagated to activate the salient region, and the region corresponding to the peak point on the input image is obtained, and then the region is Perform shape estimation to complete positioning; here, backpropagation can be approximately expressed as a process of solving conditional probability layer by layer; the specific operation of backpropagating a convolutional layer in the convolutional structure is given below, and the propagation of other layers depends on This method is carried out; there are input tensors, output tensors, convolution kernels of the convolutional layer, and the known result P(B) when backpropagating to this layer, and the process of backpropagation is expressed as :

where the transit probability is:

Among them, is the regularization term, so that the sum of the transfer probabilities to any point on A is 1; F _xi,yj is the value related to A _x,y and B _i,j on the convolution kernel; For the average response graph

At a certain point on , the activation probability map obtained when it propagates to the input layer through the iterative process above is recorded as; it reflects the deep activation significant area related to the peak point on the input image; the value of the activation probability map is normalized to [0 ,1], for all pixel points r(x,y) whose value is greater than 0.1, the shape of the region is estimated by calculating the second-order moment of the image; the calculation of the second-order moment of the image is as follows:

The parameters of the approximate ellipse of the deep activation salient region are obtained through the above second-order moment calculation, and the circumscribed rectangle of the ellipse is the final positioning result; The above positioning process is also applied to the average response map

on all positions except the peak point; at all responses to deep activation modes greater than

At the position of the mean value of , the positioning results obtained through backpropagation and the corresponding deep activation mode response scores of each position are combined with a non-maximum suppression process to filter to obtain the final positioning results; in each positioning result area By extracting features from above, instance-level features can be obtained for subsequent retrieval.

Images