CN110245678B - An Image Matching Method Based on Heterogeneous Siamese Region Selection Network - Google Patents

Abstract

The invention discloses an image matching method based on a heterogeneous twin area selection network, which belongs to the field of computer vision. Including the heterogeneous twinning network and the region matching network in series; the heterogeneous twinning network is used to extract the feature map of the template map and the feature map of the map to be matched; the region matching network is used to extract the feature map of the template map and the feature map to be matched according to the feature map of the template map. Figure, get the regional matching results; the heterogeneous twin network includes sub-network A and sub-network B in parallel with each other, each sub-network is a feature extraction module, a feature fusion module and a maximum pooling module in series, and the modules of the two sub-networks are the same. , only the convolution kernel of the first layer of the feature extraction module is different. The invention applies the heterogeneous twin region selection network to image matching, realizes the input of non-fixed scale template and the image to be matched, fully utilizes the multi-layer features of the image, effectively improves the performance of the matching method, and increases the matching success rate and speed.

Description

An Image Matching Method Based on Heterogeneous Siamese Region Selection Network

technical field

The invention belongs to the field of computer vision, and more particularly, relates to an image matching method based on a heterogeneous twin area selection network.

Background technique

Image matching refers to the process of finding images or image areas (sub-images) that are similar to a given scene area image in an image (or a batch of images). Usually, the known scene area image is called a template image, and the sub-images in the image to be searched that may correspond to it are called the scene area image to be matched of the template. Image matching is to find correspondence between two or more images of the same scene from different times or different perspectives. Specific applications include target or scene recognition, 3D structure solving in multiple images, stereo correspondence, and motion tracking.

At present, most image matching algorithms only use shallow artificial features, such as grayscale features and gradient features. Grayscale distortion and geometric distortion, the scene changes greatly in the image, resulting in a certain degree of difference between the template image and the image to be matched, and the shallow features often fail, so the current template preparation requires a lot of money Manpower, complex operation process and low efficiency; most deep neural networks currently require a high amount of data, and are often unable to identify the types of samples in the training set with a small number, that is, few samples and single samples.

SUMMARY OF THE INVENTION

In view of the defects of the prior art, the purpose of the present invention is to solve the technical problems that the prior art image matching algorithm basically cannot adapt to the changes of imaging viewpoint and scale, the scene adaptability and anti-interference ability are weak, and the matching success rate is low.

In order to achieve the above object, in a first aspect, an embodiment of the present invention provides a heterogeneous twin area selection network, and the heterogeneous twin area selection network includes a serially connected heterogeneous twin network and an area matching network;

The heterogeneous twin network is used to extract the feature map of the template map and the feature map of the map to be matched;

The region matching network is used to obtain a region matching result according to the feature map of the template map and the feature map of the map to be matched;

The heterogeneous twin network includes a sub-network A and a sub-network B in parallel with each other, each sub-network is a feature extraction module, a feature fusion module and a maximum pooling module in series, and the two sub-networks are only the first-layer convolution of the feature extraction module. The convolution kernels are different, and the rest of the modules are the same.

Specifically, the feature extraction module is used to extract the feature map of the input image, the feature fusion module is used to fuse the convolution features of the last three layers of the feature extraction module, and the maximum pooling module normalizes the scale of the fusion feature .

Specifically, the feature extraction module replaces the second layer of ResNet18 with convolution.

Specifically, the feature extraction module replaces the last layer of ResNet18 with a convolutional layer.

Specifically, the area matching network includes: a feature division module, a classification module, and a position regression module; wherein, the classification module and the position regression module are connected in parallel, processed in parallel, and connected in series after the feature division module;

The feature division module includes: a first convolution, a second convolution, a third convolution, and a fourth convolution;

The first convolution is used to extract the template classification feature map from the feature map of the template map;

The second convolution is used to extract the template position feature map from the feature map of the template map;

The third convolution is used to extract the classification feature map of the map to be matched from the feature map of the map to be matched;

The fourth convolution is used to extract the position feature map of the map to be matched from the feature map of the map to be matched;

The classification module is configured to use the template classification feature map as a convolution kernel, perform convolution with the classification feature map of the map to be matched, and output the matched category;

The position regression module is configured to use the template position feature map as a convolution kernel, perform convolution with the position feature map of the map to be matched, and output the matched position.

In a second aspect, an embodiment of the present invention provides an image matching method based on the heterogeneous twin region selection network described in the first aspect, including the following steps:

S1. Use the training sample to train the heterogeneous twinned region selection network, the training sample is a template map-to-be-matched map pair, and the label of the training sample is the location information of the corresponding region of the template map in the to-be-matched map;

S2. Input the sample to be tested into the trained heterogeneous twin area selection network, and output the matching result of the sample to be tested.

Specifically, the total loss function during training=HCE+HSL;

d=|tt ^* |=|(x ₀ -x _G )+(y _o -y _G )+(w _o -w _G )+(h _o -h _G )|

t^*是对应的标签，即样本实际的位置

x、y表示横坐标、纵坐标，w、h为宽度、长度。Among them, HCE is the classification loss function, HSL is the position regression loss, p represents the probability that the predicted sample category is positive, p ^* is the corresponding label, N represents the total number of samples, and t represents the sample position output by the network

t ^* is the corresponding label, that is, the actual position of the sample

x and y represent abscissa and ordinate, and w and h represent width and length.

Specifically, if the intersection ratio of the output position and the actual label is IOU>0.7, then p ^* =1, otherwise, p ^* =0.

Specifically, after step S1 and before step S2, the sample to be tested can also be used to optimize the heterogeneous twin area selection network, as follows:

The template image in the test sample set and the image to be matched are input to the network, and the output result and the label are calculated and compared, and the matching success probability is measured by the cross-union ratio, so as to evaluate the performance of the image matching network and decide whether to continue training the network.

In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, implements the heterogeneous twin-based method described in the second aspect above. Image matching method for region selection network.

In general, compared with the prior art, the above technical solutions conceived by the present invention have the following beneficial effects:

1. The present invention applies the heterogeneous twin region selection network to image matching, and improves it according to the problems existing in the practical application of the network model. The algorithm realizes the input of a non-fixed scale template and a graph to be matched, and makes full use of the multi-layer feature information of the image at the same time. , which effectively improves the anti-interference ability of the matching method, adapts to the viewpoint and scale changes of imaging, increases the success rate and speed of matching, reduces the demand for template quality for matching, and adapts to matching under the conditions of few samples and single samples. In the application, the labor cost can be greatly reduced.

2. In the process of training the network, the present invention proposes a new type of loss function, that is, a balanced loss function is added, including balanced cross entropy loss HCE and balanced regression loss HSL. The matching success rate has a superior effect, and effectively improves the speed of network convergence.

Description of drawings

1 is a schematic diagram of a training sample provided by an embodiment of the present invention;

2 is a schematic diagram of a test sample provided by an embodiment of the present invention;

3 is a schematic diagram of a network structure for selecting a heterogeneous twin area according to an embodiment of the present invention;

4 is a flowchart of an image matching method based on a heterogeneous twin region selection network provided by an embodiment of the present invention;

Figure 5(a) is a template diagram to be tested provided by an embodiment of the present invention;

FIG. 5(b) is a matching result of a sample to be tested provided by an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

sample generation

(1) Prepare n different scenes {P ₁ ,...,P _i ...,P _n }, the same scene P _i contains several visible light images {P _i1 ,...,P _ij ,...P _iM }, P _ij is the jth picture of scene P _i .

(2) For each image of the same scene, the artificial frame selection area {P _ij1 , ..., P _ijk , ... P _ijs } in the scene, P _ijk represents the j-th image of the scene P _i The k-th image area. In different images of the same scene, the size, brightness, and angle of the regions are different, and the regions have certain deformations.

裁剪其中选择的区域k，作为模板图

随机选取另一图像

作为待匹配图，并标记k在待匹配图

中对应区域的位置，作为标签(ground truth)。(3) Among several visible light images {P _i1 , . . . , P _ij , . . . P _iM } of the same scene P _i , randomly select an image

Crop the selected area k as a template image

Pick another image at random

As the graph to be matched, and mark k in the graph to be matched

The position of the corresponding region in , as the label (ground truth).

表示场景P_i第j幅图像区域k所对应的标签，x_G表示区域k在图像P_ij中对应区域中心的横坐标，y_G表示区域k在图像P_ij中对应区域中心的纵坐标，w_G表示区域k在图像P_ij中对应区域的宽度，h_G表示区域k在图像P_ij中对应区域的长度。以上坐标均为像素坐标。

Represents the label corresponding to the jth image area k of the scene P _i , x _G represents the abscissa of the corresponding area center of the area k in the image P _ij , y _G represents the ordinate of the corresponding area center of the area k in the image P _ij , w _G represents the width of the corresponding region of the region k in the image P _ij , and h _G represents the length of the corresponding region of the region k in the image P _ij . The above coordinates are pixel coordinates.

与对应的待匹配图

以图对形式作为网络的输入，记为

(4) The template diagram

with the corresponding graph to be matched

Take the form of graph pairs as the input of the network, denoted as

作为训练样本集，其余场景

作为测试样本集，为了在测试中验证网络对于少样本、单样本情况的适应能力，训练集与测试集的样本类别不重叠，训练集不含测试集的样本类别，例如，测试集中包含飞机、船舶等，训练集中不含飞机船舶。训练集中部分训练样本如图1所示，包括类别车手、汽车、行人。测试集中部分测试样本如图2所示，包括类别建筑、兔子、飞机。And repeat this operation, traverse the whole scene {P ₁ , P ₂ , P ₃ ..., P _n }, select some scenes

As a training sample set, the rest of the scenes

As a test sample set, in order to verify the adaptability of the network to the few-sample and single-sample situations in the test, the sample categories of the training set and the test set do not overlap, and the training set does not contain the sample categories of the test set. For example, the test set contains aircraft, Ships, etc., aircraft and ships are not included in the training set. Part of the training samples in the training set are shown in Figure 1, including categories of drivers, cars, and pedestrians. Some test samples in the test set are shown in Figure 2, including the categories of buildings, rabbits, and airplanes.

Heterogeneous Twin Region Selection Network

As shown in Figure 3, the heterogeneous twinning region selection network includes a series-connected heterogeneous twinning network and a region matching network, the heterogeneous twinning network is used to extract the feature map of the template map and the feature map of the map to be matched, and the region matching The network is used to obtain the image matching result according to the feature map of the template map and the feature map of the map to be matched. The heterogeneous twinning region selection network is only suitable for sensor images of the same spectrum, such as the visible light template map and the visible light to-be-matched map, but not suitable for hetero-spectral image matching, such as the SAR template map and the visible-light to-be-matched map.

The heterogeneous twin network includes parallel sub-network A and sub-network B, and the sub-network includes serial feature extraction module, feature fusion module and maximum pooling module. Considering that the imaging viewpoint and image scale of the template image and the image to be matched are different, and the feature gap is large, the same convolution kernel is used to extract features, and the effect is general. Therefore, the convolution kernels of the first-layer convolution of the feature extraction modules of the two sub-networks are different. , while the rest of the modules are the same.

The feature extraction module is used to extract the feature map of the input image, the fusion module is used to fuse the convolution features of the last three layers of the feature extraction module, and the maximum pooling module normalizes the scale of the fusion features.

In the embodiment of the present invention, the feature extraction module adopts a residual network structure, preferably ResNet18. Further, the second layer of ResNet18 is changed from the maximum pooling layer (downsampling) to convolution, so that the network can automatically learn the appropriate sampling kernel function; the last layer of ResNet18 is changed from a fully connected layer to a convolutional layer, and then The maximum pooling module is connected to make the network meet the image input requirements of different sizes; the parameters of the last three layers of ResNet18 are modified to make the feature maps of the latter three layers the same size. The modified network is called ResNet18v2.

The multi-layer convolution feature map fusion in the feature fusion module is better for multi-scale region matching. Suppose the last three feature layers of A network are Conv_A1, Conv_A2, Conv_A3, and the last three feature layers of B network are Conv_B1, Conv_B2, Conv_B3, then the fusion features Conv_A and Conv_B are:

Wherein, w ₁ =2, w ₂ =4, and w ₃ =6.

The block maximum pooling operation is performed on the feature map, so that the network can adapt to the input of any size template map and the image to be matched, and improve the processing speed. The block maximum pooling refers to using the maximum pooling method for feature maps of different sizes to obtain a fixed size. feature map.

The region matching network includes: a feature division module, a classification module and a position regression module, wherein the classification module and the position regression module are connected in parallel, processed in parallel, and connected in series after the feature division module

The feature division module includes: a first convolution, a second convolution, a third convolution, and a fourth convolution. The first convolution is used to extract the template classification feature map from the feature map of the template map, and the second convolution is used to extract the template classification feature map. The template location feature map is extracted from the feature map of the template map, the third convolution is used to extract the classification feature map of the to-be-matched map, and the fourth convolution is used to extract the to-be-matched map location feature map.

The classification module is used to use the template classification feature map as the convolution kernel, convolve with the classification feature map of the map to be matched, and output the matching category, including matching and non-matching.

The position regression module is used to use the template position feature map as the convolution kernel, convolve with the position feature map of the map to be matched, and output the matching position.

Balanced loss function = classification loss function HCE + location regression loss HSL

Classification loss function HCE

Consider a simple binary cross entropy loss:

Among them, x is the initial class output of the HES-RPN network, p is the positive sample class probability output by the network, the value range of p is [0, 1], p ^* is the corresponding label, and the value of p ^* is 0 or 1. If the intersection ratio of the output position and the actual label, IOU>0.7, then p ^* =1, otherwise, p ^* =0.

Its gradient (derivative) to x is:

i表示样本i。具体来说，将梯度模长的取值范围划分为若干个单位区域。对于一个样本，若它的梯度模长为g，它的密度就定义为处于它所在的单位区域内的样本数量除以这个单位区域的长度ε：So we can define a gradient modulo length,

i represents sample i. Specifically, the value range of the gradient modulus length is divided into several unit regions. For a sample, if its gradient modulus length is g, its density is defined as the number of samples in its unit area divided by the length of this unit area ε:

The initial value of α is 1, and it gradually rises to 2 as the training progresses. k represents the sample k; N represents the total number of samples, and ε is a very small number, generally 0.01. The reciprocal of the gradient density is the weight to be multiplied by the sample after calculating the loss, then the loss of the new classification:

Location regression loss HSL

The traditional commonly used regression branch loss function Smooth L1, which is a piecewise function:

Then when the distance deviation between the sample and the label is large, that is, |x|≥1, the derivative of the smoothL ₁ function is constant at 1, and the impact on the network parameter update is the same, so it is impossible to distinguish the specific difficulty of the sample. To solve this problem, a DSL loss is introduced:

d=|tt ^* |=|(x ₀ -x _G )+(y _o -y _G )+(w _o -w _G )+(h _o -h _G )|

r is the location regression loss tag, which is easy to distinguish from the classification loss. Then the new position regression loss HSL is expressed as:

in,

x_o表示在待匹配图P_NM中异构孪生区域选取网络的输出区域中心的横坐标，y_o表示在待匹配图P_NM中异构孪生区域选取网络的输出区域中心的纵坐标，w_o表示在待匹配图P_NM中异构孪生区域选取网络的输出区域的宽度，h_o表示在待匹配图P_NM中异构孪生区域选取网络的输出区域长度。t^*是对应的标签，即样本实际的位置

The initial value of β is 1, and it gradually increases to 1.5 as the training progresses. u is a decimal, generally 0.02, t represents the sample position output by the network

x _o represents the abscissa of the output area center of the heterogeneous twin area selection network in the to-be-matched graph P _NM , yo represents the ordinate of the output area center of the heterogeneous twin area selection network in the to _- be-matched graph P _NM , w _o Represents the width of the output area of the heterogeneous twin area selection network in the to-be-matched graph P _NM , and h _o represents the output area length of the heterogeneous twin area selection network in the to-be-matched graph P _NM . t ^* is the corresponding label, that is, the actual position of the sample

As shown in Figure 4, an image matching method based on a heterogeneous twin region selection network includes the following steps:

S1. Use the training sample to train the heterogeneous twinned area selection network, the training sample is a template graph-to-be-matched graph pair, and the label of the training sample is the position of the corresponding area of the template graph in the to-be-matched graph;

S2. Input the sample to be tested into the trained heterogeneous twin area selection network, and output the matching result of the sample to be tested.

Image matching using the web. Select any two images, the category is not limited, as the template, the image to be matched, input the trained image in the form of image pair to match the heterogeneous twin area selection network, and then the matching result can be obtained.

Fig. 5(a) is a diagram of a template to be tested, and Fig. 5(b) is a matching result of a sample to be tested.

The same data set is matched by using the heterogeneous twin region selection network in the present invention and SiameseFC, SiamRPN, SiamRPN++, grayscale cross-correlation, and HOG matching in the prior art, and the matching accuracy and running time are shown in Table 1.

Table 1

The running time refers to the time required to complete a matching when the size of the template image is 127*127 and the size of the image to be matched is 512*512.

After step S1 and before step S2, the sample to be tested can also be used to optimize the heterogeneous twin area selection network.

The template image in the test sample set and the image to be matched are input to the network, and the output result and the label are calculated and compared, and the matching success probability is measured by the cross-union ratio, so as to evaluate the performance of the image matching network and decide whether to continue training the network.

与实际标签

的交并比IOU，具体公式为：network output

with actual labels

The intersection and union ratio of IOU, the specific formula is:

oarea=w _o *h _o

garea=w _G *h _G

The images whose intersection and union ratio IOU is greater than 0.3 are regarded as successful matching, and the ratio of successfully matching images to all images is calculated, which is the matching success rate. The higher the success rate, the better the matching performance.

中共有T张图像，其中，A张图像交并比大于0.3，那么匹配成功率SR(success rate)即为：

Set up a test set scenario

There are T images in total, among which, the intersection ratio of A images is greater than 0.3, then the matching success rate SR (success rate) is:

With the training process, if the SR no longer increases, it means that the network performance no longer improves, stop training the network, otherwise continue to train the network.

The above are only the preferred embodiments of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, All should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (8)

1. an image matching method based on heterogeneous twin region selection network, is characterized in that, comprises the steps: S1. Use the training sample to train the heterogeneous twinned region selection network, the training sample is a template map-to-be-matched map pair, and the label of the training sample is the position information of the corresponding region of the template map in the to-be-matched map, the heterogeneous The twin region selection network includes a series of heterogeneous twin networks and a region matching network; the heterogeneous twin network is used to extract the feature map of the template map and the feature map of the map to be matched; the region matching network is used to extract the feature map of the template map. The feature map and the feature map of the map to be matched are obtained to obtain a regional matching result; the heterogeneous twin network includes a sub-network A and a sub-network B that are connected in parallel with each other, and each sub-network is a feature extraction module, a feature fusion module and a maximum pool in series in sequence The two sub-networks only have different convolution kernels of the first layer convolution of the feature extraction module, and the rest of the modules are the same; S2. Input the sample to be tested into the trained heterogeneous twin area selection network, and output the matching result of the sample to be tested. 2. image matching method as claimed in claim 1 is characterized in that, described feature extraction module is used for extracting the feature map of input image, and described feature fusion module is used for the convolution feature of last three layers of fusion feature extraction module, The max pooling module normalizes the fused feature scale. 3. The image matching method of claim 1, wherein the feature extraction module replaces the second layer of ResNet18 with convolution. 4. The image matching method of claim 1, wherein the feature extraction module replaces the last layer of ResNet18 with a convolutional layer. 5. The image matching method according to claim 1, wherein the region matching network comprises: a feature division module, a classification module and a position regression module; wherein, the classification module and the position regression module are connected in parallel, processed in parallel, and connected in series After the feature division module; the feature division module includes: a first convolution, a second convolution, a third convolution, and a fourth convolution; the first convolution is used for extracting from the feature map of the template map template classification feature map; the second convolution is used to extract the template location feature map from the feature map of the template map; the third convolution is used to extract the to-be-matched map classification feature map from the feature map of the to-be-matched map; The fourth convolution is used to extract the location feature map of the map to be matched from the feature map of the map to be matched; the classification module is used to use the template classification feature map as a convolution kernel to perform convolution with the classification feature map of the map to be matched. , and output the matching category; the position regression module is used to use the template position feature map as a convolution kernel, perform convolution with the position feature map of the map to be matched, and output the matching position. 6. The image matching method according to any one of claims 1 to 5, wherein, if the intersection ratio of the output position and the actual label is IOU>0.7, then the sample label p*=1, otherwise, the sample label p*= 0. 7. The image matching method according to any one of claims 1 to 5, characterized in that, after step S1 and before step S2, the heterogeneous twin area selection network can also be optimized with the sample to be tested, as follows: The template image in the test sample set and the image to be matched are input to the network, and the output result and the label are calculated and compared, and the matching success probability is measured by the cross-union ratio, so as to evaluate the performance of the image matching network and decide whether to continue training the network. 8. A computer-readable storage medium, characterized in that, a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented. An Image Matching Method for Heterogeneous Siamese Region Selection Networks.

Images