CN118628997A - A cross-modal place recognition method based on efficient self-attention - Google Patents

Abstract

The invention discloses a cross-modal place recognition method based on efficient self-attention, comprising: inputting an RGB image and an infrared image into a dual-stream network; performing a quantization operation on the input image, and extracting image salient features by a color contrast method; inputting the image into a ResNet50 network, performing high-dimensional feature mapping on the features output by the third stage of the ResNet50 network by a high-dimensional feature mapping module, and outputting final overall features; performing image block processing on the features output by the third stage of the ResNet50 network, and processing local shared features by an efficient self-attention module EMSA, and outputting local features; fusing image salient features and local features to obtain final local features; performing overall and local feature collaborative constraints on the final overall features and the final local features; if a specified number of training rounds is reached, then terminating the training to obtain a cross-modal place recognition model based on an efficient self-attention mechanism; otherwise, continuing to complete the training; and improving the accuracy and efficiency of the entire network place recognition.

Description

A cross-modal place recognition method based on efficient self-attention

Technical Field

The present invention relates to the field of computer vision technology, and in particular to a cross-modal place recognition method based on efficient self-attention.

Background Art

Visual Place Recognition (VPR) is a highly advanced technology that is mainly used to assist robots or visual navigation systems to accurately determine whether they are in a previously visited location. VPR uses images captured by cameras for intelligent recognition to achieve accurate location positioning. In visual simultaneous localization and mapping (SLAM) systems, VPR plays a key role, not only for relocalization, but also for effective map reuse and loop correction. Numerous technological innovations and developments have made significant progress in the field of visual place recognition (VPR). However, the VPR task still faces a series of challenges, especially the significance of intra-class differences between images, which is mainly caused by changes in appearance and perspective. Although many studies have proposed solutions to these challenges, most studies have not fully considered the impact of illumination changes.

When faced with the challenge of changing lighting conditions, the robot adopts a flexible hardware solution to adapt to different environments; in high-light environments, the robot uses RGB cameras for image acquisition; in low-light conditions, it switches to infrared cameras. The main goal of cross-modal visual place recognition with visible and infrared light (VI-VPR) is to capture images of places from cameras of different modalities and determine their locations based on them. The uniqueness of this approach is that its query set and gallery set contain images from different modalities, which effectively overcomes the limitations caused by lighting conditions. However, compared with traditional single-modal place location recognition, VI-VPR faces unique challenges, especially when dealing with cross-modal differences between RGB images and infrared images. There are significant differences between infrared and visible light images, which makes existing place recognition methods less effective when dealing with multimodal recognition problems; there is also a major disadvantage of slow feature matching, which limits the overall performance improvement.

Summary of the invention

Purpose of the invention: In order to overcome the shortcomings of the prior art, the present invention provides a cross-modal place recognition method based on efficient self-attention, proposes an efficient self-attention method, and integrates pixel color contrast as a key factor into the network, thereby improving the feature matching speed in the place recognition task without losing accuracy.

Technical solution: To achieve the above purpose, a cross-modal place recognition method based on efficient self-attention includes the following steps:

Step 1: Input the RGB image and infrared image into the two-stream network;

Step 2: quantize the input image, calculate the color distance between the RGB image and the infrared image, and the color distance between the image before and after quantization of each modality, and use the color contrast method to extract the image's salient features;

Step 3: Input the image into the ResNet50 network to extract unique features from the image of each modality;

Step 4: The features output by the third stage of the ResNet50 network are mapped to high-dimensional features by the high-dimensional feature mapping module, and the final overall features are output;

Step 5: Perform image block processing on the features output by the third stage of the ResNet50 network to obtain local shared features, and process the local shared features through the efficient self-attention module EMSA to output local features;

Step 6: Fusing the image salient features obtained in step 2 and the local features obtained in step 5 to obtain the final local features;

Step 7: Coordinate constraints between the final global features and the final local features;

Step 8: If the specified number of training rounds is reached, the training is terminated to obtain a cross-modal place recognition model based on an efficient self-attention mechanism; otherwise, return to step 1 to continue training.

Furthermore, assuming that there are n RGB location images and m infrared location images, the samples of RGB mode are represented as The infrared modal sample is represented as in, represents the i-th RGB location image, represents the jth infrared location image, and Respectively and corresponding landmarks;

In the step 2, the image is defined as:

I(x,y)＝{I _L (x,y),I _a (x,y),I _b (x,y)}

After normalization, _0≤IL (x,y), _Ia (x,y), _Ib (x,y)≤1;

The input RGB image and infrared image are quantized to obtain two quantized color images _fi and _gi . The difference between the two quantized color images _fi and _gi is evaluated by the color distance d. The color distance d is expressed as:

In the formula, L represents the brightness of the image, a represents the component from green to red, and b represents the component from blue to yellow. Further, let the number of colors in the color palette after color quantization be K, and find K colors in the color space:

(C _1L ,C _1a ,C _1b ),(C _2L ,C _2a ,C _2b ),…,(C _KL ,C _Ka ,C _Kb )

Where, 0≤C _KL ,C _Ka ,C _Kb ≤1;

Calculate the color distance between the RGB image before quantization and the image after quantization g _i , and calculate the color distance between the infrared image before quantization and the image after quantization _fi ; Taking the color image as an example, assume that the image before quantization The color distance d of the quantized image _gi satisfies the formula:

In the formula, 1≤x≤W, 1≤y≤H, 1≤j≤K, W, H are the width and height of the image; after quantization, The quantified result is processed using the color contrast method to obtain the significant features of each pixel color in the image.

Furthermore, we choose to use the Lab color space to calculate the color contrast. By comparing the color of a specific pixel with the colors of other pixels in the image, we can quantify the saliency of the pixel. We get the saliency value of each pixel color in the image. Let an element in the image be Then the significance value is:

Since the image has been color quantized, the color of each pixel is determined by the color histogram features of the image, from which the corresponding quantized color value is obtained; the proposed optimization method only needs to calculate the saliency value associated with the color corresponding to each pixel to obtain the optimized saliency value:

Where c _l is the quantized color value of pixel M, f _j is the frequency of color c _l in the image, α _j is the quantitative value of color c _l , and K is the number of quantized colors;

The image salient features in the image are extracted based on the saliency value associated with each pixel in the image and the color corresponding to each pixel.

Furthermore, in step 5, the features output in the third stage of the ResNet50 network are and Extract local shared features in the form of feature blocks and Local shared features and The local features are obtained through the efficient attention module EMSA

Let F be one of the overall feature candidate descriptors:

Where C, H, and W represent the three dimensions of channel, height, and width, respectively. A set of patch-level features {P _i , x _i , y _i } with a stride of _Sp and a size of d _x ×d _y is extracted from F. The total number of patch-level features is:

Where _Pi represents the patch-level feature set, ( _xi , _yi ) represents the coordinates of the center of the patch-level feature on the feature map;

The efficient attention module EMSA performs a convolution operation on _Pi to obtain Q, and _Pi is recorded as x. The two-dimensional input is marked Reshaped into a three-dimensional marker along the spatial dimension And input to the depth convolution operation to reduce the height H and width W dimensions; s is the reduction factor, s is adaptively set according to the feature map size or the number of stages, the kernel size, stride and padding are s+1, s and s/2 respectively; the new label mapping after spatial reduction is reshaped into a two-dimensional mark, Where n' = h/s × w/s; is fed into two sets of projection operations to obtain the key K and value V.

Furthermore, the EMSA equation is used to calculate the attention function on the query Q, key K and value V; the EMSA equation is as follows:

Where Conv(·) represents a standard 1×1 convolution operation, which is used to simulate the interaction between different attention heads;

The setting of the efficient attention module EMSA enables the function of each attention head to depend on all keys and queries at the same time; instance normalization is introduced in the dot product matrix after applying the Softmax function, marked as IN(·); the output values of each head are connected and linearly projected to form the final output.

Furthermore, the value of the EMSA equation calculation cost _St is:

Furthermore, in step 6, the saliency map extracted in step 2 is And the local features obtained in step 5 Perform feature fusion to obtain the image saliency map M of the fused color features:

The above formula is to measure the difference between two saliency values S(I _k ) and m(I _k ) of a certain pixel I _k .

Beneficial effects: The present invention adopts a cross-modal place recognition method based on efficient self-attention, which integrates pixel color contrast into the network as a key factor, and successfully solves the problem of slow feature matching speed in place recognition tasks without losing accuracy; proposes a cross-modal model EMSA based on an efficient self-attention module. Through the application of this method, place recognition tasks in complex environments can be completed faster and more accurately, which significantly promotes the progress of visual place recognition technology; and selects a K-Means clustering quantization method with lower image distortion; improves the accuracy and efficiency of the entire network place recognition.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a flowchart of the cross-modal place recognition model training based on the efficient self-attention mechanism;

Figure 2 is a schematic diagram of the high-efficiency attention EMSA model;

Figure 3 is a schematic diagram of the extraction process of the RGB image saliency map S(I);

FIG4 is a schematic diagram of the extraction process of the RGB image saliency map M(I).

DETAILED DESCRIPTION

The present invention will be further described below in conjunction with the accompanying drawings.

As shown in FIG1 , a cross-modal place recognition method based on efficient self-attention includes the following steps:

Step 1: Input the RGB image and infrared image into the two-stream network;

Step 2: quantize the input image, calculate the color distance between the RGB image and the infrared image, and the color distance between the image before and after quantization of each modality, and use the color contrast method to extract the image's salient features;

Step 3: Input the image into the ResNet50 network to accurately extract unique features from the image of each modality;

Step 4: The features output by the third stage of the ResNet50 network are mapped to high-dimensional features by the high-dimensional feature mapping module, and the final overall features are output;

Step 5: Perform image block processing on the features output by the third stage of the ResNet50 network to obtain local shared features, and process the local shared features through the efficient self-attention module EMSA to output local features;

Step 6: Fusing the image salient features obtained in step 2 and the local features obtained in step 5 to obtain the final local features;

Step 7: Coordinate constraints between the final global features and the final local features;

Step 8: If the specified number of training rounds is reached, the training is terminated to obtain a cross-modal place recognition model based on an efficient self-attention mechanism, and the location is accurately recognized based on the obtained model; otherwise, return to step 1 to continue training.

There is also a step between step six and step seven, which is to fuse the features of the fourth stage of the ResNet50 network and the final overall features through a high-dimensional intra-modal feature aggregation module to constrain the overall features.

As shown in Figure 3, suppose there are n RGB location images and m infrared location images, and the samples of RGB mode are represented as The infrared modal sample is represented as in, represents the i-th RGB location image, represents the jth infrared location image, and Respectively and The corresponding landmark; RGB images are composed of three primary colors: red, green, and blue, and contain three channels, while infrared images exist in the form of a single channel; Since the number of channels of infrared location images is 1, in the data preprocessing stage, the strategy of filling the same value is used to expand it into three channels, so as to keep the number of channels consistent with the RGB location image, which is convenient for subsequent model training; In step 1, input an RGB modal sample and an infrared modality sample

In the step 2, the extraction of image color features in the Lab color space is considered, and the image is defined as:

I(x,y)＝{I _L (x,y),I _a (x,y),I _b (x,y)}

After normalization, _0≤IL (x,y), _Ia (x,y), _Ib (x,y)≤1; the RGB image is defined as consisting of the image brightness L, the green to red component a, and the blue with yellow component b; similarly, the infrared image is defined as consisting of the image brightness L, the green to red component a, and the blue with yellow component b.

As shown in Figure 3, the input RGB image and infrared image are quantized. The quantization operation implemented on the input image uses the K-Means method to obtain two quantized color images _fi and g _i . _fi was originally an infrared image, but it has been converted into a three-channel format after image preprocessing, so it is also regarded as a color image. K-Means is a popular color quantization technology that is implemented by dividing the color space into multiple clusters, each cluster representing a specific color. The color quantization result using the K-Means method has almost no obvious visual difference from the original image, and there is no color distortion problem. The difference between the two quantized color images _fi and g _i is evaluated by the color distance d, and the color distance d is expressed as:

Where L represents the brightness of the image, a represents the component from green to red, and b represents the component from blue to yellow.

Assume that the number of colors in the color palette after color quantization is K, find K colors in the color space, and record the brightness L of each color, the component a from green to red, and the component b from blue to yellow:

(C _1L ,C _1a ,C _1b ),(C _2L ,C _2a ,C _2b ),...,(C _KL ,C _Ka ,C _Kb )

Where, 0≤C _KL ,C _Ka ,C _Kb ≤1;

As shown in FIG3 , the color distance between the RGB image before quantization and the image after quantization g _i is calculated, and the color distance between the infrared image before quantization and the image after quantization _fi is calculated; taking the color image as an example, assuming that the image before quantization The color distance d of the quantized image _gi satisfies the formula:

In the formula, 1≤x≤W, 1≤y≤H, 1≤j≤K, W, H are the width and height of the image; after quantization, The quantified result is processed using the color contrast method to obtain the significant features of each pixel color in the image.

As shown in Figure 3, the Lab color space is selected to calculate the color contrast. By comparing the color of a specific pixel with the colors of other pixels in the image, the prominence of the pixel is quantified. The Lab color space is particularly suitable for such calculations because it can more accurately depict color brightness and color differences. The Lab color space method not only provides quantitative analysis of color contrast, but also effectively reveals significant features in the image, providing important information for image processing and analysis. Then, the prominence value of each pixel color in the image is obtained. Suppose an element in the image is Then the significance value is:

When processing an image, if the saliency value of each pixel is calculated separately, the execution efficiency of the algorithm will be significantly reduced. Considering that the image has been color quantized, the color of each pixel is determined by the color histogram features of the image, and the corresponding quantized color value is obtained from it; the optimization method is proposed to optimize the above processing flow, and only the saliency value related to the color corresponding to each pixel needs to be calculated, thereby greatly reducing repeated calculations and obtaining the optimized saliency value:

Where c _l is the quantized color value of pixel M, f _j is the frequency of color c _l in the image, α _j is the quantitative value of color c _l , and K is the number of quantized colors;

According to the saliency value related to each pixel in the image and the color corresponding to each pixel, the image saliency features in the image are extracted; taking the RGB image as an example, the process of extracting the saliency map S(I) based on color contrast is shown in Figure 3.

As shown in FIG. 2 , in step 5, the features output in the third stage of the ResNet50 network are and Extract local shared features in the form of feature blocks and Local shared features and The local features are obtained through the efficient attention module EMSA

Let F be one of the overall feature candidate descriptors:

Where C, H, and W represent the three dimensions of channel, height, and width, respectively. A set of patch-level features {P _i , _xi , _yi } with a size of d _x ×d _y and a stride of _Sp is extracted from F. The total number of patch-level features is:

Where _Pi represents the patch-level feature set, ( _xi , _yi ) represents the coordinates of the center of the patch-level feature on the feature map;

As shown in Figure 2, the initial efficient attention module EMSA operation is the same as the multi-head self-attention MSA, and the convolution operation is performed to obtain Q. That is, the efficient attention module EMSA performs a convolution operation on _Pi to obtain Q. In order to compress the memory, _Pi is recorded as x, and the two-dimensional input mark Reshaped into a three-dimensional marker along the spatial dimension And input to the depth convolution operation to reduce the height H and width W dimensions; s is the reduction factor, s is adaptively set according to the feature map size or the number of stages, the kernel size, stride and padding are s+1, s and s/2 respectively; the new label mapping after spatial reduction is reshaped into a two-dimensional mark, where n' = h/s x w/s; then is fed into two sets of projection operations to obtain the key K and value V.

The EMSA equation is used to calculate the attention function on the query Q, key K and value V; the EMSA equation is as follows:

Where Conv(·) represents a standard 1×1 convolution operation, which mainly simulates the interaction between different attention heads;

The setting of the above efficient attention module EMSA is designed so that the function of each attention head can depend on all keys and queries at the same time. This method may weaken the ability of the multi-head self-attention MSA mechanism to simultaneously process diverse representation subsets from different positions to some extent. In order to make up for this deficiency and restore the diversity among heads, instance normalization is introduced in the dot product matrix after applying the Softmax function, marked as IN(·), which not only improves the sensitivity of the model to different feature subsets, but also enhances the overall attention distribution, effectively balancing the concentration and dispersion of the attention mechanism. Finally, the output values of each head are connected and linearly projected to form the final output.

The EMSA equation calculates the cost value of _St is:

Assuming s > 1, the computational cost of EMSA is much lower than that of the original MSA, especially at lower stages where s is usually higher.

As shown in FIG. 4 , in step 6, the saliency map extracted in step 2 is And the local features obtained in step 5 Perform feature fusion to obtain an image saliency map M of fused color features; calculate the residual value between the image pixel saliency value S(I _k ) and m(I _k ), and the residual value is used to evaluate the consistency between the two saliency maps; through this method, a more accurate image saliency map M fused with color features can be obtained. The image saliency map obtained by fusion of the two image saliency values S(I _k ) and m(I _k ) of a certain pixel I _k is shown in the following formula:

The above formula is to measure the difference between the two salient values S(I _k ) and m(I _k ) of a certain pixel I _k ; when the two salient values of a certain pixel are similar, it indicates that the certain pixel shows significance from the perspective of both high-level features based on deep learning and low-level features of the image based on color information; the salient value after fusion processing will be close to 1, reflecting a high degree of significance; the feature fusion diagram is shown in Figure 4.

The testing process of the cross-modal place recognition model based on the efficient self-attention mechanism:

S1, input query dataset and gallery dataset, and enter S2;

S2: Using the trained model, perform overall feature extraction and local feature extraction on all location images of the query dataset and gallery dataset input in S1, and enter S3;

S3: perform similarity matching between the overall features of the query dataset and the overall features of the gallery dataset, and proceed to S4;

S4: The candidate ranking is obtained by matching the overall features, and then re-ranked by local features, and then enters S5;

S5: According to the similarity, the matching result of each location image in the query data set and the library data set is obtained, and then S6 is entered;

S6: Calculate the model size and the inference time of feature matching, and end.

The query data set in step S1 of the test flow represents the location images to be queried, and the gallery data set represents the set of location images to be matched by the query set; in step S5 of the test flow, each image in the query data set will be matched with several images in the gallery set. The present invention uses the Recall@N curve to evaluate the performance index of the location recognition algorithm, and uses M and ms as units to evaluate the model size and inference time of the EMSA proposed by the present invention.

The common evaluation criteria for accuracy are Top-1, Top-5, Top-10, etc.; the present invention trains and evaluates the model on the public dataset KAIST, which is widely used in cross-modal place recognition tasks; compared with the existing SCHAL-Net technology, in the mode where the infrared image queries the RGB image with a threshold of 10 meters, the accuracy of the present invention is improved by 1.8% in Top-10; and in the mode where the RGB image queries the infrared image with a threshold of 10 meters, the accuracy of the present invention is improved by 1.5% and 1.7% in Top-1 and Top-10 respectively. In terms of model size, the model proposed by the present invention is 1M smaller than the SCHAL-Net technology, and the inference time is improved by 7ms; overall, the accuracy is higher than the existing SCHAL-Net technology, and the recognition efficiency of place recognition is improved while ensuring the accuracy.

Example

To evaluate the performance of the proposed network model, the present invention trained and evaluated the model on the well-known public dataset KAIST, which is widely used in cross-modal place recognition tasks; the present invention's experiments evaluated the two retrieval modes of RGB image retrieval of infrared images and infrared image retrieval of RGB images; for the sake of fairness, various baseline methods were selected for comparison to verify the effectiveness of the EMSA model of the present invention. The cross-modal place recognition algorithm proposed in the present invention is compared with several other algorithms, including the performance results of SCHAL-Net algorithm, DOLG, Patch-NetVLAD, MixVPR, etc. on the KAIST public dataset; in order to comprehensively evaluate these algorithms, the present invention uses Recall@N as the evaluation index, where the values of N are 1, 5 and 10 respectively; as shown in Table 1 below:

Table 1 Experimental results of the present invention compared with the existing algorithms in the KAIST dataset when the threshold is 10 meters

Through comparative experiments, the place recognition algorithm proposed in the present invention performs particularly well on the KAIST dataset, and is significantly better than the original algorithm model in the scenarios of RGB images querying infrared images or infrared images querying RGB images. Specifically, in the case of RGB images querying infrared images, the maximum improvement in Top-5 recall rate can be observed, reaching 1.9%; and in the scenario of infrared images querying RGB images, the improvement in Top-1 recall rate is the most significant, reaching 1.5%. This fully proves the effectiveness and superiority of the improved algorithm model of the present invention.

The algorithm proposed in this paper is subjected to a detailed comparative analysis, with particular attention paid to the running efficiency and model size, and compared with other existing methods; to ensure the fairness of the comparison, all experiments are carried out under the same hardware configuration; as shown in Table 2 below

Table 2 Comparison of operating efficiency and model size between the present invention and different algorithms

According to the data shown in Table 3, it can be clearly seen that the algorithm proposed in the present invention has significant improvements in model size and reasoning time compared with the original model; this progress is mainly due to the optimization effect brought by the proposed attention algorithm. Compared with other currently popular place recognition algorithms, the improved algorithm model of the present invention also shows obvious superiority in model volume and reasoning speed. Specifically, compared with DOLG, the EMSA model reduces about 2M in memory and saves about 5ms in reasoning time. When the EMSA algorithm is compared with the MixVPR algorithm, which performs best in recognition effect, the EMSA model size is about half of that, while the reasoning time is 16 milliseconds faster; the results not only highlight the efficiency of the algorithm proposed in the present invention, but also highlight its significant advantages in saving resources and improving running speed, which is of great significance for the deployment of algorithms in practical application scenarios. In general, the algorithm proposed in the present invention not only performs well in performance, but also shows an excellent balance in terms of operating efficiency and resource consumption.

Based on the present invention, an implementation scenario is proposed, in which a city security monitoring system automatically analyzes activities on the street through location recognition technology so as to respond to potential security threats in a timely manner; including:

Install high-resolution surveillance color and infrared cameras in key areas to ensure that activities in the area can be clearly captured; configure the connection between the cameras and the central monitoring system so that video data can be transmitted and analyzed in real time. Integrate location recognition technology into the monitoring system; may include configuring image recognition software to analyze images captured by the camera; the system needs to be "learned" or "trained" first to recognize routine activities and potential abnormal behaviors in specific scenes. The system analyzes real-time video to identify and track individuals moving in the monitoring area; uses location recognition technology to analyze individual behavior patterns, such as walking routes, dwell time, and interactions with other individuals. The system then identifies any abnormal or suspicious behavior, such as unauthorized intrusions, abnormal gatherings, or potential criminal activities, based on preset security parameters; the system can improve the accuracy of identifying abnormal behavior by learning from historical data. Finally, once abnormal behavior is identified, the system will automatically send an alert to security personnel, with relevant video clips and specific location information.

The above is only a description of the preferred embodiments of the present invention. Ordinary technicians in this technical field can make several modifications and optimizations based on the above disclosure without departing from the above basic principles. These improvements and optimizations should be regarded as the understood protection scope of the present invention.

Claims (8)

1.A cross-modal location identification method based on high-efficiency self-attention is characterized in that: the method comprises the following steps:

Step one, inputting an RGB image and an infrared image into a double-flow network;

Performing quantization operation on an input image, calculating color distances of an RGB image and an infrared image and color distances of an image before quantization and an image after quantization of each mode, and extracting image significance characteristics by adopting a color contrast method;

Inputting the images into ResNet networks, and extracting unique features from the images of each mode;

Fourthly, performing high-dimensional feature mapping on the features output by the ResNet network in the third stage through a high-dimensional feature mapping module, and outputting final overall features;

fifthly, performing image block processing on the features output by the ResNet network in the third stage to obtain local sharing features, and processing the local sharing features through an efficient self-attention module EMSA to output the local features;

step six, fusing the image salient features obtained in the step two with the local features obtained in the step five to obtain final local features;

step seven, carrying out overall feature and local feature cooperative constraint on the final overall feature and the final local feature;

step eight, if the number of training rounds reaches the designated number, finishing training to obtain a cross-modal place recognition model based on the high-efficiency self-attention mechanism; otherwise, returning to the first step to finish training continuously.

2. The method for identifying the cross-modal location based on the high-efficiency self-attention as claimed in claim 1, wherein the method comprises the following steps: let RGB site images total n, infrared site images total m, RGB mode sample expressed asThe infrared mode sample is expressed asWherein, Represents the ith RGB place image,Represents the j-th infrared location image,AndRespectively representAndA corresponding landmark;

in the second step, the image is defined as:

I(x,y)＝{I_L(x,y),I_a(x,y),I_b(x,y)}

after normalization, I _L(x,y),I_a(x,y),I_b (x, y) is more than or equal to 0 and less than or equal to 1;

The input RGB image and the infrared image are quantized to obtain quantized two color images f _i and g _i, and the difference between the quantized two color images f _i and g _i is evaluated through a color distance d, wherein the color distance d is expressed as follows:

where L represents the brightness of the picture, a represents the green to red component, and b represents the blue to yellow component.

3. A method of cross-modal location identification based on efficient self-attention as recited in claim 2 wherein: let the number of palette colors after color quantization be K, find K colors in color space:

(C_1L,C_1a,C_1b),(C_2L,C_2a,C_2b),...,(C_KL,C_Ka,C_Kb)

wherein, C _KL,C_Ka,C_Kb is more than or equal to 0 and less than or equal to 1;

Calculating the color distance of the pre-quantization RGB image and the quantized image g _i, and calculating the color distance of the pre-quantization infrared image and the quantized image f _i; taking a color image as an example, an image before quantization is set And the color distance d of the quantized image g _i satisfies the formula:

Wherein x is not less than 1 and not more than W, y is not less than 1 and not more than H, j is not less than 1 and not more than K, W and H are the width and the height of the image; after quantization, make Minimum; the quantized result is processed by using a color contrast method to obtain the significance characteristics of each pixel color in the image.

4. A method of cross-modal location identification based on efficient self-attention as recited in claim 3 wherein: selecting Lab color space to calculate color contrast, and quantifying the saliency of a specific pixel by comparing the color of the pixel with the colors of other pixels in the image; obtaining the significance value of each pixel color in the image, and setting a certain element in the image asSignificance value:

because the image is subjected to color quantization processing, the color of each pixel is determined through the color histogram feature of the image, and a corresponding quantized color value is obtained; the optimization method is provided, wherein the saliency value related to the color corresponding to each pixel is only calculated, and the saliency value after optimization is obtained:

Wherein c _l is the quantized color value of pixel M, f _j is the frequency of occurrence of color c _l in the image, α _j is the quantized value of color c _l, and K is the quantized color number;

And extracting the image saliency characteristics in the image according to each pixel in the image and the corresponding color-related saliency value of each pixel.

5. The method for identifying the cross-modal location based on the high-efficiency self-attention as claimed in claim 1, wherein the method comprises the following steps: in the fifth step, the third phase output in ResNet network is characterized byAndExtracting local shared features in the form of feature blocksAndLocal sharing featuresAndLocal feature acquisition via high-efficiency attention module EMSA

Let F be one of the global feature candidate descriptors:

Wherein, C, H, W represent three dimensions of channel, height and width respectively, a group of patch level features { P _i,x_i,y_i } with steps S _p and d _x×d_y are extracted from F, and the total number of patch level features is:

Where P _i represents the patch level feature set, (x _i,y_i) represents the coordinates of the center of the patch level feature on the feature map;

The high-efficiency attention module EMSA carries out convolution operation on P _i to obtain Q, and marks P _i as x and two-dimensional input marks Three-dimensional markers reshaped into a dimension along spaceThe data are input into a depth convolution operation, so that the height H dimension and the width W dimension are reduced; s is a reduction factor, s is adaptively set according to the size or the stage number of the feature map, and the kernel size, the stride and the filling are s+1, s and s/2 respectively; new marker mapping with spatial reductionRemodelling into two-dimensional marks to obtainWherein n' =h/s×w/s; Is fed into two sets of projection operations to obtain the key K and the value V.

6. The method for identifying the cross-modal location based on the high-efficiency self-attention as recited in claim 5, wherein: employing the EMSA equation to calculate the attention function on query Q, key K, and value V; the EMSA equation is as follows:

where Conv (-) represents a standard 1X 1 convolution operation that mimics the interaction between different attention headers;

The efficient attention module EMSA is arranged, so that the function of each attention head can depend on all Key keys and Query at the same time; an example normalization Instance Normalization is introduced into the dot product matrix after the application of the Softmax function, and is marked as IN (; the output values of each head are connected and linearly projected to form the final output.

7. The method for identifying the cross-modal location based on the high-efficiency self-attention as recited in claim 6, wherein: the value S _t of the EMSA equation computation cost is:

8. The method for identifying the cross-modal location based on the high-efficiency self-attention as claimed in claim 1, wherein the method comprises the following steps: in the sixth step, the saliency map extracted in the second step is displayed And the local features obtained in step fiveFeature fusion is carried out, and an image saliency map M with fused color features is obtained:

The above formula is to measure the difference between two significant values S (I _k) and m (I _k) for a certain pixel I _k.