CN117078959A - Multi-modal salient target detection method based on cross-modal uncertainty region correction - Google Patents

Abstract

The invention discloses a multi-mode saliency target detection method based on cross-mode uncertain region correction, wherein a cross-mode feature enhancement module utilizes an attention mechanism to enhance effective information in shallow features and simultaneously suppresses interference information; according to the cross-modal feature correction module of the uncertain region perception, in the deep single-modal feature extraction process, cross-modal correction is carried out on uncertain regions existing in two modal deep features in a bidirectional interaction mode, so that single-modal features with better discrimination are obtained; performing cross-modal fusion on the visible light image features and the depth image features according to a multi-scale cross-modal feature fusion module, and fully mining multi-scale context information and complementary information in two modes; predicting a final significance map according to the cross-modal characteristics with discrimination to obtain a significance prediction map; and obtaining network model parameters by adopting a supervised learning model for the significance prediction graph. A complete and fine saliency prediction map is obtained.

Description

A multi-modal salient target detection based on cross-modal uncertainty region correction method

Technical field

The invention relates to the field of image processing technology, and in particular to a multi-modal salient target detection method based on cross-modal uncertainty region correction.

Background technique

Salient object detection aims to simulate the human visual system to segment the most visually attractive objects in the scene. In recent years, salient target detection has been widely used in many computer vision fields, such as target recognition, video segmentation, pedestrian re-identification, semantic segmentation and image quality assessment. It has important research significance and wide application value.

Most of the early salient target detection methods based on visible light depth images mainly designed manual features and used different prior information to design salient target detection algorithms for visible light depth images. However, hand-made feature representation capabilities are limited and lack the guidance of deep semantic information, making it difficult to accurately detect salient targets and encountering performance bottlenecks. In view of the powerful representation capabilities of deep convolutional neural networks, they have been successfully applied to salient target detection and have developed rapidly.

Although convolutional neural networks have achieved good detection results, these methods still face some challenges. In the process of imaging different modal images, due to limitations of imaging conditions (such as low light, haze and other special environments) or imaging technology (such as low-resolution cameras and interference from external equipment, etc.), the imaging of different modal images During the process, the imaging sensor of a certain mode may be affected by noise, resulting in the generation of low-quality images. When these low-quality depth images are used for salient target detection, the low-quality images inevitably introduce a certain amount of noise, which in turn reduces the discriminability of the fused features and degrades the performance of the model.

In order to alleviate the above problems, some saliency detection methods based on image quality issues have been proposed. Most existing methods mainly use feature-level selection methods to solve the problem of model performance degradation caused by image quality. In addition, there are also solutions from the aspect of image enhancement. By estimating a new depth image, and then extracting features from the estimated depth image and the original depth image for fusion, the depth image is enhanced to solve the possible problems of the depth image. Image quality issues.

However, the above method only considers low-quality depth images, but when the visible light image has low visual quality, the above model cannot detect and segment salient objects well.

Contents of the invention

In view of the above problems, a method based on cross-modal uncertainty region correction is proposed in this invention.

According to one aspect of the present invention, a multi-modal salient target detection method based on cross-modal uncertainty region correction is provided. The detection method includes:

According to the multi-modal image saliency target detection method based on cross-modal uncertainty region correction, low-quality visible light images and low-quality depth images are simultaneously considered;

According to the cross-modal feature enhancement module, the attention mechanism is used to enhance effective information in shallow features and suppress interference information;

According to the cross-modal feature correction module of uncertain area perception, in the single-modal feature extraction process, the uncertain areas existing in the deep features of the two modalities are cross-modally corrected through two-way information interaction, so as to Obtain more discriminative single-modal features;

Perform cross-modal fusion on the extracted visible light image features and depth image features according to the multi-scale cross-modal feature fusion module to fully exploit the multi-scale contextual information and complementary information in the two modal features;

Perform step-by-step decoding to predict the saliency map based on the discriminative cross-modal fusion features to obtain the final saliency prediction result;

A supervised learning model is used for the saliency prediction map to obtain network model parameters.

According to the cross-modal feature enhancement module, shallow features mainly provide rich appearance information to help the network refine the boundaries of salient targets. However, as mentioned before, input visible light images and depth images sometimes have lower visual quality. , which makes the shallow features extracted from low-quality input images contain a large amount of interference information, which will further reduce the discriminability of cross-modal fusion features. Therefore, in order to alleviate this problem, the attention mechanism is used to enhance the effective information in shallow features while suppressing the interference information.

According to the cross-modal feature correction module of uncertainty area perception, deep features mainly provide semantic information, which can help the network model better locate salient targets. However, affected by the interference information in low-quality input images, there are inevitably some uncertain regions in deep features. In these uncertain areas, there are usually features of one modality image with strong discriminability, while features of another modality image have interfering information. Therefore, through information interaction, we can use features with strong discriminability to correct features with interference information to alleviate the impact of interference information on the saliency prediction of uncertain areas. Therefore, through the uncertain area-aware cross-modal feature correction module we designed, cross-modal correction is performed on the uncertain areas where deep features of the two modalities exist to obtain more discriminative single-modal features and improve model accuracy. Accuracy of saliency object detection in uncertain regions.

According to the multi-scale cross-modal feature fusion module, one of the difficulties in the salient target detection task is that the scale, shape and position of the salient target are diverse. By fully mining the multi-scale cross-modal complementary information between visible light images and depth images, it helps to solve the problem of size and size diversity of salient targets in salient target detection. Therefore, we utilize the proposed multi-scale cross-modal feature fusion module to fully mine the multi-scale cross-modal complementary information in visible light features and depth features.

The saliency prediction map uses a supervised learning model to obtain network model parameters, which specifically include:

On the training data set, the supervised learning model is used to predict the saliency map, and the algorithm network training is completed end-to-end to obtain the network model parameters:

On the training data set, a supervised learning mechanism is used to obtain the loss function L _joint between the prediction result of the saliency map and the true value in the network model:

L _joint (S,G)＝l _bce (S,G)+l _iou (S,G)

where l _bce and l _iou are the cross-entropy loss function and the intersection-to-union ratio boundary loss function respectively; the overall loss function is set to:

The multi-modal saliency target detection method based on cross-modal uncertainty region correction provided by the present invention trains the algorithm end-to-end, and obtains model parameters after training the overall saliency detection network; after training the saliency detection When changing the network parameters, in order to avoid overfitting in the training data set, data augmentation operations of horizontal flipping and random cropping were performed on the visible light images and depth images of the data set.

Description of the drawings

Figure 1 is a flow chart of a multi-modal salient target detection method based on cross-modal uncertainty region correction disclosed in the present invention;

Figure 2 is an algorithm network diagram of a multi-modal salient target detection method based on cross-modal uncertainty region correction proposed by the present invention;

Figure 3 is a framework diagram of the cross-modal feature enhancement module proposed by the present invention;

Figure 4 is a framework diagram of the cross-modal feature correction module for uncertain areas proposed by the present invention;

Figure 5 is a framework diagram of the bidirectional cross-modal feature interaction sub-module and feature selection sub-module proposed by the present invention.

Figure 6 is a framework diagram of the multi-scale cross-modal fusion module proposed by the present invention.

Figure 7 is a simulation diagram of the evaluation results proposed by the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a thorough understanding of the disclosure, and to fully convey the scope of the disclosure to those skilled in the art.

The terms "comprising" and "having" and any variations thereof in the description, claims and drawings of the present invention are intended to cover non-exclusive inclusion, for example, the inclusion of a series of steps or units.

The technical solution of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

As shown in Figure 1, a multi-modal image salient target detection method based on cross-modal uncertainty area correction includes the following steps:

Specifically, the multi-modal image salient target detection network based on cross-modal uncertainty area correction contains three key modules: a dual-branch feature extraction network, a cross-modal feature enhancement module, and an uncertainty area-aware Cross-modal feature correction module and multi-scale cross-modal fusion module.

Step 1: Dual-branch feature extraction network. The dual-branch feature extraction network includes a visible light feature-related branch, a depth feature-related branch, two cross-modal feature enhancement modules, and three uncertain region-aware cross-modal feature correction modules. Both main branches use the VGG-16 network for feature extraction. In any branch, the input of the intermediate level is the output of the corresponding previous cross-modal feature enhancement module or the uncertain area-aware cross-modal feature correction module. The output features of the five visible light/depth VGG levels are represented as r ⁱ and ^di (i=1, 2, 3, 4, 5) respectively, where i represents the corresponding feature level. r and d are represented as visible light mode and depth mode data respectively. The features after cross-modal enhancement and correction using the cross-modal feature enhancement module and the uncertain area-aware cross-modal feature correction module are respectively expressed as and/>

Step 2: First, the cross-modal feature enhancement module is used to enhance the effective information in the shallow features, while suppressing the interference information to obtain enhanced visible light features. and deep features/> The specific process is as follows:

The first step: first use the spatial attention mechanism to select and enhance the two modal features in the spatial dimension, and then use the channel attention mechanism to select important features in the channel dimension for the enhanced single-modal features. Further suppress interference information in single-modal features. Specifically, given a specific level of single-modal RGB features r ⁱ and depth features d ⁱ (i=1,2), we first calculate their shared spatial attention map SA _i :

in, Represents element-wise multiplication operation. W _sa (*) represents the spatial weight generation function:

Here Avg(*) represents the average pooling operation along the channel, and Max(*) represents the maximum pooling operation along the channel dimension. Next, the shared spatial attention map is used as the weight of RGB features and depth features to select discriminative features for enhancement while suppressing interference information, expressed as:

Then, for features enhanced in the spatial dimension and/> Perform channel attention separately and generate channel attention maps to select important single-modal features in the channel dimension. Channel attention maps/> and/> Calculated respectively by the following formula:

Step 2: Cross-modal feature correction for uncertain area perception. Deep features mainly provide semantic information and can help the network model better locate salient targets. However, affected by the interference information in low-quality input images, there are inevitably some uncertain regions in deep features. In these uncertain areas, there are usually features of one modality image with strong discriminability, while features of another modality image have interfering information. Therefore, through information interaction, we can use features with strong discriminability to correct features with interference information to alleviate the impact of interference information on the saliency prediction of uncertain areas. The proposed cross-modal feature correction module for uncertainty area perception includes a bidirectional cross-modal feature interaction submodule and a feature selection submodule. In detail, the two-way cross-modal feature interaction sub-module mainly performs cross-modal correction on the uncertain region features that exist in the deep features of the two modalities, and the feature selection sub-module mainly selects from the corrected features to correct the uncertain region better. as the characteristics of the uncertain region.

Specifically, the proposed cross-modal correction module for uncertainty area perception first establishes the interactive relationship between RGB features and depth features by calculating the common information and uncertainty area information of deep RGB features and depth features in different local areas. . Then, the uncertainty region features in the corresponding modality are obtained by constructing the weights of the uncertainty regions of the RGB features and depth features. Since we obtain the uncertain regions that exist in the deep features of the two modalities and their characteristics in the corresponding modal regions, we cannot determine in advance which modality's corresponding uncertain region features are more discriminative. Therefore, the proposed two-way cross-modal feature interaction sub-module is first used to correct the uncertain region features of the two modes using two-way interaction. Then, the designed feature selection sub-module is used to select features that are better at correcting uncertain areas, and then the original RGB features and depth features are enhanced respectively. The detailed steps are as follows:

Step 1: Use the feature conversion function to first map the deep-level RGB features r ⁱ and depth features d ⁱ (i=3,4,5) into the same feature space, that is:

R ⁱ =Conv(r ⁱ ,σ ¹ ),

D ⁱ =Conv(d ⁱ ,σ ¹ ),

Among them, Conv (*, σ ¹ ) represents a 1×1 convolution operation. As the feature conversion function, its corresponding parameter is σ ¹ , and R ⁱ and D ⁱ represent the mapped features.

Step 2: Use the mapped features R ⁱ and D ⁱ to calculate the regional features shared by the deep-level RGB features and depth features through the following formula, and then calculate the characteristics of the uncertainty regions of the respective modes.

Among them, F _co represents the shared information between RGB features and depth features. feature Information representing uncertain areas in RGB features. Features/> Information that can reflect uncertain areas in depth features.

Step 3: Obtain the spatial weights of the uncertain areas in the deep features of the two modalities through spatial attention operations, and then use this weight to select the features of the corresponding area of the other modality, that is:

in, Represents the spatial attention weight generation function described in the cross-modal feature enhancement module. Represents the discriminability characteristics of the uncertain areas in the depth features in the corresponding RGB features. /> Represents the discriminability characteristics of the uncertain areas in the RGB features in the corresponding depth features. At the same time, we obtain the features of uncertain areas in RGB features and depth features through similar operations mentioned above/> and/> Right now:

Step 4: Use the bidirectional cross-modal feature interaction sub-module pair to obtain and/> and/> Cross-modal correction is performed using two-way interaction. In the bidirectional cross-modal feature interaction submodule, we mainly perform cross-modal feature correction through cross-attention operations. Specifically, /> and/> For example, first, similar to self-attention, 1×1 convolution and reshape operations are used to transform the input features from/> Convert to/> and/> Later/> and its transposition to perform matrix multiplication and normalization operations, and further calculate to generate a position correlation matrix/> Then, using the position correlation matrix pair/> Make corrections. The whole process can be expressed as:

Among them, Nor(*) means normalizing the values in the channel correlation matrix to [0,1], and Reshape(*) means normalizing the values in the channel correlation matrix to [0,1]. Convert from size C ₁ ×H×W to C ₁ ×HW. Conv(*,σ ² ) and Conv(*,σ ³ ) represent two 1×1 convolutional layers and their parameters σ ² and σ ³ . ReLU(*) represents the ReLU activation function. Since we are using a two-way information exchange method, for/> and/> and/> and/> Applying the same correction process above, we get and/>

Step 5: Since there is uncertainty in the two-way correction, that is, it is uncertain which correction method obtains more discriminative features, so after obtaining the cross-modal interactive correction, we designed a feature selection sub-module to correct the uncertain area. characteristics to select. Taking RGB feature correction as an example, first, in the proposed feature selection module, we correct the features of the uncertain area in two ways and/> Dimensionality reduction using 1×1 convolution. Secondly, it is compressed into a one-dimensional feature vector using global average pooling operation and global max pooling operation respectively. Finally, the feature weights Wi ^∈ R ^C are generated through two cascaded fully connected layers and a Sigmoid function operation to select the more discriminative features. Therefore, the feature selection sub-module uses the weight ^Wi to select features that are more accurate in correcting the uncertain area through the following formula. The mathematical expression of this process is:

Among them, σ(*) represents the Sigmoid function, GAP(*) represents the global average pooling operation, GMP(*) represents the global maximum pooling operation, FC(*, _γω ) represents the fully connected layer and its corresponding parameters _γω . We use the corrected features in the same way and/> Obtain the corrected features of the determined area in the depth feature/>

Therefore, the corrected RGB features and depth features obtained through the above operations are respectively and/> The mathematical expression of this process is:

Step 3: Multi-scale cross-modal fusion module. In the salient target detection task, one of the difficulties lies in the diversity of scales, shapes, and locations of salient targets. By fully mining the multi-scale cross-modal complementary information between RGB images and depth images, it helps to solve the problem of size and size diversity of salient targets in salient target detection. To this end, this chapter proposes a multi-scale cross-modal feature fusion module. Fully mine the multi-scale cross-modal complementary information in RGB features and depth features. Considering that the multi-scale feature pooling module based on hole convolution can effectively extract multi-scale features. Therefore, on the basis of ASPP, this chapter proposes a multi-scale feature extraction module (Re_ASPP) based on hole convolution with residual connection, and applies it to the cross-modal fusion feature module to perform cross-modal fusion. Previously, multi-scale features were extracted from single-modal features. Re_ASPP uses four branches, each using hole convolution layers with different hole rates to extract single-modal features of different scales. At the same time, considering that the receptive fields of single-modal features at different levels are different, as shown in Table 4.1, the hole rates of the Re_ASPP module hole convolution used at different levels are different. At the same time, in Re_ASPP we use residual connection to ensure the stability of the gradient. The mathematical expression of Re_ASPP is as follows:

Among them, δ(*) and Cat(*) represent the ReLU activation function and cascade operation respectively. Represents convolutional layers with different hole rates and corresponding parameters/> Indicates a convolution layer with a convolution kernel of 1×1 and its corresponding parameters/>

Therefore, in the multi-scale cross-modal feature fusion module, we first obtain the multi-scale features of the single modality through the Re_ASPP module, and then use the multiplication operation to mine the common information of the two modalities to supplement the original single-modal features. Enhancement, and finally the cross-modal fusion features are obtained through splicing operation. The specific calculation process is as follows:

Among them, Conv(*,σ ⁵ ) and Conv(*,σ ⁶ ) represent two convolutional layers and their parameters σ ⁵ and σ ⁶ .

Finally, the cross-modal fusion features are enhanced in the channel dimension through the channel attention mechanism, namely:

Among them, W _ca (*) represents the above-mentioned channel attention weight generation function. Represents the output features of the multi-scale cross-modal fusion module.

Step 4: Significance prediction. In order to obtain more detailed information, adjacent-level multi-modal features are also integrated for saliency prediction to fully exploit and utilize the complementary relationships between multi-level features and improve the performance of the salient target detection algorithm. A total of five significant prediction results were produced during the above process, namely:

Among them, S _i represents the saliency map generated corresponding to each level, and Conv(*,θ ^s ) represents a 1×1 convolution layer. Cat(*) and Up(*) represent cascaded feature mapping operations and bilinear interpolation upsampling operations along the channel dimension, respectively.

On the training data set, a supervised learning mechanism is used to obtain the loss function L _joint between the prediction result of the saliency map and the true value in the network model:

Among them, l _bce (*) and l _iou (*) are the cross-entropy loss function and the boundary loss function based on the intersection and union ratio, respectively. The definitions of the two are:

where G(m,n)∈{0,1} is the true value of each pixel label. P(m,n)∈{0,1} is the probability of predicting each pixel of the saliency map. W and H represent the width and height of the input image respectively.

The following is a further explanation of the technical effects of the present invention in combination with simulation experiments:

1. Simulation conditions: All simulation experiments are conducted under the operating system Ubuntu 16.04.5, the hardware environment is GPUNvidia GeForce GTX 1080Ti, and are implemented using the PyTorch deep learning framework;

2. Simulation content and result analysis:

Simulation 1

The present invention and the existing saliency detection method based on visible light depth images were used to conduct saliency detection experiments on six public visible light depth image saliency detection data sets DUT-RGBD, NJU2K, NLPR, LFSD, RGBD135 and STERE. Partially Experimental results are visually compared.

Compared with the existing technology, the present invention has better detection effect on input visible light-depth image pairs of medium and low visual quality images. Thanks to the cross-modal feature enhancement module in the present invention, effective information in shallow features is enhanced and interference information is suppressed. Thanks to the cross-modal feature correction module for uncertainty area perception in the present invention, the uncertain areas existing in the deep features of the two modalities are well corrected in the form of information interaction through the cross-modal feature bidirectional interaction module. , more discriminative single-modal features are obtained, thereby improving the accuracy of saliency prediction in uncertain regions. In addition, thanks to the full mining and capture of the multi-scale and multi-modal complementary information in the visible light image depth image in the present invention, the two clues are fully combined and their respective advantages are utilized, and small targets and multiple targets in complex scenes can be better identified. were segmented, and at the same time, relatively complete saliency detection results were obtained for multi-target images. The evaluation simulation results are shown in Figure 6:

Among them, (a) RGB image; (b) Depth image; (c) MMCI prediction result; (d) TANet prediction result; (e) DMRA prediction result; (f) CPFP prediction result; (g) ICNet prediction result; ( h) CPFP prediction results; (i) S2MA prediction results; (j) D3Net prediction results; (k) A2dele prediction results; (l) ASIFNet prediction results; (m) SSF prediction results; (n) DRLF prediction results; (o )DQSD prediction results; (p) CCAFNet prediction results; (q) JL-DCF prediction results; (r) DFMNet prediction results; (s) Ours prediction results; (t) true value. It can be seen from Figure 6 that the saliency map predicted by the present invention for RGB-D images is more complete overall and the details are finer, which fully demonstrates the effectiveness and superiority of the method of the present invention.

Simulation 2

The present invention is compared with the existing multi-modal saliency detection method based on RGB-D images on the six public RGB-D image saliency detection data sets DUT-RGBD, NJU2K, NLPR, RGBD135, LFSD, and STERE. The results obtained from the sex testing experiment are objectively evaluated using recognized evaluation indicators. The evaluation simulation results are shown in Table 1:

in:

F _β represents the maximum value of the weighted harmonization of precision rate and recall rate;

E _m means combining local pixel values with image-level mean values to jointly evaluate the similarity between predictions and ground truth values;

S _α represents the structural similarity between predictions for object perception and region perception;

Metric MMCI DMRA D3Net ICNet A2dele S2MA JL-DCF SSF DQSD CCAFNet DFMNet TMFNet OURS Fβ 0.852 0.886 0.9 0.891 0.873 0.889 0.912 0.896 0.9 0.91 0.913 0.882 0.925 Em 0.915 0.927 0.936 0.926 0.916 0.93 0.949 0.935 0.936 0.943 0.949 0.91 0.953 Sα 0.858 0.886 0.9 0.894 0.869 0.894 0.91 0.899 0.899 0.909 0.912 0.91 0.918 MAE 0.079 0.051 0.046 0.052 0.051 0.053 0.038 0.043 0.05 0.037 0.039 0.041 0.034 Fβ 0.815 0.88 0.897 0.908 0.88 0.902 0.915 0.896 0.898 0.908 0.912 0.867 0.910 Em 0.913 0.947 0.953 0.952 0.945 0.953 0.963 0.953 0.952 0.956 0.961 0.944 0.956 Sα 0.856 0.899 0.912 0.923 0.896 0.915 0.926 0.914 0.916 0.921 0.923 0.921 0.924 MAE 0.059 0.031 0.03 0.028 0.028 0.03 0.024 0.026 0.029 0.026 0.026 0.027 0.024 Fβ 0.767 0.898 0.793 0.85 0.892 0.901 0.878 0.924 0.827 0.913 0.747 - 0.926 Em 0.859 0.933 0.829 0.899 0.93 0.937 0.92 0.951 0.878 0.943 0.844 - 0.950 Sα 0.791 0.889 0.773 0.852 0.885 0.903 0.881 0.915 0.845 0.903 0.791 - 0.917 MAE 0.113 0.048 0.098 0.072 0.042 0.043 0.055 0.033 0.072 0.037 0.092 - 0.034 Fβ 0.863 0.857 0.891 0.898 0.879 0.882 0.898 0.89 0.886 0.887 0.904 - 0.906 Em 0.927 0.916 0.938 0.942 0.928 0.932 0.942 0.936 0.935 0.934 0.948 - 0.946 Sα 0.873 0.845 0.899 0.903 0.879 0.89 0.9 0.893 0.892 0.892 0.908 - 0.904 MAE 0.068 0.063 0.046 0.045 0.045 0.051 0.042 0.044 0.051 0.044 0.04 - 0.037 Fβ 0.771 0.856 0.81 0.871 0.835 0.835 0.839 0.866 0.847 0.832 0.866 0.846 0.872 Em 0.839 0.9 0.862 0.903 0.879 0.873 0.879 0.9 0.878 0.876 0.902 0.865 0.909 Sα 0.787 0.847 0.825 0.878 0.836 0.837 0.833 0.859 0.851 0.826 0.87 0.849 0.873 MAE 0.132 0.075 0.095 0.071 0.074 0.094 0.084 0.066 0.085 0.087 0.068 0.084 0.062 Fβ 0.822 0.888 0.885 0.913 0.867 0.935 0.917 0.883 0.927 0.937 0.932 0.892 0.944 Em 0.928 0.945 0.946 0.96 0.923 0.973 0.96 0.941 0.973 0.977 0.973 0.968 0.982 Sα 0.848 0.901 0.898 0.92 0.885 0.941 0.924 0.905 0.935 0.938 0.938 0.936 0.944 MAE 0.065 0.029 0.031 0.027 0.028 0.021 0.021 0.025 0.021 0.018 0.019 0.021 0.016

MAE represents the average pixel absolute difference between normalized predictions.

The higher F _β , E _m and S _α are, the better, and the lower MAE is, the better. It can be seen from Table 1 that the present invention has more accurate saliency segmentation capabilities for RGB-D images, which fully demonstrates the effectiveness and superiority of the method of the present invention.

The embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above-described embodiments. Various changes can be made within the knowledge scope of those of ordinary skill in the art without departing from the gist of the present invention.

The above specific embodiments further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Within the spirit and principles of the present invention, any modifications, equivalent substitutions, improvements, etc. shall be included in the protection scope of the present invention.

Claims (9)

1. The multi-modal significance target detection method based on cross-modal uncertainty region correction is characterized by comprising the following steps of:

based on a multi-modal saliency target detection model corrected by a cross-modal uncertain region, the influence of a low-quality input image on the performance of the saliency target detection model is fully considered;

the cross-modal feature enhancement module enhances effective information in shallow features by using an attention mechanism and suppresses interference information at the same time;

the cross-modal feature correction module for the uncertain region perception carries out cross-modal correction on an uncertain region existing in the two-modal deep features in a bidirectional information interaction mode in the deep single-modal feature extraction process so as to obtain single-modal features with more discrimination and improve the accuracy of the model on the detection of the saliency target of the uncertain region;

the multi-scale cross-modal feature fusion module is used for fully mining and capturing complementary information in the visible light image and the depth image and multi-scale context information of the cross-modal fusion features; the multi-scale feature extraction module with residual connection based on hole convolution is adopted to extract multi-scale features before cross-modal fusion is carried out, and then the decoding features are carried out on the coding features through a decoder;

the fusion features are used for decoding and predicting the final significance map step by step to obtain a significance prediction map;

combining cross entropy loss and a loss function based on cross-over ratio to further train the network to obtain a more complete saliency target;

and obtaining network model parameters by adopting a supervised learning model for the significance prediction graph.

2. The multi-modal salient object detection method based on cross-modal uncertainty region correction according to claim 1, wherein the cross-modal feature enhancement module enhances effective information in shallow features by using an attention mechanism and suppresses interference information, and specifically comprises:

the two modal features are first selected and enhanced in spatial dimension using a spatial attention mechanism, and their common spatial attention profile SA is first calculated _i Then, features with discrimination are enhanced according to the shared spatial attention, and the enhanced features in the spatial dimension are obtainedAnd->

A channel attention map is then generated by a channel attention mechanismAnd->Then realizing enhancement on channel dimension through channel attention, simultaneously inhibiting interference information contained in the channel dimension, and obtaining the finally enhanced single-mode RGB (red, green and blue) characteristic>And unimodal depth profile->

3. The multi-modal salient object detection method based on cross-modal uncertainty region correction according to claim 1, wherein the cross-modal feature correction module for uncertainty region perception performs cross-modal correction on an uncertainty region with two kinds of modal deep features in an information interaction mode to obtain a single-modal feature with better discrimination, and accuracy of the salient object detection of the uncertainty region is improved.

4. The multi-modal salient object detection method based on cross-modal uncertainty region correction as claimed in claim 3, comprising the steps of:

first according to RGB features R mapped to the same space ⁱ And depth feature D ⁱ Calculating the region features shared by the deep-level RGB features and the depth features, and further calculating the features of uncertain regions of two modesAnd->

Then the spatial weight of the uncertain region in the deep features of the two modes is obtained through the spatial attention operation, and then the feature of the region corresponding to the other mode is selected by using the weightAnd->Simultaneous acquisition of RGB features and features of an uncertainty region in depth features +.>And->

Then the sub-module pair is interacted by using the bi-directional cross-modal characteristicAnd->And->The cross-modal correction is carried out by respectively utilizing a bidirectional interaction mode, the cross-modal characteristic correction is carried out by cross-attention operation, and finally the corrected characteristic is obtained as follows>The way of bi-directional information interaction is utilized, thus, for +.>And-> And->And +.>And->Applying the same correction procedure to get ∈ ->And->

And finally, selecting the features after the bidirectional cross-modal feature interaction sub-module according to the feature selection sub-module.

5. The multi-modal salient object detection method based on cross-modal uncertainty area correction of claim 4, wherein the feature selection sub-module is used for selecting the features after the bidirectional cross-modal feature interaction sub-module, and the specific steps are as follows:

first, features corrected in two ways of uncertain regionsAnd->Dimension reduction is performed by using 1×1 convolution, then the dimension reduction is performed by using global average pooling operation and global maximum pooling operation to compress the dimension reduction into one-dimensional feature vectors respectively, and finally feature weights W are generated by two cascaded full-connection layers and one Sigmoid function operation respectively ⁱ According to this weight, a more accurate feature for the correction of the uncertainty region is selected>And->The corrected RGB features and depth features are finally obtained as +.>And

6. the multi-modal salient object detection method based on cross-modal uncertainty region correction according to claim 1, wherein complementary information and multi-scale context information in a visible light image and a depth image are fully mined according to the multi-scale cross-modal feature fusion module, and specifically comprising:

firstly, respectively obtaining single-mode multi-scale characteristics through Re_ASPP modules, and then communicatingThe common information of the two modes is mined through multiplication operation to supplement and enhance the original single-mode characteristics, and finally the cross-mode fusion characteristic F is obtained through splicing operation ⁱ ；

Then enhancing the cross-modal fusion characteristics in the channel dimension through a channel attention mechanism to obtain the output characteristics of the multi-scale cross-modal fusion block

7. The multi-modal significance target detection method based on cross-modal uncertainty area correction according to claim 1, wherein the step-wise fusion of the fused cross-modal features in a decoding stage to obtain a final fused feature prediction final significance map, and the obtaining of the significance prediction map specifically comprises:

obtaining a final saliency map S for the fusion features through a 1X 1 convolution layer and a Sigmoid function ^(t) (t＝1,2,3,4,5)。

8. The multi-modal salient object detection method based on cross-modal uncertainty region correction of claim 1, wherein obtaining network model parameters for the salient prediction graph by using a supervised learning model specifically comprises:

on a training data set, adopting a supervised learning model to carry out predictive significance graph, and completing algorithm network training end to obtain network model parameters:

on a training data set, a supervised learning mechanism is adopted to calculate a loss function L of a predicted result and a true value of a significance map in a network model _joint ：

L _joint (S,G)＝l _bce (S,G)+l _iou (S,G),

Wherein l _bce (:) and l _iou Cross entropy loss function and cross-ratio based loss function, respectively;

the definition of the two is respectively:

each pixel label in which G (m, n) ε {0,1} is a true value; p (m, n) e {0,1} is the probability of predicting each pixel of the saliency map; w is the width of the input image and H is the height of the input image.

9. The multi-modal salient object detection method based on cross-modal uncertainty area correction as claimed in claim 1, wherein the multi-modal image salient object detection network based on cross-modal uncertainty area correction comprises three modules: the system comprises a dual-branch feature extraction network, a cross-modal feature enhancement module, an uncertain region-aware cross-modal feature correction module and a multi-scale cross-modal fusion module.