CN116469020A - Unmanned aerial vehicle image target detection method based on multiscale and Gaussian Wasserstein distance - Google Patents

Abstract

The invention discloses a UAV image target detection method based on multi-scale and Gaussian Wasserstein distance, which relates to the technical field of aerial image processing, combining low-level and high-level feature fusion and scale insensitivity measurement ideas, including the following steps: S1: Establishing a UAV image target data set, and preprocessing the image data; S2: Slicing the input image, and then splicing the sliced results; S3: Fusing the receptive field of multi-scale pooling information-rich features; S4: Introducing NWD measurement based on Gaussian Wasserstein distance; S5: For UAV images containing small targets in the test set, the trained improved feature extraction network is used for target prediction. The present invention adopts the above method to improve the detection accuracy of small targets, improve the depth detection algorithm for conventional scale targets, realize effective detection for targets with limited pixels, and have high accuracy and recall rate.

Description

A UAV image target based on multi-scale and Gaussian Wasserstein distance Detection method

technical field

The invention relates to the technical field of aerial image processing, in particular to a method for detecting an image target of an unmanned aerial vehicle based on multi-scale and Gaussian Wasserstein distance.

Background technique

The finite pixel target in the UAV image refers to the target that occupies few pixels in the UAV image. Under long-distance imaging conditions, especially when medium and high-altitude UAVs observe the earth with a long-distance squint, the number of pixels occupied by ground targets in the image is relatively small. Using computers to effectively analyze and process UAV image data, identifying different types of targets and marking their locations is one of the basic problems in computer vision tasks. It is widely used in various fields such as military, agriculture, forestry, maritime affairs, disaster prevention and relief, and urban planning. This also puts forward higher requirements for target detection tasks in UAV images.

Detecting small targets in complex backgrounds is an important research direction in the field of image analysis and processing. Compared with images in natural scenes, due to the longer imaging distance, UAV images have the characteristics of high background complexity, small target size, and weak features. Due to the complex imaging environment, such as weather, platform speed, height, and stability, the UAV images have problems such as low resolution, low color saturation, and environmental noise distortion, which increases the difficulty of target detection.

Existing target detection algorithms are divided into two categories based on traditional image processing and algorithms based on deep learning. Target detection methods based on traditional image processing are mostly used in the field of infrared weak and small target detection. By introducing a visual attention mechanism, the difference between the target, the background and the noise is used to selectively find the target area of interest. However, the hand-designed features have the disadvantage of insufficient representation and are easily interfered by complex backgrounds, so they cannot be directly applied to the task of UAV image target detection. The target detection algorithm based on the deep neural network performs well on conventional data sets, but the detection accuracy of small targets is low. This is because the convolutional neural network is generally composed of stacked convolution and pooling layers. As the network level deepens, the feature dimension gradually decreases, and the amount of information of the target to be tested is further reduced, making it difficult to detect.

Therefore, it is necessary to provide a UAV image target detection method based on multi-scale and Gaussian Wasserstein distance to solve the above problems.

Contents of the invention

The purpose of the present invention is to provide a method for detecting objects in UAV images based on multi-scale and Gaussian Wasserstein distance, which improves the detection accuracy of small objects, improves the depth detection algorithm for conventional-scale objects, and realizes effective detection for objects with limited pixels, with high accuracy and recall rate.

In order to achieve the above object, the present invention provides a method for detecting objects in UAV images based on multi-scale and Gauss Wasserstein distance, comprising the following steps:

S1: Establish a UAV image target data set, and preprocess the image data;

S2: Slicing the input image, and then splicing the sliced results;

S3: Integrating the receptive field of multi-scale pooling information-rich features;

S4: Introducing the NWD metric based on the Gaussian Wasserstein distance;

S5: For the UAV images containing small targets in the test set, use the trained and improved feature extraction network for target prediction.

Preferably, in step S1, the original image is overlapped and cut into a uniform size of 800×800 pixels, the target is determined according to the frequency and size of the target in the image, and the image is selected according to the proportion of the target in the image, and samples of X categories are taken as a training set, and samples of other categories are used as a test set.

Preferably, in step S2, the slicing operation is to set the Focus structure, perform down-sampling, split the high-resolution image into multiple low-resolution images, and retain the feature information of the small target.

Preferably, in step S3, an SPP module is introduced before the last convolutional layer of the backbone network to fuse feature information of different scales.

Preferably, in step S4, the process of NWD metric design is to model the bounding box as a two-dimensional Gaussian distribution, and for the horizontal bounding box, its inscribed ellipse equation is expressed as:

Where (μ _x , μ _y ) is the coordinates of the center of the ellipse, σ _x and σ _y represent the semi-axis lengths along the x and y axes respectively, μ _x =c _x , μ _y =cy , _{σ x =} w/2, σ _y =h/2.

Preferably, in step S4, the probability density function of the two-dimensional Gaussian distribution is expressed as:

where x, μ, and Σ denote the coordinates, mean vector, and covariance matrix of the Gaussian distribution, respectively.

Preferably, in step S4, when (x-μ) ^T Σ ^-1 (x-μ)=1, the horizontal bounding box R=(c _x , _cy ,w,h) is modeled as a two-dimensional Gaussian distribution N(μ,Σ), where:

Convert the similarity between two bounding boxes into the distance between two Gaussian distributions, for two two-dimensional Gaussian distributions μ ₁ = N(m ₁ ,Σ ₁ ) and μ ₂ =N(m ₂ ,Σ ₂ ), the second-order Wasserstein distance between μ ₁ and μ ₂ is simplified as:

Where ||·|| _F represents the Frobenius norm;

For Gaussian distributions N _a and N _b modeled from bounding boxes A = (cx _a , cy _a , w _a , h _a ) and B = (cx _b , cy _b , w _b , h _b ), this further simplifies to:

Normalize using its exponential form as a measure of similarity between two bounding boxes:

Among them, C is the average absolute size of the target in the data set, and the IoU curve decreases as the target size decreases, and the index decline caused by position offset increases.

Preferably, in step S4, the loss function is composed of target confidence loss, classification loss and bounding box regression loss weighting, the target confidence loss and classification loss adopt binary cross entropy, the bounding box regression loss is expressed as the normalized weighted sum of the CIoU loss and NWD loss of the predicted bounding box and the real target bounding box, and the loss function is expressed as:

Loss＝λ ₁ L _cls +λ ₂ L _obj +λ ₃ [αL _CIoU +(1-α)L _NWD ]

L _NWD ＝1-NWD(N _p , N _g )

where NWD(N _p , N _g ) represents the exponentially normalized Wasserstein distance between the predicted box and the ground truth box.

Preferably, in step S5, AP50, AP75 and mAP are used as evaluation indicators of the model to evaluate the performance of the algorithm, to test the effect of the improved feature extraction network on the test data set, and to analyze the impact of introducing the NWD metric on the model performance.

Therefore, the present invention adopts the above-mentioned multi-scale and Gaussian Wasserstein distance-based UAV image target detection method, which has the following beneficial effects;

(1) The present invention uses a bidirectional feature pyramid network (BiFPN) in the Neck network through a multi-scale feature extraction module to bidirectionally fuse low-level features with high-level features to enrich the expression of limited pixel target feature information.

(2) The present invention improves the recall rate of detection by fusing spatiotemporal information of multiple frames of images.

(3) The present invention makes the detection result reliable by extracting and combining various image visual features.

(4) The present invention uses a scale-insensitive normalized Gaussian Wasserstein distance metric in the non-maximum suppression stage and bounding box regression loss to evaluate the similarity between the predicted box and the real box, thereby improving the detection accuracy of small objects.

The technical solutions of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

Description of drawings

Fig. 1 is a kind of flow chart of the UAV image target detection method based on multi-scale and Gaussian Wasserstein distance of the present invention;

Fig. 2 is the Focus structure schematic diagram that the present invention adopts;

Fig. 3 is the SPP module structural diagram that the present invention adopts;

Fig. 4 is the schematic diagram of the position offset curve under the NWD metric index based on the Gaussian Wasserstein distance adopted by the present invention;

Detailed ways

The technical solutions of the present invention will be further described below through the accompanying drawings and embodiments.

Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the usual meanings understood by those skilled in the art to which the present invention belongs. "First", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Comprising" or "comprising" and similar words mean that the elements or items appearing before the word include the elements or items listed after the word and their equivalents, without excluding other elements or items. Words such as "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "Down", "Left", "Right" and so on are only used to indicate the relative positional relationship. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

The present invention adopts the above-mentioned UAV image target detection method based on multi-scale and Gaussian Wasserstein distance, combines low-level and high-level feature fusion and scale insensitivity measurement ideas, and includes the following steps: S1: Establish a UAV image target data set, and preprocess the image data; S2: Slicing the input image, and then splicing the sliced results; S3: Fuse the receptive field of multi-scale pooling information-rich features; S4: Introduce NWD measurement based on Gaussian Wasserstein distance; UAV imagery, using a trained improved feature extraction network for object prediction.

In step S1, the original image is overlapped and cut into a uniform size of 800×800 pixels, the target is determined according to the frequency and size of the target in the image, and the image is selected according to the proportion of the target in the image, and the samples of X categories are taken as the training set, and the samples of the other categories are used as the test set.

Based on AI-TOD data sets collected from multiple large-scale public aerial remote sensing image data sets such as DIOR, DOTA, xView, and VisDrone, a small target data set with limited pixels in UAV images is composed. According to the frequency and size of the target in the image, the target categories are mainly aircraft, ships, cars, and people.

The original image is overlapped and cut into a uniform size of 800×800 pixels. According to the proportion of the target in the image, images with a size larger than 64 pixels are selected. The dataset contains 28,036 images and 700,621 target instances. The average size of the target is 12.8 pixels, and the variance is 5.9 pixels.

In step S2, the slicing operation is to set the Focus structure, perform down-sampling, split the high-resolution image into multiple low-resolution images, and retain the feature information of the small target. Focus is a special downsampling method. The specific processing operation is shown in Figure 2. Values are taken at intervals of one pixel and merged into a low-resolution image. The corresponding number of channels becomes 4 times that of the original. By splitting the high-resolution image into multiple low-resolution images, the width and height information is unified on the channel dimension, which reduces the amount of calculation and avoids the information loss caused by downsampling. It retains more feature information of small objects and is conducive to improving network training and inference speed.

In step S3, an SPP module is introduced before the last convolutional layer of the backbone network to fuse feature information of different scales. The structure of the SSP module is shown in Figure 3. The input features first pass through a 1×1 convolutional layer, and then pass through the maximum pooling windows of three different scales of 5×5, 7×7, and 13×13 respectively. The pooled features of the three scales are connected to the input features, and then pass through a 1×1 convolutional layer to finally obtain a feature vector of a fixed size. The SPP layer enhances the feature expression ability of the feature map by fusing the receptive field of multi-scale pooling information to enrich the features.

In step S4, IoU represents the degree of overlap between the predicted frame and the real frame, which is widely used in the target detection framework based on the anchor point frame. For example, the non-maximum suppression (Non-Maximum Suppression, NMS) stage uses the IoU index to filter out the prediction frame with a high overlap rate. Usually, the IoU-based index is used in the loss function instead of the L2 loss as the bounding box regression loss. However, the IoU-based evaluation index is very sensitive to small target position deviations. Box detector performance. The similarity between two bounding boxes is calculated using the normalized Gaussian Wasserstein distance. The process of NWD metric design is to model the bounding box as a two-dimensional Gaussian distribution. For a horizontal bounding box, its inscribed ellipse equation is expressed as:

Where (μ _x , μ _y ) is the coordinates of the center of the ellipse, σ _x and σ _y represent the semi-axis lengths along the x and y axes respectively, μ _x =c _x , μ _y =cy , _{σ x =} w/2, σ _y =h/2.

In step S4, the probability density function of the two-dimensional Gaussian distribution is expressed as:

where x, μ, and Σ denote the coordinates, mean vector, and covariance matrix of the Gaussian distribution, respectively.

In step S4, when (x-μ) ^T Σ ^-1 (x-μ)=1, the horizontal bounding box R=(c _x , _cy ,w,h) can be modeled as a two-dimensional Gaussian distribution N(μ,Σ), where:

Convert the similarity between two bounding boxes into the distance between two Gaussian distributions. For two two-dimensional Gaussian distributions μ ₁ = N(m ₁ ,∑ ₁ ) and μ ₂ =N(m ₂ ,∑ ₂ ), the second-order Wasserstein distance between μ ₁ and μ ₂ is simplified as:

where ||·|| _F represents the Frobenius norm.

For Gaussian distributions N _a and N _b modeled from bounding boxes A = (cx _a , cy _a , w _a , h _a ) and B = (cx _b , cy _b , w _b , h _b ), this further simplifies to:

Normalize using its exponential form as a measure of similarity between two bounding boxes:

Among them, C is the average absolute size of the target in the data set, and the IoU curve decreases as the target size decreases, and the index decline caused by position offset increases. As shown in Figure 4, the four curves corresponding to NWD are exactly the same and are not sensitive to the scale change of the bounding box; the NWD curve is smoother and less sensitive to offset, and when the bounding box A contains the bounding box B or the two bounding boxes do not intersect, the NWD index can still reflect the similarity of the two bounding boxes and has stronger robustness.

In step S4, the loss function is composed of target confidence loss, classification loss and bounding box regression loss weighted. The target confidence loss and classification loss adopt binary cross entropy, and the bounding box regression loss is expressed as the normalized weighted sum of the CIoU loss and NWD loss of the predicted bounding box and the real target bounding box. The loss function is expressed as:

Loss＝λ ₁ L _cls +λ ₂ L _obj +λ ₃ [αL _CIoU +(1-α)L _NWD ]

L _NWD ＝1-NWD(N _p , N _g )

where NWD(N _p , N _g ) represents the exponentially normalized Wasserstein distance between the predicted box and the ground truth box.

In step S5, AP50, AP75 and mAP are used as the evaluation indicators of the model to evaluate the performance of the algorithm, test the effect of the improved feature extraction network on the test data set, and analyze the impact of introducing NWD metrics on model performance.

AP50, AP75 and mAP are used as the evaluation indicators of the model to evaluate the performance of the algorithm. The average precision (AP) under each category is the area under the P-R curve, and mAP represents the average value of the average precision (mAP) of each category; mAP in the COCO dataset means that ten mAPs are calculated and averaged at intervals of 0.05 between the IoU thresholds of 0.5 and 0.95, and AP50 and mAP75 represent 0.5 and 0.7, respectively. 5 is the average precision of each category calculated as the IoU threshold.

The algorithm is implemented on the deep learning framework PyTorch, and the hardware configuration is CPU: Intel Xeon 24 cores, 1.9GHz, 64GB RAM; GPU: GeForce RTX 3080Ti. The YOLOv5 official pre-training model is used for parameter initialization, and fine-tuning is performed on the remote sensing image target detection dataset. The initial learning rate is set to 0.01, and the Warmup strategy is used before training, and then the learning rate is dynamically attenuated through the cosine annealing algorithm. Each model is trained for 1000 cycles. In order to prevent overfitting, when the indicators on the verification set do not improve within 100 cycles, it will end early. The batch_size is set to 128 during training, and the batch_size is set to 1 during testing.

Multi-scale training is adopted, and the K-means algorithm is used to automatically cluster to generate new optimal anchor box sizes according to the real bounding box labels marked in the dataset to adapt to targets of different scales in different datasets.

Test the effect of the improved feature extraction network on the test data set, use the BiFPN structure to connect the multi-layer feature maps in two directions, and dynamically weight and fuse them according to the importance of the features, thereby improving the network feature expression ability, and adding a new detection head for the target. The experimental results are shown in Table 1. On the AI-TOD dataset, the mAP value increased by 0.3%, APm increased by 2.2%, and AP75 increased by 1.0%.

Table 1 Performance comparison of improved network structure

Analyze the impact of introducing the NWD metric on the performance of the model. In the NMS stage, the NWD metric that is not sensitive to the target scale is used instead of IoU, which can effectively avoid the increase of redundant detection frames caused by the IoU of the prediction frame and the highest score prediction frame being less than the threshold, resulting in an excessive false positive rate. For the bounding box regression loss function, the introduction of NWD loss can help alleviate the sensitivity of CIoU loss to small target position deviations, so that the network can better learn and optimize for small targets. The experimental results are shown in Table 2.

Table 2 Influence of introducing NWD metric on detection performance

The NWD metric is introduced in the NMS stage, and the mAP is 16.2%, which is 1.2% higher than the IoU metric mAP value adopted by YOLOv5. The NWD loss complements the CIoU loss in a normalized weighted manner. When the NWD loss weight is set to 0.35, the mAP is increased by 1.6% compared with the CIoU loss alone. The experimental results show that the introduction of NWD metrics in the NMS stage and the bounding box regression loss can improve the performance of small target detection to a certain extent.

The performance of the AI-TOD dataset is compared with other classic and advanced target detection methods, and the official COCOAPI-AITOD interface is used to ensure the objectivity and credibility of the model performance comparison. The comparison results are shown in Table 3.

Table 3(1) Performance comparison of different algorithms on the AI-TOD dataset

Table 3(2) Performance comparison of different algorithms on the AI-TOD dataset

The multi-category average precision mAP value of this method reaches 17.8%, and the AP ₅₀ and AP ₇₅ are 41.4% and 12.4%, respectively.与基线方法YOLOv5相比，mAP提升了3.0％，AP ₅₀提升了4.6％，AP ₇₅提升3.3％，且三个指标均高于经典的基于锚点框和非锚点框的目标检测算法；与Faster-RCNN、CascadeR-CNN等经典的多阶段目标检测方法下相比，单阶段方法，例如YOLOv3、SSD及RetinaNet的mAP值较低，对小目标的检测性能较差；无锚检测器CenterNet避免了离散尺寸的锚点框对多尺度目标鲁棒性较差的问题，基于多中心点的无锚检测器通过多中心点和偏置目标设计进一步提升检测性能极小目标检测，AP _vt指标最高，达到6.1％，本文提出的方法兼顾了整个数据集的所有尺度目标实例，整体上具有突出的性能优势。 Compared with the advanced DetectoRS algorithm, the AP _t index is increased by 4.9%, and the AP _vt is increased by 3.4%, which has a significant performance improvement in the detection of extremely small targets. The results of comparative experiments show that the method proposed in this paper has better performance than some current methods in the small target detection task of remote sensing images, thus proving the effectiveness of the method.

Therefore, the present invention adopts the above-mentioned UAV image target detection method based on multi-scale and Gaussian Wasserstein distance, and combines low-level and high-level feature fusion and scale insensitivity measurement ideas to improve the accuracy of limited-pixel small target detection in UAV images.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention rather than limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that: it can still modify or equivalently replace the technical solution of the present invention, and these modifications or equivalent replacement cannot make the modified technical solution deviate from the spirit and scope of the technical solution of the present invention.

Claims (9)

1. a kind of unmanned aerial vehicle image object detection method based on multiscale and Gaussian Wasserstein distance, it is characterized in that: comprise the following steps: S1: Establish a UAV image target data set, and preprocess the image data; S2: Slicing the input image, and then splicing the sliced results; S3: Integrating the receptive field of multi-scale pooling information-rich features; S4: Introducing the NWD metric based on the Gaussian Wasserstein distance; S5: For the UAV images containing small targets in the test set, use the trained and improved feature extraction network for target prediction. 2. according to a kind of unmanned aerial vehicle image object detection method based on multi-scale and Gaussian Wasserstein distance described in claim 1, it is characterized in that: in step S1, original image is overlapped and cut into the uniform size of 800 * 800 pixels, determines the target according to the frequency and the size that target occurs in the image, and selects image according to the proportion of target in the image, gets wherein the sample of X category as training set, all the other category samples as test set. 3. according to a kind of unmanned aerial vehicle image target detection method based on multi-scale and Gaussian Wasserstein distance described in claim 1, it is characterized in that: in step S2, slice operation is to set Focus structure, carry out down-sampling, high-resolution image is split into a plurality of low-resolution images, retains the feature information of small target. 4. according to a kind of unmanned aerial vehicle image object detection method based on multi-scale and Gaussian Wasserstein distance described in claim 1, it is characterized in that: in step S3, before the last convolutional layer of backbone network, introduce SPP module, the feature information of different scales is fused. 5. according to a kind of UAV image target detection method based on multi-scale and Gaussian Wasserstein distance described in claim 1, it is characterized in that: in step S4, the process of NWD measurement design is: The bounding box is modeled as a two-dimensional Gaussian distribution, and for a horizontal bounding box, its inscribed ellipse equation is expressed as: Where (μ _x , μ _y ) is the coordinates of the center of the ellipse, σ _x and σ _y represent the semi-axis lengths along the x and y axes respectively, μ _x =c _x , μ _y =cy , _{σ x =} w/2, σ _y =h/2. 6. according to a kind of UAV image target detection method based on multi-scale and Gaussian Wasserstein distance described in claim 5, it is characterized in that: in step S4, the probability density function of two-dimensional Gaussian distribution is expressed as: where x, μ, and Σ denote the coordinates, mean vector, and covariance matrix of the Gaussian distribution, respectively. 7. according to a kind of UAV image object detection method based on multi-scale and Gaussian Wasserstein distance described in claim 6, it is characterized in that: in step S4, (x-μ) ^T Σ ^-1 (x-μ)=1, horizontal bounding box R=(c _x , _cy , w, h) is modeled as two-dimensional Gaussian distribution N (μ, Σ), wherein: Convert the similarity between two bounding boxes into the distance between two Gaussian distributions, for two two-dimensional Gaussian distributions μ ₁ = N(m ₁ ,Σ ₁ ) and μ ₂ =N(m ₂ ,Σ ₂ ), the second-order Wasserstein distance between μ ₁ and μ ₂ is simplified as: Where ||·|| _F represents the Frobenius norm; For Gaussian distributions N _a and N _b modeled from bounding boxes A = (cx _a , cy _a , w _a , h _a ) and B = (cx _b , cy _b , w _b , h _b ), this further simplifies to: Normalize using its exponential form as a measure of similarity between two bounding boxes: where C is the average absolute size of objects in the dataset. 8. according to a kind of UAV image target detection method based on multi-scale and Gaussian Wasserstein distance described in claim 7, it is characterized in that: in step S4, loss function is made up of target confidence degree loss, classification loss and bounding box regression loss weighting, target confidence degree loss and classification loss adopt binary cross entropy, bounding box regression loss is expressed as the normalized weighted sum of CIoU loss and NWD loss of predicted bounding box and real target bounding box, and loss function is expressed as: Loss＝λ ₁ L _cls +λ ₂ L _obj +λ ₃ [αL _CIoU +(1-α)L _NWD ] L _NWD ＝1-NWD(N _p , N _g ) where NWD(N _p , N _g ) represents the exponentially normalized Wasserstein distance between the predicted box and the ground truth box. 9. according to a kind of unmanned aerial vehicle image object detection method based on multiscale and Gaussian Wasserstein distance described in claim 1, it is characterized in that: in step S5, adopt AP50, AP75 and mAP as the evaluation index evaluation algorithm performance of model, the effect of the feature extraction network of test improvement on test data set, analysis introduces the impact of NWD measurement on model performance.