CN116402859A - A Moving Target Detection Method Based on Aerial Image Sequence - Google Patents

Abstract

The present invention relates to the technical field of image processing, and specifically provides a moving target detection method based on an aerial image sequence, comprising the following steps: S1: inputting an aerial remote sensing image sequence; S2: preparing an image data set to be trained, and first cutting the image Obtain the image data set to be trained, and then cluster the training image data set into 9 fixed anchor frames according to the marked detection frame of the moving target in the image data set to be trained, and substitute them into the algorithm and train to obtain the model parameters; S3: the kth Input the frame image into the target detection network, input the k-th frame and the k-nth frame image into the motion detection network, fuse the feature maps obtained in the target detection network and the motion detection network to obtain the final network output . The invention inputs two adjacent frames of images into the network to directly obtain the information of the moving target, saves the background compensation step, reduces the detection process, obtains higher detection accuracy of the moving target, and realizes real-time detection.

Description

A Moving Target Detection Method Based on Aerial Image Sequence

technical field

The invention relates to the technical field of image processing, in particular to a moving target detection method based on aerial image sequences.

Background technique

Aerial remote sensing image sequences are generally obtained from moving base platforms such as drones or balloons, and the flying height of the moving platform is usually between hundreds of meters and tens of kilometers. Moving target detection is to detect the changing area from the sequence image, and extract the moving target from the background image. The extracted moving target information can provide a reference area for subsequent target recognition, tracking, behavior analysis and other tasks. Accurately and quickly finding the position of the moving target in the remote sensing image sequence is of great significance in both civilian and military fields; in the civilian field, it can be used for intelligent control, human-computer interaction, visual navigation, etc.; In the military field, it can be used for target surveillance, long-range early warning, precision guidance, etc. At present, the identification of moving targets in aerial remote sensing sequence images is often carried out by manual interpretation, resulting in low data utilization and poor intelligence timeliness. At the same time, manual interpretation is easily affected by the physical condition, mentality and subjective consciousness of the staff.

In order to avoid the influence of manual methods on the moving target detection results and reduce the consumption of human capital, efficient and accurate automatic moving target detection technology for aerial remote sensing images is particularly important. Compared with the task of moving object detection under a fixed base, the detection of moving objects in aerial image sequences mainly includes the following difficulties:

1. Motion imaging: UAV mobile monitoring, airborne imaging detection, etc. are all moving platform imaging. Moving platform imaging will make the background no longer remain unchanged in space and time, and the moving background will produce false motion. Reasonable Distinguishing between background motion and target motion is the key problem to be solved in moving target detection under moving platform imaging;

2. Complex background: There are a large number of objects with the same or similar characteristics as moving targets in aerial remote sensing images, and the shooting of aerial images is easily affected by weather such as clouds and fog. It is necessary to consider the influence of complex meteorological conditions on aerial images;

3. Dynamic background: In addition to the main moving objects that need attention, there are also a large number of disturbing moving backgrounds that exist in nature. These typical dynamic backgrounds include trees blown by the wind, water with waves, and moving white clouds. Dynamic background needs to be excluded according to application requirements;

4. Shadow movement: Shadows are usually detected as moving objects by conventional moving object detection techniques;

5. Real-time detection: In most cases of moving target detection, image data needs to be processed in time. The more robust and the better the detection effect, it is usually difficult to achieve real-time processing; on the one hand, it is necessary to seek perfect detection technology that can adapt to complex conditions On the other hand, it is necessary to improve and speed up the algorithm itself and hardware implementation.

At the same time, most of the existing traditional methods for detecting moving objects in aerial image sequences divide the task into two steps. First, the background motion generated by the moving platform is compensated for a static background or an updateable background image is obtained, and then the static background is used to The advanced moving object detection technology performs moving object detection on it. Such a detection method relies heavily on the real approximation of the motion model and the robust solution of the model parameters, so it is difficult to achieve accurate modeling of the same pixel in the time domain under the condition of the motion of the aerial remote sensing image imaging platform, and the compensation process is time-consuming. The time is huge, and the detection results need post-processing steps, which cannot meet the real-time requirements.

Existing methods based on deep learning to detect moving targets usually directly use general-purpose target detectors, which are difficult to achieve high detection accuracy and fast detection efficiency, and cannot be well applied in actual engineering projects. The network structure proposed so far is two-stage. The moving object detection task is divided into object detection and motion detection tasks, and the detection is performed separately. Usually, two extra frames of calculation are used to make difference images or optical flow images input into the network as its The motion detection branch, which then fuses the results of the object detection network with the motion detection results, has not yet been proposed involving end-to-end networks.

To sum up, how to design a moving target detection model based on deep learning, which can integrate multi-frame motion features and target features based on single-frame images at the feature level of the network, and realize end-to-end direct output of moving target information. A moving target detection method that can improve detection efficiency, reduce the detection process, and improve detection reliability is an urgent problem to be solved at present.

Contents of the invention

In order to solve the above problems, the present invention provides a moving target detection method based on aerial image sequences, which fuses the extracted motion features based on multi-frames and target features based on single-frame images at the feature level of the network, and finally end-to-end direct The output of moving target information does not require additional prior knowledge of platform motion for motion compensation, nor does it require post-processing steps, which can improve detection efficiency and detection reliability.

In order to achieve the above object, the present invention proposes the following technical solutions: a method for detecting moving objects based on aerial image sequences, comprising the steps of:

S1: input aerial remote sensing image sequence;

S2: Prepare the image data set to be trained;

S3: Input the k-th frame image into the target detection network, input the k-th frame and the k-nth frame image into the motion detection network, and fuse the feature maps obtained in the target detection network and the motion detection network to obtain the final network output.

Preferably, step S3 includes the following sub-steps:

S31: Extract motion information from the input kth and knth frame images, input the two frames of images into the motion feature enhancement module MFEM based on two-dimensional convolution, and obtain an enhanced motion feature map f _{64*3*152* 152} ;

S32: Input the obtained enhanced motion feature map into the improved three-dimensional convolution module MIE-Net, and extract feature maps S _76*76 , S _38*38 , S _19*19 containing motion information with different convolution depths;

S33: input the S _19*19 feature obtained in S32 to the non-local module to further integrate the motion information, and obtain the integrated S'_19*19;

S34: Perform target information feature extraction on the image of the kth frame, and obtain feature maps F _76*76 , F _38*38 , F _19*19 of different convolution depths;

S35: The feature maps of S _76*76 and S _38*38 obtained in S32, S' _19*19 obtained in S33, and F _76*76 , F _38*38 and F _19*19 obtained in S34 according to the corresponding size Stitching along the channel dimension;

S36: Input the three feature maps obtained in S35 into two convolution modules for decoding, normalize the size into YOLO format, and obtain the final output of the network.

Preferably, the MFEM module in S31 performs a difference between the features of the two frames of images, and uses the difference as the third dimension input to the three-dimensional convolutional motion information extraction module.

Preferably, the improved three-dimensional convolution module MIE-Net in S32 is specifically: decomposing a three-dimensional convolution with a convolution kernel of (3*3*3) into a convolution kernel of (3*1*1) and ( Two one-dimensional convolutions of 1*1*1) and a two-dimensional convolution with a convolution kernel of (1*2*2).

Preferably, in S31, when the number of aerial image data sets is small, if the weight of the target detection branch based on a single frame is large, add a random one-tenth of the moving target positive samples in the training image data set as the k-nth frame at the same time Input; that is, when the input kth frame and k-nth frame image are the same frame image, the kth frame and k-nth frame image are used as negative samples to make the motion information extraction branch obtain greater weight.

Preferably, when there are few images in the aerial dataset, an affine transformation algorithm is introduced:

Among them, (x, y) is the coordinate of the original pixel point, and (u, v) is the coordinate after affine transformation.

Preferably, in step S3, when the moving distance of the moving object between two adjacent frames is large, set n in the image of the k-nth frame to n<5; when the moving distance of the moving object between adjacent two frames is small, set n in the image of the k-nth frame is set as n≥5.

Preferably, the acquisition method of the image data set to be trained in S2 is as follows:

S21: Cutting the image to obtain an image data set to be trained;

S22: According to the marked detection frame of the moving target in the image data set to be trained, the training image data set is clustered into 9 fixed anchor frames, which are substituted into the algorithm and trained to obtain model parameters.

Preferably, the k-meaning method is used for the fixed anchor frame in S22 to cluster the moving objects in the image data set to be trained, and 9 typical values of object sizes are obtained as the fixed reference frame and substituted into the model.

Preferably, in S21, the image is cropped into a uniform input size: 608×608 pixels.

The beneficial effects of the present invention are:

1. The present invention fuses the extracted multi-frame-based motion features with the single-frame image-based target features at the feature level of the network, and can directly output information such as the category and position of the moving target end-to-end without additional prior knowledge of platform motion Knowledge for motion compensation, and no post-processing steps are required. By directly using the information in the two frames of original images, the motion information extracted by the motion information extraction network enhances the robustness of the network, obtains higher detection accuracy of moving objects, and reduces The detection process improves the detection speed and enables the network to achieve real-time detection.

2. The present invention can set an appropriate two-frame interval parameter according to the motion speed of the actual moving object, so that the network can simultaneously take into account the moving objects that move faster and slower; the present invention includes the acquisition of motion information based on the difference between two frames According to the characteristics of the time feature enhancement module, the time feature enhancement module is designed to enhance the motion information between two frames, and according to the characteristics of the motion information that can be extracted by the three-dimensional convolution, the three-dimensional convolution is improved to reduce the calculation cost, and the motion information is further extracted, so that the network A better motion information extraction effect is obtained, and the detection accuracy of the moving target is increased.

Description of drawings

FIG. 1 is a flow chart of moving object detection provided by an embodiment of the present invention.

Fig. 2 is a diagram of an application scenario provided by an embodiment of the present invention.

Fig. 3 is an overall framework diagram of the model provided by the embodiment of the present invention.

Fig. 4 is a diagram of a motion feature enhancement module provided by an embodiment of the present invention.

Fig. 5 is a network diagram of motion information extraction provided by an embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with accompanying drawings 1-5 and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, but not to limit the present invention.

A method for detecting moving targets based on aerial image sequences, comprising the steps of:

S1: Input the sequence of aerial remote sensing images; as shown in Figure 2, the aerial remote sensing images in this embodiment have the characteristics of complex background, wherein the vehicles shown in the boxes at a, b, c, and d are the moving targets to be extracted, The stationary vehicle next to it has the same target information, but it has different motion information. The moving target information can be extracted by combining the motion information and the target information. The moving target has a small number of pixels in the remote sensing image, and the image has the same characteristics as the moving target. The interference of static targets with the same target characteristics, coupled with the complex background, the camera is also moving relative to the background during the imaging period, the obtained aerial remote sensing image sequence is composed of the motion between the moving target and the static background and the mutual motion between the camera and the static background Combination of two movements.

S2: Prepare the image data set to be trained; the acquisition method of the image data set to be trained is as follows:

S21: Cutting the image to obtain an image data set to be trained, and cutting the image into a uniform input size: 608×608 pixels.

S22: According to the marked detection frame of the moving target in the image data set to be trained, the training image data set is clustered into 9 fixed anchor frames, which are substituted into the algorithm and trained to obtain model parameters; the fixed anchor frames are treated with k-meaning method for training The moving targets in the image data set are clustered, and the typical values of 9 target sizes are obtained as a fixed reference frame and substituted into the model; the proposed method is trained using the obtained data set, and a convolutional neural network model capable of detecting moving targets is obtained.

S3: Input the k-th frame image into the target detection network, input the k-th frame and the k-nth frame image into the motion detection network, and fuse the feature maps obtained in the target detection network and the motion detection network to obtain the final network output. The moving object detection network can extract the coordinate position, length, width, and category name of the moving object. In order to enhance its ability to detect moving objects, two-dimensional convolution is used to enhance the characteristics of motion information, and three-dimensional convolution is used to extract motion information. Two frames of images Extracting the motion information during the time, S3 specifically includes the following sub-steps:

S31: Extract motion information from the input kth and knth frame images, and input the two frames of images into the motion feature enhancement module MFEM (Motion feature enhancement module) based on two-dimensional convolution to obtain an enhanced motion feature map f _64*3*152*152 ; The MFEM module is shown in Figure 4. The TFEM module is inspired by the motion information contained in the two-frame difference method and is improved. In the MFEM module, the features of the two-frame images are differentiated, and the difference is used as The third dimension input by the three-dimensional convolutional motion information extraction module can further enhance its motion information and suppress background and static targets.

Due to the small number of aviation data sets in S2, small moving objects and difficult labeling, it is necessary to design a variety of data enhancement methods to improve the robustness of the algorithm;

When the number of aerial image data sets is small, if the weight of the target detection branch based on a single frame is large, add a random one-tenth of the moving target positive samples in the training image data set as the k-nth frame input at the same time; that is, when the input When the k-th frame and the k-n-th frame image are the same frame image, the k-th frame and the k-n-th frame image are used as negative samples to make the motion information extraction branch obtain a greater weight.

When there are few images in the aerial dataset, the affine transformation algorithm is introduced:

Among them, (x, y) is the coordinate of the original pixel point, and (u, v) is the coordinate after affine transformation.

When the moving distance of the moving object between adjacent two frames is large, set n in the image of the k-nth frame as n<5, such as setting n=1; when the moving distance of the moving object between adjacent two frames is small, set n in the image of the k-nth frame is set as n≥5, for example, n=5.

S32: Input the obtained enhanced motion feature map into the improved three-dimensional convolution module MIE-Net (Motioninformation extraction network), and extract feature maps S _76*76 , S _38*38 , which contain motion information with different convolution depths, S _19*19 ; The improved three-dimensional convolution module MIE-Net is specifically: a three-dimensional convolution with a convolution kernel of (3*3*3) is decomposed into a convolution kernel of (3*1*1) and ( Two one-dimensional convolutions of 1*1*1) and a two-dimensional convolution with a convolution kernel of (1*2*2).

S33: input the S _19*19 feature obtained in S32 to the non-local module to further integrate the motion information, and obtain the integrated S'_19*19;

S34: Using the YOLOv5s algorithm as the benchmark algorithm of the target detection branch, perform target information feature extraction on the image of the kth frame, and obtain feature maps F _76*76 , F _38*38 , F _19*19 of different convolution depths;

S35: The feature maps of S _76*76 and S _38*38 obtained in S32, S' _19*19 obtained in S33, and F _76*76 , F _38*38 and F _19*19 obtained in S34 according to the corresponding size Concatenation along the channel dimension;

S36: Input the three feature maps obtained in S35 into two convolution modules for decoding, normalize the size into YOLO format, and obtain the final output of the network.

Although the embodiments of the present invention have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limitations on the present invention. Those skilled in the art can make changes, modifications, substitutions and modifications to the above-mentioned embodiments within the scope of the present invention.

The above specific implementation manners of the present invention do not constitute a limitation to the protection scope of the present invention. Any other corresponding changes and modifications made according to the technical concept of the present invention shall be included in the protection scope of the claims of the present invention.

Claims (10)

1. The moving target detection method based on the aerial image sequence is characterized by comprising the following steps of:

s1: inputting an aerial remote sensing image sequence;

s2: preparing an image data set to be trained;

s3: inputting the kth frame image into a target detection network, inputting the kth frame and the kth-n frame images into a motion detection network, and carrying out information fusion on feature images obtained in the target detection network and the motion detection network to obtain final network output.

2. The method for detecting a moving object based on an aerial image sequence according to claim 1, wherein the step S3 comprises the following sub-steps:

s31: extracting motion information of the input k frame and k-n frame images, and inputting two frame images to a base station based on two framesIn the motion feature enhancement module MFEM of the dimensional convolution, an enhanced motion feature graph f is obtained _64*3*152*152 ；

S32: inputting the obtained enhanced motion feature map into an improved three-dimensional convolution module MIE-Net, and extracting feature maps S containing motion information with different convolution depths _76*76 ，S _38*38 ，S _19*19 ；

S33: s obtained in S32 _19*19 The characteristics are input into a non-local module to further integrate the motion information to obtain integrated S' _19*19 ；

S34: extracting target information features of the kth frame image to obtain feature images F with different convolution depths _76*76 ，F _38*38 ，F _19*19 ；

S35: s is obtained in S32 _76*76 And S is _38*38 S 'obtained in S33' _19*19 F obtained in S34 _76*76 、F _38*38 And F _19*19 Splicing the feature graphs of the channel along the dimension of the channel according to the corresponding dimension;

s36: and (3) inputting the three feature maps obtained in the step (S35) into two convolution modules for decoding, normalizing the size into a YOLO format, and obtaining the final output of the network.

3. The method for detecting a moving object based on an aerial image sequence according to claim 2, wherein the MFEM module in S31 makes a difference between features of two frames of images, and uses the difference as a third dimension input by the three-dimensional convolution motion information extraction module.

4. The method for detecting a moving object based on an aerial image sequence according to claim 2, wherein the improved three-dimensional convolution module MIE-Net in S32 is specifically: a three-dimensional convolution with a convolution kernel of (3 x 3) is decomposed into two one-dimensional convolutions with a convolution kernel of (3 x 1) and (1 x 1) and a two-dimensional convolution with a convolution kernel of (1 x 2).

5. The method for detecting a moving object based on an aerial image sequence according to claim 4, wherein when the number of aerial image data sets is small in S31, if the weight of the object detection branch based on a single frame is large, a random tenth of positive samples of the moving object in the training image data set is added and is input as the k-n frame; that is, when the input k-th frame and k-n-th frame images are the same frame image, the motion information extraction branch is given a greater weight by taking the k-th frame and k-n-th frame images as negative samples.

6. The moving object detection method based on an aerial image sequence according to any one of claims 2 to 5, wherein when aerial dataset images are small, an affine transformation algorithm is introduced:

where (x, y) is the original pixel point coordinates and (u, v) is the coordinates after affine transformation.

7. The method for detecting a moving object based on an aerial image sequence according to claim 6, wherein n in the k-n frame image is set to n < 5 when the moving distance of the moving object between two adjacent frames is large in step S3; when the motion distance of the moving object between two adjacent frames is small, n in the k-n frame image is set as n is more than or equal to 5.

8. The method for detecting a moving object based on an aerial image sequence according to claim 7, wherein the method for acquiring the image dataset to be trained in S2 is as follows:

s21: cutting the image to obtain an image data set to be trained;

s22: and clustering the training image data set into 9 anchor frames according to the detection frames marked by the moving targets in the image data set to be trained, substituting the anchor frames into an algorithm, and training to obtain model parameters.

9. The moving object detection method based on the aerial image sequence according to claim 8, wherein the fixed anchor frame in S22 clusters moving objects in the image data set to be trained by adopting a k-means method, and typical values of 9 object sizes are obtained and substituted into the model as fixed reference frames.

10. The moving object detection method based on aerial image sequence of claim 9, wherein the image is cut into uniform input sizes in S21: 608 x 608 pixels.

Images