CN114821441A - Deep learning-based airport scene moving target identification method combined with ADS-B information - Google Patents

Abstract

The invention discloses an airport scene moving target identification method based on deep learning combined with ADS-B information, which combines the thought of change detection motion and stillness judgment and a feature extraction, classification and regression module in target detection, and improves a background modeling method based on a time histogram in a change detection algorithm to obtain a more accurate background image; the existing ADS-B technology of the aircraft is adopted for auxiliary training, the feature extraction capability is enhanced, and the detection result is improved. The ideas and advantages of the change detection algorithm and the target detection algorithm are integrated.

Description

Recognition method of moving target in airport scene based on deep learning combined with ADS-B information

technical field

The invention relates to the field of airport target recognition, in particular to a deep learning-based airport scene moving target recognition method combined with ADS-B information.

Background technique

In recent years, the global civil aviation passenger capacity has continued to increase, the airport structure has become increasingly complex, and the airport scene has become more and more crowded. It has gradually become more difficult for administrators to visually monitor moving targets on the ground in the field, and the safety hazards of manual command and management have become more and more obvious. In order to meet the needs of automated surface surveillance, a surveillance system covering the entire airport surface area is applied to modern airport surface surveillance through multiple cameras. Compared with other sensors, computer vision technology has unique advantages because the way of disseminating information in video is more intuitive and realistic, and the human eye can intuitively capture the key information displayed in the video. Therefore, the intelligent application of airports based on computer vision technology has gradually become the mainstream of automated scene surveillance technology. The amount of video data obtained based on computer vision technology is very rich. On this basis, intelligent applications such as illegal intrusion reminders and collision warnings can be developed. As the basis of many video-based intelligent applications, change detection and object detection algorithms have made great progress in recent years, but there are certain limitations in airport scene surveillance applications.

The purpose of the change detection algorithm is to present and mark the foreground objects whose spatial position changes in the image sequence or video. The traditional change detection algorithm mainly uses the background subtraction technique to estimate the initial clean background image by using a small number of input video frames. Pixel segmentation is then performed between the estimated background and the input video frame, and the background model is continuously updated. In method ViBe, three background model update strategies are proposed: random background sample replacement to represent short-term and long-term history, memoryless update strategy and spatial diffusion strategy propagated through background samples, which have been detected by the latest state-of-the-art change detection The technology is widely adopted, but the traditional method is unsupervised, and the performance of the algorithm depends on the quality of the established background model. In the airport scene, the background frequently changes due to factors such as lighting and shadows, which affects the detection results. In recent years, a large number of supervised change detection techniques based on Convolutional Neural Networks (CNN) have been proposed. The change detection algorithm based on the convolutional neural network encodes the multi-scale features of the input image, and then uses the transposed convolutional neural network to decode, realizes the probability mapping of the features to the foreground pixels, and determines whether each pixel belongs to the foreground or the background one by one. Although the accuracy of the change detection algorithm based on convolutional neural network has been greatly improved compared with traditional methods, and achieved good accuracy in AGVS (airport scene surveillance data set), these methods are relatively complex and slow. It is difficult to play a role in the real-time monitoring of airport scene video.

As a classic task in computer vision, object detection has always been a research hotspot in this field. The goal of object detection is to find objects with different geometries and assign an accurate label to each detected object. This is a challenging task because objects can appear in natural images in any scale, shape, and position, and the appearance of the same class of objects can be very different. Traditional target detection algorithms usually use sliding windows to extract features for each window through feature extraction algorithms such as SIFT and HOG, and then use machine learning algorithms, such as support vector machines, to classify the extracted features, and finally get the window. Whether to contain a certain class of objects. This leads to the shortcomings of traditional target detection algorithms that have a relatively large amount of calculation, slow operation speed, and often produce multiple correct identification results; driven by the rapid development of deep learning, researchers have designed more effective target detection algorithms. The detection algorithm can be divided into two-stage and one-stage methods. The two-stage target detection algorithm first generates candidate regions that may contain objects, and then further classifies and calibrates the candidate regions to obtain the final detection result. In the first stage, there is no step of generating candidate regions, and the results are directly generated. Therefore, the former works slower, but has higher detection accuracy. Compared with the change detection algorithm of pixel-level segmentation, target detection only needs to find the area of the current target, and the computational cost is greatly reduced. However, in the specific scene of airport scene surveillance, it often occurs that most of the aircraft are still on the tarmac, and only a small number of aircraft are moving. Among them, what we are interested in is only the moving target, and the target detection algorithm will detect all the targets in the picture, which will not only cause redundancy of information, but also the display results of the algorithm are not intuitive enough.

The airport scene is different from other scenes. Most of the civil aviation aircraft are equipped with the ADS-B system, which is an automatic dependent surveillance system of broadcast type. The aircraft equipped with this system can broadcast some of its own information through the data link, which includes the position of the aircraft (four-dimensional coordinates, accuracy, dimension, altitude, time), the flight speed of the aircraft and the call sign (the identity information of the aircraft). ), and some additional information. At present, there have been some works that can map to two-dimensional image coordinates by mapping methods based on the four-dimensional image information and camera position in ADS-B. The ADS-B signal is the unique information of the moving aircraft target. Therefore, with the help of the ADS-B information, a brand-new airport surface moving target recognition strategy can be proposed to meet the special and real-time requirements of airport surface surveillance. Airport intelligent application of computer vision technology.

SUMMARY OF THE INVENTION

In view of the above deficiencies in the prior art, the present invention provides a method for recognizing moving objects in airport scenes based on deep learning combined with ADS-B information.

In order to achieve the above-mentioned purpose of the invention, the technical scheme adopted in the present invention is:

A deep learning-based airport scene moving target recognition method combined with ADS-B information, comprising the following steps:

S1. Decode the ADS-B information, and use the three-dimensional data returned by the aircraft and the position of the camera to correspond the ADS-B data to the two-dimensional image;

S2, label the scene surveillance video sequences in the airport scene surveillance data set to obtain the real bounding box, and simultaneously select the first ten video sequences as the training set, and the last ten video sequences as the test set;

S3, constructing a moving target recognition network, and using the constructed moving target recognition network to output the estimated background image;

S4, using the estimated background image obtained in step S3 and the current frame image as the moving target recognition network to calculate the category of the target candidate frame and the boundary information of the target candidate frame;

S5. Compare the target candidate frame category obtained in S4 and the boundary information of the target candidate frame with the real bounding box marked in S2, and use the cross-entropy loss function and the smooth _L1 distance to calculate the classification loss function and the regression loss function. The sum of the loss functions is used as the loss function of the motion estimation network;

S6 , back-propagating the loss function of the moving target recognition network to obtain a trained motion estimation network, and using the trained motion estimation network to perform moving target recognition on the airport scene.

Further, the moving target recognition network constructed in the described S3 comprises a cascaded motion judgment module and a target detection module, wherein, the input of the moving target recognition network is the position information and the label information after the current frame, the current frame ADS-B decoding , The first 30 frames of images in the history of the current frame.

Further, the calculation method of the estimated background image in the S3 is:

Input the recent historical frame image into the motion judgment module, and continuously pass through multiple multi-scale features, each feature captures the maximum response of the receptive field of size 1 × 1, 3 × 3 and 5 × 5, and obtains the estimated background image. .

Further, the target detection module includes a cascaded backbone network, a feature pyramid network, and a regression and classification network, wherein the backbone network is a ResNet network for calculating the convolution features of the entire input image; The lower paths and lateral connections enhance the object detection module; both regression and classification networks are multi-scale convolutional neural networks.

Further, the classification loss function and the regression loss function in S5 are respectively expressed as:

为与维度向量相同的地面实况标签，在训练阶段为地面实况的偏移量；p_i为锚点是目标的概率；

为分类损失函数；

为回归损失函数。Among them, t _i ={t _x , t _y , t _w , th} represents the coordinates of the anchor point, which is the predicted offset in the training phase;

is the ground truth label that is the same as the dimension vector, and is the offset of the ground truth in the training phase; _pi is the probability that the anchor point is the target;

is the classification loss function;

is the regression loss function.

The present invention has the following beneficial effects:

1. This method solves the problem that the target detection algorithm cannot distinguish the moving and static targets, and is more suitable for the airport scene surveillance scene.

2. The multi-scale receptive feature block cascade is used as the background modeling method, which can more accurately identify the moving target and obtain the accurate background image.

3. The aircraft-specific ADS-B technology is adopted to improve the detection accuracy of the aircraft and provide a more effective surface surveillance scheme for the airport.

Description of drawings

FIG. 1 is a schematic flowchart of a method for recognizing moving objects in an airport scene based on deep learning combined with ADS-B information according to the present invention.

FIG. 2 is the network structure of the motion estimation module of the present invention.

FIG. 3 is the network structure of the target detection module of the present invention.

FIG. 4 is a comparison result between the output of the motion estimation module of the present invention and the estimated background based on the time histogram method.

FIG. 5 is a comparison result between the moving target recognition algorithm implemented by the present invention and the RCNN target detection algorithm.

FIG. 6 is the mAP result of the present invention implementing the moving target recognition algorithm and the RCNN target detection algorithm in part of the video sequence of the AGVS data set.

Detailed ways

The specific embodiments of the present invention are described below to facilitate those skilled in the art to understand the present invention, but it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes Such changes are obvious within the spirit and scope of the present invention as defined and determined by the appended claims, and all inventions and creations utilizing the inventive concept are within the scope of protection.

A method for identifying moving targets in airport scenes based on deep learning combined with ADS-B information, as shown in Figure 1, includes the following steps:

S1. Decode the ADS-B information, and use the three-dimensional data returned by the aircraft and the position of the camera to correspond the ADS-B data to the two-dimensional image;

S2, label the scene surveillance video sequences in the airport scene surveillance data set to obtain the real bounding box, and simultaneously select the first ten video sequences as the training set, and the last ten video sequences as the test set;

Datasets play a crucial role in deep learning-based algorithms. Therefore, we select some airport scene surveillance video sequences in AGVS (airport scene surveillance data set). In this embodiment, labelme software is used to manually label the moving aircraft targets in each frame of pictures, and the first ten video sequences are set as training set, the last ten video sequences are set as the test set

S3, constructing a moving target recognition network, and using the constructed moving target recognition network to output the estimated background image;

According to the basic algorithm framework of the proposed motion judgment module and target detection module, we cascade the two to obtain the final moving target recognition network. The input of the network is the current frame, the position information and label information of the current frame after ADS-B decoding. And the first 30 frames of images in the history of the current frame, wherein, the first 30 frames of images in the history of the current frame are input to the motion estimation network as shown in Figure 2, and the output of the motion estimation network is the estimated background image. The comparison results of the time histograms are shown in Figure 4. Both of them use 30 frames of historical images to obtain the background image of the current frame.

The background estimation method based on the time histogram is widely used in the change detection algorithm. The method of the time histogram to realize the background estimation is as follows. First, at each pixel location, get a temporal histogram:

Among them, I(m,n,t) represents the pixel value at the position (m,n) of the t-th frame picture. From the pixel time histogram, the background pixel intensity at a specific location (m, n) can be obtained:

Among them, argmax(.) is the maximum value of the statistics of the pixel value l at the histogram point (m, n). Figure 4 presents the background estimation results based on the temporal histogram method. However, due to the slow speed of moving objects in airport scenes and frequent motion stills, the method based on temporal histogram cannot produce good results. Therefore, we propose a motion estimation network based on convolutional neural network. The structure is shown in Figure 2. The input of the network is the recent historical frame image, which continuously passes through multiple multi-scale receiving feature blocks. Each feature block captures the maximum response of the receptive field of size 1×1, 3×3 and 5×5, and finally obtains the estimated background feature map. . This allows us to obtain background statistics while still ensuring the adaptability of the network to different changing scenarios. The visual comparison results of the proposed motion estimation network are shown in Figure 4. It can be seen from the figure that the background estimation method based on the time histogram will appear ghost phenomenon due to the slow movement of the aircraft target. The proposed motion estimation network obtains a better background. to be accurate.

The object detection module consists of a backbone network, feature pyramid network, and regression and classification networks, as shown in Figure 3. We use the ResNet network as the backbone network, which is responsible for computing the convolutional feature map of the entire input image and providing high-dimensional image features for the feature pyramid network. Feature Pyramid Network (FPN) augments standard convolutional networks with top-down paths and lateral connections, and can be used to detect objects at different scales. At each layer of the pyramid feature map, an input feature map of 256 channels is taken and 9 anchors are set. After obtaining the feature map, first use the ADS-B signal as a guide to select a probability area, as shown in Figure 3, the red area is the probability area, and search on the feature map in this area, and finally generate by regression and classification target candidate box.

S4, use the estimated background image obtained in step S3 and the current frame image as the moving target recognition network to calculate the category of the target candidate frame and the boundary information of the target candidate frame;

S5. Compare the target candidate frame category obtained in S4 and the boundary information of the target candidate frame with the real bounding box marked in S2, and use the cross-entropy loss function and the smooth _L1 distance to calculate the classification loss function and the regression loss function. The sum of the loss functions is used as the loss function of the motion estimation network;

回归损失记作

The obtained background estimation image and the current frame image are used as the input of the backbone network and the feature pyramid network. The target detection network calculates the category of the candidate box and the bounding box information of the candidate box (the coordinates of the upper left vertex and the width and height of the box), which are calculated by These parameters are compared with the labeled ground truth, and two loss functions are calculated from the cross-entropy loss function and the smooth _L1 distance, where the classification loss is denoted as

The regression loss is recorded as

是与维度相同的向量，表示anchor，训练阶段是相对于ground truth的w偏移量。R代表smooth_L1。p_i是anchor为目标的概率，

是ground truth的标签。网络的损失函数为分类损失和回归损失二者损失函数之和：where t _i ={t _x , _ty , t _w , t _h } represents the coordinates of the anchor, and the training phase is the predicted offset.

is a vector of the same dimension as the anchor, and the training phase is the w offset relative to the ground truth. R stands for smooth _L1 . p _i is the probability that the anchor is the target,

is the ground truth label. The loss function of the network is the sum of the loss functions of classification loss and regression loss:

S6 , back-propagating the loss function of the moving target recognition network to obtain a trained motion estimation network, and using the trained motion estimation network to perform moving target recognition on the airport scene.

In this example, the back-propagation of the loss function will also pass through the motion estimation module. I use the Adam optimizer with an initial learning rate of 1×10 ^-5 for optimization, and the final detection prediction threshold is 0.5. Figure 5 compares the results of the experimental method and the target detection RCNN algorithm in the same frame of pictures. It can be seen that the algorithm in this paper not only has relatively better detection accuracy, but also can accurately determine the motion and stillness of the target.

Through the data analysis of the experimental results, it is judged that the number of historical frames most suitable for the airport scene is selected. We use the target detection evaluation index mAP to evaluate the accuracy of the detection results. Since the concept of moving target recognition is proposed for the first time in the present invention, and there is no related algorithm for comparison, the target detection algorithm RCNN is selected for comparison. The experimental results are shown in Figure 6. It can be seen that when the historical frame is selected as 30 frames, the average accuracy is the highest, and because the RCNN algorithm cannot distinguish motion and stillness, the accuracy is poor.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

In the present invention, the principles and implementations of the present invention are described by using specific embodiments, and the descriptions of the above embodiments are only used to help understand the method and the core idea of the present invention; The idea of the invention will have changes in the specific implementation and application scope. To sum up, the content of this specification should not be construed as a limitation to the present invention.

Those of ordinary skill in the art will appreciate that the embodiments described herein are intended to assist readers in understanding the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations without departing from the essence of the present invention according to the technical teaching disclosed in the present invention, and these modifications and combinations still fall within the protection scope of the present invention.

Claims (5)

1. An airport scene moving target recognition method based on deep learning combined with ADS-B information is characterized by comprising the following steps:

s1, decoding the ADS-B information, and corresponding the ADS-B information to a two-dimensional image by using three-dimensional data returned by the airplane and the position of the camera;

s2, labeling the scene monitoring video sequences in the airport scene monitoring data set to obtain a real boundary frame, and simultaneously selecting the first ten segments of video sequences as a training set and the last ten segments of video sequences as a test set;

s3, constructing a moving object recognition network, and outputting the estimated background image by using the constructed moving object recognition network;

s4, calculating the category of a target candidate frame and the boundary information of the target candidate frame by taking the estimated background image and the current frame image obtained in the step S3 as a moving target recognition network;

s5, comparing the boundary information of the target candidate frame and the target candidate frame obtained in the step S4 with the real boundary frame marked in the step S2, and utilizing a cross entropy loss function and smooth _L1 Calculating a classification loss function and a regression loss function according to the distance, and taking the sum of the two loss functions as a loss function of the motion estimation network;

and S6, learning the loss function back propagation of the moving object recognition network to obtain a trained motion estimation network, and recognizing the moving object on the airport scene by using the trained motion estimation network.

2. The method of claim 1, wherein the moving object recognition network constructed in S3 includes a motion judgment module and an object detection module in cascade, where the input of the moving object recognition network is a current frame, position information and label information after ADS-B decoding of the current frame, and a historical previous 30 frames image of the current frame.

3. The method for identifying airport surface moving objects based on deep learning combined with ADS-B information of claim 2, wherein the method for calculating the background image estimated in S3 is as follows:

inputting the recent historical frame image into a motion judgment module, continuously receiving the image through a plurality of multi-scale feature blocks, and capturing the maximum response of the receptive field with the size of 1 × 1, 3 × 3 and 5 × 5 by each feature block to obtain an estimated background image.

4. The method for identifying airport surface moving objects based on deep learning combined with ADS-B information of claim 2, wherein the object detection module comprises a trunk network, a feature pyramid network and a regression and classification network which are sequentially cascaded, wherein the trunk network is a ResNet network and is used for calculating convolution features of the whole input image; the characteristic pyramid network reinforces the target detection module through a top-down path and a transverse connection; both the regression and classification networks are multi-scale convolutional neural networks.

5. The method for identifying airport surface moving objects based on deep learning combined with ADS-B information of claim 2, wherein the classification loss function and the regression loss function in S5 are respectively expressed as:

wherein, t _i ＝{t _x ,t _y ,t _w ,t _h Representing the coordinates of the anchor points, which are predicted offsets in the training phase;

the ground truth label is the same as the dimension vector, and the offset of the ground truth is obtained in the training stage; p is a radical of _i Is the probability that the anchor point is the target;

is a classification loss function;

is a regression loss function.

Images