CN111369589B - A UAV tracking method based on multi-strategy fusion - Google Patents

Abstract

The invention discloses an unmanned aerial vehicle tracking method based on multi-strategy fusion, which comprises the steps of taking a Centernet network in a deep learning network as a structural main body, combining a frequency spectrum detection module and a steering engine control module, providing a new vision and frequency spectrum combined evaluation algorithm, effectively calculating the position of an unmanned aerial vehicle in a video image, controlling a camera steering engine to rotate through a central point of the position, accurately tracking the unmanned aerial vehicle in flight within a range of 3 kilometers, displaying the unmanned aerial vehicle in flight in a more intuitive vision tracking mode, and solving the problem of difficult tracking of the unmanned aerial vehicle in flight.

Description

A UAV tracking method based on multi-strategy fusion

technical field

The invention relates to the field of image processing, in particular to an unmanned aerial vehicle tracking method based on multi-strategy fusion.

Background technique

UAV usually refers to a powered, controllable, multi-tasking, and reusable unmanned aerial vehicle. Compared with manned aircraft, UAV has the advantages of light weight, small radar reflection cross section, low operating cost, high flexibility and no crew safety problems, and can be widely used in military tasks such as reconnaissance and attack; It can be used in many fields such as meteorological detection, disaster monitoring, geological exploration, map surveying and mapping, so it has been paid more and more attention by more and more countries and developed rapidly.

UAVs fly fast and generally have unique geometries that lack complete structural information, making tracking in flight difficult.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a UAV tracking method based on multi-strategy fusion, which aims to solve the problem that the UAV is difficult to track in flight.

In order to achieve the above purpose, the present invention provides a UAV tracking method based on multi-strategy fusion,

include:

Based on the Centernet network, the UAV image samples are trained to generate feature maps;

Obtain the UAV signal, and analyze and process the UAV signal to obtain the direction parameter of the UAV;

Control the rotation of the camera lens based on the direction parameter output of the UAV, collect the video image of the UAV, and obtain the estimated position of the UAV;

Perform image block weighting processing on the collected video images, and obtain the specific position of the UAV in the video image based on the joint evaluation algorithm of vision and spectrum;

Obtain the center coordinates of the UAV based on the opencv function and track the UAV in real time.

Wherein, the specific steps for training the UAV image samples based on the Centernet network and generating the feature map are:

Obtain the UAV image input RGB three-channel, and process it based on the convolutional neural network to output the predicted value;

Based on network forward propagation, features are extracted and feature maps are obtained.

Wherein, the specific steps of obtaining the UAV signal and analyzing and processing the UAV signal to obtain the direction parameter of the UAV are:

Detect and acquire drone signals within 3 kilometers;

Data fusion based on usage environment;

Extract UAV signal parameters through down-conversion, A/D sampling, digital channelization and array signal processing;

Perform amplitude and phase integrated direction finding processing based on the parameters of different antennas, and compare with the database to obtain the UAV model;

Positioning is carried out based on the multi-station direction finding and cross positioning system, and the direction parameters of the UAV are obtained.

Wherein, the specific steps of controlling the rotation of the camera lens based on the direction parameter output of the UAV, collecting the video image of the UAV, and obtaining the estimated position of the UAV are:

Read the output UAV direction parameters;

Calculate the difference between the orientation parameter and the center of the image;

Calculate the rotation of the gimbal through the mathematical model;

The rotation of the gimbal is controlled by the position PD algorithm. The position PD algorithm model is as follows:

用K_d表示，则有：Among them, S(n) is the control output, Kp is the proportional control parameter, Td is the differential control parameter, e(n) is the difference between the current state value and the target value, n is the control times,

Represented by K _d , there are:

S(n)=Kpe( _n )+Kd[ _e (n)-e(n-1)].

Wherein, the specific steps for obtaining the specific position of the drone in the video image based on the combined vision and spectrum evaluation algorithm by performing image block weighting processing on the collected video image are:

Divide the image into three image blocks and assign weighting parameters;

Extract the key points of each category of the feature map based on the Centernet network;

Perform confidence judgment on key points to obtain specific locations.

Wherein, the specific steps of performing confidence judgment on key points and acquiring specific positions are,

Based on the original Centernet algorithm, the first confidence judgment is made on the original image collected by the camera. If the key point in the feature map is smaller than the threshold A, the camera will be enlarged by a fixed multiple;

If the key point in the feature map is greater than the threshold A, the Centernet algorithm is used to judge the secondary confidence based on the weighted parameter and the calculation method of the feature response value;

If the secondary confidence determines that the feature peak is greater than the threshold B, use the opencv function to draw a frame and display it, and return the center coordinates (x, y) of the drone in the image.

Wherein, the specific calculation steps of the characteristic response value calculation method are,

Create the following formula:

y(t)=(ω+B(n))x(t)t≥0, n≥0

Among them, y(t) represents the feature response value, t represents the feature point sorting number, x(t) represents the response value of each feature point calculated by the original Centernet algorithm, ω represents the image block weight, B(n) represents the camera zoomed in each time The accuracy of a multiple increase, n represents the number of camera magnifications;

B(n)=(1.1) ⁿ βn≥0

where β is the initial constant.

In the UAV tracking method based on multi-strategy fusion of the present invention, spectrum detection and visual tracking can be integrated by using the UAV tracking method based on multi-strategy fusion. Use visual tracking to accurately locate the position of the UAV, and at the same time mark the position of the UAV in the video in a more intuitive form, so that the UAV user and monitor can observe the position of the UAV more clearly. .

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative efforts.

Fig. 1 is a kind of flow chart of the UAV tracking method based on multi-strategy fusion of the present invention;

Fig. 2 is a kind of operational block diagram of the UAV tracking method based on multi-strategy fusion of the present invention;

FIG. 3 is an image block diagram of a UAV tracking method based on multi-strategy fusion of the present invention.

Detailed ways

The following describes in detail the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to explain the present invention and should not be construed as limiting the present invention.

In the description of the present invention, it should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", The orientations or positional relationships indicated by "horizontal", "top", "bottom", "inside", "outside", etc. are based on the orientations or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than An indication or implication that the referred device or element must have a particular orientation, be constructed and operate in a particular orientation, is not to be construed as a limitation of the invention. In addition, in the description of the present invention, "plurality" means two or more, unless otherwise expressly and specifically defined.

Example

Please refer to Fig. 1, a UAV tracking method based on multi-strategy fusion of the present invention includes:

S101 , training the UAV image samples based on the Centernet network to generate a feature map.

Obtain the UAV image input RGB three-channel, and process it based on the convolutional neural network to output the predicted value;

24bit RGB image is also called full color image, which has three channels, namely R (red), G (green), B (blue). Use halcon program and halcon's own image to understand RGB image and gray value. The grayscale value of a three-channel image is the combination of the grayscale values of three single channels. The grayscale value is 0-255, and each channel is 0-255. The larger the value, the brighter the image looks, and the smaller the value, the darker the image. Which part of the three-channel image is seen to be darker, which proves which part of the color component is larger, and the brighter it is reflected on the single channel.

Based on network forward propagation, features are extracted and feature maps are obtained.

Network forward propagation is that a neuron has multiple inputs and one output, and the input of each neuron can be either the output of other neurons or the input of the entire neural network. The feature map is equivalent to the image formed after the convolutional neural network extracts the original image features. From the perspective of probability, each type of point in the feature map has its own probability, that is, the response value, which forms The heatmap reflects the color depth, that is, the size of the response value, and the point with the largest response value is the point where the tracking target is located.

S102 , acquiring the UAV signal, and analyzing and processing the UAV signal to obtain the direction parameter of the UAV.

Turn on the camera and turn on the spectrum detection module. At this time, the spectrum detection module owns the ownership of the steering gear control module, detects and captures the UAV signals within a range of 3 kilometers, and the signal data is fused according to the use environment; The remote control signal and the image transmission signal downloaded by the UAV are extracted by down-conversion, A/D sampling, digital channelization and array signal processing. The parameters are compared with the database to obtain the UAV model; based on the multi-station direction finding and cross positioning system, the orientation parameters of the UAV are obtained.

S103, outputting a control signal based on the direction parameter of the UAV to control the rotation of the camera lens, collecting the video image of the UAV, and obtaining the estimated position of the UAV.

Read the output UAV direction parameters, such as the direction and angle of the UAV relative to the camera; calculate the difference between the direction parameter and the image center; calculate the rotation of the gimbal through the mathematical model; use the position PD algorithm to control The PTZ rotates, and the PD algorithm model of the position is as follows:

用K_d表示，则有：

Represented by K _d , there are:

S(n)=K _p e(n)+K _d [e(n)-e(n-1)]

Among them, Kp is the proportional control parameter, Td is the differential control parameter, and e(n) is the difference between the current state value and the target value.

The center position of the camera image is the estimated position of the drone.

S104 , performing image block weighting processing on the collected video image, and obtaining a specific position of the drone in the video image based on a vision and spectrum joint evaluation algorithm.

The collected video image of size 1920*1080 takes (960,540) as the image center point, as shown in Figure 3, and divides the entire image into 3 image blocks, which are image_inside, image_middle, and image_outside. image_inside represents the middle part of the image, the image_inside image block should have a greater possibility of a drone target, and the image_outside is far from the center of the image, so there should be a small possibility of a drone target. Assuming that the parameters of image_inside, image_middle, and image_outside are ω ₁ , ω ₂ , and ω ₃ respectively, then ω ₂ =0.8ω ₁ , ω ₃ =0.4ω ₁ .

The Centernet detector compares all the feature points on the feature map with its 8 neighboring points, and if the response value of the point is greater than or equal to the value of its eight neighboring points, it is retained, and all the top 100 key points that meet the requirements are obtained.

是用上述方法检测得到的c类别的n个关键点的集合，每个关键点以整型坐标(x_i,y_i)的形式给出，则

产生如下的检测框：make

is the set of n key points of the c category detected by the above method, and each key point is given in the form of integer coordinates (x _i , y _i ), then

The following detection box is generated:

是偏移预测结果，

是尺度预测结果。in,

is the offset prediction result,

is the scale prediction result.

小于阈值的舍弃，

大于阈值的框图形成最终的无人机框图。All detection boxes form a rough frame, and then set a threshold,

Anything less than the threshold is discarded,

Blocks larger than the threshold form the final UAV block diagram.

Please refer to Figure 2. Use the Centernet algorithm to judge the confidence of the original image collected by the camera for the first time. If the key point in the feature map is smaller than the threshold value A, the camera is enlarged by a fixed multiple, n is increased by 1, and B(n) is updated; if If the key points in the feature map are greater than the threshold A, the second confidence judgment is performed. At this time, the image block weight is introduced into the video captured by the camera, and the feature response value calculation method is adopted, and the Centernet algorithm is used to judge the confidence again. Finally, if after the second confidence judgment, the characteristic peak value is greater than the threshold B, the tracking target is determined, displayed with the opencv function frame, and the center coordinates (x, y) of the drone in the image are returned. The tracking module regains control of the steering gear and re-controls the rotation of the steering gear through (x,y).

Based on the introduction of image block weights, this algorithm designs a new feature response value calculation method, as follows:

y(t)=(ω+B(n))x(t)t≥0, n≥0

Where t represents the feature point sorting number, x(t) represents the response value of each feature point calculated by the original Centernet algorithm, and y(t) represents the final response value of each feature point. ω represents the image block weight, which is divided into 3 different values according to 3 different regions image_inside, image_middle, image_outside, respectively ω ₁ , ω ₂ , ω ₃ , where ω ₂ =0.8ω ₁ , ω ₃ =0.4ω ₁ . B(n) is a linearly increasing function, which is used to represent the magnification of the camera each time (fixed multiple), the higher the resolution of the image, the higher the accuracy of the Centernet tracking algorithm, where n represents the number of times the camera is magnified , the equation is as follows:

B(n)=(1.1) ⁿ βn≥0

where β is the initial constant.

S105. Obtain the center coordinates of the UAV based on the opencv function, and track the UAV in real time.

After determining the position of the drone in the video image, use the opencv function to draw the rectangular frame where the drone is located, calculate and return the center coordinates (x, y) of the rectangular frame, and hand over the control ownership of the steering gear to the visual tracking module. Re-control the rotation of the steering gear through the center coordinate to track the drone in real time.

The above disclosure is only a preferred embodiment of the present invention, and of course, it cannot limit the scope of rights of the present invention. Those of ordinary skill in the art can understand that all or part of the process of implementing the above embodiment can be realized according to the rights of the present invention. The equivalent changes required to be made still belong to the scope covered by the invention.

Claims (5)

1. An unmanned aerial vehicle tracking method based on multi-strategy fusion is characterized in that,

the method comprises the following steps:

training an unmanned aerial vehicle image sample based on a Centeret network to generate a feature map;

acquiring an unmanned aerial vehicle signal, and analyzing and processing the unmanned aerial vehicle signal to obtain a direction parameter of the unmanned aerial vehicle;

outputting a control signal based on the direction parameter of the unmanned aerial vehicle to control the camera lens to rotate, and acquiring a video image of the unmanned aerial vehicle to obtain an estimated position of the unmanned aerial vehicle;

carrying out image block weighting processing on the acquired video image, and obtaining the specific position of the unmanned aerial vehicle in the video image based on a vision and frequency spectrum joint evaluation algorithm, wherein the method comprises the following specific steps: dividing the image into three image blocks and giving weighting parameters; extracting key points of each category of the feature map based on the Centeret network; performing confidence judgment on the key points to obtain specific positions, specifically performing first confidence judgment on an original image acquired by the camera based on a Centeret algorithm, and if the key points in the feature map are smaller than a threshold A, amplifying the camera by a fixed multiple; if the key point in the feature map is larger than the threshold A, performing secondary confidence judgment by using a Centernet algorithm based on a weighted parameter and a feature response value calculation mode; if the secondary confidence coefficient judges that the characteristic peak value is larger than the threshold value B, displaying the characteristic peak value by using an opencv function picture frame, and returning the central coordinates (x, y) of the unmanned aerial vehicle in the image;

and acquiring the central coordinate of the unmanned aerial vehicle based on the opencv function, and tracking the unmanned aerial vehicle in real time.

2. The unmanned aerial vehicle tracking method based on multi-strategy fusion as claimed in claim 1, wherein the specific steps of training the unmanned aerial vehicle image sample based on the centret network and generating the feature map are as follows:

acquiring three channels of an unmanned aerial vehicle image input RGB, and outputting a predicted value based on convolutional neural network processing;

and extracting features based on network forward propagation to obtain a feature map.

3. The unmanned aerial vehicle tracking method based on multi-strategy fusion of claim 1, wherein the specific steps of obtaining the unmanned aerial vehicle signal, analyzing and processing the unmanned aerial vehicle signal to obtain the direction parameter of the unmanned aerial vehicle are as follows:

detecting and acquiring unmanned aerial vehicle signals within a range of 3 kilometers;

performing data fusion based on the use environment;

extracting unmanned aerial vehicle signal parameters through down-conversion, A/D sampling, digital channelization and array signal processing;

carrying out amplitude-phase integrated direction finding processing based on parameters of different antennas, and comparing the processed direction finding processing with a database to obtain the model of the unmanned aerial vehicle;

and positioning based on a multi-station direction-finding cross positioning system to obtain the direction parameters of the unmanned aerial vehicle.

4. The unmanned aerial vehicle tracking method based on multi-strategy fusion as claimed in claim 1, wherein the specific steps of outputting a control signal based on the directional parameter of the unmanned aerial vehicle to control the camera lens to rotate, acquiring the video image of the unmanned aerial vehicle, and obtaining the estimated position of the unmanned aerial vehicle are as follows:

reading the direction parameters of the unmanned aerial vehicle;

calculating the difference between the direction parameter and the image center;

calculating the rotation quantity of the holder through a mathematical model;

controlling the rotation of the holder through a position PD algorithm, wherein the position PD algorithm model is as follows:

wherein S (n) is a control output, Kp is a proportional control parameter, Td is a derivative control parameter, e (n) is a difference between a current state value and a target value, n is a control number,

by K_dThis means that there are:

S(n)＝K_pe(n)+K_d[e(n)-e(n-1)]。

5. the unmanned aerial vehicle tracking method based on multi-strategy fusion of claim 1, wherein the calculation formula of the characteristic response value calculation mode is,

y(t)＝(ω+B(n))x(t)t≥0,n≥0

wherein y (t) represents a characteristic response value, t represents a characteristic point ranking number, x (t) represents a response value of each characteristic point calculated by the original Centernet algorithm, ω represents an image block weight, B (n) represents the accuracy of the camera increased by each magnification, and n represents the number of times of the camera magnification;

B(n)＝(1.1)ⁿβn≥0

where β is an initial constant.

Images