CN116883297A - Multi-target automatic verification method based on infrared and visible light fusion - Google Patents

Abstract

The invention discloses a multi-target automatic verification method based on infrared and visible light fusion, which includes: a target search stage, constructing a target detection algorithm based on the decision-level fusion of visible light images and infrared images, and realizing the search and detection of the presence or absence of targets in the task area. , and conduct fusion target detection on multiple targets; in the target verification stage, after the detection results in the fusion target detection stage meet the judgment conditions for entering the verification stage, the fusion target detection results are superimposed on the infrared image, and the target based on the infrared image is The tracking algorithm is used for tracking; multiple discovered targets are sequentially identified through zooming to obtain fine-grained information about the target, and the maneuvering target tracking model estimates and predicts the target's position. The present invention can effectively solve the problem of re-identification of highly similar targets, achieve faster and more accurate target lock switching, and improve the perception ability of UAVs in real scenarios.

Description

A multi-target automatic verification method based on infrared and visible light fusion

Technical field

The invention relates to the technical fields of computer vision, target positioning, and autonomous control of drones, and in particular, to a multi-target automatic verification method based on infrared and visible light fusion.

Background technique

The air-to-ground and sea multi-target automatic verification method refers to the technology in which UAVs use variable-focus photoelectric pods to automatically conduct fine-grained identification of multiple targets in the mission area and locate their geographical locations.

In the current multi-target verification methods based on UAV optoelectronic platforms, most of them use manual control to verify the targets in sequence. In the only automated methods, they also follow the large field of view search to find multiple targets. target, reduce the field of view to fine-grained identification of a single target, and re-enlarge the field of view to change the target for verification. In this paradigm, existing methods include methods to consistently represent targets before and after changes in the field of view through deep learning-based target re-identification algorithms and by combining target localization, that is, estimating the geography of each target obtained by target detection. position, and then construct a consistent representation of the target before and after the field of view changes.

Existing methods have major limitations and cannot adapt to the capability requirements of UAVs for autonomous verification of multiple targets. For methods based on target re-identification, the computing power of airborne embedded devices is limited. After running the target detection and target tracking algorithms, it is difficult to run the target re-identification model while ensuring real-time performance; for methods combined with target positioning, it is difficult to run the target re-identification model in real-time. It is said that most of the existing target positioning algorithms can only guarantee the target at the center of the field of view, that is, at the optical axis. For targets at the edge of the image, the positioning error is large and cannot be effectively used for target re-identification.

At present, most UAVs can be equipped with dual-light pods or even three-light pods, and infrared and visible light can perceive the environment separately with different field of view angles. However, existing methods cannot detect visible light and infrared images. Fusion only expands the richness of the UAV's information perception, but does not improve the flexibility of the UAV's environmental perception.

This patent aims to solve the following technical problems:

1. Poor multi-modal fusion adaptability: Existing UAVs mostly detect targets on the ground and sea through single light, or through pixel-level or feature-level modal fusion for target detection. The former cannot be well utilized. The information between infrared and visible light is complementary. The latter has higher computational cost and cannot flexibly adjust the model according to the environment.

2. It is difficult to achieve consistent representation of targets: When UAVs conduct reconnaissance of earth and sea targets, due to the high degree of similarity of multiple targets (multiple tanks, multiple uniformed personnel, multiple similar ships, multiple sea surface buoys, etc.) and the real-time performance of the algorithm Due to various reasons, it is difficult to construct a consistent representation of the target before and after the field of view changes through the target re-identification algorithm based on deep learning, which in turn leads to a low degree of autonomy in the UAV reconnaissance process.

In this problem, there are two sub-problems. a: When the target is discovered in the target search stage and is ready to enter the target verification stage, the target detection changes from visible light and infrared fusion detection to infrared image detection and tracking, and the target may be found in visible light. But no target was found in the infrared image/> At this time, it enters the target verification stage. Visible light is no longer different from target tracking. How to maintain target tracking in infrared images/> tracking. b: During the movement and hovering of the drone, there is a target/> , which at a certain moment/> Leave the field of view of the infrared image and in/> Within a short period of time, it re-enters the field of view, how to re-identify it and re-identify it as a target/> instead of/> , so as to avoid repeated verification.

3. UAV multi-task reconnaissance has poor flexibility: When performing reconnaissance of Earth-sea targets, the computing platform on the UAV needs to run target detection algorithms, target tracking algorithms, fine-grained target recognition algorithms, etc. at the same time. The existing algorithms cannot fully To ensure the flexibility of tasks using multi-sensor pods, a new algorithm framework needs to be constructed to give full play to the respective advantages of infrared and visible light to perform tasks more flexibly.

Contents of the invention

The technical problem to be solved by the present invention is to provide a multi-target automatic verification method based on infrared and visible light fusion in view of the defects in the existing technology.

The technical solutions adopted by the present invention to solve the technical problems are:

The present invention provides a multi-target automatic verification method based on infrared visible light fusion. The method includes the following steps:

In the target search stage, a target detection algorithm based on the decision-level fusion of visible light images and infrared images is constructed to realize the search and detection of the presence or absence of targets in the task area, and perform fusion target detection on multiple targets;

In the target verification stage, after the detection results of the fused target detection stage meet the judgment conditions for entering the verification stage, the fused target detection results are superimposed on the infrared image and tracked by the target tracking algorithm based on the infrared image; multiple targets discovered are Fine-grained recognition is performed sequentially through zooming to obtain fine-grained information about the target, and the target's position is estimated and predicted by the maneuvering target tracking model.

Further, the target detection algorithm based on the decision-level fusion of visible light images and infrared images in the method of the present invention is specifically:

Preserve accurate results for targets detected only in visible images or only in infrared images;

Perform weighted fusion on the accurate results of the same target detected simultaneously in visible light images and infrared images, including fusion of frames and fusion of target confidence;

The obtained detection results are merged and used as the detection results of all corresponding targets in the fused image to achieve fast target detection based on decision-level fusion.

Further, in the method of the present invention, the fusion target detection result is superimposed on the infrared image, and the method for tracking by the target tracking algorithm based on the infrared image is specifically:

After the detection results in the fusion target detection stage meet the judgment conditions for entering the verification stage, the fusion target detection results are superimposed on the infrared image and tracked by the target tracking algorithm based on the infrared image. In order to prevent the target from being detected in the infrared image The characteristics of the target are not obvious/> Tracking failed. When performing target locking verification, priority is given to locking the target/> Perform fine-grained target identification on it; if there are multiple such targets when entering the verification stage, the target position will be recorded after verifying the first target, and the target search stage will be re-entered. Until the verification stage is entered, the target can be detected based on infrared Images are tracked;

If there is a target To re-enter the infrared field of view, first give the target number/> , when locking the target for fine-grained identification, first judge whether the target has been verified based on the target positioning result. If it meets:

It is considered that for target/> Re-enter the field of view, otherwise, it is considered that the target is a new target or the target _position estimation model error is too large, the target number remains unchanged, waiting for further verification, Pos _Ti represents the position of target Ti .

Further, in the method of the present invention, the method of sequentially performing fine-grained identification of multiple discovered targets through zooming is specifically:

The fine-grained target recognition model used is pre-loaded according to the content of the reconnaissance mission, including open scene character recognition model, person recognition model, and face detection model. Vessels entering the reconnaissance range are verified. If people in the deck area of the ship are detected, then The face detection model is further used to identify faces, and the acquired ship images, ship hull numbers, number of people on board, and personnel face information are saved and transmitted back to the command center.

Further, the target search stage of the method of the present invention specifically includes the following steps:

Step 1: Adjust the visible light and infrared lenses to the maximum field of view and enter the target search state;

Step 2: Obtain visible and infrared images separately with/> And perform image preprocessing separately;

Step 3: Separate images with/> Enter the visible light target detection model/> and infrared target detection model/> Carry out model inference and obtain the detection frame set/> with/> Then take the union of the two after aligning them by resolution, and compare the intersection/> Partial detection results are fused, that is, when the target is detected in both infrared and visible light images, the detection frame is fused. The center position of the fused frame is weighted by the infrared and visible light results, and the prediction confidence score /> , of which/> Preset according to specific situations; detected targets are divided into three groups,/> Represents the target group detected in the visible light image but not detected in the infrared image, /> Represents a target group detected in the infrared image but not detected in the visible light image, /> Represents target groups detected by both;

Step 4: Track multiple targets in the field of view through the target tracking algorithm, and sort and number them in the order of , for the targets in each group, sort them from high to low in terms of confidence;

Step 5: Adjust the focal length of visible light and infrared adaptively, repeat step 4, and enter the target verification stage when the number of times the target is stably found after multiple consecutive readings is greater than the threshold.

Further, the target verification stage of the method of the present invention specifically includes the following steps:

Step 6: According to the target ID generated by the fusion target detection result, the ID is lock gaze on the target, when locking for the first time/> ;

Step 7: In the infrared image, keep track of the target and zoom in by changing the focal length of the visible light lens Perform fine-grained target identification on the target to obtain the specific category of the target, including ship model, shipboard number, and quantity information on board. While obtaining the fine-grained information, the target based on multi-model particle filtering is constructed through the positioning algorithm and The state estimation algorithm estimates the target's position and speed information, and predicts the target's next position;

Step 8: Get the ID as After the target information is obtained, the tracking status of the infrared image is switched synchronously to the lock ID of/> target, repeat steps 6 and 7.

The beneficial effects produced by the present invention are:

1. The method constructed in the present invention can construct a consistent representation of the target through positioning algorithm, infrared visible light fusion target detection algorithm and other methods when the UAV conducts multi-target verification, and effectively solves the problem of re-identification of highly similar targets.

2. The present invention gives full play to the flexibility of the dual-light pod. Through independent control of infrared and visible light, it can achieve faster and more accurate target locking switching, and fully utilizes the tracking-friendly characteristics of infrared images and the wireless granularity characteristics of visible light images.

3. The present invention fully combines target positioning with target detection and tracking, improving the perception ability of the drone in real scenarios.

4. The present invention greatly improves the degree of autonomy of the UAV reconnaissance system and can complete rapid independent verification of multiple targets in the mission area in a short period of time.

Description of the drawings

The present invention will be further described below in conjunction with the accompanying drawings and examples. In the accompanying drawings:

Figure 1 is a target detection algorithm based on decision-level fusion of visible light images and infrared images;

Figure 2 is a schematic diagram of the IOU fusion of infrared and visible red target detection results;

Figure 3 is a schematic diagram of infrared and visible light performing different tasks in different fields of view.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention and are not intended to limit the present invention.

Embodiment 1

The multi-target automatic verification method based on infrared visible light fusion according to the embodiment of the present invention includes two stages, namely the target search stage and the target verification stage. The target search stage mainly solves the problem of whether the target is present in the task area, and the target verification stage It mainly performs fine-grained identification of multiple targets discovered through zooming to obtain fine-grained information such as ship model, hull number, tank model, etc. By combining algorithm models such as weapon firepower classification, different back-end tasks such as threat assessment and alarms can be further realized. .

In response to technical question 1, a target detection algorithm based on the decision-level fusion of visible light images and infrared images is constructed. First, the accurate results of targets detected only in visible images or only in infrared images are retained; then, the target detected in visible light images and infrared images is The accurate results of the same target detected at the same time are weighted and fused, including the fusion of frames and the fusion of target confidence. Finally, the detection results are merged and used as the detection results of all corresponding targets in the fused image to achieve a decision-level based Fusion of fast object detection.

The algorithm framework diagram is shown in Figure 1, in which the image enhancement algorithm can be manually selected according to weather conditions (such as adaptive defogging algorithm in foggy conditions, etc.), and the small target detection capability, algorithm real-time performance, and complex background can be weighed according to task requirements. Different target detection network models are selected according to the degree, etc. Compared with feature-level or pixel-level infrared and visible light fusion, this method is more flexible in practical applications.

Infrared and visible light image fusion target detection is mainly divided into three categories, namely pixel-level image fusion, feature-level image fusion, and decision-level image fusion. Pixel-level image fusion refers to the process of registering infrared and visible light before inputting into the network model, and processing the corresponding pixels to obtain the fused image. This method is more accurate, but it requires a large amount of calculation and is not practical. During the application process, due to reasons such as the focal length of infrared visible light and the processing delay of the algorithm, the two images obtained may not be obtained at the same time and at the same focal length, which brings greater difficulties to registration; feature-level fusion refers to the alignment through volume After extracting features, only important information is retained, and some redundant information is removed, and then the features are fused. This compresses the detailed information in the source image to a certain extent and lacks a certain degree of flexibility in practical applications. Decision-level image fusion can obtain detection results through different network models for infrared and visible light images, and then fuse the results according to actual needs. It has high flexibility and is especially suitable for the wireless sensors involved in the present invention. In human-machine target reconnaissance scenarios, through the target detection algorithm based on the decision-level fusion of visible light images and infrared images, the image data under the two spectrums of infrared and visible light can be fully utilized in the target search stage, and it can also reduce the problems caused by the angle of view, focal length, and time. It can solve the fusion problems caused by factors such as delay and resolution, and can freely choose different models according to the actual scene. For example, when the task area to be reconnoited has a complex background, choose a network with better robustness under the complex background. model, and when the background of the area to be detected is relatively simple, it can maximize the small target detection capability to obtain a larger effective detection range.

In response to technical issue 2, a strategy is proposed for infrared and visible light to perform different tasks respectively. That is, infrared images rely on the advantage of more obvious differences between targets and backgrounds to maintain tracking of multiple targets with a large field of view, while visible light images have richer details. Carry out more fine-grained target recognition, such as tank model, ship model, hull number, buoy identification, etc. and obtain the target's position information, and use the maneuvering target tracking model to estimate and predict its position, as shown in Figure 3.

The maneuvering target tracking model can choose an interactive multi-model IMM method that is more robust to operating state uncertainty. Specifically, in order to avoid the estimation time delay caused by the tracking algorithm with maneuver detection and the degradation of tracking performance during the maneuver detection process To reduce the problem, an adaptive maneuvering target tracking algorithm based on interactive multi-model (IMM) is used to achieve adaptive estimation of the target maneuvering state through the interaction of different target models.

Regarding technical issue 2a, after the detection results in the fusion target detection stage meet the judgment conditions for entering the verification stage, the fusion target detection results are superimposed on the infrared image and tracked by the target tracking algorithm based on the infrared image. In order to prevent Hit the target The characteristics of the target are not obvious/> Tracking failed. When performing target locking verification, priority is given to locking the target/> Perform fine-grained target identification on it; if there are multiple such targets when entering the verification stage, the target position will be recorded after verifying the first target, and the target search stage will be re-entered. Until the verification stage is entered, the target can be detected based on infrared Image is tracked; for technical issue 2b, if target exists/> To re-enter the infrared field of view, first give the target number/> , when locking the target for fine-grained identification, first judge whether the target has been verified based on the target positioning result. If

It is considered that for target/> Re-enter the field of view, otherwise, the target is considered to be a new target or the target position estimation model error is too large, and the target number remains unchanged, waiting for further verification.

In response to technical question 3, in order to increase the flexibility of UAV multi-mission reconnaissance, the deep learning algorithm mentioned in the present invention can be replaced by other models, and can be adjusted according to different tasks and different environmental conditions to conduct daily monitoring of a certain sea surface. Take the patrol reconnaissance scenario as an example, in which the fine-grained target recognition model is pre-loaded according to the content of the reconnaissance mission. You can verify the ships entering the reconnaissance range by loading the open scene character recognition model, person recognition model, face detection model, etc. People arriving on the ship's deck and other areas will further use the face detection model to identify their faces, and the acquired information such as the ship image, ship's hull number, number of people on the ship, and their faces will be saved and sent back to the command center.

Embodiment 2

The multi-target automatic verification method based on infrared visible light fusion according to the embodiment of the present invention includes the following steps:

In the target search stage, through the target detection algorithm based on infrared and visible light fusion, in order to effectively utilize the information of infrared and visible light, reduce the computational cost and consider the flexibility in practical scenarios, this method chooses the decision-level fusion method with the lowest computational cost instead of pixel Level or feature level image fusion, the algorithm framework is shown in Figure 1. in

Step 1: Adjust the visible light and infrared lenses to the maximum field of view and enter the target search state;

Step 2: Obtain visible and infrared images separately with/> And perform image preprocessing separately to reduce the impact of weather and sensor noise;

Step 3: Separate images with/> Enter the visible light target detection model/> and infrared target detection model/> Carry out model inference and obtain the detection frame set/> with/> Then take the union of the two after aligning them by resolution, and compare the intersection/> Part of the detection results are fused, that is, when the target is detected in both infrared and visible light images, the detection frame is fused. The center position of the fused frame is represented by the weighted infrared and visible light results, as shown in Figure 2. The prediction confidence score is the same Reason,/> , of which/> Preset according to specific situations; so far, the detected targets have been divided into three groups,/> Represents the target group detected in the visible light image but not detected in the infrared image, /> Represents a target group detected in the infrared image but not detected in the visible light image, /> Represents target groups detected by both;

Step 4: Track multiple targets in the field of view through the target tracking algorithm, and sort and number them in the order of , for the targets in each group, sort them from high to low in terms of confidence.

Step 5: Adjust the focal length of visible light and infrared adaptively, repeat step 4, and enter the target verification stage when the number of times the target is stably found after multiple consecutive readings is greater than the threshold.

In the target verification phase, the infrared camera always maintains a large field of view and uses the target tracking algorithm to track the target in the infrared image to maintain the perception of the global target. The visible light camera adjusts the focal length to reduce the field of view and enlarge the target to detect it. Fine-grained recognition, after the recognition is completed, track the target information in the infrared image to replace the locked target, and perform fine-grained recognition in sequence.

Step 6: According to the target ID generated by the fusion target detection result, the ID is lock gaze on the target (when locking for the first time/> );

Step 7: In the infrared image, keep track of the target and zoom in by changing the focal length of the visible light lens Perform fine-grained target identification on the target to obtain the specific category of the target, such as ship model, shipboard number, number on board, etc., and while obtaining fine-grained information, use the positioning algorithm and the built-in multi-model particle filtering based The target state estimation algorithm estimates the target's position and speed information, and predicts the target's next position;

Step 8: Get the ID as After the target information is synchronized with the tracking status of the infrared image, switch the lock ID to/> target, repeat steps 6 and 7.

It should be understood that the sequence number of each step in the above embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any influence on the implementation process of the embodiment of the present application. limited.

It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the protection scope of the appended claims of the present invention.

Claims (6)

1. The multi-target automatic verification method based on infrared and visible light fusion is characterized by comprising the following steps of:

in the target searching stage, a target detection algorithm based on visible light image and infrared image decision level fusion is constructed, so that whether targets exist in a task area or not is searched and detected, and fusion target detection is carried out on a plurality of targets;

a target verification stage, wherein after the detection result of the fusion target detection stage meets the judgment condition of entering the verification stage, the fusion target detection result is superimposed on the infrared image and tracked by a target tracking algorithm based on the infrared image; and carrying out fine granularity identification on the discovered multiple targets sequentially through zooming to obtain fine granularity information of the targets, and carrying out position estimation and prediction on the targets by a maneuvering target tracking model.

2. The multi-target automatic verification method based on infrared-visible light fusion according to claim 1, wherein the method is characterized in that a target detection algorithm based on visible light image and infrared image decision level fusion specifically comprises:

retaining accurate results of objects detected only in the visible image or only in the infrared image;

weighting and fusing the accurate results of the same object detected in the visible light image and the infrared image simultaneously, wherein the weighting and fusing comprise fusion of a frame and fusion of a target confidence coefficient;

and combining the obtained detection results to serve as detection results of all corresponding targets in the fusion image so as to realize rapid target detection based on decision-level fusion.

3. The method for automatically verifying multiple targets based on infrared-visible light fusion according to claim 1, wherein the method comprises the steps of superimposing a fusion target detection result on an infrared image and tracking by a target tracking algorithm based on the infrared image:

after the detection result of the fusion target detection stage meets the judging condition of entering the verification stage, the fusion target detection result is superimposed on the infrared image and tracked by a target tracking algorithm based on the infrared image, so as to prevent targets in the infrared image from being causedIs not significantly responsible for the target->Tracking failure, when target-lock verification is performed, target is preferentially locked +.>Carrying out fine-grained target identification on the target; if a plurality of targets exist when entering the verification stage, recording the target position after the first target is verified, and re-entering the target searching stage until entering the verification stage, wherein the targets can be tracked based on infrared images;

if there is a targetRe-enter the infrared field of view, first given the object number +.>When the target is locked for fine granularity identification, firstly judging whether the target is verified according to the target positioning result, and if so:

then consider asFor the purpose of->Re-entering the view field, otherwise, considering the target as a new target or the error of the target position estimation model is overlarge, the number of the target is unchanged, waiting for further verification,Pos _Ti representing objectsT _i Is a position of (c).

4. The method for automatically verifying multiple targets based on infrared and visible light fusion according to claim 1, wherein the method for sequentially carrying out fine-granularity identification on the multiple targets by zooming is specifically as follows:

and preloading a fine-grained target recognition model adopted according to the content of the reconnaissance task, wherein the model comprises an open scene character recognition model, a personnel recognition model and a face detection model, the ship entering the reconnaissance range is verified, if personnel in a deck area of the ship are detected, the face is further recognized through the face detection model, and the acquired ship image, ship board number, number of the personnel on the ship and personnel face information are stored and returned to the command center.

5. The multi-target automatic verification method based on infrared-visible light fusion according to claim 1, wherein the target search stage of the method specifically comprises the following steps:

step 1: the visible light and the infrared lens are adjusted to the maximum view field, and the object searching state is entered;

step 2: respectively obtain visibilityLight and infrared imageAnd->And respectively performing image preprocessing;

step 3: respectively image the imagesAnd->Input visible light target detection model->And infrared target detection modelModel reasoning is carried out to obtain a detection frame set +.>And->Then, the two are merged after resolution alignment, and the intersection is +.>Fusing partial detection results, namely fusing the detection frames when the infrared and visible light images detect targets, wherein the central position of the fused frames is represented by weighting the infrared and visible light results, and predicting confidence scores->Wherein->Presetting according to specific conditions; the targets that have been detected are divided into three groups，/>A target group representing a detected in the visible light image but not in the infrared image,/->Indicating the detected target group in the infrared image but not in the visible image,/->Representing a set of targets detected by both;

step 4: tracking a plurality of targets in the visual field through a target tracking algorithm, and numbering the targets in sequence, wherein the sequence is thatFor the targets within each group, ordering from high to low confidence;

step 5: and (3) adjusting the focal lengths of visible light and infrared rays in a self-adaptive manner, repeating the step (4), and entering a target verification stage when the number of times of continuous reading and repeated detection and stable finding of the target is larger than a threshold value.

6. The method for automatically verifying multiple targets based on infrared-visible light fusion according to claim 5, wherein the target verification stage of the method specifically comprises the following steps:

step 6: according to the target ID generated by fusing the target detection result, the pair ID isIs fixation locked, first fixation +.>；

Step 7: in the infrared image, the target is kept locked and tracked, and the focus of the visible light lens is changed to enlarge the targetThe number target carries out fine-granularity target identification to obtain specific categories of the target, including ship model, ship board number and ship quantity information, and estimates the position and speed information of the target through a positioning algorithm and a constructed target state estimation algorithm based on multi-model particle filtering while obtaining fine-granularity information, and predicts the next position of the target;

step 8: acquisition of ID asAfter the target information of (a), the ID is switched synchronously with the tracking state of the infrared image to be lockedRepeating steps 6, 7.