CN117876774A - High-speed object detection method based on multi-branch feature fusion and re-parameterization - Google Patents

Abstract

The invention discloses a high-speed target detection method based on multi-branch feature fusion and re-parameterization. The method comprises the steps of firstly reading a target image under high-speed movement, inputting the image into a network model for reasoning, and reasoning out the target in the image and labeling the target through a network. According to the invention, the E-ELAN structure in Yolov7 is improved to obtain the IM-ELAN structure, so that not only is the comprehensive and richer characteristic representation obtained through multi-branch characteristic fusion, but also the parameter quantity and the calculated quantity of multi-head self-attention are reduced through a pooling layer, finally, the multi-scale local characteristic is extracted by using a heavy parameterization help model on the premise of not influencing the speed, the morphological understanding of the model on the targets in different sizes is enhanced, the detection effect of the high-speed deformation targets is improved, and the actual detection requirement of the high-speed targets in real life is met.

Description

High-speed object detection method based on multi-branch feature fusion and re-parameterization

Technical Field

The invention belongs to the field of image processing and pattern recognition in computer vision, and relates to a high-speed target detection method based on multi-branch feature fusion and re-parameterization.

Background technique

With the rapid development of deep learning in recent years, detection algorithms based on deep learning have been increasingly used in real life. In autonomous driving, the recognition of pedestrians on the road, other fast-moving vehicles, and traffic signals requires a high-speed target detection algorithm to respond quickly in order to reserve reaction time for subsequent operations. In sports competitions, high-speed deformable target detection can help referees more accurately locate athletes, balls and other moving objects, so as to make more fair judgments. However, since various high-speed deformable targets themselves are very different from those at rest, there are problems of deformation and smearing. And from the perspective of practical application, it is also necessary to ensure a faster detection speed of the model.

In the framework of convolutional neural networks, the underlying features are rich in detailed information, including edge, position, and texture information. These features have significant advantages in capturing targets at rest or normal speed. However, under high-speed motion conditions, due to factors such as changes in ambient light and equipment limitations, the target may experience deformation, smearing, and other phenomena. At this time, the underlying features are easily affected by these noises. For example, deformation may cause a significant change in texture and edge information, which makes most detection models originally designed for targets at normal speed unable to effectively cope with target detection under high-speed motion. However, high-level features have stronger robustness after multiple nonlinear transformations and pooling operations, and can resist interference from factors such as image noise, brightness changes, and deformation smearing. In addition, high-level features also contain a large amount of semantic information and global information, including object shape information and scene information. Taking car detection as an example, even if the car shape in the image is elongated or flattened, its overall outline still maintains the unique characteristics of the car. The detection model can rely on these global features to more accurately identify the deformed car. Therefore, the fusion of low-level features and high-level features can obtain a more comprehensive and richer feature representation, further improving the ability to handle deformation problems.

In addition, general single-scale local feature extraction only extracts features within a window of a certain size, that is, it only extracts features from a fixed scale. In actual scenes, due to the influence of factors such as distance and angle, the same target may appear in different sizes in the image. The information extracted by single-scale features shows certain limitations when facing targets of different sizes. Multi-scale feature extraction extracts features of targets at different scales (such as different resolutions or spatial ranges). For example, at a small scale, the model may focus on a small part or a specific texture of an object; at a large scale, the model may obtain more global or contextual information. Therefore, multi-scale local feature extraction helps to enhance the understanding of the morphology and structure of high-speed targets at different sizes, thereby improving the robustness of high-speed target detection.

Summary of the invention

Aiming at the deformation and smear problems of targets in high-speed motion, the present invention proposes an IM-ELAN structure based on Yolov7, which further enhances the deformation feature extraction capability of the network model for high-speed targets without slowing down the reasoning speed.

In a first aspect of the present invention, a high-speed target detection method based on multi-branch feature fusion and re-parameterization is provided, the method comprising the following steps:

Step 1: Obtain a high-speed deformation target data set;

Step 2: Improve the E-ELAN structure in the network model of yolov7 to improve the detection effect of the model on high-speed targets;

The E-ELAN structure is mainly composed of four branches. An IM module is added to the first branch, and the remaining three branches are convolution extraction branches, which fuse the features extracted by the four branches.

The IM module includes an IMA submodule and an MSCR submodule. The IMA submodule directly processes any two pairs of relationships between input elements through a multi-head self-attention mechanism; the MSCR submodule helps the model extract multi-scale local information without affecting the reasoning speed.

Step 3: Train the improved yolov7 network on COCO large-scale target detection data to obtain a pre-trained model;

Step 4: Based on the pre-trained model, the final model is trained on the high-speed deformation target dataset;

Step 5: Input the image to be detected that may contain high-speed deformation targets into the final model to detect high-speed targets.

A second aspect of the present invention provides a computer device, comprising a memory, a processor, and computer executable instructions stored in the memory and executable on the processor, wherein the processor implements the above-mentioned high-speed target detection method when executing the instructions.

According to a third aspect of the present invention, a computer storage medium is provided, wherein the computer storage medium stores computer executable instructions, and wherein the computer executable instructions implement the above-mentioned high-speed target detection method when executed.

The beneficial effect of the present invention is that the existing detection model cannot well detect targets that are deformed and have smears under high-speed movement. The optimization structure proposed by the present invention modifies the backbone network of the Yolov7 network, not only obtaining a more comprehensive and richer feature representation through multi-branch feature fusion, but also reducing the number of parameters and calculations of multi-head self-attention through the pooling layer, and finally using re-parameterization to help the model extract multi-scale local features without affecting the speed.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the network structure and training process of the present invention, the following briefly introduces the drawings required in the implementation case.

FIG1 is a flowchart of high-speed deformable target detection of the present application.

FIG2 is a diagram of the IM-ELAN structure designed in an embodiment of the present application.

FIG3 is an IMA-MSCR module designed in an embodiment of the present application.

FIG. 4 is an IMA module designed in an embodiment of the present application.

FIG5 is a diagram of a MSCR module designed in an embodiment of the present application.

FIG. 6 is a partial sample diagram of the high-speed deformation target data set used in the embodiment of the present application.

FIG7 is a visualization effect of an embodiment of the present application on the PASCAL VOC dataset and a self-built high-speed target dataset.

Detailed ways

In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

The embodiment of the present application provides a high-speed target detection method based on multi-branch feature fusion and re-parameterization, the method comprising the following steps:

Step 1. Obtain a video of a high-speed moving target in real life, manually collect deformation images of the target under different conditions, and obtain a high-speed deformation target dataset. Some sample images are shown in Figure 6. The distribution of the dataset is shown in Table 1.

Table 1 High-speed target dataset

Step 2. Improve the E-ELAN structure in the Yolov7 network to obtain the IM-ELAN structure.

The IM-ELAN structure is shown in Figure 2. First, the input image X is subjected to three convolutions to obtain the feature map C ₂ , and then C ₂ is split into two feature maps F ₁ and F ₂ according to the number of channels. Among them, F ₁ is used as the first branch, and the feature map F _im is obtained through the IM module; F ₂ is used as the second branch, and the feature map F ₃ is obtained after two convolutions; F ₃ is used as the third branch, and F ₃ is subjected to two convolutions to obtain the feature map F ₄ as the fourth branch. Then the four branches are spliced on the channel, and a convolution is performed to obtain the feature map F ₅ . _{F 5} will be extracted through the shuffle and pooling branches respectively to obtain the feature map F _5-1 and the feature map F _5-2 , and finally the two are added on the channel to obtain the feature map C ₃ .

This embodiment not only obtains a more comprehensive and richer feature representation through multi-branch feature fusion, but also reduces the number of parameters and calculations of multi-head self-attention through the pooling layer. Finally, re-parameterization is used to help the model extract multi-scale local features without affecting the speed, enhance the morphological understanding of the target in different sizes, and thus improve the detection effect of high-speed deformed targets.

In a preferred embodiment, the IM-ELAN uses the stochastic gradient descent method SGD, and warmup is used for the first three iterations. The training hyperparameter settings are shown in Table 2.

Table 2 Training hyperparameter settings

Furthermore, the IM module consists of two parts: an IMA submodule and an MSCR submodule, as shown in FIG3 .

The IMA submodule directly processes any two pairs of relationships between input elements through a multi-head self-attention mechanism, which makes them particularly suitable for processing long-distance dependencies, which is especially useful when the objects to be identified are distributed in different parts of the image, such as a train blocked by vegetation in the middle. At the same time, in order to ensure the real-time performance of the model, the pooling module is introduced to reduce the number of parameters of the self-attention module. The MSCR submodule helps the network model extract multi-scale local information without affecting the inference speed.

In a preferred embodiment, the IMA submodule adopts the same idea as the Vision Transformer, and the structure is shown in FIG4 .

First, the pixels in the feature map _F1 are regarded as elements in each token. By flattening and merging the width and height, they are constructed into the input of Self-Attention, that is, a vector sequence E, as shown in Formula 1.

E = [token ₁ , token ₂ , .. , token _wh ] = flatten (w, h, c) (1)

After adding the position encoding, three weight-sharing linear layers are used to obtain the three matrices Q, K, and V. Q, K, and V represent the query matrix, key-value matrix, and value matrix, respectively.

The existing self-attention mechanism is to directly perform calculations, which leads to high computational complexity. This embodiment hopes to help the network model improve spatial correlation without reducing computational efficiency. Therefore, the K and V matrices are improved by using average pooling with a pooling kernel of 3 and a step size of 2, reducing the width and height dimensions of K and V to 1/3 of the original, and obtaining K′ and V′.

After adopting the above improvements, the number of model parameters and the amount of calculation are reduced, and the speed of calculating Self-Attention in the model is improved. The calculation complexity of Q and K ^T matrices before the improvement is O((w×h) ² ×c), and the calculation complexity of Q and (K′) ^T matrices after the improvement is O(((w×h) ² ×c)/3). Similarly, the calculation complexity of the subsequent V′ calculation is also reduced by three times, which greatly reduces the calculation complexity of Self-Attention.

Subsequently, a multi-head attention mechanism was adopted to concatenate the calculation results of the four groups of heads and fuse them with a linear layer to obtain the feature map F _ima .

The feature map F _ima obtained by the IMA sub-module will then pass through the MSCR sub-module. The MSCR sub-module will be divided into two processes: training process and inference process, as shown in Figure 5.

In training, the F _ima is first placed in a fully connected layer and a BN layer. In order to save parameters, a grouped fully connected layer is used. That is, the feature map of each channel is flattened, and then a fully connected operation is performed inside each channel. In one example, a 1×1 grouped convolution is used to replace the grouped fully connected layer, but its essence is a grouped fully connected layer.

Pass F _ima through 1×1 group convolution and BN layer, and then transform the obtained feature vector into the shape of the original input to obtain F _fc . As shown in Formula 2:

F _fc = flatten(BN(GroupConv(flatten(Fi _ima )))) (2)

At the same time, F _ima is input into another multi-scale convolution branch, where the convolution kernels in the multi-scale convolution module are 1, 3, and 5 respectively. The use of multi-scale convolution in this application can help the network model learn local features of different sizes. At the same time, in order to consider the subsequent need to fuse the feature map of the convolution branch with F _fc , it is necessary to perform padding operations of different sizes on different pairs of convolutions, and finally pass through the BN layer, as shown in Formula 3:

With the help of the multi-scale convolution module, this application can better focus on the deformation characteristics of high-speed targets of different scales, thereby helping to detect targets in high-speed motion.

Then the obtained F _fc and After adding, we get F _im , as shown in Formula 4:

The above is the training process of the MSCR module.

In inference, the MSCR submodule reparameterizes The branch is merged into the F _fc branch. The specific theory is as follows: Since matrix multiplication (MMUL) is additive, the weight matrix is also additive, as shown in Formula 5. Since convolution is essentially a sparse matrix multiplication, there is an equivalent weight conversion between convolution and fully connected matrix multiplication, as shown in Formula 6:

MMUL( _Fima ⁽ⁱⁿ⁾ , _W1 )+MMUL( _Fima ⁽ⁱⁿ⁾ , _W2 )=MMUL(M ⁽ⁱⁿ⁾ , _W1 + _W2 ) (5)

MMUL( _Fima ⁽ⁱⁿ⁾ ,W)＝CONV( _Fima ⁽ⁱⁿ⁾ ) (6)

According to the above two formulas, namely matrix additivity and convolution weight convertibility, the convolution weight is weighted to the full connection during forward reasoning, as shown in Formula 7:

F _im = flatten(BN(FC_CONV(flatten(F _ima )))) (7)

Traditional MLP usually considers the extraction of global features, only mapping the input dimension to the corresponding dimension, without considering the location information and lacking local feature information. Different from the traditional MLP, this application uses the prior knowledge of multi-scale convolution, which can effectively help the network model extract the local features of the target. At the same time, in order to take into account the speed, in the process of forward reasoning, the weight of the convolution is integrated into the full connection by using a re-parameterization method.

In summary, during training, the multi-scale local feature extraction module and the multi-layer perceptron module are run in parallel; during inference, the convolution weights are integrated into the full connection. The MSCR submodule further helps the IMA-MSCR module to fully utilize spatial correlation while enabling the model to have multi-scale local features.

In order to prove the effectiveness of the IM-ELAN design, ablation experiments are performed on the original E-ELAN and the improved IM-ELAN structures on the high-speed target dataset. The experimental results are shown in Table 3.

Table 3 Ablation experiment table

It can be observed from the table that the improved IM-ELAN module improves the accuracy by 2.1% compared with E-ELAN, which further illustrates that the IM module effectively improves the high-speed target detection effect of the network model.

Step 3: Train the improved yolov7 network on the COCO large-scale object detection data to obtain a pre-trained model.

Step 4: Based on the pre-trained model, train the final model on the high-speed deformation target dataset.

The regression loss uses CIoU Loss, and the category and target confidence loss uses BCE Loss. Then the losses are fused and different weights are set to balance each loss value to prevent a certain loss value from accounting for too large a proportion, that is:

Total Loss = λ ₁ L _cls + λ ₂ L _obj + λ ₃ L _loc ;

Where _Lcla is the classification loss, _Lobj is the target confidence loss, _Lloc is the regression loss, _λ1 is the weight of the classification loss, the default value is 0.125, _λ2 is the weight of the target confidence loss, the default value is 0.1, and _λ3 is the weight of the regression loss, the default value is 0.05.

Step 5: Input the image to be detected that may contain high-speed deformation targets into the improved model to detect the targets. Specifically:

(1) High-speed target detection system reads target images in real time;

(2) The target image is input into the network model for forward reasoning;

(3) Enter the network to determine whether there is an object in the target image. If so, proceed to step (4); otherwise, proceed to step (5);

(4) The detection system marks the target and indicates that the target exists in the image;

(5) Check whether there are any unread images. If yes, return to step (1); otherwise, terminate the detection.

The visualization comparison of the PASCAL VOC dataset and the self-built high-speed target dataset is shown in Figure 7. IM-ELAN detects the train blocked by the tree and the ball deforming at high speed, proving that the IM-ELAN structure helps the model extract multi-scale local information and can better handle long-distance dependencies, and ultimately can better detect high-speed moving targets.

An embodiment of the present application also provides a computer device, including a memory, a processor, and computer executable instructions stored in the memory and executable on the processor, wherein the processor implements the above-mentioned high-speed target detection method when executing the instructions.

The embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores computer executable instructions, and is characterized in that the above-mentioned high-speed target detection method is implemented when the computer executable instructions are executed.

In summary, this application is based on the target detection model YOLOv7, adopts feature fusion and multi-scale local feature extraction, combines the advantages of low-level features and high-level features to achieve a more comprehensive feature description, and uses re-parameterization to extract local features of different scales without affecting the speed to enhance the understanding of the morphology and structure of high-speed targets in different sizes, thereby improving the detection accuracy of targets under high-speed motion.

Claims (10)

1. The high-speed target detection method based on multi-branch feature fusion and re-parameterization is characterized by comprising the following steps:

step 1, acquiring a high-speed deformation target data set;

step 2, improving an E-ELAN structure in a yolov7 network model, and improving the detection effect of the model on a high-speed target;

the E-ELAN structure mainly comprises four branches, an IM module is added to the first branch, the other three branches are convolution extraction branches, and features extracted by the four branches are fused;

the IM module comprises an IMA sub-module and an MSCR sub-module, and the IMA sub-module directly processes any two pairs of relations between input elements through a multi-head self-attention mechanism; the MSCR sub-module helps the model to extract multi-scale local information on the premise of not affecting the reasoning speed;

step 3, training an improved yolov7 network on COCO large-scale target detection data to obtain a pre-training model;

step 4, training a final model on the high-speed deformation target data set based on the pre-training model;

and 5, inputting an image to be detected, which possibly contains the high-speed deformation target, into a final model, and detecting the high-speed target.

2. The high-speed target detection method based on multi-branch feature fusion and re-parameterization according to claim 1, wherein: the high-speed deformation target data set in the step 1 is obtained as follows:

and acquiring videos of high-speed moving targets in real life, and manually acquiring deformation images of the targets under different conditions to obtain the high-speed deformation target data set.

3. The high-speed target detection method based on multi-branch feature fusion and re-parameterization according to claim 1, wherein: features extracted from the four branches are fused in step 2 using aggregation and scrambling.

4. The high-speed target detection method based on multi-branch feature fusion and re-parameterization according to claim 1, wherein: the IMA submodule constructs the input of Self-Attention by combining the width and height flattening of the input feature map, and after position coding, the input feature map is subjected to three weight sharing linear layers to obtain Q, K, V matrixes respectively.

5. The high-speed target detection method based on multi-branch feature fusion and re-parameterization according to claim 4, wherein: the K, V matrix is improved, and the average pooling with the pooling core of 3 and the step length of 2 is adopted to reduce the width and height dimensions of K, V to 1/3 of the original dimensions.

6. The multi-branch feature fusion and re-parameterization based high-speed target detection method according to any one of claims 1-5, wherein: the MSCR sub-module uses re-parameterization; during training, the multi-scale local feature extraction module and the multi-layer perceptron module are parallel; when reasoning, the convolved weights are fused into the full connection.

7. The high-speed target detection method based on multi-branch feature fusion and re-parameterization according to claim 1, wherein: in the training process of step 4, the regression Loss uses CIoU Loss, and the category and target confidence Loss uses BCE Loss.

8. The high-speed target detection method based on multi-branch feature fusion and re-parameterization according to claim 7, wherein: and fusing the losses, and setting different weights to balance each loss value so as to prevent the overlarge proportion of a certain loss value.

9. A computer device comprising a memory, a processor, and computer executable instructions stored on the memory and executable on the processor, wherein the processor, when executing the instructions, implements the high speed target detection method of any one of claims 1 to 8.

10. A computer storage medium storing computer executable instructions which when executed implement the high speed target detection method of any one of claims 1 to 8.