CN109685145B - Small object detection method based on deep learning and image processing - Google Patents

Abstract

The invention relates to the field of image processing, in particular to a small object detection method based on deep learning and image processing, which reserves more details by replacing the convolution of 5x5 of an original increment module with two convolution kernels of 3x3, meanwhile, in order to accelerate the training speed and the output consistency, BN is added at the end of each branch, namely Batch Normalization, and introduces a residual error network structure to increase the accuracy, in addition, the invention adopts deconvolution to enhance the context information of the upper layer and the bottom layer of the two adjacent layers, the result of deconvolution of the upper layer is aligned with the pixels of the bottom layer and added one by one, and the obtained new characteristic diagram is used as the detected characteristic diagram, the invention can improve the identification of the small object, and improve the accuracy rate of the traditional SSD for detecting the small object on the premise of not influencing the high FPS of the traditional SSD.

Description

A small object detection method based on deep learning and image processing

technical field

The invention relates to the field of image processing, and more particularly, to a small object detection method based on deep learning and image processing.

Background technique

At present, the commonly used algorithm for object detection is SSD, namely Single Shot MultiBoxDetection. SSD is an end-to-end detection framework based on deep learning. Its framework is mainly divided into two parts: the first part is the convolutional neural network (VGG16) located in the front end, which is used for feature extraction of the target, and the back end is a multi- The scale feature detection network performs feature extraction under different scale conditions on the feature layers generated by the previous network; Non maximum suppression (NMS) results.

However, since SSD uses a multi-scale detection method, this method will reduce the amount of calculation and have a high FPS, and because the detection is performed on feature maps of different scales, the convolution receptive field on the feature maps of different scales It will be different, especially in the high-level convolutional layer, its receptive field will be very large, and the extracted features are also very abstract, so it is very insensitive to the detection of small objects and details.

SUMMARY OF THE INVENTION

In order to solve the problem that the SSD detection algorithm in the prior art is not sensitive to small object detection, the present invention provides a small object detection method based on deep learning and image processing.

In order to achieve the above purpose of the invention, the technical scheme adopted is:

A small object detection method based on deep learning and image processing, comprising the following steps:

Step S1: Obtain a data set, which includes the marked object category information and the original picture of the coordinate information of the upper left (x _min , y _min ) and lower right (x _max , y _max ) points of the target frame, from the data Select a picture with label information arbitrarily in the training set of the set, and adjust the picture to the size of 300x300 as input;

Step S2: Divide the picture into 4 parts P1, P2, P3, P4 with a size of 150×150 along the horizontal (0, 150) (300, 150) and vertical directions (150, 0) (150, 300); Taking (75, 75) (225, 75) (75, 225) (225, 225) as the image of the coordinates of the four vertices as the fifth part P5;

Step S3: According to the two coordinate information (x _min , y _min ) and (x _max , y _max ) of the upper left and lower right of the target frame of each input picture, it is judged whether the object in the picture has been divided, and according to the object Modify the coordinates in the case of segmentation;

Step S4: Interpolate the picture by using the cubic interpolation method, so that the divided pictures P1, P2, P3, P4 and P5 with a size of 150x150 are the same size as the original picture of 300x300, and are named as F1, F2, F3, F4, F5, at the same time multiply the modified coordinates obtained in step S3 by 2 and update;

Step S5: Extract features from each of the five pictures F1, F2, F3, F4, and F5 through the VGG16 network, and then perform convolution with a 3x3x1024 convolution kernel to obtain a Conv6 feature map with a size of 19x19x1024, and then continue to use 1x1x1024 The size of the convolution kernel is convolved to obtain a Conv7 feature map of size 19x19x1024;

Step S6: Stack the 1x1, 3x3, and 3x3 convolution kernels together to form three branches. At the end of each branch, BN, that is, Batch Normalization, is added to perform batch normalization processing, and each branch is connected and fused while introducing residuals. Poor network structure, this structure is named IRBNet convolution structure;

Step S7: The Conv7 feature map with a size of 19x19x1024 obtained in step S5 is extracted through the IRBNet convolution structure, and the feature map Conv8 with a size of 10x10x512 is obtained; IRBNet convolution obtains a feature map Conv10 with a size of 3x3x256; Conv10 is convolved with IRBNet to obtain a feature map Conv11 with a size of 1x1x256;

Step S8: Deconvolve the high-level feature map with a convolution method with a convolution kernel of 3x3 and a stride of 4, so that it is doubled to make it the same size as the previous bottom layer, and then the pixels at the corresponding positions are phased one by one. Plus, the size of the new feature map obtained is consistent with the size of the underlying feature map, and the structure is named HDPANet;

Step S9: Add another feature map of size 19x19x1024 from the feature map Conv8 to Conv7 through step S8 to obtain the feature map Conv7D, and the feature map Conv9 obtains another feature map of size 10x10x512 through step S8 and add Conv8 to obtain the feature map Conv8D, the feature map Conv10 obtains another feature map of size 5x5x256 after step S8 and adds it to Conv9 to obtain the feature map Conv9D, and the feature map Conv11 obtains another feature map of size 3x3x256 after step S8 and Conv10 is added to obtain the feature map Conv10D ;

Step S10: Perform convolution with a 3x3 convolution kernel in the Conv4_3, Conv10D and Conv11 feature layers to obtain a feature map with a channel number of 4x (class+4), and use a 3x3 convolution kernel in the Conv7D, Conv8D, Conv9D feature layers. Perform convolution to obtain a feature map with a channel number of 6x (class+4);

Step S11: F1, F2, F3, F4, F5 obtain their corresponding loss function loss through steps S1 to S10; during backpropagation, the sum total_loss of the five loss function losses is optimized by the stochastic gradient descent algorithm, and the training is also set. The number of iterations epoch, the network parameters obtained when the total_loss is stable is the optimal solution;

Step S12: Select pictures without label information in the data set, perform steps S1 and S2 to perform picture segmentation, and put the segmented pictures into the network trained in steps S1 to S10, and then pass through the non-maximum value. Suppress and filter, and finally get the five pictures of F1, F2, F3, F4, F5 with the predicted category label and predicted coordinates (x _{pred_min} , y _{pred_min} ), (x _{pred_max} , y _{pred_max} );

Step S13: fuse the pictures according to the predicted category labels and predicted coordinates of the five pictures F1, F2, F3, F4, and F5, and the final result is the final result of the detection.

Preferably, the specific steps of modifying the coordinates in step S3 are as follows:

1) If _xmin <150, _xmax >150, and _ymin , _ymax <150 or _xmin <150, _xmax >150, _ymin , _ymax >150, then the object in the image is segmented along the vertical direction For the left and right parts, let the new coordinates be (x _min , y _min ), (150, y _max ) and (150, y _min ), (x _max , y _max ), the category information does not change;

2) If x _min , x _max < 150, y _min < 150, y _max > 150 or x _min , x _max > 150, y _min < 150, y _max > 150, the object in the image is horizontally divided into upper and lower parts. part, let the new coordinates be (x _min , y _min ), (x _max , 150), and (x _min , 150), (x _max , y _max ), and the category information does not change;

3) If x _min <150, y _min <150, x _max > 150, y _max > 150, it means that the object in the image is cut into four parts in the horizontal direction and the vertical direction together, let the new coordinates be (x _min , y _min ), (150, 150) and (150, y _min ), (x _max , 150) and (x _min , 150), (150, y _max ) and (150, 150), (x _max , y _max ), categories Information remains unchanged.

Preferably, the specific steps for obtaining the loss function loss and total_loss in the step S11 are as follows:

Loss is divided into confidence loss and location loss.

代表第i个预测框匹配到了第j个真实框为p类别的GT box；c表示置信度，l表示预测框，g表示真框；Among them, L(x,c,l,g) represents Loss, L _conf represents confidence loss, confidence loss is the softmax loss algorithm, L _loc represents location loss, and N is the number of priorboxes that match to GroundTruth in confidence loss; and the α parameter uses To adjust the ratio between confidence loss and location loss,

Represents that the i-th predicted box matches the j-th real box to the GT box of the p category; c represents the confidence, l represents the prediction box, and g represents the true box;

表示通过softmax方法产生的概率值，Pos表示是正样本，Neg表示负样本，N是confidence loss中匹配到到Ground Truth的prior box数量当

时成立，

表示第i个预测框属于类别p的概率、p表示类别中的第p个类别；

Represents the probability value generated by the softmax method, Pos represents a positive sample, Neg represents a negative sample, and N is the number of prior boxes matched to Ground Truth in confidence loss.

was established,

Indicates the probability that the i-th prediction box belongs to category p, and p indicates the p-th category in the category;

表示第i个预测框与第j个真实框关于类别k是否匹配，匹配为1，不匹配为0，

表示预测框，

表示真实框的偏移框；m表示属于(cx,cy,w,h)中的一个取值，

表示第j个真实框的偏移框的中心点x的坐标，

表示第j个真实框的偏移框的中心点y的坐标，

表示第j个真实框的偏移框的宽度，

表示第j个真实框的偏移框的高度，

表示第i个预测框的中心点x坐标偏移量，

表示第i个预测框的中心点y坐标偏移量，

表示第i个预测框宽度偏移量，

表示第i个预测框的高度偏移量，

表示第j个真实框中心点x坐标，

表示第j个真实框的中心点y坐标，

表示第j个真实框的宽度，

表示第j个真实框的高度；Among them, cx represents the x coordinate of the center point of the box, cy represents the y coordinate of the center point, w represents the width, h represents the height, i represents the ith prediction box, j represents the jth real box, and d _i represents the offset,

Indicates whether the i-th predicted frame matches the j-th real frame with respect to category k, the match is 1, the mismatch is 0,

represents the prediction box,

Represents the offset frame of the real frame; m represents a value belonging to (cx, cy, w, h),

Represents the coordinates of the center point x of the offset box of the jth ground truth box,

Represents the coordinates of the center point y of the offset box of the jth real box,

represents the width of the offset box of the jth ground truth box,

represents the height of the offset box of the jth ground truth box,

Indicates the x-coordinate offset of the center point of the i-th prediction box,

Indicates the y-coordinate offset of the center point of the i-th prediction frame,

Indicates the width offset of the i-th prediction box,

represents the height offset of the i-th prediction box,

Represents the x-coordinate of the center point of the jth real box,

Represents the y-coordinate of the center point of the jth real box,

represents the width of the jth ground truth box,

Represents the height of the jth ground truth box;

The five loss functions obtained by processing F1, F2, F3, F4, and F5 are recorded as L ₁ (x,c,l,g), L ₂ (x,c,l,g), L ₃ (x,c) ,l,g), L ₄ (x,c,l,g),L ₅ (x,c,l,g), the total loss function is written as:

Total_loss=L ₁ (x,c,l,g)+L ₂ (x,c,l,g)+L ₃ (x,c,l,g)+L ₄ (x,c,l,g)+ L ₅ (x,c,l,g).

Preferably, in the step S13, the specific steps of fusing the pictures are as follows:

(1) If the predicted coordinates of each picture of F1, F2, F3, and F4 are x _{pred_max} , y _{pred_max} <300 and x _{pred_min} , y _{pred_min} > 0, then combine F1, F2, F3, F4 into one picture according to the original position, Then reduce the size of the fused image by 4 times to obtain the size of the original image of 300x300, and reduce the predicted coordinates by 4 times, and the final result is the final result of the detection;

(2) Detect the categories label1 and label2 of the objects on the boundary of the left and right parts. If label1 is equal to label2, it means the same category. Compare the size of the coordinate information of the two objects, and extend to the smaller direction based on the larger frame (x _max -x _min ) length, then fill in the four sides of the picture, combine F1, F2, F3, F4 into one picture according to the original position, and reduce the size of the fused whole picture by 4 times to get the original The size of the picture is 300x300, and the modified coordinates are reduced by 4 times, and the final result is the final result of the detection;

(3) Detect the categories label1 and label2 of the objects on the boundary of the upper and lower parts. If label1 is equal to label2, it indicates the same category. Compare the size of the coordinate information of the two objects, and extend y _max in the direction of the smaller frame based on the larger frame. Subtract the length of y _min , and then fill it up; reduce the size of the entire image after fusion by 4 times to obtain the size of the original image 300x300, and reduce the modified coordinates by 4 times, and the final result is the final result of the detection ;

(4) If the predicted coordinates of each picture of F1, F2, F3, F4 (x _{pred_min} , y _{pred_min} )=(300, 300) or (x _{pred_max} , y _{pred_max} )=(300, 300), it means that the object is top left, bottom left, The upper right and lower right parts are divided at the same time; the detection result of the picture F5 in the middle part is used as the detection result of the intermediate object, and the size of the fused whole picture is reduced by 4 times to obtain the size of the original picture 300x300, and the obtained coordinates The information is reduced by 4 times at the same time, and the final result is the final result of the detection.

Preferably, the α=1.

Compared with the prior art, the beneficial effects of the present invention are:

The present invention retains more details by replacing the 5x5 convolution of the original Inception module with two 3x3 convolution kernels. At the same time, in order to speed up the training speed and output consistency, BN is added at the end of each branch, namely BatchNormalization, performing batch normalization processing, and introducing a residual network structure at the same time to increase the accuracy, and the present invention uses deconvolution to enhance the context information of the upper layer and the lower layer of the two adjacent layers, and the result of the upper layer deconvolution is combined with the lower layer volume. The accumulated pixels are aligned one by one, and the obtained new feature map is used as the detected feature map, which can improve the recognition of small objects. The present invention improves the detection of small objects by traditional SSD without affecting the high FPS of traditional SSD. 's accuracy.

Description of drawings

FIG. 1 is a flow chart of the present invention.

FIG. 2 is a schematic diagram of a segmentation point for segmentation of an image according to the present invention.

FIG. 3 is a flow chart of the image segmentation network of the present invention.

FIG. 4 is a structural diagram of the residual network of the present invention.

Figure 5 is the structure diagram of IRBNet.

Figure 6 is a flowchart of high-level deconvolutional pixel addition.

FIG. 7 is a flowchart of solving the predicted category label and predicted coordinates.

Detailed ways

The accompanying drawings are for illustrative purposes only, and should not be construed as limitations on this patent;

The present invention will be further elaborated below in conjunction with the accompanying drawings and embodiments.

Example 1

As shown in Figures 1 to 7, a small object detection method based on deep learning and image processing includes the following steps:

Step S1: Obtain a data set, which includes the marked object category information and the original picture of the coordinate information of the upper left (x _min , y _min ) and lower right (x _max , y _max ) points of the target frame, from the data Select a picture with label information arbitrarily in the training set of the set, and adjust the picture to the size of 300x300 as input;

Step S2: As shown in FIG. 2, the picture is divided into 4 parts P1, P2 with a size of 150×150 along the horizontal (0, 150) (300, 150) and vertical directions (150, 0) (150, 300), P3, P4; In addition, take (75, 75) (225, 75) (75, 225) (225, 225) as the image of the coordinates of the four vertices as the fifth part P5;

Step S3: According to the two coordinate information (x _min , y _min ) and (x _max , y _max ) of the upper left and lower right of the target frame of each input picture, it is judged whether the object in the picture has been divided, and according to the object Modify the coordinates in the case of segmentation;

Step S4: As shown in Figure 3, use the cubic interpolation method to interpolate the picture, so that the divided pictures P1, P2, P3, P4, P5 with a size of 150x150 are the same size as the original picture 300x300, and are named as F1, F2, F3, F4, F5, at the same time multiply the modified coordinates obtained in step S3 by 2 and update;

Step S5: Extract features from each of the five pictures F1, F2, F3, F4, and F5 through the VGG16 network, and then perform convolution with a 3x3x1024 convolution kernel to obtain a Conv6 feature map with a size of 19x19x1024, and then continue to use 1x1x1024 The size of the convolution kernel is convolved to obtain a Conv7 feature map of size 19x19x1024;

Step S6: As shown in Figure 4 and Figure 5, the 1x1, 3x3, and 3x3 convolution kernels are stacked together to form three branches, and BN is added at the end of each branch, that is, Batch Normalization for batch normalization. , each branch connection is fused and the residual network structure is introduced at the same time, and the structure is named IRBNet convolution structure;

Step S7: The Conv7 feature map with a size of 19x19x1024 obtained in step S5 is extracted through the IRBNet convolution structure, and the feature map Conv8 with a size of 10x10x512 is obtained; IRBNet convolution obtains a feature map Conv10 with a size of 3x3x256; Conv10 is convolved with IRBNet to obtain a feature map Conv11 with a size of 1x1x256;

Step S8: As shown in Figure 6, the high-level feature map is deconvolved with a convolution method with a convolution kernel of 3x3 and a stride of 4, so that it is doubled to make it the same size as the previous bottom layer, and then the corresponding position The pixels are added one by one, and the size of the new feature map obtained is consistent with the size of the underlying feature map, and the structure is named HDPANet;

Step S9: Add another feature map of size 19x19x1024 from the feature map Conv8 to Conv7 through step S8 to obtain the feature map Conv7D, and the feature map Conv9 obtains another feature map of size 10x10x512 through step S8 and add Conv8 to obtain the feature map Conv8D, the feature map Conv10 obtains another feature map of size 5x5x256 after step S8 and adds it to Conv9 to obtain the feature map Conv9D, and the feature map Conv11 obtains another feature map of size 3x3x256 after step S8 and Conv10 is added to obtain the feature map Conv10D ;

Step S10: Perform convolution with a 3x3 convolution kernel in the Conv4_3, Conv10D and Conv11 feature layers to obtain a feature map with a channel number of 4x (class+4), and use a 3x3 convolution kernel in the Conv7D, Conv8D, Conv9D feature layers. Perform convolution to obtain a feature map with a channel number of 6x (class+4);

Step S11: F1, F2, F3, F4, F5 obtain their corresponding loss function loss through steps S1 to S10; during backpropagation, the sum total_loss of the five loss function losses is optimized by the stochastic gradient descent algorithm, and the training is also set. The number of iterations epoch, the network parameters obtained when the total_loss is stable is the optimal solution;

Step S12: As shown in Figure 7, select pictures without label information in the data set, perform steps S1 and S2 to perform picture segmentation, and put the segmented pictures into the network trained in steps S1 to S10, Then filter by non-maximum value suppression, and finally get the predicted category label and predicted coordinates (x _{pred_min} , y _{pred_min} ), (x _{pred_max} , y _{pred_max} ) of the five pictures F1, F2, F3, F4, and F5;

Step S13: fuse the pictures according to the predicted category labels and predicted coordinates of the five pictures F1, F2, F3, F4, and F5, and the final result is the final result of the detection.

As a preferred embodiment, the specific steps of modifying the coordinates in step S3 are as follows:

1) If _xmin <150, _xmax >150, and _ymin , _ymax <150 or _xmin <150, _xmax >150, _ymin , _ymax >150, then the object in the image is segmented along the vertical direction For the left and right parts, let the new coordinates be (x _min , y _min ), (150, y _max ) and (150, y _min ), (x _max , y _max ), the category information does not change;

2) If x _min , x _max < 150, y _min < 150, y _max > 150 or x _min , x _max > 150, y _min < 150, y _max > 150, the object in the image is horizontally divided into upper and lower parts. part, let the new coordinates be (x _min , y _min ), (x _max , 150), and (x _min , 150), (x _max , y _max ), and the category information does not change;

3) If x _min < 150, y _min < 150, x _max > 150, y _max > 150, it means that the object in the image is cut into four parts in the horizontal and vertical directions, and the new coordinates are (x _min , y _min ), (150, 150) and (150, y _min ), (x _max , 150) and (x _min , 150), (150, y _max ) and (150, 150), (x _max , y _max ), categories Information remains unchanged.

As a preferred embodiment, the specific steps for obtaining the loss function loss and total_loss in the step S11 are as follows:

Loss is divided into confidence loss and location loss.

代表第i个预测框匹配到了第j个真实框为p类别的GT box；c表示置信度，l表示预测框，g表示真框；Among them, L(x, c, l, g) represents Loss, L _conf represents confidence loss, confidence loss is the softmax loss algorithm, L _loc represents location loss, and N is the number of priorboxes that match to GroundTruth in confidence loss; and the α parameter uses To adjust the ratio between confidence loss and location loss,

Represents that the i-th predicted box matches the j-th real box to the GT box of the p category; c represents the confidence, l represents the prediction box, and g represents the true box;

表示通过softmax方法产生的概率值，Pos表示是正样本，Neg表示负样本，N是confidence loss中匹配到到Ground Truth的prior box数量当

时成立，

表示第i个预测框属于类别p的概率、p表示类别中的第p个类别；

Indicates the probability value generated by the softmax method, Pos indicates a positive sample, Neg indicates a negative sample, and N is the number of prior boxes matched to Ground Truth in confidence loss.

was established,

Indicates the probability that the i-th prediction box belongs to category p, and p indicates the p-th category in the category;

表示第i个预测框与第j个真实框关于类别k是否匹配，匹配为1，不匹配为0，

表示预测框，

表示真实框的偏移框；m表示属于(cx,cy,w,h)中的一个取值，

表示第j个真实框的偏移框的中心点x的坐标，

表示第j个真实框的偏移框的中心点y的坐标，

表示第j个真实框的偏移框的宽度，

表示第j个真实框的偏移框的高度，

表示第i个预测框的中心点x坐标偏移量，

表示第i个预测框的中心点y坐标偏移量，

表示第i个预测框宽度偏移量，

表示第i个预测框的高度偏移量，

表示第j个真实框中心点x坐标，

表示第j个真实框的中心点y坐标，

表示第j个真实框的宽度，

表示第j个真实框的高度；Among them, cx represents the x coordinate of the center point of the box, cy represents the y coordinate of the center point, w represents the width, h represents the height, i represents the ith prediction box, j represents the jth real box, and d _i represents the offset,

Indicates whether the i-th predicted frame matches the j-th real frame with respect to category k, the match is 1, the mismatch is 0,

represents the prediction box,

Represents the offset frame of the real frame; m represents a value belonging to (cx, cy, w, h),

Represents the coordinates of the center point x of the offset box of the jth ground truth box,

Represents the coordinates of the center point y of the offset box of the jth real box,

represents the width of the offset box of the jth ground truth box,

represents the height of the offset box of the jth ground truth box,

Represents the x-coordinate offset of the center point of the i-th prediction box,

Indicates the y-coordinate offset of the center point of the i-th prediction frame,

Indicates the width offset of the i-th prediction box,

represents the height offset of the i-th prediction box,

Represents the x-coordinate of the center point of the jth real box,

Represents the y-coordinate of the center point of the jth real box,

represents the width of the jth ground truth box,

Represents the height of the jth ground truth box;

The five loss functions obtained by processing F1, F2, F3, F4, and F5 are recorded as L ₁ (x,c,l,g), L ₂ (x,c,l,g), L ₃ (x,c) ,l,g), L ₄ (x,c,l,g),L ₅ (x,c,l,g), the total loss function is written as:

Total_loss=L ₁ (x,c,l,g)+L ₂ (x,c,l,g)+L ₃ (x,c,l,g)+L ₄ (x,c,l,g)+ L ₅ (x,c,l,g).

As a preferred embodiment, the specific steps of merging the pictures in the step S13 are as follows:

(1) If the predicted coordinates of each picture of F1, F2, F3, and F4 are x _{pred_max} , y _{pred_max} <300 and x _{pred_min} , y _{pred_min} > 0, then combine F1, F2, F3, F4 into one picture according to the original position, Then reduce the size of the fused image by 4 times to obtain the size of the original image of 300x300, and reduce the predicted coordinates by 4 times, and the final result is the final result of the detection;

(2) Detect the categories label1 and label2 of the objects on the boundary of the left and right parts. If label1 is equal to label2, it means the same category. Compare the size of the coordinate information of the two objects, and extend to the smaller direction based on the larger frame (x _max -x _min ) length, then fill in the four sides of the picture, combine F1, F2, F3, F4 into one picture according to the original position, and reduce the size of the fused whole picture by 4 times to get the original The size of the picture is 300x300, and the modified coordinates are reduced by 4 times, and the final result is the final result of the detection;

(3) Detect the categories label1 and label2 of the objects on the boundary of the upper and lower parts. If label1 is equal to label2, it indicates the same category. Compare the size of the coordinate information of the two objects, and extend y _max in the direction of the smaller frame based on the larger frame. Subtract the length of y _min , and then fill it up; reduce the size of the entire image after fusion by 4 times to obtain the size of the original image 300x300, and reduce the modified coordinates by 4 times, and the final result is the final result of the detection ;

(4) If the predicted coordinates of each picture of F1, F2, F3, F4 (x _{pred_min} , y _{pred_min} )=(300, 300) or (x _{pred_max} , y _{pred_max} )=(300, 300), it means that the object is top left, bottom left, The upper right and lower right parts are divided at the same time; the detection result of the picture F5 in the middle part is used as the detection result of the intermediate object, and the size of the fused whole picture is reduced by 4 times to obtain the size of the original picture 300x300, and the obtained coordinates The information is reduced by 4 times at the same time, and the final result is the final result of the detection.

As a preferred embodiment, the α=1.

Obviously, the above-mentioned embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those of ordinary skill in the art, changes or modifications in other different forms can also be made on the basis of the above description. There is no need and cannot be exhaustive of all implementations here. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the claims of the present invention.

Claims (5)

1. a small object detection method based on deep learning and image processing, is characterized in that, comprises the following steps: Step S1: Obtain a data set, which includes the marked object category information and the original picture of the coordinate information of the upper left (x _min , y _min ) and lower right (x _max , y _max ) points of the target frame, from the data Select a picture with label information arbitrarily in the training set of the set, and adjust the picture to the size of 300x300 as input; Step S2: Divide the picture into 4 parts P1, P2, P3, P4 with a size of 150×150 along the horizontal (0, 150) (300, 150) and vertical directions (150, 0) (150, 300); Taking (75, 75) (225, 75) (75, 225) (225, 225) as the image of the coordinates of the four vertices as the fifth part P5; Step S3: According to the two coordinate information (x _min , y _min ) and (x _max , y _max ) of the upper left and lower right of the target frame of each input picture, it is judged whether the object in the picture has been divided, and according to the object Modify the coordinates in the case of segmentation; Step S4: Interpolate the picture by using the cubic interpolation method, so that the divided pictures P1, P2, P3, P4 and P5 with a size of 150x150 are the same size as the original picture of 300x300, and are named as F1, F2, F3, F4, F5, at the same time multiply the modified coordinates obtained in step S3 by 2 and update; Step S5: Extract features from each of the five pictures F1, F2, F3, F4, and F5 through the VGG16 network, and then perform convolution with a 3x3x1024 convolution kernel to obtain a Conv6 feature map with a size of 19x19x1024, and then continue to use 1x1x1024 The size of the convolution kernel is convolved to obtain a Conv7 feature map of size 19x19x1024; Step S6: Stack the 1x1, 3x3, and 3x3 convolution kernels together to form three branches. At the end of each branch, BN, that is, Batch Normalization, is added to perform batch normalization processing, and each branch is connected and fused while introducing residuals. Poor network structure, this structure is named IRBNet convolution structure; Step S7: The Conv7 feature map with a size of 19x19x1024 obtained in step S6, extracts features through the IRBNet convolution structure, and obtains a feature map Conv8 with a size of 10x10x512; IRBNet convolution obtains a feature map Conv10 with a size of 3x3x256; Conv10 is convolved with IRBNet to obtain a feature map Conv11 with a size of 1x1x256; Step S8: Deconvolve the high-level feature map with a convolution method with a convolution kernel of 3x3 and a stride of 4, so that it is doubled to make it the same size as the previous bottom layer, and then the pixels at the corresponding positions are phased one by one. Plus, the size of the new feature map obtained is consistent with the size of the underlying feature map, and the structure is named HDPANet; Step S9: Add another feature map of size 19x19x1024 from the feature map Conv8 to Conv7 through step S8 to obtain the feature map Conv7D, and the feature map Conv9 obtains another feature map of size 10x10x512 through step S8 and add Conv8 to obtain the feature map Conv8D, the feature map Conv10 obtains another feature map of size 5x5x256 after step S8 and adds it to Conv9 to obtain the feature map Conv9D, and the feature map Conv11 obtains another feature map of size 3x3x256 after step S8 and Conv10 is added to obtain the feature map Conv10D ; Step S10: Perform convolution with a 3x3 convolution kernel in the Conv4_3, Conv10D and Conv11 feature layers to obtain a feature map with a channel number of 4x (class+4), and use a 3x3 convolution kernel in the Conv7D, Conv8D, Conv9D feature layers. Perform convolution to obtain a feature map with a channel number of 6x (class+4); Step S11: F1, F2, F3, F4, F5 obtain their corresponding loss function loss through steps S1 to S10; during backpropagation, the sum total_loss of the five loss function losses is optimized by the stochastic gradient descent algorithm, and the training is also set. The number of iterations epoch, the network parameters obtained when the total_loss is stable is the optimal solution; Step S12: Select pictures without label information in the data set, perform steps S1 and S2 to perform picture segmentation, and put the segmented pictures into the network trained in steps S1 to S10, and then pass through the non-maximum value. Suppress and filter, and finally get the five pictures of F1, F2, F3, F4, F5 with the predicted category label and predicted coordinates (x _{pred_min} , y _{pred_min} ), (x _{pred_max} , y _{pred_max} ); Step S13: fuse the pictures according to the predicted category labels and predicted coordinates of the five pictures F1, F2, F3, F4, and F5, and the final result is the final result of the detection. 2. according to a kind of small object detection method based on deep learning and image processing described in claim 1, it is characterized in that, the concrete steps of described step S3 modifying coordinates are as follows: 1) If _xmin <150, _xmax >150, and _ymin , _ymax <150 or _xmin <150, _xmax >150, _ymin , _ymax >150, then the object in the image is segmented along the vertical direction For the left and right parts, let the new coordinates be (x _min , y _min ), (150, y _max ) and (150, y _min ), (x _max , y _max ), the category information does not change; 2) If x _min , x _max < 150, y _min < 150, y _max > 150 or x _min , x _max > 150, y _min < 150, y _max > 150, the object in the image is horizontally divided into upper and lower parts. part, let the new coordinates be (x _min , y _min ), (x _max , 150), and (x _min , 150), (x _max , y _max ), and the category information does not change; 3) If x _min <150, y _min <150, x _max > 150, y _max > 150, it means that the object in the image is cut into four parts in the horizontal direction and the vertical direction together, let the new coordinates be (x _min , y _min ), (150, 150) and (150, y _min ), (x _max , 150) and (x _min , 150), (150, y _max ) and (150, 150), (x _max , y _max ), category Information remains unchanged. 3. according to a kind of small object detection method based on deep learning and image processing described in claim 2, it is characterized in that, described step S11 seeks the concrete steps of loss function loss and total_loss as follows: Loss is divided into confidence loss and location loss.

Among them, L(x,c,l,g) means Loss, L _conf means confidence loss, confidence loss is the softmaxloss algorithm, L _loc means location loss, N is the number of priorboxes matched to GroundTruth in confidence loss; and the α parameter is used for Adjust the ratio between confidence loss and location loss,

Represents that the i-th predicted box matches the j-th real box to the GT box of the p category; c represents the confidence, l represents the prediction box, and g represents the true box;

Indicates the probability value generated by the softmax method, Pos indicates a positive sample, Neg indicates a negative sample, and N is the number of prior boxes matched to Ground Truth in confidence loss.

was established,

Indicates the probability that the i-th prediction box belongs to category p, and p indicates the p-th category in the category;

Among them, cx represents the x coordinate of the center point of the box, cy represents the y coordinate of the center point, w represents the width, h represents the height, i represents the ith prediction box, j represents the jth real box, and d _i represents the offset,

Indicates whether the i-th predicted frame matches the j-th real frame with respect to category k, the match is 1, the mismatch is 0,

represents the prediction box,

Represents the offset frame of the real frame; m represents a value belonging to (cx, cy, w, h),

Represents the coordinates of the center point x of the offset box of the jth ground truth box,

Represents the coordinates of the center point y of the offset box of the jth real box,

represents the width of the offset box of the jth ground truth box,

represents the height of the offset box of the jth ground truth box,

Indicates the x-coordinate offset of the center point of the i-th prediction box,

Indicates the y-coordinate offset of the center point of the i-th prediction frame,

Indicates the width offset of the i-th prediction box,

represents the height offset of the i-th prediction box,

Represents the x-coordinate of the center point of the jth real box,

Represents the y-coordinate of the center point of the jth real box,

represents the width of the jth ground truth box,

Represents the height of the jth ground truth box; The five loss functions obtained by processing F1, F2, F3, F4, and F5 are recorded as L ₁ (x,c,l,g), L ₂ (x,c,l,g), L ₃ (x,c) ,l,g), L ₄ (x,c,l,g),L ₅ (x,c,l,g), the total loss function is written as: Total_loss=L ₁ (x,c,l,g)+L ₂ (x,c,l,g)+L ₃ (x,c,l,g)+L ₄ (x,c,l,g)+ L ₅ (x,c,l,g). 4. according to a kind of small object detection method based on deep learning and image processing described in claim 3, it is characterized in that, described step S13 carries out the concrete steps of picture fusion as follows: (1) If the predicted coordinates of each picture of F1, F2, F3, and F4 are x _{pred_max} , y _{pred_max} <300 and x _{pred_min} , y _{pred_min} > 0, then combine F1, F2, F3, F4 into one picture according to the original position, Then reduce the size of the fused image by 4 times to obtain the size of the original image of 300x300, and reduce the predicted coordinates by 4 times, and the final result is the final result of the detection; (2) Detect the categories label1 and label2 of the objects on the boundary of the left and right parts. If label1 is equal to label2, it means the same category. Compare the size of the coordinate information of the two objects, and extend to the smaller direction based on the larger frame (x _max -x _min ) length, then fill in the four sides of the picture, combine F1, F2, F3, F4 into one picture according to the original position, and reduce the size of the fused whole picture by 4 times to get the original The size of the picture is 300x300, and the modified coordinates are reduced by 4 times, and the final result is the final result of the detection; (3) Detect the categories label1 and label2 of the objects on the boundary of the upper and lower parts. If label1 is equal to label2, it indicates the same category. Compare the size of the coordinate information of the two objects, and extend y _max in the direction of the smaller frame based on the larger frame. Subtract the length of y _min , and then fill it up; reduce the size of the entire image after fusion by 4 times to obtain the size of the original image 300x300, and reduce the modified coordinates by 4 times, and the final result is the final result of the detection ; (4) If the predicted coordinates of each picture of F1, F2, F3, F4 (x _{pred_min} , y _{pred_min} )=(300, 300) or (x _{pred_max} , y _{pred_max} )=(300, 300), it means that the object is top left, bottom left, The upper right and lower right parts are divided at the same time; the detection result of the picture F5 in the middle part is used as the detection result of the intermediate object, and the size of the fused whole picture is reduced by 4 times to obtain the size of the original picture 300x300, and the obtained coordinates The information is reduced by 4 times at the same time, and the final result is the final result of the detection. 5 . The small object detection method based on deep learning and image processing according to claim 4 , wherein the α=1. 6 .

Images