CN108564097B - Multi-scale target detection method based on deep convolutional neural network - Google Patents

Abstract

The invention discloses a multi-scale target detection method based on a deep convolutional neural network, which comprises the following steps: 1) acquiring data; 2) processing data; 3) constructing a model; 4) defining a loss function; 5) training a model; 6) and (5) verifying the model. The method combines the capability of extracting high-level semantic information of the image by the deep convolutional neural network, the capability of generating a candidate region by a region generation network, the capability of repairing and mapping a pooling layer of an interested region with content perception capability and the accurate classification capability of a multi-task classification network, and more accurately and efficiently completes multi-scale target detection.

Description

A multi-scale target detection method based on deep convolutional neural network

technical field

The invention relates to the technical field of computer image processing, in particular to a multi-scale target detection method based on a deep convolutional neural network.

Background technique

Object detection and recognition is one of the important topics in the field of computer vision computing. With the development of human science and technology, the important technology of target detection is constantly being fully utilized. People use it in various scenarios to achieve various expected goals, such as battlefield warning, security detection, traffic control, video surveillance, etc. aspect.

In recent years, with the rapid development of deep learning, deep convolutional neural networks have also made further breakthroughs in target detection and recognition technology. Using the deep convolutional neural network, the high-level semantic feature information of the picture can be extracted, and then the high-level semantic information can be used to detect the target. The deeper the neural network is, the more representative the feature information it expresses, but the problem is that it expresses very coarsely for small-scale objects, and even some features of small-scale objects are lost. The size and scale are very sensitive, and the feature information extracted by the neural network of objects of different sizes and scales has great differences, resulting in low accuracy of small-scale object detection, which greatly reduces the robustness and effectiveness of target detection.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and propose a multi-scale target detection method based on a deep convolutional neural network. It is good at detecting the limitation of homogeneous objects with large differences in size and scale.

In order to achieve the above purpose, the technical solution provided by the present invention is: a multi-scale target detection method based on a deep convolutional neural network, comprising the following steps:

1) Data acquisition

Training a deep convolutional neural network requires a lot of training data, so large-scale natural image or video image data needs to be used. If the obtained image data does not have label data, it needs to be manually labeled, and then divided into training data sets and verification data sets;

2) Data processing

Convert the image and label data of the image dataset into the format required for training a deep convolutional neural network through preprocessing;

3) Model building

According to the training target and the input and output form of the model, a deep convolutional neural network suitable for multi-scale target detection is constructed;

4) Define the loss function

Define the required loss function according to the training target and the architecture of the model;

5) Model training

Initialize the parameters of each layer of the network, iteratively input training samples, calculate the loss value of the network according to the loss function, and then calculate the gradient of the parameters of each network layer through back propagation, and update the parameters of each layer of the network through the stochastic gradient descent method. ;

6) Model Validation

Use the validation dataset to validate the trained model and test its generalization performance.

Described step 2) comprises the following steps:

2.1) Scale the images in the dataset to the size of m×n pixels in length and width, and the label data is also scaled to the corresponding size according to the corresponding ratio;

2.2) In the scaled image, randomly crop the place containing the label to obtain a rectangular image of a×b pixel size, a<=m, b<=n;

2.3) Randomly flip the cropped image horizontally with a probability of 0.5;

2.4) Convert the randomly flipped image from [0, 255] to the range of [-1, 1].

Described step 3) comprises the following steps:

3.1) Construct feature extraction network model

The feature extraction network is equivalent to an encoder, which is used to extract high-level semantic information from the input image and save it into a low-dimensional encoding. The input of the feature extraction network is the image processed in step 2). Some information will be lost in the deep coding, so in order to reduce the preservation of more information, low-dimensional and low-dimensional feature codes are output; in order to realize the transformation from input to a series of outputs, the feature extraction network contains multiple cascaded downsampling The downsampling layer consists of convolutional layers, batch regularization layers, nonlinear activation function layers, and pooling layers in series. The batch regularization layer plays a role in stabilizing and accelerating model training by normalizing the mean and standard deviation of the input samples of the same batch. The addition of a nonlinear activation function layer prevents the model from degenerating into a simple linear model. , to improve the description ability of the model, the role of the pooling layer is to reduce the size of the feature map, which can increase the receptive field of the convolution kernel;

3.2) Constructing a regional generative network model

The region generation network is responsible for finding all the objects and their positions in the input image; the region generation network inputs the feature map, and then maps each point on the feature map back to the original image, obtains the coordinates of these points, and then takes the coordinates around these points. Some candidate boxes of different sizes and different aspect ratios are set in advance, and the probability score that each box is an object is calculated; among them, the input of the region generation network is the output of step 3.1) feature extraction network, and a series of candidate boxes are output. The coordinates of and a series of candidate boxes are the probability scores of the object;

In order to realize a series of transformations from input to output, the region generation network model includes 3 functional structures in series, including convolutional layers, batch regularization layers, and nonlinear activation function layers. The first functional structure is to perform 3 × The feature fusion of 3 sizes, fuses the surrounding information, and serves as the input of the second and third functional structures respectively. The second functional structure realizes the output of the coordinate information of the rectangular frame, and the third functional structure realizes the output corresponding to the rectangular frame is an object. probability score;

3.3) Construct a content-aware region of interest pooling layer

The role of the content-aware region of interest pooling layer is to map the target region of the original image to the low-dimensional coding region obtained in step 3.1), and then pool to a fixed size, while the content-aware capability It is manifested in the following two aspects:

3.3.1) Information Completion

Information completion is to complete the information lost by small targets during low-dimensional coding, so that the detection of small targets is more accurate; for the low-dimensional coding feature map mapped from the target area of the original image to the step 3.1), if its One of the length and width is greater than z, the value of z is determined according to the network requirements, and the other is less than z, then it is enlarged to a square whose side length is max (length, width) by deconvolution, and then pooled Operation; if its length and width are both less than z, the length and width are enlarged to twice the original size by deconvolution, and then the pooling operation is performed; if both its length and width are greater than z, the subsequent pooling operation is performed directly ;

3.3.2) Division size

Divide the size of the target area of the original image output in the step 3.2), according to the average value of the area of all the label boxes in the prepared training data set, if the area of the rectangular box output in the step 3.2) is smaller than the average value, mark it as a small target output, and those greater than or equal to the mean are marked as large target outputs;

3.4) Constructing a multi-task classification network

The multi-task classification network is to identify large-scale and small-scale targets respectively, and prevent classification errors caused by different low-dimensional codes of large-scale and small-scale targets; according to the two types of rectangular boxes obtained in step 3.3), input the two classification networks respectively. ; The score of the output category of the classification network is used for the classification task, and the position of the refinement box is used for the regression task. In order to complete the classification and regression tasks, the network includes a fully connected layer, a nonlinear activation function layer, a signal loss layer, and a fully connected layer. The layer plays the role of mapping the learned "distributed feature representation" to the sample label space. The addition of the nonlinear activation function layer prevents the model from degenerating into a simple linear model and improves the description ability of the model. The signal loss layer is 0.5 Probability makes neurons not work, makes the training process converge faster, and prevents overfitting;

Finally, the output results of the size classification network are fused as the final output;

Described step 4) comprises the following steps:

4.1) Define the loss function of the region generation network

The region generation network is used to obtain the coordinates of the region of interest in the input image and the score of whether the region is foreground in the low-dimensional encoding, that is, the regression task and the classification task, and the loss function is defined to make the output box as close to the standard reference box as possible. ; therefore, the loss function for the regression task can be defined as the smoothed Manhattan distance loss loss (SmoothL1Loss), the formula is as follows:

Among them, L _reg is the regression loss, v and t respectively represent the position of the prediction frame and the position of its corresponding standard reference frame, x and y represent the upper left corner coordinate value, and w and h represent the width and height of the rectangular frame, respectively;

The loss function of the classification task is defined as the soft maximization loss (SoftmaxLoss), and the formula is as follows:

x' _i =x' _i -max(x' ₁ ,...,x' _n )

L _cls = _-logpi

Among them, x' is the output of the network, n is the total number of categories, p is the probability of each category, and L _cls is the classification loss;

4.2) Define the loss function of the classification network

The score of the output category of the classification network is used for the classification task, and the position of the refinement box is used for the regression task. The loss function is defined so that the output category is as consistent as possible with the label data, and the output box position is as close as possible. The position of the standard reference frame is the same; as in step 4.1), the loss function of the regression task can be defined as SmoothL1Loss, and the loss function of the classification task can be defined as SoftmaxLoss;

4.3) Define the total loss function

The two area generation network loss functions defined in step 4.1) and step 4.2) and the two classification network loss functions can be combined in a weighted manner, so that the network can complete the task of multi-scale target detection in the picture;

Described step 5) comprises the following steps:

5.1) Initialize the parameters of each layer of the model

The initialization of the parameters of each layer adopts the method used in the traditional deep convolutional neural network. For the convolutional layer parameters of the feature extraction network, the convolutional layer parameter values of the VGG16 network model pre-trained in ImageNet are used as the initial values. The convolutional layer in the region generation network and the fully connected layer of the classification network are initialized with a Gaussian distribution with a mean of 0 and a standard deviation of 0.02, while the parameters of all batch regularization layers are initialized with a mean of 1 and a standard deviation of The Gaussian distribution of 0.02 is initialized;

5.2) Train the network model

Randomly input the original image processed in step 2), obtain the corresponding low-dimensional coding features through the feature extraction network in step 3.1), generate a batch of candidate regions for the selection box in the region generation network in step 3.2), and pass step 4.1) Calculate the corresponding loss value, and then pass these regions through the content-aware region of interest pooling layer in step 3.3) to obtain another low-dimensional encoding feature of a fixed size, and then go through the classification network in step 3.4) to obtain the target's Classification and refinement of the marquee position, and calculate the corresponding loss value through step 4.2). Finally, the loss values of these two parts are processed in step 4.3) to obtain the final loss value, and the gradient of each layer parameter in the network model in step 3) can be obtained by back-propagation of this value, and then the stochastic gradient descent algorithm is used to make the obtained The gradient optimizes the parameters of each layer to realize a round of training of the network model;

5.3) Repeat step 5.2) until the ability of the network on multi-scale object detection reaches the desired goal.

The concrete practice of described step 6) is as follows:

Randomly take some original images from the verification data set, and after processing in step 2), input them to the network model trained in step 5), and let the network model detect the position of the target in the figure and predict its category. The corresponding label data are compared to judge the multi-scale target detection ability of the trained network model.

Compared with the prior art, the present invention has the following advantages and beneficial effects:

1. A new network layer is proposed--a content-aware region of interest pooling layer (CAROIPooling, Content-Aware ROIPooling layer), which realizes the mapping from the original image area to the low-dimensional coding area and then pools it to a fixed size. In particular, it will complete the information of the position-encoding feature maps of small-scale objects, so as to achieve the purpose of more accurate and comprehensive low-dimensional encoding feature maps, and this network layer is also applicable to other target detection networks.

2. A multi-branch target detection network is proposed. Different branches are responsible for large-scale and small-scale target detection tasks, so as to more accurately distinguish and detect large-scale objects and small-scale objects, breaking through the limitations of existing methods.

Description of drawings

Fig. 1 is the flow chart of the method of the present invention.

Figure 2 is a schematic diagram of the feature extraction network.

Figure 3 is a schematic diagram of the area generation network.

Figure 4 is a schematic diagram of the classification network.

Detailed ways

The present invention will be further described below in conjunction with specific embodiments.

As shown in FIG. 1 , the details of the multi-scale target detection method based on a deep convolutional neural network provided by this embodiment are as follows:

Step 1: Obtain the highway video data set, then obtain its video frames, perform manual annotation, and divide it into a training data set and a verification data set.

Step 2: Convert the image and label data of the image dataset into a format required for training a deep convolutional neural network through preprocessing, including the following steps:

In step 2.1, the images in the dataset are scaled to a size of 768 × 1344 pixels in length and width, and the label data is also scaled to the corresponding size according to the corresponding scale.

Step 2.2, in the scaled image, randomly crop the place containing the label to obtain a square image with a size of 768 × 768 pixels.

Step 2.3, randomly flip the cropped image horizontally with a probability of 0.5.

Step 2.4, transform the randomly flipped image from [0,255] to the range of [-1,1].

Step 3, building a network model, including a feature extraction network, a region generation network, and a multi-task classification network, including the following steps:

Step 3.1, construct a feature extraction network. The input to the feature extraction network is an image of 3 × 768 × 768, and the output is a series of low-dimensional encoded feature maps (512 × 48 × 48 and 512 × 24 × 24). The network includes multiple cascaded downsampling layers. The downsampling layer is composed of convolutional layers, batch regularization layers, nonlinear activation function layers, and pooling layers in series. The following is a specific example of a feature extraction network model, as shown in Figure 2.

Step 3.2, construct the region generation network. The input of the region generation network is the feature map of 512×48×48/512×24×24, and the output is the matrix information of 36×48×48/36×24×24 and 18×48×48/18×24×24. The network consists of 3 concatenated structures (convolutional layer, batch regularization layer, non-linear activation function layer). The following is a specific example of a region generative network model, as shown in Figure 3.

Step 3.3, construct a multi-task classification network. This example uses two classification networks, their input is a vector of length 512 × 7 × 7, and the output is a vector A of length 4 and a vector B of length 4, where the four values in vector A represent the background respectively. , car, bus, train class scores, the 4 values in the vector B represent the position of a marquee (coordinates x and y of the upper left point, and the width and height of the marquee w and h). The network includes a fully connected layer, a nonlinear activation function layer, and an information loss layer. The following is a specific example of the multi-task classification network model in this example, as shown in Figure 4.

Step 4, define the loss function of the region generation network and the classification network, including the following steps:

Step 4.1, define the loss function of the region generation network. Define the loss function to make the output box as close as possible to the position of the standard reference frame. Here, SmoothL1Loss is used to define the loss function to make the foreground score of the output box as close to the label data as possible, and SoftmaxLoss is used here.

Step 4.2, define the loss function of the classification network. The loss function is defined so that the foreground score of the output box is as close as possible to the label data, and the category is 4 categories. Define the loss function so that the output box is as close as possible to the position of the standard reference box.

Step 4.3, define the total loss function. Weighted summation of the above 4 losses. The formula is expressed as follows:

Loss=(w ₁ ×L _cls +w ₂ ×L _reg ) _{area generation network loss} +(w ₁ ×L _cls +w ₂ ×L _reg ) _{classification network loss}

Among them, Loss is the total loss value, w1, w2, w3, and w4 are the weights. In this example, w1=w2=w3=w4=1, L _cls is the classification loss value, and L _reg is the regression loss value.

Step 5, train the network model, including the following steps:

Step 5.1, initialize the parameters of each layer of the model, the convolutional layer parameters of the feature extraction network use the convolutional layer parameter values of the VGG16 network model pre-trained on a large database ImageNet as the initial value, and the convolutional layer and The fully connected layer of the classification network is initialized with a Gaussian distribution with a mean of 0 and a standard deviation of 0.02, while the parameters of all batch regularization layers are initialized with a Gaussian distribution with a mean of 1 and a standard deviation of 0.02.

Step 5.2, train the network model to randomly input the original image processed in step 2, input the network model of step 3, output the category information and the coordinate information of the regression box, and then calculate the final loss value through step 4, and pass this value through back propagation The gradient of the parameters of each layer in the network model in step 3 can be obtained, and then the obtained gradient can be used to optimize the parameters of each layer through the stochastic gradient descent algorithm, and then a round of training of the network model can be realized.

Step 5.3, continuous iterative training, that is, repeat step 5.2 until the ability of the network on multi-scale target detection reaches the expected target.

Step 6: Validate the trained model using the validation dataset to test its generalization performance.

The specific method is to randomly take some original images from the verification data set, and after processing in step 2, input them into the network model trained in step 5, and let the network model detect the position of the target in the picture and predict its category. The multi-scale target detection ability of the trained network model is judged by comparing the output results with the corresponding label data.

The above-mentioned embodiments are only preferred embodiments of the present invention, and are not intended to limit the scope of implementation of the present invention. Therefore, any changes made according to the shape and principle of the present invention should be included within the protection scope of the present invention.

Claims (3)

1. a multi-scale target detection method based on deep convolutional neural network, is characterized in that, comprises the following steps: 1) Data acquisition Training a deep convolutional neural network requires a lot of training data, so large-scale natural image or video image data needs to be used. If the obtained image data does not have label data, it needs to be manually labeled, and then divided into training data sets and verification data sets; 2) Data processing Convert the image and label data of the image dataset into the format required for training a deep convolutional neural network through preprocessing; 3) Model building According to the training target and the input and output form of the model, a deep convolutional neural network suitable for multi-scale target detection is constructed, including the following steps: 3.1) Construct feature extraction network model The feature extraction network is equivalent to an encoder, which is used to extract high-level semantic information from the input image and save it into a low-dimensional encoding. The input of the feature extraction network is the image processed in step 2). Some information will be lost in the deep coding, so in order to reduce the preservation of more information, low-dimensional and low-dimensional feature codes are output; in order to realize the transformation from input to a series of outputs, the feature extraction network contains multiple cascaded downsampling The downsampling layer consists of convolutional layers, batch regularization layers, nonlinear activation function layers, and pooling layers in series. The batch regularization layer plays a role in stabilizing and accelerating model training by normalizing the mean and standard deviation of the input samples of the same batch. The addition of a nonlinear activation function layer prevents the model from degenerating into a simple linear model. , to improve the description ability of the model, the role of the pooling layer is to reduce the size of the feature map, which can increase the receptive field of the convolution kernel; 3.2) Constructing a regional generative network model The region generation network is responsible for finding all the objects and their positions in the input image; the region generation network inputs the feature map, and then maps each point on the feature map back to the original image, obtains the coordinates of these points, and then takes the coordinates around these points. Some candidate boxes of different sizes and different aspect ratios are set in advance, and the probability score that each box is an object is calculated; among them, the input of the region generation network is the output of step 3.1) feature extraction network, and a series of candidate boxes are output. The coordinates of and a series of candidate boxes are the probability scores of the object; In order to realize a series of transformations from input to output, the region generation network model includes 3 functional structures in series, including convolutional layers, batch regularization layers, and nonlinear activation function layers. The first functional structure is to perform 3 × The feature fusion of 3 sizes, fuses the surrounding information, and serves as the input of the second and third functional structures respectively. The second functional structure realizes the output of the coordinate information of the rectangular frame, and the third functional structure realizes the output corresponding to the rectangular frame is an object. probability score; 3.3) Construct a content-aware region of interest pooling layer The role of the content-aware region of interest pooling layer is to map the target region of the original image to the low-dimensional coding region obtained in step 3.1), and then pool to a fixed size, while the content-aware capability It is manifested in the following two aspects: 3.3.1) Information Completion Information completion is to complete the information lost by small targets during low-dimensional coding, so that the detection of small targets is more accurate; for the low-dimensional coding feature map mapped from the target area of the original image to the step 3.1), if its One of the length and width is greater than z, the value of z is determined according to the network requirements, and the other is less than z, then it is enlarged to a square whose side length is max (length, width) by deconvolution, and then pooled Operation; if its length and width are both less than z, the length and width are enlarged to twice the original size by deconvolution, and then the pooling operation is performed; if both its length and width are greater than z, the subsequent pooling operation is performed directly ; 3.3.2) Division size Divide the size of the target area of the original image output in the step 3.2), according to the average value of the area of all the label boxes in the prepared training data set, if the area of the rectangular box output in the step 3.2) is smaller than the average value, mark it as a small target output, and those greater than or equal to the mean are marked as large target outputs; 3.4) Constructing a multi-task classification network The multi-task classification network is to identify large-scale and small-scale targets respectively, and prevent classification errors caused by different low-dimensional codes of large-scale and small-scale targets; according to the two types of rectangular boxes obtained in step 3.3), input the two classification networks respectively. ; The score of the output category of the classification network is used for the classification task, and the position of the refinement box is used for the regression task. In order to complete the classification and regression tasks, the network includes a fully connected layer, a nonlinear activation function layer, a signal loss layer, and a fully connected layer. The layer plays the role of mapping the learned "distributed feature representation" to the sample label space. The addition of the nonlinear activation function layer prevents the model from degenerating into a simple linear model and improves the description ability of the model. The signal loss layer is 0.5 Probability makes neurons not work, makes the training process converge faster, and prevents overfitting; Finally, the output results of the size classification network are fused as the final output; 4) Define the loss function According to the training target and the architecture of the model, the required loss function is defined, including the following steps: 4.1) Define the loss function of the region generation network The region generation network is used to obtain the coordinates of the region of interest in the input image and the score of whether the region is foreground in the low-dimensional encoding, that is, the regression task and the classification task, and the loss function is defined to make the output box close to the position of the standard reference box; Therefore, the loss function of the regression task can be defined as the smoothed Manhattan distance loss SmoothL1Loss, the formula is as follows:

Among them, L _reg is the regression loss, v and t respectively represent the position of the prediction frame and the position of its corresponding standard reference frame, x and y represent the upper left corner coordinate value, and w and h represent the width and height of the rectangular frame, respectively; The loss function of the classification task is defined as the soft maximization loss SoftmaxLoss, the formula is as follows: x' _i =x' _i -max(x' ₁ ,...,x' _n )

L _cls = _-logpi Among them, x' is the output of the network, n is the total number of categories, p is the probability of each category, and L _cls is the classification loss; 4.2) Define the loss function of the classification network The score of the output category of the classification network is used for the classification task, and the position of the refinement box is used for the regression task. The loss function is defined to make the output category and label data consistent, and the output box position and the standard reference frame position. Consistent; also as in step 4.1), the loss function of the regression task can be defined as SmoothL1Loss, and the loss function of the classification task can be defined as SoftmaxLoss; 4.3) Define the total loss function The two area generation network loss functions defined in step 4.1) and step 4.2) and the two classification network loss functions can be combined in a weighted manner, so that the network can complete the task of multi-scale target detection in the picture; 5) Model training Initialize the parameters of each layer of the network, iteratively input training samples, calculate the loss value of the network according to the loss function, and then calculate the gradient of the parameters of each network layer through back propagation, and update the parameters of each layer of the network through the stochastic gradient descent method. , including the following steps: 5.1) Initialize the parameters of each layer of the model The initialization of the parameters of each layer adopts the method used in the traditional deep convolutional neural network. For the convolutional layer parameters of the feature extraction network, the convolutional layer parameter values of the VGG16 network model pre-trained in ImageNet are used as the initial values. The convolutional layer in the region generation network and the fully connected layer of the classification network are initialized with a Gaussian distribution with a mean of 0 and a standard deviation of 0.02, while the parameters of all batch regularization layers are initialized with a mean of 1 and a standard deviation of The Gaussian distribution of 0.02 is initialized; 5.2) Train the network model Randomly input the original image processed in step 2), obtain the corresponding low-dimensional coding features through the feature extraction network in step 3.1), generate a batch of candidate regions for the selection box in the region generation network in step 3.2), and pass step 4.1) Calculate the corresponding loss value, and then pass these regions through the content-aware region of interest pooling layer in step 3.3) to obtain another low-dimensional encoding feature of a fixed size, and then go through the classification network in step 3.4) to obtain the target's Classification and refinement of the marquee position, and calculate the corresponding loss value through step 4.2); finally, the loss value of these two parts is processed in step 4.3) to obtain the final loss value, which can be obtained through back propagation. Step 3 ) the gradients of the parameters of each layer in the network model, and then optimize the parameters of each layer with the gradient obtained by the stochastic gradient descent algorithm, and then a round of training of the network model can be realized; 5.3) Repeat step 5.2) until the ability of the network on multi-scale target detection reaches the expected target; 6) Model Validation Use the validation dataset to validate the trained model and test its generalization performance. 2. a kind of multi-scale target detection method based on deep convolutional neural network according to claim 1, is characterized in that, described step 2) comprises the following steps: 2.1) Scale the images in the dataset to the size of m×n pixels in length and width, and the label data is also scaled to the corresponding size according to the corresponding ratio; 2.2) In the scaled image, randomly crop the place containing the label to obtain a rectangular image of a×b pixel size, a<=m, b<=n; 2.3) Randomly flip the cropped image horizontally with a probability of 0.5; 2.4) Convert the randomly flipped image from [0, 255] to the range of [-1, 1]. 3. a kind of multi-scale target detection method based on deep convolutional neural network according to claim 1, is characterized in that, the concrete practice of described step 6) is as follows: Randomly take out some original images from the verification data set, and after processing in step 2), input them into the network model trained in step 5), and let the network model detect the position of the target in the figure and predict its category, through the output result and The corresponding label data are compared to judge the multi-scale target detection ability of the trained network model.

Images