CN110210603A - Counter model construction method, method of counting and the device of crowd - Google Patents

Abstract

The present invention relates to a crowd counting model construction method, counting method and device. Firstly, the crowd images in the pre-stored data set are marked with heads to obtain a crowd density map, and the crowd images and the crowd density map corresponding to the crowd images are combined. Combining to obtain the target data set; then performing data amplification processing on the training set obtained by dividing the target data set according to the first preset ratio to obtain the target training set; finally using the training crowd image and the training crowd density map in the target training set, The multi-scale feature aggregation convolutional neural network model after parameter initialization is trained to obtain the counting model of the crowd. By adopting the technical solution of the present invention, when the model is trained and applied, it can not only solve the problem of the scale change of the characters in the image, but also reduce the amount of computation, improve the recognition efficiency, and do not need to use perspective images in the training process, which improves the accuracy of the model. applicability.

Description

Crowd counting model construction method, counting method and device

technical field

The invention relates to the field of computer vision, in particular to a method for constructing a crowd counting model, a counting method and a device.

Background technique

Crowd counting is to calculate the number of heads from images or video frames, and crowd density is the distribution of crowds in a certain time and space. Accurately estimating the number of crowds and crowd density is one of the important indicators to measure the quality of a security system. Accurately estimating the number and density of crowds plays an extremely important role in public safety and traffic control. At the same time, by analyzing the distribution of crowd density in large shopping malls, customers' purchasing preferences and tendencies can be obtained and potential commercial value can be explored.

In the process of image processing of crowds, the scale change of the characters in the image is difficult to solve by the traditional algorithm based on single-column convolutional neural network, and multi-column network architecture is usually used for processing, but multi-column network is more computationally intensive than single-column network. Heavy and low recognition efficiency. And the existing model training process needs to use the perspective view on the training scene and the test scene, but it is difficult to obtain the perspective view in the actual application, which makes the applicability of the model poor.

Therefore, how to reduce the amount of computation, improve the recognition efficiency and the applicability of the model is a technical problem to be solved urgently by those skilled in the art.

Contents of the invention

In view of this, the object of the present invention is to provide a crowd counting model building method, counting method and device to solve the problems of large amount of computation, low recognition efficiency and poor model applicability in the prior art.

To achieve the above object, the present invention adopts the following technical solutions:

A method for constructing a population counting model, comprising:

Perform head annotation processing on the crowd images in the pre-stored data set to obtain a crowd density map;

combining the crowd image with a crowd density map corresponding to the crowd image to obtain a target data set;

performing data amplification processing on the training set obtained by dividing the target data set according to the first preset ratio to obtain the target training set;

Using the images of the training crowd and the density map of the training crowd in the target training set, the multi-scale feature aggregation convolutional neural network model after parameter initialization is trained to obtain the counting model of the crowd.

Further, in the method described above, performing data amplification processing on the training set obtained by dividing the target data set according to the first preset ratio to obtain the target training set includes:

Each of the original training crowd images in the training set and the original training crowd density map corresponding to the original training crowd images are respectively cropped by a preset cropping method to obtain at least two cropped images and at least two crop density map;

performing flipping processing on the cropped image to obtain a flipped image, and flipping the cropped density map to obtain a flipped density map;

According to the second preset ratio of the original training crowd image, the cropped image and the flipped image are enlarged to obtain an enlarged image, and the cropped density map and the flipped density map are enlarged to obtain an enlarged density map ;

storing the enlarged image and the enlarged density map in the training set to obtain a target training set;

Wherein, the training crowd image includes the original training crowd image and the enlarged image; the training crowd density map includes the original training crowd density map and the enlarged density map.

Further, in the above-mentioned method, the multi-scale feature aggregation convolutional neural network model after parameter initialization is trained by using the training crowd image and the training crowd density map in the target training set to obtain the crowd's Counting models, including:

Input the training crowd image and the training crowd density map in the target training set into the multi-scale feature aggregation convolutional neural network model after parameter initialization to perform first-level training to obtain the first training Model;

Using the Euclidean distance loss formula to optimize the model parameters of the first training model at the first level to obtain the first optimization model;

Iteratively executing the first-level training and the first-level optimization processes according to a preset first number of iterations to obtain a first target model;

Inputting the training crowd image and the training crowd density map into the first target model to perform second-level training to obtain a second training model;

Using the Euclidean distance loss formula and the relative number of people loss formula to perform second-level optimization on the model parameters of the second training model to obtain a second optimization model;

performing iterative execution on the second-level training and the second-level optimization process according to the preset second number of iterations to obtain a second target model;

The second target model is used as the population count model.

Further, in the above-mentioned method, before using the second target model as the counting model of the crowd, it also includes:

inputting the test crowd images in the test set obtained by dividing the target data set according to the first preset ratio into the second target model to obtain a target crowd density map;

Determine the test accuracy of the model according to the target crowd density map and the test crowd density map corresponding to the test crowd images in the test set;

Judging whether the test accuracy rate is greater than the preset accuracy rate;

Correspondingly, said using the second target model as the population counting model includes:

If the test accuracy rate is greater than the preset accuracy rate, the second target model is used as the population count model.

Further, in the method described above, the Euclidean distance loss formula is:

Wherein, Θ represents a parameter model; F(X _i ; Θ) represents an output model; Xi represents the training crowd image of the _{i-th input; F i represents} the training corresponding to the training crowd image of the i-th input Crowd density map; N represents the number of input data;

The relative headcount loss formula is:

Wherein, F _D (X _i ; Θ) represents the number of predictions; D _i represents the number of true value; N represents the number of input data; the denominator is D _i +1 to prevent the denominator from being zero; the training of the relative number of people loss The weight is L=1.0*L(Θ)+0.1*L _D (Θ).

The present invention also provides a method for counting people, including:

Obtain the image of the crowd to be counted;

Based on the pre-built counting model of the crowd, the image of the crowd to be counted is input to obtain a density map of the crowd to be counted; the counting model of the crowd is constructed by the counting model construction method of the above crowd;

Using an accumulative calculation method, the values in each pixel in the population density map to be counted are accumulated to obtain the total value of the pixel points, and the total value of the pixel points is used as the number of people in the image of the population to be counted.

The present invention also provides a device for constructing a population counting model, including:

The first processing module is used to perform head labeling processing on the crowd images in the pre-stored data set to obtain a crowd density map;

A data combination module, configured to combine the crowd image and the crowd density map corresponding to the crowd image to obtain a target data set;

A data amplification module, configured to perform data amplification processing on the training set obtained by dividing the target data set according to a first preset ratio to obtain a target training set;

The second processing module is used to use the training crowd image and training crowd density map in the target training set to train the multi-scale feature aggregation convolutional neural network model after parameter initialization to obtain a crowd counting model.

Further, in the above-mentioned device, the second processing module includes: a first training unit, a first optimization unit, an iterative processing unit, a second training unit, a second optimization unit, and a model determination unit;

The first training unit is configured to input the training crowd image and the training crowd density map in the target training set into the multi-scale feature aggregation convolutional neural network model after parameter initialization processing to perform the second One level of training to obtain the first training model;

The first optimization unit is configured to use the Euclidean distance loss formula to perform first-level optimization on the model parameters of the first training model to obtain a first optimization model;

The iterative processing unit is configured to iteratively execute the first-level training and first-level optimization processes according to a preset first number of iterations to obtain a first target model;

The second training unit is configured to input the training crowd image and the training crowd density map into the first target model to perform second-level training to obtain a second training model;

The second optimization unit is configured to use the Euclidean distance loss formula and the relative number of people loss formula to optimize the model parameters of the second training model at a second level to obtain a second optimization model;

The iterative processing unit is further configured to iteratively execute the process of the second-level training and the second-level optimization according to a preset second number of iterations to obtain a second target model;

A model determining unit, configured to use the second target model as a counting model of the population.

Further, in the above-mentioned device, the multi-scale feature aggregation convolutional neural network model includes a feature mapping module, a multi-scale feature aggregation module and a density map regression module;

The feature mapping module adopts the first X convolutional layers of the VGG-16 structure, where X is 4 or 6;

The convolution layer is a 3*3 convolution kernel;

The multi-scale feature aggregation module adopts at least two scale branches;

The density map regression module adopts atrous convolution of column A and layer B, wherein both A and B are positive integers.

The present invention also provides a crowd counting device, including: an image acquisition module, a density map determination module, and a people counting module;

The image acquisition module is used to acquire images of people to be counted;

The density map determination module is used to input the image of the crowd to be counted to obtain a density map of the crowd to be counted based on the pre-built counting model of the crowd; the counting model of the crowd is constructed by the above-mentioned crowd counting model construction method;

The population counting module is used to accumulate the values of each pixel in the population density map to be counted by using an accumulation calculation method to obtain the total value of the pixels, and use the total value of the pixels as the number of people in the image of the population to be counted .

The crowd counting model construction method, counting method and device of the present invention, first, carry out head labeling processing on the crowd images in the pre-stored data set, obtain the crowd density map, and combine the crowd image and the crowd density map corresponding to the crowd image , to get the target data set, in which, this scheme is to label the heads in the crowd images, and the occlusion of the body does not affect the result, so there is no need to use perspective images on the training scene and test scene; then the target data set according to the first The training set obtained by a preset ratio is subjected to data amplification processing to obtain the target training set; finally, the multi-scale feature aggregation convolutional neural network after parameter initialization is used to use the training crowd images and training crowd density maps in the target training set. The model is trained to obtain the counting model of the crowd. Among them, the multi-scale feature aggregation convolutional neural network model is a single-column network architecture with multi-scale feature aggregation function. By adopting the technical solution of the present invention, when the model is trained and applied, it can not only solve the problem of the scale change of the characters in the image, but also reduce the amount of computation, improve the recognition efficiency, and do not need to use perspective images in the training process, which improves the accuracy of the model. applicability.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is the flow chart of embodiment 1 of the counting model construction method of the crowd of the present invention;

Figure 2 is a schematic structural diagram of the multi-scale feature aggregation module in the multi-scale feature aggregation convolutional neural network model;

Fig. 3 is the structural representation of stacking pool;

Fig. 4 is the flow chart of embodiment 2 of the counting model construction method of the crowd of the present invention;

Fig. 5 is a flow chart of an embodiment of the crowd counting method of the present invention;

Fig. 6 is a structural schematic diagram of Embodiment 1 of the crowd counting model construction device of the present invention;

Fig. 7 is a schematic structural diagram of Embodiment 2 of the crowd counting model construction device of the present invention;

Fig. 8 is a structural schematic diagram of an embodiment of the crowd counting device of the present invention.

Detailed ways

In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Apparently, the described embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other implementations obtained by persons of ordinary skill in the art without making creative efforts fall within the protection scope of the present invention.

FIG. 1 is a flow chart of Embodiment 1 of the method for constructing a population counting model in the present invention. As shown in Figure 1, the crowd counting model construction method of the present embodiment may specifically include the following steps:

S101. Perform head labeling processing on the crowd images in the pre-stored data set to obtain a crowd density map;

In this embodiment, head labeling processing is first performed on crowd images in a pre-stored data set, wherein the data set is a pre-stored collection containing many crowd images. Then process the crowd density map corresponding to the crowd image by the following formula:

Among them, F(x) represents the crowd density map; M represents the total number of head annotations in the crowd image; x _i represents the pixel point; δ(xx _i ) represents the head position of the ith person; Represents the Gaussian kernel; Indicates the average distance from the i-th person's head position to the k nearest neighbors.

S102. Combine the crowd image and the crowd density map corresponding to the crowd image to obtain a target data set;

Through the above steps, after the crowd density map corresponding to the crowd image is obtained, all the crowd images in the data set and the crowd density map corresponding to the crowd image are combined and stored to obtain a new data set, namely the target data set.

S103. Perform data amplification processing on the training set obtained by dividing the target data set according to the first preset ratio to obtain the target training set;

Through the above steps, after the target data set is obtained, the target data set is divided into a training set and a test set according to a first preset ratio, and then the training set is subjected to data amplification processing, thereby obtaining the target training set. In this embodiment, the first preset ratio is preferably 7:3, that is, the ratio of the data in the training set and the test set into which the target data set is divided is 7:3. In addition, in general, the training set into which the target data set is divided has more data than the test set.

S104. Using the images of the training crowd and the density map of the training crowd in the target training set, train the multi-scale feature aggregation convolutional neural network model after parameter initialization to obtain a counting model of the crowd.

Through the above steps, after obtaining the target training set, use the training crowd image in the target training set and the training crowd density map corresponding to the training crowd image to train the multi-scale feature aggregation convolutional neural network model after parameter initialization. Among them, the parameter initialization process is to initialize and set the parameters of the model. In this embodiment, tensorflow is used to implement the model code. In the initial setting of the model parameters, the initial learning rate is preferably 10 ^-4 , the attenuation coefficient is preferably 0.9, and the attenuation speed is preferably 20 . After the multi-scale feature aggregation convolutional neural network model is trained, the counting model of the crowd can be obtained.

After parameter initialization, the multi-scale feature aggregation convolutional neural network model includes three parts: the front end, the middle end and the back end. The front end is the feature mapping module, the middle end is the multi-scale feature aggregation module FME, and the back end is the density map regression module.

In this embodiment, the feature mapping module uses the first 4 convolutional layers of the VGG-16 structure or the first 6 convolutional layers of the VGG-16 structure, where the stacked pool is used instead of the maximum pooling layer, without introducing additional parameters , which can increase the scale invariance of the model. The convolution layer in this module adopts 3*3 convolution kernel stacking, which can deepen the network depth and increase the nonlinearity of the model. Multiple 3*3 convolutional layers have fewer parameters than a large-size filter filter. Assuming that the input and output feature maps of the convolutional layer are the same size as C, then three 3*3 convolutional layer parameters The number is 27C ² , and the parameter of a 7*7 convolution layer is 49C ² , so three 3*3 convolution kernels can be regarded as a decomposition of a 7*7 convolution kernel (the middle layer has nonlinear decomposition, and play the role of implicit regularization), this replacement can make the model have fewer parameters. Moreover, the number of single-layer neurons in the network is large enough, and the generated feature abstraction level is high enough, which improves the accuracy and speed of the network, and has fewer network parameters. The convolutional layer of the feature mapping module in the multi-scale feature aggregation convolutional neural network model of this embodiment is a 3*3 convolution kernel, through which image features are remapped. In order to ensure the nonlinearity of the model, each convolution layer is followed by a nonlinear activation function layer. This embodiment uses Rectified linear units (ReLU), which can accelerate the convergence of the network.

In this embodiment, the multi-scale feature aggregation convolutional neural network model preferably adopts a four-layer multi-scale feature aggregation module FME, and the multi-scale feature aggregation module FME uses at least two scale branches. The multi-scale feature aggregation module in this embodiment preferably adopts Four branches, Figure 2 is a schematic structural diagram of the multi-scale feature aggregation module in the multi-scale feature aggregation convolutional neural network model, as shown in Figure 2, each branch of the multi-scale feature aggregation module only uses 1*1 and 3*3 The convolution kernel, the first branch of this module only uses a 1*1 convolution kernel, in order to preserve the feature scale of the previous layer to cover small targets. The remaining three branches all use the stack of 3*3 convolution kernels to imitate the receptive field of large convolution kernels. The imitations are 3*3, 5*5, and 7*7 convolution kernels, but they add a layer in front of them. 1*1 convolution reduces the feature dimension by half. At the same time, for the sake of simplicity, the number of channels of each branch is set to be equal, and a ReLU function layer is followed by each convolution kernel. Intuitively speaking, the multi-scale feature aggregation module FME is an integration of receptive fields of different sizes. This module can capture the multi-scale appearance of crowds in dense crowds, which is conducive to crowd counting.

In this embodiment, the density map regression module uses atrous convolution of column A and layer B, where both A and B are positive integers. It has been verified by experiments that the density map regression module of this embodiment is preferably 3 columns and 5 layers of atrous convolution. Multi-layer atrous convolution can increase the receptive field without affecting the resolution, and higher quality can be obtained through atrous convolution. population density map. The dilated convolution is defined as follows:

Among them, w(i,j) represents the filter; y(m,n) represents x(m,n) as input plus a filter w(i,j) for hollow volume with M and N as the length and width respectively The output of the product; r represents the expansion rate.

If the dilation rate r=1, a dilated convolution is an ordinary convolution. Dilated convolutions are a good alternative to pooling layers and can lead to significant improvements in accuracy in segmentation tasks. Although pooling layers (such as max pooling layers) are widely used to maintain invariance and control overfitting, they also greatly reduce the spatial resolution, meaning that the spatial information of the feature maps is lost. Unwrapping layers can reduce the loss of information, but the additional complexity and execution latency may not be appropriate in all cases. Atrous convolution is a better choice, which uses sparse kernels to alternate pooling and convolutional layers. In terms of maintaining the resolution of the feature map, compared with the scheme of convolution + pooling + deconvolution, dilated convolution has obvious advantages. In addition, through empirical research, the best effect is obtained when the expansion rate is 2, so the expansion rate r is preferably 2 in this embodiment.

Table 1 is the architecture table of the multi-scale feature aggregation convolutional neural network model, as shown in Table 1:

Table 1

Among them, since the value of the density map is always positive, it is necessary to add a ReLU activation function after the last 1*1 convolutional layer to strengthen the regression of the density map.

The detailed parameter settings are listed in Table 1, where all convolutional layers use padding to keep the original size. The parameters of the convolutional layer in the table are expressed as "conv(kernel size)-(filter number)-(dilation rate)", where kernel size represents the size of the convolution kernel, filter number represents the number of channels, and dilation rate represents the expansion rate; stacked- Pooling is a stacking pool, and the parameters of the stacking pool are expressed as a kernel set.

Stacked pooling is a stack of pooling layers. Except for a pooling kernel, its pooling operation is calculated on the downsampled feature map, where the continuous calculation formula of the feature map in the middle is:

Among them, the arrow s represents the downsampling rate s; ρ represents the ordinary maximum pooling layer; k' _i represents the kernel size; s' _i represents the step size; Y' ₀ = X is the input feature map; the kernel size k' _i corresponds to A certain transformation on ki; step size s' _{i =1} =s, s'_i>1 =1.

The output connection intermediate feature map of the stacked pool is calculated as:

Figure 3 is a schematic diagram of the structure of the stacked pool, as shown in Figure 3, the kernel set K in the stacked pool = {2, 2, 3}, the step size S = {2, 1, 1}, as shown by experiments, in Figure 3 The stacked pool configuration works well in this example model. Among them, M and N represent the length and width respectively; Map pool represents the maximum pooling; 2ⅹ2 and 3ⅹ3 represent the kernel size in the stacking pool respectively, stride represents the step size; channel average represents the average value of the channel.

In the method for constructing a crowd counting model in this embodiment, first, the crowd images in the pre-stored data set are marked with heads to obtain a crowd density map, and the crowd image and the crowd density map corresponding to the crowd image are combined to obtain the target data set; and then perform data amplification processing on the training set obtained by dividing the target data set according to the first preset ratio to obtain the target training set; finally, use the training crowd images and training crowd density maps in the target training set to process the parameter initialization The multi-scale feature aggregation convolutional neural network model is trained to obtain the counting model of the crowd. Among them, in this embodiment, the head in the crowd image is marked, and the occlusion of the body does not affect the result, so there is no need to use perspective images on the training scene and test scene, which improves the applicability of the model. The multi-scale feature aggregation convolutional neural network model is a single-column network architecture with multi-scale feature aggregation function. When the model is trained and applied, it can not only solve the problem of scale changes of people in the image, but also reduce the amount of calculation and improve the recognition efficiency.

Fig. 4 is a flow chart of the second embodiment of the crowd counting model construction method of the present invention. As shown in Fig. 4, the crowd counting model construction method of this embodiment is based on the embodiment described in Fig. 1 and further detailed The technical solution of the present invention is described in detail.

As shown in Figure 4, the crowd counting model construction method of this embodiment may specifically include the following steps:

S201. Perform head labeling processing on the crowd images in the pre-stored data set to obtain a crowd density map;

The execution process of this step is the same as the execution process of S101 shown in FIG. 1 , and will not be repeated here.

S202. Combine the crowd image and the crowd density map corresponding to the crowd image to obtain a target data set;

The execution process of this step is the same as the execution process of S102 shown in FIG. 1 , and will not be repeated here.

S203. Using a preset clipping method, respectively clip each original training crowd image and the original training crowd density map corresponding to the original training crowd image in the training set obtained by dividing the target data set according to the first preset ratio, to obtain at least two a cropped image and at least two cropped density maps;

Through the above steps, after the target data set is obtained, the target data set is divided according to the first preset ratio. The specific division method has been described in the above embodiments and will not be elaborated here again. After division, the training set is obtained. The training set includes the original training crowd image and the original training crowd density map corresponding to the original crowd image. Through the preset cutting method, each original training crowd image and the original training crowd density map are respectively cut to obtain at least Two cropped images and at least two cropped density maps, the cropped images correspond to the cropped density maps, and how to crop the original training crowd image is how to crop the original training crowd density map corresponding to the original training crowd image. In this embodiment, the preset cropping method is preferably a nine-square grid cutting method, that is, the original training crowd image and the original training crowd density map are cut into nine square grids, and each original training crowd image is cut out to 9 cropped images, and each original training crowd image The crowd density maps are also cropped to produce 9 cropped density maps.

S204, flipping the cropped image to obtain a flipped image, and flipping the cropped density map to obtain a flipped density map;

Through the above steps, after the cropped image and the cropped density map corresponding to the cropped image are obtained, the cropped image is flipped to obtain a flipped image; the cropped density map is flipped to obtain a flipped density map.

S205. Expand the cropped image and flipped image according to the second preset ratio of the original training crowd image to obtain an enlarged image, and expand the cropped density map and flipped density map to obtain an enlarged density map;

Through the above steps, after the cropped image, cropped density map, flipped image and flipped density map are obtained, the cropped image and flipped image are enlarged according to the second preset ratio of the original training crowd image to obtain the enlarged image; according to the ratio of the original training crowd image The second preset ratio expands the cropped density map and flipped density map to obtain the enlarged density map. In this embodiment, the second preset ratio of the original training crowd image is preferably 90% of the original training crowd image, that is, the cropped image, cropped density map, flipped image, and flipped density map are all enlarged to 90% of the original training crowd image.

S206. Store the enlarged image and the enlarged density map in the training set to obtain the target training set;

Through the above steps, after the enlarged image and the enlarged density map corresponding to the enlarged image are obtained, the enlarged image and the enlarged density map are stored in the training set to obtain a new training set, that is, the target training set. Wherein, the training crowd image in the target training set includes the original training crowd image and the enlarged image; the training crowd density map in the target training set includes the original training crowd density map and the enlarged density map.

S207. Input the images of the training crowd and the density map of the training crowd in the target training set into the multi-scale feature aggregation convolutional neural network model after parameter initialization processing to perform first-level training to obtain the first training model;

Through the above steps, after the target training set is obtained, the training crowd images and training crowd density maps in the target training set are input into the multi-scale feature aggregation convolutional neural network model after parameter initialization processing, and the first level of training is performed on the model. training to obtain the first training model. Wherein, for each training, a training crowd image and a training crowd density map corresponding to the training crowd image are selected from the training set and input into the model.

S208. Using the Euclidean distance loss formula to optimize the model parameters of the first training model at the first level to obtain the first optimization model;

Through the above steps, after the first training model is obtained, the model parameters of the first training model are optimized at the first level by using the Euclidean distance loss formula, so as to obtain the first optimization model.

The Euclidean distance loss formula is:

Wherein, Θ represents a parameter model; F(X _i ; Θ) represents an output model; Xi represents the _i -th input training crowd image; F _i represents the training crowd density map corresponding to the i-th input training crowd image; N represents The number of input data.

S209. Perform iterative execution of the first-level training and first-level optimization processes according to the preset number of first iterations to obtain the first target model;

Through the above steps, after the first-level training and the first-level optimization are performed, the process of the first-level training and the first-level optimization is iteratively executed according to the preset number of iterations to obtain the first target model. The preset first number of iterations in this embodiment is preferably 200,000 times.

Iteration is the activity of repeating a feedback process, usually with the aim of approaching a desired goal or result. Each repetition of the process is called an "iteration", and the result of each iteration is used as the initial value for the next iteration. For a subroutine that needs to be executed repeatedly in a specific computer program, perform a repetition, that is, repeat the loop in the program until a certain condition is met, which is also called iteration. In this embodiment, the first-level training and the first-level optimization are continuously performed until the number of iterative executions satisfies the preset first number of iterations.

S210. Input the training crowd image and the training crowd density map into the first target model to perform second-level training to obtain a second training model;

Through the above steps, after the first target model is obtained, the training crowd image and the training crowd density map are input into the first target model, and the second level of training is performed on the model, thereby obtaining the second training model. Wherein, for each training, a training crowd image and a training crowd density map corresponding to the training crowd image are selected from the training set and input into the model.

S211. Using the Euclidean distance loss formula and the relative number of people loss formula to perform second-level optimization on the model parameters of the second training model to obtain a second optimization model;

Through the above steps, after the second training model is obtained, the model parameters of the second training model are optimized at the second level by using the Euclidean distance loss formula and the relative number loss formula to obtain the second optimization model. Using the relative number loss formula to optimize the model parameters can focus on learning samples with large prediction errors. In the scenario where the absolute number of people is very sparse, the accuracy rate can be significantly improved by using the relative number loss in the network. In this embodiment, not only the Euclidean distance loss formula is used to optimize the model parameters, but also the relative number of people loss formula is used to optimize the model parameters, which can improve the prediction accuracy in the scene of absolute sparse number of people.

The relative headcount loss formula is:

Among them, F _D (X _i ; Θ) represents the number of people predicted; D _i represents the number of people of true value; N represents the number of input data; the denominator is D _i +1 in order to prevent the denominator from being zero; the training weight of the relative number of people loss is L=1.0*L(Θ)+0.1*L _D (Θ).

S212. Iteratively execute the second-level training and second-level optimization processes according to the second preset number of iterations to obtain a second target model;

Through the above steps, after the second-level training and the second-level optimization are performed, the process of the second-level training and the second-level optimization is iteratively executed according to the second preset number of iterations, so as to obtain the second target model. The second preset number of iterations in this embodiment is preferably 100,000 times.

In this embodiment, in addition to training first and using the Euclidean distance loss formula to optimize iterations, then training and using the Euclidean distance loss formula and the relative number loss formula to optimize iterations, it is also possible to directly use the training and use the Euclidean distance loss formula and the relative number loss formula Optimization iterations, but the number of iterations is preferably more than 300,000 times, so as to achieve the convergence of the model.

S213. Input the test crowd images in the test set obtained by dividing the target data set according to the first preset ratio into the second target model to obtain a target crowd density map;

Through the above steps, after the second target model is obtained, the test population images in the test set obtained by dividing the target data set according to the first preset ratio are input into the second target model, and after the second target model processes the test crowd images, the obtained Target population density map.

S214. Determine the test accuracy of the model according to the target crowd density map and the test crowd density map corresponding to the test crowd images in the test set;

After the target population density map is obtained through the above steps, compare and analyze the target population density map and the test population density map corresponding to the test population images in the test set to determine the test accuracy of the second target model.

S215. Determine whether the test accuracy rate is greater than the preset accuracy rate; if yes, execute step S216; if not, execute step S210;

Through the above steps, after obtaining the test accuracy rate, compare the test accuracy rate with the preset accuracy rate, and judge whether the test accuracy rate is greater than the preset accuracy rate, if the test accuracy rate is greater than the preset accuracy rate, then perform step S216; If the accuracy rate is less than or equal to the preset accuracy rate, it means that the accuracy rate of the second target model is not up to the standard, and the second target model is used as the first target model to perform step S210 to reacquire data and retrain the model.

S216. Use the second target model as a population counting model.

Through the above steps, if it is judged that the test accuracy rate is greater than the preset accuracy rate, it means that the accuracy rate of the second target model meets the requirements, and the second target model is used as the population counting model.

In the method for constructing a crowd counting model in this embodiment, first, the crowd images in the pre-stored data set are marked with heads to obtain a crowd density map. In this way, marking the heads in the crowd images does not affect the result due to body occlusion, so There is no need to use perspective images on the training scene and test scene, which improves the applicability of the model. Crop, flip and expand the original training population image and the original training population density map in the training set divided by the target data set to obtain the enlarged image and the enlarged density map, and store the enlarged image and the enlarged density map in the training set to obtain the target training population Set, so as to realize the data amplification of the training set and ensure that the training data of the model is sufficient. Iteratively perform first-level training and first-level optimization on the multi-scale feature aggregation convolutional neural network model to obtain the first target model, and then iteratively perform second-level training and second-level optimization on the first target model, The second objective model is obtained. During the optimization process, the Euclidean distance loss formula and the relative number loss formula are implemented to optimize the model at the same time, thereby improving the prediction accuracy of the number of people. Input the test population images in the test set divided by the target data set into the second target model to obtain the target population density map, and determine the test accuracy of the model according to the target population density map and the test population density map in the test set. If the test accuracy rate is greater than the preset accuracy rate, the second target model is used as the counting model of the crowd, so that the accuracy of the model can be tested, thereby ensuring the accuracy of the counting results. The multi-scale feature aggregation convolutional neural network model used in this embodiment is a single-column network architecture with multi-scale feature aggregation function. When the model is trained and applied, it can not only solve the problem of the scale change of the characters in the image, but also reduce the amount of calculation. , to improve the recognition efficiency.

Fig. 5 is a flow chart of an embodiment of the crowd counting method of the present invention. As shown in Fig. 5, the crowd counting method of this embodiment may specifically include the following steps:

S301. Obtain images of people to be counted;

In this embodiment, it is first necessary to obtain images of people to be counted. Wherein, the image of the crowd to be counted is an image that needs to be counted.

S302. Based on the pre-built counting model of the crowd, input the image of the crowd to be counted, and obtain a density map of the crowd to be counted;

Through the above steps, after the image of the crowd to be counted is obtained, the image of the crowd to be counted is input into the counting model of the crowd constructed by the counting model construction method of the crowd in the above embodiment, and through the processing of the counting model of the crowd, the counting model of the crowd to be counted is obtained. The density map of the crowd to be counted corresponding to the counted crowd image. Wherein, the population density map to be counted is composed of many points, each point represents a person, and the position of the point is the head position of the person in the image of the population to be counted.

S303. Use an accumulation calculation method to accumulate the values of each pixel in the population density map to be counted to obtain the total value of the pixels, and use the total value of the pixels as the number of people in the image of the population to be counted.

Through the above steps, after obtaining the population density map to be counted, use the cumulative calculation method to accumulate the values in each pixel in the population density map to be counted to obtain the total value of the pixels, and use the total value of the pixels as the number of people in the image of the population to be counted .

The counting method of the crowd in this embodiment first obtains the image of the crowd to be counted, and inputs the image of the crowd to be counted into the counting model of the crowd pre-built by the counting model construction method of the crowd to obtain the density map of the crowd to be counted, and finally, Using the cumulative calculation method, the values in each pixel in the population density map to be counted are accumulated to obtain the total value of the pixel points, and the total value of the pixel points is used as the number of people in the image of the population to be counted. In this embodiment, the head in the crowd image is marked, and the occlusion of the body does not affect the result, so when constructing the counting model of the crowd, it is not necessary to use perspective images on the training scene and the test scene, which improves the applicability of the model, and The multi-scale feature aggregation convolutional neural network model used in the crowd counting model is a single-column network architecture with multi-scale feature aggregation function. When the model is trained and applied, it can not only solve the problem of scale changes of people in the image, but also reduce reduce the amount of computation and improve the recognition efficiency.

To be more comprehensive, corresponding to the method for constructing a crowd counting model provided in the embodiment of the present invention, the present application also provides a crowd counting model constructing device.

Figure 6 is a schematic structural view of Embodiment 1 of the crowd counting model construction device of the present invention, as shown in Figure 6, the crowd counting model construction device of this embodiment includes a first processing module 11, a data combination module 12, a data amplification module 13 and a second processing module 14.

The first processing module 11 is used to perform head labeling processing on the crowd images in the pre-stored data set to obtain a crowd density map;

The data combination module 12 is used to combine the crowd image and the crowd density map corresponding to the crowd image to obtain the target data set;

The data amplification module 13 is used to perform data amplification processing on the training set obtained by dividing the target data set according to the first preset ratio to obtain the target training set;

The second processing module 14 is configured to use the training crowd image and training crowd density map in the target training set to train the multi-scale feature aggregation convolutional neural network model after parameter initialization to obtain a crowd counting model.

In the device for constructing a crowd counting model in this embodiment, first, the first processing module 11 performs head labeling processing on the crowd images in the pre-stored data set to obtain a crowd density map, and the crowd images are combined with the crowd images by the data combination module 12. The corresponding crowd density maps are combined to obtain the target data set; then the training set obtained by dividing the target data set according to the first preset ratio is subjected to data amplification processing through the data amplification module 13 to obtain the target training set; finally through the second The processing module 14 uses the images of the training crowd and the density map of the training crowd in the target training set to train the multi-scale feature aggregation convolutional neural network model after parameter initialization to obtain a counting model of the crowd. Among them, in this embodiment, the head in the crowd image is marked, and the occlusion of the body does not affect the result, so there is no need to use perspective images on the training scene and test scene, which improves the applicability of the model. The multi-scale feature aggregation convolutional neural network model is a single-column network architecture with multi-scale feature aggregation function. When the model is trained and applied, it can not only solve the problem of scale changes of people in the image, but also reduce the amount of calculation and improve the recognition efficiency.

Fig. 7 is a schematic structural diagram of the second embodiment of the crowd counting model construction device of the present invention. As shown in Fig. 7, the crowd counting model construction device of this embodiment is based on the embodiment described in Fig. 6 and has a data amplification module 13 includes a cropping unit 131 , a flipping unit 132 , an expanding unit 133 and a storage unit 134 .

The clipping unit 131 is configured to clip each original training crowd image in the training set and the original training crowd density map corresponding to the original training crowd image through a preset clipping method to obtain at least two cropped images and at least two cropped densities picture;

The flipping unit 132 is configured to flip the cropped image to obtain a flipped image, and flip the cropped density map to obtain a flipped density map;

The expansion unit 133 is configured to expand the cropped image and flipped image according to the second preset ratio of the original training crowd image to obtain an enlarged image, and expand the cropped density map and flipped density map to obtain an enlarged density map;

The storage unit 134 is configured to store the enlarged image and the enlarged density map in the training set to obtain a target training set.

Further, in the device for constructing a crowd counting model in this embodiment, the second processing module 14 includes a first training unit 141, a first optimization unit 142, an iterative processing unit 143, a second training unit 144, a second optimization unit 145 and The model determination unit 146 ; the device for constructing a population counting model in this embodiment also includes a model testing module 15 , a determination module 16 and a judgment module 17 .

The first training unit 141 is used to input the images of the training crowd and the density map of the training crowd in the target training set into the multi-scale feature aggregation convolutional neural network model after parameter initialization for the first level of training to obtain the first training Model;

Wherein, the training crowd image includes an original training crowd image and an enlarged image; the training crowd density map includes an original training crowd density map and an enlarged density map.

The first optimization unit 142 is configured to use the Euclidean distance loss formula to perform first-level optimization on the model parameters of the first training model to obtain the first optimization model;

The iterative processing unit 143 is configured to iteratively execute the first-level training and first-level optimization processes according to the preset first number of iterations to obtain the first target model;

The second training unit 144 is configured to input the training crowd image and the training crowd density map into the first target model for second-level training to obtain a second training model;

The second optimization unit 145 is configured to use the Euclidean distance loss formula and the relative number of people loss formula to perform second-level optimization on the model parameters of the second training model to obtain a second optimization model;

The iterative processing unit 143 is further configured to iteratively execute the second-level training and second-level optimization processes according to a preset second number of iterations to obtain a second target model;

The model testing module 15 is used to input the test crowd images in the test set obtained by dividing the target data set according to the first preset ratio into the second target model to obtain the target crowd density map;

Determining module 16, for determining the test accuracy of the model according to the target crowd density map and the test crowd density map corresponding to the test crowd image in the test set;

Judging module 17, for judging whether the test accuracy rate is greater than the preset accuracy rate;

The model determining unit 146 is configured to use the second target model as the population counting model if the test accuracy rate is greater than the preset accuracy rate.

In the device for constructing a crowd counting model in this embodiment, first, the first processing module 11 performs head labeling processing on the crowd images in the pre-stored data set to obtain a crowd density map. It doesn't matter, so there is no need to use the perspective view on the training scene and test scene, which improves the applicability of the model. The cutting unit 131, the flipping unit 132 and the expanding unit 133 cut, flip and expand the original training crowd image and the original training crowd density map in the training set divided by the target data set to obtain the enlarged image and the enlarged density map, and the storage unit 134 will expand The image and the enlarged density map are stored in the training set to obtain the target training set, so as to realize the data amplification of the training set and ensure that the training data of the model is sufficient. The iterative processing unit 143 iteratively performs first-level training and first-level optimization on the multi-scale feature aggregation convolutional neural network model to obtain the first target model, and then iteratively performs second-level training and second-level optimization on the first target model. Level optimization, get the second target model, realize the Euclidean distance loss formula and the relative number loss formula to optimize the model at the same time during the optimization process, so as to improve the prediction accuracy of the number of people. The model testing module 15 inputs the test crowd images in the test set divided by the target data set into the second target model, and the determination module 16 determines the test accuracy of the model according to the obtained target crowd density map and the test crowd density map in the test set, If the judging module 17 judges that the test accuracy rate is greater than the preset accuracy rate, the model determination unit 146 uses the second target model as the counting model of the crowd, so that the accuracy of the model can be checked, thereby ensuring the accuracy of the counting result. The multi-scale feature aggregation convolutional neural network model used in this embodiment is a single-column network architecture with multi-scale feature aggregation function. When the model is trained and applied, it can not only solve the problem of the scale change of the characters in the image, but also reduce the amount of calculation. , to improve the recognition efficiency.

To be more comprehensive, corresponding to the crowd counting method provided in the embodiment of the present invention, the present application also provides a crowd counting device.

FIG. 8 is a schematic structural diagram of an embodiment of the crowd counting device of the present invention. As shown in FIG.

An image acquisition module 21, configured to acquire an image of a crowd to be counted;

The density map determination module 22 is used to input the image of the crowd to be counted based on the pre-built counting model of the crowd to obtain the density map of the crowd to be counted; the counting model of the crowd is constructed by the counting model construction method of the crowd in the above embodiment;

The people counting module 23 is used for accumulating the values of each pixel in the density map of the people to be counted by using the cumulative calculation method to obtain the total value of the pixels, and using the total value of the pixels as the number of people in the image of the people to be counted.

In the crowd counting device of this embodiment, the image acquisition module 21 acquires the images of the crowds to be counted, and the density map determination module 22 inputs the images of the crowds to be counted into the counting model of the crowds pre-built by the counting model construction method of the crowds, and obtains For counting the crowd density map, finally, the people counting module 23 uses the cumulative calculation method to accumulate the values in each pixel in the crowd density map to be counted to obtain the total value of the pixels, and use the total value of the pixels as the number of people in the crowd image to be counted. In this embodiment, the head in the crowd image is marked, and the occlusion of the body does not affect the result, so when constructing the counting model of the crowd, it is not necessary to use perspective images on the training scene and the test scene, which improves the applicability of the model, and The multi-scale feature aggregation convolutional neural network model used in the crowd counting model is a single-column network architecture with multi-scale feature aggregation function. When the model is trained and applied, it can not only solve the problem of scale changes of people in the image, but also reduce reduce the amount of computation and improve the recognition efficiency.

Regarding the apparatus in the foregoing embodiments, the specific manner in which each module executes operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

It can be understood that, the same or similar parts in the above embodiments can be referred to each other, and the content that is not described in detail in some embodiments can be referred to the same or similar content in other embodiments.

It should be noted that, in the description of the present invention, the terms "first", "second" and so on are only used for description purposes, and should not be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" means at least two.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments or portions of code comprising one or more executable instructions for implementing specific logical functions or steps of the process , and the scope of preferred embodiments of the invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including substantially concurrently or in reverse order depending on the functions involved, which shall It is understood by those skilled in the art to which the embodiments of the present invention pertain.

It should be understood that various parts of the present invention can be realized by hardware, software, firmware or their combination. In the embodiments described above, various steps or methods may be implemented by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques known in the art: Discrete logic circuits, ASICs with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the program can be executed when the program is executed. When, one or a combination of the steps of the method embodiment is included.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing module, each unit may exist separately physically, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the integrated modules are realized in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium.

The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, and the like.

In the description of this specification, descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims (10)

1. A counting model building method of a crowd, is characterized in that, comprises: Perform head annotation processing on the crowd images in the pre-stored data set to obtain a crowd density map; combining the crowd image with a crowd density map corresponding to the crowd image to obtain a target data set; performing data amplification processing on the training set obtained by dividing the target data set according to the first preset ratio to obtain the target training set; Using the images of the training crowd and the density map of the training crowd in the target training set, the multi-scale feature aggregation convolutional neural network model after parameter initialization is trained to obtain the counting model of the crowd. 2. The method according to claim 1, characterized in that, performing data amplification processing on the training set obtained by dividing the target data set according to the first preset ratio to obtain the target training set, comprising: Each of the original training crowd images in the training set and the original training crowd density map corresponding to the original training crowd images are respectively cropped by a preset cropping method to obtain at least two cropped images and at least two crop density map; performing flipping processing on the cropped image to obtain a flipped image, and flipping the cropped density map to obtain a flipped density map; According to the second preset ratio of the original training crowd image, the cropped image and the flipped image are enlarged to obtain an enlarged image, and the cropped density map and the flipped density map are enlarged to obtain an enlarged density map ; storing the enlarged image and the enlarged density map in the training set to obtain a target training set; Wherein, the training crowd image includes the original training crowd image and the enlarged image; the training crowd density map includes the original training crowd density map and the enlarged density map. 3. The method according to claim 1, wherein, using the training crowd image and the training crowd density map in the target training set, the multi-scale feature aggregation convolutional neural network model after parameter initialization is processed Train to get the counting model of the crowd, including: Input the training crowd image and the training crowd density map in the target training set into the multi-scale feature aggregation convolutional neural network model after parameter initialization to perform first-level training to obtain the first training Model; Using the Euclidean distance loss formula to optimize the model parameters of the first training model at the first level to obtain the first optimization model; Iteratively executing the first-level training and the first-level optimization processes according to a preset first number of iterations to obtain a first target model; Inputting the training crowd image and the training crowd density map into the first target model to perform second-level training to obtain a second training model; Using the Euclidean distance loss formula and the relative number of people loss formula to perform second-level optimization on the model parameters of the second training model to obtain a second optimization model; performing iterative execution on the second-level training and the second-level optimization process according to the preset second number of iterations to obtain a second target model; The second target model is used as the population count model. 4. The method according to claim 3, wherein, before using the second target model as the counting model of the crowd, further comprising: inputting the test crowd images in the test set obtained by dividing the target data set according to the first preset ratio into the second target model to obtain a target crowd density map; Determine the test accuracy of the model according to the target crowd density map and the test crowd density map corresponding to the test crowd images in the test set; Judging whether the test accuracy rate is greater than the preset accuracy rate; Correspondingly, said using the second target model as the population counting model includes: If the test accuracy rate is greater than the preset accuracy rate, the second target model is used as the population count model. 5. method according to claim 3, is characterized in that, described Euclidean distance loss formula is: Wherein, Θ represents a parameter model; F(X _i ; Θ) represents an output model; Xi represents the training crowd image of the _{i-th input; F i represents} the training corresponding to the training crowd image of the i-th input Crowd density map; N represents the number of input data; The relative headcount loss formula is: Wherein, F _D (X _i ; Θ) represents the number of predictions; D _i represents the number of true value; N represents the number of input data; the denominator is D _i +1 to prevent the denominator from being zero; the training of the relative number of people loss The weight is L=1.0*L(Θ)+0.1*L _D (Θ). 6. A crowd counting method, characterized in that, comprising: Obtain the image of the crowd to be counted; Based on the pre-built counting model of the crowd, input the image of the crowd to be counted to obtain a density map of the crowd to be counted; the counting model of the crowd is constructed by the crowd counting model construction method described in any one of claims 1-5; Using an accumulative calculation method, the values in each pixel in the population density map to be counted are accumulated to obtain the total value of the pixel points, and the total value of the pixel points is used as the number of people in the image of the population to be counted. 7. A crowd counting model building device, characterized in that it comprises: The first processing module is used to perform head labeling processing on the crowd images in the pre-stored data set to obtain a crowd density map; A data combination module, configured to combine the crowd image and the crowd density map corresponding to the crowd image to obtain a target data set; A data amplification module, configured to perform data amplification processing on the training set obtained by dividing the target data set according to a first preset ratio to obtain a target training set; The second processing module is used to use the training crowd image and training crowd density map in the target training set to train the multi-scale feature aggregation convolutional neural network model after parameter initialization to obtain a crowd counting model. 8. The device according to claim 7, wherein the second processing module comprises: a first training unit, a first optimization unit, an iterative processing unit, a second training unit, a second optimization unit and a model determination unit ; The first training unit is configured to input the training crowd image and the training crowd density map in the target training set into the multi-scale feature aggregation convolutional neural network model after parameter initialization processing to perform the second One level of training to obtain the first training model; The first optimization unit is configured to use the Euclidean distance loss formula to perform first-level optimization on the model parameters of the first training model to obtain a first optimization model; The iterative processing unit is configured to iteratively execute the first-level training and first-level optimization processes according to a preset first number of iterations to obtain a first target model; The second training unit is configured to input the training crowd image and the training crowd density map into the first target model to perform second-level training to obtain a second training model; The second optimization unit is configured to use the Euclidean distance loss formula and the relative number of people loss formula to optimize the model parameters of the second training model at a second level to obtain a second optimization model; The iterative processing unit is further configured to iteratively execute the process of the second-level training and the second-level optimization according to a preset second number of iterations to obtain a second target model; A model determining unit, configured to use the second target model as a counting model of the population. 9. The device according to claim 8, wherein the multi-scale feature aggregation convolutional neural network model includes a feature mapping module, a multi-scale feature aggregation module and a density map regression module; The feature mapping module adopts the first X convolutional layers of the VGG-16 structure, where X is 4 or 6; The convolution layer is a 3*3 convolution kernel; The multi-scale feature aggregation module adopts at least two scale branches; The density map regression module adopts atrous convolution of column A and layer B, wherein both A and B are positive integers. 10. A crowd counting device, comprising: an image acquisition module, a density map determination module and a people counting module; The image acquisition module is used to acquire images of people to be counted; The density map determination module is used to input the image of the crowd to be counted based on the pre-built counting model of the crowd to obtain a density map of the crowd to be counted; the counting model of the crowd is as described in any one of claims 1-5. The counting model construction method construction of the crowd; The population counting module is used to accumulate the values of each pixel in the population density map to be counted by using an accumulation calculation method to obtain the total value of the pixels, and use the total value of the pixels as the number of people in the image of the population to be counted .