CN108416307B - An aerial image pavement crack detection method, device and equipment - Google Patents

Abstract

The embodiment of the invention provides a method, a device and equipment for detecting a pavement crack of an aerial image. The method comprises the following steps: extracting deep high-dimensional features of a road surface area of the aerial photographing road surface image, and obtaining a high-dimensional feature map according to the deep high-dimensional features; based on deep high-dimensional characteristics of the pavement area, screening positive and negative samples of the high-dimensional characteristic diagram to distinguish a pavement crack target from a pavement background; and classifying and positioning the pavement crack target in a coordinate manner to obtain classification information and coordinate information of the pavement crack target. The method and the device can be applied to pavement crack detection of aerial images of high-altitude motion backgrounds and complex scenes, are more suitable for acquiring images by an unmanned aerial vehicle-mounted system compared with various commonly used crack detection algorithms, and have better crack robustness of aerial image detection.

Description

An aerial image pavement crack detection method, device and equipment

technical field

Embodiments of the present invention relate to the fields of deep learning and pattern recognition, and in particular, to a method, device and device for detecting road cracks in aerial images.

Background technique

At present, one of the main damage forms of highway pavement is pavement cracks. Among them, the types of highway cracks in my country are mainly transverse and longitudinal cracks. If the cracks can be found at the early stage and their development can be tracked, the road maintenance cost will be greatly reduced, and the driving safety of the expressway will be guaranteed at the same time. Therefore, it is extremely important to conduct regular investigation and maintenance of the road surface condition.

The pavement crack detection method has been developed from the initial manual detection method; with the application of image processing technology, the combination of vehicle-mounted acquisition device and image processing technology is applied to pavement crack detection, which greatly improves the detection efficiency. In recent years, UAV technology has developed rapidly, and the applications combined with it have been greatly enriched. Compared with other methods, the road crack detection device combined with the UAV acquisition method has the advantages of fast efficiency, large field of view and storage. The data volume has declined. However, compared with vehicle-mounted images, there are roadside scenery, vehicles, wires, shadows and other disturbances, and the noise is also very rich.

Commonly used crack identification methods mainly focus on the applications of threshold segmentation, feature detection, texture analysis and seed growth, in addition to machine learning and the use of fuzzy sets. However, these existing methods are basically based on the detection and development of vehicle-mounted acquisition device images, and cannot be applied to aerial images with more interference and noise.

SUMMARY OF THE INVENTION

Aiming at the problems existing in the prior art, the embodiments of the present invention provide a method, device and equipment for detecting road surface cracks in aerial images, which combine a series of advantages of aerial photography acquisition methods to make crack detection more efficient and convenient, and improve the robustness of traditional image processing algorithms Sexual issues.

In a first aspect, an embodiment of the present invention provides a method for detecting road cracks in aerial images, including:

extracting deep high-dimensional features of the pavement area of the aerial photographed road surface image, and obtaining a high-dimensional feature map according to the deep high-dimensional features;

Based on the deep high-dimensional features of the pavement area, the high-dimensional feature map is screened for positive and negative samples to distinguish the pavement crack target and the pavement background;

Classify and coordinate positioning of the road surface crack target to obtain classification information and coordinate information of the road surface crack target.

In a second aspect, an embodiment of the present invention provides an aerial image pavement crack detection device, including:

The high-dimensional feature map module is used to extract the deep high-dimensional features of the road surface area of the aerial photographic road image, and obtain the high-dimensional feature map according to the deep high-dimensional features;

a crack identification module for screening positive and negative samples of the high-dimensional feature map based on the deep high-dimensional features of the pavement area, so as to distinguish the pavement crack target from the pavement background; and

The classification and positioning module is used for classifying and coordinate positioning of the road surface crack target, and obtaining classification information and coordinate information of the road surface crack target.

In a third aspect, an embodiment of the present invention provides an aerial image pavement crack detection device, characterized in that it includes:

at least one processor; and

at least one memory communicatively coupled to the processor, wherein:

The memory stores program instructions that can be executed by the processor, and the processor can invoke the program instructions to execute the method for detecting road cracks in aerial images according to the first aspect of the embodiment of the present invention and any optional embodiments thereof. the method described.

In a fourth aspect, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions execute the method for detecting road cracks in aerial images according to the first aspect of the embodiments of the present invention and the method of any alternative embodiment thereof.

An embodiment of the present invention provides a method for detecting road surface cracks in aerial images. By extracting high-dimensional feature maps of the road surface area, the road surface crack target and the road surface background are preliminarily distinguished, and the crack target is classified and accurately positioned, so as to realize the detection of the road surface by aerial photography. Crack target location and classification, and obtain the coordinate information of the crack target. The embodiments of the present invention can be applied to road crack detection in aerial images of high-altitude motion backgrounds and complex scenes. Compared with various commonly used crack detection algorithms, the embodiments of the present invention are more suitable for unmanned aerial systems to collect images, and have better aerial image detection capabilities. robustness.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1 is a flowchart of a method for detecting road cracks in aerial images according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of an embodiment of identifying road cracks in an aerial image according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a framework of an aerial image pavement crack detection device according to an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are the Some, but not all, embodiments are disclosed. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

1 is a flowchart of a method for detecting road cracks in aerial images according to an embodiment of the present invention. The method for detecting road cracks in aerial images as shown in FIG. 1 includes:

S100, extracting deep high-dimensional features of the road surface area of the aerial photographed road surface image, and obtaining a high-dimensional feature map according to the deep high-dimensional features;

The embodiments of the present invention are suitable for crack detection of aerial photographed road images. The aerial photographic images are characterized by high altitude, long distance, sports performance shooting, and complex scenes, that is, the background outside the road surface is complex.

Preferably, before step S100, the method further includes: performing rough segmentation on the aerial photographed road surface image to eliminate roadside invalid areas, and obtain the road surface area in the aerial photographed road surface image.

Specifically, the aerial photographed road surface image includes the road surface area and the roadside invalid area other than the road surface area. In the embodiment of the present invention, the roadside invalid area other than the road surface area in the aerial photographed road surface image is eliminated, and the rest is the road surface area; on this basis Above, perform high-dimensional feature extraction on the pavement area, and obtain a high-dimensional feature map according to the extracted deep high-dimensional features. The high-dimensional feature is: the overall semantic information image feature of the crack target which is different from the local information feature of the shallow edge.

S200, based on the deep high-dimensional features of the pavement area, screening positive and negative samples of the high-dimensional feature map to distinguish the pavement crack target and the pavement background;

The embodiment of the present invention performs positive and negative sample screening, that is, preliminary identification of the pavement crack target, and distinguishes the pavement crack target from the pavement background. The positive sample is the crack target, and the negative sample is the pavement background.

S300 , classify and coordinate the pavement crack target, and obtain classification information and coordinate information of the pavement crack object.

After preliminary positioning in step S200 to obtain the crack target and the road background, step S300 classifies the crack target based on the preliminary positioning, precisely locates the position of the crack target, and finally obtains detailed classification information and coordinate position information of the crack target. The classification information refers to the type classification of the crack target, such as a transverse crack or a longitudinal crack, etc. The coordinate location is the precise position coordinates of the quantitative crack target.

An embodiment of the present invention provides a method for detecting road surface cracks in aerial images. By extracting high-dimensional feature maps of the road surface area, the road surface crack target and the road surface background are preliminarily distinguished, and the crack target is classified and accurately positioned, so as to realize the detection of the road surface by aerial photography. Crack target location and classification, and obtain the coordinate information of the crack target. The embodiments of the present invention can be applied to road crack detection in aerial images of high-altitude motion backgrounds and complex scenes. Compared with various commonly used crack detection algorithms, the embodiments of the present invention are more suitable for unmanned aerial systems to collect images, and have better aerial image detection capabilities. robustness.

Based on the above embodiment, the classification and coordinate positioning of the road surface crack target to obtain the classification information and coordinate information of the road surface crack target further include:

S400: Calculate the length of the road surface crack target according to the classification information and coordinate information of the road surface crack target.

Based on the detailed classification information and coordinate position information of the above-mentioned crack target, step S400 further calculates the length of the road surface crack target, so that the embodiment of the present invention can finally obtain the following information according to the aerial photographed road surface image: the crack target is located, the detailed classification of the crack target, Precise coordinate position and quantitative length data.

The embodiment of the present invention overcomes the difficulty of image processing caused by the UAV acquisition method, can be applied to the detection of road cracks with high-altitude motion background and complex scenes, and has stronger robustness than various commonly used crack detection algorithms, and can obtain Better recognition of cracks in aerial images. When the invention is applied to crack detection in aerial photography, it can not only provide the observer with a more prominent image of the crack target, but also quantitatively analyze the crack length to provide a reference for subsequent road maintenance.

In an optional embodiment, in step S100, the deep high-dimensional features of the road surface area of the aerial photographed road surface image are extracted, and a high-dimensional feature map is obtained according to the deep high-dimensional features, which specifically includes:

S100.1, using a convolutional neural network to construct a feature extraction network, and adding a rough road segmentation layer based on the K-means clustering algorithm in the feature extraction network;

S100.2, using the rough road segmentation layer to filter out the roadside invalid area of the aerial photographed road surface image to obtain the road surface area of the aerial photographed road surface image;

S100.3, using the feature extraction network to combine the low-latitude features of the road surface area into high-latitude features to obtain a high-dimensional feature map.

In step S100.1, the embodiment of the present invention performs learning and training based on a convolutional neural network. By analyzing the mature network structure and combining with the task of detecting small targets in complex scenes, a convolutional neural network is used to construct a feature extraction network. The extraction network is used to extract high-dimensional features of the pavement area. Further, in the embodiment of the present invention, combined with the K-means algorithm, a rough road segmentation layer based on the K-means clustering algorithm is added to the feature extraction network. The rough road segmentation layer satisfies the characteristics of high recall rate and can remove roadside invalid area.

Step S100.2 uses the rough road segmentation layer to filter out the roadside invalid area of the aerial photographed road surface image to obtain the road surface area of the aerial photographed road surface image, and the specific processing is as follows:

In order to ensure that all the pavement areas will be extracted into high-dimensional features by the feature extraction network, the road rough segmentation layer based on the K-means clustering algorithm is used to roughly segment the input aerial pavement image to remove the invalid areas on the roadside.

For a given sample set, according to the distance between samples, use the initial K cluster centers to calculate the distance between each cluster center and each data element, and assign the K closest to each cluster center in each iteration. The data elements form K clusters, then recalculate the assigned cluster centers, and iteratively assign all data elements until all the clusters do not change, divide the sample set into fixed K clusters, and let the points in the clusters as far as possible close together, and make the distance between clusters as large as possible.

Assuming the number of clusters K, the maximum number of iterations N, the input sample set data is:

D={x ₁ ,x ₂ …x _m } (1)

Among them, x _i is the input sample, i=0, 2, ..., m.

The k samples randomly selected from the dataset D as the centroid vector are:

μ={μ ₁ , μ ₂ ... μ _k } (2)

For each of the N iterations, the cluster division is initialized as:

And calculate the distance between each input sample calculation sample x _i (i=1,2...m) and each centroid vector μ _j (j=1,2...k), its expression is:

The sample _xi with the smallest distance d _ij among all samples is divided into the corresponding category λ _i , and the sample cluster update rule is:

The update rule for sample particles is:

Repeat the above steps (4), (5) and (6) until N iterations are completed or the clusters are not updated in the iterative process, and the final K segmentation clusters are obtained as:

C={C ₁ , C ₂ . . . C _k } (7)

In general, it is to update the training of the target's minimum square error E to the minimum, and its expression is:

Among them, β is the distance type of the cost function. When β=1, the Manhattan distance is used, and when β=2, the Euclidean distance is used.

According to the cluster update rule, it can be seen that after the K-means coarse segmentation layer, image areas with similar features are divided into the same cluster, the pavement area generally has similar feature distribution, and the roadside area features are distributed more discretely. The cluster with K=2 can be roughly segmented, and the features similar to the pavement area and the features dissimilar to it can be divided into two categories, and the segmentation effect of the pavement area with high recall rate can be obtained.

S100.3, using the feature extraction network to combine the low-latitude features of the road surface area into high-latitude features to obtain a high-dimensional feature map, and the specific processing is as follows:

The feature extraction network constructed by convolutional neural network is used to extract high-dimensional features of the pavement area. The neurons are composed of layers, and the weights and biases are trained and updated according to the back-propagation algorithm. The training uses the local and overall relationship of the input data to combine low-dimensional features into high-dimensional features to obtain different dimensions. Spatial correlation between features. By sharing the same level of convolution kernel parameters to achieve local connection and weight sharing, the prior knowledge is introduced into the convolutional neural network, which greatly reduces the difficulty of network training, and also makes the convolutional neural network suitable for image data. .

In the hierarchy of convolutional neural networks, there are mainly four basic hierarchical structures: convolutional layer; pooling layer; fully connected layer; activation layer.

A convolutional layer is a filter that can be trained to learn parameters. There are two main types of convolution kernels according to the filling method: convolution kernels with no edge filling and convolution kernels with edges filled with 0 pixels according to half of the size of the convolution kernel. The latter has an advantage as a filling method to prevent image After the multi-layer convolution operation, the image size is severely reduced to ensure that the feature map is not affected by the convolution operation. The size of the convolution kernels at the same level is the same, and one convolution kernel is responsible for extracting an image shape feature. There is no size limit on the size of the convolution kernels between different levels.

Suppose the two-dimensional convolution H is used to operate the two-dimensional feature image F, and its expression is:

Among them, F _layer is the two-dimensional feature map of the layer layer, F _layer+1 is the two-dimensional feature map of the (layer+1) layer, i and j are the image coordinate points corresponding to the convolution center, and m and n are the two-dimensional convolution respectively. length and width dimensions.

After the feature map goes through the convolutional layer, the size of the feature map of the next layer is:

height _layer+1 =(height _layer -m+2*padding)/stride+1 (10)

width _layer+1 =(width _layer -n+2*padding)/stride+1 (11)

Among them, the height _layer is the height of the two-dimensional feature map of the layer layer, the height _layer+1 is the height of the two-dimensional feature map of the (layer+1) layer, the width _layer is the width of the two-dimensional feature map of the layer layer, and the width _layer+1 is ( layer+1) The width of the two-dimensional feature map of the layer, padding is the edge padding size of the feature map, and stride is the step size of the convolution calculation.

As a transition layer between adjacent convolutional layers, the pooling layer can effectively compress the number of data and network parameters, and also help to prevent the network from overfitting. Pooling operations can be divided into two types: maximum pooling and average pooling. The former divides the feature map into corresponding regions according to the size of the pooling block, and selects the data with the largest feature in each region as the pooled feature map. The latter is to average the features in each corresponding region, and select the mean value as the pooled feature map parameter. Max pooling is suitable for extracting the features of the image object, and average pooling is suitable for averaging the features of the image background.

The fully connected layer is one of the initial forms of the neural network. Through the fully connected layer, the high-dimensional features of the previous network output can be connected into a slender feature vector and mapped to a linearly separable space for the output layer to realize the The classification of feature targets, and its output size is the type of data classification.

The activation layer mainly adds nonlinear factors to the linear model, removes redundancy in the data, and retains and maps the features extracted by the convolution layer to make the model fit better. In order to avoid gradient disappearance and neuron misfire, and make the loss curve converge faster, the ELU activation function is used, and its expression is:

Among them, x is the parameter information of the input activation layer, and α is the preset parameter of the activation layer. At the same time, Dropout and BatchNormalization methods are used to avoid the risk of overfitting and reduce the fitting situation of the deep network to the data.

Dropout randomly deactivates a number of proportional neurons in the current layer, calculates the loss of the predicted value and the label value through forward propagation, and updates the parameters of the network through back propagation, and the process is repeated. The deactivation is random at each iteration, resulting in the parameters being iteratively updated with different network structures, which can avoid overfitting on the one hand; reduce the complex co-adaptation relationship between neurons, and the weight update will not be affected by the existence of The influence of interconnected neurons, filtering features that make the model perform more general, forces the network to learn more robust features.

Batch Normalization is a method of normalizing the distribution of data. During deep network training, since the data distribution of the input layer has no fixed distribution, the distribution of the current layer will also change randomly. As a result, the training model needs to constantly adapt to learn new data distribution, and the learning rate should be set very small. Also very high. BatchNormalization is to solve this problem, by normalizing the data distribution to a fixed interval before the activation function, model training does not need to adapt to different distributions each time, so that it is better to use gradients for parameter update.

In the training phase, the mean value of the mini-batch samples is calculated as:

where m is the number of mini-batch samples.

The standard deviation of mini-batch samples calculated from the sample mean is:

Among them, δ is a preset parameter, X is each sample in the data set, and the sample distribution after normalization is:

The sample distribution for the replacement batch sample normalization is:

为样本标准差的学习参数。Among them, γ is the learning parameter of the sample mean,

is the learning parameter of the sample standard deviation.

The gradient update algorithm adopts the adaptive learning rate method RMSprop with faster convergence, and uses the gradient of the previous moment to update the current position gradient and learning rate. The gradient update formula is:

为当前时刻位置的梯度平方，η为学习速率，θ_t为前一时刻权重参数，θ_t+1为当前时刻权重参数。Among them, E[g ² ] _t-1 is the mean value of the gradient squares at the previous moment, E[g ² ] _t is the mean value of the gradient squares at the current moment,

is the square of the gradient at the current moment, η is the learning rate, θ _t is the weight parameter at the previous moment, and θ _t+1 is the weight parameter at the current moment.

In addition, for the convolution of small-sized feature maps, the combination of small convolution kernels is used to replace the large convolution kernel, and the combination of n×1 and 1×n convolution kernels at the back end of the network replaces the n×n convolution kernel. Effectively reduce the number of network parameters and speed up network convergence.

Based on the above embodiment, in step S200, the high-dimensional feature map is screened for positive and negative samples based on the deep high-dimensional features of the pavement area, so as to distinguish the pavement crack target and the pavement background, specifically including:

S200.1, based on the deep high-dimensional features of the pavement area, use the anchor sliding window to traverse the high-dimensional feature map, and obtain a candidate sample frame with a preset area scale and a preset aspect ratio, the candidate sample frame That is, the candidate sample area;

Wherein, the preset area scale may be 3 kinds of area scales; preferably, the preset area scale is 128 ² , 256 ² , and 512 ² . The preset aspect ratio may be three aspect ratios; preferably, the preset aspect ratio is 1:1, 1:3, and 3:1.

Specifically, the division basis of the positive and negative samples is:

The IOU of the candidate sample frame and any calibration sample frame is greater than the first preset threshold and 2 candidate sample regions with the largest IOU of the candidate sample frame and the remaining calibration sample frame are divided into positive samples;

The candidate sample frame and the candidate sample frame whose IOU of the calibration sample frame except the positive sample is smaller than the second preset threshold are divided into negative samples.

in,

IOU=(candidate sample frame∩calibration sample frame)/(candidate sample frame∪calibration sample frame).

Preferably, the first preset threshold is 0.7; the second preset threshold is 0.3.

S200.2, using the classification loss function, localization loss function and multi-task loss function of the regional nomination network of the feature extraction network, train the candidate sample frame to screen positive and negative samples, and obtain the pavement crack target sample and the pavement background samples, in which the positive samples are the pavement crack target samples, and the negative samples are the pavement background samples.

Specifically, the classification loss function of the regional nomination network is:

l _cls (p)=-(1-p _u ) ^γ logp _u ;

Among them, p _u is the probability that the calibration sample frame is a positive and negative sample, and γ is the Focal Loss training parameter;

The localization loss function of the regional nomination network is:

为前景和背景预测样本框的回归修正参数，u为第u个预测样本框，t_i为标定样本框的坐标信息，

损失函数为：Among them, _vi is the coordinate information of the candidate sample frame,

is the regression correction parameter of the foreground and background prediction sample frames, u is the u-th prediction sample frame, t _i is the coordinate information of the calibration sample frame,

The loss function is:

The multi-task loss function of the region nomination network is:

为离散的标记值。Among them, n _cls is the number of all samples, n _reg is the number of positive samples, μ is the normalized value of 0.2, p _i is the probability of being predicted as a crack target,

are discrete tag values.

Based on the above features, the specific processing of step S200.1 is:

First, according to the high-dimensional feature map extracted by the last convolutional layer of the feature extraction network, a fixed-size anchor sliding window is used to traverse the entire feature map, and the center of each sliding window generates different local receptive fields according to different levels. The expression for:

size _l-1 = stride×(size _l -1)+size _conv -2×padding (19)

Among them, size _l-1 is the size of the upper layer of the receptive field, size _l is the output size of the feature map, stride is the convolution moving step size, and padding is the convolution padding size.

The center of each sliding window corresponds to 9 kinds of candidate sample frames, and in the receptive field area corresponding to the center of the sliding window, the above-mentioned 3 kinds of aspect ratios and 3 kinds of area anchor sample frames are used to select candidate sample regions. Combined with the ^slender and ^slender characteristics of the fracture target, the preferred area scale and aspect ratio among the ^three are selected. 1.

Then, the obtained coordinates of the candidate region are mapped into a 1024-dimensional feature vector, which corresponds to the number of output channels of the convolution layer, and two loss functions need to be calculated separately: the classification loss function and the localization loss function. First, it is necessary to select the positive and negative samples that can be used for training in the candidate sample frame generated by the anchor mechanism, and use the IOU to calculate the overlap rate between the candidate sample frame and the adjacent training data calibration frame to select the samples for training. The expression is:

IOU=(candidate sample frame∩calibration sample frame)/(candidate sample frame∪calibration sample frame) (20)

The specific basis for dividing positive and negative samples is: dividing the candidate sample frame and any calibration sample frame IOU>0.7 and the candidate sample frame and the remaining calibration sample frame IOU The largest two candidate sample frames are divided into training positive samples; the candidate sample frame is divided into training positive samples; The sample frame and the candidate sample frame with the IOU<0.3 of the calibration sample frame except the positive sample are divided into training negative samples.

Based on the above features, the specific processing of step S200.2 is:

The classification loss function of the regional nomination network is used to distinguish the two types of samples, the crack target and the background area, and the actual samples of the calibration sample frame corresponding to the divided positive and negative samples are used. The sample of the calibration target is defined as 1, and the background area is defined as 0. Each calibration sample box corresponds to a discrete probability distribution, which can be expressed as:

p=(p ₀ ,p ₁ ) (21)

Among them, p ₀ is the probability that it is the background region, and p ₁ is the probability that it is the crack target region.

Secondly, because the proportion of positive and negative samples divided by the anchor mechanism cannot be guaranteed to be balanced, resulting in the problem of unbalanced categories, and most of the negative samples are areas that are simple and easy to judge as negative samples. Due to the large number of these simple samples, they will also affect the The loss function training has an impact, so that the loss function cannot converge to a better result. Therefore, combined with the idea of Focal Loss, the classification loss function of the regional nomination network is expressed as:

l _cls (p)=-(1-p _u ) ^γ logp _u (22)

Among them, p _u is the probability that the calibration sample frame is a positive and negative sample, γ is the Focal Loss training parameter, and the initial value is 5.

Finally, the position of the candidate sample frame divided by the receptive field is corrected by combining the training positive and negative samples with the positioning loss. Each training positive and negative sample corresponds to a set of coordinate frame size coordinate vectors, which are calibrated using the divided positive and negative sample coordinates and the data set. The sample coordinate mapping is calculated, and the mapping relationship between its borders is:

为边框修正样本框坐标信息，g(G_x,G_y,G_w,G_h)为标定样本框坐标信息，每个坐标信息包含4个定量坐标信息，分别为左上角横纵坐标和样本框宽高尺寸。Among them, f(P _x , P _y , P _w , P _h ) is the divided positive and negative sample frame coordinate information,

Correct the coordinate information of the sample frame for the frame, g(G _x , G _y , G _w , G _h ) is the coordinate information of the calibration sample frame, and each coordinate information contains 4 quantitative coordinate information, which are the abscissa and ordinate of the upper left corner and the sample frame. width and height dimensions.

Referring to the frame regression correction method in Faster-RCNN, a series of translation and scale scaling are used to convert into two groups of four feature vectors respectively.

The regression correction parameters for the calibration sample frame are:

The regression correction parameters of the predicted sample frame are:

The loss function of border correction adopts the L1 _loss function which is more robust to discrete points, and its expression is:

损失函数表达式为：in,

The loss function expression is:

The multi-task loss function for this region nomination network is then

为离散的标记值，其表达式为：Among them, n _cls is the number of all samples, n _reg is the number of positive samples, μ is the normalized value of 0.2, p _i is the probability of being predicted as a crack target,

is a discrete tag value whose expression is:

The positive and negative sample regions required for training are screened by the multi-task loss function of the regional nomination network, and the crack target and the road background are preliminarily classified, and the initial regression positioning of the crack target sample box is realized by combining with the calibration sample frame.

Based on the above embodiment, in step S300, the classification and coordinate positioning of the road surface crack target is performed to obtain the classification information and coordinate information of the road surface crack target, which specifically includes:

S300.1, using the ROI pooling layer of the feature extraction network to regularize the positive and negative samples screened by the regional nomination network into a feature map of uniform size, and perform classification output to obtain classification information including transverse cracks, longitudinal cracks and pavement background;

Specifically, when the sample frame generated by the regional nomination network for training classification and bounding box regression is combined with the convolutional neural network, the ROI pooling layer of the spatial pyramid pooling structure is used to divide the feature map into n×n size on average. Grid, the feature map part in a small grid is processed by maximum pooling, and only the extracted maximum is retained.

The output feature map of the last convolutional layer of the feature extraction network and the frame coordinates of the original image corresponding to the candidate sample frame output by the regional nomination network pass through the ROI pooling layer, the feature size is unified, and connected to the fully connected layer, of which the fully connected layer The output size is the number of categories of the classification network plus the background classification, that is, the output classification network parameters are 3 kinds, including transverse cracks, longitudinal cracks and pavement background. Compared with the crack target and the pavement background preliminarily classified in step S200, the classification in this step is more refined.

S300.2, using a classification loss function to classify the specific crack category of the positive sample, and perform frame regression to correct the coordinate information of the crack target frame.

Specifically, the classification loss function is used to classify the specific crack category of the positive sample candidate sample frame, each candidate sample frame corresponds to a discrete probability distribution, and its expression is:

P=(P ₀ ,P ₁ ,P ₃ ) (36)

Among them, P ₀ is the probability of being a background region, P ₁ is the probability that it is a transverse crack region, and P ₂ is the probability that it is a longitudinal crack region.

In the classification of crack targets, there is an imbalance in the number of transverse crack samples and longitudinal crack samples, so Focal Loss is used to design the classification loss function, and its expression is:

L _cls (P)=-(1-P _U ) ^γ logP _U (37)

Among them, _PU is the probability that the candidate sample frame is a positive and negative sample.

Use the loss function of the frame regression network to locate the coordinates of the classified positive sample candidate sample frame, and use the regression correction parameters T=(T _x ,T _y , _Tw ,T _h ) of the corresponding calibration sample frame described in S2.1 and The regression correction parameter V=(V _x , V _y , V _w , V _h ) of the predicted sample frame is corrected, and its expression is:

For this crack classification network, the multi-task loss function expression combining the classification loss function and the localization loss function is:

为离散的标记值，其表达式为：Among them, N _cls is the number of all samples, N _reg is the number of positive samples, _Pi is the probability of being predicted as a crack target,

is a discrete tag value whose expression is:

The crack targets initially identified by the regional nomination network are classified by the multi-task loss function, and the types of crack targets are specifically divided. Combined with the calibration sample frame, the secondary regression positioning of the crack target sample frame is realized, and the coordinate information of the crack target frame is corrected.

Based on the above embodiment, in step S400, calculating the length of the road surface crack target according to the classification information and coordinate information of the road surface crack target specifically includes:

S400.1, according to the classification information and coordinate information of the pavement crack target, use the morphological hit-miss transformation to extract the single-pixel skeleton of the crack target, and calculate the pixel length of the crack target; the crack target includes transverse cracks , longitudinal cracks;

S400.2, according to the pixel coordinates of the aerial photographed road surface image and the road length of the actual road surface, convert the pixel length of the crack target to obtain the length of the crack target.

Specifically, in step S400.1, through the morphological expansion operation, the fracture fragments broken during the crack detection process are connected, and the morphologically processed hit and miss transformation method is used to refine the expanded complete crack, so that the When the geometric scale of the crack remains unchanged, the redundant edge pixel information is removed, and the multi-pixel width problem caused by the crack target is removed. The crack skeleton composed of a single pixel width can more accurately reflect the detailed characteristics of the crack.

For the crack detection target, the structural elements in multiple directions are used to iteratively process, and the expression is:

Among them, f is the input structure image, s is a number of appropriate structure elements, and c is the number of iterations.

Suppose the definition of the structural element sequence s is:

{s}={s ¹ ,s ² ,...s ⁿ } (42)

then exists:

Iterative operation is performed on all the result elements using the methods of formula (3.21) to formula (3.23), and if there is no convergence, the operation is repeated for each structural element in turn until the result does not change.

Secondly, the Huber weight function of the least square fitting method is used to fit the discrete points as the skeleton curve. The standard least squares principle requires that the distance between each discrete point and the fitted skeleton curve is the smallest. However, it does not have good robustness for outliers far away from the skeleton curve, and it is necessary to set a weight threshold for processing.

The Huber weight function is:

Among them, τ represents the distance threshold, and δ is the distance between adjacent curves.

According to the image crack recognition effect and the distance δ of each adjacent curve, including noise interference and crack target, an appropriate threshold is selected to constrain it.

When the point-to-curve distance is less than or equal to a threshold τ, the weight is assigned to 1. When the point-to-curve distance is greater than the threshold, the weight function is equal to the reciprocal of the distance multiplied by the threshold. The farther the distance, the smaller the value. Preferably, τ=3.

The final step S400.2 combines the pixel length of the crack target calculated in S400.1 with parameters such as the flying height and speed of the drone, and converts the ratio of the image pixel to the actual length to obtain the actual length of the crack target.

The embodiment of the present invention can be combined with a display device to display relevant information, including road length, number of cracks, types of cracks, and information on the length of each section of cracks.

To sum up, a deep learning method for detecting road cracks in aerial images proposed by the embodiments of the present invention uses the K-means road rough segmentation layer with high recall rate to remove the invalid areas on the roadside; combined with the small target detection task in complex scenes Provide a feature extraction network structure to extract high-dimensional features of the pavement area; provide a region nomination structure that satisfies the feature extraction network of aerial image pavement crack detection to generate candidate sample frames, and preliminarily screen positive and negative samples of crack targets and pavement backgrounds; The classification and positioning structure is used to detect the crack target and the quadratic bounding box regression is used, and the single-pixel skeleton of the crack is extracted from the detection result area to calculate the length data of the crack target.

FIG. 2 is a schematic diagram of an embodiment of identifying road cracks in an aerial image according to an embodiment of the present invention. Please refer to FIG. 2 . An embodiment of the present invention samples a road surface image from an aerial photograph as an input. The invalid area on the roadside in the aerial image, combined with the small target detection task in complex scenes, a feature extraction network structure is designed to extract the high-dimensional features of the road surface area; according to step S200, a feature extraction method that meets the detection of road cracks in the aerial image is provided. The regional nomination structure of the network generates a candidate sample frame for the feature map of the last layer of the network in step 2, and preliminarily screens the positive and negative samples of the crack target and the road background; according to step S300, the classification and positioning structure is used to classify the crack target and the secondary frame. Regression; according to step S400, the single-pixel skeleton of cracks is extracted from the detection result area using the hit and miss transformation, and the crack length is further calculated according to the image coordinates and the actual road surface size ratio; finally, the crack image and related information are output, including the road length and the number of cracks. , the types of cracks and the information of the length of each section of cracks for display equipment to display.

For aerial images, the embodiments of the present invention can overcome the difficulty of image processing caused by the collection method of unmanned aerial vehicles, can be applied to the detection of road cracks with high-altitude motion backgrounds and complex scenes, and have better robustness than various commonly used crack detection algorithms. Great for better recognition of cracks in aerial images. The invention is applied to aerial photographic crack detection, not only can automatically detect, classify and locate crack targets, but also can quantitatively analyze the crack length to provide reference for subsequent road maintenance.

The embodiment of the present invention performs operation processing on the road surface image based on deep learning, which mainly includes three steps: firstly, extracting the deep high-dimensional feature map of the road surface area; secondly, generating and screening valid positive and negative samples for training and preliminarily distinguishing the road surface Crack target and pavement background, thirdly, classify the crack target and locate the specific coordinates, and further calculate the length of the crack target skeleton. In each processing step, different algorithm networks are used to process and analyze the image data, and different hyperparameters are set to perform initial distinction, detailed classification and coordinate positioning of the pavement crack target and pavement background, so as to realize automatic accurate classification and positioning of pavement cracks. and quantitative skeleton calculations.

The algorithm selected in the embodiment of the present invention is especially suitable for the identification of road cracks in aerial images with a large amount of data and a lot of interference noise, which solves the deficiencies of the prior art and has good beneficial effects.

The embodiment of the present invention also provides an aerial image pavement crack detection device, including:

The high-dimensional feature map module is used to extract the deep high-dimensional features of the road surface area of the aerial photographic road image, and obtain the high-dimensional feature map according to the deep high-dimensional features;

a crack identification module for screening positive and negative samples of the high-dimensional feature map based on the deep high-dimensional features of the pavement area, so as to distinguish the pavement crack target from the pavement background; and

The classification and positioning module is used for classifying and coordinate positioning of the road surface crack target, and obtaining classification information and coordinate information of the road surface crack target.

The device in the embodiment of the present invention can be used to implement the technical solution of the embodiment of the method for detecting road cracks in aerial images shown in FIG.

FIG. 3 is a schematic diagram of a framework of an aerial image pavement crack detection device according to an embodiment of the present invention. Referring to FIG. 3 , an embodiment of the present invention provides an aerial image pavement crack detection device, including: a processor 310 , a communications interface 320 , a memory 330 and a bus 340 , wherein the processor 310 , the communication interface 320 and the memory 330 complete the communication with each other through the bus 340 . The processor 310 may invoke the logic instructions in the memory 330 to perform the following method, including: extracting deep high-dimensional features of the road surface area of the aerial road surface image, obtaining a high-dimensional feature map according to the deep high-dimensional features; based on the road surface area The deep high-dimensional features of the high-dimensional feature map are screened for positive and negative samples to distinguish the pavement crack target and the pavement background; the pavement crack target is classified and coordinate location is obtained to obtain the classification information and Coordinate information.

An embodiment of the present invention discloses a computer program product, where the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer, The computer can execute the methods provided by the above method embodiments, for example, including: extracting the deep high-dimensional features of the road surface area of the aerial road image, and obtaining a high-dimensional feature map according to the deep high-dimensional features; The high-dimensional feature map is screened for positive and negative samples to distinguish the pavement crack target and the pavement background; the pavement crack object is classified and coordinate positioning is performed to obtain the classification information and coordinate information of the pavement crack object.

Embodiments of the present invention provide a non-transitory computer-readable storage medium, where the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the methods provided by the foregoing method embodiments, for example The method includes: extracting the deep high-dimensional features of the pavement area of the aerial photographic road image, and obtaining a high-dimensional feature map according to the deep high-dimensional features; and performing positive and negative samples on the high-dimensional feature map based on the deep high-dimensional features of the pavement area. Screening to distinguish the pavement crack target from the pavement background; classify and coordinate the pavement crack object to obtain classification information and coordinate information of the pavement crack object.

Those of ordinary skill in the art can understand that the implementation of the above device embodiments or method embodiments is merely illustrative, wherein the processor and the memory may be physically separated components or may not be physically separated, that is, they may be located in One place, or it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in computer-readable storage media, such as U disk, mobile hard disk , ROM/RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to perform the methods described in various embodiments or parts of embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (7)

1. an aerial image pavement crack detection method, is characterized in that, comprises: extracting deep high-dimensional features of the pavement area of the aerial photographed road surface image, and obtaining a high-dimensional feature map according to the deep high-dimensional features; Based on the deep high-dimensional features of the pavement area, the high-dimensional feature map is screened for positive and negative samples to distinguish the pavement crack target and the pavement background; classifying and locating the coordinates of the road surface crack target to obtain classification information and coordinate information of the road surface crack target; Wherein, extracting the deep high-dimensional features of the road surface area of the aerial photographic road surface image, and obtaining a high-dimensional feature map according to the deep high-dimensional features, specifically includes: A feature extraction network is constructed by using a convolutional neural network, and a rough road segmentation layer based on the K-means clustering algorithm is added to the feature extraction network; Use the rough road segmentation layer to filter out the roadside invalid area of the aerial photographed road surface image to obtain the road surface area of the aerial photographed road surface image; Using the feature extraction network to combine the low-dimensional features of the road surface area into high-dimensional features to obtain a high-dimensional feature map; The high-dimensional feature map is screened for positive and negative samples based on the deep high-dimensional features of the pavement area, so as to distinguish the pavement crack target and the pavement background, specifically including: Described based on the deep high-dimensional features of the pavement area, using the anchor sliding window to traverse the high-dimensional feature map to obtain candidate sample frames with a preset area scale and a preset aspect ratio; The candidate sample frame is trained by using the classification loss function, localization loss function and multi-task loss function of the regional nomination network of the feature extraction network to screen positive and negative samples, and obtain road surface crack target samples and road background samples, wherein the positive samples are is the pavement crack target sample, and the negative sample is the pavement background sample; The classification and coordinate positioning of the road surface crack target to obtain the classification information and coordinate information of the road surface crack target specifically include: Use the ROI pooling layer of the feature extraction network to regularize the positive and negative samples screened by the regional nomination network into a feature map of uniform size, and perform classification output to obtain classification information including transverse cracks, longitudinal cracks and pavement background; The classification loss function is used to classify the specific crack category of the positive sample, and the frame regression is performed to correct the coordinate information of the crack target frame. 2. The method according to claim 1, wherein the classification and coordinate positioning of the road surface crack target are performed to obtain classification information and coordinate information of the road surface crack target, and further comprising: According to the classification information and coordinate information of the pavement crack object, the length of the pavement crack object is calculated. 3. The method according to claim 1, wherein the division basis of the positive and negative samples is: The IOU of the candidate sample frame and any calibration sample frame is greater than the first preset threshold and 2 candidate sample regions with the largest IOU of the candidate sample frame and the remaining calibration sample frame are divided into positive samples; dividing the candidate sample frame and the candidate sample frame whose IOU of the calibration sample frame except the positive sample is less than the second preset threshold into a negative sample; in, IOU=(candidate sample frame∩calibration sample frame)/(candidate sample frame∪calibration sample frame). 4. The method according to claim 2, wherein the calculating the length of the road surface crack target according to the classification information and coordinate information of the road surface crack target specifically comprises: According to the classification information and coordinate information of the pavement crack target, the single-pixel skeleton of the crack target is extracted by morphological hit-miss transformation, and the pixel length of the crack target is calculated; the crack target includes transverse cracks and longitudinal cracks; According to the pixel coordinates of the aerial photographed road surface image and the road length of the actual road surface, the pixel length of the crack target is converted to obtain the length of the crack target. 5. An aerial image pavement crack detection device, characterized in that, comprising: The high-dimensional feature map module is used to extract the deep high-dimensional features of the road surface area of the aerial photographic road image, and obtain the high-dimensional feature map according to the deep high-dimensional features; a crack identification module for screening positive and negative samples of the high-dimensional feature map based on the deep high-dimensional features of the pavement area, so as to distinguish the pavement crack target from the pavement background; and A classification and positioning module, used for classifying and coordinate positioning of the road surface crack target, and obtaining classification information and coordinate information of the road surface crack target; Wherein, the high-dimensional feature map module is also used for, A feature extraction network is constructed by using a convolutional neural network, and a rough road segmentation layer based on the K-means clustering algorithm is added to the feature extraction network; Use the rough road segmentation layer to filter out the roadside invalid area of the aerial photographed road surface image to obtain the road surface area of the aerial photographed road surface image; Using the feature extraction network to combine the low-dimensional features of the road surface area into high-dimensional features to obtain a high-dimensional feature map; The classification and positioning module is also used for: Described based on the deep high-dimensional features of the pavement area, using the anchor sliding window to traverse the high-dimensional feature map to obtain candidate sample frames with a preset area scale and a preset aspect ratio; The candidate sample frame is trained by using the classification loss function, localization loss function and multi-task loss function of the regional nomination network of the feature extraction network to screen positive and negative samples, and obtain road surface crack target samples and road background samples, wherein the positive samples are is the pavement crack target sample, and the negative sample is the pavement background sample; The classification and positioning module is also used for: Use the ROI pooling layer of the feature extraction network to regularize the positive and negative samples screened by the regional nomination network into a feature map of uniform size, and perform classification output to obtain classification information including transverse cracks, longitudinal cracks and pavement background; The classification loss function is used to classify the specific crack category of the positive sample, and the frame regression is performed to correct the coordinate information of the crack target frame. 6. An aerial image pavement crack detection device, characterized in that, comprising: at least one processor; and at least one memory communicatively coupled to the processor, wherein: The memory stores program instructions executable by the processor, and the processor invokes the program instructions to be able to perform the method as claimed in any one of claims 1 to 4. 7 . A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, the computer instructions cause the computer to execute any one of claims 1 to 4 . Methods.

Images