CN114677659A - Model construction method, system and intelligent terminal for road surface flatness detection - Google Patents

Abstract

The invention discloses a model construction method, a system and an intelligent terminal for detecting the flatness of a pavement, wherein the method comprises the following steps: acquiring a multi-frame original image of a target scene, extracting an RGB (red, green and blue) image from the original image, and generating a depth map from the original image; generating a data set by the RGB image and the depth map, dividing the data set into a training set, a verification set and a test set according to a preset proportion, and labeling each data set; and respectively inputting RGB-D data into a training set and a verification set based on a pre-stored classification detection network model, and obtaining a road surface flatness detection model through model training. The technical problem of lower road surface flatness detection accuracy among the prior art is solved.

Description

Model construction method, system and intelligent terminal for road surface flatness detection

technical field

The invention relates to the technical field of automatic driving assistance, in particular to a model building method, system and intelligent terminal for road flatness detection.

Background technique

With the development of autonomous driving technology, people's requirements for the safety and comfort of assisted driving vehicles are also increasing. Autonomous driving is the product of the deep integration of the automotive industry with emerging information technologies such as artificial intelligence and the Internet of Things. It is the main direction of the development of intelligence and networking in the global automotive and transportation fields. It can improve the driving experience and safety by detecting the smoothness of the road. Sexuality has also become one of the core issues in the field of autonomous driving. Judging from the current urban road conditions, road surface irregularities caused by road damage and speed bumps or manhole covers necessary for urban infrastructure will affect the driving experience of drivers and the riding experience of passengers, and even increase the risk factor of car driving. . Therefore, effectively detecting the smoothness of the road surface plays a crucial role in the driving of the vehicle.

However, traditional image processing algorithms have strict data requirements, and interference factors such as road shadows and uneven illumination make it difficult to give accurate detection results. The classic deep learning model is mainly for classification and identification of large-scale and large targets, and it is easy to cause missed detection and misjudgment for grooves and cracks with different shapes on the road surface.

SUMMARY OF THE INVENTION

To this end, the embodiments of the present invention provide a model construction method, system and intelligent terminal for road surface flatness detection, in order to at least partially solve the technical problem of low road surface flatness detection accuracy in the prior art.

In order to achieve the above purpose, the embodiments of the present invention provide the following technical solutions:

A model building method for road surface flatness detection, the method comprising:

Obtaining multiple frames of original images of the target scene, extracting RGB images from the original images, and generating a depth map from the original images;

generating a data set from the RGB image and the depth map, dividing the data set into a training set, a verification set and a test set according to a preset ratio, and labeling each data set;

Based on the pre-stored classification detection network model, the RGB-D data are input into the training set and the validation set respectively, and the road flatness detection model is obtained through model training.

Further, acquiring multiple frames of original images of the target scene, extracting RGB images from the original images, and generating a depth map from the original images, specifically including:

Use the binocular camera to collect the binocular road image, and use the binocular road image as the original image;

extracting an RGB image from the original image according to the color three-channel principle;

A depth map is generated from the original image according to a stereo matching and three-dimensional reconstruction method, and the depth map has the same size as the RGB image.

Further, generate a data set from the RGB image and the depth map, and divide the data set into a training set, a verification set and a test set according to a preset ratio, specifically including:

Generate a dataset from the left-eye RGB image and depth map;

The dataset is divided into training set, validation set and test set in a ratio of 3:1:1.

Further, label each data set, including:

The training set and the validation set of the left-eye images are labeled, and the labeled categories include at least one of longitudinal cracks, transverse cracks, network cracks, pits, speed bumps, and manhole covers.

Further, input the RGB-D data into the training set and the validation set respectively, including:

The RGB image is uniformly scaled to the same size, and the RGB image and disparity map are filled with black edges adaptively to obtain a feature map, and the feature map is input into the training set and the validation set for training.

Further, the output of the classification detection network model adopts CIoU_Loss as the loss function of the bounding box:

为权重系数，其中v是衡量长宽比一致性的参数：Among them, A and B are the areas of the predicted bounding box and the ground truth, respectively, |A∩B| represents the area of the intersection of the two boxes, |A∪B| represents the area of the union of the two boxes, and IoU is the intersection of the areas of the two boxes and compare. D1 is the square of the Euclidean distance between the center points of the two boxes, D2 is the square of the length of the diagonal of the smallest rectangle that can just contain the two boxes,

is the weight coefficient, where v is a parameter that measures the consistency of the aspect ratio:

这两个惩罚项。In the above formula, w ^gt is the width of the real annotation frame, h ^gt is the height of the real annotation frame; w is the width of the target detection frame, and h is the height of the target detection frame. On the basis of IoU, the CIoU_Loss function increases the distance between the center points of the two frames and the aspect ratio.

these two penalties.

Further, the method further includes a post-processing step, and the post-processing step includes:

For the screening of multi-object boxes, a weighted non-maximum suppression algorithm is used to remove some redundant bounding boxes. Compared with the traditional non-maximum suppression, the weighted non-maximum suppression does not directly remove the prediction box whose IoU is greater than the threshold and the same category as the real annotation box in the process of executing the rectangular box culling, but according to The confidence of the network prediction is weighted to obtain a new rectangular frame, which is used as the final predicted rectangular frame, and then other redundant frames are eliminated.

The present invention also provides a model building system for road surface flatness detection, the system comprising:

a data acquisition unit for acquiring multiple frames of original images of the target scene, extracting RGB images from the original images, and generating a depth map from the original images;

a data set generation unit, configured to generate a data set from the RGB image and the depth map, and divide the data set into a training set, a verification set and a test set according to a preset ratio, and label each data set;

The model output unit is used to input the RGB-D data into the training set and the verification set based on the pre-stored classification detection network model, and obtain the road surface flatness detection model through model training.

The present invention also provides an intelligent terminal, the intelligent terminal includes: a data acquisition device, a processor and a memory;

The data collection device is used to collect data; the memory is used to store one or more program instructions; the processor is used to execute one or more program instructions, so as to execute the above method.

The present invention also provides a computer-readable storage medium containing one or more program instructions for executing the method as described above.

The model construction method for road surface flatness detection provided by the present invention obtains multiple frames of original images of a target scene, extracts RGB images from the original images, and generates a depth map from the original images; The image and the depth map generate a data set, and the data set is divided into a training set, a verification set and a test set according to a preset ratio, and each data set is marked; based on the pre-stored classification detection network model, the RGB- D data are input into training set and validation set respectively, and the road surface flatness detection model is obtained through model training. The method uses RGB-D images for target detection, and has a good effect on detecting targets such as road cracks, potholes, speed bumps, and manhole covers, and solves the technical problem of low road flatness detection accuracy in the prior art.

Description of drawings

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that are required to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only exemplary, and for those of ordinary skill in the art, other implementation drawings can also be obtained according to the extension of the drawings provided without creative efforts.

The structures, proportions, sizes, etc. shown in this specification are only used to cooperate with the contents disclosed in the specification, so as to be understood and read by those who are familiar with the technology, and are not used to limit the conditions for the implementation of the present invention, so there is no technical The substantive meaning above, any modification of the structure, the change of the proportional relationship or the adjustment of the size should still fall within the technical content disclosed in the present invention without affecting the effect and the purpose that the present invention can produce. within the range that can be covered.

1 is a flow chart of a specific embodiment of a model building method for road surface flatness detection provided by the present invention;

FIG. 2 is a schematic diagram of a combination of an RGB image and a depth map to form an RGB-D image;

Figure 3 is a network architecture diagram;

4 is a schematic diagram of feature transfer;

5 is a schematic diagram of feature fusion;

FIG. 6 is a structural block diagram of a specific embodiment of the model building system for road surface flatness detection provided by the present invention.

Detailed ways

The embodiments of the present invention are described below by specific specific embodiments. Those who are familiar with the technology can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the present invention. , not all examples. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In order to solve the problem of low detection accuracy of road surface flatness, the present invention provides a road surface flatness detection method based on deep learning, which trains a network based on RGB-D image data. The detection accuracy is high, the detection speed is fast, and it is suitable for the urban complex road environment.

In a specific embodiment, as shown in FIG. 1 , the model construction method for road surface flatness detection provided by the present invention includes the following steps:

S1: Acquire multiple frames of original images of the target scene, extract RGB images from the original images, and generate a depth map from the original images.

Specifically, the binocular road image is collected by using a binocular camera, and the binocular road image is used as the original image; RGB images are extracted from the original image according to the principle of color three-channel; A depth map is generated, the depth map is the same size as the RGB image.

S2: Generate a data set from the RGB image and the depth map, divide the data set into a training set, a verification set and a test set according to a preset ratio, and label each data set.

A dataset is generated from the left-eye RGB image and the depth map; the dataset is divided into training set, validation set and test set according to the ratio of 3:1:1.

When labeling each data set, label the training set and validation set of left-eye images, and the labeling categories include at least one of longitudinal cracks, transverse cracks, network cracks, pits, speed bumps, and manhole covers.

When inputting the RGB-D data into the training set and the validation set respectively, firstly, the RGB images are uniformly scaled to the same size, and the RGB images and disparity maps are adaptively filled with black borders to obtain feature maps, and the feature maps are input into the training set and the disparity map. Train on the validation set.

In the model training process, it is first necessary to prepare a data set, use a binocular camera to collect binocular road images, and generate a depth map according to the relevant principles of stereo matching and 3D reconstruction. The depth image and the binocular image have the same size. The left-eye image and depth map dataset are divided into training set, validation set and test set according to the ratio of 3:1:1, and the training set and validation set of left-eye image are manually marked according to the unified standard. The labeling categories include: longitudinal cracks (D00), transverse cracks (D10), network cracks (D20), potholes (D40), speed bumps, manhole covers.

After the data set is prepared, add network channels for model training. Generally, the input of the deep learning network is a color three-channel RGB image. Using this kind of data, the scale and position information of the target object are captured relatively vaguely, and the false detection rate is high. In response to this problem, the present invention adds a channel at the input end to store the depth information of the image, that is, the input is an RGB-D image. As shown in Figure 2, RGB-D is a combination of an RGB image and a depth map, including three RGB images. The channel and a depth channel representing the distance between the pixel and the imaging device have a total of four channels, and the depth map and RGB image are processed in a similar way in the network. The information in the depth map and the RGB image are complementary, with the RGB image providing distinct appearance features and the depth image conveying the geometric structure. Experiments using RGB-D data can better overcome the uncertainty caused by the occlusion and overlap of the target object, and are more sensitive to the contour and surface features of the object. According to the registration of the depth image and the color image, the ICP (Iterative Closest Point) algorithm is used to align the 3D model with the points within the 2D detection window. In addition, since the RGB image and the depth map are registered, there is a one-to-one correspondence between the pixels, and the location information of the real annotation box and the predicted bounding box of the two are also consistent.

S3: Based on the pre-stored classification detection network model, the RGB-D data are input into the training set and the validation set respectively, and the road flatness detection model is obtained through model training.

和

在此缩放系数下，原始图像的尺寸被缩小到416＊312，故需对其高度方向填充黑边；原本需要填充的高度为416－312＝104，取其整除32的余数8并除以2，得到最终在图像高度方向两端所需填充的最小黑边数值。After inputting the training set and validation set data, randomly rotate, crop, splicing, and arrange the training set data at the input end, uniformly scale the original image to the same size (416*416), and adaptively provide the original input image and parallax The information is filled with the least black borders to reduce information redundancy, and finally input into the Backbone and Neck network structures for training. For example, the original input image size is 800*600, then the scaling factors of its width and height are respectively

and

Under this scaling factor, the size of the original image is reduced to 416*312, so it needs to be filled with black borders in the height direction; the original height to be filled is 416-312=104, take the remainder 8 of the division of 32 and divide by 2 , to get the minimum black border value that needs to be filled at both ends of the image height direction.

The network structure is shown in Figure 3. First, in the SubNet1 structure of Backbone, the thickness of the feature map is deepened by slicing operation to enhance the network's ability to learn features, and then the convolution operation is performed. The convolution here adopts the CSPNet structure, which improves and solves the problem of excessive calculation amount caused by redundant gradient information, and integrates the change of gradient into the feature map from beginning to end, so as to ensure the features while achieving lightweight calculation. learning ability. The specific implementation is shown in Figure 4. The feature is mapped into two parts, one part is processed by Conv convolution and BN (batch normalization) batch normalization, and the activation function Leaky relu is selected, and the other part is directly concat spliced with the output of the convolution.

The Neck part consists of a top-down feature pyramid connected with a bottom-up feature pyramid. The top-level features are fused with the shallow features through upsampling, and the strong localization information of the shallow features is passed on through downsampling, so that It can make better use of high-level strong semantic information and low-level strong localization information. In addition, the ROI pooling layer is used to pool the features of each layer to obtain a feature matrix of the same size, and finally these features are fused. This operation can effectively enhance the adaptation of the network to the features and obtain better classification. , regression, prediction effect.

The output uses CIoU_Loss as the loss function of the bounding box,

为权重系数，其中v是衡量长宽比一致性的参数：Among them, A and B are the areas of the predicted bounding box and the ground truth, respectively, |A∩B| represents the area of the intersection of the two boxes, |A∪B| represents the area of the union of the two boxes, and IoU is the intersection of the areas of the two boxes and compare. D1 is the square of the Euclidean distance between the center points of the two boxes, D2 is the square of the length of the diagonal of the smallest rectangle that can just contain the two boxes,

is the weight coefficient, where v is a parameter that measures the consistency of the aspect ratio:

这两个惩罚项。In the above formula, w ^gt is the width of the real annotation frame, h ^gt is the height of the real annotation frame; w is the width of the target detection frame, and h is the height of the target detection frame. On the basis of IoU, the CIoU_Loss function increases the distance between the center points of the two frames and the aspect ratio.

these two penalties.

Further, the method further includes a post-processing step, and the post-processing step includes:

For the screening of multi-object boxes, a weighted non-maximum suppression algorithm is used to remove some redundant bounding boxes. Compared with the traditional non-maximum suppression, the weighted non-maximum suppression does not directly remove the prediction box whose IoU is greater than the threshold and the same category as the real annotation box in the process of executing the rectangular box culling, but according to The confidence of the network prediction is weighted to obtain a new rectangular frame, which is used as the final predicted rectangular frame, and then other redundant frames are eliminated.

After the model is constructed, in order to verify the accuracy of the model, the model can also be evaluated. The specific methods are as follows:

Among them, TP is True Positive, indicating the number of samples that are actually positive samples and predicted to be positive samples; FP is False Positive, indicating the number of samples that are actually negative samples predicted to be positive samples; FN is False Negative, indicating that the actual positive samples are predicted to be negative samples the number of samples. Recall is the model recall rate, which reflects the proportion of positive samples that are predicted to be positive. The higher the Recall, the stronger the model's ability to identify positive samples. Precision is the precision, which reflects the proportion of true positive samples that are predicted to be positive samples. F1-score, or F1 score, is a measure of classification problems. It calculates the harmonic mean of precision and recall, with a maximum of 1 and a minimum of 0.

The method provided by the present invention has the following technical effects:

1. Improve the accuracy of road detection by designing a target detection network based on RGB-D images.

Specifically, in many of the previous technologies, only monocular vision is used, and the spatial point coordinate distribution is fitted through a machine learning model combined with geometric constraints. In contrast, binocular vision has additional parallax information, which can reconstruct the depth information of the scene, so it can obtain stronger spatial constraints than monocular vision.

Aiming at the problem that the RGB-based pavement detection model has a large probability of missed detection and false detection, the present invention can provide the target contour and the target surface concave feature according to the depth map, and the color image can provide the target surface texture feature. RGB-D based object detection network. RGB is a three-channel color image. The three channels correspond to the three primary color values of Red, Green, and Blue. The value of each primary color is an integer between 0 and 255. RGB-D has an additional Depth channel on the basis of RGB, which uses the depth value Depth as a pixel, which reflects the actual distance between the camera and the object.

Among them, b and f are the internal parameters of the camera, representing the baseline distance and focal length, respectively; disparity is the binocular disparity obtained by stereo matching. The information in Depth and RGB images are complementary, with RGB images providing distinct appearance features and depth images conveying geometric structure.

The present invention detects the target object in the 2D image plane by treating the depth image as an extra channel of the color image, and uses the ICP algorithm to align the 3D model with the points within the 2D detection window. The purpose of the ICP algorithm is to merge the three-dimensional data in different coordinate systems into the same coordinate system. The algorithm selects the corresponding relationship point pairs repeatedly, calculates the optimal rigid transformation until it meets the convergence accuracy requirements of the correct registration, and outputs this. The rigid transformation matrix under the condition is used to transform the original 3D data.

Experiments using RGB-D data can better overcome the uncertainty caused by the occlusion and overlap of the target object, and are more sensitive to the contour and surface features of the object. By extracting more sufficient feature information, the road surface flatness can be effectively improved detection accuracy.

The experimental results show that the network can be well generalized to different data sets after proper training, and can be based on the current training weights without additional training, and the detection effect is also better than the network using only RGB images.

2. Use data augmentation to enrich the training set to ensure data accuracy.

In view of the small amount of data and the lack of model generalization ability, this network enhances the data set at the input end, that is, random rotation, cropping, splicing, and arrangement of training samples. The specific implementation method is: read 6 training samples each time, randomly rotate these 6 images by any angle, and randomly crop some areas in the samples, and fill them with 0 pixel values; then randomize these six images. Each sample image has its corresponding real annotation frame. After splicing 6 images, a new image is obtained, and the frame corresponding to this image is also obtained, and then the new image is passed in Learning in the network is equivalent to passing in 6 images with rich backgrounds at the same time, which enhances the diversity of the scene and greatly enriches the training data set. In addition, a random scaling strategy is adopted to increase the number of small targets and make the network more robust.

3. Double pyramids are connected in series to make full use of feature information to enhance feature fusion performance.

Aiming at the problem of low detection and positioning accuracy of small obstacles, small potholes and cracks on the road, the present invention can provide strong positioning information according to low-level features, and high-level features can provide strong semantic information, and uses two feature pyramids . As shown in the Neck part of Fig. 1, first the top-down pyramid transfers the high-level strong semantic features through upsampling, and then a bottom-up pyramid is connected in series to supplement the previous pyramid, and in order to shorten the Paths between feature layers, establishing lateral connections from bottom-level features to top-level features. The fusion method between high-level features and low-level features is shown in Figure 5.

It can be seen from FIG. 5 that the double pyramid obtains the next layer Ni+1 by fusing a lower layer Ni and a higher layer P _{i + 1} . Taking the calculation between N ₂ and N ₃ as an example, firstly, N ₂ is down-sampled by a 3*3 convolution with a step size of 2, and then P ₃ and the feature matrix obtained by down-sampling are processed by unit addition. Fusion, and then use a 3*3 convolution to fuse the features to enhance the representation ability of the fused features. The concatenation of two feature pyramids can greatly preserve the diversity and completeness of features, resulting in more accurate prediction of target bounding boxes.

4. The CIoU loss function is adopted to improve the detection performance in the case of overlapping targets.

Aiming at the situation of missing detection when the target objects on the road are densely overlapped, the present invention adopts the CIoU_Loss function in the bounding box regression part. It adds an impact factor v based on the DIoU_Loss function, which can be used to measure the proportional consistency between the Anchor box and the target box.

The CIoU_Loss function adds two penalty items on the basis of ordinary IoU. One is the penalty item for the distance of the center point for the overlapping of the bounding box, and the other is the penalty item for the relative proportion of the predicted frame and the real label frame. This part of the improvement makes CIoU_Loss With faster convergence speed and more robust detection effect, the regression loss can be more accurate and converge faster when the target frame overlaps or is occluded.

In addition to the above method, the present invention also provides a model building system for road surface flatness detection, as shown in FIG. 6 , the system includes:

a data acquisition unit 100 for acquiring multiple frames of original images of a target scene, extracting RGB images from the original images, and generating a depth map from the original images;

A data set generating unit 200 is used to generate a data set from the RGB image and the depth map, and divide the data set into a training set, a verification set and a test set according to a preset ratio, and label each data set ;

The model output unit 300 is configured to input the RGB-D data into the training set and the verification set respectively based on the pre-stored classification detection network model, and obtain a road surface flatness detection model through model training.

In the above specific embodiment, the model construction system for road flatness detection provided by the present invention obtains multiple frames of original images of the target scene, extracts RGB images from the original images, and generates the original images. Depth map; generate a data set from the RGB image and the depth map, and divide the data set into a training set, a verification set and a test set according to a preset ratio, and label each data set; based on the pre-stored classification Detect the network model, input the RGB-D data into the training set and the validation set respectively, and obtain the road flatness detection model through model training. The method uses RGB-D images for target detection, and has a good effect on detecting targets such as road cracks, potholes, speed bumps, and manhole covers, and solves the technical problem of low road flatness detection accuracy in the prior art.

The present invention also provides an intelligent terminal, the intelligent terminal includes: a data acquisition device, a processor and a memory;

The data collection device is used to collect data; the memory is used to store one or more program instructions; the processor is used to execute one or more program instructions, so as to execute the above method.

Corresponding to the foregoing embodiments, an embodiment of the present invention further provides a computer storage medium, where the computer storage medium contains one or more program instructions. Wherein, the one or more program instructions are used for performing the above method by a binocular camera depth calibration system.

In this embodiment of the present invention, the processor may be an integrated circuit chip, which has signal processing capability. The processor may be a general-purpose processor, a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field programmable gate array (FPGA for short), or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

Various methods, steps, and logical block diagrams disclosed in the embodiments of the present invention can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present invention may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The processor reads the information in the storage medium, and completes the steps of the above method in combination with its hardware.

The storage medium may be memory, eg, may be volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory.

Among them, the non-volatile memory may be a read-only memory (Read-Only Memory, referred to as ROM), a programmable read-only memory (Programmable ROM, referred to as PROM), an erasable programmable read-only memory (Erasable PROM, referred to as EPROM) , Electrically Erasable Programmable Read-Only Memory (Electrically EPROM, EEPROM for short) or flash memory.

The volatile memory may be a random access memory (Random Access Memory, RAM for short), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM, referred to as SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, referred to as DDRSDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, referred to as ESDRAM), synchronous connection dynamic random access memory (Synchlink DRAM) , referred to as SLDRAM) and direct memory bus random access memory (DirectRambus RAM, referred to as DRRAM).

The storage medium described in the embodiments of the present invention is intended to include, but not be limited to, these and any other suitable types of memory.

Those skilled in the art should appreciate that, in one or more of the above examples, the functions described in the present invention may be implemented by a combination of hardware and software. When the software is applied, the corresponding functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage medium can be any available medium that can be accessed by a general purpose or special purpose computer.

The above specific embodiments further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention, and are not intended to limit the protection scope of the present invention. On the basis of the technical solutions of the present invention, any modifications, equivalent replacements, improvements, etc. made shall be included within the protection scope of the present invention.

Claims (10)

1. a model construction method for road surface flatness detection, is characterized in that, described method comprises: Obtaining multiple frames of original images of the target scene, extracting RGB images from the original images, and generating a depth map from the original images; generating a data set from the RGB image and the depth map, dividing the data set into a training set, a verification set and a test set according to a preset ratio, and labeling each data set; Based on the pre-stored classification detection network model, the RGB-D data are input into the training set and the validation set respectively, and the road flatness detection model is obtained through model training. 2. The method for building a model according to claim 1, wherein obtaining multiple frames of original images of the target scene, extracting RGB images from the original images, and generating a depth map from the original images, specifically comprising: Use the binocular camera to collect the binocular road image, and use the binocular road image as the original image; extracting an RGB image from the original image according to the color three-channel principle; A depth map is generated from the original image according to a stereo matching and three-dimensional reconstruction method, and the depth map has the same size as the RGB image. 3. The method for building a model according to claim 1, wherein a dataset is generated from the RGB image and the depth map, and the dataset is divided into a training set, a verification set and a test set according to a preset ratio set, including: Generate a dataset from the left eye RGB image and depth map; The dataset is divided into training set, validation set and test set in a ratio of 3:1:1. 4. The model construction method according to claim 3, wherein each data set is marked, specifically comprising: The training set and the validation set of the left-eye images are labeled, and the labeled categories include at least one of longitudinal cracks, transverse cracks, network cracks, pits, speed bumps, and manhole covers. 5. The method for building a model according to claim 4, wherein the RGB-D data are input into the training set and the verification set, respectively, comprising: The RGB image is uniformly scaled to the same size, and the RGB image and disparity map are filled with black borders adaptively to obtain a feature map, and the feature map is input into the training set and validation set for training. 6. The model construction method according to claim 5, wherein the output of the classification detection network model adopts CIoU_Loss as the loss function of the bounding box:

Among them, A and B are the areas of the predicted bounding box and the ground truth, respectively, |A∩B| represents the area of the intersection of the two boxes, |A∪B| represents the area of the union of the two boxes, and IoU is the intersection of the areas of the two boxes and compare. D1 is the square of the Euclidean distance between the center points of the two boxes, D2 is the square of the length of the diagonal of the smallest rectangle that can just contain the two boxes,

is the weight coefficient, where v is a parameter that measures the consistency of the aspect ratio:

In the above formula, w ^gt is the width of the real annotation frame, h ^gt is the height of the real annotation frame; w is the width of the target detection frame, and h is the height of the target detection frame. On the basis of IoU, the CIoU_Loss function increases the distance between the center points of the two frames and the aspect ratio.

these two penalties. 7. The model construction method according to claim 6, wherein the method further comprises a post-processing step, and the post-processing step comprises: For the screening of multi-object boxes, a weighted non-maximum suppression algorithm is used to remove some redundant bounding boxes. Compared with the traditional non-maximum suppression, the weighted non-maximum suppression does not directly eliminate the prediction box whose IoU is greater than the threshold and the same category as the real annotation box in the process of executing the rectangular box culling, but according to The confidence of the network prediction is weighted to obtain a new rectangular frame, which is used as the final predicted rectangular frame, and then other redundant frames are eliminated. 8. A model building system for road surface flatness detection, wherein the system comprises: a data acquisition unit for acquiring multiple frames of original images of the target scene, extracting RGB images from the original images, and generating a depth map from the original images; a data set generation unit, configured to generate a data set from the RGB image and the depth map, and divide the data set into a training set, a verification set and a test set according to a preset ratio, and label each data set; The model output unit is used to input the RGB-D data into the training set and the verification set based on the pre-stored classification detection network model, and obtain the road surface flatness detection model through model training. 9. An intelligent terminal, characterized in that the intelligent terminal comprises: a data acquisition device, a processor and a memory; The data collection device is used to collect data; the memory is used to store one or more program instructions; the processor is used to execute one or more program instructions to execute any one of claims 1-7 the method described. 10. A computer-readable storage medium, wherein the computer storage medium contains one or more program instructions, and the one or more program instructions are used to execute any one of claims 1-7 Methods.

Images