CN113537037A - Pavement disease identification method, system, electronic device and storage medium - Google Patents
Pavement disease identification method, system, electronic device and storage medium Download PDFInfo
- Publication number
- CN113537037A CN113537037A CN202110786241.3A CN202110786241A CN113537037A CN 113537037 A CN113537037 A CN 113537037A CN 202110786241 A CN202110786241 A CN 202110786241A CN 113537037 A CN113537037 A CN 113537037A
- Authority
- CN
- China
- Prior art keywords
- disease
- pavement
- picture
- sample
- disease identification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
本发明实施例提供一种路面病害识别方法、系统、电子设备及存储介质，其中方法包括：确定待识别的路面病害图片；将所述待识别的路面病害图片输入至病害识别模型中，得到所述病害识别模型输出的路面病害识别结果；其中，所述病害识别模型是基于路面病害的样本图片及所述样本图片对应的病害类别标注训练得到的。同时公开了与上述方法构成同一发明构思的系统，包括图片确定单元和病害识别单元。本发明解决了现有路面裂缝的自动分割算法存在通用性、精度和效率不高的问题。
Embodiments of the present invention provide a pavement disease identification method, system, electronic device, and storage medium, wherein the method includes: determining a pavement disease picture to be identified; inputting the to-be-identified pavement disease picture into a disease identification model to obtain the The road surface disease identification result output by the disease identification model; wherein, the disease identification model is obtained by training based on the sample pictures of the road surface diseases and the labeling of the disease category corresponding to the sample pictures. At the same time, a system with the same inventive concept as the above method is disclosed, including a picture determination unit and a disease identification unit. The invention solves the problems of low versatility, precision and efficiency of the existing automatic segmentation algorithm of road surface cracks.
Description
Technical Field
The invention relates to the technical field of highway image recognition, in particular to a pavement disease recognition method, a system, electronic equipment and a storage medium.
Background
With the increasing total mileage of roads, the task of road maintenance management is increasingly aggravated, and how to quickly, efficiently, accurately and timely detect and evaluate the road surface damage condition is particularly important. At present, pavement detection mainly depends on a detection vehicle to carry out full-section data acquisition, automation is realized, but pavement damage detection and segmentation mainly depend on manual identification, and when more time and human resources are consumed, the efficiency is lower, and the subjective factor obviously influences the result. At present, a crack segmentation technology becomes a large research hotspot, and various recognition algorithms are provided.
Although the automatic segmentation algorithm of the pavement cracks is developed in recent years, the problem that the traditional image processing method is difficult to eliminate still exists: the road surface condition is easy to change, and the image is easy to be interfered by external conditions such as illumination, shadow and the like when being obtained. At this time, the automatic segmentation algorithm of the road surface crack is not high in universality, and contradictions exist between precision and speed. For example, a complete pavement crack image cannot be obtained by using a CNN recognition segmentation method, which is not favorable for implementation of a later path planning process. If CNN is used to predict the class of each pixel of the image, a manually designed feature extractor is still required for pre-processing, whereas CNN is used only as a classifier. Furthermore, its network architecture is strictly related to the input image size, which prevents generalization of the method. Using CNN to predict whether a single pixel belongs to a crack on local information presents the problem of ignoring the spatial relationship between pixels and overestimating the crack width.
Disclosure of Invention
The invention provides a pavement damage identification method, a pavement damage identification system, electronic equipment and a storage medium, which are used for solving the problems of low universality, low precision and low efficiency of the existing automatic segmentation algorithm of pavement cracks.
The invention provides a pavement disease identification method, which comprises the following steps:
determining a pavement disease picture to be identified;
inputting the pavement disease picture to be identified into a disease identification model to obtain a pavement disease identification result output by the disease identification model;
the disease identification model is obtained based on a sample picture of the road surface disease and a disease category marking training corresponding to the sample picture.
Further, the disease identification model comprises a picture preprocessing model, a candidate frame optimization model, a transmission connection block and a target detection model;
the step of inputting the pavement disease picture to be identified into a disease identification model to obtain a pavement disease identification result output by the disease identification model comprises the following steps:
inputting the pavement disease picture to be identified into the picture preprocessing model, and outputting a preprocessed pavement disease picture;
inputting the preprocessed pavement disease picture into the candidate frame optimization model, and outputting the optimized candidate frame in each characteristic layer and the corresponding characteristic graph of each characteristic layer;
inputting the feature map of each feature layer into the transmission connection block, and outputting the feature map of the deformable convolution of each feature layer;
and inputting the optimized candidate frame in each characteristic layer and the characteristic diagram of the deformable convolution of each characteristic layer into the target detection model, and outputting the pavement disease identification result.
Further, inputting the feature map of each feature layer into the transmission connection block, and outputting the feature map of deformable convolution of each feature layer, including:
generating the offset of each pixel in the pavement disease picture to be identified based on the feature map of each feature layer;
and performing convolution operation on the characteristic diagram of each characteristic layer and the offset of each pixel in the pavement disease picture to be identified to obtain a characteristic diagram of deformable convolution of each characteristic layer.
Further, the disease identification model is obtained based on a sample picture of the road surface disease and a disease category marking training corresponding to the sample picture, and comprises the following steps:
after the distribution and the size of a candidate frame are set based on a sample picture of the pavement diseases and the disease category labels corresponding to the sample picture, a positive and negative balance sample of the sample picture is obtained based on a self-adaptive selection matching strategy;
and carrying out neural network training on positive and negative balanced samples based on the sample image to obtain the disease identification model.
Further, after the sample picture based on the road surface disease and the disease category label corresponding to the sample picture are subjected to distribution and size setting of the candidate frame, a positive and negative balance sample of the sample picture is obtained based on a self-adaptive selection matching strategy, and the method comprises the following steps:
presetting the distribution setting of a priori candidate frames with a specific aspect ratio based on a sample picture of the pavement diseases and a characteristic picture labeled by the disease types corresponding to the sample picture;
detecting whether each prior candidate frame contains a disease or not and the center deviation of the detected disease relative to the prior candidate frame based on the distribution setting of the prior candidate frames with the specific aspect ratio, and adjusting the size according to the statistical clustering result of the aspect ratio;
and adaptively selecting a training sample according to the center offset and a preset threshold value to obtain a positive and negative balanced sample of the sample image.
Further, adaptively selecting a training sample according to the center offset and a preset threshold to obtain a positive and negative balanced sample of the sample image, including:
presetting a marking frame, and selecting k prior candidate frames from each layer of a sample picture of the road surface disease and a characteristic picture of the disease category marking corresponding to the sample picture based on the Euclidean distance between the center of the prior candidate frame and the center of the marking frame to obtain k X L prior candidate frames as candidate positive samples; wherein L is the number of layers of the characteristic diagram;
obtaining a confidence threshold value set and a mean value and a variance of the confidence threshold value set based on the candidate positive sample and the preset labeling frame, and taking the sum of the mean value and the variance as a preset threshold value;
and taking the candidate positive samples which are greater than the preset threshold value in the confidence coefficient threshold value set and correspond to the preset threshold value and the candidate positive samples in the labeling frame as training samples to obtain positive and negative balanced samples of the sample image.
Further, after the road surface disease picture to be identified is input into a disease identification model and a road surface disease identification result output by the disease identification model is obtained, the method further comprises the following steps:
and merging the diseases of the pavement disease identification results, finding the positions of the diseases, and finding out disease information including actual length, actual area and severity according to the positions of the diseases.
The invention also provides a pavement disease identification system, comprising:
the image determining unit is used for determining a pavement disease image to be identified;
the disease identification unit is used for inputting the pavement disease picture to be identified into a disease identification model to obtain a pavement disease identification result output by the disease identification model;
the disease identification model is obtained based on a sample picture of the road surface disease and a disease category marking training corresponding to the sample picture.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor executes the program to realize the steps of the pavement damage identification method.
The present invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the pavement damage identification method as recited in any one of the above.
According to the pavement disease identification method, the pavement disease identification system, the electronic equipment and the storage medium, the pavement disease image is input into the disease identification model, so that the pavement disease identification result output by the disease identification model is obtained, and the disease identification model is obtained based on the sample image of the pavement disease and the disease category marking training corresponding to the sample image. The invention realizes the improvement of the accuracy and the efficiency of the pavement disease identification.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a schematic flow chart of a pavement damage identification method provided by the invention;
FIG. 2 is a block diagram of a disease identification model provided by the present invention;
FIG. 3 is a schematic structural diagram of a transmission connection block provided by the present invention;
FIG. 4 is a schematic structural diagram of the deformable convolution DConv operator provided by the present invention;
FIG. 5 is a schematic diagram of a training process of a disease recognition model provided by the present invention;
FIG. 6 is an exemplary illustration of a lateral crack defect in a lane line provided by the present invention;
FIG. 7 is a histogram statistical representation of aspect ratios provided by the present invention;
FIG. 8 is a schematic structural diagram of a pavement damage recognition system provided by the present invention;
fig. 9 is a schematic structural diagram of an electronic device provided by the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The following describes a pavement damage identification method provided by the present invention with reference to fig. 1 to 9.
The embodiment of the invention provides a pavement disease identification method. Fig. 1 is a method for identifying a pavement fault according to an embodiment of the present invention, and as shown in fig. 1, the method includes:
the disease identification model is obtained based on a sample picture of the road surface disease and a disease category marking training corresponding to the sample picture.
It should be noted that the neural network model for training according to the embodiment of the present invention is a network architecture based on CNN, but is a supervised learning in the network learning process, so that data is needed to support the training of the network. While there are now some open source data sets such as coco, voc, etc., they do not contain data sets of the road surface. To solve this problem, a dataset based on road surface impairments is created. Data amplification is required because the samples of the road surface diseases are severely unbalanced. Data augmentation can improve network performance, and because the fracture geometry is not fixed, no inversion or affine transformation is required. Random chunking operations are less desirable and prevent the wrong region from being taken. The apparent characteristics of the pictures are different due to different pavement materials. If the disease is detected by adopting the model trained in the same way, the detection accuracy is reduced. If a plurality of models are trained, the recognition efficiency is reduced, and the types of the road materials are not fixed, so that all the roads cannot be included. The method for enhancing, reducing noise and sharpening the image in the picture preprocessing comprises the following steps: the ruts, impurities and road shadows in the asphalt pavement can weaken and shield the road diseases. These disadvantages can cause some chance of false detection. The small-range influence factors such as oil stains adopt a method of screening through the area, and ruts, shadows and the like adopt an improved SGRSR method and an increased gray level equalization algorithm to increase the contrast of diseases.
The method provided by the embodiment of the invention is based on the sample picture of the pavement diseases and the disease identification model obtained by marking and training the disease types corresponding to the sample picture, so that the precision and the efficiency of pavement disease identification can be effectively improved.
Based on any of the above embodiments, as shown in fig. 2, the disease identification model includes a picture preprocessing model 210, a candidate frame optimization model 220, a transmission connection block 230, and a target detection model 240;
the step of inputting the pavement disease picture to be identified into a disease identification model to obtain a pavement disease identification result output by the disease identification model comprises the following steps:
inputting the pavement disease picture to be identified into the picture preprocessing model 210, and outputting a preprocessed pavement disease picture;
it should be noted that, due to different pavement materials for paving the highway, different pavement paving times, different time for acquiring the pavement data, and the influence of factors such as illumination and shadow, the quality of the image may be greatly different. Therefore, in order to better identify cracks and make the identification system more robust, the picture needs to be preprocessed. The picture preprocessing is mainly divided into the following processing methods: enhancing an image, reducing noise of the image, sharpening the image, eliminating shadow and zooming the image.
1. Image enhancement
The image is enhanced to emphasize or sharpen edges, contours, contrast and the like, so that the image is convenient to analyze and process.
Histogram equalization is typically used to increase the local contrast of an image. This method works well for images where the background and foreground are too bright or too dark. In the process of image acquisition, due to the influence of factors such as illumination, vehicle speed and the like, the problem of improper exposure often occurs. Histogram equalization may better show details in road overexposure or underexposure. More of the data is that the image edge is darker and the middle is brighter, and histogram equalization can be accelerated according to a CLAHE interpolation method, so that the dark area of the image is adjusted to be suitable for brightness.
2. Noise reduction of images
The noise reduction processing of the image weakens the details of the image. But the better the noise reduction effect, the more blurred the image. Both should be considered in the process of image noise reduction.
The wavelet transformation can well improve the contrast of the image and complete the noise reduction of the image, so that the detail characteristics of the image are more obvious. By adopting a wavelet coefficient processing method combining soft and hard thresholds, the image can be fidelity to a certain degree while the image has higher signal-to-noise ratio. The set gain factor can effectively supplement weak information of the image or reduce the influence of overexposure of the image so as to facilitate the processing of the image.
3. Sharpening images
No matter which kind of noise reduction treatment can make the detail of picture disappear, and tiny crack information can be lost in the crack of bituminous paving in the noise reduction treatment process. In order to obtain better accuracy and make the crack details of the picture clearer, sharpening processing of the picture is used.
The directions of cracks of the asphalt pavement are different, and the Laplacian algorithm is a linear secondary differential operator. The method has the same rotation invariance as the gradient operator, thereby meeting the requirements of sharpening the image edge in different directions, and the obtained boundary can contain more detail information.
4. Shadow elimination
The improved SGRSR algorithm can better eliminate the road shadow area and can not lose the information of cracks. And eliminating the influence of the pavement cracks and noise points on the division of the shadow region by adopting an MMDilate algorithm and a gauSmooth algorithm. Meanwhile, a metSegment is adopted to solve a shadow area and a non-shadow area. And solving the average value of the pixel brightness according to the bright parameter to realize the self-adaptive threshold value. And the elimination of the road surface shadow is realized by using an igeoLevel model and an AllillumComp. Finally, the adjustment of the threshold value needs to be optimized, and the repair label of the asphalt pavement is lost due to the full-automatic adjustment of the threshold value.
5. Image scaling
The size of the original data picture is 2512x3140, and if the original picture is input to the neural network, the picture is down-sampled inside the network. Down-sampling can cause a significant loss of information from the image, particularly the crack information. The proportion of crack information in the whole image is very small, and once the sampling is reduced, the crack disappears. A crop scaling process is performed before entering the network. The size of 1024x1024 is adopted, clipping is firstly carried out, and then an INTER _ AREA interpolation method is adopted. INTER _ AREA may produce less moire at zoom out, but its effect is similar to INTER _ near effect when the image is zoomed in.
Inputting the preprocessed pavement damage picture into the candidate frame optimization model 220, and outputting the optimized candidate frame in each characteristic layer and the corresponding characteristic graph of each characteristic layer;
inputting the feature map of each feature layer into the transmission connection block 230, and outputting a feature map of deformable convolution of each feature layer;
and inputting the optimized candidate frame in each feature layer and the feature map of the deformable convolution of each feature layer into the target detection model 240, and outputting the pavement damage identification result.
Specifically, the neural network framework model mainly comprises two modules, namely a candidate frame optimization module ARM and a target detection module ODM: the ARM module is dedicated to the binary task and filters a large amount of simple negative samples for the subsequent ODM module; and meanwhile, primary frame correction is carried out, so that a better frame regression starting point is provided for a subsequent ODM module. The ODM module takes the ARM-optimized a priori candidate box anchor as input, focusing on the multi-classification task and further border correction. The ODM module is not time-consuming operation similar to RoIPooling of candidate regions, but is directly connected through a transmission connection block TCB to convert ARM characteristics and fuse high-level characteristics to obtain characteristics with rich receptive fields, sufficient details and abstract contents for further classification and regression.
Based on any one of the above embodiments, inputting the feature map of each feature layer into the transmission connection block, and outputting the feature map of the deformable convolution of each feature layer, including:
generating the offset of each pixel in the pavement disease picture to be identified based on the feature map of each feature layer;
and performing convolution operation on the characteristic diagram of each characteristic layer and the offset of each pixel in the pavement disease picture to be identified to obtain a characteristic diagram of deformable convolution of each characteristic layer.
Specifically, to establish a link between the ARM and the ODM, a TCB (transport connection block) is introduced, converting functions from different layers of the ARM into a form required by the ODM so that the ODM can share features from the ARM. The ODM consists of the output of a TCB followed by a prediction layer that generates the score of the target class and the shape offset relative to the coordinates of the refined anchors. The TCB transmits features in the anchor refinement module that on the one hand can predict the location, size and class label of the target in the ODM, while another function of the TCB inherits large-scale context by adding features of higher layers to the transmitted features to improve the accuracy of detection. As shown in fig. 3, in order to match the dimensions of the features of the higher layer to those of the transmitted features, the higher layer feature map is augmented using an inverse convolution operation and their corresponding elements are summed, after which a convolutional layer is added to ensure the perceptibility of the detected features.
A deformable convolution DConv operator is introduced into the TCB module, and the deformable convolution structure of the operator is divided into two parts, as shown in fig. 4, the upper part generates an offset based on an input feature map, and the lower part obtains an output feature map through deformable convolution based on the feature map and the offset. The original picture data (dimension B × H × W × C), denoted as U, is subjected to a common convolution operation to obtain a corresponding output result (B × H × W × 2C), denoted as V. V is the offset of each pixel in the original image data (2C because there are x and y directions). The pixel index value of the picture in U is added to V to obtain the offset position (coordinate value in the original picture U), and the position value needs to be limited within the picture size range, and here, a bilinear interpolation method is adopted. And obtaining a new picture M after all the pixels with the positions are obtained, and inputting the new picture M into the common convolution to obtain a final result.
Ordinary convolution samples the fixed position of the input feature map, the receptive fields of all the activation units are the same, but since different positions may correspond to different scales or deformed objects, adaptation to the scale or receptive field size is required for accurate localization. The advantage of using a deformable convolution is: the sampling position of the deformable convolution better conforms to the shape and the size of the road diseases, and especially has obvious enhancement on the extraction of the features under the condition that the shape and the size of the current road diseases are different.
The improvement point of the embodiment of the invention on the neural network structure is a TCB module, all 3-by-3 ordinary convolutions in the TCB module are replaced by deformable convolutions, and the aim is to establish strict upper and lower image semantic modeling capability.
Based on any of the above embodiments, as shown in fig. 5, the disease identification model is obtained based on a sample picture of a road surface disease and a disease category marking training corresponding to the sample picture, and includes the following steps:
and 520, training a neural network based on the positive and negative balance samples of the sample image to obtain the disease identification model.
Specifically, the training optimization of the built neural network framework model is divided into two parts: (1) optimization of anchor settings, and (2) matching strategies for anchors.
(1) Optimization of Anchor settings: the anchors are priori candidate frames with certain width-height ratio, the anchors are distributed on the output characteristic diagram, and the detection network can judge whether each anchor contains an object, and the detected object deviates from the center point of the anchor and has the length-width ratio. The anchors are set during the training process to guide the detection network to learn the object classes it contains and the relative displacement (for localization) of the center point of the detected object. The anchors, which are arranged according to the distribution of the sizes of the object frames of the data set, can only make the model more effective if they are closer to the size and aspect ratio of the object to be detected.
(2) and (3) matching strategy of anchor: for the rationalization distribution strategy of positive and negative samples of the anchor, in the training process, after a feature map is obtained by detecting a network, a positioning head and a classification head need to find the position of a training target from the feature map and judge the category, and at the moment, the network needs to be told which of predicted results bbox is right and which is wrong. Assuming that only one target exists on the currently trained input feature map, and 5000 prediction results are generated by the detection algorithm, for the network just started to be trained, it is difficult to make 5000 target frames predicted by the network all on the target, most of the 5000 prediction results which are possibly randomly distributed are negative samples, and the network can only obtain a small amount of distribution conditions of the target features to be detected, so that the imbalance of the positive and negative samples is caused, and the result is that the accuracy of the training model is not high enough.
Based on any of the above embodiments, after the sample picture based on the road surface disease and the disease category label corresponding to the sample picture are subjected to distribution and size setting of the candidate frame, and a positive-negative equilibrium sample of the sample picture is obtained based on a self-adaptive selection matching strategy, including:
presetting the distribution setting of a priori candidate frames with a specific aspect ratio based on a sample picture of the pavement diseases and a characteristic picture labeled by the disease types corresponding to the sample picture;
detecting whether each prior candidate frame contains a disease or not and the center deviation of the detected disease relative to the prior candidate frame based on the distribution setting of the prior candidate frames with the specific aspect ratio, and adjusting the size according to the statistical clustering result of the aspect ratio;
specifically, for example, a fault such as a transverse crack is present in the lane line, and an example of the fault is shown in fig. 6, and histogram statistics of the width-height and width-height ratio is performed according to the single type of labeled data, as shown in fig. 7. In fig. 7, it can be seen from the histogram statistics that the width-to-height ratio of the label of the current type of disease is approximately in the range of 0.5-0.6, and therefore, it is suggested that the setting of the anchor can be finely adjusted in the range of 0.5-0.6. For the size of the anchor, the width and height statistics of the upper graph can be looked up, and the clustering result is clustered by using a clustering algorithm (k-means), so that a reasonable size is set. The clustering results for breadth and height are given in table 1 below:
TABLE 1
And adaptively selecting a training sample according to the center offset and a preset threshold value to obtain a positive and negative balanced sample of the sample image.
That is, embodiments of the present invention determine a preselected box category and confidence value, and filter out preselected boxes that belong to the background, and filter out boxes with a lower confidence threshold, where the threshold is the value of the IOU. And (3) performing descending order arrangement on the remaining frames according to the confidence degree, reserving top-k prediction frames, filtering the prediction frames with larger overlapping degree through an NMS (non-maximum suppression) algorithm, wherein the last remaining prediction frame is the detection result, namely the positive sample of the candidate frame.
Based on any of the above embodiments, adaptively selecting a training sample according to the center offset and a preset threshold to obtain a positive-negative equilibrium sample of the sample image includes:
presetting a marking frame, and selecting k prior candidate frames from each layer of a sample picture of the road surface disease and a characteristic picture of the disease category marking corresponding to the sample picture based on the Euclidean distance between the center of the prior candidate frame and the center of the marking frame to obtain k X L prior candidate frames as candidate positive samples; wherein L is the number of layers of the characteristic diagram;
obtaining a confidence threshold value set and a mean value and a variance of the confidence threshold value set based on the candidate positive sample and the preset labeling frame, and taking the sum of the mean value and the variance as a preset threshold value;
and taking the candidate positive samples which are greater than the preset threshold value in the confidence coefficient threshold value set and correspond to the preset threshold value and the candidate positive samples in the labeling frame as training samples to obtain positive and negative balanced samples of the sample image.
Specifically, for the phenomenon that the road surface disease detection has the imbalance of positive and negative samples, after the Anchor is reasonably set, the positive and negative samples need to be reasonably divided for network learning training, an ATSS (adaptive selection training sample) positive and negative sample matching strategy is adopted, and the matching strategyThe process is as follows: firstly, for a label box g, k anchor boxes are selected from each layer in the output featuremap, if k anchor boxes are selected from each layer, if L layers are output, k x L positive samples of the anchor boxes are taken as candidates, and k are selected from each layer according to the Euclidean distance between the center of the anchor boxes and the center of the label box. Next, the IOU of the candidate positive sample and the label box g is calculated, and the set of the IOU is DgAnd (4) showing. Next, set D is calculatedgMean value m ofgSum variance vg. Next, the mean value mgAnd variance vgIs taken as a threshold value, and is denoted as tg. Next, if set DgIs greater than threshold tgAnd the positive samples are inside the labeling boxes, then the candidate boxes are the positive samples used in training, and the anchors which are not positive samples are negative samples.
It should be noted that the candidate frames are obtained according to the distance between the anchor center and the labeled frame center, because the quality of the candidate anchors closer to the center is higher, which is helpful for training. The sum of the mean and variance is used as a threshold choice to select high quality training samples. The training samples are limited to be centered in the labeling box, and the high-quality training samples are selected.
Based on any of the above embodiments, after the to-be-identified pavement disease picture is input into a disease identification model and a pavement disease identification result output by the disease identification model is obtained, the method further includes:
and merging the diseases of the pavement disease identification results, finding the positions of the diseases, and finding out disease information including actual length, actual area and severity according to the positions of the diseases.
Specifically, the network identification result is information of one overlapped small frame, and the results output by the two models are weighted, so that the identification result has multiple small frames which are not beneficial to the evaluation of the technical indexes of the diseases, and therefore the diseases are further combined. And the merging strategy merges the net cracks and the auxiliary net crack positioning according to the expert experience, and then merges the frames of the same kind of label. According to the current standard, the repaired cracks are combined when being identified as cracks, and the block repair and the strip repair are combined after being logically distinguished and judged. The combination is only to find the approximate position of a certain disease, and the accurate information of the disease is also found according to the position. Such as actual length, actual area, severity, etc.
After a merging strategy, selecting a corresponding position of the picture, only leaving the shape of the disease on the preprocessed picture, performing histogram projection on the processed picture, obtaining the accurate curve width (pixel level) of the crack in the x-axis projection, and obtaining the length of the crack curve in the y-axis projection.
And when the network identifies, dividing the long crack into a horizontal crack and a vertical crack, judging whether the cracks are communicated or not according to the coordinates of the two cracks, if so, combining the two cracks, and returning the curve length of the cracks.
In the returned label of the reticular fracture, the actual area needs to be calculated, the boundary point of the reticular fracture can be returned according to the histogram projection, two fitting curves are calculated according to polynomial fitting, and finally the area of the reticular fracture is calculated according to integral. And for the cracks, semantic segmentation is further carried out, disease segmentation is carried out at the pixel level, the histogram projection and the semantic segmentation result are subjected to weighted calculation, and the geometric dimension of the actual disease is finally calculated.
The method is adopted to identify and analyze the pavement diseases, and tests are carried out on 10000 pictures, 50000 pictures and 100000 pictures, and the results show that the detection accuracy reaches 90 percent to 95 percent, the missing report rate reaches 0.02 percent to 0.05 percent, and the false report rate reaches 0.08 percent to 0.12 percent. The efficiency is about 40 times faster than manual work, and the standard requirements are basically met. The invention is simple to realize, has effective detection and meets the application requirement.
The following describes a pavement damage identification system provided by the present invention, and the following description and the above-described pavement damage identification method can be referred to correspondingly.
Fig. 8 is a schematic structural diagram of a pavement damage identification system according to an embodiment of the present invention, and as shown in fig. 8, the system includes an image determination unit 810 and a damage identification unit 820;
the image determining unit is used for determining a pavement disease image to be identified;
the disease identification unit is used for inputting the pavement disease picture to be identified into a disease identification model to obtain a pavement disease identification result output by the disease identification model;
the disease identification model is obtained based on a sample picture of the road surface disease and a disease category marking training corresponding to the sample picture.
The system provided by the embodiment of the invention is based on the sample picture of the pavement diseases and the disease identification model obtained by marking and training the disease types corresponding to the sample picture, so that the precision and the efficiency of pavement disease identification can be effectively improved.
Based on any one of the above embodiments, the disease identification unit includes a picture preprocessing module, a candidate frame optimization module, a transmission connection block, and a target detection module;
the image preprocessing module is used for inputting the pavement disease image to be identified and outputting a preprocessed pavement disease image;
the candidate frame optimization module is used for inputting the preprocessed road surface disease picture and outputting the optimized candidate frame in each characteristic layer and the corresponding characteristic graph of each characteristic layer;
the transmission connection block is used for inputting the feature map of each feature layer and outputting the feature map of the deformable convolution of each feature layer;
and the target detection model is used for inputting the optimized candidate frame in each characteristic layer and the characteristic diagram of the deformable convolution of each characteristic layer and outputting the pavement disease identification result.
Based on any of the above embodiments, the transmission connection block includes an offset module and a convolution operation module;
the offset module is used for generating the offset of each pixel in the pavement disease picture to be identified based on the feature map of each feature layer;
and the convolution operation module is used for performing convolution operation on the characteristic diagram of each characteristic layer and the offset of each pixel in the pavement disease picture to be identified to obtain the characteristic diagram of the deformable convolution of each characteristic layer.
Based on any one of the above embodiments, the disease identification model is obtained by training based on a sample picture of a road surface disease and a disease category label corresponding to the sample picture, and includes the following steps:
after the distribution and the size of a candidate frame are set based on a sample picture of the pavement diseases and the disease category labels corresponding to the sample picture, a positive and negative balance sample of the sample picture is obtained based on a self-adaptive selection matching strategy;
and carrying out neural network training on positive and negative balanced samples based on the sample image to obtain the disease identification model.
Based on any of the above embodiments, after the sample picture based on the road surface disease and the disease category label corresponding to the sample picture are subjected to distribution and size setting of the candidate frame, and a positive-negative equilibrium sample of the sample picture is obtained based on a self-adaptive selection matching strategy, including:
presetting the distribution setting of a priori candidate frames with a specific aspect ratio based on a sample picture of the pavement diseases and a characteristic picture labeled by the disease types corresponding to the sample picture;
detecting whether each prior candidate frame contains a disease or not and the center deviation of the detected disease relative to the prior candidate frame based on the distribution setting of the prior candidate frames with the specific aspect ratio, and adjusting the size according to the statistical clustering result of the aspect ratio;
and adaptively selecting a training sample according to the center offset and a preset threshold value to obtain a positive and negative balanced sample of the sample image.
Based on any of the above embodiments, adaptively selecting a training sample according to the center offset and a preset threshold to obtain a positive-negative equilibrium sample of the sample image includes:
presetting a marking frame, and selecting k prior candidate frames from each layer of a sample picture of the road surface disease and a characteristic picture of the disease category marking corresponding to the sample picture based on the Euclidean distance between the center of the prior candidate frame and the center of the marking frame to obtain k X L prior candidate frames as candidate positive samples; wherein L is the number of layers of the characteristic diagram;
obtaining a confidence threshold value set and a mean value and a variance of the confidence threshold value set based on the candidate positive sample and the preset labeling frame, and taking the sum of the mean value and the variance as a preset threshold value;
and taking the candidate positive samples which are greater than the preset threshold value in the confidence coefficient threshold value set and correspond to the preset threshold value and the candidate positive samples in the labeling frame as training samples to obtain positive and negative balanced samples of the sample image.
Based on any of the above embodiments, the system further comprises a merging and analyzing module;
and the merging and analyzing module is used for merging the diseases of the pavement disease identification results, finding the positions of the diseases, and finding out the disease information including the actual length, the actual area and the severity according to the positions of the diseases.
In conclusion, due to the fact that the influence factors in the actual road surface are too many, some image preprocessing algorithms are added to reduce the influence of the noise points, and therefore the robustness of recognition is improved; the road surface data are identified through the convolutional neural network, so that the identification efficiency and the accuracy can be improved; the constructed convolutional neural network structure is modified to better adapt to pictures with different sizes, so that the robustness of the recognition system is improved; through the fusion of the dual-model results and an intelligent merging strategy, the bottleneck of neural network identification can be broken, and the missing report is reduced; and the geometric dimension of the diseases is accurately calculated, and a report can be directly returned.
Fig. 9 illustrates a physical structure diagram of an electronic device, and as shown in fig. 9, the electronic device may include: a processor (processor)910, a communication interface (communication interface)920, a memory (memory)930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other via the communication bus 940. Processor 910 may invoke logic instructions in memory 930 to perform a pavement virus identification method comprising: determining a pavement disease picture to be identified; inputting the pavement disease picture to be identified into a disease identification model to obtain a pavement disease identification result output by the disease identification model; the disease identification model is obtained based on a sample picture of the road surface disease and a disease category marking training corresponding to the sample picture.
Furthermore, the logic instructions in the memory 930 may be implemented in software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, when the program instructions are executed by a computer, the computer being capable of executing the road surface virus identification method provided by the above methods, the method including: determining a pavement disease picture to be identified; inputting the pavement disease picture to be identified into a disease identification model to obtain a pavement disease identification result output by the disease identification model; the disease identification model is obtained based on a sample picture of the road surface disease and a disease category marking training corresponding to the sample picture.
In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which when executed by a processor is implemented to perform the above-mentioned road surface virus identification method, the method including: determining a pavement disease picture to be identified; inputting the pavement disease picture to be identified into a disease identification model to obtain a pavement disease identification result output by the disease identification model; the disease identification model is obtained based on a sample picture of the road surface disease and a disease category marking training corresponding to the sample picture.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A pavement disease identification method is characterized by comprising the following steps:
determining a pavement disease picture to be identified;
inputting the pavement disease picture to be identified into a disease identification model to obtain a pavement disease identification result output by the disease identification model;
the disease identification model is obtained based on a sample picture of the road surface disease and a disease category marking training corresponding to the sample picture.
2. The pavement disease identification method according to claim 1, wherein the disease identification model includes a picture preprocessing model, a candidate frame optimization model, a transmission connection block, and a target detection model;
the step of inputting the pavement disease picture to be identified into a disease identification model to obtain a pavement disease identification result output by the disease identification model comprises the following steps:
inputting the pavement disease picture to be identified into the picture preprocessing model, and outputting a preprocessed pavement disease picture;
inputting the preprocessed pavement disease picture into the candidate frame optimization model, and outputting the optimized candidate frame in each characteristic layer and the corresponding characteristic graph of each characteristic layer;
inputting the feature map of each feature layer into the transmission connection block, and outputting the feature map of the deformable convolution of each feature layer;
and inputting the optimized candidate frame in each characteristic layer and the characteristic diagram of the deformable convolution of each characteristic layer into the target detection model, and outputting the pavement disease identification result.
3. The pavement disease identification method according to claim 2, wherein inputting the feature map of each feature layer into the transmission connection block and outputting the feature map of the deformable convolution of each feature layer includes:
generating the offset of each pixel in the pavement disease picture to be identified based on the feature map of each feature layer;
and performing convolution operation on the characteristic diagram of each characteristic layer and the offset of each pixel in the pavement disease picture to be identified to obtain a characteristic diagram of deformable convolution of each characteristic layer.
4. The method for identifying the pavement diseases according to claim 1, wherein the disease identification model is obtained based on a sample picture of the pavement diseases and a disease category marking training corresponding to the sample picture, and comprises the following steps:
after the distribution and the size of a candidate frame are set based on a sample picture of the pavement diseases and the disease category labels corresponding to the sample picture, a positive and negative balance sample of the sample picture is obtained based on a self-adaptive selection matching strategy;
and carrying out neural network training on positive and negative balanced samples based on the sample image to obtain the disease identification model.
5. The method for identifying the road surface diseases according to claim 4, wherein the step of obtaining the positive and negative balance samples of the sample image based on the adaptive selection matching strategy after setting the distribution and the size of the candidate frame based on the sample image of the road surface diseases and the disease category labels corresponding to the sample image comprises the following steps:
presetting the distribution setting of a priori candidate frames with a specific aspect ratio based on a sample picture of the pavement diseases and a characteristic picture labeled by the disease types corresponding to the sample picture;
detecting whether each prior candidate frame contains a disease or not and the center deviation of the detected disease relative to the prior candidate frame based on the distribution setting of the prior candidate frames with the specific aspect ratio, and adjusting the size according to the statistical clustering result of the aspect ratio;
and adaptively selecting a training sample according to the center offset and a preset threshold value to obtain a positive and negative balanced sample of the sample image.
6. The method for identifying the pavement diseases according to claim 5, wherein adaptively selecting a training sample according to the center offset and a preset threshold value to obtain a positive and negative balance sample of the sample image comprises:
presetting a marking frame, and selecting k prior candidate frames from each layer of a sample picture of the road surface disease and a characteristic picture of the disease category marking corresponding to the sample picture based on the Euclidean distance between the center of the prior candidate frame and the center of the marking frame to obtain k X L prior candidate frames as candidate positive samples; wherein L is the number of layers of the characteristic diagram;
obtaining a confidence threshold value set and a mean value and a variance of the confidence threshold value set based on the candidate positive sample and the preset labeling frame, and taking the sum of the mean value and the variance as a preset threshold value;
and taking the candidate positive samples which are greater than the preset threshold value in the confidence coefficient threshold value set and correspond to the preset threshold value and the candidate positive samples in the labeling frame as training samples to obtain positive and negative balanced samples of the sample image.
7. The method for identifying the pavement diseases according to claim 1, wherein after the pavement disease picture to be identified is input into a disease identification model and a pavement disease identification result output by the disease identification model is obtained, the method further comprises:
and merging the diseases of the pavement disease identification results, finding the positions of the diseases, and finding out disease information including actual length, actual area and severity according to the positions of the diseases.
8. A pavement damage recognition system, comprising:
the image determining unit is used for determining a pavement disease image to be identified;
the disease identification unit is used for inputting the pavement disease picture to be identified into a disease identification model to obtain a pavement disease identification result output by the disease identification model;
the disease identification model is obtained based on a sample picture of the road surface disease and a disease category marking training corresponding to the sample picture.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the pavement damage identification method according to any one of claims 1 to 7 when executing the program.
10. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the steps of the pavement disease identification method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110786241.3A CN113537037A (en) | 2021-07-12 | 2021-07-12 | Pavement disease identification method, system, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110786241.3A CN113537037A (en) | 2021-07-12 | 2021-07-12 | Pavement disease identification method, system, electronic device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113537037A true CN113537037A (en) | 2021-10-22 |
Family
ID=78098689
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110786241.3A Pending CN113537037A (en) | 2021-07-12 | 2021-07-12 | Pavement disease identification method, system, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113537037A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114863122A (en) * | 2022-05-05 | 2022-08-05 | 湖南大学 | Intelligent high-precision pavement disease identification method based on artificial intelligence |
| CN115345881A (en) * | 2022-10-18 | 2022-11-15 | 上海交强国通智能科技有限公司 | Pavement disease detection method based on computer vision |
| CN117671495A (en) * | 2023-12-01 | 2024-03-08 | 中路高科交通检测检验认证有限公司 | Real-time pavement disease automatic detection method and system based on edge calculation technology |
| CN117710756A (en) * | 2024-02-04 | 2024-03-15 | 成都数之联科技股份有限公司 | Target detection and model training method, device, equipment and medium |
| CN118053038A (en) * | 2024-03-13 | 2024-05-17 | 兰州交通大学 | Automatic pavement disease image identification method of C/S and B/S fusion architecture |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109685124A (en) * | 2018-12-14 | 2019-04-26 | 斑马网络技术有限公司 | Road disease recognition methods neural network based and device |
| WO2019218835A1 (en) * | 2018-05-16 | 2019-11-21 | 腾讯科技（深圳）有限公司 | Image recognition method, storage medium and computer device |
| CN112215819A (en) * | 2020-10-13 | 2021-01-12 | 中国民航大学 | Airport pavement crack detection method based on deep feature fusion |
| CN112906823A (en) * | 2021-03-29 | 2021-06-04 | 苏州科达科技股份有限公司 | Target object recognition model training method, recognition method and recognition device |
| CN113096126A (en) * | 2021-06-03 | 2021-07-09 | 四川九通智路科技有限公司 | Road disease detection system and method based on image recognition deep learning |
-
2021
- 2021-07-12 CN CN202110786241.3A patent/CN113537037A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019218835A1 (en) * | 2018-05-16 | 2019-11-21 | 腾讯科技（深圳）有限公司 | Image recognition method, storage medium and computer device |
| CN109685124A (en) * | 2018-12-14 | 2019-04-26 | 斑马网络技术有限公司 | Road disease recognition methods neural network based and device |
| CN112215819A (en) * | 2020-10-13 | 2021-01-12 | 中国民航大学 | Airport pavement crack detection method based on deep feature fusion |
| CN112906823A (en) * | 2021-03-29 | 2021-06-04 | 苏州科达科技股份有限公司 | Target object recognition model training method, recognition method and recognition device |
| CN113096126A (en) * | 2021-06-03 | 2021-07-09 | 四川九通智路科技有限公司 | Road disease detection system and method based on image recognition deep learning |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114863122A (en) * | 2022-05-05 | 2022-08-05 | 湖南大学 | Intelligent high-precision pavement disease identification method based on artificial intelligence |
| CN115345881A (en) * | 2022-10-18 | 2022-11-15 | 上海交强国通智能科技有限公司 | Pavement disease detection method based on computer vision |
| CN117671495A (en) * | 2023-12-01 | 2024-03-08 | 中路高科交通检测检验认证有限公司 | Real-time pavement disease automatic detection method and system based on edge calculation technology |
| CN117710756A (en) * | 2024-02-04 | 2024-03-15 | 成都数之联科技股份有限公司 | Target detection and model training method, device, equipment and medium |
| CN117710756B (en) * | 2024-02-04 | 2024-04-26 | 成都数之联科技股份有限公司 | Target detection and model training method, device, equipment and medium |
| CN118053038A (en) * | 2024-03-13 | 2024-05-17 | 兰州交通大学 | Automatic pavement disease image identification method of C/S and B/S fusion architecture |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113537037A (en) | Pavement disease identification method, system, electronic device and storage medium | |
| CN106651872B (en) | Pavement crack identification method and system based on Prewitt operator | |
| CN111310558A (en) | An intelligent extraction method of pavement diseases based on deep learning and image processing | |
| CN107481238A (en) | Image quality measure method and device | |
| CN110705583A (en) | Cell detection model training method and device, computer equipment and storage medium | |
| CN113435407B (en) | Small target identification method and device for power transmission system | |
| CN111695373B (en) | Zebra stripes positioning method, system, medium and equipment | |
| CN111259878A (en) | Method and equipment for detecting text | |
| CN113505702A (en) | Pavement disease identification method and system based on double neural network optimization | |
| CN108764328A (en) | The recognition methods of Terahertz image dangerous material, device, equipment and readable storage medium storing program for executing | |
| CN110599453A (en) | Panel defect detection method and device based on image fusion and equipment terminal | |
| CN113221881B (en) | A multi-level smartphone screen defect detection method | |
| CN118275449B (en) | Copper strip surface defect detection method, device and equipment | |
| CN114926407A (en) | Steel surface defect detection system based on deep learning | |
| CN112488220B (en) | A small target detection method based on deep learning | |
| CN117094975A (en) | Method and device for detecting surface defects of steel and electronic equipment | |
| CN114387591A (en) | License plate recognition method, system, device and storage medium | |
| CN115797314A (en) | Part surface defect detection method, system, equipment and storage medium | |
| CN118392891A (en) | Chip packaging defect detection method and system based on machine vision | |
| CN112863194A (en) | Image processing method, device, terminal and medium | |
| CN115661765A (en) | Pavement disease detection method | |
| CN117197064A (en) | Automatic non-contact eye red degree analysis method | |
| CN116993970A (en) | Oil and gas pipeline excavator pressure detection method and system based on yolov5 | |
| CN108537815A (en) | A kind of video image foreground segmentation method and device | |
| CN114723728A (en) | Method and system for detecting CD line defects of silk screen of glass cover plate of mobile phone camera |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20211022 |
|
| RJ01 | Rejection of invention patent application after publication |