CN119417830B

CN119417830B - A method for judging bolt looseness based on detecting the minimum bounding box

Info

Publication number: CN119417830B
Application number: CN202510021260.5A
Authority: CN
Inventors: 洪诚康; 汪华靖; 涂文豪; 李鑫; 陈晓彬; 杨轩; 万辰飞
Original assignee: Zhongshu Zhike Hangzhou Technology Co ltd
Current assignee: Zhongshu Zhike Hangzhou Technology Co ltd
Priority date: 2025-01-07
Filing date: 2025-01-07
Publication date: 2025-04-11
Anticipated expiration: 2045-01-07
Also published as: CN119417830A

Abstract

The present invention provides a method for judging bolt looseness based on detecting a minimum bounding box, comprising a bolt item point map acquisition step, detecting each bolt item point map on a train photographed by a camera according to a preset target detection model; a marking line bounding box marking step, marking each marking line in the bolt item point map with a minimum bounding box according to a preset minimum bounding box model; a marking line category analysis step, judging the number of minimum bounding boxes in the bolt item point map as the number of marking lines, and judging whether the category of each marking line is the same; a marking line selection and judgment step, performing type screening on the bolt item point map according to a marking line category judgment result and the number of marking lines to obtain a marking type, and selecting the minimum bounding box of two marking lines according to a marking type matching corresponding marking line judgment strategy to perform marking line offset judgment; the present invention has the advantage that bolts in different situations can be classified and processed more meticulously, thereby reducing false alarms of bolt looseness caused by viewing angle problems.

Description

Bolt loosening judging method based on minimum bounding box detection

Technical Field

The invention relates to the technical field of bolt loosening detection at the bottom of a train, in particular to a bolt loosening judging method based on a detection minimum bounding box.

Background

Along with the rapid development of railway transportation industry, the running speed and frequency of the train are continuously improved, the requirements on the running safety and the maintenance efficiency of the train are higher and higher, the bottom parts of the train bear complex mechanical load and environmental influence in the running process of the train, the problems of loosening, abrasion, falling off and the like are easy to occur, the running performance of the train can be influenced by the abnormality of the bottom bolt of the train, and serious safety accidents can be even caused, so that the state of the bottom bolt of the train is timely and accurately detected, and the method has important significance for guaranteeing the safe running of the train.

Current inspection includes traditional manual inspection and machine vision and deep learning based inspection of bottom pieces of trains.

The method mainly comprises the following steps of low efficiency, high cost, high manual detection cost, high omission rate, easy omission and false detection due to the fact that manual detection is limited by experience and attention of personnel, and safety risk that maintenance personnel need to enter the bottom of a train to operate in the manual detection process, and certain safety risk exists.

The detection of the train bottom part related to the deep learning mainly depends on the comparison of a template image output by a model and an actual image, or analysis is carried out through a point, but the number and the condition of the corresponding drawn marking lines are different when a plurality of different conditions exist in the current bolt part, and the analysis result is low in accuracy due to the existing model matching comparison or the point analysis.

Disclosure of Invention

Aiming at the defects existing in the prior art, the invention aims to provide a bolt loosening judging method based on a detection minimum bounding box.

In order to achieve the above purpose, the present invention provides the following technical solutions:

a bolt loosening judging method based on a detection minimum bounding box comprises the following steps:

A bolt item point diagram obtaining step, namely detecting each bolt item point diagram of the train shot by the camera according to a preset target detection model;

marking each marking line in the bolt item point diagram by using a minimum bounding box according to a preset minimum bounding box model;

a step of analyzing the category of the marking lines, which is to judge the number of the minimum bounding boxes in the bolt item point diagram as the number of the marking lines and judge whether the category of each marking line is the same or not;

and a marking line selection judging step, namely carrying out type screening on the bolt item dot diagram according to a marking line type judging result and the marking line number to obtain a marking type, and selecting the minimum bounding box of two marking lines according to the marking type matching corresponding marking line judging strategy to carry out marking line offset judgment.

Further, the marking line selection judging step comprises a type screening strategy, wherein the type screening strategy comprises the steps of counting the marking line numbers of different types as effective marking lines, and sequentially obtaining marking types according to the number sequence of the effective marking lines, wherein the marking types comprise three marking lines, two marking lines and one marking line.

Further, the marking line judging strategy comprises a line selecting judging sub-strategy, a direct judging sub-strategy and an neglecting judging sub-strategy, the marking type corresponding to the line selecting judging sub-strategy is three marking types, the line selecting judging sub-strategy comprises selecting two adjacent and effective marking lines to carry out offset judgment, the marking type corresponding to the direct judging sub-strategy is two marking types, the direct judging sub-strategy comprises carrying out offset judgment on only two effective marking lines, the neglecting judging sub-strategy corresponds to the marking type to be one marking type, and the neglecting judging sub-strategy comprises neglecting to carry out offset judgment on only one marking line.

Further, the minimum bounding box output by the minimum bounding box model is matched with a marking line category, and the marking line category comprises a bottom marking line of a bolt fixing position, a nut marking line and a screw marking line.

Further, the line selection judging sub-strategy comprises judging whether screw rod type mark lines exist in three effective mark lines, if so, selecting the screw rod type mark lines to carry out offset judgment with nut type mark lines, and if not, selecting the nut type mark lines to carry out offset judgment with bottom type mark lines at the bolt fixing positions.

Further, the method also comprises a step of calculating the joint coefficient of the marking line,

And the marking line connection coefficient calculating step is used for obtaining the smallest bounding box corresponding to the two marking lines selected and drawn in the initial image as a first frame group, obtaining the smallest bounding box corresponding to the two marking lines selected and drawn in the initial image in the actual detection image as a second frame group, respectively calculating the parallelism of the long sides at the same side and the distance between the adjacent short sides of the two smallest bounding boxes in the first frame group and the second frame group, calculating to obtain connection coefficients of the two marking lines in the actual detection image through a connection algorithm, and outputting whether the marking lines deviate or not according to the connection coefficients.

Further, the engagement algorithm is configured to:

,

Wherein, For the angle between the long sides of the same side of the two bounding boxes in the initial image,For the distance between the adjacent short sides of two bounding boxes in the initial image,To actually detect the angle between the long sides of the same sides of two bounding boxes in the image,To actually detect the distance between the adjacent short sides of two bounding boxes in an image,The range is between 0 and 1 for the engagement coefficient.

Further, the method also comprises a marking line engagement verification step,

And in the marking line connection verification step, the minimum bounding box in the first frame group and the second frame group is subjected to image processing to delete the area diagram outside the type in the minimum bounding box to obtain the corresponding marking line area, the area ratio of the two marking line areas in the first frame group is subjected to a first ratio, the area ratio of the two marking line areas in the second frame group is subjected to a second ratio, the difference between the first ratio and the second ratio is compared with a preset threshold value, and verification correctness or verification error is output according to a comparison result.

Further, the verification algorithm is configured to:

,

Wherein, In the first ratio of the values of the first and second values,Is a second ratio.

Further, the long side of the minimum bounding box is parallel to the marking line, and the minimum bounding box encloses the whole marking line.

The method has the advantages that the marking lines are marked in the minimum bounding box form in the bolt item dot diagram, the types and the number of the marking lines are analyzed, five types of marking lines are distinguished, bolts in different conditions can be classified more carefully, false alarms caused by loosening of the bolts due to the problem of visual angles in the past are reduced, and in addition, the marking line judgment is carried out by selecting the corresponding marking line judgment strategy.

Drawings

FIG. 1 is a general flow chart of the present invention;

FIG. 2 is a first case diagram of the present invention with three reticle classes labeled;

FIG. 3 is a second scenario diagram of the present invention, labeled types of three reticle classes;

FIG. 4 is a third scenario diagram of the present invention, labeled types of three reticle classes;

FIG. 5 is a fourth scenario featuring three classes of reticle labeled in the present invention;

FIG. 6 is a first situation diagram of the present invention with two reticle classes labeled;

FIG. 7 is a second scenario diagram of the present invention, with the label types being two reticle classes;

FIG. 8 is a third scenario featuring two classes of reticle labeled according to the present invention;

FIG. 9 is a fourth scenario featuring two classes of reticle labeled according to the present invention;

FIG. 10 is a diagram of the present invention with a marker type of one reticle class;

FIG. 11 is a diagram of a second decision logic according to the present invention.

Detailed Description

The invention will now be described in further detail with reference to the drawings and examples. Wherein like parts are designated by like reference numerals. It should be noted that the words "front", "back", "left", "right", "upper" and "lower" used in the following description refer to directions in the drawings, and the words "bottom" and "top", "inner" and "outer" refer to directions toward or away from, respectively, the geometric center of a particular component.

Because the existing detection of the bottom part of the train in deep learning mainly depends on the comparison of the template image output by the model and the actual image or analysis is carried out through the item points, but the existing bolt part has a plurality of different conditions, the number and the conditions of the corresponding drawn marking lines are different, and the analysis result is low in accuracy through the existing model matching comparison or item point analysis, the invention designs the bolt loosening judging method based on the detection minimum bounding box, which comprises the following steps:

And (3) in the step of manufacturing the bolt small image, firstly, bolt loosening marking lines are drawn on the bottom of the train and the bolts screwed on the train, the images of the bottom and the side of the train are collected under different illumination conditions by a collecting robot or the train running through a collecting area, the images are ensured to comprise bolts with various shapes and specifications, all the bolts on the subway are detected by a pre-trained YOLOV target detection method, and then the small image of the bolt part is intercepted for later manufacturing of a bolt marking line data set.

Marking each marking line on a bolt by using X-AnyLabeling software, marking each marking line on the bolt by adopting a mode of a minimum surrounding frame (OBB frame), so that each marked rectangular frame surrounds a whole bolt marking line under the condition of the minimum area, the shape of the marking line is better adapted to the rectangular frame, the OBB frame is not required to be parallel to any coordinate axis in the coordinates, the long side of the OBB frame is parallel to the bolt marking line, and the marking line is divided into 5 categories of marking lines on ① bolt nuts, marking lines on the side surfaces of ② bolt nuts, marking lines on the side surfaces of ③ screw rods, marking lines on the side surfaces of ④ nut nuts, and marking lines on the base of a ⑤ bolt fixing position;

The marking requirements are that two adjacent marking lines in the same bolt position, namely the same marking line, are marked as the same surrounding frame (rotating frame) when the two marking lines are interrupted, marking lines in different bolt positions are connected, marking lines in different positions are marked separately according to the types, and surrounding frames (rotating frames) need to be marked on all marking lines in a bolt image.

After marking, each bolt picture corresponds to a json file with the same name, the json file contains information of each bolt marking line, the json file is required to be converted into a YOLOv rotating frame txt format, the data format contains categories of the bolt marking lines, and four corner points of the rotating frame correspond to the upper right, the lower left and the upper left of the sequence.

The marked images and the corresponding txt files are respectively manufactured into a training set, a verification set and a test set, and are distributed according to the proportion of 7:1.5:1.5 so as to carry out model training and evaluation.

Training a data set of the minimum bounding box through a YOLOV-OBB neural network, and training a model to detect whether the minimum bounding box of each marking line is different from a traditional target detection mode in a bolt image, wherein a detection result of the method YOLOV-OBB is provided with a rotating detection box, each detection box is provided with coordinates of four corner points, and a rotation angle of the rotating box and the positive direction of an X axis of the image is used for judging whether the marking line is loosened or not.

As shown in fig. 1, the method further comprises a step of acquiring a bolt item point diagram, wherein each bolt item point diagram on the train shot by an actual camera is detected according to a preset target detection model;

Marking a marking line surrounding frame, namely marking each marking line in the bolt item dot diagram by using a minimum surrounding frame according to the constructed minimum surrounding frame model to obtain a surrounding frame (rotating frame) corresponding to each marking line, wherein the minimum surrounding frame output by the minimum surrounding frame model is matched with marking line categories, and the marking line categories comprise a bottom marking line, a nut marking line and a screw marking line at a bolt fixing position;

A marker line category analysis step of judging the number of the minimum bounding boxes in the bolt item point diagram as the number of marker lines, judging whether the category of each marker line is the same or not, and judging whether the category is the same or not and is different from the 5 category distinction in the data set manufacturing step of the minimum bounding boxes, wherein the category only considers the component category of the marker line, namely the marker line category comprises a bottom category marker line of a bolt fixing position, a nut category marker line and a screw category marker line;

A marking line selection judging step, namely analyzing a bolt item dot diagram according to a marking line type judging result and marking line numbers to obtain marking types through a type screening strategy, wherein the type screening strategy comprises counting marking line numbers of different types as effective marking lines, and sequentially obtaining marking types according to the effective marking line numbers, wherein the marking types comprise three marking lines, two marking lines and one marking line according to the number sequence, as shown in fig. 2, 3, 4 and 5, one bolt item dot plot comprises three marking lines, the marking types comprise three marking lines, as shown in fig. 6, 7, 8 and 9, one bolt item dot plot comprises two marking lines, but the types of the two marking lines in fig. 8 and 9 are similar, so the marking types of fig. 6 and 7 are two marking lines, and the marking types of fig. 8, 9 and 10 are one marking line;

Selecting a minimum bounding box of two mark lines according to a mark type matching corresponding mark line judging strategy to carry out mark line offset judgment, wherein the mark line judging strategy comprises a line selecting judging sub-strategy, a direct judging sub-strategy and an neglect judging sub-strategy, the mark type corresponding to the line selecting judging sub-strategy is three mark lines, the line selecting judging sub-strategy comprises selecting two adjacent and effective mark lines to carry out offset judgment, specifically judging whether a screw type mark line exists in the three effective mark lines, if the screw type mark line exists, selecting the screw type mark line to carry out offset judgment with a nut type mark line, if the screw type mark line does not exist, selecting the nut type mark line to carry out offset judgment with a bottom type mark line at a bolt fixing position, directly judging the mark type corresponding to the sub-strategy to be two mark lines, the direct judging sub-strategy comprises carrying out offset judgment on only two effective mark lines, the neglect judging sub-strategy corresponds to the mark type to be one mark line, and the neglect judging sub-strategy comprises carrying out offset judgment on only one mark line.

In addition, according to the 5 categories in the data set creation step of the minimum bounding box shown in fig. 2 to 10, the following cases are specifically classified:

When two marking lines are different in category and are not category ① and ②, whether the OBB frames of the two marking lines deviate or not is judged (because the relative positions of the marking lines of the category ①② and the category ②④ are fixed, the marking lines cannot deviate along with the loosening of the bolt, the marking lines cannot be used as a loosening judgment basis, and the category 0 ② is adopted under the condition of the same bolt point, class 0 ③, class ①②, The category ①③ is not simultaneously present, the situation that whether the mark line is shifted can not be judged in the above six situations can not be judged, when three mark lines and more mark lines exist, the three mark lines exist in the bolt at most, except when the mark lines are worn and the like, the three mark lines exist, only the mark line frame with the highest confidence degree of each category can be selected according to the result of target detection, and because the condition of the bolts at the vehicle bottom is unified, only the following situations exist, namely, when the categories 0, ① and ④ exist simultaneously, when the categories 0 and ① exist simultaneously, the relative positions of the categories 0 and ① are fixed, only the deviation needs to be considered, and when the categories ① and ④ exist simultaneously, and when the categories ② and ③ exist simultaneously, the categories ④ do not need to be considered, and because the relative positions of the categories ② and ④ are fixed, only the deviation needs to be considered. the method for judging whether the mark line is deviated comprises two modes, namely, a mark line joint coefficient calculation step, namely, when a minimum bounding box model of the train is constructed, an initial image of each bolt is acquired and used for recording and training, so that the minimum bounding box corresponding to two drawn mark lines selected in the initial image is acquired as a first box group, the minimum bounding box corresponding to the same two mark lines selected in the initial image in an actual detection image is acquired as a second box group, the parallelism (the angle between the same side long edges) of the same side long edges of the two minimum bounding boxes in the first box group and the second box group and the distance between adjacent short edges are calculated, the joint coefficient of the two mark lines in the actual detection image is calculated through a joint algorithm, and whether the mark line is deviated or not is output according to the joint coefficient, and the joint algorithm is configured as follows:

,

Wherein, For the angle between the long sides of the same side of the two bounding boxes in the initial image,For the distance between the adjacent short sides of two bounding boxes in the initial image,To actually detect the angle between the long sides of the same sides of two bounding boxes in the image,To actually detect the distance between the adjacent short sides of two bounding boxes in an image,For the engagement coefficient, the range is between 0 and 1,To calculate the cosine similarity between the initial angle and the actual angle,In order to normalize the initial short-side distance,To calculate an exponential decay value from the actual short side distance, whereinTaking out 0 of the mixture,Taking 0.1, when K is equal to 1, the bolt is not loosened and offset, when K is equal to 0, the bolt is completely loosened and offset, and when K is at a value of 0-1, the larger the value is, the smaller the offset is.

As shown in fig. 4 and 5, because the screw is thin, under the conditions of shooting angle, bolt loosening, etc., part of the marking line on the screw may not be displayed in the image, that is, the surrounding frame (rotating frame) does not completely surround the whole marking line, and if the width of the marking line is too large, the possibility of bolt shifting still occurs, therefore, the method further comprises a marking line engagement verification step, image processing is performed on the smallest surrounding frame in the first frame group and the second frame group, so as to delete the area map outside the type in the smallest surrounding frame to obtain the corresponding marking line area, wherein the image preprocessing comprises noise removal, a filter (such as a mean filter, a median filter, a gaussian filter, etc.) is applied to remove noise in the image, then image enhancement is performed, such as using techniques of contrast enhancement, histogram equalization, etc., then edge monitoring is performed, an edge detection algorithm (such as Canny edge detector, sobel operator, presitt operator, etc.) is applied to identify the image of the marking line in the image, finally, image segmentation is performed on the marking line in the image, the marking line in the image is detected by the image, the area is calculated to obtain the area ratio value of the marking line in the first frame group 35, the first frame group is calculated to obtain the area of the first frame 35, the area of the marking line is compared with the first frame 35, the area in the second frame group is calculated to the area 35, and the area of the first frame 35 is calculated to the area 35, and the area of the first frame group is compared with the area 35, compared with the area 35 in the area of the first frame group is calculated by comparing the area 35, and then calculating the areas of the categories ② and ③ in the second frame group to obtain a second ratio, comparing the difference between the first ratio and the second ratio with a preset threshold value, outputting verification correctness or verification error according to the comparison result, and configuring a verification algorithm as follows:

,

The second judgment logic comprises:

The length of each marker line in the image and the rotation angle of the marker line are calculated. Each detection result after being detected by YOLOV-OBB targets has coordinates of four corner points of the rotating frame and angles of the rotating frame rotating along the positive direction of the X axis of the image, the angles are divided into positive and negative, positive numbers indicate that the rotating frame rotates clockwise along the positive direction of the X axis under the condition of being parallel to the coordinate axes, negative numbers indicate that the rotating frame rotates anticlockwise, the OBB frames aim at detecting the smallest bounding frame of the marking line, therefore, the long side of the detecting frame is the length of the marking line, the width of the detecting frame is the width of the marking line, and the marking line extends along the direction of the marking when being drawn, therefore, the long side direction of the rotating frame defaults to the extending direction of the marking line, through the coordinates of the four corner points of the rotating frame, the long side and the short side of the frame can be calculated, and meanwhile, the rotating angle of the extending direction of the marking line in the image can be calculated, and the rotating angle of the rotating frame under the image coordinate system is the angle of the target detection output if the included angle between the long side of the rotating frame and the X axis is smaller than the included angle between the long side and the Y axis. If the included angle between the long side and the X axis of the rotating frame is larger than the included angle between the long side and the Y axis, and if the angle of the target detection output is larger than 0, the rotating angle of the rotating frame under the image coordinate system is the angle of the target detection output plus 90 degrees, otherwise, the rotating angle is reduced by 90 degrees.

Judging whether the two mark lines are offset and summarizing into two cases, wherein the two mark lines are in adjacent states, a certain included angle is formed between the mark lines after the two mark lines are offset, or the mark lines are separated and then are in parallel states, such as a category ①④, a category ②③ and a category ③④, and the two mark lines are not adjacent but are parallel in an image, and a certain included angle is formed between the mark lines after the mark lines are offset. Such as category 0 ④.

As shown in FIG. 11, in the first case, the two rotating frames extend along the long side, the extending length is 30% of the length of the rotating frames, it is determined whether the two rotating frames are intersected, if the marking lines are intersected at this time, the marking lines are not deviated, if the marking lines are not intersected after extension, a second confirmation is required, the obtained rotation angles of the marking lines are determined whether the marking lines are deviated by the angle difference value in a threshold screening manner, and in the second case, it is only required to rotate the two marking lines by the two rotation angles of the marking lines, and whether the marking lines are deviated by the threshold screening manner.

The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above examples, and all technical solutions belonging to the concept of the present invention belong to the protection scope of the present invention. It should be noted that modifications and adaptations to the present invention may occur to one skilled in the art without departing from the principles of the present invention and are intended to be within the scope of the present invention.

Claims

1. A bolt loosening judging method based on a detection minimum bounding box is characterized by comprising the following steps:

Marking a marking line surrounding frame, namely marking each marking line in the bolt item point diagram by a minimum surrounding frame according to a preset minimum surrounding frame model, wherein the minimum surrounding frame output by the minimum surrounding frame model is matched with a marking line category, and the marking line category comprises a bottom marking line, a nut marking line and a screw marking line at a bolt fixing position;

a marking line selection judging step, namely carrying out type screening on the bolt item dot diagram according to a marking line type judging result and the marking line number to obtain a marking type, wherein the marking type comprises three marking lines, two marking lines and one marking line, and selecting the minimum bounding box of the two marking lines according to the marking type matching corresponding marking line judging strategy to carry out marking line offset judgment;

The method also comprises a step of calculating the joint coefficient of the mark line,

The method comprises the steps of calculating the joint coefficients of the marking lines, namely, obtaining a minimum bounding box corresponding to two marking lines selected and drawn in an initial image as a first frame group, obtaining a minimum bounding box corresponding to the same two marking lines selected in the initial image in an actual detection image as a second frame group, respectively calculating the parallelism of long sides on the same side and the distance between adjacent short sides of the two minimum bounding boxes in the first frame group and the second frame group, calculating the joint coefficients of the two marking lines in the actual detection image through a joint algorithm, and outputting whether the marking lines deviate or not according to the joint coefficients;

The engagement algorithm is configured to:

,

2. The method for judging bolt loosening based on the minimum bounding box detection as claimed in claim 1, wherein the marking line selection judging step comprises a type screening strategy, wherein the type screening strategy comprises the steps of counting the marking line numbers of different types as effective marking lines, and sequentially obtaining marking types according to the effective marking line numbers and the number sequence.

3. The method for judging bolt looseness based on the minimum bounding box detection is characterized in that the marking line judging strategy comprises a line selecting judging sub-strategy, a direct judging sub-strategy and an neglecting judging sub-strategy, the marking type corresponding to the line selecting judging sub-strategy is three marking lines, the line selecting judging sub-strategy comprises the step of selecting two adjacent and effective marking lines to conduct offset judgment, the marking type corresponding to the direct judging sub-strategy is two marking lines, the direct judging sub-strategy comprises the step of conducting offset judgment on only two effective marking lines, the neglecting judging sub-strategy corresponds to the marking type to be one marking line, and the neglecting judging sub-strategy comprises the step of neglecting conducting offset judgment on only one marking line.

4. The method for judging bolt looseness based on the minimum bounding box detection of claim 3, wherein the line selection judging sub-strategy comprises judging whether a screw rod type mark line exists in three effective mark lines, if so, selecting the screw rod type mark line to conduct offset judgment with a nut type mark line, and if not, selecting the nut type mark line to conduct offset judgment with a bottom type mark line at a bolt fixing position.

5. The method for judging bolt looseness based on detecting a minimum bounding box of claim 1, further comprising a mark line engagement verification step,

6. The method for judging bolt loosening based on the minimum bounding box detection of claim 5, wherein the marking line engagement verification step is configured with:

,

7. The method for judging bolt looseness based on detecting a minimum bounding box according to claim 1, wherein the long side of the minimum bounding box is parallel to a marking line, and the minimum bounding box encloses the whole marking line.