CN118691917B - Bridge structure damage identification method and system based on machine vision - Google Patents

Abstract

The present application provides a method and system for bridge structure damage identification based on machine vision, which obtains image blocks in machine vision images that need to complete bridge structure damage identification, obtains t image implicit representations for representing different types of features of the image blocks and the image block connectivity matrix corresponding to the machine vision image; and obtains a merged implicit representation corresponding to the image blocks based on the t image implicit representations and the image block connectivity matrix corresponding to the image blocks. The merged implicit representation is used to perform bridge structure damage identification on the image blocks, and then perform bridge structure damage identification on the machine vision image to obtain an identification result. The present application performs bridge structure damage identification on machine vision images, rather than comparing the machine vision images with existing damage features. When the existing features cannot cover unknown damage conditions, the above implementation process can also be used to perform bridge structure damage identification on machine vision images, thereby increasing the reliability of bridge structure damage identification.

Description

Bridge structure damage identification method and system based on machine vision

Technical Field

The present application relates to the field of machine vision technology, and in particular to a method and system for identifying bridge structure damage based on machine vision.

Background Art

In the field of bridge structure health monitoring, machine vision technology has been widely used in recent years as a non-contact and efficient detection method. The existing machine vision-based bridge structure damage identification technology still has some challenges. On the one hand, the types and forms of bridge structure damage are diverse, including cracks, corrosion, deformation, etc., and the manifestations of these damages in machine vision images are also different. On the other hand, due to the complexity and diversity of bridge structures, there are also many background noises and interference factors in machine vision images, which brings great difficulties to damage identification. Traditional machine vision damage identification methods usually achieve damage identification and classification by comparing the image to be identified with the existing damage feature library. However, this method has obvious limitations. When the damage features in the image to be identified do not match the features in the existing feature library, this method often cannot accurately identify the damage, resulting in missed detection or false detection. Especially when facing unknown or novel damage types, the reliability of this method is greatly reduced. Therefore, a new machine vision-based bridge structure damage identification technology is needed, which can overcome the limitations of traditional methods and improve the accuracy and reliability of damage identification.

Summary of the invention

In view of this, the embodiment of the present application provides a bridge structure damage identification method and system based on machine vision. The technical solution of the present application is implemented as follows:

In a first aspect, an embodiment of the present application provides a method for identifying damage to a bridge structure based on machine vision, comprising: acquiring a machine vision image of a bridge structure to be identified; performing image implicit representation extraction on image blocks in the machine vision image to obtain t image implicit representations corresponding to the image blocks, wherein t∈(0,+∞); the t image implicit representations are used to represent different types of features, and the image blocks are obtained by dividing the machine vision image into blocks; merging the t image implicit representations to obtain a merged implicit representation corresponding to the image blocks, and performing adaptive feature integration on the merged implicit representation to obtain a feature for highlighting the representation. The adaptive implicit representation of the merged implicit representation is characterized; the image block connectivity matrix corresponding to the machine vision image and the output implicit representation corresponding to the image block are merged step by step to obtain the merged implicit representation corresponding to the image block; the output implicit representation is obtained through the adaptive implicit representation and the merged implicit representation, and the image block connectivity matrix is used to characterize the similarity between the image blocks; bridge structure damage is identified for the image blocks according to the merged implicit representation to obtain image block damage identification results, and the damage identification result of the bridge structure to be identified is obtained according to the image block damage identification results of each image block.

In a second aspect, the present application provides a computer system, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the steps in the above-described method when executing the program.

The beneficial effects of the present application include at least: the present application obtains image blocks in machine vision images that need to complete bridge structure damage recognition, obtains t image implicit representations for representing different types of features of the image blocks and the image block connectivity matrix corresponding to the machine vision image; then, based on the t image implicit representations corresponding to the image blocks and the image block connectivity matrix, obtains the merged implicit representation corresponding to the image blocks. The merged implicit representation is used to perform bridge structure damage recognition on the image blocks, and then perform bridge structure damage recognition on the machine vision image to obtain a recognition result. The present application performs bridge structure damage recognition on machine vision images, rather than comparing the machine vision images with existing damage features. When the existing features cannot cover unknown damage conditions, the present application can also use the above-mentioned implementation process to perform bridge structure damage recognition on machine vision images, thereby increasing the reliability of bridge structure damage recognition.

It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

FIG1 is a schematic diagram of an implementation flow of a method for identifying bridge structure damage based on machine vision provided in an embodiment of the present application.

FIG. 2 is a schematic diagram of a hardware entity of a computer system provided in an embodiment of the present application.

DETAILED DESCRIPTION

In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

The embodiment of the present application provides a bridge structure damage identification method based on machine vision, which can be executed by a processor of a computer system. The computer system can refer to a server, a laptop, a tablet computer, a desktop computer, a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, a portable gaming device) and other devices with data processing capabilities.

FIG1 is a schematic diagram of an implementation flow of a bridge structure damage identification method based on machine vision provided in an embodiment of the present application. As shown in FIG1 , the method includes:

Step S100: Acquire a machine vision image of the bridge structure to be identified.

In step S100, the computer system controls and coordinates the machine vision equipment (such as high-definition cameras, infrared thermal imagers, etc.) installed on the bridge. These devices are usually installed at key locations of the bridge, such as piers, main beam connections, supports, etc., to capture detailed images of the bridge structure. The computer system sets the acquisition parameters of the machine vision equipment according to the preset configuration file or real-time instructions, including resolution, frame rate, exposure time, etc. For example, in order to capture subtle cracks or deformations, the system can set a higher resolution and an appropriate exposure time. According to actual needs, image acquisition can be set to be executed on a scheduled basis (such as automatic acquisition during low-flow periods in the early morning every day) or triggered by specific events (such as instant acquisition when abnormal vibration is detected). The computer system controls the acquisition timing through an internal clock or an external sensor signal. The collected machine vision image data will be transmitted to the central processing server in real time. In this process, the computer system is responsible for ensuring the integrity and security of the data and using encrypted communication protocols to prevent data leakage. At the same time, the system will store the image data in a specific format (such as JPEG, PNG) in the database of the server for subsequent processing and analysis.

Step S200: extracting implicit image representations of image blocks in the machine vision image to obtain t implicit image representations corresponding to the image blocks, where t∈(0,+∞); the t implicit image representations are used to represent different types of features, and the image blocks are obtained by dividing the machine vision image into blocks.

In step S200, the computer system is responsible for in-depth analysis of each image block in the machine vision image, and extracts multiple implicit image representations through specific algorithms or models. These implicit representations respectively describe the characteristics of the image blocks in different dimensions or aspects, providing a rich information basis for subsequent damage identification. The following is a detailed explanation of this step, combined with specific application scenarios.

The computer system first divides the original image into blocks according to a preset block strategy (such as sliding overlapping windows of fixed size). Assume that the image is divided into multiple image blocks of 128×128 pixels, and these blocks partially overlap to ensure the continuity of boundary information. For each image block, the computer system uses texture analysis algorithms (such as gray level co-occurrence matrix, local binary pattern LBP, etc.) to extract its texture features. These features are represented in the form of vectors, for example, a vector of length 64, each element of which corresponds to a quantized value of a texture attribute. In order to capture the shape information of the image block, the system can use edge detection algorithms (such as Canny edge detector) combined with shape descriptors (such as Hu moments) to generate shape feature vectors. These feature vectors may contain geometric properties of edges, moment invariants, etc., which are used to describe the overall shape and outline of the block. For color images, the system also extracts color features of each block. This can be achieved through methods such as color histograms, color moments, or color aggregation vectors. For example, using the histogram in the HSV color space, the color space is divided into multiple intervals, and the number of pixels in each interval is counted to form a color feature vector. In some advanced applications, the system can also try to extract semantic information from image patches. This involves, for example, a pre-trained model of a deep learning model, such as a convolutional neural network (CNN), for identifying objects or scenes in an image. The output of an intermediate layer of the CNN is used as a feature map, which can be further processed to obtain a semantic feature vector, which implicitly represents the high-level semantic content of the image patch. For each image patch, the computer system merges the different types of implicit representations mentioned above (texture, shape, color, semantics, etc.). This usually means concatenating these feature vectors into a longer vector as a comprehensive representation of the patch. For example, if the texture feature vector is 64 in length, the shape feature vector is 10 in length, and the color feature vector is 256 in length, then the length of the merged implicit representation vector will be 64+10+256=330.

Step S300: merging t implicit representations of the image to obtain a merged implicit representation corresponding to the image block, and performing adaptive feature integration on the merged implicit representation to obtain an adaptive implicit representation for highlighting the merged implicit representation.

In step S300, the computer system further processes the multiple implicit image representations extracted in step S200 (each representing a specific feature dimension of the image block, a total of t types), aiming to integrate these multi-dimensional features and strengthen key information through adaptive feature integration technology, thereby generating a more comprehensive and representative adaptive implicit representation. This step is crucial for subsequent bridge structure damage identification because it ensures that all relevant features are effectively integrated and utilized. First, the computer system merges the t implicit image representations of each image block. These implicit representations may include texture feature vectors, shape feature vectors, color feature vectors, etc., each of which captures different aspects of the image block. The merging operation usually means simply splicing these vectors together to form a longer vector, that is, merging the implicit representations. For example, suppose the texture feature vector of an image block is f _texture = [0.2, 0.3, …, 0.1] (length n), the shape feature vector is f _shape = [0.5, −0.2, …, 0.4] (length m), and the color feature vector is f _color = [20, 50, …, 40] (length p). Then, the merged implicit representation will be the concatenation of these three vectors: f _merged = [f _texture , f _shape , f _color ] = [0.2, 0.3, …, 0.1, 0.5, −0.2, …, 0.4, 20, 50, …, 40].

Next, the computer system performs adaptive feature integration on the merged implicit representation to highlight key features and reduce redundant information. This step usually relies on complex machine learning models or algorithms, such as the Self-Attention mechanism or the Multi-Head Attention part in the Transformer model. The system then calculates the similarity score between the requested feature and all index features (usually using the dot product and applying a scaling factor d _k , where d _k is the dimension of the index feature) to determine the contribution of each output feature to the current query. This step is actually performing a weighted summation operation, with the weights obtained by normalizing the similarity scores through the softmax function. The calculation formula is as follows:

;

Finally, the system obtains a new vector, namely the adaptive implicit representation, which focuses more on the features that are important for the task of bridge structure damage identification. This vector will be used as input for further damage identification analysis in subsequent steps.

The adaptive implicit representation not only integrates the multi-dimensional features of image blocks, but also strengthens key information through advanced techniques such as self-attention mechanism, so that the subsequent damage identification model can more accurately capture subtle damage signs of the bridge structure.

Step S400: The image block connectivity matrix corresponding to the machine vision image and the output implicit representation corresponding to the image block are merged step by step to obtain a merged implicit representation corresponding to the image block; the output implicit representation is obtained through the adaptive implicit representation and the merged implicit representation, and the image block connectivity matrix is used to characterize the similarity between the image blocks.

Step S400 aims to utilize the connectivity information between image blocks to further enhance the understanding of the damage characteristics of the bridge structure, thereby providing more comprehensive and accurate information for subsequent damage identification.

First, the computer system constructs an image patch connectivity matrix, which is a two-dimensional array with the number of rows and columns equal to the number of image patches. Each element in the matrix represents the similarity or connectivity between the corresponding two image patches. This similarity can be calculated in many ways, such as based on the spatial distance between image patches, the similarity of color histograms, the matching degree of texture features, etc. For example, adjacent image patches have higher values for the corresponding elements in the connectivity matrix, while non-adjacent patches have lower values or zero. The computer system uses the image patch connectivity matrix and the output implicit representation of each image patch to merge level by level. This process involves, for example, a graph neural network (GNN) or similar graph processing models, because these models can naturally process graph structured data, where nodes (image patches in this case) are connected to each other through edges (connectivity). Specifically, the system can first use a graph convolutional network (GCN) as part of the merging submodule. GCN updates the feature representation of each node (image patch) by aggregating the feature information of its neighboring nodes. This process can be iterated multiple times, and each iteration will update the node features according to the current connectivity matrix and node features, thereby realizing the step-by-step merging of features. In the embodiment of the present application, by iterating such graph convolution operations multiple times, the system can capture the complex spatial relationships between image blocks, and incorporate these relationships into the feature representation of each block, and finally obtain a merged implicit representation. These merged implicit representations not only contain the feature information of the block itself, but also incorporate the influence of the surrounding blocks, thereby more comprehensively reflecting the overall state of the bridge structure.

Step S500: performing bridge structure damage identification on the image blocks according to the combined implicit representation to obtain image block damage identification results, and obtaining damage identification results of the bridge structure to be identified according to the image block damage identification results of each image block.

In step S500, the computer system uses the merged implicit representation obtained from the previous steps to perform detailed damage identification analysis on the image blocks of the bridge structure. This process is the core link of the entire machine vision bridge structure damage identification process, which directly determines the accuracy and reliability of the identification results. First, the computer system inputs the merged implicit representation of each image block into a pre-trained damage identification model. This model is usually a complex machine learning model, such as a deep neural network (DNN), a convolutional neural network (CNN), or a support vector machine (SVM). The specific choice depends on the characteristics of the data and the requirements of the recognition task. In the embodiment of the present application, since the image data has a high degree of spatial correlation, CNN is a common choice.

The task of the damage recognition model is to classify or regress the merged implicit representation of each image block to determine whether the block is damaged, and may further subdivide the type and degree of damage. For classification tasks, the model outputs the probability that each block belongs to different damage categories; for regression tasks, it may directly output a quantitative indicator of the degree of damage. For example, if CNN is used for classification, the last layer of the model can be a softmax layer, which outputs the probability distribution of each block belonging to categories such as "no damage", "slight cracks", and "severe cracks". The regression model may directly output a continuous value representing the severity of the damage.

After obtaining the damage identification results for each image block, the computer system further aggregates these local results to form an overall damage assessment of the entire bridge structure. This involves, for example, weighted averaging of the block results, voting mechanisms, or more complex fusion strategies.

For example, the system can assign different weights to image blocks according to their location and importance in the bridge structure, and then summarize the weighted damage identification results. Alternatively, the system can use a spatial smoothing technique to reduce the impact of local outliers caused by image noise or identification errors on the overall results.

Finally, the computer system organizes the aggregated damage identification results into an easy-to-understand report format and outputs it to the bridge management department or maintenance personnel. This report should include information such as the overall damage of the bridge structure, specific damage location, damage type and degree, so that the management department can take appropriate maintenance and repair measures.

The damage identification model example can use a CNN-based classification model, which learns the mapping relationship between the feature representation of a large number of bridge structure damage images and damage categories during the training phase. For each image block, the model outputs a probability distribution, such as [0.1, 0.8, 0.1], indicating that the block has a 10% probability of no damage, an 80% probability of slight cracks, and a 10% probability of severe cracks. The system determines the damage of the block based on these probability values and generates an overall damage identification report based on this. Through the above steps, the computer system can accurately and efficiently identify the damage of the bridge structure, providing a strong guarantee for the safe operation of the bridge.

In an implementation of the present application, the t implicit image representations include texture implicit representations; based on this, the step S200 of extracting the image implicit representations from the image blocks in the machine vision image to obtain the t implicit image representations corresponding to the image blocks may include:

Step ^S1 210: transmitting the machine vision image to the image implicit representation extraction module of the bridge structure damage recognition model;

Step ^S1 220: in the image implicit representation extraction module, obtaining w texture units of the image block in the machine vision image, w∈[1,+∞); the w texture units are used to represent the texture of the image block;

Step ^S1 230: Obtaining texture unit implicit representations corresponding to the w texture units respectively, performing feature extraction on the w texture unit implicit representations respectively, and obtaining texture extraction implicit representations corresponding to the w texture unit implicit representations respectively;

Step ^S1 240: performing feature dimension reduction on the w texture extraction implicit representations to obtain the texture implicit representation corresponding to the image block.

In step ^S1210 , the computer system passes the collected machine vision image to a specially designed bridge structure damage identification model. This model is a complex machine learning system that can process image data and identify signs of damage in the bridge structure. In order to focus on the extraction of texture features, a specific image implicit representation extraction module is included in the model.

For example, suppose a bridge structure damage recognition model based on convolutional neural network (CNN) is used. CNN has been widely used in the field of computer vision due to its powerful image feature extraction capabilities. In this model, the image implicit representation extraction module can be a combination of one or more convolutional layers, which are responsible for preliminary processing of the input machine vision image to generate the feature map required for subsequent analysis.

The implementation of the implicit representation extraction module for images relies on deep learning frameworks (such as TensorFlow, PyTorch, etc.), which provide all the tools needed to build and train neural networks. In CNN, this module may contain multiple convolutional layers, activation layers (such as ReLU), pooling layers, etc., which are stacked together in a serialized manner. The convolutional layer performs convolution operations with the input image through sliding windows and convolution kernels to extract local features of the image; the activation layer introduces nonlinear factors, allowing the network to learn complex patterns; the pooling layer reduces the size of the feature map by downsampling, reducing the amount of calculation and increasing the robustness of the features.

After the image is passed into the model, in step ^S1 220, the next task is to process it in blocks in the image implicit representation extraction module and extract texture units representing its texture characteristics from each block. Texture units are small areas with specific texture patterns in the image, which together constitute the texture features of the image block. In order to effectively extract texture units, the system first needs to divide the machine vision image into multiple overlapping or non-overlapping image blocks. The size and step size of these blocks can be adjusted according to the specific task to ensure that the entire image can be covered and sufficient local details can be captured. For example, a fixed-size sliding window (such as 64x64 pixels) can be set to slide on the image with a certain step size (such as 32 pixels) to generate a series of image blocks. After obtaining the image blocks, the system identifies representative texture units from them. This usually involves statistical analysis or pattern matching of the pixel values in the blocks. A simple method is to cluster the pixel values in the blocks into several clusters through a clustering algorithm (such as K-means), each cluster corresponding to a texture unit. However, in more complex scenes, more advanced texture analysis techniques may be required, such as local binary patterns (LBP), gray-level co-occurrence matrices (GLCM), or deep learning-based texture descriptors. For example, suppose that in an image patch, the system identifies three texture units: a rough texture representing the concrete surface, a mottled texture representing the corrosion of the steel bar, and a linear texture that may be caused by a crack. These texture units are distinguished from other areas by their gray-level distribution, edge characteristics, or statistical properties.

After the texture units are identified, step ^S1 230 generates an implicit representation for each texture unit and enhances its representativeness through a feature extraction process. The implicit representation is an abstract data structure used to capture the essential characteristics of the texture unit. The implicit representation of the texture unit can be in various forms, such as histograms, moment features, texture descriptors, etc. These representation methods aim to simplify the complex visual patterns of the texture unit into mathematical objects that are easy to process and compare. For example, a gray level co-occurrence matrix (GLCM) can be used to generate a texture descriptor, which reflects the coarseness, directionality, contrast and other characteristics of the texture by counting the gray level dependencies between pixel pairs. After obtaining the implicit representation of the texture unit, the system further extracts features from it to generate a more compact and discriminative implicit representation of the texture extraction. This usually involves applying convolution operations, pooling operations or other types of feature transformations. However, in this step, since the texture unit itself is relatively small and has clear features, a convolutional neural network as complex as that for processing the entire image may not be required. Instead, some simple feature extraction methods such as local feature aggregation descriptors (LBP histograms), SIFT features, etc. can be used. In order to maintain consistency with the overall process and demonstrate the potential of deep learning techniques in feature extraction, it can still be assumed that the system uses a lightweight convolutional neural network to extract features from the implicit representation of texture units. This network may only contain a few convolutional and pooling layers to capture the key features in the texture units.

For example, suppose that for the three texture units previously identified (concrete roughness, steel rust, crack texture), the system generates their implicit representations (such as GLCM descriptors). These descriptors are then input into a lightweight convolutional neural network for feature extraction. The network learns the unique spatial patterns and statistical properties of each texture unit through the convolutional layer, and outputs a more compact and discriminative implicit representation of texture extraction.

After obtaining the texture extraction implicit representation of each texture unit, the system merges these high-dimensional feature vectors into a comprehensive texture implicit representation to represent the texture characteristics of the entire image block in step ^S1 240. However, since direct merging can lead to excessively high feature dimensions and redundant information, feature dimensionality reduction processing is usually required.

Feature dimension reduction is a method of reducing the dimension of feature space, aiming to retain important information in the original data while removing noise and redundancy. In the embodiments of the present application, commonly used feature dimension reduction methods include principal component analysis (PCA), linear discriminant analysis (LDA), pooling, and nonlinear dimension reduction techniques (such as t-SNE, ISOMAP, etc.), which are not specifically limited. For example, assume that after feature extraction, each texture unit corresponds to a 128-dimensional texture extraction implicit representation. The system applies PCA to reduce the dimensionality of these representations and selects the first 32 principal components to approximate the original data. In this way, each image block is mapped to a 32-dimensional low-dimensional space, and its implicit texture representation is also transformed into a 32-dimensional vector. This vector integrates the feature information of all texture units in the image block and removes redundant and noisy parts, providing a more compact and effective feature representation for subsequent damage identification tasks.

In a second implementation of the present application, the t implicit image representations may further include implicit shape representations; based on this, the step S200, extracting implicit image representations from image blocks in the machine vision image to obtain t implicit image representations corresponding to the image blocks, may include:

Step ^S2 210: transmitting the machine vision image to the image implicit representation extraction module of the bridge structure damage recognition model;

Step ^S2 220: in the image implicit representation extraction module, obtaining r region shapes of the image blocks in the machine vision image, r∈[1,+∞); the r region shapes are used to represent image blocks of different shapes;

Step ^S2 230: extracting features of the r region shapes respectively to obtain implicit graphical representations corresponding to the r region shapes respectively;

Step ^S2 240: performing feature dimension reduction on the r implicit representations of the graphics to obtain the implicit representation of the shape corresponding to the image block.

Step ^S2210 can refer to the aforementioned step ^S1210 . After the image is passed into the model, the next step is to further process the image within the image implicit representation extraction module to identify different shape regions in the image. These region shapes may correspond to specific components in the bridge structure (such as piers, main beam sections) or damage signs (such as cracks, deformed areas).

In order to effectively identify the shape of the region in the image, the computer system can use a variety of image processing techniques. A common method is to combine edge detection algorithms (such as Canny edge detector) and shape matching algorithms. First, the edge information in the image is extracted by the edge detection algorithm, which usually corresponds to the outline of the object. Then, the extracted edges are combined into specific shape regions using shape matching algorithms (such as Hough transform, shape context algorithm, etc.). In addition, with the development of deep learning technology, shape segmentation models based on convolutional neural networks have gradually become mainstream. These models can directly segment different shape regions from the original image without explicit edge detection steps. For example, network structures such as U-Net and Mask R-CNN have achieved remarkable results in the field of medical image segmentation, and can also be applied to the shape segmentation task of bridge structure images.

In the bridge image, the system can first identify the edge contours of key components such as piers and main beams through edge detection algorithms. Then, these contours are combined into specific shape regions using shape matching algorithms. Alternatively, if a deep learning model is used for shape segmentation, the system directly outputs different shape regions in the image. These region shapes may include regular geometric shapes (such as rectangles, circles) and irregular crack shapes.

After identifying the regional shapes in the image, step ^S2 230 performs feature extraction on these shape regions to generate corresponding graphic implicit representations. Graphic implicit representation is a high-level feature vector or matrix that can describe the geometric characteristics and spatial distribution of the shape region. For each identified regional shape, the computer system can use a variety of feature extraction methods to generate its graphic implicit representation. These methods include but are not limited to:

Geometric feature extraction: Calculate the basic geometric parameters of the shape area, such as the perimeter, area, and bounding box size. These parameters directly reflect the size and scale information of the shape.

Moment feature extraction: Moment invariants such as Hu moments and Zernike moments are used to describe the rotation, scaling, and translation invariance of shapes. These moment features are very useful for identifying the same shape from different perspectives.

Contour feature extraction: Encode the shape contour through Fourier descriptors, chain codes, etc. These methods can capture the curvature and direction change information of the contour.

Deep learning feature extraction: Convolutional neural networks are used to learn features of shape regions. The network extracts high-level semantic features of the shape through multi-layer convolution and pooling operations, and outputs a compact feature vector as an implicit representation of the graph.

For example, for each shape region in a bridge image (such as a bridge pier, crack region, etc.), the system can first calculate its basic geometric parameters (such as perimeter, area) as preliminary features. Then, the Hu moment is used to extract the invariant features of the shape to cope with problems such as image rotation and scaling. If a deep learning model is used for feature extraction, the system directly uses a pre-trained convolutional neural network to process the shape region and output a high-dimensional feature vector as a graphical implicit representation of the region.

After the graphic implicit representation of each shape region is extracted, since these representations may have high dimensionality and contain redundant information, step ^S2 240 is required to perform feature dimensionality reduction processing to generate a compact shape implicit representation.

In a third implementation scheme of the present application, the t implicit image representations also include color implicit representations; based on this, the step S200, extracting the image implicit representations from the image blocks in the machine vision image to obtain the t implicit image representations corresponding to the image blocks, may include:

Step ^S3 210: transmitting the machine vision image to the image implicit representation extraction module of the bridge structure damage recognition model;

Step ^S3 220: In the image implicit representation extraction module, the machine vision image is divided into blocks to obtain g image blocks in the machine vision image, g∈[1,+∞); the g image blocks are used to merge into the machine vision image, and the g image blocks are image blocks of different color blocks in the machine vision image;

Step ^S3 230: Obtain the color implicit representation corresponding to the target image block from the color implicit representations corresponding to the g image blocks, and determine the color implicit representation corresponding to the target image block as the color implicit representation corresponding to the image block in the machine vision image.

Step ^S3210 can refer to the aforementioned step ^S1210 . In step ^S3220 , in the image implicit representation extraction module, the machine vision image is divided into blocks to obtain g image blocks in the machine vision image. Since the bridge structure image is usually large in size and contains rich detail information, directly extracting color features from the entire image may be inefficient due to excessive calculation. Therefore, before extracting the color implicit representation, the image is divided into blocks.

Computer systems can use a variety of methods to divide images into blocks. A simple and commonly used method is uniform grid division, which divides the image into multiple small blocks of equal size and neatly arranged (i.e., image blocks). Each image block contains part of the image content, and there are overlapping areas between them to ensure the continuity of information. In addition, block division can also be performed adaptively based on the image content, such as dynamically determining the size and position of the blocks based on changes in image edges, textures, or colors.

After the image blocks are obtained, the corresponding color implicit representation is generated for each block in step ^S3 230. However, in the third implementation, the color features of each block are not directly extracted and represented independently; instead, the focus is on how to identify the target block related to a specific damage sign (such as rust) from the color features of all blocks and obtain its corresponding color implicit representation.

In order to extract the implicit color representation, the computer system can use a variety of color feature description methods. These methods include but are not limited to color histogram, color moment, color aggregation vector, etc. However, in the specific scenario of bridge structure damage identification, more attention may be paid to color features that can reflect the corrosion phenomenon. Therefore, a method based on color space conversion and color clustering can be used to identify the color area related to corrosion. First, the system can convert the image blocks from the RGB color space to a color space that is more suitable for corrosion detection (such as HSV or Lab color space). In these color spaces, the corrosion phenomenon is usually manifested as hue and saturation values within a specific range. Then, the system can perform cluster analysis on the converted color values, classify pixels with similar colors into one category, and thus identify the color area related to corrosion. Next, in order to generate the implicit color representation, the system can extract features from the identified rusted area. These features may include the average color value of the rusted area, the variance of the color distribution, the area proportion of the rusted area, etc. These features together constitute the color implicit representation of the rusted area for subsequent damage identification tasks. However, in practical applications, it may not be necessary to perform a complete color implicit representation extraction process for each image block. Instead, a more efficient method can be adopted: first, the possible rust areas (i.e., target image blocks) are roughly located through a fast color screening and region growing algorithm, and then only these areas are subjected to detailed color implicit representation extraction.

In the bridge structure image, the system first traverses all image blocks and uses color screening algorithms (such as threshold-based color range judgment) to quickly locate blocks that may contain rust. These blocks are marked as target image blocks and serve as the focus of subsequent color implicit representation extraction. Then, the system performs color space conversion and color clustering analysis on each target image block to identify specific rusted areas. Finally, the system performs feature extraction and encoding processing on the rusted area (such as using color histograms or color moments as color implicit representations) to generate color implicit representations corresponding to each target image block.

It should be noted that in actual applications, there may be multiple target image blocks containing similar corrosion phenomena. In order to comprehensively evaluate the corrosion of the entire bridge structure, the system can further aggregate or fuse the color implicit representations of all target image blocks (such as through weighted averaging, maximum pooling, etc.) to generate a global color implicit representation as the basis for bridge structure damage identification.

In a fourth implementation scheme of the present application, the t image implicit representations further include semantic implicit representations; based on this, the step S200 of extracting image implicit representations from image blocks in the machine vision image to obtain t image implicit representations corresponding to the image blocks may include:

Step ^S4 210: transmitting the machine vision image to the image implicit representation extraction module of the bridge structure damage recognition model;

Step ^S4 220: In the image implicit representation extraction module, image semantic representation is performed on the image blocks in the machine vision image to obtain initial semantic implicit representations corresponding to the image blocks;

Step ^S4 230: determining the image coordinates of the image block in the machine vision image, encoding the image coordinates of the image block, and obtaining an implicit representation of the coordinates corresponding to the image block;

Step ^S4240 : Obtain the positioning vector corresponding to the image block, merge the initial semantic implicit representation corresponding to the image block, the coordinate implicit representation corresponding to the image block and the positioning vector corresponding to the image block, and obtain the semantic implicit representation corresponding to the image block.

Step S ⁴ 210 can refer to the aforementioned step S ¹ 210. In step S ⁴ 220, the computer system processes the image in blocks and performs semantic analysis on each image block to obtain its initial semantic implicit representation. The semantic implicit representation not only contains the visual feature information of the image block, but also attempts to understand the actual meaning and contextual relationship represented by these features. In order to obtain the initial semantic implicit representation of the image block, the computer system can adopt a variety of methods. For example, a pre-trained convolutional neural network (CNN) is used to extract the visual features of the image block, and a deep learning model (such as ResNet, DenseNet, etc.) is used to extract the visual features of the image block, and these features are converted into initial semantic implicit representations through components such as fully connected layers or attention mechanisms.

In step S ⁴ 230, the computer system determines the position of each image block in the original image (i.e., the image coordinates), and encodes the coordinate information to generate an implicit representation of the coordinates. In order to encode the coordinate information of the image blocks, the computer system can adopt a variety of methods. One method is to directly use the coordinate values (such as x, y coordinates) as part of the encoding. However, this method may not be effective enough when processing high-dimensional data because the coordinate values themselves do not contain rich spatial structure information. Alternatively, position embedding is used. Alternatively, the coordinate values of the image blocks can be converted into two-dimensional grid indexes (such as dividing the image into multiple small grids and assigning a unique index number to each grid), and then these index numbers are converted into embedded vectors. These embedded vectors can be generated by a lookup table or an embedding layer.

For example, in a bridge structure image, the system first determines the coordinate position of each image block (such as the pixel coordinates of the upper left corner and the lower right corner). Then, the system converts these coordinate values into two-dimensional grid indices and converts these indices into implicit coordinate representations through the embedding layer. For example, assuming that the image is divided into 10×10 grid areas, each image block can be assigned a unique grid index (such as (3, 5)). The system converts this index into a fixed-length embedding vector as an implicit coordinate representation through the embedding layer. This vector can reflect the relative position information of the image block in the image.

In step S ⁴ 240, the computer system obtains the positioning vector of each image block and merges it with the initial semantic implicit representation and the coordinate implicit representation to generate a final semantic implicit representation. The positioning vector is a special embedding vector that is used to represent the identity or role of the image block in the overall image or a specific context. By merging these three representation forms, the system can generate a more comprehensive and semantically rich feature vector for subsequent damage identification tasks. The method of obtaining the positioning vector is similar to the coordinate implicit representation. The coordinate implicit representation mainly focuses on the spatial position information of the image block, while the positioning vector focuses more on the identity or role information of the image block in the overall image or a specific context. In practical applications, the positioning vector can be obtained in a variety of ways:

Predefined embedding vectors: Each image block can be assigned a predefined embedding vector as a localization vector. These embedding vectors can be generated by random initialization or based on some rules (such as the size, shape, etc. of the block).

Learned embedding vectors: The embedding vectors learned during training can also be used as localization vectors. For example, when training a bridge structure damage recognition model, each image block can be treated as an independent symbol, and an embedding vector is generated for each symbol through the embedding layer as a localization vector.

Context-dependent embedding vectors: In some cases, the localization vector may be closely related to the contextual information around the image patch. In this case, methods such as attention mechanisms or graph neural networks can be used to capture the contextual information and generate context-dependent localization vectors.

After obtaining the initial semantic implicit representation, coordinate implicit representation, and positioning vector, the computer system merges them to generate the final semantic implicit representation. The merging process can be achieved through a simple splicing operation or optimized through more complex feature fusion methods (such as attention mechanism, bilinear pooling, etc.). Regardless of the method used, the merged semantic implicit representation should be able to fully and accurately reflect the visual features, spatial location information, and contextual semantic information of the image blocks.

For example, in a bridge structure image, the system generates an initial semantic implicit representation and a coordinate implicit representation for each image block, and obtains a positioning vector in some way (such as through a predefined embedding vector or a learned embedding vector). The system then merges these three representations to generate a final semantic implicit representation. For example, the system can simply concatenate the initial semantic implicit representation, the coordinate implicit representation, and the positioning vector to obtain a longer feature vector; or use an attention mechanism to perform a weighted summation of the three representations to generate a weighted feature vector as the final semantic implicit representation. Regardless of which method is used, this representation will be used in subsequent damage identification tasks to determine whether there are signs of damage in the image block.

Through the detailed explanation and example of the above steps, we can clearly see how the fourth implementation of step S200 extracts semantic implicit representation from machine vision images. This process not only involves the application of complex image processing technology and deep learning models, but also fully considers the comprehensive representation of semantic information, spatial position information and contextual semantic information of image blocks. The semantic implicit representation finally generated will provide more comprehensive and accurate feature information support for the subsequent bridge structure damage identification task.

As an implementation scheme of the present application, the step S300 of merging t implicit representations of the image to obtain a merged implicit representation corresponding to the image block, and adaptively integrating the features of the merged implicit representation to obtain an adaptive implicit representation for highlighting the merged implicit representation may include:

Step S310: passing the t implicit representations of the images into an implicit representation merging module of the bridge structure damage recognition model.

Step S320: In the implicit representation merging module, t implicit representations of the image are merged to obtain a merged implicit representation corresponding to the image block.

Step S330: weighting the combined implicit representation with a weight matrix to obtain h adaptive execution features corresponding to the combined implicit representation, h∈[1,+∞).

Step S340: Adaptively weighting the h adaptive execution features to obtain an adaptive implicit representation for highlighting the combined implicit representation.

In the embodiment of the present application, step S300 effectively merges multiple types of implicit image representations (such as texture implicit representation, shape implicit representation, color implicit representation, semantic implicit representation, etc.), and generates a more representative adaptive implicit representation through adaptive feature integration technology. This process not only improves the richness and expressiveness of features, but also provides a more solid data foundation for subsequent damage identification.

Specifically, in step S310, the computer system passes the multiple implicit representations of images (t in total) extracted in step S200 to the implicit representation merging module in the bridge structure damage identification model. This module is specially designed to handle the fusion problem of multiple implicit representations of images, and aims to generate a merged implicit representation that integrates all feature information through an effective merging strategy.

The implicit representation merging module can be a complex system that integrates multiple feature fusion techniques, including but not limited to concatenation, weighted summation, principal component analysis (PCA), and attention mechanism.

For example, it is assumed that in the embodiment of the present application, texture implicit representation (feature vector length is n1), shape implicit representation (feature vector length is n2), color implicit representation (feature vector length is n3) and semantic implicit representation (feature vector length is n4) have been extracted from a certain image block. These four implicit representations respectively reflect the feature information of the image block in different dimensions. The computer system passes these implicit representations as input to the implicit representation merging module, preparing for subsequent merging operations.

In step S320, inside the implicit representation merging module, the computer system merges the t implicit representations of the input images. The purpose of the merging is to generate a merged implicit representation that integrates all feature information, which will serve as the input for the subsequent adaptive feature integration.

For example, a simple merging method is direct concatenation. That is, the implicit representations of t images are connected end to end in a certain order to form a longer feature vector. For example, if the texture implicit representation is [t1, t2, ..., tn1], the shape implicit representation is [s1, s2, ..., sn2], the color implicit representation is [c1, c2, ..., cn3], and the semantic implicit representation is [m1, m2, ..., mn4], then the merged implicit representation is [t1, t2, ..., tn1, s1, s2,..., sn2, c1, c2, ..., cn3, m1, m2, ..., mn4]. Direct concatenation can lead to excessively high feature dimensions and increase the computational complexity of subsequent processing. Therefore, in practical applications, other more efficient merging methods can be considered, such as weighted summation, principal component analysis (PCA) dimensionality reduction, etc.

Assume that the merged implicit representation is a vector of length N (N=n1+n2+n3+n4), which integrates the feature information of the image block in four dimensions: texture, shape, color, and semantics. This vector will be used as input in the subsequent steps to generate adaptive implicit representations.

After the merged implicit representation is obtained, step S330 performs weighted processing on it with a weight matrix to generate an adaptive execution feature (i.e., a self-attention input vector). This process involves, for example, linear transformation and weight assignment, which aims to highlight the key features in the merged implicit representation and provide preparation for subsequent adaptive weighting operations.

Weight matrix weighting can be achieved through linear transformation. Specifically, a weight matrix W can be defined (whose shape may match the feature dimension of the merged implicit representation), and then the merged implicit representation is multiplied by the weight matrix to obtain the weighted feature vector. In the self-attention mechanism, three different weight matrices are involved: the query weight matrix (Wq), the key weight matrix (Wk), and the value weight matrix (Wv), which are used to generate request features (Query), index features (Key), and output features (Value), respectively.

In the embodiment of the present application, it can be assumed that the self-attention mechanism is used to generate adaptive execution features. Specifically, the merged implicit representation is first linearly transformed through the query weight matrix Wq to generate the request feature; then, the merged implicit representation is linearly transformed through Wk to generate the index feature (i.e., key); the output feature can be directly used by the merged implicit representation itself or further transformed through the value weight matrix Wv.

Assume that after weighting by the weight matrix, h adaptive execution features are obtained (i.e., self-attention features that need to be input, which can be a vector). These features may have the same dimensions as the merged implicit representation or different dimensions after further transformation. They will be used as input for subsequent self-attention processing to generate adaptive implicit representations.

After obtaining h adaptive execution features, step S340 performs adaptive weighting operations on these features, such as applying a self-attention mechanism to generate adaptive implicit representations. The self-attention mechanism calculates the similarity (or attention score) between the request feature and the index feature, and performs a weighted summation of the output features based on these similarities, thereby achieving dynamic selection and importance adjustment of the input features.

First, the dot product (or scaled dot product) between the request feature and the index feature is calculated, and then the softmax function is applied to normalize these dot product values into a probability distribution. Next, the output features are weighted summed using these attention weights to obtain the weighted feature vector as the output.

In practical applications, the merged implicit representation can be used to generate request features, index features, and output features through multiple linear transformation layers. Then, the weighted feature vector is generated as the adaptive implicit representation according to the calculation process of the self-attention mechanism. This representation not only integrates the feature information of multiple implicit representations of the image, but also strengthens and highlights the key features through the self-attention mechanism.

Assume that the adaptive implicit representation obtained after self-attention processing is a feature vector with the same dimension as the merged implicit representation or after further transformation. This vector integrates the feature information of the image block in multiple dimensions and strengthens the key features through the self-attention mechanism. It will be used as feature input in the subsequent bridge structure damage identification task.

In an optional implementation, the h adaptive execution features include index features, output features, and request features. Based on this, the step S340 of performing adaptive weighting operations on the h adaptive execution features to obtain an adaptive implicit representation for highlighting the merged implicit representation may include:

Step S341: obtaining an inverse feature corresponding to the index feature, performing inner product calculation on the request feature and the inverse feature to obtain an inner product matrix, wherein the inner product matrix is used to represent the similarity between the t implicit representations of the images;

Step S342: normalizing the inner product matrix to obtain a normalized result corresponding to the inner product matrix, performing inner product calculation on the normalized result and the output feature to obtain an adaptive merging result for merging and representing the implicit representations of the t images;

Step S343: performing multi-layer perceptron processing on the adaptive merging result to obtain an adaptive implicit representation for highlighting the merged implicit representation.

In the self-attention mechanism, index features (Key), output features (Value) and request features (Query) play a core role. In the embodiment of the present application, these feature vectors represent different aspects of the implicit representation of the image, such as texture, shape, color, etc. In order to calculate the similarity between these feature vectors, the system first needs to process the index feature and perform an inner product calculation with the request feature.

The inverse feature is the transpose of the index feature. After the system obtains the inverse feature of the index feature, it calculates the inner product with the request feature. The inner product operation measures the similarity or correlation between two vectors. In the embodiment of the present application, this means that the system is evaluating the degree of association between the implicit representations of different images (represented by their corresponding index features and request features).

After the inner product matrix is generated, in step S342, it is normalized to ensure the stability of subsequent calculations. The normalization process, for example, involves applying a softmax function to normalize the rows (or columns) of the inner product matrix into a probability distribution form. Then, the system uses the normalized result and the output feature to perform an inner product calculation to generate an adaptive merging result.

Assume that the inner product matrix is A, and the softmax normalized result of its i-th row is softmax(ai), where ai is the i-th row vector of A. Then the normalized inner product matrix can be expressed as softmax(A), where each row is a probability distribution.

The normalized inner product matrix is inner-producted with the output features (Value vectors) to generate an adaptive merge result. This process is equivalent to weighted summation of the output features according to the attention weights.

After obtaining the adaptive merging results (ie, the weighted output feature vectors), step S343 further processes these results to enhance their expressiveness and nonlinear characteristics. Multilayer perceptron (MLP) is a commonly used processing method that can transform and learn input features through multiple hidden layers.

The multilayer perceptron is composed of multiple fully connected layers, each of which contains a certain number of neurons and introduces nonlinear characteristics through activation functions (such as ReLU, sigmoid, etc.). In the embodiment of the present application, the system can use the adaptive merging result as the input of the multilayer perceptron and transform and learn it through a series of hidden layers. Ultimately, the output of the multilayer perceptron is the desired adaptive implicit representation.

Assuming that the adaptive merging results in two vectors (corresponding to the two requested features above), the system uses these vectors as the input of the multilayer perceptron. The multilayer perceptron transforms and processes these inputs through its internal hidden layers and activation functions, and finally outputs two adaptive implicit representation vectors. These vectors not only contain the information of the implicit representation of the original image, but also enhance their expressiveness and robustness through the self-attention mechanism and multilayer perceptron processing.

As an implementation scheme of the present application, the step S400 of merging the image block connectivity matrix corresponding to the machine vision image and the output implicit representation corresponding to the image block step by step to obtain the merged implicit representation corresponding to the image block may include:

Step S410: The image block connectivity matrix corresponding to the machine vision image and the output implicit representation corresponding to the image block are input into a step-by-step merging module of the bridge structure damage recognition model, wherein the step-by-step merging module includes a feature extraction submodule, v step-by-step merging submodules and a merging submodule, v∈[1,+∞).

Step S420: In the feature extraction submodule, feature extraction is performed on the output implicit representation corresponding to the image block to obtain a feature extracted implicit representation corresponding to the image block.

Step S430: In the v step-by-step merging sub-modules, based on the image block connectivity matrix corresponding to the machine vision image and the feature extraction implicit representation corresponding to the image block, the hierarchical implicit representations corresponding to the v step-by-step merging sub-modules are obtained, and the hierarchical implicit representation corresponding to the target step-by-step merging sub-module in the v step-by-step merging sub-modules is determined as the step-by-step merging implicit representation corresponding to the image block; the target step-by-step merging sub-module is the last step-by-step merging sub-module among the v step-by-step merging sub-modules.

Step S440: In the merging submodule, the step-by-step merged implicit representation corresponding to the image block and the feature extraction implicit representation corresponding to the image block are merged to obtain a merged implicit representation corresponding to the image block.

In step S410, the computer system passes the key input data (image block connectivity matrix and output implicit representation) to the step-by-step merging module in the bridge structure damage identification model. This module is specially designed to process the spatial relationship between image blocks and their respective feature information to generate a higher level merged implicit representation.

The step-by-step merging module is a complex neural network that contains multiple submodules, each of which has a specific processing task. Specifically, the module includes a feature extraction submodule, v step-by-step merging submodules (such as the convolutional network GCN in the figure), and a merging submodule. These submodules work together to achieve the conversion process from the original input to the final merged implicit representation.

In step S420, after receiving the output implicit representation of the image block, the feature extraction submodule first performs further feature extraction operations on it. The purpose of this step is to extract more refined and distinguishing feature information from the output implicit representation for subsequent processing and analysis.

The feature extraction submodule may use a variety of techniques to achieve feature extraction, including but not limited to linear transformation, nonlinear activation function, convolution operation, etc. However, in this specific scenario, since the output implicit representation is already a high-dimensional vector or matrix that integrates multiple features, the feature extraction process may focus more on dimensionality reduction and transformation of features in order to better adapt to subsequent graph convolutional network processing.

In step S430, after obtaining the feature extraction implicit representation of the image block, the system passes these representations and the image block connectivity matrix to v level-by-level merging submodules for processing. These submodules are usually implemented using graph neural networks (such as GCN) because they can naturally process graph structured data and effectively capture the spatial relationship between nodes (image blocks).

The graph convolutional network is a neural network specifically designed for graph structured data processing. It aggregates information about node neighbors through layer-by-layer convolution operations to generate an updated representation of each node. In an embodiment of the present application, the GCN can use the image block connectivity matrix as the adjacency matrix input of the graph, the feature extraction implicit representation as the initial feature input of the node, and then update the node representation through multiple rounds of convolution operations.

In the step-by-step merging submodule, the system constructs graph structure data based on the image block connectivity matrix and feature extraction implicit representation. Then, starting from the first step-by-step merging submodule, GCN is applied layer by layer to aggregate node information and update representation. Each layer of GCN generates a new node representation based on the node representation and adjacency matrix of the current layer. As the number of layers increases, the representation of the node will gradually integrate the information of more neighboring nodes, thus forming a more comprehensive and accurate hierarchical implicit representation.

After being processed by v level-by-level merging submodules, the system uses the hierarchical implicit representation output by the last (i.e., target) level-by-level merging submodule as the level-by-level merging implicit representation corresponding to the image block. This representation not only contains the feature information of the image block itself, but also integrates the information of its neighboring blocks and the spatial relationship information in the connectivity matrix of the entire image block.

In step S440, after obtaining the step-by-step merged implicit representation of the image blocks, the system does not directly use it as the final output. Instead, it merges this representation with the original feature extraction implicit representation to generate a more complete and rich merged implicit representation.

The merging submodule may use a variety of methods to merge two implicit representations. A simple and common method is concatenation, which directly connects the two representations end to end to form a longer vector. However, this method can lead to excessive feature dimensions and increased computational complexity. Therefore, in practical applications, the system can use more complex feature fusion techniques (such as attention mechanism, bilinear pooling, etc.) to optimize the merging process.

For example, suppose that the implicit representation of the step-by-step merging obtained after processing by v step-by-step merging submodules is a 256-dimensional feature vector, and the original feature extraction implicit representation is a 128-dimensional feature vector. In the merging submodule, the system can concatenate these two vectors to obtain a 384-dimensional merged implicit representation. Considering the issues of computational efficiency and feature redundancy, the system can first reduce the dimensionality of the two vectors (such as by PCA dimensionality reduction), and then concatenate them or use other fusion methods.

The resulting merged implicit representation will be used as one of the input data for the subsequent damage identification step. It not only contains the feature information of the image block itself (embodied by the feature extraction implicit representation), but also integrates the spatial relationship information between the image blocks (embodied by the step-by-step merging of implicit representations). This comprehensive feature representation will help improve the accuracy and robustness of damage identification.

In an optional implementation, the v step-by-step merging submodules include a step-by-step merging submodule C _b , and b is not greater than v. Based on this, in step S430, in the v step-by-step merging submodules, based on the image block connectivity matrix corresponding to the machine vision image and the feature extraction corresponding to the image block, the hierarchical implicit representations corresponding to the v step-by-step merging submodules are obtained, which may include two cases:

Case 1: When the step-by-step merging submodule C _b is the first step-by-step merging submodule among the v step-by-step merging submodules, the step-by-step merging submodule C _b performs context feature extraction on the image block connectivity matrix corresponding to the machine vision image and the feature extraction implicit representation corresponding to the image block, and obtains the hierarchical implicit representation corresponding to the step-by-step merging submodule C _b ;

Case 2: When the step-by-step merging submodule C _b is not the first step-by-step merging submodule among the v step-by-step merging submodules, the step-by-step merging submodule C _b performs context feature extraction on the image block connectivity matrix corresponding to the machine vision image and the hierarchical implicit representation corresponding to the step-by-step merging submodule C _a to obtain the hierarchical implicit representation corresponding to the step-by-step merging submodule C _b ; the step-by-step merging submodule C _a is the previous step-by-step merging submodule of the step-by-step merging submodule C _b .

In case 1, it is assumed that in the initial stage of bridge structure damage identification, the computer system has completed the feature extraction of image blocks and constructed an image block connectivity matrix representing the spatial relationship between image blocks. Now, these input data are sent to the first step-by-step merging submodule C _b for processing to generate a preliminary hierarchical implicit representation. In this application, it is obvious that a and b in the step-by-step merging submodules C _a and C _b both represent the sequence number of the step-by-step merging submodules. For example, when v=5, a and b cannot be greater than 5, then the values of a and b are 1~5, for example, a=1, b=2, at this time it can be understood that the step-by-step merging submodule C _a is the first step-by-step merging submodule, and the step-by-step merging submodule C _b is the second step-by-step merging submodule, because 2-1=1, so the step-by-step merging submodule C _a is the previous step-by-step merging submodule C _b . As for the C in the step-by-step merging submodules C _a and C _b , it represents the step-by-step merging submodule. To make it easier to understand, let's take a common example. There are 5 athletes at the sports meeting. They are all called Athletes. For the sake of simplicity, they are all considered A (that is, the abbreviation of Athletes). In order to distinguish them, they are divided according to the aforementioned serial number method. A certain athlete Zhang San, his abbreviation is A ₁ , Li Si's abbreviation is A ₂ , and so on (of course, this is only a division method. The serial numbers of Zhang San and Li Si can also be other, but none of them is greater than 5). Then, it is easy to understand that due to normal logical settings, the number after A cannot exceed 5. Corresponding to this application, the C in the step-by-step merging of sub-modules _Ca and _Cb is analogous to the above A. In addition, it is obvious that a, b, and v are all positive integers, that is, they cannot be fractions and decimals.

The computer system passes the image block connectivity matrix and the feature extraction implicit representation of the image blocks as input to the first level-by-level merging submodule C _b . The image block connectivity matrix is a two-dimensional array, in which the elements represent the connectivity between image blocks (such as adjacent relations, distances, etc.); the feature extraction implicit representation is a high-dimensional vector or matrix that contains various feature information of the image blocks.

In the first step-by-step merging submodule C _b , the system uses a graph convolutional network (GCN) or other graph neural network to extract contextual features from the input data. Specifically, GCN extracts implicit representations of image blocks as node features based on the graph structure defined by the image block connectivity matrix and the features of the image blocks, and performs graph convolution operations. In this process, GCN considers the neighbor information of each image block and updates the representation of the current node by aggregating the features of neighboring nodes.

After one or more rounds of graph convolution operations, the first level-by-level merging submodule _Cb outputs hierarchical implicit representations corresponding to each image block. These representations not only contain the feature information of the image block itself, but also incorporate the information of its neighboring blocks and the spatial relationship information in the connectivity matrix of the entire image block.

For example, suppose the bridge image is divided into four image blocks (A, B, C, D), and the connectivity between them is represented by the image block connectivity matrix. After feature extraction, each image block obtains a 128-dimensional feature vector. In the first step-by-step merging submodule C _b , the system uses a graph neural network containing two layers of GCN for processing.

First layer GCN: According to the image block connectivity matrix and the initial feature vector, the representation of each image block is updated through graph convolution operation. Assuming that the convolution kernel size of GCN is 3 (i.e., considering the first-order neighbors of each node), the new representation of each node will be the weighted average of its own features and the features of its first-order neighbors (the weight coefficient is learned by GCN).

The second layer of GCN: further performs graph convolution operations based on the updated node representation and image block connectivity matrix to generate the final hierarchical implicit representation. These representations will be used in subsequent damage identification tasks.

In case 2, after the first level-by-level merging submodule processes the data, a preliminary hierarchical implicit representation is generated. Next, these representations will be passed as input to the subsequent level-by-level merging submodule (such as C _b ) for further processing. This process will be repeated until all level-by-level merging submodules have completed processing and generated the final merged implicit representation.

For non-first level-by-level merging submodule C _b , its input data includes two parts: one is the image block connectivity matrix (same as case 1), and the other is the hierarchical implicit representation from the previous level-by-level merging submodule _Ca. This means that the C _b module will perform further feature extraction and representation update based on the output of the _Ca module.

Similar to case 1, the C _b module also uses graph convolutional networks or other graph neural networks to extract contextual features from the input data. However, the difference is that this time the C _b module considers the updated hierarchical implicit representation (i.e., the output of the C _a module) and the spatial relationship in the image block connectivity matrix. Through a new round of graph convolution operations, the C _b module will further fuse the information between image blocks to generate a more refined hierarchical implicit representation.

After processing, the C _b module outputs new hierarchical implicit representations for each image block. These representations are more comprehensive and accurate than the outputs of the C _a module because they incorporate more levels of contextual information. As the processing flow progresses, each level-by-level merging submodule generates its own hierarchical implicit representation until the last module completes processing.

Let's continue with the example of the four image blocks (A, B, C, D) of the bridge image. Assume that the first level-by-level merging submodule has generated preliminary hierarchical implicit representations (each block corresponds to a 128-dimensional feature vector). Now, these representations are passed to the second level-by-level merging submodule C _b for processing.

The C _b module first receives the hierarchical implicit representation and image block connectivity matrix from the C _a module. Then, it uses the graph convolutional network to perform a new round of feature extraction and representation update. Assuming that the C _b module also contains two layers of GCN, it will generate a more refined hierarchical implicit representation based on the output of the C _a module and the image block connectivity matrix. Through the processing of multiple step-by-step merging sub-modules, the system is able to gradually fuse the contextual information between image blocks. Each module will optimize and improve on the basis of the previous module to generate a more accurate and comprehensive hierarchical implicit representation. This step-by-step merging strategy helps to improve the accuracy and robustness of damage identification.

In summary, the step-by-step merging submodule in step S430 generates a hierarchical implicit representation by gradually integrating the image block connectivity matrix and the implicit representation of feature extraction of the image blocks. This process not only considers the feature information of the image blocks themselves, but also incorporates the spatial relationship information between them and multi-level context information. The processing system through multiple step-by-step merging submodules can generate a more comprehensive and accurate feature representation, thereby providing strong support for the subsequent bridge structure damage identification task.

In an optional implementation, the number of image blocks is m, and the m image blocks include a target image block, m∈(1,+∞); the image block connectivity matrix corresponding to the machine vision image includes connectivity parameters of the target image block for the m image blocks respectively; the feature extraction implicit representation corresponding to the image block includes the feature extraction implicit representation corresponding to the target image block. Then, in case 1, the context feature extraction is performed on the image block connectivity matrix corresponding to the machine vision image and the feature extraction implicit representation corresponding to the image block by the step-by-step merging submodule C _b to obtain the hierarchical implicit representation corresponding to the step-by-step merging submodule C _b , which may include:

Step S4311: obtaining connectivity features of the target image block for m image blocks respectively in the step-by-step merging submodule C _b .

Step S4312: In the step-by-step merging submodule C _b , the feature extraction implicit representation corresponding to the target image block and the m connectivity features are merged respectively to obtain merged connectivity features corresponding to the m image blocks.

Step S4313: performing weighted summation on the m merged connectivity features according to the m connectivity parameters to obtain image association features corresponding to the target image block.

Step S4314: activating the image association features corresponding to the target image block to obtain the hierarchical sub-implicit representation corresponding to the target image block, wherein the hierarchical sub-implicit representation corresponding to the target image block belongs to the hierarchical implicit representation corresponding to the step-by-step merging submodule C _b .

In step S4311, the computer system extracts connectivity features related to the target image block from the image block connectivity matrix. The image block connectivity matrix is a two-dimensional array, the number of rows and columns of which are equal to the number of image blocks m, and each element in the matrix represents a connectivity parameter between two image blocks. The value range of these parameters is usually [0,1], where 0 indicates no relationship between the two image blocks and 1 indicates a close relationship.

In step S4312, the computer system merges the feature extraction implicit representation corresponding to the target image block with the connectivity feature extracted from the connectivity matrix. The merging operation is usually implemented by splicing, that is, the feature extraction implicit representation is connected end to end with each connectivity feature vector to form a new vector.

For simplicity of explanation, a simplified merging approach is assumed: each connectivity parameter is added as an additional feature dimension to the end of the feature extraction implicit representation.

For example, assume that the feature extraction corresponding to the target image block B is implicitly represented as a 128-dimensional feature vector fB. After obtaining the connectivity parameters [0.8, 0, 0.5, 0.3] of the four blocks A, B, C, and D of block B, the system adds these parameters to the end of fB as additional feature dimensions to obtain a new merged connectivity feature vector fB′, whose dimension is 132 (128-dimensional original features + 4-dimensional connectivity features). However, please note that in practical applications, since the connectivity parameters are scalar values, it may be necessary to first convert them into vectors that match the dimensions of the original feature vectors through some method (such as an embedding layer) and then concatenate them. But here, a simplified merging method is adopted.

Another approach is to use graph neural networks such as graph convolutional networks (GCN) to naturally process this graph structure data. In GCN, the connectivity matrix is used as the adjacency matrix input of the graph, and the feature extraction implicit representation is used as the initial feature input of the node. GCN automatically fuses the features of a node with the features of its neighboring nodes through graph convolution operations without explicit feature splicing.

In step S4313, the m merged connectivity features are weighted and summed according to the m connectivity parameters to obtain the image association features corresponding to the target image block. That is, the m connectivity parameters are used as weights to perform weighted summation on the m merged connectivity features.

In step S4314, the computer system activates the image-related features corresponding to the target image blocks. The activation function is a nonlinear transformation function commonly used in neural networks, which can introduce nonlinear factors so that the neural network can learn more complex patterns. In image recognition or feature extraction tasks, commonly used activation functions include ReLU (Rectified Linear Unit), Sigmoid, Tanh, etc. These functions have their own advantages and disadvantages, but in most cases, the ReLU function is widely used due to its simplicity and effectiveness. If a graph neural network such as a graph convolutional network is used to process the connectivity relationship between image blocks, the activation function is usually built into the output of the graph convolutional layer. That is, after the graph convolutional layer aggregates and transforms the node features, the activation function is automatically applied to introduce nonlinear factors.

In summary, the implementation of case 1 is to gradually merge the feature extraction implicit representation and connectivity features of the target image blocks, and apply activation functions to generate hierarchical sub-implicit representations. However, please note that in practical applications, a more common and effective method is to use graph neural networks such as graph convolutional networks to automatically handle the connectivity relationship and feature fusion problems between image blocks. Through the application of multi-layer graph convolution operations and nonlinear activation functions, graph neural networks can gradually generate more accurate and comprehensive node representations (i.e., hierarchical implicit representations), thereby providing strong support for subsequent damage identification tasks.

As an implementation scheme of the present application, the step S500, performing bridge structure damage identification on the image blocks according to the combined implicit representation to obtain image block damage identification results, may include:

Step S510: passing the merged implicit representation into a damage identification module of a bridge structure damage identification model.

Step S520: In the damage identification module, multi-layer perceptron processing is performed on the combined implicit representation to obtain damage features corresponding to the image blocks.

Step S530: normalize the damage feature to obtain a normalized damage feature corresponding to the damage feature.

Step S540: performing bridge structure damage identification on the image blocks according to the normalized damage features to obtain image block damage identification results.

In step S510, the computer system transmits the combined implicit representation generated in step S400 to a damage identification module in the bridge structure damage identification model, such as a classifier.

In step S520, within the damage identification module, a multilayer perceptron (MLP) as a feedforward artificial neural network is used to further process and analyze the merged implicit representation. The multilayer perceptron can learn the complex mapping relationship between the input data (i.e., the merged implicit representation) and the output target (i.e., the damage feature) through multiple layers of nonlinear transformation.

A multilayer perceptron usually consists of an input layer, several hidden layers, and an output layer. In the input layer, the merged implicit representation is passed to the network as input data; in the hidden layer, the network transforms and combines the input data through a series of nonlinear activation functions (such as ReLU, Sigmoid, etc.) and weight matrices; in the output layer, the network outputs the transformed feature vector, i.e., the damage feature.

For example, suppose the merged implicit representation is a 256-dimensional feature vector, which represents the comprehensive information of the image block in multiple dimensions. This vector is fed into the multi-layer perceptron network. The network first receives this vector through the input layer, and then processes it through several hidden layers (each hidden layer may contain hundreds or even thousands of neurons, and nonlinearity is introduced through the ReLU activation function), and finally generates a lower-dimensional feature vector (such as 32 dimensions) at the output layer. This vector is the damage feature corresponding to the image block.

After obtaining the damage features, for the convenience and accuracy of subsequent processing, step S530 normalizes these features. Normalization refers to scaling the feature values to a uniform range (such as [0, 1] or [-1, 1]) to eliminate the impact of dimensions and numerical ranges between different features. A commonly used normalization method is Min-Max Normalization. In the embodiment of the present application, it is assumed that the damage feature vector obtained is f=[f1,f2,…,f32], where each element fi represents a specific damage-related feature. The minimum-maximum normalization formula can be applied to each feature element to obtain a normalized damage feature vector fnorm.

After the normalized damage features are obtained, step S540 uses these features to identify bridge structure damage on the image blocks. This process, for example, involves a classifier or regressor model that can predict whether the image block has damage and its damage type or degree based on the input normalized damage features.

In practical applications, different models can be selected for damage identification according to the specific requirements of the task (such as the presence or absence of damage, the type of damage, the degree of damage, etc.). For binary classification problems (such as determining whether an image block is damaged), models such as logistic regression, support vector machine (SVM) or convolutional neural network (CNN) classifiers in deep learning can be used. For multi-classification problems (such as determining the specific type of damage), models such as softmax regression or multi-class SVM can be used. For regression problems (such as predicting the degree of damage), models such as linear regression, ridge regression or random forest can be used.

The assumed task is to determine whether there is damage in the image block and identify its damage type (such as cracks, rust, etc.). In this case, a multi-class SVM classifier can be selected as the damage recognition model. The model has learned the mapping relationship between normalized damage features and damage categories during the training phase. In the test phase (i.e., the actual damage recognition process), the model receives the normalized damage features as input and outputs a probability distribution vector, which represents the probability that the image block belongs to different damage categories. Finally, the category with the highest probability is selected as the final damage recognition result.

For example, suppose the probability distribution vector output by the normalized damage feature vector fnorm after being processed by the multi-class SVM classifier is [0.1, 0.8, 0.1], which corresponds to the three categories of no damage, crack, and rust. Based on this probability distribution vector, it can be judged that the image block has the highest probability of crack damage (probability is 0.8), so it is identified as crack damage.

Through the detailed explanation and example of the above steps, we can clearly see how step S500 transforms the merged implicit representation into a specific image block damage recognition result. This process not only involves technical details such as feature extraction and normalization processing of multi-layer perceptrons, but also involves advanced machine learning techniques such as the selection and application of classifier or regressor models.

In an optional implementation, the normalized damage feature includes the support corresponding to y damage categories, y∈(1,+∞). Then, the step S540, performing bridge structure damage identification on the image blocks according to the normalized damage feature to obtain image block damage identification results, may include:

Step S541: Determine the maximum support among the y supports of the normalized damage features.

Step S542: When the maximum support is greater than the reference support, bridge structure damage identification is performed on the image blocks according to the damage category corresponding to the maximum support to obtain image block damage identification results.

Step S543: When the maximum support is not greater than the reference support, the image block is determined as an image block damage identification result.

In step S541, the computer system processes the normalized damage features, which represent the possibility of the image block belonging to different damage categories in the form of support. The support is a value between 0 and 1, which is used to quantify the degree of match between the image block and each damage category.

For example, a large bridge has a complex structure, and common damage types include cracks, rust, and spalling. In order to monitor these damages, a high-definition camera is installed on the bridge, and images are collected regularly for analysis. After the previous steps, the normalized damage features of each image block have been obtained. These features include the support for y damage categories such as cracks, rust, and spalling.

For example, the normalized damage features of an image block may be as follows (assuming y=3, that is, only considering three damage categories: crack, rust, and spalling):

Normalized damage feature = [0.7, 0.2, 0.1], where the first element 0.7 represents the support of the image block belonging to crack damage, the second element 0.2 represents the support of corrosion damage, and the third element 0.1 represents the support of spalling damage. In step S541, the computer system will traverse this feature vector to find the maximum value, that is, the maximum support. In this example, the maximum support is 0.7, and the corresponding damage category is crack.

In step S542, after determining the maximum support, the computer system compares it with a preset reference support. The reference support is a threshold used to determine whether there are obvious signs of damage in the image block. Only when the maximum support exceeds this threshold will the system consider that the image block has damage of the corresponding category. The setting of the reference support is usually based on a large amount of experimental data and expert experience. It should be high enough to ensure the accuracy of recognition, but not too high to avoid missing real damage. In actual applications, the reference support may need to be adjusted according to different bridge types, image quality, environmental conditions and other factors.

Assume that the reference support is set to 0.5. In step S542, the computer system compares the maximum support of 0.7 with the reference support of 0.5. Since 0.7 is greater than 0.5, the system believes that there is obvious crack damage in the image block. Therefore, the system will identify the damage of the image block according to the damage category (i.e., crack) corresponding to the maximum support, and generate the corresponding damage identification result. This result can be presented to the bridge management personnel in the form of a report, indicating the specific location of the image block, the type of damage, and the degree of damage (although the degree of damage is not directly calculated in this step, the system can estimate it based on other information or algorithms).

In step S543, when the maximum support is not greater than the reference support, the image block is determined as the image block damage identification result. If the maximum support does not exceed the reference support, it means that the degree of match between the image block and all preset damage categories is not high enough, that is, there is no obvious sign of damage. In this case, the computer system can mark the image block as "no damage" or "normal" and use it as part of the image block damage identification result. In step S543, the computer system compares the maximum support of 0.3 with the reference support of 0.5, and finds that 0.3 is less than 0.5. Therefore, the system determines the image block as "no damage" or "normal" and generates the corresponding damage identification result. This result will also be reported to bridge managers to help them understand the overall health of the bridge.

In the embodiment of the present application, the training process of the bridge structure damage identification model mentioned above can refer to the following steps:

Step T100: obtaining a damage recognition model for a bridge structure to be trained, a bridge image template, and a damage priori mark corresponding to the bridge image template;

Step T200: in the bridge structure damage recognition model to be trained, extracting implicit representations of images from image blocks in the bridge image template, and obtaining t implicit representations of template images corresponding to the image blocks in the bridge image template, wherein t∈(0,∞); the t implicit representations of the template images are used to represent different types of features, and the image blocks in the bridge image template are obtained by dividing the bridge image template into blocks;

Step T300: merging t implicit representations of the template images to obtain a template merged implicit representation corresponding to the image blocks in the bridge image template, and performing adaptive feature integration on the template merged implicit representation to obtain a template adaptive implicit representation for highlighting and characterizing the template merged implicit representation;

Step T400: The template image block connectivity matrix corresponding to the bridge image template and the template output implicit representation corresponding to the image blocks in the bridge image template are merged step by step to obtain the template merged implicit representation corresponding to the image blocks in the bridge image template; the template output implicit representation is obtained by the template adaptability implicit representation and the template merged implicit representation; the template image block connectivity matrix is used to characterize the similarity between the image blocks in the bridge image template;

Step T500: According to the template, the implicit representation and the damage priori mark are merged to update the parameters of the bridge structure damage identification model to be trained to obtain the bridge structure damage identification model.

In step T100, the computer system first needs to prepare all resources required for training, including a bridge structure damage recognition model to be trained (which can be a neural network model based on deep learning, such as convolutional neural network CNN, graph convolutional network GCN, etc.), a series of bridge image templates (as training samples) and the damage prior labels corresponding to these image templates.

Assume that a hybrid neural network model combining CNN and GCN is used. The CNN part is used to extract visual features of image blocks, while the GCN part is used to process the spatial relationship between image blocks. A large number of high-definition images of different bridge structures (such as suspension bridges, cable-stayed bridges, arch bridges, etc.) are collected. These images cover the appearance of bridges under different lighting conditions and different viewing angles.

Damage prior marking is done by bridge engineers or expert teams to annotate each bridge image template, clearly indicating which areas in the image have damage (such as cracks, rust, spalling, etc.) and the type of damage. These markings are usually in the form of pixel level or region level, such as through mask images or bounding boxes to represent the damaged area.

In step T200, the computer system uses the feature extraction module (such as the first few layers of CNN) in the bridge structure damage recognition model to be trained to process each bridge image template into blocks, and extracts the image implicit representation for each image block. These implicit representations contain the feature information of the image block in different dimensions.

Each bridge image template is divided into multiple image blocks of fixed size (such as 64×64 pixels) or adaptively resized according to the content. For each image block, CNN is used to extract multiple visual features such as texture, color, and shape. For example, multiple convolutional layers are used to gradually reduce the spatial resolution of the image while increasing the number of channels of the feature map to capture more abstract and advanced feature representations. After each image block is processed by CNN, a high-dimensional feature vector (such as 128 or 256 dimensions) is obtained, which is the implicit representation of the image block.

In step T300, the computer system merges the implicit representations of multiple template images extracted in step T200, and strengthens key features through adaptive feature integration technology (such as self-attention mechanism), thereby obtaining template merged implicit representation and template adaptive implicit representation.

The implicit representations of all image blocks in the same image template are concatenated or weighted averaged to generate a merged implicit representation of the image template. The self-attention mechanism is used to weight different features in the merged implicit representation. Specifically, the request feature, index feature, and output feature are first generated through linear transformation, and then the similarity matrix between the request feature and the index feature is calculated and normalized through the softmax function to obtain the attention weight. Finally, these attention weights are used to weight the output features to obtain the adaptive implicit representation.

In step T400, the computer system uses the image block connectivity matrix (characterizing the spatial relationship between image blocks) and the template output implicit representation (obtained through the template adaptability implicit representation and the template merging implicit representation) to extract and fuse contextual features by merging sub-modules step by step (such as the convolutional network GCN in the figure).

Construct a two-dimensional array as the image block connectivity matrix, in which the element values represent the connectivity (or similarity) between image blocks. For example, the connectivity value between adjacent image blocks is high, while the connectivity value between non-adjacent blocks is low. Pass the template output implicit representation and the image block connectivity matrix as input to the GCN model. GCN gradually fuses the information between image blocks through multi-layer graph convolution operations. In each layer, GCN updates the representation of each node by aggregating the information of neighboring nodes according to the graph structure defined by the connectivity matrix and the node features of the current layer (that is, the implicit representation of the image block). After multi-layer GCN processing, the template merged implicit representation that integrates the context information is finally obtained.

In step T500, the computer system uses the template to merge implicit representations as feature inputs, combines damage prior markers to calculate loss functions (such as cross entropy loss, mean square error loss, etc.), and updates and optimizes model parameters through a back-propagation algorithm.

Assume that the cross entropy loss function is used to measure the difference between the model prediction and the true damage mark. For each image block (or its corresponding region) in each image template, the cross entropy loss between the predicted damage probability and the true damage mark is calculated.

According to the gradient information of the loss function, the error is back-propagated from the output layer to the input layer layer by layer through the back-propagation algorithm, and the model parameters are updated according to the gradient descent method or other optimization algorithms. This process is repeated multiple times (i.e. multiple training cycles) until the loss function value converges to a stable low value or reaches the preset training round.

Through repeated iteration and optimization of the above steps, the computer system can train an accurate and robust bridge structure damage identification model. The model can automatically extract key feature information from the input bridge image and integrate the contextual relationship to achieve accurate identification of bridge structure damage. In practical applications, this model can be deployed in the bridge monitoring system to monitor the health status of the bridge structure in real time and provide strong support for the maintenance and management of the bridge.

An embodiment of the present application provides a computer system, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

Figure 2 is a schematic diagram of the hardware entity of a computer system provided in an embodiment of the present application. As shown in Figure 2, the hardware entity of the computer system 1000 includes: a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can be run on the processor 1001, and the processor 1001 implements the steps in the method of any of the above embodiments when executing the program.

The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims (9)

1. The bridge structure damage identification method based on machine vision is characterized by comprising the following steps of:

Collecting a machine vision image of a bridge structure to be identified;

Extracting implicit image representations of image blocks in the machine vision image to obtain t implicit image representations corresponding to the image blocks, wherein t is E (0, + & gt); t images are implicitly expressed to express different types of characteristics, and the image blocks are obtained by dividing the machine vision image into blocks;

combining t implicit image representations to obtain a first combined implicit representation corresponding to the image block, and performing adaptive feature integration on the first combined implicit representation to obtain an adaptive implicit representation for highlighting the first combined implicit representation;

Step-by-step combination is carried out on the image block connectivity matrix corresponding to the machine vision image and the output implicit expression corresponding to the image block, so as to obtain a second combined implicit expression corresponding to the image block; the output implicit representation is obtained through the adaptive implicit representation and the first combined implicit representation, and the image block connectivity matrix is used for representing the similarity between the image blocks;

performing bridge structure damage identification on the image blocks according to the second combined implicit expression to obtain image block damage identification results, and obtaining damage identification results of the bridge structure to be identified according to the image block damage identification results of each image block;

The step-by-step combination of the image block connectivity matrix corresponding to the machine vision image and the output implicit expression corresponding to the image block to obtain a second combined implicit expression corresponding to the image block, which comprises the following steps:

the step-by-step merging module is used for implicitly representing the image block connectivity matrix corresponding to the machine vision image and the output corresponding to the image block to be transmitted into the bridge structure damage identification model, and comprises a feature extraction sub-module, v step-by-step merging sub-modules and merging sub-modules, wherein v is E [1, + ];

In the feature extraction submodule, carrying out feature extraction on the output implicit expression corresponding to the image block to obtain feature extraction implicit expression corresponding to the image block;

extracting implicit expressions from v progressive merging sub-modules based on an image block connectivity matrix corresponding to the machine vision image and features corresponding to the image blocks, obtaining hierarchical implicit expressions corresponding to v progressive merging sub-modules respectively, and determining the hierarchical implicit expressions corresponding to the target progressive merging sub-modules in the v progressive merging sub-modules into progressive merging implicit expressions corresponding to the image blocks; the target progressive merging sub-module is the last progressive merging sub-module in the v progressive merging sub-modules;

and in the merging sub-module, merging the step-by-step merged implicit representation corresponding to the image block and the feature extraction implicit representation corresponding to the image block to obtain a second merged implicit representation corresponding to the image block.

2. The method of claim 1, wherein t of the implicit representations of images comprise implicit representations of textures;

The extracting the implicit image representation of the image blocks in the machine vision image to obtain t implicit image representations corresponding to the image blocks comprises the following steps:

the machine vision image is transmitted into an image implicit representation extraction module of the bridge structure damage identification model;

in the image implicit expression extraction module, w texture units of image blocks in the machine vision image are obtained, wherein w is epsilon [1, + ] in the image implicit expression extraction module; w texture units are used for representing textures of the image blocks;

acquiring implicit expression representations of texture units corresponding to w texture units respectively, and performing feature extraction on the implicit expression representations of the w texture units respectively to obtain implicit expression representations of texture extraction corresponding to the implicit expression representations of the w texture units respectively;

Feature dimension reduction is carried out on w implicit texture extraction representations to obtain implicit texture representations corresponding to the image blocks;

t said implicit representations of images also include implicit representations of shapes;

The extracting the implicit image representation of the image blocks in the machine vision image to obtain t implicit image representations corresponding to the image blocks comprises the following steps:

the machine vision image is transmitted into an image implicit representation extraction module of the bridge structure damage identification model;

in the image implicit expression extraction module, r area shapes of image blocks in the machine vision image are obtained, wherein r is E [1, + ]; r said region shapes are used to represent image tiles comprising different shapes;

Respectively extracting features of r region shapes to obtain graphic implicit representations corresponding to the r region shapes;

performing feature dimension reduction on r implicit graphic representations to obtain implicit shape representations corresponding to the image blocks;

t said implicit representations of images also include implicit representations of colors;

The extracting the implicit image representation of the image blocks in the machine vision image to obtain t implicit image representations corresponding to the image blocks comprises the following steps:

the machine vision image is transmitted into an image implicit representation extraction module of the bridge structure damage identification model;

In the image implicit representation extraction module, partitioning the machine vision image to obtain g image partitions in the machine vision image, wherein g is [1 ] + ]; g image blocks are used for being fused into the machine vision image, and g image blocks are image blocks of different color blocks in the machine vision image;

acquiring color implicit representations corresponding to target image blocks from color implicit representations corresponding to g image blocks respectively, and determining the color implicit representations corresponding to the target image blocks as the color implicit representations corresponding to the image blocks in the machine vision image;

t said implicit representations of images also include semantic implicit representations;

The extracting the implicit image representation of the image blocks in the machine vision image to obtain t implicit image representations corresponding to the image blocks comprises the following steps:

the machine vision image is transmitted into an image implicit representation extraction module of the bridge structure damage identification model;

In the image implicit expression extraction module, image semantic expression is carried out on the image blocks in the machine vision image, and initial semantic implicit expression corresponding to the image blocks is obtained;

Determining image coordinates of the image blocks in the machine vision image, and carrying out coordinate information coding on the image coordinates of the image blocks to obtain implicit representation of the coordinates corresponding to the image blocks;

And obtaining a positioning vector corresponding to the image block, and combining the initial semantic implicit expression corresponding to the image block, the coordinate implicit expression corresponding to the image block and the positioning vector corresponding to the image block to obtain the semantic implicit expression corresponding to the image block.

3. The method according to claim 1, wherein said merging said t implicit representations of said image to obtain a first merged implicit representation corresponding to said image block, and said adaptively feature integrating said first merged implicit representation to obtain an adaptive implicit representation used to highlight said first merged implicit representation, comprises:

The implicit representation merging module is used for transmitting t implicit representations of the images into the bridge structure damage identification model;

in the implicit representation merging module, merging t implicit representations of the images to obtain a first merged implicit representation corresponding to the image block;

weighting the first combined implicit representation by a weight matrix to obtain h adaptive execution characteristics corresponding to the first combined implicit representation, wherein h is E [1, + ];

And carrying out adaptive weighting operation on the h adaptive execution characteristics to obtain an adaptive implicit representation used for highlighting the first combined implicit representation.

4. The method of claim 3, wherein h of the adaptive execution features include an index feature, an output feature, and a request feature;

performing adaptive weighting operation on the h adaptive execution features to obtain an adaptive implicit representation for highlighting the first combined implicit representation, including:

obtaining an inverse feature corresponding to the index feature, and performing inner product calculation on the request feature and the inverse feature to obtain an inner product matrix, wherein the inner product matrix is used for representing similarity among t implicit representations of the images;

normalizing the inner product matrix to obtain a normalized result corresponding to the inner product matrix, and performing inner product calculation on the normalized result and the output characteristics to obtain an adaptive merging result for merging and representing t implicit representations of the images;

and carrying out multi-layer perceptron processing on the adaptive merging result to obtain the adaptive implicit representation for highlighting the first merged implicit representation.

5. The method of claim 1, wherein v of said progressive merging sub-modules comprises a progressive merging sub-module C _b, wherein b represents a sequence number of the progressive merging sub-module, and b is not greater than v; extracting implicit expressions from v progressive merging sub-modules based on an image block connectivity matrix corresponding to the machine vision image and features corresponding to the image blocks, and obtaining hierarchical implicit expressions respectively corresponding to the v progressive merging sub-modules, wherein the method comprises the following steps:

When the progressive merging sub-module C _b is the first progressive merging sub-module of v progressive merging sub-modules, performing, by using the progressive merging sub-module C _b, contextual feature extraction on an image block connectivity matrix corresponding to the machine vision image and feature extraction implicit representation corresponding to the image block, so as to obtain a hierarchical implicit representation corresponding to the progressive merging sub-module C _b;

When the progressive merging sub-module C _b is not the first progressive merging sub-module of the v progressive merging sub-modules, performing, by using the progressive merging sub-module C _b, extracting context features from an image block connectivity matrix corresponding to the machine vision image and a hierarchical implicit representation corresponding to the progressive merging sub-module C _a, so as to obtain a hierarchical implicit representation corresponding to the progressive merging sub-module C _b; the progressive merging sub-module C _a is the last progressive merging sub-module of the progressive merging sub-module C _b.

6. The method of claim 5, wherein the number of image tiles is m, the m image tiles comprising target image tiles, m e (1, + -infinity); the image block connectivity matrix corresponding to the machine vision image comprises connectivity parameters of the target image blocks for m image blocks respectively; the feature extraction implicit representation corresponding to the image block comprises the feature extraction implicit representation corresponding to the target image block;

The step-by-step merging sub-module C _b performs context feature extraction on the image block connectivity matrix corresponding to the machine vision image and the feature extraction implicit expression corresponding to the image block, to obtain a step-by-step implicit expression corresponding to the step-by-step merging sub-module C _b, including:

Acquiring connectivity characteristics of the target image blocks respectively aiming at m image blocks in the step-by-step merging sub-module C _b;

in the step-by-step merging sub-module C _b, respectively merging the implicit representation extracted from the features corresponding to the target image blocks and m connectivity features to obtain m merged connectivity features respectively corresponding to the image blocks;

weighting and summing the m combined connectivity characteristics according to the m connectivity parameters to obtain image association characteristics corresponding to the target image blocks;

Activating the image association features corresponding to the target image blocks to obtain the hierarchical implicit representation corresponding to the target image blocks, wherein the hierarchical implicit representation corresponding to the target image blocks belongs to the hierarchical implicit representation corresponding to the step-by-step merging sub-module C _b.

7. The method according to claim 1, wherein the performing bridge structure damage recognition on the image blocks according to the second combined implicit representation to obtain an image block damage recognition result includes:

The second combined implicit representation is transmitted into a damage recognition module of the bridge structure damage recognition model;

In the damage identification module, performing multi-layer perceptron processing on the second combined implicit representation to obtain damage characteristics corresponding to the image blocks;

normalizing the damage characteristics to obtain normalized damage characteristics corresponding to the damage characteristics;

And carrying out bridge structure damage identification on the image blocks according to the normalized damage characteristics to obtain an image block damage identification result.

8. The method of claim 7, wherein the step of determining the position of the probe is performed, the normalized lesion characterization includes a respective corresponding support for y lesion categories, y is an element of (1, +++). The bridge structure damage identification is carried out on the image blocks according to the normalized damage characteristics to obtain an image block damage identification result, and the method comprises the following steps:

Determining the maximum support degree in y support degrees of the normalized damage characteristics;

When the maximum support degree is greater than the reference support degree, carrying out bridge structure damage identification on the image blocks according to the damage category corresponding to the maximum support degree to obtain an image block damage identification result;

And when the maximum support degree is not greater than the reference support degree, determining the image block as an image block damage identification result.

9. A computer system comprising a memory and a processor, the memory storing a computer program executable on the processor, wherein the processor implements the steps of the method of any one of claims 1 to 8 when the program is executed.