Disclosure of Invention
The embodiment of the invention provides a predictive maintenance management method and system for an industrial robot, which at least can solve part of problems in the prior art.
In a first aspect of an embodiment of the present invention, there is provided an industrial robot predictive maintenance management method, including:
Acquiring multi-modal heterogeneous data of an industrial robot in real time through a preset sensing network, performing intelligent preprocessing on the multi-modal heterogeneous data, combining blockchain right-determining synchronization and generating an countermeasure network enhancement equalization operation to obtain first preprocessed data, performing cross-modal correlation fusion on the first preprocessed data through a multi-modal graph convolutional neural network to generate fusion data, extracting complementary redundant information among the fusion data, and constructing a semantic feature map corresponding to the industrial robot;
Constructing a health state prediction model corresponding to an industrial robot comprising a plurality of subsystems based on a converter model, adding the semantic feature map as input into the health state prediction model, mapping the semantic feature map to an embedded space by an input embedding layer through linear transformation, extracting time sequence information, combining self-attention layers to perform feature interaction to obtain interaction features, and performing transformation and feature enhancement on the interaction features through a feedforward neural network layer to obtain health state probability distribution of each subsystem;
Based on the health state probability distribution, generating comprehensive health state distribution through Bayesian inference fusion, combining an abnormality detection model based on active learning, combining active exploration and uncertainty sampling to identify an abnormality mode and key turning points in the comprehensive health state distribution, generating an optimal maintenance strategy based on health state influence information and combining a reinforcement learning algorithm, carrying out collaborative maintenance on the industrial robot through a game theory method, and uploading a maintenance result to a server in real time and storing the current state of the industrial robot.
In an alternative embodiment of the present invention,
Acquiring multi-mode heterogeneous data of the industrial robot in real time through a preset sensing network, performing intelligent preprocessing on the multi-mode heterogeneous data, combining blockchain acknowledgement synchronization and generating countermeasure network enhancement equalization operation, and obtaining first preprocessed data comprises the following steps:
A plurality of sensor composition sensing networks of types are deployed in advance at key parts and joints of an industrial robot, and multi-modal heterogeneous data of the industrial robot are obtained in real time through the sensing networks, wherein the multi-modal heterogeneous data comprise acceleration, vibration, temperature, current and position information;
Carrying out intelligent preprocessing on the acquired multi-mode heterogeneous data, carrying out noise reduction and outlier rejection on the original data through wavelet transformation operation, mapping heterogeneous data with different sampling rates and time stamps onto the same time scale by combining a dynamic time warping algorithm to obtain aligned data, and carrying out normalization processing on the aligned data to generate standard data;
And carrying out data enhancement on the standard data through a depth generation countermeasure network, expanding the scale of the standard data, balancing data category distribution by combining an interpolation synthesis algorithm, carrying out data encryption through an asymmetric encryption algorithm, and carrying out synchronization and trusted sharing at multiple ends based on a distributed consensus mechanism of a blockchain to obtain the first preprocessing data.
In an alternative embodiment of the present invention,
Performing cross-modal correlation fusion on the first preprocessing data through a multi-modal graph convolutional neural network to generate fusion data, and extracting complementary redundant information among the fusion data and constructing a semantic feature map corresponding to the industrial robot comprises the following steps:
Mapping the first preprocessing data into a pre-initialized heterogeneous graph structure, wherein each sensor node is used as a node of the graph, the node attribute is a multidimensional time sequence feature vector of a current sensor, the edge is the association strength between the sensors, the edge weight is the correlation and the dependency relationship between the nodes, and the node feature and the edge weight matrix are initialized;
Applying a multi-modal graph convolutional neural network to the heterogeneous graph structure, respectively extracting local features in each mode through a cross-modal convolutional kernel and a convolutional kernel function corresponding to each mode, distributing importance weights for the local features in each mode by combining an attention mechanism, performing cross-modal correlation fusion on convolution results of a plurality of modes according to the importance weights to generate joint semantic representation of each node, stacking a plurality of multi-modal graph convolutional neural network layers and adaptively aggregating node features to generate cross-modal features;
For the cross-modal features, calculating a correlation matrix between different modes based on pearson correlation coefficients, carrying out thresholding treatment on the correlation matrix and generating a sparse correlation matrix, carrying out community division on the sparse correlation matrix by combining a graph clustering algorithm, clustering the cross-modal features into clusters, wherein features in each cluster represent a group of complementary redundant semantic information, carrying out importance analysis on the features in each cluster, identifying key complementary redundant features by combining a feature selection method based on a graph, and adding the key complementary redundant features into a pre-initialized redundant set;
And constructing a semantic feature map, filling an instance layer of the semantic feature map by using the key complementary redundant features based on the redundant set, definitely defining type layers and attribute values, introducing a time dimension, and establishing semantic association among instances to obtain the semantic feature map.
In an alternative embodiment of the present invention,
Constructing a health state prediction model corresponding to an industrial robot comprising a plurality of subsystems based on a converter model, adding the semantic feature map as input into the health state prediction model, mapping the semantic feature map to an embedded space by an input embedding layer through linear transformation, extracting time sequence information, combining self-attention layers to perform feature interaction to obtain interaction features, transforming and feature enhancing the interaction features through a feedforward neural network layer, and obtaining health state probability distribution of each subsystem comprises the following steps:
Based on a converter model, constructing an encoder-decoder structure, adding a semantic feature map corresponding to an industrial robot into the converter model, mapping nodes in the semantic feature map into low-dimensional dense vectors through a node embedding technology, reserving a graph structure, designing the structure of the encoder and the decoder, introducing a position coding mechanism, adding a position embedding vector into an input embedding layer, and obtaining a health state prediction model corresponding to each subsystem;
Adding the semantic feature map as input into the health state prediction model, mapping nodes in the map into a low-dimensional embedding space through linear transformation by an input embedding layer, aggregating neighbor node information by combining a map attention network, updating embedded representation of each node, adding corresponding time stamp information in each node embedding, constructing a time sequence embedding matrix, determining time sequence information corresponding to the current node based on the time sequence embedding matrix, encoding the semantic feature map embedded with the time sequence information, dividing the map into a plurality of sub-graphs, wherein each sub-graph corresponds to a subsystem of an industrial robot, and extracting features and interacting features of each sub-graph based on an attention mechanism to obtain the interactive features;
based on the interaction characteristics, extracting association modes among different subsystems, adding the interaction characteristics into the health state prediction model, recursively generating the health state of the next time step, combining the association modes, combining an attention mechanism to dynamically adjust the influence degree of the different subsystems on the current prediction, and repeating the prediction until the preset maximum iteration times are reached, so as to obtain the health state probability distribution corresponding to each subsystem.
In an alternative embodiment of the present invention,
The embedded representation of aggregating neighbor node information and updating each node in conjunction with the graph attention network is shown in the following formula:
;
Wherein, Represents the updated hidden state of node i, h i represents the hidden state of node i, K represents the number of attention headers, sigma represents the activation function,Indicating that the kth attention header is at time step t, the attention weight of node i to neighbor node j, W k indicates the weight matrix of the kth attention header, h j indicates the hidden state of node j,Representing the attention weight of the kth attention head to itself by node i at time step t, N (i) represents the set of neighbor nodes of node i.
In an alternative embodiment of the present invention,
Based on the health state probability distribution, generating comprehensive health state distribution through Bayesian inference fusion, combining an abnormality detection model based on active learning, combining active exploration and uncertainty sampling to identify an abnormality mode and key turning points in the comprehensive health state distribution, generating an optimal maintenance strategy based on health state influence information and combining a reinforcement learning algorithm to cooperatively maintain the industrial robot through a game theory method, and uploading a maintenance result to a server in real time and storing the current state of the industrial robot comprises the following steps:
the health state probability distribution of each subsystem is used as an independent random variable, a joint probability distribution model is established, priori knowledge is introduced to construct priori distribution probability, posterior probability distribution corresponding to each subsystem is calculated based on Bayesian theorem by combining the prior distribution probability and health state likelihood functions of the subsystems, and the posterior probability distribution is used as the comprehensive health state distribution;
Based on the comprehensive health state distribution, an abnormal detection model based on active learning is constructed, the comprehensive health state distribution is modeled through an unsupervised learning method, a sample with the maximum information amount is selected for marking according to an active exploration strategy, the uncertainty of a current sample is evaluated through edge probability, the sample with the highest uncertainty is repeatedly evaluated and selected, and a manual marking label is added;
Updating the abnormal detection model according to the re-labeled sample, adaptively adjusting decision boundaries and model super-parameters of the abnormal detection model by combining an incremental learning algorithm, detecting the comprehensive health state distribution according to the updated abnormal detection model, and determining drift trend and the key turning points of the current health state distribution by combining a drift detection mechanism;
the method comprises the steps of obtaining fault risks, task importance, maintenance cost and residual service life corresponding to industrial robots, recording the fault risks, task importance, maintenance cost and residual service life as health state influence information, taking the health state influence information as a state space, taking maintenance operation as an action space, defining state transition probability and rewarding functions, updating function values between the state and the action through a reinforcement learning algorithm, generating an optimal maintenance strategy, determining a maintenance rewarding mechanism and a cooperation strategy between each industrial robot by combining a game theory method, repeatedly optimizing, obtaining a global optimal maintenance decision, carrying out cooperative maintenance on the industrial robots, and uploading the state parameters of the industrial robots after maintenance to a server and storing.
In an alternative embodiment of the present invention,
Based on the Bayes theorem, the posterior probability distribution corresponding to each subsystem is calculated by combining the prior probability and the health state likelihood function of each subsystem, and the posterior probability distribution is shown as the following formula:
;
Wherein S g represents the health state of the g-th subsystem, O represents the comprehensive health state evidence observed, P (o|s g) represents the likelihood that the comprehensive health state evidence O is observed under the condition that the g-th subsystem is in the health state S g, P (S g |o) represents the posterior probability that the g-th subsystem is in the health state S g under the condition that the comprehensive health state evidence O, P (S g) represents the prior probability that the g-th subsystem is in the health state S g, P (o|s j) represents the likelihood that the comprehensive health state evidence O is observed under the condition that the j-th subsystem is in the health state S j, P (S j) represents the prior probability that the j-th subsystem is in the health state S j, and n represents the total number of subsystem states.
In a second aspect of the embodiment of the present invention, there is provided an industrial robot predictive maintenance management system, including:
The system comprises a first unit, a second unit and a third unit, wherein the first unit is used for acquiring multi-modal heterogeneous data of the industrial robot in real time through a preset sensing network, performing intelligent preprocessing on the multi-modal heterogeneous data, combining blockchain right-determining synchronization and generating an countermeasure network enhancement equalization operation to obtain first preprocessed data, performing cross-modal correlation fusion on the first preprocessed data through a multi-modal graph convolutional neural network to generate fusion data, extracting complementary redundant information among the fusion data and constructing a semantic feature map corresponding to the industrial robot;
The second unit is used for constructing a health state prediction model corresponding to the industrial robot comprising a plurality of subsystems based on the converter model, adding the semantic feature map into the health state prediction model as input, mapping the semantic feature map into an embedded space by an input embedding layer through linear transformation, extracting time sequence information, combining a self-attention layer to perform feature interaction to obtain interaction features, and performing transformation and feature enhancement on the interaction features through a feedforward neural network layer to obtain health state probability distribution of each subsystem;
The third unit is used for generating comprehensive health state distribution through Bayesian inference fusion based on the health state probability distribution, identifying an abnormal mode and key turning points in the comprehensive health state distribution by combining an abnormal detection model based on active learning and combining active exploration and uncertainty sampling, generating an optimal maintenance strategy by combining a reinforcement learning algorithm based on health state influence information, carrying out cooperative maintenance on the industrial robot through a game theory method, uploading a maintenance result to a server in real time, and storing the current state of the industrial robot.
In a third aspect of an embodiment of the present invention,
There is provided an electronic device including:
A processor;
A memory for storing processor-executable instructions;
wherein the processor is configured to invoke the instructions stored in the memory to perform the method described previously.
In a fourth aspect of an embodiment of the present invention,
There is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method as described above.
According to the invention, the data enhancement and equalization operation is carried out by adopting the generation countermeasure network, the problems of data sparseness and unbalance are effectively solved, a good foundation is laid for subsequent analysis and modeling, the multimodal graph convolutional neural network is utilized to carry out cross-modal association fusion on the preprocessed data, complementary and redundant information among different modality data is fully mined, high-level semantic association among modalities can be effectively captured, more comprehensive and compact fusion data representation is generated, the fusion is carried out by adopting a Bayesian inference method, comprehensive health state distribution is generated, priori knowledge and observation data can be effectively combined, more reliable and comprehensive health state estimation is obtained, reinforcement learning can be realized through continuous interaction with the environment, and a decision sequence with maximized long-term benefit is learned, meanwhile, the maintenance cost is remarkably reduced, the availability of the robot is improved, in a comprehensive way, the health management level of the industrial robot is remarkably improved, the service life of the industrial robot is prolonged to the maximum extent, the external shutdown time is reduced, the production efficiency is improved, and the operation cost is reduced.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The technical scheme of the invention is described in detail below by specific examples. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.
Fig. 1 is a flow chart of a predictive maintenance management method for an industrial robot according to an embodiment of the invention, as shown in fig. 1, the method includes:
S1, acquiring multi-modal heterogeneous data of an industrial robot in real time through a preset sensing network, performing intelligent preprocessing on the multi-modal heterogeneous data, combining blockchain right-determining synchronization and generating an countermeasure network enhancement equalization operation to obtain first preprocessed data, performing cross-modal correlation fusion on the first preprocessed data through a multi-modal graph convolutional neural network to generate fusion data, extracting complementary redundant information among the fusion data, and constructing a semantic feature map corresponding to the industrial robot;
The multi-modal heterogeneous data is data with different formats and characteristics from different sources, the blockchain right synchronization is that the blockchain technology is utilized to manage and synchronize ownership and access rights of the data, the generation of the countermeasure network enhancement balanced operation is a data enhancement technology, the generation of the countermeasure network is used to generate synthesized data, the training data set is expanded, the generalization capability of a model is improved, the multi-modal graph convolutional neural network is a neural network model for processing the multi-modal data, the data of different modalities is mapped to the same graph structure, the graph rolling operation is utilized to extract cross-modal characteristic representation, the cross-modal correlation fusion is that the correlation relationship among different modalities is mined in the multi-modal data, the cross-modal information is fused, the complementary redundant information is that the information carried by different modalities may have certain complementarity and redundancy in the multi-modal data, the semantic characteristic graph is a structural knowledge graph representation method, and a semantic graph is formed through the association among concepts of building equipment parts, fault modes, maintenance operations and the like.
In an alternative embodiment of the present invention,
Acquiring multi-mode heterogeneous data of the industrial robot in real time through a preset sensing network, performing intelligent preprocessing on the multi-mode heterogeneous data, combining blockchain acknowledgement synchronization and generating countermeasure network enhancement equalization operation, and obtaining first preprocessed data comprises the following steps:
A plurality of sensor composition sensing networks of types are deployed in advance at key parts and joints of an industrial robot, and multi-modal heterogeneous data of the industrial robot are obtained in real time through the sensing networks, wherein the multi-modal heterogeneous data comprise acceleration, vibration, temperature, current and position information;
Carrying out intelligent preprocessing on the acquired multi-mode heterogeneous data, carrying out noise reduction and outlier rejection on the original data through wavelet transformation operation, mapping heterogeneous data with different sampling rates and time stamps onto the same time scale by combining a dynamic time warping algorithm to obtain aligned data, and carrying out normalization processing on the aligned data to generate standard data;
And carrying out data enhancement on the standard data through a depth generation countermeasure network, expanding the scale of the standard data, balancing data category distribution by combining an interpolation synthesis algorithm, carrying out data encryption through an asymmetric encryption algorithm, and carrying out synchronization and trusted sharing at multiple ends based on a distributed consensus mechanism of a blockchain to obtain the first preprocessing data.
The sensing network is a distributed network system composed of a large number of sensor nodes and is used for collecting, processing and transmitting information of the physical world, the wavelet transformation operation is a time-frequency analysis tool, time domain signals can be converted into time-frequency domain representations, local characteristics of the signals are extracted, the dynamic time warping algorithm is a time sequence alignment method and is used for processing time sequence data under different time scales or speeds, the interpolation synthesis algorithm is a data filling and reconstruction method and is used for processing missing or sparse data, the asymmetric encryption algorithm is an encryption method and is used for conducting encryption and decryption operations by using different keys and is used for protecting confidentiality of the data, and the distributed consensus mechanism based on block chains is a method for achieving consensus in the distributed system and is realized through a block chain technology.
The method comprises the steps that multiple types of sensors, such as an acceleration sensor, a vibration sensor, a temperature sensor, a current sensor and a position sensor, are deployed in advance at key parts and joints of the industrial robot, such as a speed reducer, a motor and a bearing, and multi-mode heterogeneous data of the industrial robot are obtained in real time through a sensing network, wherein the multi-mode heterogeneous data comprise acceleration, vibration, temperature, current and position information;
Performing intelligent preprocessing on the acquired multi-modal heterogeneous data, performing noise reduction and outlier rejection on the original data by utilizing wavelet transformation operation, mapping heterogeneous data with different sampling rates and time stamps onto the same time scale by adopting a dynamic time warping algorithm, performing normalization processing on the aligned data, scaling the data of different modalities into the same numerical range, and eliminating dimension influence;
The method comprises the steps of carrying out data enhancement on standard data by utilizing a depth generation countermeasure network (GAN), expanding the scale of the standard data, constructing a GAN model, comprising a generator and a discriminator, continuously improving the capability of the generator for generating vivid data through countermeasure training of the generator and the discriminator, expanding the scale of a standard data set, carrying out class balance on the enhanced data by adopting an interpolation synthesis algorithm, generating new synthesized samples in a feature space by adopting the interpolation algorithm for fault classes with fewer samples, increasing the data quantity of sparse classes, reducing the number of the samples in a random undersampling mode for normal classes with more samples, and balancing the data distribution of each class;
Encrypting the balanced data by using an asymmetric encryption algorithm, protecting confidentiality and integrity of the data, generating a public key and a private key pair, using the public key for encrypting the data, using the private key for decrypting the data, encrypting the data, converting plaintext data into ciphertext, setting up a blockchain network comprising a plurality of participating nodes such as an industrial robot manufacturer, an equipment owner, a maintenance service provider and the like, designing and deploying an intelligent contract, prescribing rules and rights of data sharing such as data ownership, access control and a use range, adopting a proper consensus algorithm such as workload certification (PoW), rights and interests certification (PoS) and the like, achieving data synchronization and consistency consensus among the plurality of nodes, uploading the encrypted data to the blockchain network, and performing data synchronization among the nodes by a consensus mechanism to obtain first preprocessing data.
In the embodiment, the multi-type sensor forming sensing network is deployed at key parts and joints of the industrial robot, multi-mode heterogeneous data comprising acceleration, vibration, temperature, current, position information and the like are collected in real time, comprehensive sensing and monitoring of the running state of the robot are achieved, wavelet transformation is adopted to reduce noise and abnormal values of original data, high-frequency noise and abnormal points in the data are eliminated, the signal-to-noise ratio and reliability of the data are improved, depth generation is utilized to enhance standard data by the countermeasure network, the scale and diversity of the standard data are expanded, data equalization is achieved through sparse type oversampling and majority type undersampling, accuracy and reliability of fault diagnosis are improved, based on a distributed consensus mechanism of a block chain, data synchronization and credible sharing among a plurality of nodes are achieved, in conclusion, the embodiment achieves comprehensive sensing, quality improvement, enhancement expansion, safe sharing and intelligent application of the running data of the industrial robot, provides key data support for predictive maintenance management of the industrial robot, improves timeliness, accuracy and efficiency of maintenance, and reduces unexpected risk.
In an alternative embodiment of the present invention,
Performing cross-modal correlation fusion on the first preprocessing data through a multi-modal graph convolutional neural network to generate fusion data, and extracting complementary redundant information among the fusion data and constructing a semantic feature map corresponding to the industrial robot comprises the following steps:
Mapping the first preprocessing data into a pre-initialized heterogeneous graph structure, wherein each sensor node is used as a node of the graph, the node attribute is a multidimensional time sequence feature vector of a current sensor, the edge is the association strength between the sensors, the edge weight is the correlation and the dependency relationship between the nodes, and the node feature and the edge weight matrix are initialized;
Applying a multi-modal graph convolutional neural network to the heterogeneous graph structure, respectively extracting local features in each mode through a cross-modal convolutional kernel and a convolutional kernel function corresponding to each mode, distributing importance weights for the local features in each mode by combining an attention mechanism, performing cross-modal correlation fusion on convolution results of a plurality of modes according to the importance weights to generate joint semantic representation of each node, stacking a plurality of multi-modal graph convolutional neural network layers and adaptively aggregating node features to generate cross-modal features;
For the cross-modal features, calculating a correlation matrix between different modes based on pearson correlation coefficients, carrying out thresholding treatment on the correlation matrix and generating a sparse correlation matrix, carrying out community division on the sparse correlation matrix by combining a graph clustering algorithm, clustering the cross-modal features into clusters, wherein features in each cluster represent a group of complementary redundant semantic information, carrying out importance analysis on the features in each cluster, identifying key complementary redundant features by combining a feature selection method based on a graph, and adding the key complementary redundant features into a pre-initialized redundant set;
And constructing a semantic feature map, filling an instance layer of the semantic feature map by using the key complementary redundant features based on the redundant set, definitely defining type layers and attribute values, introducing a time dimension, and establishing semantic association among instances to obtain the semantic feature map.
The multi-dimensional time sequence feature vector refers to a series of time sequence features extracted from multi-modal heterogeneous data, each time step corresponds to one multi-dimensional feature vector, the edge weight matrix refers to a matrix representing connection relation strength among nodes in graph structure data, the cross-modal convolution kernel is a convolution kernel function used for multi-modal data fusion, feature data from different modalities can be processed simultaneously, the convolution kernel function is core operation in a convolution neural network and is used for extracting local features, the sparse correlation matrix is a sparse matrix representing correlation among variables, most elements are zero, only few elements represent significant correlation, the key complementary redundancy feature refers to complementarity and redundancy existing among the different modal features in the multi-modal data, the instance layer refers to a hierarchy representing specific examples in knowledge, and community division refers to a process of dividing the nodes into different communities or modules in a complex network.
Mapping the first preprocessing data into a pre-initialized heterogeneous graph structure, wherein each sensor node is used as one node of the graph, the attribute of the node is a multidimensional time sequence feature vector of the current sensor, the multidimensional time sequence feature vector comprises feature data acquired by the sensor at different time steps, edges in the graph represent the association strength between the sensors, the weight of the edges represents the correlation and the dependency relationship between the nodes, and a node feature matrix and an edge weight matrix are initialized;
Applying a multi-modal graph convolutional neural network on a heterogeneous graph structure, carrying out feature extraction and fusion on multi-modal data, designing a corresponding convolutional kernel function for each mode, extracting local features in the mode, distributing importance weights for the local features through an attention mechanism in a convolutional result of each mode, carrying out weighted fusion on the convolutional result of different modes based on the attention weights, generating joint semantic representation of each node, realizing effective integration of cross-mode information, stacking a plurality of multi-modal graph convolutional neural network layers, extracting and fusing advanced semantic features layer by layer, generating final cross-mode features by self-adaptively aggregating the feature representation of each node, and capturing deep association and complementary information among different modes;
Calculating the correlation among different modal features based on pearson correlation coefficients to obtain a correlation matrix, carrying out thresholding treatment on the correlation matrix, setting the correlation lower than a threshold value to zero, generating a sparse correlation matrix, carrying out community division on the sparse correlation matrix by combining graph clustering algorithms such as spectral clustering, louvain algorithm and the like, aggregating cross-modal features into different clusters, wherein the features in each cluster represent a group of complementary redundant semantic information, evaluating the importance of each feature in the cluster by a feature selection method based on a graph, selecting a plurality of features with highest score as key complementary redundant features according to the importance score, and adding the selected key complementary redundant features into a pre-initialized redundant set to form a redundant set containing multi-modal key features;
Filling an instance layer of a semantic feature map by using key complementary redundant features in a redundancy set, wherein each feature is used as an instance node, carrying attribute information such as a mode, a time step, a feature value and the like to which the feature belongs, clearly defining types and layers of the semantic feature map, such as a sensor type, a fault type, a component type and the like, establishing inheritance and inclusion relations among the types, defining an attribute and a value range of each type, such as a measuring range of the sensor, a sampling frequency, a severity of the fault and the like, perfecting attribute description of the instance node, introducing a time dimension into the semantic feature map, representing state changes of the instance node in different time steps, connecting the instance nodes in different time steps through time edges to form a time sequence relation, establishing semantic association edges such as a causal relation, a correlation relation, a synonymous relation and the like among the instance nodes according to domain knowledge and expert experience, enriching semantic expression capability of the map, and finally obtaining the semantic feature map.
In the embodiment, the multi-modal sensor data is mapped into the heterogeneous graph structure, the sensors are represented by nodes, the sensors are represented by edges, the graphical representation of the multi-modal data is realized, the extraction of local features and the fusion of cross-modal features are realized by cross-modal convolution kernels and modal specific convolution kernel functions, the complementary and redundant relations among different modalities are identified by computing correlation matrixes among the cross-modal features and performing sparsification processing and community division, the semantic feature patterns form structural knowledge representation in the industrial robot predictive maintenance field, semantic support is provided for intelligent fault diagnosis and maintenance decision, and in sum, the deep fusion and semantic enhancement of the industrial robot multi-modal sensor data are realized, the knowledge patterns facing predictive maintenance are constructed, and important technical support is provided for intelligent fault diagnosis, health assessment and maintenance policy optimization.
S2, constructing a health state prediction model corresponding to the industrial robot comprising a plurality of subsystems based on a converter model, adding the semantic feature map as input into the health state prediction model, mapping the semantic feature map to an embedded space by an input embedding layer through linear transformation, extracting time sequence information, combining a self-attention layer to perform feature interaction to obtain interaction features, and performing transformation and feature enhancement on the interaction features through a feedforward neural network layer to obtain health state probability distribution of each subsystem;
The converter model is a deep learning model based on a self-attention mechanism, and is generally applied to the fields of natural language processing, computer vision and the like, the health state prediction model is a machine learning model used for predicting the current and future health states of an industrial robot in predictive maintenance of the industrial robot, the input embedding layer is a layer for mapping discrete input tokens to continuous vector representation in the converter model, the embedding space is a continuous vector space to which the input embedding layer maps discrete tokens, and the interaction characteristics are interaction information between different positions calculated by the self-attention mechanism in the converter model.
In an alternative embodiment of the present invention,
Constructing a health state prediction model corresponding to an industrial robot comprising a plurality of subsystems based on a converter model, adding the semantic feature map as input into the health state prediction model, mapping the semantic feature map to an embedded space by an input embedding layer through linear transformation, extracting time sequence information, combining self-attention layers to perform feature interaction to obtain interaction features, transforming and feature enhancing the interaction features through a feedforward neural network layer, and obtaining health state probability distribution of each subsystem comprises the following steps:
Based on a converter model, constructing an encoder-decoder structure, adding a semantic feature map corresponding to an industrial robot into the converter model, mapping nodes in the semantic feature map into low-dimensional dense vectors through a node embedding technology, reserving a graph structure, designing the structure of the encoder and the decoder, introducing a position coding mechanism, adding a position embedding vector into an input embedding layer, and obtaining a health state prediction model corresponding to each subsystem;
Adding the semantic feature map as input into the health state prediction model, mapping nodes in the map into a low-dimensional embedding space through linear transformation by an input embedding layer, aggregating neighbor node information by combining a map attention network, updating embedded representation of each node, adding corresponding time stamp information in each node embedding, constructing a time sequence embedding matrix, determining time sequence information corresponding to the current node based on the time sequence embedding matrix, encoding the semantic feature map embedded with the time sequence information, dividing the map into a plurality of sub-graphs, wherein each sub-graph corresponds to a subsystem of an industrial robot, and extracting features and interacting features of each sub-graph based on an attention mechanism to obtain the interactive features;
based on the interaction characteristics, extracting association modes among different subsystems, adding the interaction characteristics into the health state prediction model, recursively generating the health state of the next time step, combining the association modes, combining an attention mechanism to dynamically adjust the influence degree of the different subsystems on the current prediction, and repeating the prediction until the preset maximum iteration times are reached, so as to obtain the health state probability distribution corresponding to each subsystem.
The position coding mechanism is to assign a unique position code to each position of the input sequence in the converter model to introduce position information, and the position embedding vector is to generate a unique vector representation for each position in the position coding mechanism.
On the basis of a converter model, constructing an encoder-decoder structure for predicting the health state of the industrial robot, wherein the encoder is responsible for encoding an input semantic feature map into a hidden representation, the decoder generates a prediction result of the health state according to the hidden representation, the semantic feature map corresponding to the industrial robot is added into the converter model, the semantic representation capability of the model is enhanced by utilizing node information and side information in the map, the nodes in the semantic feature map are mapped into low-dimensional dense vectors through a node embedding technology, meanwhile, map structure information is reserved, each node is represented as a vector with a fixed dimension, and each element in the vector represents the coordinates of the node in an embedding space;
Introducing a position coding mechanism into the structures of an encoder and a decoder to capture position information in an input sequence, adding a position embedded vector in an input embedded layer besides mapping a node into an embedded vector, adding or splicing the position embedded vector and the node embedded vector to obtain final input representation, constructing a corresponding health state prediction model for each subsystem of the industrial robot, wherein the model of each subsystem is based on a converter structure and comprises the encoder and the decoder, the encoder receives a semantic feature sub-graph corresponding to the subsystem as input, extracting features through a self-attention mechanism and a feedforward neural network, generating health state prediction of the next time step according to the output of the encoder, and obtaining a health state sequence in a future period of the subsystem through repeated iterative prediction;
Inputting semantic feature patterns into a health state prediction model, mapping pattern nodes into a low-dimensional embedding space through an input embedding layer, aggregating neighbor information of each node through a graph attention network, updating embedding representation of the nodes, adding corresponding timestamp information in the embedding representation of each node, constructing a time sequence embedding matrix, wherein each row of the time sequence embedding matrix represents embedding vectors of one node at different time steps, each column represents embedding vectors of different nodes at the same time step, encoding the semantic feature patterns of the embedding time sequence information, dividing the patterns into a plurality of sub-graphs, each sub-graph corresponds to one sub-system of the industrial robot, carrying out feature extraction and feature interaction on each sub-graph based on an attention mechanism, and calculating correlation and interaction strength among different sub-graphs through the attention mechanism to obtain interaction features representing a correlation mode among the sub-systems;
The extracted interaction characteristics are added into a health state prediction model to serve as additional input information, when the health state of the next time step is predicted, the node embedding and the historical health state of the current time step are utilized, the interaction characteristics are combined, the association mode among the subsystems is captured, the influence degree of different subsystems on the current prediction is dynamically adjusted, interaction among the subsystems can be fully considered by a prediction result, the health state of the next time step is recursively generated until the preset maximum iteration times are reached, and the health state probability distribution of each subsystem is obtained.
In the embodiment, the semantic feature map corresponding to the industrial robot is added to the converter model, the semantic representation capability of the model is enriched by utilizing node information and side information in the map, the node embedding technology maps the map nodes into low-dimensional dense vectors, meanwhile, map structure information is reserved, the model can fully utilize structured knowledge of the map, quality of feature representation and accuracy of prediction are improved, a position coding mechanism is introduced, the model can capture position information in an input sequence, node representations of different time steps are distinguished, feature extraction and feature interaction are carried out on each sub-map through an attention mechanism, interaction features representing a correlation mode among sub-systems are obtained, dependency and influence mechanisms among different sub-systems are fully considered, context information of a cross-subsystem is provided, global feature and correlation of health state prediction are enhanced, evolution rules of map structuring knowledge and time sequence data are fully utilized, correlation modes among sub-systems are captured, future predictive maintenance decision support is provided, and accurate prediction and comprehensive evaluation of the industrial robot health state are realized.
In an alternative embodiment of the present invention,
The embedded representation of aggregating neighbor node information and updating each node in conjunction with the graph attention network is shown in the following formula:
;
Wherein, Represents the updated hidden state of node i, h i represents the hidden state of node i, K represents the number of attention headers, sigma represents the activation function,Indicating that the kth attention header is at time step t, the attention weight of node i to neighbor node j, W k indicates the weight matrix of the kth attention header, h j indicates the hidden state of node j,Representing the attention weight of the kth attention head to itself by node i at time step t, N (i) represents the set of neighbor nodes of node i.
In this embodiment, different importance weights are allocated to neighbor nodes of each node through an attention mechanism, so that selective aggregation of neighbor information is achieved, an adaptive weight allocation mechanism enables a graph attention network to flexibly adjust contribution degrees of neighbor nodes to central node representation according to semantic similarity and structural relation of the nodes, semantic richness and distinguishability of node representation are improved, the graph attention network can capture diversified interaction modes among the nodes from different semantic subspaces through introducing a plurality of attention heads, information of different semantic subspaces can be synthesized, robustness and expression capability of representation are enhanced, hidden states of the nodes can be propagated and updated in a graph structure in multiple steps through iteratively applying the graph attention network, in this embodiment, local and global information of the graph structure are fully utilized, diversified interaction modes among the nodes are captured through the adaptive attention weight allocation and the attention mechanism, accuracy, robustness and reliability of industrial robot health state prediction can be effectively improved, and reliability of intelligent transportation and prediction are provided for maintenance and prediction.
S3, based on the health state probability distribution, generating comprehensive health state distribution through Bayesian inference fusion, combining an abnormality detection model based on active learning, combining active exploration and uncertainty sampling to identify an abnormality mode and key turning points in the comprehensive health state distribution, generating an optimal maintenance strategy based on health state influence information and combining a reinforcement learning algorithm, carrying out collaborative maintenance on the industrial robot through a game theory method, uploading a maintenance result to a server in real time, and storing the current state of the industrial robot.
The Bayesian inference is an inference method based on probability theory and statistics, the prior knowledge and observation data are fused, the probability distribution of unknown quantity is updated to obtain posterior probability distribution, the active exploration is a method for reducing uncertainty and optimizing decisions by actively selecting mobile phone information through behaviors, the uncertainty sampling is an active learning strategy, the sample with the largest uncertainty in model prediction is selected for labeling and learning, the generalization capability and the robustness of the model are improved, the key turning points are important time nodes in the evolution process of the health state of the industrial robot and are usually represented as abrupt change or abnormal fluctuation of the state, the game theory is a mathematical theory for researching strategy interaction and decision optimization among a plurality of participants, and in predictive maintenance of the industrial robot, the game theory method can be applied to collaborative decision and optimization among a plurality of stakeholders, and the collaborative maintenance refers to predictive maintenance tasks of the industrial robot are jointly completed through information sharing, resource coordination and task division.
In an alternative embodiment of the present invention,
Based on the health state probability distribution, generating comprehensive health state distribution through Bayesian inference fusion, combining an abnormality detection model based on active learning, combining active exploration and uncertainty sampling to identify an abnormality mode and key turning points in the comprehensive health state distribution, generating an optimal maintenance strategy based on health state influence information and combining a reinforcement learning algorithm to cooperatively maintain the industrial robot through a game theory method, and uploading a maintenance result to a server in real time and storing the current state of the industrial robot comprises the following steps:
the health state probability distribution of each subsystem is used as an independent random variable, a joint probability distribution model is established, priori knowledge is introduced to construct priori distribution probability, posterior probability distribution corresponding to each subsystem is calculated based on Bayesian theorem by combining the prior distribution probability and health state likelihood functions of the subsystems, and the posterior probability distribution is used as the comprehensive health state distribution;
Based on the comprehensive health state distribution, an abnormal detection model based on active learning is constructed, the comprehensive health state distribution is modeled through an unsupervised learning method, a sample with the maximum information amount is selected for marking according to an active exploration strategy, the uncertainty of a current sample is evaluated through edge probability, the sample with the highest uncertainty is repeatedly evaluated and selected, and a manual marking label is added;
Updating the abnormal detection model according to the re-labeled sample, adaptively adjusting decision boundaries and model super-parameters of the abnormal detection model by combining an incremental learning algorithm, detecting the comprehensive health state distribution according to the updated abnormal detection model, and determining drift trend and the key turning points of the current health state distribution by combining a drift detection mechanism;
the method comprises the steps of obtaining fault risks, task importance, maintenance cost and residual service life corresponding to industrial robots, recording the fault risks, task importance, maintenance cost and residual service life as health state influence information, taking the health state influence information as a state space, taking maintenance operation as an action space, defining state transition probability and rewarding functions, updating function values between the state and the action through a reinforcement learning algorithm, generating an optimal maintenance strategy, determining a maintenance rewarding mechanism and a cooperation strategy between each industrial robot by combining a game theory method, repeatedly optimizing, obtaining a global optimal maintenance decision, carrying out cooperative maintenance on the industrial robots, and uploading the state parameters of the industrial robots after maintenance to a server and storing.
The joint probability distribution model is used for describing the correlation and probability distribution among a plurality of random variables, the health state likelihood function represents the conditional probability distribution of observed data under a given health state, the unsupervised learning is a machine learning method for finding hidden modes and structures from unlabeled data, the edge probability refers to the probability distribution of a certain variable in the joint probability distribution and does not consider the values of other variables, the incremental learning is a continuous learning machine learning model, the model can be continuously updated when new data arrive without retraining the whole model, the decision boundary refers to a dividing line or a hyperplane for dividing different categories or states in a feature space, and the drift detection refers to a mechanism for identifying the change of the data distribution along time and is used for timely finding the change of a data generating process.
Dividing an industrial robot system into a plurality of subsystems, such as a mechanical arm, a sensor, a controller and the like, carrying out probability modeling on the health state of each subsystem, regarding the health state as an independent random variable, introducing priori knowledge, such as historical maintenance data, expert experience and the like, constructing prior probability distribution of the health state of each subsystem, calculating posterior probability distribution of the health state of each subsystem based on Bayesian theorem by combining the prior probability and the health state likelihood function of each subsystem, and combining the posterior probability distribution of all the subsystems to obtain comprehensive health state distribution of the whole industrial robot system;
based on the comprehensive health state distribution obtained in advance, an unsupervised learning method, such as a Gaussian mixture model, a hidden Markov model and the like, is adopted to model the comprehensive health state distribution, a sample with the highest value for judging an abnormal detection model, namely a sample with the largest information amount is selected according to an active exploration strategy, such as uncertainty sampling, information gain and the like, the uncertainty of the currently selected sample is evaluated through calculating the edge probability, the sample with the highest uncertainty is selected, the selected sample is marked manually, whether the selected sample is in an abnormal state or not is determined, the marked sample is added into a training set, and the abnormal detection model is updated;
Adopting an increment learning algorithm, such as online learning, progressive learning and the like, adjusting decision boundaries and model super parameters of an abnormal detection model according to a newly marked sample, detecting comprehensive health state distribution by using an updated abnormal detection model, identifying potential abnormal states, introducing a drift detection mechanism, such as concept drift detection, covariance drift detection and the like, monitoring the change trend of the comprehensive health state distribution, and determining the drift trend and key turning points of the health state distribution by combining the drift detection result;
acquiring health state influence information such as fault risk, task importance, maintenance cost and residual life of an industrial robot, taking optional maintenance operations such as preventive maintenance, corrective maintenance and the like as a state space of reinforcement learning, defining state transition probability and rewarding functions as an action space of reinforcement learning, describing the influence of different maintenance operations on the health state and the cost, selecting a reinforcement learning algorithm, generating an optimal maintenance strategy by continuously interacting with the environment, updating a value function between the state and the action, and determining the optimal maintenance operation to be adopted under different health states;
For a scene of cooperative work of a plurality of industrial robots, a game theory method is introduced, such as Nash equilibrium, cooperation and competition relation among modeling robots are established, an allocation scheme and a cooperation strategy of maintenance resources are determined according to health states and maintenance requirements of the robots, a maintenance rewarding mechanism and a cooperation strategy of a multi-robot system are calculated through a game theory algorithm, a maintenance decision of the multi-robot is optimized in an iterative mode, global benefits and local benefits are comprehensively considered, a globally optimal maintenance strategy is obtained, cooperative maintenance is carried out on the industrial robots according to the generated optimal maintenance strategy, such as preventive replacement of parts, sensor calibration and the like, state parameters of the robots, such as running time, load, vibration and the like, are collected in a maintenance process, and the state parameters of the robots after maintenance are uploaded to a central server, and health state information of the robots is updated and stored.
In the embodiment, the overall health condition of the robot can be more accurately depicted by the overall modeling method, a reliable basis is provided for subsequent abnormal detection and maintenance decision making, the most valuable samples are intelligently selected for marking, learning efficiency and accuracy of an abnormal detection model are remarkably improved, the accurate assessment of the health condition of the robot, early warning of abnormality, dynamic optimization of maintenance strategies and collaborative management of multiple robots are achieved by continuously integrating the newly marked samples, the decision boundary and model parameters are adjusted, the high accuracy and timeliness are always kept, multiple factors such as fault risk, task importance, maintenance cost and residual life are comprehensively considered, a reinforcement learning algorithm is adopted for solving, a globally optimal maintenance decision can be obtained, maintenance cost can be remarkably reduced, the availability and production efficiency of the robot are improved, during collaborative maintenance of multiple robots, the game theory method is introduced, maintenance resource allocation and task collaboration among different robots can be effectively coordinated, and maximization of overall benefits is achieved.
In an alternative embodiment of the present invention,
Based on the Bayes theorem, the posterior probability distribution corresponding to each subsystem is calculated by combining the prior probability and the health state likelihood function of each subsystem, and the posterior probability distribution is shown as the following formula:
;
Wherein S g represents the health state of the g-th subsystem, O represents the comprehensive health state evidence observed, P (o|s g) represents the likelihood that the comprehensive health state evidence O is observed under the condition that the g-th subsystem is in the health state S g, P (S g |o) represents the posterior probability that the g-th subsystem is in the health state S g under the condition that the comprehensive health state evidence O, P (S g) represents the prior probability that the g-th subsystem is in the health state S g, P (o|s j) represents the likelihood that the comprehensive health state evidence O is observed under the condition that the j-th subsystem is in the health state S j, P (S j) represents the prior probability that the j-th subsystem is in the health state S j, and n represents the total number of subsystem states.
In the embodiment, the prior probability and the likelihood function can be combined through the bayesian theorem, so that the posterior probability distribution considers the prior knowledge and fully utilizes the observation data, thereby obtaining more reliable and comprehensive health state estimation, quantifying the uncertainty of the health state of each subsystem through the posterior probability distribution, providing more abundant information, supporting more flexible and fine decision, enabling the health state estimation to be adaptively learned and improved in a recursive updating mode, and in conclusion, the embodiment fully utilizes the prior knowledge and the observation data, quantifying the uncertainty of the health state, supporting probabilistic decision making and providing a solid foundation for intelligent maintenance of the industrial robot.
Fig. 2 is a schematic structural diagram of an industrial robot predictive maintenance management system according to an embodiment of the present invention, as shown in fig. 2, the system includes:
The system comprises a first unit, a second unit and a third unit, wherein the first unit is used for acquiring multi-modal heterogeneous data of the industrial robot in real time through a preset sensing network, performing intelligent preprocessing on the multi-modal heterogeneous data, combining blockchain right-determining synchronization and generating an countermeasure network enhancement equalization operation to obtain first preprocessed data, performing cross-modal correlation fusion on the first preprocessed data through a multi-modal graph convolutional neural network to generate fusion data, extracting complementary redundant information among the fusion data and constructing a semantic feature map corresponding to the industrial robot;
The second unit is used for constructing a health state prediction model corresponding to the industrial robot comprising a plurality of subsystems based on the converter model, adding the semantic feature map into the health state prediction model as input, mapping the semantic feature map into an embedded space by an input embedding layer through linear transformation, extracting time sequence information, combining a self-attention layer to perform feature interaction to obtain interaction features, and performing transformation and feature enhancement on the interaction features through a feedforward neural network layer to obtain health state probability distribution of each subsystem;
The third unit is used for generating comprehensive health state distribution through Bayesian inference fusion based on the health state probability distribution, identifying an abnormal mode and key turning points in the comprehensive health state distribution by combining an abnormal detection model based on active learning and combining active exploration and uncertainty sampling, generating an optimal maintenance strategy by combining a reinforcement learning algorithm based on health state influence information, carrying out cooperative maintenance on the industrial robot through a game theory method, uploading a maintenance result to a server in real time, and storing the current state of the industrial robot.
In a third aspect of an embodiment of the present invention,
There is provided an electronic device including:
A processor;
A memory for storing processor-executable instructions;
wherein the processor is configured to invoke the instructions stored in the memory to perform the method described previously.
In a fourth aspect of an embodiment of the present invention,
There is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method as described above.
The present invention may be a method, apparatus, system, and/or computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions embodied thereon for performing various aspects of the present invention.
It should be noted that the above embodiments are merely for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the technical solution described in the above embodiments may be modified or some or all of the technical features may be equivalently replaced, and these modifications or substitutions do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiments of the present invention.