CN113095331A - Visual question answering method, system and equipment for appearance defects of electric equipment and storage medium thereof - Google Patents

Abstract

The invention provides a visual question and answer method for appearance defects of electric equipment, comprising the steps of: acquiring an image of appearance defects of electric equipment, and preprocessing the image; extracting features from the processed image to obtain image features; For the image, obtain the problem information related to the image, and perform feature extraction on the problem information to obtain text features; based on the bilinear pooling network, the image features and the text features are fused to obtain Multimodal features; establishing and training a visual question answering model to obtain the trained visual question answering model; inputting the multimodal features into the trained visual question answering model to obtain a visual question answering result. The present invention provides a visual question and answer method for appearance defects of electric equipment, which can screen and evaluate appearance defects of electric equipment through images, so that the running state of electric equipment can be monitored in real time.

Description

A visual question answering method, system, device and storage medium for appearance defects of electrical equipment

technical field

The present invention relates to the technical field of visual question and answer, in particular to a visual question and answer method, system, device and storage medium for appearance defects of electric equipment.

Background technique

In recent years, with the continuous expansion of the scale of my country's industrial and technological industries and the continuous expansion of urban areas, the demand for civil, commercial and military power energy is increasing day by day, and the scale of my country's power grid is also constantly upgrading and expanding to meet the growing demand. At present, my country's equipment coverage is very large, and the number of various substation equipment is also huge, such as power stations, wires, towers, etc. Such a long transmission line is almost completely in an outdoor environment, and is very vulnerable to factors such as bad weather, animal and plant invasion, and floating objects, which can lead to equipment failure, aging, and damage. On the one hand, experienced operation and maintenance personnel are needed to identify defects, and on the other hand, it is difficult to obtain the defect status of equipment in real time or preventively by relying solely on manual labor. At the same time, it also requires huge labor costs and maintenance costs, and there is also a threat to the life safety of frontline workers. Therefore, in order to improve the safety of personnel and equipment in unattended or less-attended power stations, it is necessary to monitor the operating status and hidden faults of substation equipment in real time.

With the improvement of the intelligence of power stations and the gradual popularization of intelligent inspection technology, the use of artificial intelligence technologies such as image recognition and text mining to assist in judging the health status of equipment has begun to appear. At present, power station operation and maintenance personnel usually use hand-held terminal equipment such as cameras and video cameras to collect images of the main electrical equipment in the station; in addition, a large number of inspection robots and distributed cameras have been arranged in the power station, responsible for image collection and On-site monitoring. Visual images can reflect equipment defects, often including oil leaks, rust, circuit breakers, and damaged components. However, at present, the appearance defects of electrical equipment are still in the stage of perceptual intelligence, that is, the computer has the ability of vision and hearing similar to human beings, for example, what is heard, corresponding to speech recognition; what is seen, corresponding to the classification and detection of images and semantic segmentation. However, it has not yet risen to the stage of cognitive intelligence, that is, emphasizing knowledge, reasoning and other skills, requiring machines to understand and think. Cognitive intelligence involves semantic understanding, knowledge expression, associative reasoning, intelligent question answering, autonomous learning, etc. It is related to human language, knowledge, and logic, and is the highest stage of artificial intelligence. As one of cognitive intelligence applications, visual question answering is still almost blank in the research on appearance defects of electrical equipment.

VQA (Visual Question Answering) means that given a picture and a natural language question related to the picture, the computer can generate a correct answer. Obviously, this is a typical multimodal problem that combines CV (Computer Vision, Computer Vision) and NLP (Natural Language Processing, Natural Language Processing) technologies, and computers need to learn to understand images and text at the same time. Because of this, the concept of VQA was not formally proposed until 2015, when related technologies achieved breakthrough development. Although visual question answering has been greatly developed in the past two years, there are still two major problems. The first is the lack of training data: although the data scale of the existing intelligent visual question answering dataset has reached about 1 million, it still has a huge gap compared with traditional tasks such as image classification and target detection; Model parameters are often in the tens of millions, and a small amount of training data often fails to give full play to the performance of the model. Another problem is that the answers provided by the machine are not explanatory: due to the black-box design of the deep learning model, the reasons and reasons for the machine's answer are often difficult to give; it involves the promotion of visual question answering in the industry and the There are very few related studies used.

Therefore, in order to solve the problems of high false detection and missed detection caused by a large number of manual operations in the prior art, and the traditional inspection equipment cannot provide the evaluation and analysis of hidden dangers of power transmission lines and power transmission and transformation equipment, the present invention designs a power equipment. Visual question answering method, system, device and storage medium for appearance defect.

SUMMARY OF THE INVENTION

In view of the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a visual question and answer method, system, device and storage medium for the appearance defects of power equipment, which are used to solve the high false detection caused by a large number of manual operations in the prior art. and missed inspections, and traditional inspection equipment cannot provide assessment and analysis of hidden dangers of transmission lines and power transmission and transformation equipment.

In order to achieve the above-mentioned purpose and other related purposes, the present invention provides a visual question and answer method for appearance defects of electrical equipment, comprising the steps of:

acquiring an image of the appearance defect of the power equipment, and preprocessing the image;

Perform feature extraction on the processed image to obtain image features;

According to the image, obtain problem information related to the image, and perform feature extraction on the problem information to obtain text features;

Based on a bilinear pooling network, the image features and the text features are fused to obtain multimodal features;

establishing a visual question answering model and training it to obtain the trained visual question answering model;

The multimodal feature is input into the trained visual question answering model to obtain a visual question answering result.

In an embodiment of the present invention, the preprocessing of the image includes the steps of:

screening the images to remove images that do not contain detection objects;

Performing dark channel dehazing and adaptive gamma correction processing on the screened image to improve the clarity and contrast of the image;

performing position transformation and color transformation on the processed image to enhance the information of the image;

Cropping the transformed image so that the size of the image satisfies the size requirement when extracting features from the image;

Marking the cropped image to obtain a marked image;

Manual inspection of the labeled image is performed to remove abnormal data in the labeled image.

In an embodiment of the present invention, performing feature extraction on the processed image to obtain image features includes the following steps:

Detecting the processed image through a target detection model based on a convolutional neural network to identify objects in the image;

According to the identified object, feature extraction is performed on the image to obtain the image feature.

In an embodiment of the present invention, performing feature extraction on the problem information to obtain text features includes the steps of:

Vectorize the corpus to get word vectors;

According to the word vector, the question information is vectorized through the text vector layer of the LSTM-Attention neural network to obtain a sentence sequence;

The sentence sequence is input into the LSTM network, and the sentence sequence is semantically encoded by the LSTM network to obtain the output information of the hidden layer, which is expressed as:

h _it = LSTM(x _it ), t∈[1, m]

Wherein, m is the number of words in the sentence sequence, x _it is the word input into the LSTM network at time t, h _it is the output information of the hidden layer output by the LSTM network at time t;

Through the word Attention layer of the LSTM-Attention neural network, the output information of the hidden layer is nonlinearly transformed and normalized to obtain the weight coefficient of the word-level hidden layer output, and the word attention is obtained according to the weight coefficient. Mechanism matrix, expressed as:

Wherein, α _it is the weight coefficient output by the word-level hidden layer, and s _i is the sentence vector of the word attention mechanism matrix;

Inputting the sentence vector into the LSTM network, and the LSTM network updates the sentence vector according to the output information of the hidden layer at the previous moment to obtain the updated sentence vector;

Through the sentence Attention layer of the LSTM-Attention neural network, nonlinear transformation and normalization are performed on the updated sentence vector to obtain the weight coefficient of the sentence-level hidden layer output, and the sentence attention is obtained according to the weight coefficient. The force mechanism matrix, expressed as:

Wherein, α _i is the weight coefficient output by the sentence-level hidden layer, and _hi is the updated sentence vector;

According to the word attention mechanism matrix and the sentence attention mechanism matrix, feature extraction is performed on the question information to obtain the text feature.

In an embodiment of the present invention, the step of inputting the sentence sequence into an LSTM network, and performing semantic encoding on the sentence sequence through the LSTM network to obtain the output information of the hidden layer further includes the steps:

Using the sentence sequence as the input node data of the LSTM network;

Learning the before and after information of the sentence sequence through the LSTM network to obtain the before and after information between the sentence sequences;

Semantic encoding is performed on the sentence sequence according to the context information.

In an embodiment of the present invention, the method of fusing the image feature and the text feature based on a bilinear pooling network to obtain a multimodal feature includes the following steps:

mapping the image feature and the text feature to the same feature space, so that the dimension of the image feature matches the dimension of the text feature;

The image features and the text features whose dimensions are matched are input into the bilinear pooling network to output the fused multimodal features, which are expressed as:

q=Fusion(m,n)

Among them, m is the image feature, n is the text feature, Fusion represents feature fusion, and q is the multimodal feature.

In an embodiment of the present invention, inputting the multimodal features into the trained visual question answering model to obtain a visual question answering result includes the steps:

Inputting the multimodal feature into the trained visual question answering model;

According to the multi-modal feature, the answer to be selected is determined, and the answer to be selected constitutes a domain to be selected;

predict the probability distribution of the correct answer on the candidate field;

According to the probability distribution, the candidate answer corresponding to the maximum probability is used as the result of the visual question answering.

The present invention also provides a visual question answering system for appearance defects of electrical equipment, including:

an image acquisition module, used for acquiring an image of the appearance defect of the power equipment, and preprocessing the image;

an image feature extraction module, configured to perform feature extraction on the processed image to obtain image features;

a text feature extraction module, configured to obtain problem information related to the image according to the image, and perform feature extraction on the problem information to obtain text features;

a feature fusion module, configured to fuse the image features and the text features based on a bilinear pooling network to obtain multimodal features;

a model building module, used to build a visual question answering model and train it to obtain the trained visual question answering model;

The visual question answering module is used for inputting the multimodal feature into the trained visual question answering model to obtain a visual question answering result.

In an embodiment of the present invention, a visual question answering device for appearance defects of electrical equipment includes: a processor, the processor is coupled to a memory, the memory stores program instructions, and when the program instructions stored in the memory are When the processor is executed, the visual question answering method for the appearance defect of the electrical equipment as described above is realized.

In one embodiment of the present invention, a computer-readable storage medium is characterized by comprising a program, which when the program runs on a computer, enables the computer to execute the above-mentioned visual question and answer method for appearance defects of electrical equipment.

As described above, the present invention provides a visual question answering method, system, device, and storage medium for electrical equipment appearance defects. The image features are extracted from the marked power equipment scene images, and fused with the text features of the problem information to obtain The corresponding answer prediction is obtained, and the question and answer effect is good; secondly, the method can also monitor the running state of the substation equipment in real time, prevent the phenomenon of false detection and missed detection, and improve the work efficiency.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a schematic flowchart of a visual question and answer method for appearance defects of electrical equipment provided by the present invention.

FIG. 2 is a schematic flowchart of step S1 in an embodiment of the present invention.

FIG. 3 is a schematic flowchart of step S3 in an embodiment of the present invention.

FIG. 4 is a schematic flowchart of step S33 in the embodiment of the present invention.

FIG. 5 is a schematic flowchart of step S6 in an embodiment of the present invention.

FIG. 6 is a schematic structural diagram of the principle and structure of a visual question answering system for appearance defects of electrical equipment provided by the present invention.

Component label description

11 Image acquisition module 14 Feature fusion module

12 Image Feature Extraction Module 15 Model Building Module

13 Text Feature Extraction Module 16 Visual Question Answering Module

Detailed ways

The embodiments of the present invention are described below through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments may be combined with each other under the condition of no conflict.

It should be noted that the drawings provided in the following embodiments are only used to illustrate the basic concept of the present invention in a schematic way, so the drawings only show the components related to the present invention rather than the number, shape and number of components in actual implementation. For dimension drawing, the type, quantity and proportion of each component can be changed at will in actual implementation, and the component layout may also be more complicated.

As shown in Figure 1, the present invention provides a visual question and answer method for appearance defects of electrical equipment, comprising the steps of:

S1. Obtain an image of the appearance defect of the power equipment, and preprocess the image;

S2. Perform feature extraction on the processed image to obtain image features;

S3. Obtain problem information related to the image according to the image, and perform feature extraction on the problem information to obtain text features;

S4. Based on bilinear pooling network, image features and text features are fused to obtain multimodal features;

S5. Establish a visual question answering model and train it to obtain a trained visual question answering model;

S6. Input the multimodal feature into the trained visual question answering model to obtain a visual question answering result.

As shown in Figure 2, further, step S1 also includes:

S11. Screen the images to remove images that do not contain the detection object;

S12. Perform dark channel dehazing and adaptive gamma correction processing on the filtered image to improve the clarity and contrast of the image;

S13, performing position transformation and color transformation on the processed image to enhance the information of the image;

S14. Crop the transformed image so that the size of the image meets the size requirements for the feature extraction of the image;

S15, marking the cropped image to obtain a marked image;

S16. Perform manual inspection on the marked image to remove abnormal data in the marked image.

In an embodiment of the present invention, for step S11, the images are screened to remove images that do not contain detection objects, wherein the detection objects include glass insulators, composite insulators, wires, suspension clamps, anti-vibration hammers, towers, bolts , line corridors and other related foreign objects and defects.

In an embodiment of the present invention, for step S12, a dehazing algorithm is used to perform dark channel dehazing on the image with fog, so as to improve the clarity of the image, and at the same time, adaptive gamma correction processing is performed on the image with poor imaging, In order to improve the contrast of the image, the accuracy of subsequent image segmentation and information quantification can be improved.

In an embodiment of the present invention, for step S13, when the basic features of the image are guaranteed to remain unchanged, position transformation and color transformation are performed on the image to enhance the information of the image. Among them, position transformation includes rotation, translation, and horizontal mirroring; rotation refers to taking any point in the image as the rotation center, and rotating all pixels of the image by the same angle, and filling zero-value positions without pixel points. For continuous targets such as insulators, rotating the image can expand the angular diversity of the target without changing the characteristics of the target; translation refers to moving all pixels in random steps in the vertical and horizontal directions to expand the target space The diversity of positions; horizontal mirroring refers to the symmetry of the central axis of the image, and the positions of the pixels on both sides are exchanged, which can expand the diversity of image shooting directions. In this embodiment, the color transformation includes color dithering, contrast adjustment, brightness adjustment, and saturation adjustment; color dithering refers to randomly selecting the value of one channel in the three RGB channels of the image to perform up-down floating transformation. In this embodiment, The range of floating transformation does not exceed 20, such as 16; and the adjustment of contrast, brightness and saturation is mainly achieved by simulating the changes of sunlight intensity in real scenes.

In an embodiment of the present invention, further, step S2 further includes:

S21. Detect the processed image through a target detection model based on a convolutional neural network to identify objects in the image;

S22. Perform feature extraction on the image according to the recognized object to obtain image features.

In an embodiment of the present invention, an object detection model based on a convolutional neural network, Faster-RCNN, is used to identify objects in an image, and according to the identified objects, feature extraction is performed on the image to obtain image features. Expressed as;

V ^h =RCNN(I) (1)

Among them, I is the image, RCNN represents the detection and recognition of the image, and V ^h is the extracted image feature.

In this embodiment, the extracted image features are represented as a K×2048 matrix, which indicates that each image includes K vectors, and the dimension of each vector is 2048 dimensions.

As shown in Figure 3, further, step S3 also includes:

S31, vectorize the corpus to obtain word vectors;

S32. According to the word vector, vectorize the question information through the text vector layer of the LSTM-Attention neural network to obtain a sentence sequence;

S33. Input the sentence sequence into the LSTM network, and perform semantic encoding on the sentence sequence through the LSTM network to obtain the output information of the hidden layer;

S34. Through the word Attention layer of the LSTM-Attention neural network, the output information of the hidden layer is nonlinearly transformed and normalized to obtain the weight coefficient of the word-level hidden layer output, and the word attention mechanism matrix is obtained according to the weight coefficient. ;

S35. Input the sentence vector in the word attention mechanism matrix into the LSTM network, and the LSTM network updates the sentence vector according to the output information of the hidden layer at the previous moment to obtain the updated sentence vector;

S36. Perform nonlinear transformation and normalization operations on the updated sentence vector through the sentence Attention layer of the LSTM-Attention neural network to obtain the weight coefficient of the sentence-level hidden layer output, and obtain the sentence attention mechanism according to the weight coefficient. matrix;

S37. According to the word attention mechanism matrix and the sentence attention mechanism matrix, feature extraction is performed on the question information to obtain text features.

In an embodiment of the present invention, for step S31, the Skip-Gram unsupervised model in word2vec is used to vectorize the corpus to obtain word vectors. Then, according to the word vector, the problem information is vectorized through the text vectorization layer of the LSTM-Attention neural network.

As shown in Figure 4, further, step S33 also includes:

S331. Use the sentence sequence as the input node data of the LSTM network;

S332, learning the before and after information of the sentence sequence through the LSTM network, so as to obtain the before and after information between the sentence sequences;

S333. Perform semantic coding on the sentence sequence according to the before and after information.

In an embodiment of the present invention, the sentence sequence (x _i1 , x _i2 ,..., x _it ) is first used as the input node data of the LSTM network, wherein x _i1 , x _i2 ,..., x _it are the sentences in the sentence sequence For each word, the LSTM network will then learn the before and after information of the sentence sequence according to the input data, obtain the before and after information between the sentence sequences, and perform semantic coding on the sentence sequence according to the obtained before and after information. Output the hidden layer output information of the corresponding node, which is expressed as:

h _it = LSTM(x _it ), t∈[1, m] (2)

Among them, m is the number of words in the sentence sequence, x _it is the word input to the LSTM network at time t, and h _it is the output information of the hidden layer output by the LSTM network at time t.

In an embodiment of the present invention, for step S34, since the contribution of each word to the problem information is different, in order to extract the features of important words, the word Attention layer of the LSTM-Attention neural network is used to extract hidden words. The layer output information is nonlinearly transformed and normalized to obtain the weight coefficient of the word-level hidden layer output, which is expressed as:

u _it =tanh(W _w h _it +b _w ) (3)

Among them, tanh is a nonlinear function, W _w is the weight of the hidden layer of the LSTM network, b _w is the bias of the hidden layer of the LSTM network, u _it is the implicit representation obtained by nonlinear transformation of the hidden layer output information h _it , u _w is the randomly initialized attention mechanism matrix, and α _it is the weight coefficient output by the word-level hidden layer.

Then, according to the weight coefficient output of the word-level hidden layer and the output information of the hidden layer, the word attention mechanism matrix is obtained, which is expressed as:

s _i =∑ _t α _it h _it (5)

where _si is the sentence vector in the word attention mechanism matrix.

In an embodiment of the present invention, for step S36, since the contribution of different sentences to the importance of the question information is also different, the sentence Attention layer of the LSTM-Attention neural network is used to nonlinearly perform the updated sentence vector. Transform and normalize operations to obtain the weight coefficient of the sentence-level hidden layer output, which is expressed as:

u _i =tanh(W _w h _i +b _w ) (6)

Among them, hi is the sentence vector updated by the LSTM network, _ui is the implicit representation obtained by nonlinear transformation of the updated sentence vector _hi , _{u s} is the randomly initialized sentence attention mechanism matrix, α _i is The weight coefficients for the output of the sentence-level hidden layer.

Then, according to the weight coefficient output by the sentence-level hidden layer and the updated sentence vector, the sentence attention mechanism matrix is obtained, which is expressed as:

v=∑ _i α _i h _i (8)

where _si is the sentence vector in the word attention mechanism matrix.

In an embodiment of the present invention, according to the word attention mechanism matrix and the sentence attention mechanism matrix, feature extraction is performed on the question information, and after text features are obtained, in order to compress high-dimensional text features into low-dimensional, the non-dimensionality of the data is enhanced. Linear ability, add a nonlinear layer before the Softmax classification layer, map the feature vector to a vector of length c, and finally calculate the distribution probability of each corresponding category through the Softmax classification layer, which is expressed as:

为提取出的文本特征向量，w_c、b_c分别为非线性层的权重和偏置，y_i为输出的概率分布。in,

is the extracted text feature vector, w _c , b _c are the weights and biases of the nonlinear layer, respectively, and y _i is the output probability distribution.

In an embodiment of the present invention, further, step S4 further includes:

S41, map the image feature and the text feature to the same feature space, so that the dimension of the image feature matches the dimension of the text feature;

S42 , inputting the image features and text features with matching dimensions into a bilinear pooling network to obtain a fused multimodal feature.

When using the bilinear pooling network for feature fusion, the dimensions of different features need to be matched, so the image features and text features are first mapped to the same feature space so that they have the same dimension, and then the two features are input into the bilinear feature space. In the linear pooling network, the fused multimodal features are obtained, which are expressed as:

q=Fusion(m,n) (10)

Among them, m is the image feature, n is the text feature, Fusion represents feature fusion, and q is the fused multimodal feature.

As shown in Figure 5, further, step S6 also includes:

S61. Input the fused multimodal features into the trained visual question answering model;

S62, according to the multimodal feature, determine the answer to be selected, and the answer to be selected constitutes the domain to be selected;

S63, predict the probability distribution of the correct answer on the field to be selected;

S64. According to the probability distribution, the candidate answer corresponding to the maximum probability is used as the result of the visual question answering.

In an embodiment of the present invention, after the multimodal features are input into the trained visual question answering model, the answers to be selected will be determined, and the answers to be selected will form the fields to be selected, and the correct answers will be predicted to fall in these fields to be selected first Then the candidate answer corresponding to the maximum probability can be used as the result of the final visual question and answer, which is expressed as:

为正确答案，M为待选域，P为概率分布。Among them, I is the acquired image, Q is the problem information related to the image,

is the correct answer, M is the field to be selected, and P is the probability distribution.

The step division of the above method is only for the purpose of describing clearly, and can be combined into one step or split some steps during implementation, and decomposed into multiple steps, as long as they contain the same logical relationship, they are all within the protection scope of the present invention; Adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are within the protection scope of this invention.

As shown in FIG. 6 , the present invention also provides a visual question answering system for appearance defects of electrical equipment, including: an image acquisition module 11 , an image feature extraction module 12 , a text feature extraction module 13 , a feature fusion module 14 , and a model building module 15 and visual question answering module 16. The image acquisition module 11 is used to acquire the image of the appearance defect of the power equipment, and preprocess the image; the image feature extraction module 12 is used to perform feature extraction on the processed image to obtain image features; the text feature extraction module 13 It is used to obtain the problem information related to the image according to the image, and perform feature extraction on the problem information to obtain text features; the feature fusion module 14 is used to combine the image features based on the bilinear pooling network. Fusion with the text features to obtain multimodal features; the model building module 15 is used to build a visual question answering model and train it to obtain the trained visual question answering model; the visual question answering module 16 is used to The modal features are input into the trained visual question answering model to obtain a visual question answering result.

Further, the text feature extraction module 13 further includes: a vectorization unit, a text vector unit, a semantic encoding unit, a word Attention unit, an update unit, a sentence Attention unit, and a feature extraction unit. The vectorization unit is used to vectorize the corpus to obtain the word vector; the text vector unit is used to vectorize the question information through the text vector layer of the LSTM-Attention neural network according to the word vector to obtain the sentence sequence; the semantic coding unit is used to The sentence sequence is input into the LSTM network, and the sentence sequence is semantically encoded by the LSTM network to obtain the output information of the hidden layer; the word Attention unit is used to pass the word Attention layer of the LSTM-Attention neural network. Non-linear transformation of the output information of the hidden layer and The normalization operation is used to obtain the weight coefficient of the word-level hidden layer output, and the word attention mechanism matrix is obtained according to the weight coefficient; the update unit is used to input the sentence vector into the LSTM network, and the LSTM network outputs the hidden layer according to the previous moment. The information updates the sentence vector to obtain the updated sentence vector; the sentence Attention unit is used to perform nonlinear transformation and normalization on the updated sentence vector through the sentence Attention layer of the LSTM-Attention neural network to obtain the sentence. The weight coefficient of the output of the hidden layer of the level, and the sentence attention mechanism matrix is obtained according to the weight coefficient; the feature extraction unit is used to perform feature extraction on the question information according to the word attention mechanism matrix and the sentence attention mechanism matrix to obtain text features.

It should be noted that, in order to highlight the innovative part of the present invention, this embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in this embodiment. module.

In addition, those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the system described above, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here. In the embodiments provided by the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

Modules described as separate components may or may not be physically separated, and components shown as modules may or may not be physical modules, that is, they may be located in one place, or may be distributed to multiple network modules. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically alone, or two or more modules may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, or can be implemented in the form of software functional units.

If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, removable hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program codes.

In an embodiment of the present invention, an agricultural machinery tracking control device includes: a processor, the processor is coupled to a memory, the memory stores program instructions, and when the program instructions stored in the memory are processed by the processor When executed, the visual question answering method for the appearance defect of the electrical equipment can be realized.

As described above, the present invention provides a visual question answering method, system, device, and storage medium for electrical equipment appearance defects. The image features are extracted from the marked power equipment scene images, and fused with the text features of the problem information to obtain The corresponding answer prediction is obtained, and the question and answer effect is good; secondly, the method can also monitor the running state of the substation equipment in real time, prevent the phenomenon of false detection and missed detection, and improve the work efficiency. Therefore, the present invention effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

The above-mentioned embodiments merely illustrate the principles and effects of the present invention, but are not intended to limit the present invention. Anyone skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical idea disclosed in the present invention should still be covered by the claims of the present invention.

Claims (10)

1. A visual question answering method for appearance defects of electric equipment is characterized by comprising the following steps:

acquiring an image of the appearance defect of the power equipment, and preprocessing the image;

performing feature extraction on the processed image to obtain image features;

according to the image, problem information related to the image is obtained, and feature extraction is carried out on the problem information to obtain text features;

fusing the image features and the text features based on a bilinear pooling network to obtain multi-modal features;

establishing a visual question-answer model and training the visual question-answer model to obtain the trained visual question-answer model;

and inputting the multi-modal characteristics into the trained visual question-answering model to obtain a visual question-answering result.

2. The visual question answering method for the appearance defects of the electric power equipment as claimed in claim 1, wherein the preprocessing of the image comprises the steps of:

screening the images to remove images not containing the detection object;

carrying out dark channel defogging and self-adaptive gamma correction processing on the screened image so as to improve the definition and contrast of the image;

performing position transformation and color transformation on the processed image to enhance the information of the image;

cutting the transformed image to enable the specification size of the image to meet the specification size requirement when the image is subjected to feature extraction;

marking the cut image to obtain a marked image;

and carrying out manual inspection on the marked image to remove abnormal data in the marked image.

3. The visual question answering method for the appearance defects of the electric power equipment according to claim 1, wherein the step of performing feature extraction on the processed image to obtain image features comprises the following steps:

detecting the processed image through a target detection model based on a convolutional neural network to identify an object in the image;

and according to the identified object, performing feature extraction on the image to obtain the image feature.

4. The visual question answering method for the appearance defects of the electric power equipment according to claim 1, wherein the step of performing feature extraction on the question information to obtain text features comprises the following steps:

vectorizing the corpus to obtain a word vector;

vectorizing the question information through a text vector layer of an LSTM-Attention neural network according to the word vector to obtain a sentence sequence;

inputting the sentence sequence into an LSTM network, and performing semantic coding on the sentence sequence through the LSTM network to obtain hidden layer output information, wherein the hidden layer output information is expressed as:

h_it＝LSTM(x_it)，t∈[1，m]

wherein m is the number of words in the sentence sequence, x_itInputting the words, h, of the LSTM network for time t_itThe hidden layer output information is output by the LSTM network at the time t;

carrying out nonlinear transformation and normalization operation on the hidden layer output information through the word Attention layer of the LSTM-Attention neural network to obtain a weight coefficient output by the word level hidden layer, and obtaining a word Attention mechanism matrix according to the weight coefficient, wherein the expression is as follows:

wherein alpha is_itWeight coefficient, s, output for said word-level hidden layer_iSentence vectors for the word attention mechanism matrix;

inputting the sentence vector into the LSTM network, and updating the sentence vector by the LSTM network according to the hidden layer output information at the previous moment to obtain the updated sentence vector;

through the sentence Attention layer of the LSTM-Attention neural network, carrying out nonlinear transformation and normalization operation on the updated sentence vector to obtain a weight coefficient output by the sentence-level hidden layer, and obtaining a sentence Attention mechanism matrix according to the weight coefficient, wherein the expression is as follows:

wherein alpha is_iWeight coefficient, h, for the sentence-level hidden layer output_iThe sentence vector after updating;

and according to the word attention mechanism matrix and the sentence attention mechanism matrix, performing feature extraction on the question information to obtain the text features.

5. The visual question-answering method for the appearance defects of the electric power equipment as claimed in claim 4, wherein the step of inputting the sentence sequence into an LSTM network, and semantically coding the sentence sequence through the LSTM network to obtain hidden layer output information further comprises the steps of:

taking the sentence sequence as input node data of the LSTM network;

learning the front and back information of the sentence sequences through the LSTM network to obtain the front and back information between the sentence sequences;

and according to the previous and next information, performing semantic coding on the sentence sequence.

6. The visual question-answering method for the appearance defects of the electric power equipment as claimed in claim 1, wherein the fusion of the image features and the text features based on the bilinear pooling network to obtain multi-modal features comprises the steps of:

mapping the image features and the text features to the same feature space, such that dimensions of the image features match dimensions of the text features;

inputting the image features and the text features with matched dimensions into the bilinear pooling network to output the fused multi-modal features, which are expressed as:

q＝Fusion(m，n)

wherein m is the image feature, n is the text feature, Fusion represents feature Fusion, and q is the multimodal feature.

7. The visual question-answering method for the appearance defects of the electric power equipment as claimed in claim 1, wherein the step of inputting the multi-modal features into the trained visual question-answering model to obtain a visual question-answering result comprises the steps of:

inputting the multi-modal features into the trained visual question-answering model;

determining answers to be selected according to the multi-modal characteristics, wherein the answers to be selected form a domain to be selected;

predicting the probability distribution of correct answers on the domain to be selected;

and according to the probability distribution, taking the answer to be selected corresponding to the maximum probability as the result of the visual question and answer.

8. A visual question-answering system for appearance defects of electrical equipment, the system comprising:

the image acquisition module is used for acquiring an image of the appearance defect of the power equipment and preprocessing the image;

the image feature extraction module is used for extracting features of the processed image to obtain image features;

the text feature extraction module is used for acquiring problem information related to the image according to the image and extracting features of the problem information to obtain text features;

the feature fusion module is used for fusing the image features and the text features based on a bilinear pooling network to obtain multi-modal features;

the model establishing module is used for establishing a visual question-answer model and training the visual question-answer model to obtain the trained visual question-answer model;

and the visual question-answering module is used for inputting the multi-modal characteristics into the trained visual question-answering model to obtain a visual question-answering result.

9. A visual question answering device for appearance defects of electric equipment is characterized in that: comprising a processor coupled with a memory, the memory storing program instructions that, when executed by the processor, implement the method of any of claims 1 to 7.

10. A computer-readable storage medium characterized by: comprising a program which, when run on a computer, causes the computer to perform the method of any one of claims 1 to 7.

Images