CN116541507A - A visual question answering method and system based on dynamic semantic graph neural network - Google Patents

Abstract

The invention discloses a visual question answering method and system based on a dynamic semantic graph neural network. The method includes: constructing a Glove word embedding model, a Bi-GRU model, a bilinear attention model, a graph attention network model, and a multi-layer A dynamic semantic graph neural network model of a perceptron model; based on a plurality of training samples, the dynamic semantic graph neural network model used for visual question answering prediction is trained to obtain a target visual question answering model; wherein each training sample includes: training The image and the training question text corresponding to the training image; the target image and the test question text corresponding to the target image are input into the target visual question answering model to obtain the target visual question answering result. The present invention can effectively improve the performance of the visual question answering model, and can improve the accuracy of the visual question answering.

Description

A visual question answering method and system based on dynamic semantic graph neural network

technical field

The invention relates to the technical fields of computer vision and natural language processing, in particular to a visual question answering method and system based on a dynamic semantic graph neural network.

Background technique

Visual Question Answering (VQA) is a comprehensive learning task involving computer vision and natural language processing, that is, given an image and a question about the image scene, the model predicts the correct answer. The visual question answering system first needs to analyze the semantic information of the question text, then perform multimodal fusion with the visual information extracted from the image, and finally get the correct answer through a classifier. Since existing methods usually simply use the attention mechanism for cross-modal fusion in the face of complex problems, these methods aim to improve cross-modal representations, without paying attention to how to construct the interactive relationship between image targets, and without spatial reasoning .

Traditional visual question answering methods usually use deep convolutional neural networks and recurrent neural networks to compute image and question representations respectively, and then use attention to perform cross-modal fusion between visual and linguistic features to predict the final answer. While these approaches can solve visual question answering tasks, they lack reasoning about explicitly detected objects and the interactive semantic and spatial relationships between them. Therefore, some papers propose to generate a probabilistic scene graph from the image and perform a series of inference operations on it. The generated scene graph contains the probability distribution of predicted objects, attributes and relationships. However, when constructing a scene graph, there is a lack of exploring local semantic features of fine-grained image regions. Extracting global image features will generate high-level image features, while ignoring discriminative local regions in the image, and failing to capture fine-grained distinguishing regions. At the same time, the traditional single solid-state scene graph can provide global semantic information from the image, but the semantic expression of the scene graph cannot adaptively change the key information in different problems, especially when there are too many instances of the scene graph, each instance Inter-reasoning relationships often cannot provide effective information for different questions, and will also put forward stricter requirements for later question-guided reasoning.

Therefore, it is urgent to provide a technical solution to solve the above technical problems.

Contents of the invention

In order to solve the above technical problems, the present invention provides a visual question answering method and system based on a dynamic semantic graph neural network.

A technical scheme of a visual question answering method based on a dynamic semantic graph neural network of the present invention is as follows:

Construct a dynamic semantic graph neural network model including Glove word embedding model, Bi-GRU model, bilinear attention model, graph attention network model and multi-layer perceptron model;

Based on a plurality of training samples, the dynamic semantic graph neural network model used for visual question answering prediction is trained to obtain a target visual question answering model; wherein each training sample includes: a training image and a training question text corresponding to the training image ;

Inputting the target image and the text of the question sentence to be tested corresponding to the target image into the target visual question answering model to obtain a target visual question answering result.

The beneficial effect of a kind of visual question answering method based on dynamic semantic graph neural network of the present invention is as follows:

The method of the present invention can effectively improve the performance of the visual question answering model, and can improve the accuracy of the visual question answering.

On the basis of the above solution, a visual question answering method based on a dynamic semantic graph neural network of the present invention can also be improved as follows.

Further, it also includes:

The plurality of training samples are obtained, and the real value of the visual question answering result corresponding to each training sample is marked to obtain the real visual question answering result of each training sample.

Further, based on a plurality of training samples, the dynamic semantic graph neural network model used for visual question answering prediction is trained to obtain the step of the target visual question answering model, including:

Input the training question text of any training sample into the Glove word embedding model, obtain a plurality of word embedding vectors of the training sample and input it into the Bi-GRU model to capture structural information, and obtain multiple word embedding vectors of the training sample Text word feature vectors and sentence-level feature vectors, and generate question graphs;

Input the nodes and edges in the real scene graph corresponding to the training image of any training sample to the Glove word embedding model to carry out word embedding processing, and obtain the initial node feature vector and the initial edge feature vector of the training sample scene graph;

Based on the bilinear attention model, process each text word feature vector and initial scene graph of any training sample to obtain the second scene graph of the training sample;

The second scene graph, a plurality of text word feature vectors and sentence-level feature vectors of any training sample are input into the graph attention network model and the multi-layer perceptron model in sequence to obtain the training visual question answering result of the training sample ;

Based on the training visual question answering result and the real visual question answering result of any training sample, the loss value of the training sample is obtained until the loss value of each training sample is obtained;

Based on all the loss values, the dynamic semantic graph neural network model is optimized to obtain the optimized dynamic semantic graph neural network model, and the optimized dynamic semantic graph neural network model is used as the dynamic semantic graph neural network Model and return to execute the step of inputting the training question text of any training sample to the Glove word embedding model, until the preset iterative training condition is met, the optimized dynamic semantic graph neural network model is determined as The target visual question answering model.

The beneficial effect of adopting the above-mentioned further technical solution is: to further solve the problem of adapting the scene graph of the image to different problems related to the image, make full use of the problem change to form a dynamic scene graph, and enhance the word-level semantic information in the problem and the image. The deep interaction of local regions of interest can improve the accuracy of visual question answering.

Further, input the training question text of any training sample into the Glove word embedding model, obtain a plurality of word embedding vectors of the training sample and input them into the Bi-GRU model to capture structural information, and obtain the training sample's Multiple text word feature vectors and sentence-level feature vectors, and the steps of generating a question graph include:

Separating word processing is performed on the training question text of any of the training samples to obtain a plurality of word data of the training sample ; where /> Indicates the number of word data of the training sample, /> Indicates that the first /> in the training sample training question text, /> Indicates the first /> in the training sample The /> in the training question text word data;

Each word data of any of the training samples respectively input to the Glove word embedding model to obtain the word embedding vector of each word data /> ; where /> Indicates the first /> in the training sample The /> in the training question text The word embedding vector corresponding to word data, /> , /> is the dimension of the word embedding vector;

Embedding vectors of multiple words of any of the training samples Input to the Bi-GRU model to capture structural information, and obtain the sentence-level feature vector corresponding to the training sample /> and word feature vectors /> , and generate the problem graph ; where /> , /> Indicates the first /> in the training sample The /> in the training question text text word feature vector, /> , /> is the dimension of the text word feature vector, /> Indicates the number of words.

Further, the nodes and edges in the real scene graph corresponding to the training image of any training sample are input to the Glove word embedding model for word embedding processing, and the training sample contains initial node feature vectors and initial edge feature vectors The initial scene graph steps include:

According to the training image of any training sample, the nodes and edges in the real scene graph corresponding to the label information are input to the Glove word embedding model for word embedding processing, and the training sample contains the initial node feature vector. and initial edge eigenvectors /> The initial scene graph for /> ; where /> , , /> , /> Indicates the number of candidate boxes, /> is the dimension of the label word embedding vector in the real scene graph, /> , /> .

Further, based on the bilinear attention model, each text word feature vector and the initial scene graph of any training sample are processed to obtain the second scene graph of the training sample, including:

The initial scene graph of any training sample respectively with each word feature vector of the training sample Input the bilinear attention model for processing to obtain the target node feature vector of the training sample /> ; The bilinear attention model is: /> ;

in, is a learnable weight matrix, /> On behalf of No. /> The first training question text in word data for /> The resulting attention score, /> , /> Indicates the first /> No. /> in the second scene diagram The target node feature vector of a target object;

Obtaining an initial scene graph of any training sample based on a first preset formula The initial edge eigenvector of Each initial relation eigenvector of /> The edge attention weights /> , and use the second preset formula and the side attention weights of the training sample/> , for the corresponding initial relation eigenvectors /> Transform respectively to obtain the target relationship feature vector of the training sample /> ;

in, , /> , /> ; The first preset formula is: /> , is a learnable weight matrix, /> On behalf of No. /> No. /> of the question word and initial relation feature vector/> The resulting attention score, pair /> normalized to get ; The second preset formula is: /> ; where /> Indicates the first /> object in the second scene graph /> to target /> The relationship feature vector of ;

According to the target node feature vector of any training sample and target relation eigenvector/> , to obtain multiple second scene graphs of the training sample /> ; where /> .

Further, the second scene graph, multiple text word feature vectors and sentence-level feature vectors of any training sample are sequentially input into the graph attention network model and the multi-layer perceptron model to obtain the training data corresponding to the training sample. Steps for visual question answering results, including:

A plurality of text word feature vectors and the second scene graph of any training sample are input into the graph attention network model for iterative processing to obtain a plurality of target word feature vectors, and each of them is calculated by the multi-layer perceptron model The target word feature vector and the weight score of adjacent nodes, and filter all adjacent nodes based on the channel attention mechanism, and obtain and combine multiple node feature vectors Aggregate to get visual Q&A target features/> , and the sentence-level feature vector corresponding to the training sample /> and visual question answering target features/> Input to softmax for processing, and obtain the training visual question answering result corresponding to the training sample /> ; where /> , .

A technical scheme of a visual question answering system based on a dynamic semantic graph neural network of the present invention is as follows:

Including: construction module, training module and operation module;

The building block is used for: constructing the dynamic semantic graph neural network model comprising Glove word embedding model, Bi-GRU model, bilinear attention model, graph attention network model and multi-layer perceptron model;

The training module is used to: based on a plurality of training samples, train the dynamic semantic graph neural network model used for visual question answering prediction to obtain a target visual question answering model; wherein each training sample includes: training images and the training The training question text corresponding to the image;

The running module is configured to: input the target image and the text of the question sentence corresponding to the target image into the target visual question answering model to obtain the target visual question answering result.

The beneficial effect of a kind of visual question answering system based on dynamic semantic graph neural network of the present invention is as follows:

The system of the present invention can effectively improve the performance of the visual question answering model, and can improve the accuracy of the visual question answering.

On the basis of the above solution, a visual question answering system based on a dynamic semantic graph neural network of the present invention can also be improved as follows.

Further, it also includes: a processing module;

The processing module is configured to: obtain the plurality of training samples, and mark the real value of the visual question answering result corresponding to each training sample, so as to obtain the real visual question answering result of each training sample.

Further, the training module includes: a first training module, a second training module, a third training module, a fourth training module, a loss calculation module and an iterative training module;

The first training module is used to: input the training question text of any training sample into the Glove word embedding model, obtain a plurality of word embedding vectors of the training sample and input them into the Bi-GRU model to capture the structure Information, get multiple text word feature vectors and sentence-level feature vectors of the training sample, and generate a question graph;

The second training module is used to: input the nodes and edges in the real scene diagram corresponding to the training image of any training sample to the Glove word embedding model to carry out word embedding processing, and obtain the initial node of the training sample Initial scene graph of eigenvectors and initial edge eigenvectors;

The third training module is used to: process each text word feature vector and initial scene graph of any training sample based on a bilinear attention model to obtain a second scene graph of the training sample;

The fourth training module is used to: input the second scene graph, a plurality of text word feature vectors and sentence-level feature vectors of any training sample into the graph attention network model and the multi-layer perceptron model in sequence, Obtain the training visual question answering result of the training sample;

The loss calculation module is used to: obtain the loss value of the training sample based on the training visual question answering result and the real visual question answering result of any training sample, until the loss value of each training sample is obtained;

The iterative training module is used to: optimize the dynamic semantic graph neural network model based on all loss values, obtain the optimized dynamic semantic graph neural network model, and transfer the optimized dynamic semantic graph neural network model to As the dynamic semantic graph neural network model and return to the step of inputting the training question text of any training sample to the Glove word embedding model, until the preset iterative training condition is met, the optimized The dynamic semantic graph neural network model is determined as the target visual question answering model.

Description of drawings

Fig. 1 shows a schematic flow chart of an embodiment of a visual question answering method based on a dynamic semantic graph neural network provided by the present invention;

Fig. 2 shows a schematic flow chart of step 120 in an embodiment of a visual question answering method based on a dynamic semantic graph neural network provided by the present invention;

FIG. 3 shows a schematic structural diagram of an embodiment of a visual question answering system based on a dynamic semantic graph neural network provided by the present invention.

Detailed ways

Fig. 1 shows a schematic flowchart of an embodiment of a visual question answering method based on a dynamic semantic graph neural network provided by the present invention. As shown in Figure 1, the method includes the following steps:

Step 110: Construct a dynamic semantic graph neural network model including Glove word embedding model, Bi-GRU model, bilinear attention model, graph attention network model and multi-layer perceptron model.

Among them, the dynamic semantic graph neural network model in this embodiment is composed of multiple models, including but not limited to: Glove word embedding model, Bi-GRU (bidirectional dynamic gated recurrent unit) model, bilinear attention model, graph Pay attention to network models and multi-layer perceptron models. The function of each model is described in detail later, so I won't go into details here.

Step 120: Based on a plurality of training samples, train the dynamic semantic graph neural network model used for visual question answering prediction to obtain a target visual question answering model.

Among them, ① each training sample includes: the initial scene graph corresponding to the training image and the training question text corresponding to the training image. ②Initial scene graph: The image scene graph used for visual question answering training, which uses a graph structure to represent the relationship between different targets in the training image. ③ The training question text is: the question text associated with the training image, and the number of the training question text corresponding to any training image is at least one. ③ The target visual question answering model is: the visual question answering model obtained after training with training samples.

Step 130: Input the target image and the text of the question to be tested corresponding to the target image into the target visual question answering model to obtain the target visual question answering result.

Among them, ① The target image corresponds to an initial scene graph, and the initial scene graph is: an image scene graph for visual question answering test. ② The question text to be tested is: at least one question text corresponding to the target image. ③ The target visual question answering result is: the final answer corresponding to the text of the question sentence to be tested, which is the predicted value.

For example, when the content of a target image contains the text of the question sentence to be tested as "Where is the green tent in the picture?", the target visual question answer result is: "bottom" (the corresponding real value is: bottom); When the content of the target image contains the text of the question sentence to be tested is "Is the person in the picture wearing a hat?", the target visual question answer result is: "yes" (the corresponding true value is: yes). For another example, when the content of a target image contains the text of the question sentence to be tested as "What device does the personhold?", the result of the target visual question answer is: "camera" (the corresponding real value is: camera); when When the content of the target image contains the text of the question to be tested as "What is the color of the car?", the target visual question answer result is: "white" (the corresponding real value is: white).

Preferably, it also includes:

The plurality of training samples are obtained, and the real value of the visual question answering result corresponding to each training sample is marked to obtain the real visual question answering result of each training sample.

Among them, the real visual question answering result is: the real value of the visual question answering result of each training sample marked by the user.

Preferably, as shown in Figure 2, step 120 includes:

Step 121: Input the training question text of any training sample into the Glove word embedding model, obtain multiple word embedding vectors of the training sample and input them into the Bi-GRU model to capture structural information, and obtain the training sample Multiple text word feature vectors and sentence-level feature vectors of , and generate a question graph.

Among them, ①Glove word embedding model is used to generate corresponding word embedding vectors for each word in the question text. ②The Bi-GRU model is used to capture the structural information of the word embedding vector, so as to obtain the text word feature vector and sentence-level feature vector corresponding to the question text to generate a question graph.

Specifically, step 121 includes:

Step 1211: Perform word separation processing on the training question text of any training sample to obtain multiple word data of the training sample .

in, Indicates the number of word data of the training sample, /> Indicates that the first /> in the training sample training question text, /> Indicates the first /> in the training sample The /> in the training question text word data.

Specifically, perform word separation processing on each training question text corresponding to any training sample, convert all words in each training question text into lowercase, and obtain multiple word data corresponding to the training sample , /> Indicates the first /> in the training sample The total number of word data in the training question text.

It should be noted that, in this embodiment, the maximum length of the training question text is 14 (ie is 14), and can also be used for /> Make settings, there are no restrictions here.

Step 1212: each word data of any training sample respectively input to the Glove word embedding model to obtain the word embedding vector of each word data /> .

in, Indicates the first /> in the training sample The /> in the training question text The word embedding vector corresponding to word data, /> , /> is the dimension of the word embedding vector.

Step 1213: embedding a plurality of words of any training sample into vectors Input to the Bi-GRU model to capture structural information, and obtain the sentence-level feature vector corresponding to the training sample /> and word feature vectors /> , and generate the problem graph /> .

in, , Indicates the first /> in the training sample The /> in the training question text text word feature vector, /> , is the dimension of the text word feature vector, /> Indicates the number of words.

Step 122: Input the nodes and edges in the real scene graph corresponding to the training image of any of the training samples to the Glove word embedding model for word embedding processing, and obtain the initial node feature vector and initial edge features of the training sample Vector initial scene graph.

Specifically, the nodes and edges in the real scene graph containing label information corresponding to the training image of any training sample are input to the Glove word embedding model for word embedding processing, and the initial node feature vector containing the training sample is obtained. and initial edge eigenvectors /> The initial scene graph for /> .

in, , /> , /> Indicates the number of candidate boxes, /> is the dimension of the label word embedding vector in the real scene graph, /> , /> .

Step 123: Based on the bilinear attention model, process each text word feature vector and the initial scene graph of any training sample to obtain a second scene graph of the training sample.

Specifically, the initial scene graph of any training sample Respectively with each word feature vector of the training sample /> Input the bilinear attention model for processing to obtain the target node feature vector of the training sample ;

The bilinear attention model is: ;

in, is a learnable weight matrix, /> On behalf of No. /> The first training question text in word data for /> The resulting attention score, /> , /> Indicates the first /> No. /> in the second scene diagram The target node feature vector of a target object;

Obtaining an initial scene graph of any training sample based on a first preset formula The initial edge eigenvector of Each initial relation eigenvector of /> The edge attention weights /> , and use the second preset formula and the side attention weights of the training sample/> , for the corresponding initial relation eigenvectors /> Transform respectively to obtain the target relationship feature vector of the training sample /> ;

in, , /> , /> ; The first preset formula is: /> , is a learnable weight matrix, /> On behalf of No. /> No. /> of the question word and initial relation feature vector/> The resulting attention score, pair /> normalized to get ;

The second preset formula is: ; where /> Indicates the first /> object in the second scene graph /> to target /> The relationship feature vector of ;

According to the target node feature vector of any training sample and target relation eigenvector/> , to obtain multiple second scene graphs of the training sample /> ; where /> .

It should be noted that: ①Introducing a bilinear attention model, the first image of the same image will be question/> Word vectors in /> Enter the bilinear attention model with the initial scene graph, and after fusing the keywords in the question, get the updated node features/> with edge eigenvectors /> , to highlight the position of the key nodes of the scene graph, so as to generate the corresponding second scene graph according to the problem /> .

Firstly, the attention weight of the question and the node is calculated, which will generate the attention score of different image regions based on the question. Through the attention score, we can only focus on some scene instances most relevant to the given different questions: ; where the relevance score is computed /> After that, normalize them to get attention weights /> , adapting the node characteristics of the problem /> can be entered by the feature /> and attention weights The weighted average of is calculated as:

;

;

;

②In addition to the need to seek attention weights for nodes, attention mechanisms must also be applied to relationship edges, because relationships are equally important for answering questions. In order to capture the deep interactive relationship between the problem and the scene graph, find and problem For related relationship edges, use the following formula to calculate edge attention weights /> :

;

in, is a learnable weight matrix. /> On behalf of No. /> No. /> of the question words and initial edge feature vectors/> The resulting attention score will then be /> Normalize to get attention weights /> , initial scene graph edge feature vector /> In the fusion section /> word in question /> converts to /> :

③ From the above method can be obtained according to the The second scene graph established by this problem is , /> , /> . The second scene graph adds sentence-level interaction with different questions, making the original complex scene graph more sparse, paying more attention to the content of the question, and predicting the answer more effectively.

Step 124: Input the second scene graph, multiple text word feature vectors, and sentence-level feature vectors of any training sample into the graph attention network model and the multi-layer perceptron model in sequence to obtain the training data of the training sample. Visual question answering results.

Specifically, a plurality of text word feature vectors and a second scene graph of any training sample are input into the graph attention network model for iterative processing to obtain a plurality of target word feature vectors, and pass the multi-layer perceptron model Calculate the weight score of each target word feature vector and adjacent nodes, and filter all adjacent nodes based on the channel attention mechanism, and obtain multiple node feature vectors Aggregate to get visual Q&A target features/> , and the sentence-level feature vector corresponding to the training sample /> and visual question answering target features/> Input to softmax for processing, and obtain the training visual question answering result corresponding to the training sample /> .

It should be noted:

① Adopt graph attention network (GAT). Unlike the graph attention network, the update of the scene graph nodes in this embodiment is based on the word data of the question text Generated word feature vectors /> For iterative operation, instructions play two roles in the node update process. First, instruction vectors are introduced when generating attention distribution to control how much information can flow into adjacent nodes. Second, channel attention is used to filter unimportant information when nodes are updated.

Sequential Fusion Using In-Graph Attention Network Model Instruction vectors are used to update the nodes. at /> During the iteration, we will pass the /> No. /> of the question instruction vectors /> Connect to page /> Each node output by iteration and edge features/> .

in, , /> Respectively represent the fusion of the first /> No. /> of the question instruction vectors /> node features and edge features, and as the first /> input to the iterative process. /> Represents a concatenation of vectors.

②Calculate the weight scores of nodes and adjacent nodes, that is, nodes feature pair node /> The importance of two of nodes /> For fusion, the fusion method in this paper adopts the multi-layer perceptron (MLP) model. Use softmax for all neighbor nodes /> For normalization, the attention function expression is:

; where /> Indicated at the /> node after iterations /> feature pair node /> attention score, /> represents a node /> The set of all neighbor nodes of .

③ Before updating the central node according to the neighbor nodes, we introduce the channel attention mechanism to filter the neighbor nodes on the channel. The graph attention network (GAT) calculates the weighted average of the transformed features of neighbor nodes as a node A new vector representation of : /> ; where /> is a trainable weight matrix, /> is a non-linear activation function. passing by /> After the iterative neural network, the model can finally get the final state of all nodes /> .

④ get message delivery After iteration, first pass the message vector to the final state of all graph nodes Aggregate to final state /> , and then use the question vector to predict the answer /> :

in, is the predicted answer probability, and the model finally selects the label with the highest probability as the final predicted answer (training visual question answering result).

Step 125: Obtain the loss value of the training sample based on the training visual question answering result and the real visual question answering result of any training sample, until the loss value of each training sample is obtained.

Among them, ① the result of training visual question answering is: the predicted value of the visual question answering corresponding to the training sample. ②The result of the real visual question answering is: the real value of the visual question answering corresponding to the training sample, that is, the annotation information of the training sample.

Specifically, based on the preset loss function, the training visual question answering result and the real visual question answering result of any training sample, the loss value of the training sample is obtained, and the above process is repeated until the loss value of each training sample is obtained.

Step 126: Based on all loss values, optimize the dynamic semantic graph neural network model to obtain an optimized dynamic semantic graph neural network model, and use the optimized dynamic semantic graph neural network model as the dynamic semantic graph neural network model The graph neural network model returns to step 121 until the preset iterative training condition is met, and the optimized dynamic semantic graph neural network model is determined as the target visual question answering model.

Specifically, based on all loss values, the dynamic semantic graph neural network model is optimized to obtain the optimized dynamic semantic graph neural network model, and it is judged whether the optimized dynamic semantic graph neural network model meets the preset iterative training conditions, and if so, Then the optimized dynamic semantic graph neural network model is determined as the target visual question answering model; if not, the optimized dynamic semantic graph neural network model is used as the dynamic semantic graph neural network model and returns to step 121 until the preset iteration is satisfied When training conditions, the optimized dynamic semantic graph neural network model is determined as the target visual question answering model.

It should be noted that this embodiment tests the proposed algorithm on the public GQA dataset. The images in the GQA dataset come from the Visual Genome dataset, and a normalization process is applied starting from the scene graph annotations provided in Visual Genome. It contains 113K images, about 12K questions, divided into about 80%, 10%, 10% for training, validation and testing. The overall vocabulary consists of 3097 words, including 1702 object classes, 310 relations and 610 object attributes. Each image in the training set and verification set of GQA is equipped with information annotations of the scene graph, which describe the labels and attributes of the target objects in the scene, as well as the paired relationship information between target entities.

The method of this embodiment is compared with the following mainstream visual question answering algorithm models based on GQA datasets, including Human, LCGN and GraphVQA.

As shown in Table 1 below, the method of this example shows better performance compared to other state-of-the-art methods. The following results show that the method of this embodiment can adaptively model the semantic dependencies between visual objects through the question-guided dynamic scene graph, better understand the scene related to the question, and effectively improve the accuracy of visual question answering.

Table 1:

The technical solution of this embodiment enables the scene graph of the image to adapt to different problems related to the image through the collaborative attention mechanism, and makes full use of the problem change to form a dynamic scene graph, which enhances the word-level semantic information in the problem and the local image on the image. The deep interaction of the region of interest can effectively improve the performance of the visual question answering model and improve the accuracy of the visual question answering.

FIG. 3 shows a schematic structural diagram of an embodiment of a visual question answering system based on a dynamic semantic graph neural network provided by the present invention. As shown in FIG. 3 , the system 200 includes: a construction module 210 , a training module 220 and an operation module 230 .

The building block 210 is used for: constructing the dynamic semantic graph neural network model comprising Glove word embedding model, Bi-GRU model, bilinear attention model, graph attention network model and multi-layer perceptron model;

The training module 220 is used for: based on a plurality of training samples, the dynamic semantic graph neural network model used for visual question answering prediction is trained to obtain a target visual question answering model; wherein, each training sample includes: a training image and the The training question text corresponding to the training image;

The running module 230 is configured to: input the target image and the text of the question sentence corresponding to the target image into the target visual question answering model to obtain the target visual question answering result.

Preferably, it also includes: a processing module;

The processing module is configured to: obtain the plurality of training samples, and mark the real value of the visual question answering result corresponding to each training sample, so as to obtain the real visual question answering result of each training sample.

Preferably, the training module 220 includes: a first training module, a second training module, a third training module, a fourth training module, a loss calculation module and an iterative training module;

The first training module is used to: input the training question text of any training sample into the Glove word embedding model, obtain a plurality of word embedding vectors of the training sample and input them into the Bi-GRU model for structural information Capture, obtain multiple text word feature vectors and sentence-level feature vectors of the training sample, and generate a question graph;

The second training module is used to: input the nodes and edges in the real scene diagram corresponding to the training image of any training sample to the Glove word embedding model to carry out word embedding processing, and obtain the initial node of the training sample Initial scene graph of eigenvectors and initial edge eigenvectors;

The third training module is used to: process each text word feature vector and initial scene graph of any training sample based on a bilinear attention model to obtain a second scene graph of the training sample;

The fourth training module is used to: input the second scene graph, a plurality of text word feature vectors and sentence-level feature vectors of any training sample into the graph attention network model and the multi-layer perceptron model in sequence, Obtain the training visual question answering result of the training sample;

The loss calculation module is used to: obtain the loss value of the training sample based on the training visual question answering result and the real visual question answering result of any training sample, until the loss value of each training sample is obtained;

The iterative training module is used to: optimize the dynamic semantic graph neural network model based on all loss values, obtain the optimized dynamic semantic graph neural network model, and transfer the optimized dynamic semantic graph neural network model to As the dynamic semantic graph neural network model and return to the step of inputting the training question text of any training sample to the Glove word embedding model, until the preset iterative training condition is met, the optimized The dynamic semantic graph neural network model is determined as the target visual question answering model.

The technical solution of this embodiment enables the scene graph of the image to adapt to different problems related to the image through the collaborative attention mechanism, and makes full use of the problem change to form a dynamic scene graph, which enhances the word-level semantic information in the problem and the local image on the image. The deep interaction of the region of interest can effectively improve the performance of the visual question answering model and improve the accuracy of the visual question answering.

For the above-mentioned steps of realizing the corresponding functions of each parameter and each module in a visual question answering system 200 based on a dynamic semantic graph neural network in this embodiment, you can refer to the above about a visual question answering method based on a dynamic semantic graph neural network The parameters and steps in the embodiments will not be repeated here.

A storage medium provided by an embodiment of the present invention includes: an instruction is stored in the storage medium, and when the computer reads the instruction, the computer is made to perform steps such as a visual question answering method based on a dynamic semantic graph neural network, specifically Reference can be made to the parameters and steps in the above embodiment of a visual question answering method based on a dynamic semantic graph neural network, and details are not repeated here.

Computer storage media such as: USB flash drive, mobile hard disk, etc.

Those skilled in the art know that the present invention can be implemented as a method, system and storage medium.

Therefore, the present invention can be embodied in the following forms, namely: it can be complete hardware, it can also be complete software (including firmware, resident software, microcode, etc.), and it can also be a combination of hardware and software. Called a "circuit", "module" or "system". Furthermore, in some embodiments, the present invention can also be implemented in the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied therein. Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more leads, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), Erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention, those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

Claims (10)

1. A visual question-answering method based on a dynamic semantic graph neural network is characterized by comprising the following steps:

constructing a dynamic semantic graph neural network model comprising a Glove word embedding model, a Bi-GRU model, a bilinear attention model, a graph meaning network model and a multi-layer perceptron model;

training the dynamic semantic graph neural network model for visual question-answering prediction based on a plurality of training samples to obtain a target visual question-answering model; wherein each training sample comprises: training images and training question texts corresponding to the training images;

and inputting the target image and the text of the question to be tested corresponding to the target image into the target visual question-answering model to obtain a target visual question-answering result.

2. The visual question-answering method based on dynamic semantic graph neural network according to claim 1, further comprising:

and acquiring the plurality of training samples, and marking the true value of the visual question-answer result corresponding to each training sample to obtain the true visual question-answer result of each training sample.

3. The visual question-answering method based on a dynamic semantic graph neural network according to claim 2, wherein the step of training the dynamic semantic graph neural network model for visual question-answering prediction based on a plurality of training samples to obtain a target visual question-answering model comprises:

Inputting training question text of any training sample into the Glove word embedding model to obtain a plurality of word embedding vectors of the training sample, inputting the word embedding vectors into the Bi-GRU model to capture structural information, obtaining a plurality of text word feature vectors and sentence-level feature vectors of the training sample, and generating a question graph;

inputting nodes and edges in the real scene graph corresponding to the training image of any training sample into the Glove word embedding model to perform word embedding processing to obtain an initial scene graph of the training sample, wherein the initial scene graph comprises initial node feature vectors and initial edge feature vectors;

processing each text word feature vector of any training sample and the initial scene graph based on the bilinear attention model to obtain a second scene graph of the training sample;

sequentially inputting a second scene graph, a plurality of text word feature vectors and sentence level feature vectors of any training sample into the graph annotation network model and the multi-layer perceptron model to obtain a training visual question-answering result of the training sample;

obtaining a loss value of the training sample based on the training visual question-answering result and the real visual question-answering result of any training sample until the loss value of each training sample is obtained;

And optimizing the dynamic semantic graph neural network model based on all the loss values to obtain an optimized dynamic semantic graph neural network model, taking the optimized dynamic semantic graph neural network model as the dynamic semantic graph neural network model, and returning to execute the step of inputting training question text of any training sample into the Glove word embedding model until a preset iteration training condition is met, and determining the optimized dynamic semantic graph neural network model as the target vision question-answering model.

4. The visual question-answering method based on dynamic semantic graph neural network according to claim 3, wherein the step of inputting training question text of any training sample to the Glove word embedding model to obtain a plurality of word embedding vectors of the training sample and inputting the word embedding vectors to the Bi-GRU model to capture structural information to obtain a plurality of text word feature vectors and sentence-level feature vectors of the training sample, and generating a question graph comprises the steps of:

performing separator word processing on the training question text of any training sample to obtain a plurality of word data of the training sampleThe method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Number of word data representing the training sample, +. >Indicating +.>Training question text->Representing the +.>The +.f. in the text of the training question>Individual word data;

each word data of any training sample is processedRespectively inputting the words into the Glove word embedding model to obtain each wordWord embedding vector of data->The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Representing the +.>The first training question textWord embedding vector corresponding to the individual word data, +.>，/>Embedding a dimension of a vector for a word;

embedding multiple words of the training samples into vectorsInputting the sentence-level feature vector into the Bi-GRU model to capture structural information, and obtaining the sentence-level feature vector corresponding to the training sample>Sum word feature vector->And generates the problem graphThe method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)> ，/>Representing the +.>The +.f. in the text of the training question>Individual text word feature vectors,/>，/>For the dimension of the text word feature vector, +.>Representing the number of words.

5. The visual question-answering method based on a dynamic semantic graph neural network according to claim 4, wherein the step of inputting nodes and edges in the real scene graph corresponding to the training image of any training sample to the Glove word embedding model to perform word embedding processing to obtain an initial scene graph of the training sample, wherein the initial scene graph comprises initial node feature vectors and initial edge feature vectors comprises:

Inputting nodes and edges in the real scene graph corresponding to the training image of any training sample and containing the labeling information into the Glove word embedding model to perform word embedding processing to obtain an initial node feature vector containing the training sampleAnd an initial edge feature vector +.>Is->The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>，，/>，/>Representing the number of candidate boxes, +.>Embedding vector dimension for tag words in real scene graph, < ->，/>。

6. The visual question-answering method based on a dynamic semantic graph neural network according to claim 5, wherein the step of processing each text word feature vector of any training sample with an initial scene graph based on a bilinear attention model to obtain a second scene graph of the training sample comprises:

an initial scene graph of any training sampleEach word feature vector associated with the training sample>Inputting the bilinear attention model for processing to obtain the target node characteristic vector of the training sample>The method comprises the steps of carrying out a first treatment on the surface of the The bilinear attention model is: />；

Wherein,,is a weight matrix which can be learned, +.>Represents->The +.f. in the text of the training question>Individual word data +.>The resulting attention score,/- >，/>Indicate->Second scene graph +.>Target node feature vectors of the individual target objects;

based on a first preset formula, acquiring an initial scene graph of any training sampleIs>Is +.>Is +.>And side note force weight of the training sample using a second predetermined formula ++>For the corresponding initial relation feature vector +.>Respectively converting to obtain target relation feature vector +.>；

Wherein,,，/>，/>the method comprises the steps of carrying out a first treatment on the surface of the The first preset formula is:

，is a weight matrix which can be learned, +.>Represents->First of all of the questions>Individual words and initial relation feature vector +.>The resulting attention score, for->Normalizing to obtainThe method comprises the steps of carrying out a first treatment on the surface of the The second preset formula is: />The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Indicate->Target +.>To the target->Is a relation feature vector of (1);

target node feature vector according to any training sampleAnd target relation feature vector->Obtaining a plurality of second scene graphs of the training sample +.>The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>。

7. The visual question-answering method based on the dynamic semantic graph neural network according to claim 6, wherein the step of sequentially inputting the second scene graph, the plurality of text word feature vectors and the sentence-level feature vectors of any training sample into the graph-meaning network model and the multi-layer perceptron model to obtain the training visual question-answering result corresponding to the training sample comprises the following steps:

Inputting the text word feature vectors and the second scene graph of any training sample into the graph annotation network model for iterative processing to obtain a plurality of target word feature vectors, calculating the weight scores of each target word feature vector and adjacent nodes through the multi-layer perceptron model, filtering all adjacent nodes based on a channel attention mechanism to obtain a plurality of node feature vectorsPolymerizing to obtain visual question-answering target feature +.>And the sentence-level feature vector corresponding to the training sample is +.>And visual question-answering target feature->Inputting into softmax for processing to obtain training visual question-answer result corresponding to the training sample>The method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>，。

8. A visual question-answering system based on a dynamic semantic graph neural network, comprising: the system comprises a construction module, a training module and an operation module;

the construction module is used for: constructing a dynamic semantic graph neural network model comprising a Glove word embedding model, a Bi-GRU model, a bilinear attention model, a graph meaning network model and a multi-layer perceptron model;

the training module is used for: training the dynamic semantic graph neural network model for visual question-answering prediction based on a plurality of training samples to obtain a target visual question-answering model; wherein each training sample comprises: training images and training question texts corresponding to the training images;

The operation module is used for: and inputting the target image and the text of the question to be tested corresponding to the target image into the target visual question-answering model to obtain a target visual question-answering result.

9. The visual question-answering system based on dynamic semantic graph neural network according to claim 8, further comprising: a processing module;

the processing module is used for: and acquiring the plurality of training samples, and marking the true value of the visual question-answer result corresponding to each training sample to obtain the true visual question-answer result of each training sample.

10. The visual question-answering system based on dynamic semantic graph neural network according to claim 9, wherein the training module comprises: the system comprises a first training module, a second training module, a third training module, a fourth training module, a loss calculation module and an iteration training module;

the first training module is used for: inputting training question text of any training sample into the Glove word embedding model to obtain a plurality of word embedding vectors of the training sample, inputting the word embedding vectors into the Bi-GRU model to capture structural information, obtaining a plurality of text word feature vectors and sentence-level feature vectors of the training sample, and generating a question graph;

The second training module is used for: inputting nodes and edges in the real scene graph corresponding to the training image of any training sample into the Glove word embedding model to perform word embedding processing to obtain an initial scene graph of the training sample, wherein the initial scene graph comprises initial node feature vectors and initial edge feature vectors;

the third training module is used for: processing each text word feature vector of any training sample and the initial scene graph based on the bilinear attention model to obtain a second scene graph of the training sample;

the fourth training module is used for: sequentially inputting a second scene graph, a plurality of text word feature vectors and sentence level feature vectors of any training sample into the graph annotation network model and the multi-layer perceptron model to obtain a training visual question-answering result of the training sample;

the loss calculation module is used for: obtaining a loss value of the training sample based on the training visual question-answering result and the real visual question-answering result of any training sample until the loss value of each training sample is obtained;

the iterative training module is used for: and optimizing the dynamic semantic graph neural network model based on all the loss values to obtain an optimized dynamic semantic graph neural network model, taking the optimized dynamic semantic graph neural network model as the dynamic semantic graph neural network model, and returning to execute the step of inputting training question text of any training sample into the Glove word embedding model until a preset iteration training condition is met, and determining the optimized dynamic semantic graph neural network model as the target vision question-answering model.