CN114757209B - Man-machine interaction instruction analysis method and device based on multi-mode semantic role recognition - Google Patents

Abstract

The invention provides a human-computer interaction instruction parsing method and device based on multimodal semantic role recognition, and relates to the technical field of semantic analysis in natural language processing. Including: constructing a complete set of instruction semantic role annotation paradigm according to the characteristics of human-computer interaction instructions; according to the instruction semantic role annotation paradigm, combined with image acquisition, the single-modal form of the semantic role annotation model is extended to visual text multimodality The multi-modal form of visual text of the semantic role annotation model is trained and learned, and the multi-modal semantic role recognition is completed to perform semantic analysis of human-computer interaction instructions. The invention innovatively attempts to use the paradigm of multi-modal semantic role labeling to perform semantic analysis on human-computer interaction instructions, so as to convert the original machine-incomprehensible instructions into machine-understandable semantic structured output, which is more convenient and safe. , Quickly execute the user's intent.

Description

Human-computer interaction instruction analysis method and device based on multi-modal semantic role recognition

technical field

The invention relates to the technical field of semantic analysis in natural language processing, in particular to a method and device for man-machine interaction instruction analysis based on multi-modal semantic role recognition.

Background technique

Semantic role labeling is a shallow semantic analysis technique used to extract the predicate-argument structure implied in a sentence. Among them, the predicate is the core word in a sentence that can trigger a semantic event, and the argument is the role that participates in the semantic event, including the agent and the recipient. In general, the core of semantic role labeling technology is to enable machines to understand "who did what, when and where" in a sentence. At present, there are already many applications that try to use semantic role labeling as a key link in their technical links, such as knowledge question answering, dialogue robots, machine translation, etc.

With the development of technology, human-computer interaction technology has gradually become an important way for users to control unmanned devices (such as robots and drones). Commands are issued through voice, so that the unmanned device understands the operator's intentions and executes the corresponding command, which can free the hands of the operator and control the unmanned device more conveniently, safely and quickly. However, the development of existing instruction parsing technologies is limited, and it is impossible to parse out machine-understandable semantic structures from instructions in a targeted manner. The present invention plans to use the advantages of the semantic role labeling technology itself to achieve high-precision analysis of the intent and semantics of control instructions, so that unmanned devices can better serve users and perform more abstract and difficult operations.

At present, the overall process of semantic role labeling is mainly divided into two types. One is based on the pipeline method, using the sequence labeling method to identify the predicates in the sentence, and then identify the semantic role (argument) in the sentence, which will lead to error propagation. The problem is serious. The other is to construct a semantic graph to simultaneously extract predicates and their corresponding semantic roles. First, enumerate all possible predicate and argument candidate fragments of a sentence as nodes in the graph, and then use the relationship between predicate fragments and semantic role fragments to The semantic role relationship among them is used as the edges in the graph, and finally the semantic graph formed by precise decoding can get a structured output. Most of the current unmanned devices have both vision and language perception. However, most of the existing semantic role labeling methods are oriented to a single text setting, ignoring the important complementary relationship between image information and text information.

At present, the annotation paradigms of semantic role annotation datasets are mostly oriented to the general field, and there are still large gaps in special fields such as unmanned equipment command and control instructions.

Contents of the invention

Aiming at the problem that there is still a large gap under the command and control command of unmanned equipment in the prior art, the present invention proposes a method and device for man-machine interaction command parsing based on multi-modal semantic role recognition.

In order to solve the above technical problems, the present invention provides the following technical solutions:

On the one hand, a method for parsing human-computer interaction instructions based on multimodal semantic role recognition is provided, and the method is applied to electronic devices, including the following steps:

S1: According to the characteristics of human-computer interaction instructions, construct an instruction semantic role annotation paradigm;

S2: According to the semantic role labeling paradigm of the instruction, combined with image collection, the single-modal form of the semantic role labeling model is extended to a multi-modal form of visual text;

S3: Train and learn the visual text multimodal form of the semantic role labeling model, and complete the multimodal semantic role recognition and semantic analysis of human-computer interaction instructions.

Optionally, in step S1, according to the characteristics of the human-computer interaction instruction, construct an instruction semantic role labeling paradigm, including:

S11: Use the labeling method of VerbAtlas semantic role labeling data as the labeling benchmark;

S12: Extend and modify the pre-stored Chinese semantic role labeling paradigm, so that the extended and modified Chinese semantic role labeling paradigm is applicable to the semantic analysis of human-computer interaction instructions, and obtain the instruction semantic role labeling paradigm.

Optionally, in step S2, according to the instruction semantic role labeling paradigm, combined with image collection, the single-modal form of the semantic role labeling model is extended to a visual-text dual-modal form, including:

S21: According to the instruction semantic role annotation paradigm, collect images through an unmanned system, use Faster-RCNN to obtain a sequence target area, form the sequence target area into an image area sequence, and extract features of the image sequence;

S22: Use the extracted image sequence features to assist in identifying the semantic role on the semantic text side, and extend the single-modal form of the semantic role labeling model to a dual-modal form of visual text.

Optionally, in step S3, train and learn the visual text multimodal form of the semantic role labeling model, complete multimodal semantic role recognition and perform semantic analysis on human-computer interaction instructions, including:

S31: Construct a pre-training model according to the visual text multimodal form of the semantic role labeling model;

；利用BERT预训练模型 对所述指令I进行编码，获得指令I中每个词对应的词向量序列

； S32: Instructions for inputting the pre-trained model

; Utilize the BERT pre-training model to encode the instruction I, and obtain the word vector sequence corresponding to each word in the instruction I

;

，其中

， 获得每个跨度的特征向量；其中，所述跨度的大小均为预设值； S33: enumerate all spans in instruction I

,in

, to obtain the feature vector of each span; wherein, the size of the span is a preset value;

S34: According to the feature vector of each span, generate candidate vectors corresponding to the predicate node and the semantic role node in the semantic graph;

S35: Introduce a loss function to improve the training loss of the model, and complete multi-modal semantic role recognition and semantic analysis of human-computer interaction instructions.

以及语义角色候 选向量

，其中：

；

。 Optionally, in S34, two different MLP layers are used to obtain predicate candidate vectors respectively

and semantic role candidate vectors

,in:

;

.

Optionally, in S35, a loss function is introduced to improve the training loss of the model, including:

Construct a semantic role labeling loss function to judge the integrity of the predicate and argument structure predicted by the model;

、语义角色

以及两者关系

的三元组

进行打分；交叉熵来计算每个三元组的损失，所述语 义角色标注损失函数如下述公式（1）所示： Wherein, comprise a MLP layer score layer and a Biaffine score layer; The MLP layer score layer is used to judge the semantic framework of the current predicate node, and the Biaffine score layer is used to each predicate in the sentence

, semantic role

and the relationship between the two

triplet of

Scoring; cross-entropy to calculate the loss of each triplet, the semantic role labeling loss function is shown in the following formula (1):

Optionally, in S35, a loss function is introduced to improve the training loss of the model, including:

Construct a modal matching function for modal matching of image and text cross-modal feature pairs. The label of this function is defined as if the segment corresponding to the semantic role contains the object corresponding to the target area, then the output label is 1, otherwise The label is 0; through the paradigm of multi-task learning, define the loss function of the mode matching function as shown in the following formula (2):

On the one hand, a multi-modal semantic role recognition-based human-computer interaction instruction parsing device is provided, the device is applied to electronic equipment, and the device includes:

Instruction Semantic Role Labeling Paradigm Building Module, which is used to construct instruction semantic role labeling paradigm according to the characteristics of human-computer interaction instructions;

A multimodal building block, used to expand the single-modal form of the semantic role labeling model to a multimodal form of visual text according to the instruction semantic role labeling paradigm and in combination with image acquisition;

The model training module is used to train and learn the visual text multimodal form of the semantic role labeling model, and complete the multimodal semantic role recognition and semantic analysis of human-computer interaction instructions.

Optionally, 9. The device according to claim 8, wherein the instruction semantic role labeling paradigm building module is used to use the labeling method of VerbAtlas semantic role labeling data as the labeling reference;

The pre-stored Chinese semantic role labeling paradigm is extended and modified, so that the extended and modified Chinese semantic role labeling paradigm is suitable for the semantic analysis of human-computer interaction instructions, and the instruction semantic role labeling paradigm is obtained.

Optionally, the multi-modal construction module is used to collect images through an unmanned system according to the instruction semantic role labeling paradigm, use Faster-RCNN to obtain sequence target areas, and form the sequence target areas into an image area sequence, for all Extract the features of the above image sequence;

Through the extracted image sequence features, the semantic role on the semantic text side is assisted to identify, and the single-modal form of the semantic role labeling model is extended to a visual-text dual-modal form.

In one aspect, an electronic device is provided, the electronic device includes a processor and a memory, the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the above multi-based Human-computer interaction instruction parsing method for modal semantic role recognition.

On the one hand, a computer-readable storage medium is provided, and at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to realize the above-mentioned human-computer interaction based on multimodal semantic role recognition. Interactive command parsing method.

The above-mentioned technical solutions of the embodiments of the present invention have at least the following beneficial effects:

Among the above solutions, the present invention innovatively proposes a semantic dependency graph representation scheme of integrated discourse chapters, which extends the sentence semantic dependency graph to the entire discourse, fully considering the incompleteness of discourse semantic information in dialogue scenarios. For the first time, the present invention proposes an integrated semantic dependency graph joint analysis model that integrates within discourses and between discourses, and uses an end-to-end modeling method to connect sentence semantics and discourse semantics together. In addition, the teacher-student network based on knowledge distillation used in the present invention can also meet the high requirements for efficiency and delay in the practical application of dialogue systems.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

Fig. 1 is a flowchart of a human-computer interaction instruction parsing method based on multimodal semantic role recognition provided by an embodiment of the present invention;

Fig. 2 is a flow chart of a human-computer interaction instruction analysis method based on multimodal semantic role recognition provided by an embodiment of the present invention;

Fig. 3 is a multimodal semantic role labeling model diagram of a human-computer interaction instruction analysis method based on multimodal semantic role recognition provided by an embodiment of the present invention;

Fig. 4 is a multimodal semantic role structured output diagram of a human-computer interaction instruction analysis method based on multimodal semantic role recognition provided by an embodiment of the present invention;

Fig. 5 is an example diagram of human-computer interaction realized by multi-modal semantic role labeling in a human-computer interaction instruction analysis method based on multi-modal semantic role recognition provided by an embodiment of the present invention;

Fig. 6 is a block diagram of a human-computer interaction instruction parsing device based on multimodal semantic role recognition provided by an embodiment of the present invention;

Fig. 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will describe in detail with reference to the drawings and specific embodiments.

An embodiment of the present invention provides a method for parsing human-computer interaction instructions based on multimodal semantic role recognition. The method can be implemented by an electronic device, and the electronic device can be a terminal or a server. As shown in Figure 1, the flow chart of the human-computer interaction instruction parsing method based on multi-modal semantic role recognition, the processing flow of the method may include the following steps:

S101: According to the characteristics of human-computer interaction instructions, construct an instruction semantic role labeling paradigm;

S102: According to the semantic role labeling paradigm of the instruction, combined with image collection, the single-modal form of the semantic role labeling model is extended to a multi-modal form of visual text;

S103: Carry out training and learning on the visual text multimodal form of the semantic role labeling model, and complete multimodal semantic role recognition and semantic analysis of human-computer interaction instructions.

Optionally, in step S101, according to the characteristics of the human-computer interaction instruction, construct an instruction semantic role annotation paradigm, including:

S111: Use the labeling method of the VerbAtlas semantic role labeling data as the labeling benchmark;

S112: Extend and modify the pre-stored Chinese semantic role labeling paradigm, so that the extended and modified Chinese semantic role labeling paradigm is applicable to the semantic analysis of human-computer interaction instructions, and obtain the instruction semantic role labeling paradigm.

Optionally, in step S102, according to the instruction semantic role labeling paradigm, combined with image collection, the single-modal form of the semantic role labeling model is extended to a visual-text dual-modal form, including:

S121: According to the instruction semantic role labeling paradigm, collect images through an unmanned system, use Faster-RCNN to obtain sequence target areas, form the sequence target areas into an image area sequence, and extract features of the image sequence;

S122: Use the extracted image sequence features to assist in identifying the semantic role on the semantic text side, and extend the single-modal form of the semantic role labeling model to a dual-modal form of visual text.

Optionally, in step S103, training and learning the visual text multimodal form of the semantic role labeling model, completing multimodal semantic role recognition and semantic analysis of human-computer interaction instructions, including:

S131: Construct a pre-training model according to the visual text multimodal form of the semantic role labeling model;

；利用BERT预训练模 型对所述指令I进行编码，获得指令I中每个词对应的词向量序列

； S132: Instructions for inputting the pre-trained model

; Utilize the BERT pre-training model to encode the instruction I, and obtain the word vector sequence corresponding to each word in the instruction I

;

，其中

， 获得每个跨度的特征向量；其中，所述跨度的大小均为预设值； S133: enumerate all spans in instruction I

,in

, to obtain the feature vector of each span; wherein, the size of the span is a preset value;

S134: According to the feature vector of each span, generate candidate vectors corresponding to the predicate node and the semantic role node in the semantic graph;

S135: Introduce a loss function to improve the training loss of the model, and complete multi-modal semantic role recognition and semantic analysis of human-computer interaction instructions.

以及语义角色 候选向量

，其中：

；

。 Optionally, in S134, two different MLP layers are used to obtain predicate candidate vectors respectively

and semantic role candidate vectors

,in:

;

.

Optionally, in S135, a loss function is introduced to improve the training loss of the model, including:

Construct a semantic role labeling loss function to judge the integrity of the predicate and argument structure predicted by the model;

、语义角色

以及两者关系

的三元组

进行打分；交叉熵来计算每个三元组的损失，所述语 义角色标注损失函数如下述公式（1）所示： Wherein, comprise a MLP layer score layer and a Biaffine score layer; The MLP layer score layer is used to judge the semantic framework of the current predicate node, and the Biaffine score layer is used to each predicate in the sentence

, semantic role

and the relationship between the two

triplet of

Scoring; cross-entropy to calculate the loss of each triplet, the semantic role labeling loss function is shown in the following formula (1):

Optionally, in S135, a loss function is introduced to improve the training loss of the model, including:

Construct a modal matching function for modal matching of image and text cross-modal feature pairs. The label of this function is defined as if the segment corresponding to the semantic role contains the object corresponding to the target area, then the output label is 1, otherwise The label is 0; through the paradigm of multi-task learning, define the loss function of the mode matching function as shown in the following formula (2):

In the embodiment of the present invention, an innovative attempt is made to introduce image information into the existing single-modal semantic role labeling model, so that image information is used to assist the semantic role labeling model to perform semantic analysis on input sentences. Try to use the paradigm of multi-modal semantic role labeling to semantically analyze human-computer interaction instructions, so as to convert instructions that cannot be understood by the machine into semantically structured output that the machine can understand, so as to achieve more convenient, safe and fast execution of the user intention of.

An embodiment of the present invention provides a method for parsing human-computer interaction instructions based on multimodal semantic role recognition. The method can be implemented by an electronic device, and the electronic device can be a terminal or a server. As shown in Figure 2, the flow chart of the human-computer interaction instruction analysis method based on multi-modal semantic role recognition, the processing flow of the method may include the following steps:

S201: Using the labeling method of the VerbAtlas semantic role labeling data as a labeling benchmark.

In a feasible implementation manner, the present invention first aims at human-computer interaction instructions, and builds a complete set of instruction semantic role labeling paradigms based on its own characteristics. Most of the previous semantic role annotation paradigms are oriented to general fields (such as news), etc., and the setting of their semantic roles lies in better generality. But in the field of human-computer interaction, each type of instruction has its own particularity in its semantic role, which cannot be covered by the semantic role in the general field.

S202: Extend and modify the pre-stored Chinese semantic role labeling paradigm, so that the extended and modified Chinese semantic role labeling paradigm is applicable to the semantic analysis of human-computer interaction instructions, and obtain the instruction semantic role labeling paradigm.

In a feasible implementation, the present invention intends to expand and modify the existing Chinese semantic role labeling paradigm, making it suitable for semantic analysis of human-computer interaction instructions.

The preliminary plan is to use the VerbAtlas semantic role labeling data labeling method as the labeling benchmark of the present invention, which is mainly based on the following two considerations: (1) The labeling benchmark adds the concept of semantic framework to the predicate recognition, making the specific semantics of each predicate more accurate , thereby alleviating the ambiguity problem caused by different contexts of predicates. (2) The labeling benchmark is designed for multilingual scenes, which can facilitate the design of the Chinese instruction-oriented labeling paradigm in the present invention. Table 1 shows the semantic framework initially set by the present invention, that is, semantic roles. It covers simple displacement instructions such as moving forward and moving, as well as difficult operation instructions such as taking and opening; the semantic role includes the control equipment involved in the semantic event, the control means, and the time and place when the command is executed.

S203: According to the instruction semantic role labeling paradigm, collect images through an unmanned system, use Faster-RCNN to obtain sequence target areas, form the sequence target areas into an image area sequence, and extract features of the image sequence;

S204: Use the extracted image sequence features to assist in identifying the semantic role on the semantic text side, and extend the single-modal form of the semantic role labeling model to a dual-modal form of visual text.

In a feasible implementation, in terms of model architecture, the present invention adopts the twin-tower model as shown in FIG. 3 to solve the fusion of graphic and text features between multi-modal semantic role tasks. The overall architecture is mainly divided into three parts, the image sequence feature extraction on the image side, the semantic map feature extraction on the language side, and finally the training function for feature fusion.

，本发明 采用现有的Faster-RCNN获得一序列目标区域，将其组成图像区域序列

，并 获得区域序列对应的特征序列

。对于特征序列中的区域特征

，本发明利用 一层MLP层，对做进一步的特征抽象，得到最终的图像特征

: In a feasible implementation, image sequence features: for images observed by unmanned systems

, the present invention uses the existing Faster-RCNN to obtain a sequence of target areas, which are composed of image area sequences

, and obtain the feature sequence corresponding to the region sequence

. For the region features in the feature sequence

, the present invention uses a layer of MLP layer to further abstract the features to obtain the final image features

:

S205: Construct a pre-training model according to the visual-text multimodal form of the semantic role labeling model;

；利用BERT预训练模 型对所述指令I进行编码，获得指令I中每个词对应的词向量序列

； S206: Instructions for inputting the pre-trained model

; Utilize the BERT pre-training model to encode the instruction I, and obtain the word vector sequence corresponding to each word in the instruction I

;

，其中

， 获得每个跨度的特征向量；其中，所述跨度的大小均为预设值； S207: enumerate all spans in instruction I

,in

, to obtain the feature vector of each span; wherein, the size of the span is a preset value;

S208: Generate candidate vectors corresponding to predicate nodes and semantic role nodes in the semantic graph according to the feature vectors of each span.

，利用BERT预训练模型对其进行编码，获得指令中每个词对应的 词向量序列

。接着枚举出指令中所有的跨度

，其中

，由句子中多个词组成的。每个跨度的最大长度和最小长度都是 预先设定好的。对于每个跨度

，将其特征向量表示为： In a feasible implementation, text sequence features: the present invention adopts the current classic semantic graph neural network construction idea of end-to-end semantic role labeling to obtain hidden predicates and their corresponding arguments in sentences. For the input command

, use the BERT pre-training model to encode it, and obtain the word vector sequence corresponding to each word in the instruction

. Then enumerate all spans in the instruction

,in

, consisting of multiple words in the sentence. The maximum and minimum lengths of each span are preset. for each span

, expressing its eigenvectors as:

，

表示每个跨度起始单词和结尾单词所对应的隐藏层表示，

表示每个跨度对应的长度特征，

则是利用Self-Attention机制，计算对于跨度 内每个词的注意力，并根据注意力加权平均得到的向量。 in

,

denote the hidden layer representations corresponding to each span start word and end word,

Represents the length feature corresponding to each span,

It uses the Self-Attention mechanism to calculate the attention for each word in the span, and calculates the vector obtained by weighting the attention.

，需要生成语义图中谓词节点和语义角色节点对应 的候选向量，因此本发明采用两个不同的MLP层分别得到谓词候选向量以及语义角色候选 向量

以及

：For each span the corresponding representation

, it is necessary to generate candidate vectors corresponding to predicate nodes and semantic role nodes in the semantic graph, so the present invention uses two different MLP layers to obtain predicate candidate vectors and semantic role candidate vectors respectively

as well as

:

S209: Introduce a loss function to improve the training loss of the model, and complete multi-modal semantic role recognition and semantic analysis of human-computer interaction instructions.

以及语 义角色候选向量

，其中：

；

。 In a feasible implementation, two different MLP layers are used to obtain predicate candidate vectors respectively

and semantic role candidate vectors

,in:

;

.

Among them, MLP ^P is a multi-layer feed-forward neural network for obtaining predicate representations, and MLP ^R is a multi-layer feed-forward neural network for obtaining semantic role representations.

In a feasible implementation, a loss function is introduced to improve the training loss of the model, including:

Construct a semantic role labeling loss function to judge the integrity of the predicate and argument structure predicted by the model;

、语义角色

以及两者关系

的三元组

进行打分；交叉熵来计算每个三元组的损失，所述语 义角色标注损失函数如下述公式（1）所示： Wherein, comprise a MLP layer score layer and a Biaffine score layer; The MLP layer score layer is used to judge the semantic framework of the current predicate node, and the Biaffine score layer is used to each predicate in the sentence

, semantic role

and the relationship between the two

triplet of

Scoring; cross-entropy to calculate the loss of each triplet, the semantic role labeling loss function is shown in the following formula (1):

进行打分，具体定义如下： In a feasible implementation manner, in terms of training loss, the present invention defines two loss functions for training the model. The first is the semantic role labeling loss function that judges the integrity of the predicate and argument structure predicted by the model, which includes an MLP layer score layer to judge the semantic framework of the current predicate node, and a Biaffine score layer to evaluate each A triplet of predicates, semantic roles, and their relations

To score, the specific definition is as follows:

表示用于获取语义框架类别得分的多层前馈神经网络；

是 Biaffine权重矩阵，

是线性权重矩阵，

是偏置项。获得每个关系对应的评分后，本发 明采用交叉熵来计算每个三元组的损失：in,

represents a multi-layer feed-forward neural network for obtaining semantic frame category scores;

is the Biaffine weight matrix,

is the linear weight matrix,

is a bias item. After obtaining the score corresponding to each relationship, the present invention uses cross-entropy to calculate the loss of each triplet:

和

表示对应的的语义框架以及语义角色集合。 in

and

Represents the corresponding semantic framework and semantic role set.

In a feasible implementation, a loss function is introduced to improve the training loss of the model, including:

Construct a modal matching function for modal matching of image and text cross-modal feature pairs. The label of this function is defined as if the segment corresponding to the semantic role contains the object corresponding to the target area, then the output label is 1, otherwise The label is 0; through the paradigm of multi-task learning, define the loss function of the mode matching function as shown in the following formula (2):

进行打分， In a feasible implementation, the second type is a modal matching function for image and text cross-modal feature pairs, and the label of this function is defined in the present invention as if the segment corresponding to the semantic role contains the corresponding object, the output label is 1, otherwise the label is 0. The present invention also utilizes a Biaffine layer to calculate triplets of the image region features, semantic roles and the relationship between the two

to score,

Similarly, the corresponding loss function is:

The final loss function is defined in the present invention using the paradigm of multi-task learning:

用于调节两种损失函数在模型训练中所发挥的权重。 in

It is used to adjust the weight of the two loss functions in model training.

In the embodiment of the present invention, the goal of multi-modal semantic role labeling is to give an input instruction and obtain a semantically structured output of the instruction, so that the machine can understand and execute it. The structured output results of multimodal semantic role recognition are shown in Figure 4.

In the embodiment of the present invention, FIG. 5 shows an example of parsing human-computer interaction instructions by the multimodal semantic role labeling model of the present invention. For the instruction issued by the user, the multi-modal semantic role analysis system of the present invention identifies the predicates, the corresponding semantic framework, and the semantic roles belonging to the semantic framework, and organizes them into a machine-recognizable structured output.

In the embodiment of the present invention, since most existing semantic role labeling models are based on single-modal settings, an innovative attempt is made to introduce image information into the existing single-modal semantic role labeling models, thereby using image information to assist semantic The role labeling model performs semantic analysis on the input sentence. Try to use the paradigm of multi-modal semantic role labeling to semantically analyze human-computer interaction instructions, so as to convert instructions that cannot be understood by the machine into semantically structured output that the machine can understand, so as to achieve more convenient, safe and fast execution of the user intention of.

Fig. 6 is a block diagram of a human-computer interaction instruction parsing device based on multimodal semantic role recognition according to an exemplary embodiment. Referring to Figure 6, the device 300 includes:

Paradigm construction module 310, configured to construct a complete set of instruction semantic role labeling paradigms according to the characteristics of human-computer interaction instructions;

The multimodal construction module 320 is used to expand the single-modal form of the semantic role labeling model into a multimodal form of visual text according to the semantic role labeling paradigm of the instruction and in combination with image collection;

The model training module 330 is used to train and learn the visual text multi-modal form of the semantic role labeling model, complete multi-modal semantic role recognition and perform semantic analysis on human-computer interaction instructions.

Optionally, the paradigm building module 310 is used to use the labeling method of VerbAtlas semantic role labeling data as the labeling benchmark;

The pre-stored Chinese semantic role labeling paradigm is extended and modified, so that the extended and modified Chinese semantic role labeling paradigm is suitable for the semantic analysis of human-computer interaction instructions, and a complete set of instruction semantic role labeling paradigms is obtained.

Optionally, the multimodal construction module 320 is configured to collect images through an unmanned system according to the instruction semantic role labeling paradigm, use Faster-RCNN to obtain sequence target areas, and form the sequence target areas into an image area sequence, for The image sequence features are extracted;

Through the extracted image sequence features, the semantic role on the semantic text side is assisted to identify, and the single-modal form of the semantic role labeling model is extended to a visual-text dual-modal form.

Optionally, the model training module 330 is configured to construct a pre-training model according to the visual-text multimodal form of the semantic role labeling model;

；利用BERT预训练模型对所 述指令I进行编码，获得指令I中每个词对应的词向量序列

； Instructions for the input of the pre-trained model

; Utilize the BERT pre-training model to encode the instruction I, and obtain the word vector sequence corresponding to each word in the instruction I

;

，其中

， 获得每个跨度的特征向量；其中，所述跨度的大小均为预设值； Enumerate all spans in instruction I

,in

, to obtain the feature vector of each span; wherein, the size of the span is a preset value;

According to the feature vector of each span, generate the candidate vector corresponding to the predicate node and the semantic role node in the semantic graph;

Introduce a loss function to improve the training loss of the model, and complete multi-modal semantic role recognition and semantic analysis of human-computer interaction instructions.

以及语义角色候选向量

，其中：

；

。 Optionally, the model training module 330 is used to adopt two different MLP layers to obtain predicate candidate vectors respectively

and semantic role candidate vectors

,in:

;

.

Optionally, the model training module 330 is used to construct a semantic role labeling loss function to judge the integrity of the predicate and argument structure predicted by the model;

、语义角色

以及两者关系

的三元组

进行打分；交叉熵来计算每个三元组的损失，所述语 义角色标注损失函数如下述公式（1）所示： Wherein, comprise a MLP layer score layer and a Biaffine score layer; The MLP layer score layer is used to judge the semantic framework of the current predicate node, and the Biaffine score layer is used to each predicate in the sentence

, semantic role

and the relationship between the two

triplet of

Scoring; cross-entropy to calculate the loss of each triplet, the semantic role labeling loss function is shown in the following formula (1):

Optionally, the model training module 330 is used to construct a modality matching function for modality matching of image and text cross-modal feature pairs, and the label of the function is defined as if the segment corresponding to the semantic role contains the target region For the corresponding object, the output label is 1, otherwise the label is 0; through the paradigm of multi-task learning, the loss function of the modal matching function defined in the following formula (2):

In the embodiment of the present invention, since most existing semantic role labeling models are based on single-modal settings, an innovative attempt is made to introduce image information into the existing single-modal semantic role labeling models, thereby using image information to assist semantic The role labeling model performs semantic analysis on the input sentence. Try to use the paradigm of multi-modal semantic role labeling to semantically analyze human-computer interaction instructions, so as to convert instructions that cannot be understood by the machine into semantically structured output that the machine can understand, so as to achieve more convenient, safe and fast execution of the user intention of.

FIG. 7 is a schematic structural diagram of an electronic device 400 provided by an embodiment of the present invention. The electronic device 400 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 401 and one or more memory 402, wherein at least one instruction is stored in the memory 402, and the at least one instruction is loaded and executed by the processor 401 to realize the following human-machine based on multimodal semantic role recognition The steps of the interactive instruction parsing method:

S1: According to the characteristics of human-computer interaction instructions, construct a complete set of instruction semantic role labeling paradigms;

S2: According to the semantic role labeling paradigm of the instruction, combined with image collection, the single-modal form of the semantic role labeling model is extended to a multi-modal form of visual text;

S3: Train and learn the visual text multimodal form of the semantic role labeling model, and complete the multimodal semantic role recognition and semantic analysis of human-computer interaction instructions.

In an exemplary embodiment, there is also provided a computer-readable storage medium, such as a memory including instructions, the above-mentioned instructions can be executed by a processor in the terminal to complete the above-mentioned human-computer interaction instruction parsing method based on multi-modal semantic role recognition . For example, the computer readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

Those of ordinary skill in the art can understand that all or part of the steps for implementing the above embodiments can be completed by hardware, and can also be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. The above-mentioned The storage medium mentioned may be a read-only memory, a magnetic disk or an optical disk, and the like.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within range.

Claims (9)

1. A human-computer interaction instruction analysis method based on multimodal semantic role recognition, characterized in that, comprising the following steps: S1: According to the characteristics of human-computer interaction instructions, construct an instruction semantic role annotation paradigm; S2: According to the semantic role labeling paradigm of the instruction, combined with image collection, the single-modal form of the semantic role labeling model is extended to a multi-modal form of visual text; S3: Train and learn the visual text multimodal form of the semantic role labeling model, and complete the semantic analysis of human-computer interaction instructions for multimodal semantic role recognition; In the step S3, the multimodal form of visual text of the semantic role labeling model is trained and learned, and the multimodal semantic role recognition is completed to perform semantic analysis on human-computer interaction instructions, including: S31: Construct a pre-training model according to the visual text multimodal form of the semantic role labeling model; S32: Instructions for inputting the pre-trained model

; Utilize the BERT pre-training model to encode the instruction I, and obtain the word vector sequence corresponding to each word in the instruction I

; S33: enumerate all spans in instruction I

,in

, to obtain the feature vector of each span; wherein, the size of the span is a preset value; S34: According to the feature vector of each span, generate candidate vectors corresponding to the predicate node and the semantic role node in the semantic graph; S35: Introduce a loss function to improve the training loss of the model, and complete multi-modal semantic role recognition and semantic analysis of human-computer interaction instructions. 2. The method according to claim 1, characterized in that, in the step S1, according to the characteristics of the human-computer interaction instruction, constructing an instruction semantic role annotation paradigm, including: S11: Use the labeling method of VerbAtlas semantic role labeling data as the labeling benchmark; S12: Extend and modify the pre-stored Chinese semantic role labeling paradigm, so that the extended and modified Chinese semantic role labeling paradigm is applicable to the semantic analysis of human-computer interaction instructions, and obtain the instruction semantic role labeling paradigm. 3. The method according to claim 2, characterized in that, in the step S2, according to the instruction semantic role labeling paradigm, combined with image acquisition, the single-modal form of the semantic role labeling model is extended to a visual-text dual-mode state forms, including: S21: According to the instruction semantic role annotation paradigm, collect images through an unmanned system, use Faster-RCNN to obtain a sequence target area, form the sequence target area into an image area sequence, and extract features of the image sequence; S22: Use the extracted image sequence features to assist in identifying the semantic role on the semantic text side, and extend the single-modal form of the semantic role labeling model to a dual-modal form of visual text. 4. The method according to claim 1, characterized in that, in said S34, two different layer perceptron MLP layers are used to obtain the predicate candidate vector respectively

and semantic role candidate vectors

,in:

. 5. The method according to claim 4, characterized in that, in said S35, introducing a loss function to improve the training loss of the model, including: Construct a semantic role labeling loss function to judge the integrity of the predicate and argument structure predicted by the model; Wherein, comprise a MLP layer score layer and a Biaffine score layer; The MLP layer score layer is used to judge the semantic framework of the current predicate node, and the Biaffine score layer is used to each predicate in the sentence

semantic role

and the relationship between the two

triplet of

Scoring; cross-entropy to calculate the loss of each triplet, the semantic role labeling loss function is shown in the following formula (1):

. 6. The method according to claim 4, characterized in that, in said S35, introducing a loss function to improve the training loss of the model, including: Construct a modal matching function for modal matching of image and text cross-modal feature pairs. The label of this function is defined as if the segment corresponding to the semantic role contains the object corresponding to the target area, then the output label is 1, otherwise the label is 0; through the paradigm of multi-task learning, define the loss function of the mode matching function as shown in the following formula (2):

. 7. A human-computer interaction instruction parsing device based on multimodal semantic role recognition, characterized in that the device is suitable for any one of the above-mentioned methods in claims 1-6, and the device includes: Instruction Semantic Role Labeling Paradigm Building Module, which is used to construct instruction semantic role labeling paradigm according to the characteristics of human-computer interaction instructions; A multimodal building block, used to expand the single-modal form of the semantic role labeling model to a multimodal form of visual text according to the instruction semantic role labeling paradigm and in combination with image acquisition; The model training module is used to train and learn the visual text multimodal form of the semantic role labeling model, and complete the multimodal semantic role recognition and semantic analysis of human-computer interaction instructions. 8. The device according to claim 7, characterized in that, the instruction semantic role labeling paradigm building block is used to root the labeling method using VerbAtlas semantic role labeling data as the labeling reference; The pre-stored Chinese semantic role labeling paradigm is extended and modified, so that the extended and modified Chinese semantic role labeling paradigm is suitable for the semantic analysis of human-computer interaction instructions, and the instruction semantic role labeling paradigm is obtained. 9. The device according to claim 7, wherein the multimodal construction module is configured to collect images through an unmanned system according to the instruction semantic role labeling paradigm, and use Faster-RCNN to obtain the sequence target area, Composing the sequence target area into a sequence of image regions, and extracting the features of the sequence of images; Through the extracted image sequence features, the semantic role on the semantic text side is assisted to identify, and the single-modal form of the semantic role labeling model is extended to a visual-text dual-modal form.

Images