CN114973402A - A visual language navigation system and method based on modal-aligned action prompts - Google Patents

Abstract

The invention provides a visual language navigation system and method based on modal alignment action prompt, the system includes action prompt set generation module, input order to the action prompt set generation module, the intelligent agent retrieves the action prompt set correlated to order from the action prompt library before the navigation begins; the visual language navigation module of modal alignment action prompt, the action prompt set is through prompting the code module, the output prompt characteristic links together with output instruction characteristic of the text code module; the hint-based instruction features and the output visual features of the visual coding module are provided to the multi-layer transformer for action decision-making. The optimization learning module, namely a modal alignment loss module and a continuous consistency loss module, realizes effective action prompt learning; the invention mainly provides explicit modal alignment action prompt to improve the accuracy of intelligent agent navigation and generalization capability in different environments.

Description

A visual language navigation system and method based on modal-aligned action prompts

technical field

The present invention relates to the field of visual language navigation, and more particularly, to a visual language navigation system and method based on modal alignment action prompts.

Background technique

Visual-language navigation is a challenging task that requires embodied subjects to navigate to target locations following natural language instructions. For successful navigation, the agent should in turn make correct action decisions to move through a dynamically changing scene by understanding the intent of a given instruction and incrementally basing the instruction on observations around it.

在预训练模型中引入循环函数使智能体具有时间感知功能。虽然对象级别对齐能力可能在预训练过程中被显著提高，这些智能体仍然是在隐式地学习动作级别的模态对齐，这在很大程度上限制了不同场景下的行动决策的鲁棒性。Early visual-linguistic navigation methods explored different data augmentation strategies, efficient learning paradigms and useful model architectures for improving agent performance. Inspired by the significant progress achieved in large-scale cross-modal pre-training models in visual-linguistic tasks, a growing body of work attempts to introduce pre-training paradigms and models into visual-linguistic navigation tasks. PREVALENT self-supervised pre-training of the model on a large number of image-language-action triples.

Introducing a recurrent function into the pretrained model makes the agent time-aware. Although object-level alignment capabilities may be significantly improved during pre-training, these agents are still implicitly learning action-level modal alignment, which largely limits the robustness of action decisions in different scenarios .

The prior art discloses a patent for a navigation method integrating vision and language multimodality, which belongs to the fields of robot navigation, natural language processing and computer vision; the patent first installs a binocular camera on the robot, and uses the robot to train A multi-modal fusion neural network model; select any real scene, issue natural language navigation instructions to the robot and convert them into corresponding semantic vectors; use the RGB images obtained by the robot at each moment, and convert them into corresponding features; , RGB image feature features are fused to obtain the action feature at the current moment; after correcting the action feature by using the prompt, the neural network model finally outputs the robot's action at the current moment, and the robot performs the action until the navigation task is completed. However, there are few reports in this patent on how to implement a more robust visual language navigation model, improve the accuracy and generalization ability, and have good interpretability.

SUMMARY OF THE INVENTION

The present invention provides a visual-language navigation system based on modal-aligned action prompts, which can force an agent to explicitly learn cross-modal action knowledge to improve action decision-making during navigation.

Another object of the present invention is to provide a navigation method for the above system.

In order to achieve above-mentioned technical effect, technical scheme of the present invention is as follows:

A visual language navigation system based on modal-aligned action prompts, comprising:

Action prompt set generation module, inputting an instruction to the action prompt set generation module, the agent retrieves the action prompt set related to the instruction from the action prompt library before navigation starts;

The visual language navigation module of the modal alignment action cue, the action cue set is connected through the cue coding module, and the output cue feature is connected with the output instruction feature of the text coding module; the cue-based instruction feature and the output visual feature of the visual coding module are provided to Multi-layer transformers are used to make action decisions;

The optimized learning modules, namely the modal alignment loss module and the continuous consistency loss module, enable effective action cue learning.

Further, the visual language navigation module of the modal alignment action prompt includes:

Text encoding module This module receives the input of language information, uses the multi-layer transformer neural network to encode separately, and obtains the corresponding feature vector;

The prompt decoding module is composed of two single-modality sub-prompt encoders and a multi-modal prompt encoder. The image sub-prompt and text sub-prompt respectively obtain the sub-prompt features through the corresponding single-modal auto-encoder. Input into the multi-modal prompt encoder to obtain prompt features;

Visual coding module, this module receives the input of visual observation information, encodes it through the visual encoder, and obtains the corresponding feature vector.

Further, the optimization learning module includes:

Modal alignment loss module, when the action prompt already has matching image and text sub-prompts, using the InfoNCE loss to align them in the feature space, the action prompt can become more discriminative;

A continuous consistency loss module that prompts the agent to sequentially focus on relevant action cues in the retrieved cue set based on its observations.

A visual language navigation method based on modal-aligned action prompts, comprising the following steps:

S1: At the beginning of navigation, the agent obtains the instruction, and retrieves the action prompt set related to the instruction from the action prompt library through the action prompt generation module;

S2: Through the visual encoding module and the text encoding module, the neural network respectively encodes the input image information and instruction information, and obtains visual encoding, instruction encoding, and state features respectively;

S3: Through the prompt coding mode, the image sub-prompt and the text sub-prompt in the action prompt set respectively obtain the sub-prompt feature through the corresponding single-modal auto-encoder, and then input into the multi-modal prompt encoder after connection to obtain the prompt feature;

S4: connect the above-mentioned instruction code and the prompt code to obtain the prompt-based instruction feature, and connect the above-mentioned state feature with the visual code to obtain the state visual feature;

S5: By modally aligning the visual language navigation module of action cues, the state visual feature is updated based on the cross-modal attention between itself and cue-based instruction features, and the attention is decomposed into two parts, the first part weights the instruction encoding Update, which is used to update the state features, the second part performs weighted updates on the image and text sub-cue features, which are used to calculate the sequential consistency loss, and feed the state visual features into another self-attention module to obtain the state features with respect to the visual features. Attention score, which is the probability of action prediction based on cues;

S6: By optimizing the learning model, combining the commonly used imitation learning loss and reinforcement learning loss, as well as the unique modal alignment loss and continuous consistency loss of the present invention, weighted summation is performed to obtain the total training target, and the model is updated and optimized to improve Agent Navigation Performance and Generalization.

Further, the step S1 includes the following sub-steps:

S100: Construction of action prompt library, in order to align images and action phrases to form action prompts, a two-branch scheme is designed to collect image and text sub-prompts: First, for an instruction path instance in the training dataset, use a pre-created Visual object visual object/location location vocabulary to find the visual object/location mentioned in the instruction, for each visual object/location, get the relevant image and text sub-cues respectively, using cross-modal alignment with excellent 0-shot Capable CLIP for locating object/position related images, to accommodate the inference process of CLIP, replace the token {CLASS} token in the phrase "a photo of {CLASS}" with a visual object/position whose class label is c , the probability that an image B belongs to class c in the action sequence is calculated by:

where τ1 is the temperature parameter, sim is the cosine similarity, _b , wc are the image features and phrase features generated by CLIP, respectively, M is the size of the vocabulary, and then the image with the greatest similarity to the phrase is selected as the image sub-prompt, To obtain textual subcues, a simple nearest verb search scheme is used, i.e. to find the nearest verb before a specific object/position word, which is in a pre-built verb vocabulary, and finally, has the same visual object/position and action Image and text subprompts form an aligned action prompt;

其中N为该集合的大小。S101: Retrieval of action prompt set, at the beginning of navigation, the agent retrieves the action prompt related to the instruction from the action prompt library, and calculates the sentence between each object/position related action phrase in the prompt library and the text sub-prompt Similarity for retrieving the set of action cues associated with the instruction

where N is the size of the set.

Further, the step S2 includes the following sub-steps:

S200: Encoding of visual input, for time step t, each image view O _t,i in the candidate view will use a pre-trained convolutional neural network CNN or transformer to extract image features v _t,i , and then v _t,i are mapped to visual encoding by visual encoder F _v :

V _t,i =F _v (v _t,i ; θ _v )

代表时间t下的候选视觉编码；where θ _v is the parameter of F _v , a set of

represents the candidate visual code at time t;

S201: The encoding of language input. During initialization, the instruction code X and the initialized state feature s ₀ are obtained by inputting the instruction sequence I and [CLS] and [SEP] tokens to the self-attention module in the transformer:

表示self-attention模组的参数，s₀将会在时间步骤t被更新为s_t。Where Concat( ) represents the connection concatenation operation,

Represents the parameters of the self-attention module, s ₀ will be updated to s _{t at time step t} .

Further, the step S3 includes the following steps:

通过提示编码器得到提示编码

该提示编码器由两个单模态子提示编码器和一个多模态提示编码器组成，

其中图像子提示和文本子提示分别为

和

和

首先通过单模态子提示编码器得到子提示特征

和

use

Get hint code from hint encoder

The cue encoder consists of two unimodal sub-cue encoders and a multi-modal cue encoder,

The image sub-prompt and text sub-prompt are respectively

and

and

First obtain the sub-cue features through a unimodal sub-cue encoder

and

和

输送到多模态提示编码器E^p(·)，得到提示编码

where E ⁱ (·) uses the parameter θ ⁱ and E ^u (·) uses the parameter θ ^u , representing the image sub-hint encoder and the text sub-hint encoder, respectively, and then the

and

Send to the multimodal prompt encoder E ^p ( ), get the prompt code

where θ ^p is the parameter of E ^p ( ), Concat ( ) is the concatenation operation, the encoder E ⁱ ( ), E ^u ( ) and E ^p ( ) consist of a linear layer followed by a dropout operation, to reduce overfitting.

Further, the step S4 includes the following sub-steps:

和指令编码X的基础上，通过简单地将X和

连接起来，得到基于提示的指令特征X^p。coding at the prompt

and instruction encoding X, by simply combining X and

concatenated to obtain hint-based instruction features X ^p .

Further, the step S5 includes the following sub-steps:

更新：The state visual feature _{Kt is} based on cross-modal attention between _Kt and ^Xp

renew:

分解为

和

获得不同的基于注意力机制增强的特征，参与指令特征

是通过对X进行

加权得到的，基于注意力机制增强的图像子提示特征

和基于注意力机制增强的文本子提示特征

通过对

和

进行

加权获得，

和

用于计算顺序一致性损失L_c，和baseline智能体一样，

用于更新状态特性，最后，将

输入

得到基于提示的动作预测概率

followed by

Decomposed into

and

Obtain different attention-based enhanced features, participating in instruction features

is performed on X by

Weighted, enhanced image sub-cue features based on attention mechanism

and enhanced text sub-cue features based on attention mechanism

through the pair

and

conduct

weighted gain,

and

Used to calculate the sequential consistency loss L _c , like the baseline agent,

is used to update the state property, and finally, the

enter

Get cue-based action prediction probabilities

Further, the step S6 includes the following sub-steps:

S600: Modal alignment loss, prompting action cues that already have matching image and text subcues to align in feature space, following the contrastive learning paradigm used in CLIP, making pairs of image and text features similar, while unpaired images and text Text feature alienation, using infoNCE loss to facilitate feature alignment of image and text sub-cues in each action cue:

表示动作提示p_n的成对的图像和文本子提示的特征，

表示非配对的子提示，通过模态对齐损失，动作提示可以变得更加具有识别力，从而知道学习动作级别的模态对齐；where τ2 is the temperature parameter,

The features representing pairs of image and text sub-prompts of action cue p _n ,

Represents unpaired sub-prompts, through the modal alignment loss, the action cue can become more discriminative, so as to know the modal alignment of the learned action level;

以及基于注意力机制增强的指导特征

必须接近：S601: Sequential consistency loss, since instructions usually point to different visual landmarks sequentially, the action cues in the retrieved action cue set {p _n } are also related to different objects/positions. Focusing on the relevant action cues in the retrieved cue set in order, a sequential consistency loss is proposed, that is, the sum of the two unimodal consistency losses; at each time step t, the text sub-cue feature enhanced by the attention mechanism

and guidance features enhanced by attention mechanism

must be close to:

用于提高基于注意力机制增强的图像子提示特征

和基于注意力机制增强的视觉特征之间的相似性，则顺序一致性损失L_c为：definition,

Image sub-cue features for improving attention-based enhancement

and the similarity between visual features enhanced by the attention mechanism, then the sequential consistency loss L _c is:

S602: The total objective uses the navigation loss L _n , namely the imitation loss L _IL and the reinforcement learning loss L _RL , and the total training objective is:

L=L _RL +λ ₁ L _IL +λ ₂ L _c +λ ₃ L _a

where λ ₁ , λ ₂ and λ ₃ are the loss weights to balance the losses.

Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

The present invention proposes modal-aligned action cues to force the agent to explicitly learn cross-modal action knowledge to improve action decision-making during navigation, develop cue-based navigation in visual-language navigation tasks; develop a modal Alignment loss and continuous consistency loss for efficient learned action cues. The contrastive language-image pre-training (CLIP) model is used to ensure the quality of action prompts; it effectively improves the navigation performance of agents based on R2R and RxR, and has good interpretability and generalization ability.

Description of drawings

1 is an architectural diagram of a visual language navigation system based on a modal alignment-based action prompt of the present invention;

2 is a flow chart of steps of a visual language navigation method based on modal alignment of action prompts of the present invention;

3 is an example diagram of a visual language navigation module for modal alignment action prompts in a specific embodiment of the present invention;

4 is an example diagram of the construction of an action prompt library of an action prompt set generation module in a specific embodiment of the present invention;

FIG. 5 is a comparative display of a result sample of the result navigation using the visual language navigation method and the baseline method in the specific embodiment of the present invention.

Detailed ways

The accompanying drawings are for illustrative purposes only, and should not be construed as limitations on this patent;

In order to better illustrate this embodiment, some parts of the drawings are omitted, enlarged or reduced, which do not represent the size of the actual product;

It will be understood by those skilled in the art that some well-known structures and their descriptions may be omitted from the drawings.

The technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments.

Example 1

As shown in Figure 1, a visual language navigation system based on modal-aligned action prompts includes:

The action cue collection module 10, in order to produce a high-quality action cue library, adopts the newly developed contrastive language image pre-training CLIP model with powerful cross-modal object/position level alignment capability for locating object/position-related images. To better align images and action phrases to form action prompts, a two-branch scheme is designed to collect image and text sub-prompts. First, for an instruction path instance in the training dataset, a pre-created visual object/location vocabulary is used to find the visual object/location mentioned in the instruction. Then for each visual object/location, the associated image and textual subcues are obtained respectively. When the navigation starts, input an instruction to the action prompt set generation module, and the agent retrieves the action prompt related to the instruction from the action prompt library built in advance to form an action prompt set.

The visual language navigation module 11 of the modal alignment action prompt obtains the prompt feature through a prompt encoder, and is connected with the output instruction feature of the text encoding module to obtain the prompt-based instruction feature. This feature and the output visual features of the visual encoding module are provided to the multi-layer transformer for action decision-making.

The optimization learning module 12, namely the modal alignment loss module and the continuous consistency loss module, realizes effective action cue learning.

In a specific embodiment of the present invention, specifically, the visual language navigation module 11 of the modal alignment action prompt further includes:

Text encoding module 110, the module receives the input of language information, uses self-supervised neural network for encoding, and obtains corresponding text feature vectors and state features.

The prompt encoding module 111 is composed of two single-modality sub-prompt encoders and a multi-modal prompt encoder. The image sub-prompt and the text sub-prompt respectively obtain the sub-prompt features through the corresponding single-modal auto-encoder, and connect the Then input into the multi-modal prompt encoder to obtain prompt features.

Visual coding module 112, this module receives the input of visual observation information, performs coding through the pre-trained visual feature encoder, and obtains the corresponding feature vector.

In a specific embodiment of the present invention, specifically, the optimization learning module 12 further includes:

Modal alignment loss module 120, when action cues already have matching image and text subcues, they may not be aligned in the feature space. To address this issue, following the contrastive learning paradigm used in CLIP to make pairs of image and text features similar, while unpaired image and text features are dissimilar, an infoNCE loss is used to facilitate the comprehension of image and text sub-cues in each action cue Feature alignment. With the modality alignment loss, action cues can be made more discriminative, knowing to learn action-level modal alignment.

In the continuous consistency loss module 121, since the instructions usually point to different visual landmarks sequentially, the action cues in the retrieved action cue set are also related to different objects/locations. To motivate the agent to sequentially focus on relevant action cues in the retrieved cue set according to its observations, a sequential consistency loss is proposed, which is the sum of two unimodal consistency losses. Taking the text modality as an example, at each time step t, the text sub-cue features as well as the guidance features must be close; a similar loss is defined in the image modality to improve the similarity between the image sub-cue features and visual features.

Example 2

As shown in Figure 2, a visual language navigation method based on modal alignment action prompts includes the following steps:

Step S1: Retrieve a relevant action prompt set according to the input instruction information.

Specifically, step S1 further includes:

Step S100, construction of an action prompt library. To better align images and action phrases to form action prompts, a two-branch scheme is designed to collect image and text sub-prompts. First, for an instruction path instance in the training dataset, a pre-created visual object/location vocabulary is used to find the visual object/location mentioned in the instruction. Then for each visual object/location, the associated image and textual subcues are obtained separately, as described below.

Note that the ground-truth path sequence contains a collection of single-view images, each representing an action that needs to be performed at a specific time step. Therefore, to derive image sub-cues in action cues, only object/location-related images, which themselves contain action information, are retrieved from the ground-truth path sequence. Compared to resorting to existing object classifiers or detectors trained on a fixed set of item categories, we use CLIP with excellent 0-shot cross-modal alignment capability for locating objects/position-related images. To accommodate the inference process of CLIP, the token {CLASS} token in the phrase "a photo of {CLASS}" is replaced with a visual object/location whose class label is c. The probability that an image B belongs to class c in the action sequence is calculated by:

where τ1 is the temperature parameter, sim is the cosine similarity, _b , wc are the image features and phrase features generated by CLIP, respectively, M is the size of the vocabulary, and then the image with the greatest similarity to the phrase is selected as the image sub-prompt.

To obtain textual subcues, a simple nearest-verb search scheme is used to find the nearest verb (in a pre-built verb vocabulary) before a specific object/location word. Finally, image and text subcues with the same visual object/position and action form an aligned action cue.

其中N为该集合的大小。Step S101, retrieval of the action prompt set. At the beginning of navigation, the agent retrieves the action cues associated with the instruction from the action cue library. Calculates the sentence similarity between each object/location-related action phrase in the cue library and the textual sub-cues for retrieving the set of instruction-related action cues

where N is the size of the set.

Step S2, respectively encode the input image information and instruction information through a neural network.

Specifically, step S2 further includes:

Step S200, coding of visual input, for time step t, each image view O _t,i in the candidate view will use a pre-trained convolutional neural network CNN or transformer to extract image features v _t,i , Then v _{t, i} are mapped to visual encoding by visual encoder F _v :

V _t,i =F _v (v _t,i ; θ _v )

代表时间t下的候选视觉编码。where θ _v is the parameter of F _v , a set of

represents the candidate visual encoding at time t.

Step S201, the encoding of language input, during initialization, the instruction encoding X and the initialized state feature s ₀ are obtained by inputting the instruction sequence I and [CLS] and [SEP] tokens to the self-attention module in the transformer:

表示self-attention模组的参数，s₀将会在时间步骤t被更新为s_t。Where Concat( ) represents the connection concatenation operation,

Represents the parameters of the self-attention module, s ₀ will be updated to s _{t at time step t} .

通过提示编码器得到提示编码

该提示编码器由两个单模态子提示编码器和一个多模态提示编码器组成，

其中图像子提示和文本子提示分别为

和

和

首先通过单模态子提示编码器得到子提示特征

和

In step S3, the action prompt set is encoded by the modal encoder. use

Get hint code from hint encoder

The cue encoder consists of two unimodal sub-cue encoders and a multi-modal cue encoder,

The image sub-prompt and text sub-prompt are respectively

and

and

First obtain the sub-cue features through a unimodal sub-cue encoder

and

和

输送到多模态提示编码器E^p(·)，得到提示编码

where E ⁱ (·) uses the parameter θ ⁱ and E ^u (·) uses the parameter θ ^u , representing the image sub-hint encoder and the text sub-hint encoder, respectively, and then the

and

Send to the multimodal prompt encoder E ^p ( ), get the prompt code

where θ ^p is the parameter of E ^p ( ), Concat ( ) is the concatenation operation, the encoder E ⁱ ( ), E ^u ( ) and E ^p ( ) consist of a linear layer followed by a dropout operation, to reduce overfitting.

和指令编码X的基础上，通过简单地将X和

连接起来，得到基于提示的指令特征X^p。Step S4, at the prompt encoding

and instruction encoding X, by simply combining X and

concatenated to obtain hint-based instruction features X ^p .

更新：Step S5, the state visual feature _{Kt is} based on the cross-modal attention between _Kt and ^Xp

renew:

分解为

和

获得不同的基于注意力机制增强的特征，参与指令特征

是通过对X进行

加权得到的，基于注意力机制增强的图像子提示特征

和基于注意力机制增强的文本子提示特征

通过对

和

进行

加权获得，

和

用于计算顺序一致性损失L_c，和baseline智能体一样，

用于更新状态特性，最后，将

输入

得到基于提示的动作预测概率

followed by

Decomposed into

and

Obtain different attention-based enhanced features, participating in instruction features

is performed on X by

Weighted, enhanced image sub-cue features based on attention mechanism

and enhanced text sub-cue features based on attention mechanism

through the pair

and

conduct

weighted gain,

and

Used to calculate the sequential consistency loss L _c , like the baseline agent,

is used to update the state property, and finally, the

enter

Get cue-based action prediction probabilities

In step S6, the weighted sum total target of each loss is calculated, and the model is updated and optimized to improve the navigation performance and generalization ability of the agent.

Specifically, step S6 further includes:

Step S600, the modal alignment loss, urges the action prompt to align the already matched image and text sub-prompts in the feature space, following the contrastive learning paradigm used in CLIP, making pairs of images and text features similar, while unpaired images Alienated from text features, the infoNCE loss is used to facilitate feature alignment of image and text sub-cues in each action cue:

表示动作提示p_n的成对的图像和文本子提示的特征，

表示非配对的子提示，通过模态对齐损失，动作提示可以变得更加具有识别力，从而知道学习动作级别的模态对齐。where τ2 is the temperature parameter,

The features representing pairs of image and text sub-prompts of action cue p _n ,

Representing unpaired sub-cues, action cues can be made more discriminative through a modality alignment loss, knowing to learn action-level modal alignment.

以及基于注意力机制增强的指导特征

必须接近：Step S601, loss of sequential consistency, since the instructions usually point to different visual signs sequentially, the action cues in the retrieved action cue set {p _n } are also related to different objects/positions. , sequentially focus on the relevant action cues in the retrieved cue set, and propose a sequential consistency loss, which is the sum of the two unimodal consistency losses; at each time step t, the text sub-cue feature enhanced by the attention mechanism

and guidance features enhanced by attention mechanism

must be close to:

用于提高基于注意力机制增强的图像子提示特征

和基于注意力机制增强的视觉特征之间的相似性，则顺序一致性损失L_c为：definition,

Image sub-cue features for improving attention-based enhancement

and the similarity between visual features enhanced by the attention mechanism, then the sequential consistency loss L _c is:

Step S602, the overall target uses the navigation loss L _n , namely the imitation loss L _IL and the reinforcement learning loss L _RL , and the overall training target is:

L=L _RL +λ ₁ L _IL +λ ₂ L _c +λ ₃ L _a

where λ ₁ , λ ₂ and λ ₃ are the loss weights to balance the losses.

Example 3

As shown in Figure 1, a visual language navigation system based on modal-aligned action prompts includes:

The action cue collection module 10, in order to produce a high-quality action cue library, adopts the newly developed contrastive language image pre-training CLIP model with powerful cross-modal object/position level alignment capability for locating object/position-related images. To better align images and action phrases to form action prompts, a two-branch scheme is designed to collect image and text sub-prompts. First, for an instruction path instance in the training dataset, a pre-created visual object/location vocabulary is used to find the visual object/location mentioned in the instruction. Then for each visual object/location, the associated image and textual subcues are obtained respectively. When the navigation starts, input an instruction to the action prompt set generation module, and the agent retrieves the action prompt related to the instruction from the action prompt library built in advance to form an action prompt set.

The visual language navigation module 11 of the modal alignment action prompt obtains the prompt feature through a prompt encoder, and is connected with the output instruction feature of the text encoding module to obtain the prompt-based instruction feature. This feature and the output visual features of the visual encoding module are provided to the multi-layer transformer for action decision-making.

The optimization learning module 12, namely the modal alignment loss module and the continuous consistency loss module, realizes effective action cue learning.

In a specific embodiment of the present invention, specifically, the visual language navigation module 11 of the modal alignment action prompt further includes:

Text encoding module 110, the module receives the input of language information, uses self-supervised neural network for encoding, and obtains corresponding text feature vectors and state features.

The prompt encoding module 111 is composed of two single-modality sub-prompt encoders and a multi-modal prompt encoder. The image sub-prompt and the text sub-prompt respectively obtain the sub-prompt features through the corresponding single-modal auto-encoder, and connect the Then input into the multi-modal prompt encoder to obtain prompt features.

Visual coding module 112, this module receives the input of visual observation information, performs coding through the pre-trained visual feature encoder, and obtains the corresponding feature vector.

In a specific embodiment of the present invention, specifically, the optimization learning module 12 further includes:

Modal alignment loss module 120, when action cues already have matching image and text subcues, they may not be aligned in the feature space. To address this issue, following the contrastive learning paradigm used in CLIP to make pairs of image and text features similar, while unpaired image and text features are dissimilar, an infoNCE loss is used to facilitate the comprehension of image and text sub-cues in each action cue Feature alignment. With the modality alignment loss, action cues can be made more discriminative, knowing to learn action-level modal alignment.

In the continuous consistency loss module 121, since the instructions usually point to different visual landmarks sequentially, the action cues in the retrieved action cue set are also related to different objects/locations. To motivate the agent to sequentially focus on relevant action cues in the retrieved cue set according to its observations, a sequential consistency loss is proposed, which is the sum of two unimodal consistency losses. Taking the text modality as an example, at each time step t, the text sub-cue features as well as the guidance features must be close; a similar loss is defined in the image modality to improve the similarity between the image sub-cue features and visual features.

As shown in FIG. 2, the above-mentioned navigation method of the visual language navigation system based on the action prompt of modal alignment includes the following steps:

Step S1: Retrieve a relevant action prompt set according to the input instruction information.

Specifically, step S1 further includes:

Step S100, construction of an action prompt library. To better align images and action phrases to form action prompts, a two-branch scheme is designed to collect image and text sub-prompts. First, for an instruction path instance in the training dataset, a pre-created visual object/location vocabulary is used to find the visual object/location mentioned in the instruction. Then for each visual object/location, the associated image and textual subcues are obtained separately, as described below.

Note that the ground-truth path sequence contains a collection of single-view images, each representing an action that needs to be performed at a specific time step. Therefore, to derive image sub-cues in action cues, only object/location-related images, which themselves contain action information, are retrieved from the ground-truth path sequence. Compared to resorting to existing object classifiers or detectors trained on a fixed set of item categories, we use CLIP with excellent 0-shot cross-modal alignment capability for locating objects/position-related images. To accommodate the inference process of CLIP, the token {CLASS} token in the phrase "a photo of {CLASS}" is replaced with a visual object/location whose class label is c. The probability that an image B belongs to class c in the action sequence is calculated by:

where τ1 is the temperature parameter, sim is the cosine similarity, _b , wc are the image features and phrase features generated by CLIP, respectively, M is the size of the vocabulary, and then the image with the greatest similarity to the phrase is selected as the image sub-prompt.

To obtain textual subcues, a simple nearest-verb search scheme is used to find the nearest verb (in a pre-built verb vocabulary) before a specific object/location word. Finally, image and text subcues with the same visual object/position and action form an aligned action cue.

其中N为该集合的大小。Step S101, retrieval of the action prompt set. At the beginning of navigation, the agent retrieves the action cues associated with the instruction from the action cue library. Calculates the sentence similarity between each object/location-related action phrase in the cue library and the textual sub-cues for retrieving the set of instruction-related action cues

where N is the size of the set.

Step S2, respectively encode the input image information and instruction information through a neural network.

Specifically, step S2 further includes:

Step S200, coding of visual input, for time step t, each image view O _t,i in the candidate view will use a pre-trained convolutional neural network CNN or transformer to extract image features v _t,i , Then v _{t, i} are mapped to visual encoding by visual encoder F _v :

V _t,i =F _v (v _t,i ; θ _v )

代表时间t下的候选视觉编码。where θ _v is the parameter of F _v , a set of

represents the candidate visual encoding at time t.

Step S201, the encoding of language input, during initialization, the instruction encoding X and the initialized state feature s ₀ are obtained by inputting the instruction sequence I and [CLS] and [SEP] tokens to the self-attention module in the transformer:

表示self-attention模组的参数，s₀将会在时间步骤t被更新为s_t。Where Concat( ) represents the connection concatenation operation,

Represents the parameters of the self-attention module, s ₀ will be updated to s _{t at time step t} .

通过提示编码器得到提示编码

该提示编码器由两个单模态子提示编码器和一个多模态提示编码器组成，

其中图像子提示和文本子提示分别为

和

和

首先通过单模态子提示编码器得到子提示特征

和

In step S3, the action prompt set is encoded by the modal encoder. use

Get hint code from hint encoder

The cue encoder consists of two unimodal sub-cue encoders and a multi-modal cue encoder,

The image sub-prompt and text sub-prompt are respectively

and

and

First obtain the sub-cue features through a unimodal sub-cue encoder

and

和

输送到多模态提示编码器E^p(·)，得到提示编码

where E ⁱ (·) uses the parameter θ ⁱ and E ^u (·) uses the parameter θ ^u , representing the image sub-hint encoder and the text sub-hint encoder, respectively, and then the

and

Send to the multimodal prompt encoder E ^p ( ), get the prompt code

where θ ^p is the parameter of E ^p ( ), Concat ( ) is the concatenation operation, the encoder E ⁱ ( ), E ^u ( ) and E ^p ( ) consist of a linear layer followed by a dropout operation, to reduce overfitting.

和指令编码X的基础上，通过简单地将X和

连接起来，得到基于提示的指令特征X^p。Step S4, at the prompt encoding

and instruction encoding X, by simply combining X and

concatenated to obtain hint-based instruction features X ^p .

更新：Step S5, the state visual feature _{Kt is} based on the cross-modal attention between _Kt and ^Xp

renew:

分解为

和

获得不同的基于注意力机制增强的特征，参与指令特征

是通过对X进行

加权得到的，基于注意力机制增强的图像子提示特征

和基于注意力机制增强的文本子提示特征

通过对

和

进行

加权获得，

和

用于计算顺序一致性损失L_c，和baseline智能体一样，

用于更新状态特性，最后，将

输入

得到基于提示的动作预测概率

followed by

Decomposed into

and

Obtain different attention-based enhanced features, participating in instruction features

is performed on X by

Weighted, enhanced image sub-cue features based on attention mechanism

and enhanced text sub-cue features based on attention mechanism

through the pair

and

conduct

weighted gain,

and

Used to calculate the sequential consistency loss L _c , like the baseline agent,

is used to update the state property, and finally, the

enter

Get cue-based action prediction probabilities

In step S6, the weighted sum total target of each loss is calculated, and the model is updated and optimized to improve the navigation performance and generalization ability of the agent.

Specifically, step S6 further includes:

Step S600, the modal alignment loss, urges the action prompt to align the already matched image and text sub-prompts in the feature space, following the contrastive learning paradigm used in CLIP, making pairs of images and text features similar, while unpaired images Alienated from text features, the infoNCE loss is used to facilitate feature alignment of image and text sub-cues in each action cue:

表示动作提示p_n的成对的图像和文本子提示的特征，

表示非配对的子提示，通过模态对齐损失，动作提示可以变得更加具有识别力，从而知道学习动作级别的模态对齐。where τ2 is the temperature parameter,

The features representing pairs of image and text sub-prompts of action cue p _n ,

Representing unpaired sub-cues, action cues can be made more discriminative through a modality alignment loss, knowing to learn action-level modal alignment.

以及基于注意力机制增强的指导特征

必须接近：Step S601, loss of sequential consistency, since the instructions usually point to different visual signs sequentially, the action cues in the retrieved action cue set {p _n } are also related to different objects/positions. , sequentially focus on the relevant action cues in the retrieved cue set, and propose a sequential consistency loss, which is the sum of the two unimodal consistency losses; at each time step t, the text sub-cue feature enhanced by the attention mechanism

and guidance features enhanced by attention mechanism

must be close to:

用于提高基于注意力机制增强的图像子提示特征

和基于注意力机制增强的视觉特征之间的相似性，则顺序一致性损失L_c为：definition,

Image sub-cue features for improving attention-based enhancement

and the similarity between visual features enhanced by the attention mechanism, then the sequential consistency loss L _c is:

Step S602, the overall target uses the navigation loss L _n , namely the imitation loss L _IL and the reinforcement learning loss L _RL , and the overall training target is:

L=L _RL +λ ₁ L _IL +λ ₂ L _c +λ ₃ L _a

where λ ₁ , λ ₂ and λ ₃ are the loss weights to balance the losses.

FIG. 3 is an example diagram of a visual language navigation module for modal alignment action prompts in a specific embodiment of the present invention.

This figure shows a comparison of action decisions between baseline agents and . With action prompts related to "going to the stairs", the present invention can select the correct action to successfully navigate in a given observation.

FIG. 4 is an example diagram of the construction of an action prompt library of an action prompt set generation module in a specific embodiment of the present invention.

The present invention uses a two-branch scheme to collect image and text sub-prompts. First, for an instruction path instance in the training data set, a newly developed contrastive language image pre-training CLIP with strong cross-modal object/location-level alignment capabilities is used. Model that replaces the token {CLASS} token in the phrase "a photo of {CLASS}" with a visual object/location whose class label is c. Calculate the probability that an image B belongs to class c in the action sequence, and then select the image with the greatest similarity to the phrase as the image sub-prompt. For textual subprompts, a nearest verb search scheme is used, i.e. finding the nearest verb (in a pre-built verb vocabulary) before a particular object/position word.

FIG. 5 is a comparative display of a result sample of the result navigation using the visual language navigation method and the baseline method in the specific embodiment of the present invention. By introducing action prompts, the present invention can accurately make action decisions and complete successful navigation. The present invention performs the correct "walking through the window" action in the first two navigation steps with the help of the action cues related to "walking through the window". However, the baseline agent failed to perform the "walking through the window" action during navigation, resulting in an incorrect trajectory.

The same or similar reference numbers correspond to the same or similar parts;

The positional relationship described in the accompanying drawings is only for exemplary illustration, and should not be construed as a limitation on this patent;

Obviously, the above-mentioned embodiments of the present invention are only examples for clearly illustrating the present invention, rather than limiting the embodiments of the present invention. For those of ordinary skill in the art, changes or modifications in other different forms can also be made on the basis of the above description. There is no need and cannot be exhaustive of all implementations here. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the claims of the present invention.

Claims (10)

1. a visual language navigation system based on the action prompt of modal alignment, is characterized in that, comprises: Action prompt set generation module, inputting an instruction to the action prompt set generation module, the agent retrieves the action prompt set related to the instruction from the action prompt library before navigation starts; The visual language navigation module of the modal alignment action cue, the action cue set is connected through the cue coding module, and the output cue feature is connected with the output instruction feature of the text coding module; the cue-based instruction feature and the output visual feature of the visual coding module are provided to Multi-layer transformers are used to make action decisions; The optimized learning modules, namely the modal alignment loss module and the continuous consistency loss module, enable effective action cue learning. 2. The visual language navigation system based on the action prompt of modal alignment according to claim 1, wherein the visual language navigation module of the modal alignment action prompt comprises: Text encoding module This module receives the input of language information, uses the multi-layer transformer neural network to encode separately, and obtains the corresponding feature vector; The prompt decoding module is composed of two single-modality sub-prompt encoders and a multi-modal prompt encoder. The image sub-prompt and text sub-prompt respectively obtain the sub-prompt features through the corresponding single-modal auto-encoder. Input into the multi-modal prompt encoder to obtain prompt features; Visual coding module, this module receives the input of visual observation information, encodes it through the visual encoder, and obtains the corresponding feature vector. 3. The visual language navigation system based on the action prompt of modal alignment according to claim 2, is characterized in that, described optimization learning module comprises: Modal alignment loss module, when the action prompt already has matching image and text sub-prompts, using the InfoNCE loss to align them in the feature space, the action prompt can become more discriminative; A continuous consistency loss module that prompts the agent to sequentially focus on relevant action cues in the retrieved cue set based on its observations. 4. a visual language navigation method applying the described system of claim 3, is characterized in that, comprises the following steps: S1: At the beginning of navigation, the agent obtains the instruction, and retrieves the action prompt set related to the instruction from the action prompt library through the action prompt generation module; S2: Through the visual encoding module and the text encoding module, the neural network respectively encodes the input image information and instruction information, and obtains visual encoding, instruction encoding, and state features respectively; S3: Through the prompt coding mode, the image sub-prompt and the text sub-prompt in the action prompt set respectively obtain the sub-prompt feature through the corresponding single-modal auto-encoder, and then input into the multi-modal prompt encoder after connection to obtain the prompt feature; S4: connect the above-mentioned instruction code and the prompt code to obtain the prompt-based instruction feature, and connect the above-mentioned state feature with the visual code to obtain the state visual feature; S5: By modally aligning the visual language navigation module of action cues, the state visual feature is updated based on the cross-modal attention between itself and cue-based instruction features, and the attention is decomposed into two parts, the first part weights the instruction encoding Update, which is used to update the state features, the second part performs weighted updates on the image and text sub-cue features, which are used to calculate the sequential consistency loss, and feed the state visual features into another self-attention module to obtain the state features with respect to the visual features. Attention score, which is the probability of action prediction based on cues; S6: By optimizing the learning model, combining the commonly used imitation learning loss and reinforcement learning loss, as well as the unique modal alignment loss and continuous consistency loss of the present invention, weighted summation is performed to obtain the total training target, and the model is updated and optimized to improve Agent Navigation Performance and Generalization. 5. The visual language navigation method based on the action prompt of modal alignment according to claim 4, wherein the step S1 comprises the following sub-steps: S100: Construction of action prompt library, in order to align images and action phrases to form action prompts, a two-branch scheme is designed to collect image and text sub-prompts: First, for an instruction path instance in the training dataset, use a pre-created Visual object visual object/location location vocabulary to find the visual object/location mentioned in the instruction, for each visual object/location, get the relevant image and text sub-cues respectively, using cross-modal alignment with excellent 0-shot Capable CLIP for locating object/position related images, to accommodate the inference process of CLIP, replace the token {CLASS} token in the phrase "a photo of {CLASS}" with a visual object/position whose class label is c , the probability that an image B belongs to class c in the action sequence is calculated by:

where τ1 is the temperature parameter, sim is the cosine similarity, _b , wc are the image features and phrase features generated by CLIP, respectively, M is the size of the vocabulary, and then the image with the greatest similarity to the phrase is selected as the image sub-prompt, To obtain textual subcues, a simple nearest verb search scheme is used, i.e. to find the nearest verb before a specific object/position word, which is in a pre-built verb vocabulary, and finally, has the same visual object/position and action Image and text subprompts form an aligned action prompt; S101: Retrieval of action prompt set, at the beginning of navigation, the agent retrieves the action prompt related to the instruction from the action prompt library, and calculates the sentence between each object/position related action phrase in the prompt library and the text sub-prompt Similarity for retrieving the set of action cues associated with the instruction

where N is the size of the set. 6. The visual language navigation method based on the action prompt of modal alignment according to claim 4, wherein the step S2 comprises the following sub-steps: S200: Encoding of visual input, for time step t, each image view O _t,i in the candidate view will use a pre-trained convolutional neural network CNN or transformer to extract image features v _t,i , and then v _t,i are mapped to visual encoding by visual encoder F _v : V _t,i =F _v (v _t,i ; θ _v ) where θ _v is the parameter of F _v , a set of

represents the candidate visual code at time t; S201: The encoding of language input. During initialization, the instruction code X and the initialized state feature s ₀ are obtained by inputting the instruction sequence I and [CLS] and [SEP] tokens to the self-attention module in the transformer:

Where Concat( ) represents the connection concatenation operation,

Represents the parameters of the self-attention module, s ₀ will be updated to s _{t at time step t} . 7. The visual language navigation method based on the action prompt of modal alignment according to claim 4, wherein the step S3 comprises the steps of: use

Get hint code from hint encoder

The cue encoder consists of two unimodal sub-cue encoders and a multi-modal cue encoder,

The image sub-prompt and text sub-prompt are respectively

and

and

First obtain the sub-cue features through a unimodal sub-cue encoder

and

where E ⁱ (·) uses the parameter θ ⁱ and E ^u (·) uses the parameter θ ^u , representing the image sub-hint encoder and the text sub-hint encoder, respectively, and then the

and

Send to the multimodal prompt encoder E ^p ( ), get the prompt code

where θ ^p is the parameter of E ^p ( ), Concat ( ) is the concatenation operation, the encoder E ⁱ ( ), E ^u ( ) and E ^p ( ) consist of a linear layer followed by a dropout operation, to reduce overfitting. 8. The visual language navigation method based on the action prompt of modal alignment according to claim 4, wherein the step S4 comprises the following sub-steps: coding at the prompt

and instruction encoding X, by simply combining X and

concatenated to obtain hint-based instruction features X ^p . 9. The visual language navigation method based on the action prompt of modal alignment according to claim 4, wherein the step S5 comprises the following sub-steps: The state visual feature _{Kt is} based on cross-modal attention between _Kt and ^Xp

renew:

followed by

Decomposed into

and

Obtain different attention-based enhanced features, participating in instruction features

is performed on X by

Weighted, enhanced image sub-cue features based on attention mechanism

and enhanced text sub-cue features based on attention mechanism

through the pair

and

conduct

weighted gain,

and

Used to calculate the sequential consistency loss L _c , like the baseline agent,

is used to update the state property, and finally, the

enter

Get cue-based action prediction probabilities

10. The visual language navigation method based on the action prompt of modal alignment according to claim 4, wherein the step S6 comprises the following sub-steps: S600: Modal alignment loss, prompting action cues that already have matching image and text sub-cues to align in feature space, following the contrastive learning paradigm used in CLIP, making pairs of image and text features similar, while unpaired images and text Text feature alienation, using infoNCE loss to facilitate feature alignment of image and text sub-cues in each action cue:

where τ2 is the temperature parameter,

The features representing pairs of image and text sub-prompts of action cue p _n ,

Represents unpaired sub-prompts, through the modal alignment loss, the action cue can become more discriminative, so as to know the modal alignment of the learned action level; S601: Sequential consistency loss, since instructions usually point to different visual landmarks sequentially, the action cues in the retrieved action cue set {p _n } are also related to different objects/positions. Focusing on the relevant action cues in the retrieved cue set in order, a sequential consistency loss is proposed, that is, the sum of the two unimodal consistency losses; at each time step t, the text sub-cue feature enhanced by the attention mechanism

and guidance features enhanced by attention mechanism

must be close to:

definition,

Image sub-cue features for improving attention-based enhancement

and the similarity between visual features enhanced by the attention mechanism, then the sequential consistency loss L _c is:

S602: The total objective uses the navigation loss L _n , namely the imitation loss L _IL and the reinforcement learning loss L _RL , and the total training objective is: L=L _RL +λ ₁ L _IL +λ ₂ L _c +λ ₃ L _a where λ ₁ , λ ₂ and λ ₃ are the loss weights to balance the losses.

Images