CN118379563B - Navigation model training method and device, electronic equipment and storage medium - Google Patents

Abstract

The present invention relates to the field of visual navigation technology, and provides a navigation model training method, device, electronic device and storage medium, the method comprising: inputting sample images and sample text information in each sample image-text pair into a visual encoder and a text encoder in a navigation model respectively to extract sample image features and sample text features; substituting the sample image features and sample text features corresponding to each sample image-text pair into a contrastive learning loss function, and completing pre-training of the visual encoder and the text encoder when the contrastive learning loss function converges; and training the navigation model based on the pre-trained visual encoder and text encoder. The navigation model trained by the training method of the present invention can accurately judge whether the image under the current viewing angle conforms to the content described by the text information, thereby accurately predicting the next waypoint of the robot.

Description

Navigation model training method, device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of visual navigation technology, and in particular to a navigation model training method, device, electronic equipment and storage medium.

Background Art

Visual language navigation is an embodied artificial intelligence task that controls robots (drones or self-driving vehicles, etc.) through human language text information. Taking drones as an example, this task inputs the model with the drone's overhead view image from the starting point, the starting angle, and the text information of the destination. The model predicts the drone's next waypoint; then the model inputs the text information of the destination, the next waypoint, and all previous waypoints (the starting point and the target point are both waypoints) with the overhead view image and starting angle, and the model predicts the drone's next waypoint until it reaches the target point. Among them, a long-distance action process of robot navigation is divided into several actions, and the stopping point of each action is the waypoint.

At present, there are mainly two problems with visual language navigation technology.

1. Stopping early before reaching the destination. When the model mistakenly considers the current view image as the image corresponding to the destination before reaching the actual destination, early stopping will occur.

2. When the destination is reached but the model fails to effectively identify it and continues to move forward, when the real destination is reached, the model believes that the destination image is not related to the destination in the text information, so it will skip the destination and continue to move forward.

The reasons for the above two problems are that the model fails to accurately determine the relationship between the destinations in the image and text information at the current perspective.

Summary of the invention

The present invention provides a navigation model training method, device, electronic device and storage medium, which are used to solve the problem that the model in the prior art cannot accurately determine the association between the destinations in the image and text information under the current viewing angle.

The invention provides a navigation model training method, which comprises the following steps.

The sample image and sample text information in each sample image-text pair are respectively input into the visual encoder and the text encoder in the navigation model to extract the sample image features and the sample text features.

The sample image features and sample text features corresponding to each sample image-text pair are substituted into the contrastive learning loss function. When the contrastive learning loss function converges, the pre-training of the visual encoder and the text encoder is completed.

The navigation model is trained based on the pre-trained visual encoder and text encoder.

According to a navigation model training method provided by the present invention, the sample images in the sample image-text pairs are constructed based on the corresponding images of waypoints in a preset sample route, the sample text information in the sample image-text pairs is constructed based at least on the sample description text corresponding to the starting waypoint of the navigation stage in the sample route where the waypoint is located, and the navigation stage is divided by the human-computer dialogue waypoints preset in the sample route.

According to a navigation model training method provided by the present invention, each navigation stage in the sample route corresponds to a sample image-text pair, and the sample image in the sample image-text pair is formed by stitching images based on at least the starting point waypoint and the target waypoint of the corresponding navigation stage.

According to a navigation model training method provided by the present invention, the sample image in the sample image-text pair is formed by stitching images of a starting waypoint, a target waypoint and at least one intermediate waypoint of the corresponding navigation stage, and the intermediate waypoint is a waypoint between the starting waypoint and the target waypoint of the corresponding navigation stage.

According to a navigation model training method provided by the present invention, the sample text information in the sample image-text pair is formed by splicing the sample description texts corresponding to the starting point waypoints of the corresponding navigation stage and the previous navigation stage.

According to a navigation model training method provided by the present invention, the contrastive learning loss function is determined based on a first similarity loss function of a visual encoder and a second similarity loss function of a text encoder.

According to a navigation model training method provided by the present invention, before the sample image and sample text information in each sample image-text pair are respectively input into the visual encoder and text encoder in the navigation model to extract the sample image features and sample text features, it also includes: performing image classification pre-training on the visual encoder.

According to a navigation model training method provided by the present invention, before the sample image and sample text information in each sample image-text pair are respectively input into the visual encoder and text encoder in the navigation model to extract the sample image features and the sample text features, it also includes: pre-training the visual encoder for target detection based on a sample data set in the navigation field.

According to a navigation model training method provided by the present invention, the visual encoder is pre-trained for target detection based on a sample data set in the navigation field, including: pre-training the visual encoder with overhead view samples and the corresponding true values of each target category and target frame coordinates in the overhead view samples as labels.

The present invention also provides a navigation model training device, comprising the following units.

The pre-trained feature extraction unit is used to input the sample image and sample text information in each sample image-text pair into the visual encoder and text encoder in the navigation model respectively to extract the sample image features and sample text features.

The pre-training loss calculation unit is used to substitute the sample image features and sample text features corresponding to each sample image-text pair into the contrastive learning loss function, and complete the pre-training of the visual encoder and the text encoder when the contrastive learning loss function converges.

A model training unit is used to train the navigation model based on the pre-trained visual encoder and text encoder.

The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the navigation model training method as described above is implemented.

The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the navigation model training method as described in any one of the above is implemented.

The navigation model training method, device, electronic device and storage medium provided by the present invention are pre-trained by contrast learning of the visual encoder and text encoder in the navigation model before the traditional training of the navigation model. The goal of contrast learning is to make the feature vector distance of the positive sample image-text pair closer and the feature vector distance of the negative sample image-text pair farther. Therefore, through the contrast learning pre-training, for the matching image-text pair, the correlation between the image features and the text features respectively output by the visual encoder and the text encoder is strengthened, and for the unmatched image-text pair, the correlation between the image features and the text features respectively output by the visual encoder and the text encoder is weakened, so that the navigation model can accurately judge whether the image under the current perspective conforms to the content described by the text information. The navigation model as a whole can accurately judge the correlation between the image under the current perspective and the semantics of the text information, so as to accurately predict the next waypoint of the robot, avoiding the problem of stopping in advance before reaching the destination and continuing to move forward without effective recognition after reaching the destination. Moreover, a better initialization model can be obtained after pre-training, which lays a training foundation for the subsequent overall training of the navigation model and can effectively improve the convergence speed and test accuracy of the model training.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a schematic diagram of a navigation model used in the navigation model training method provided by the present invention.

FIG. 2 is a flow chart of the navigation model training method provided by the present invention.

FIG3 is a schematic diagram of pre-training of comparative learning of a visual encoder and a text encoder in the navigation model training method provided by the present invention.

FIG. 4 is a schematic diagram of constructing sample image-text pairs based on sample routes in the navigation model training method provided by the present invention.

FIG5 is a schematic diagram of image classification pre-training of a visual encoder in the navigation model training method provided by the present invention.

FIG6 is a schematic diagram of image target detection pre-training for a visual encoder in the navigation model training method provided by the present invention.

FIG. 7 is a schematic diagram showing the navigation model training method provided by the present invention in stages.

FIG8 is a schematic diagram of the structure of the navigation model training device provided by the present invention.

FIG. 9 is a schematic diagram of the structure of an electronic device provided by the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Since the navigation model based on visual language predicts the next waypoint of the robot's action based on human language text information and the current visual image perceived by the robot, the navigation model at least includes: a visual encoder for image recognition and feature extraction and a text encoder for text information recognition and feature extraction. As shown in FIG1 , it is a schematic diagram of the structure of a navigation model based on visual language, including: a visual encoder 110, a text encoder 120, an angle encoder 130 and a multimodal information fusion module 140, the angle encoder 130 is used to identify the direction angle of the robot at each waypoint and extract the angle feature, and the multimodal information fusion module 140 is used to fuse the image feature, the text feature and the angle feature and output the coordinate value of the next waypoint of the robot until it reaches the target point. Among them, the visual encoder 110 can be MViT (Multiscale Vision Transformer), the text encoder 120 can be Roberta, and the angle encoder 130 can be Projector. MViT is a deep learning model based on the Transformer architecture, specifically designed for computer vision tasks. Unlike traditional convolutional neural networks, MViT uses the Transformer's self-attention mechanism to capture global features in images and processes images at different scales, making it perform better when processing large-scale data sets.

The navigation model training method of the embodiment of the present invention is illustrated below using the above-mentioned navigation model structure as an example, but is not limited to the above-mentioned navigation model structure. As long as the navigation model structure includes the visual encoder 110 and the text encoder 120, or includes corresponding functional modules of visual encoding and text encoding, it is applicable to the navigation model training method of the embodiment of the present invention.

The navigation model training method according to the embodiment of the present invention, as shown in FIGS. 2 and 3 , includes the following steps S210 to S230 .

Step S210: Input the sample image and sample text information in each sample image-text pair into the visual encoder 110 and the text encoder 120 in the navigation model respectively to extract the sample image features and the sample text features. The sample image-text pair is a picture-text pair composed of the sample image and the sample text information. The sample image-text pair composed of the matched sample image and the sample text information is the positive sample data, and the sample image-text pair composed of the unmatched sample image and the sample text information is the negative sample data. When the sample image has part or all of the content described in the sample text information, the sample image and the sample text information match, otherwise they do not match. For example, for the sample text information of "the destination is at three o'clock in your direction, which is a small rectangle parallel to the road", as long as there are sample images of roads and/or rectangular objects, they will match the sample text information.

Step S220: Substitute the sample image features and sample text features corresponding to each sample image-text pair into the contrastive learning loss function, and complete the pre-training of the visual encoder 110 and the text encoder 120 when the contrastive learning loss function converges. In this embodiment, through contrastive learning training, when the contrastive learning loss function converges, for the positive sample image-text pair, the correlation between the sample image features and the sample text features output by the visual encoder 110 and the text encoder 120 is strengthened.

Step S230: Train the navigation model based on the pre-trained visual encoder 110 and text encoder 120. In this step, the navigation model can be trained using an existing training method. Since the visual encoder 110 and text encoder 120 have been pre-trained in the above steps S210 and S220, the corresponding image features and text features have a strong correlation for the matching image and text information, so that the trained navigation model can accurately determine the correlation between the image and text information semantics under the current perspective, avoiding incorrect determination of the target point, resulting in premature stopping before reaching the destination, and continuing to move forward without effective recognition after reaching the destination.

In the navigation model training method of this embodiment, before the traditional training of the navigation model, the visual encoder 110 and the text encoder 120 in the navigation model are pre-trained by contrastive learning through step S210 and step S220. The goal of contrastive learning is to make the feature vector distance of the positive sample image-text pair closer and the feature vector distance of the negative sample image-text pair farther. Therefore, through the contrastive learning pre-training, for the matching image-text pair, the correlation between the image features and the text features output by the visual encoder 110 and the text encoder 120 is strengthened, and for the unmatched image-text pair, the correlation between the image features and the text features output by the visual encoder 110 and the text encoder 120 is weakened, so that the navigation model can accurately judge whether the image under the current perspective conforms to the content described by the text information, thereby accurately predicting the next waypoint of the robot, avoiding the problem of stopping in advance before reaching the destination and continuing to move forward without effective recognition after reaching the destination. Moreover, after pre-training, a better initialization model can be obtained, which lays a training foundation for the subsequent overall training of the navigation model and can effectively improve the convergence speed and test accuracy of the model training.

In some embodiments, the sample images in the sample image-text pair are constructed based on the images corresponding to the waypoints in the preset sample routes, and the sample text information in the sample image-text pair is constructed based on at least the sample description text corresponding to the starting point waypoint of the navigation stage in the sample route where the waypoint is located, and the navigation stage is divided by the preset human-computer dialogue waypoints in the sample route. In this embodiment, an AVDN dataset in the field of visual dialogue navigation can be used, and the AVDN dataset includes multiple sample routes.

Among them, the sample route is the sample data representing the robot's action route. The human-machine dialogue waypoint is when marking the sample data. When the robot reaches a certain waypoint, it thinks that it has reached the target point, and asks the artificial platform whether it has reached the target point. When the artificial platform determines that it has not reached the target point, it replies with a text message (for example: the target point is in the southeast direction of the current navigation point). After receiving the reply, the robot navigates again and enters the next navigation stage. The human-machine dialogue waypoint is the starting waypoint of the next navigation stage and the target waypoint of the previous navigation stage. The dialogue information of this time is the sample description text of the target point corresponding to the dialogue waypoint. As shown in Figure 4, a sample route from the starting point to the target point, all points between the starting point and the target point are waypoints, and each waypoint in the sample route includes the corresponding image and coordinates. In the sample route, the starting point and the human-machine dialogue waypoints correspond to the sample description text of the target point respectively. The sample description text of the target point corresponding to the starting point is the original description information of the target point, for example: the destination is at your three o'clock direction, which is a small rectangle parallel to the road. Therefore, the simplest way to construct sample image-text pairs is to use the image corresponding to any navigation point as the sample image, and the sample description text corresponding to the starting point waypoint of the navigation stage where the any navigation point is located as the sample text information.

In some embodiments, each navigation stage in the sample route corresponds to a sample image-text pair, and the sample image in the sample image-text pair is formed by stitching together at least the images of the starting waypoint and the target waypoint of the corresponding navigation stage, and the corresponding sample text information can still be the sample description text corresponding to the starting waypoint of the navigation stage. Specifically, as shown in FIG4 , each navigation stage corresponds to a sample image-text pair, so there are three sample image-text pairs in FIG4 . For the first navigation stage, the sample image in the sample image-text pair is formed by stitching together at least the images of the starting point and the first human-computer dialogue waypoint, and the sample text information is the sample description text corresponding to the starting point. The stitching here means that the image boundaries are connected to form an image.

In this embodiment, each navigation stage can generate at least one positive sample image-text pair, and the mismatched image-text pairs are used as negative sample image-text pairs. During contrastive learning pre-training, the navigation model can more easily learn the correlation between sample images and sample text information from the contrastive learning training of positive and negative samples. Since the image semantic information corresponding to the starting waypoint and the target waypoint is most appropriate to the text description and has the highest correlation with the text, after contrastive learning pre-training, the image features and text features extracted by the visual encoder 110 and the text encoder 120 respectively have a stronger correlation for the matching image and text information.

In some embodiments, the sample image in the sample image-text pair is formed by stitching images of the starting waypoint, the target waypoint, and at least one intermediate waypoint of the corresponding navigation phase, wherein the intermediate waypoint is a waypoint between the starting waypoint and the target waypoint of the corresponding navigation phase, for example, two intermediate waypoints can be randomly selected, and a total of four images are stitched into a whole image as a sample image. Although the correlation between the intermediate waypoints and the sample text information is lower, the random selection of the intermediate waypoints also improves the richness of the selected images, so that the navigation model can learn more image information during the training process, and to a certain extent enhances the robustness of the navigation model.

In some embodiments, the sample text information in the sample image-text pair is formed by splicing the sample description texts corresponding to the starting point waypoints of the corresponding navigation stage and the previous navigation stage. That is, a sample image-text pair includes not only the sample description text corresponding to the starting point waypoint of the current navigation stage, but also the sample description text corresponding to the starting point waypoint of the previous navigation stage. The sample text information contains richer description text information of the target point, so that after the contrast learning pre-training, the image features and text features extracted by the visual encoder 110 and the text encoder 120 respectively have stronger correlation for the matching image and text information.

In some embodiments, the contrastive learning loss function is determined based on the first similarity loss function of the visual encoder 110 and the second similarity loss function of the text encoder 120. Specifically, the contrastive learning loss function as follows.

Among them, sim () is the similarity function, which is calculated by matrix dot multiplication, exp () represents the natural exponential function, and represent the first similarity loss function and the second similarity loss function respectively, represents the i -th sample image-text pair among N sample image-text pairs, represents the sample image of the i -th sample image-text pair, represents the sample text information of the i -th sample image-text pair, is the learnable temperature parameter.

In some embodiments, the navigation model includes: an angle encoder 130, a multimodal information fusion module 140, the visual encoder 110 and the text encoder 120. Based on the navigation model structure, step 230 specifically includes the following steps.

For any waypoint in the sample route, the sample images and the sample direction angles of the drone corresponding to the waypoint and the waypoints before the waypoint in the navigation stage are respectively input into the visual encoder and the angle encoder to extract sample image features and sample angle features; the sample description texts corresponding to the waypoint and all previous waypoints are spliced into sample text information, and the sample text information is input into the text encoder to extract sample text features. Specifically, in this embodiment, the sample routes in the AVDN data set can also be used for training, and each time a waypoint is predicted, the sample description texts corresponding to the waypoint and all previous waypoints (the starting point and all human-machine dialogue waypoints) are spliced into sample text information. During the training process, the navigation model can learn more semantic information related to the sample description texts, and associate more text semantic information with the sample images, so that the navigation model can more accurately predict the next waypoint.

The sample image features, sample text features and sample angle features are input into the multimodal information fusion module to obtain the predicted coordinate value of the next waypoint of any waypoint output by the multimodal information fusion module.

The actual coordinate value of the next waypoint of any waypoint in the sample route and the predicted coordinate value are substituted into the preset action prediction loss function. When the action prediction loss function converges, the training is completed. The action prediction loss function is as follows.

Where, T is the number of samples currently executed, ( a , b ) is the true value of the two-dimensional coordinates of the next position of the drone, ( , ) is the predicted value of the two-dimensional coordinate model of the next position of the UAV.

In some embodiments, before step S210, it also includes: performing image classification pre-training on the visual encoder 110. Specifically, as shown in Figure 5, this embodiment uses the ImageNet data set to pre-train the visual encoder 110 for large-scale image classification tasks, and the visual encoder 110 can be MViT. ImageNet is a database containing more than 14 million images, covering more than 1,000 categories, and is one of the largest and most widely used image recognition data sets. The images in this data set are manually annotated, and each image corresponds to a category label, which provides rich data resources for image classification training. In order to enable the visual encoder 110 to better extract image features, in this embodiment, the visual encoder 110 is pre-trained for image classification. In the image classification pre-training, the cross entropy loss function is used to compare the classification results, and gradient back propagation is performed. The cross entropy loss function measures the difference between the category probability distribution output by the model and the true label. Its mathematical expression is as follows.

Where N is the number of samples, C is the number of categories, is the one-hot encoding of the true label c of sample i , is the probability that the model predicts that sample i is of category c .

In this embodiment, through image classification pre-training, the visual encoder 110 can learn the low-level features of the image (local pixel changes in the image) and high-level semantic information (what the image is), thereby paving the way for subsequent model pre-training and training stages, making it easier for the visual encoder 110 or the entire navigation model to adapt to task-specific data sets in subsequent stages, while improving the visual encoder 110's ability to abstract and generalize image features.

In some embodiments, before step S210, it also includes: pre-training the visual encoder 110 for target detection in images based on a sample dataset in the navigation field, so as to further improve the feature extraction capability of the visual encoder 110 for the domain-specific dataset.

For the UAV navigation task, the image comes from the top view formed by aerial aerial photography. Therefore, in order to train a navigation model suitable for UAV navigation, it is preferred to use the top view samples and the corresponding target category true values and target frame coordinate true values in the top view samples as labels to pre-train the visual encoder 110, which may be MViT.

The specific training process is shown in Figure 6. Since satellite images are also bird's-eye views, high-resolution satellite images of the xView dataset and their corresponding label information are used for pre-training of specific target detection tasks in the field of drone navigation. The xView dataset is a satellite image dataset collected from the WorldView-3 satellite at 0.3 meters above the ground. The fine-grained categories used for target detection in this dataset include buildings, vehicles, and infrastructure, etc. These images cover a wide range of geographical locations around the world, providing complex backgrounds. The image resolution in the dataset is very high, allowing for detailed analysis of small and complex destinations. Target detection pre-training contains two loss functions: the bounding box classification loss function and the bounding box regression loss function. Paired images, category labels y , and bounding box label coordinates t are input into the MViT model to calculate the loss function. Among them, the bounding box classification loss function is expressed as follows.

Among them, N is the number of bounding boxes, C is the number of categories, represents the label of category c of the i -th bounding box, is the predicted probability that the i -th predicted bounding box belongs to category c .

The bounding box regression loss function expression is as follows.

in, is the number of positive samples, pos is the index of the number of positive samples, represents the label coordinates of the i -th bounding box, Represents the predicted coordinates of the predicted i -th bounding box. is the smooth L1 loss function, which is expressed as follows.

The final loss function is: .

As shown in FIG. 7 , it is a schematic diagram of a method for training a drone navigation model in a specific embodiment, which includes the following training steps.

The first stage of pre-training: the image classification pre-training of the visual encoder 110 is performed using the ImageNet dataset.

Second stage pre-training: Use the xView dataset to perform target detection pre-training on the visual encoder 110 that has completed the first stage pre-training. The samples in the xView dataset are all high-resolution satellite images. Therefore, before pre-training in this stage, the high-resolution satellite images are smoothed and an image is evenly divided into four blocks to match the resolution of the image during the first stage pre-training. After being divided into four blocks, the label information is converted. After the image is divided into four blocks, the coordinates of the bounding box of the target in each block are converted to the coordinates relative to the image block.

The third stage pre-training: the AVDN dataset is used to perform comparative learning pre-training on the text encoder 120 and the visual encoder 110 completed in the second stage pre-training.

Navigation model training stage: The UAV navigation model is trained based on the visual encoder 110 and text encoder 120 pre-trained in the third stage.

In this embodiment, in the first stage of pre-training, large-scale image data is used to perform image classification training on the visual encoder 110, so that the visual encoder 110 can extract better visual features; in the second stage of pre-training, a satellite map dataset is used to perform target detection tasks on the visual encoder 110, further improving the feature extraction capability of the visual encoder 110 for domain-specific datasets; in the third stage of pre-training, an unmanned aerial vehicle visual language navigation AVDN dataset is used to perform comparative learning training on text and image data, and for matching images and text information, the association between the image features and text features output by the text encoder and the visual encoder respectively is strengthened, so that the navigation model can accurately determine whether the image at the current perspective conforms to the text description, and the three stages of pre-training lay a foundation for subsequent overall model training, which can effectively improve the convergence speed and test accuracy of model training.

The navigation model training method of the above embodiment is not only suitable for the navigation model of unmanned aerial vehicles, but also suitable for the navigation models of other robots, for example: the navigation model of automated working machinery operating in the field, and the navigation model of fully automatic driving vehicles.

The navigation model training device provided by the present invention is described below. The navigation model training device described below and the navigation model training method described above can be referred to each other.

The navigation model training device according to the embodiment of the present invention, as shown in FIG8 , includes the following modules.

The pre-trained feature extraction unit 810 is used to input the sample image and sample text information in each sample image-text pair into the visual encoder and text encoder in the navigation model respectively, so as to extract the sample image features and sample text features.

The pre-training loss calculation unit 820 is used to substitute the sample image features and sample text features corresponding to each sample image-text pair into the contrastive learning loss function, and complete the pre-training of the visual encoder and the text encoder when the contrastive learning loss function converges.

The model training unit 830 is used to train the navigation model based on the pre-trained visual encoder and text encoder.

The navigation model training device of the embodiment of the present invention performs contrast learning pre-training on the visual encoder 110 and the text encoder 120 in the navigation model before the traditional training of the navigation model. The goal of contrast learning is to make the feature vector distance of the positive sample image-text pair closer and the feature vector distance of the negative sample image-text pair farther. Therefore, through the contrast learning pre-training, for the matching image-text pair, the correlation between the image features and the text features respectively output by the visual encoder 110 and the text encoder 120 is strengthened, and for the unmatched image-text pair, the correlation between the image features and the text features respectively output by the visual encoder 110 and the text encoder 120 is weakened, so that the navigation model can accurately judge whether the image at the current perspective conforms to the content described by the text information. The navigation model as a whole can accurately judge the correlation between the image at the current perspective and the semantics of the text information, thereby accurately predicting the next waypoint of the robot, avoiding the problem of stopping in advance before reaching the destination and continuing to move forward without effective recognition after reaching the destination. Moreover, a better initialization model can be obtained after pre-training, which lays a training foundation for the subsequent overall training of the navigation model and can effectively improve the convergence speed and test accuracy of the model training.

Optionally, the sample image in the sample image-text pair is constructed based on the corresponding image of the waypoint in the preset sample route, and the sample text information in the sample image-text pair is constructed based at least on the sample description text corresponding to the starting waypoint of the navigation stage in the sample route where the waypoint is located, and the navigation stage is divided by the human-computer dialogue waypoints preset in the sample route.

Optionally, each navigation stage in the sample route corresponds to a sample image-text pair, and the sample image in the sample image-text pair is formed by stitching images of at least a starting point waypoint and a target waypoint of the corresponding navigation stage.

Optionally, the sample image in the sample image-text pair is formed by stitching images of a starting waypoint, a target waypoint and at least one intermediate waypoint of the corresponding navigation phase, and the intermediate waypoint is a waypoint between the starting waypoint and the target waypoint of the corresponding navigation phase.

Optionally, the sample text information in the sample image-text pair is formed by splicing sample description texts corresponding to respective starting point waypoints of the corresponding navigation phase and the previous navigation phase.

Optionally, the contrastive learning loss function is determined based on a first similarity loss function of the visual encoder and a second similarity loss function of the text encoder.

Optionally, the navigation model training device also includes: a classification pre-training unit, which is used to perform image classification pre-training on the visual encoder before inputting the sample image and sample text information in each sample image-text pair into the visual encoder and text encoder in the navigation model respectively to extract sample image features and sample text features.

Optionally, the navigation model training device also includes: a target detection pre-training unit, which is used to perform target detection pre-training on the visual encoder based on a sample data set in the navigation field before inputting the sample image and sample text information in each sample image-text pair into the visual encoder and text encoder in the navigation model respectively to extract sample image features and sample text features.

Optionally, the object detection pre-training unit is specifically used to pre-train the visual encoder using the top view samples and the corresponding true values of each object category and the true values of the object frame coordinates in the top view samples as labels.

FIG9 illustrates a schematic diagram of a physical structure of an electronic device. As shown in FIG9 , the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the navigation model training method, which includes the following steps.

The sample image and sample text information in each sample image-text pair are respectively input into the visual encoder and the text encoder in the navigation model to extract the sample image features and the sample text features.

The sample image features and sample text features corresponding to each sample image-text pair are substituted into the contrastive learning loss function. When the contrastive learning loss function converges, the pre-training of the visual encoder and the text encoder is completed.

The navigation model is trained based on the pre-trained visual encoder and text encoder.

In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the navigation model training method provided by the above methods, which includes the following steps.

The sample image and sample text information in each sample image-text pair are respectively input into the visual encoder and the text encoder in the navigation model to extract the sample image features and the sample text features.

The sample image features and sample text features corresponding to each sample image-text pair are substituted into the contrastive learning loss function. When the contrastive learning loss function converges, the pre-training of the visual encoder and the text encoder is completed.

The navigation model is trained based on the pre-trained visual encoder and text encoder.

On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the navigation model training method provided by the above methods is implemented, and the method includes the following steps.

The sample image and sample text information in each sample image-text pair are respectively input into the visual encoder and the text encoder in the navigation model to extract the sample image features and the sample text features.

The sample image features and sample text features corresponding to each sample image-text pair are substituted into the contrastive learning loss function. When the contrastive learning loss function converges, the pre-training of the visual encoder and the text encoder is completed.

The navigation model is trained based on the pre-trained visual encoder and text encoder.

The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A navigation model training method, characterized by comprising: Inputting the sample image and sample text information in each sample image-text pair into the visual encoder and text encoder in the navigation model respectively to extract sample image features and sample text features; Substituting the sample image features and sample text features corresponding to each sample image-text pair into the contrastive learning loss function, and completing the pre-training of the visual encoder and the text encoder when the contrastive learning loss function converges; Training the navigation model based on the pre-trained visual encoder and text encoder; Among them, the sample images in the sample image-text pair are constructed based on the corresponding images of waypoints in a preset sample route, the sample text information in the sample image-text pair is constructed based on at least the sample description text corresponding to the starting waypoint of the navigation stage in the sample route where the waypoint is located, and the navigation stage is divided by the human-computer dialogue waypoints preset in the sample route; each navigation stage in the sample route corresponds to a sample image-text pair, and the sample images in the sample image-text pair are formed by stitching together the images of at least the starting waypoint and the target waypoint of the corresponding navigation stage. 2. The navigation model training method according to claim 1 is characterized in that the sample image in the sample image-text pair is formed by stitching images of a starting waypoint, a target waypoint and at least one intermediate waypoint of the corresponding navigation stage, and the intermediate waypoint is a waypoint between the starting waypoint and the target waypoint of the corresponding navigation stage. 3. The navigation model training method according to claim 1 or 2 is characterized in that the sample text information in the sample image-text pair is formed by splicing the sample description texts corresponding to the starting point waypoints of the corresponding navigation stage and the previous navigation stage. 4. The navigation model training method according to claim 1 is characterized in that the contrastive learning loss function is determined based on a first similarity loss function of a visual encoder and a second similarity loss function of a text encoder. 5. The navigation model training method according to claim 1 is characterized in that before the sample image and sample text information in each sample image-text pair are respectively input into the visual encoder and the text encoder in the navigation model to extract the sample image features and the sample text features, it also includes: The visual encoder is pre-trained for image classification. 6. The navigation model training method according to claim 1, characterized in that before the sample image and sample text information in each sample image-text pair are respectively input into the visual encoder and the text encoder in the navigation model to extract the sample image features and the sample text features, it also includes: The visual encoder is pre-trained for target detection based on a sample dataset in the navigation field. 7. The navigation model training method according to claim 6, characterized in that the visual encoder is pre-trained for target detection based on a sample data set in the navigation field, comprising: The visual encoder is pre-trained using the top view samples and the corresponding true values of each target category and the true values of the target frame coordinates in the top view samples as labels. 8. A navigation model training device, characterized in that it comprises: A pre-trained feature extraction unit, used for inputting sample image and sample text information in each sample image-text pair into a visual encoder and a text encoder in the navigation model respectively, so as to extract sample image features and sample text features; A pre-training loss calculation unit, used for substituting the sample image features and sample text features corresponding to each sample image-text pair into a contrastive learning loss function, and completing the pre-training of the visual encoder and the text encoder when the contrastive learning loss function converges; A model training unit, used for training the navigation model based on the visual encoder and text encoder obtained by pre-training; Among them, the sample images in the sample image-text pair are constructed based on the corresponding images of waypoints in a preset sample route, the sample text information in the sample image-text pair is constructed based on at least the sample description text corresponding to the starting waypoint of the navigation stage in the sample route where the waypoint is located, and the navigation stage is divided by the human-computer dialogue waypoints preset in the sample route; each navigation stage in the sample route corresponds to a sample image-text pair, and the sample images in the sample image-text pair are formed by stitching together the images of at least the starting waypoint and the target waypoint of the corresponding navigation stage. 9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the navigation model training method according to any one of claims 1 to 7 when executing the program. 10. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the navigation model training method according to any one of claims 1 to 7 is implemented.