CN117351161A - Mapping methods, devices, storage media and electronic devices based on visual semantics - Google Patents

Abstract

The application provides a diagram building method and device based on visual semantics, a storage medium and electronic equipment, and relates to the field of intelligent driving. The method comprises the following steps: acquiring a plurality of paths of looking-around images of a target vehicle at the current moment acquired in a target environment; performing aerial view stitching on the multipath surrounding images to generate a first key frame image; determining a plurality of historical frame images without a vehicle bottom blind area corresponding to the first key frame image; based on a plurality of historical frame images, texture mapping is carried out on the blind areas of the vehicle bottom in the first key frame image, and a second key frame image is obtained; converting the second key frame image into a local semantic map based on the initial pose of the target vehicle; and splicing the local semantic maps of the target vehicle at a plurality of moments in the target environment to generate a global map of the target environment. According to the scheme, the influence of feature shielding and truncation caused by the visual blind area can be eliminated, and the continuity and accuracy of the semantic features of the second key frame image are ensured.

Description

Mapping methods, devices, storage media and electronic devices based on visual semantics

Technical field

This application relates to the field of intelligent driving technology, and specifically to a mapping method, device, storage medium and electronic equipment based on visual semantics.

Background technique

With the development of intelligent driving technology, vehicles are commonly equipped with panoramic surround-view imaging systems, allowing drivers to better understand the environment around the vehicle. However, current panoramic surround-view imaging systems still have a visual blind spot under the car.

In related applications, for panoramic mosaics, deep neural networks can be used to segment static road sign elements, and the segmentation results can be used as matching features between images. However, when there is a blind spot under the car in the panoramic surround view spliced image, the relevant features will be truncated or completely blocked, thereby increasing the error rate in the feature matching process, causing the mapping error to increase or even fail.

Contents of the invention

In view of this, embodiments of the present application provide a mapping method, device, storage medium and electronic device based on visual semantics.

In the first aspect, an embodiment of the present application provides a mapping method based on visual semantics, which includes: obtaining multi-channel surround images of the target vehicle at the current moment collected in the target environment; performing bird's-eye view splicing of the multi-channel surround images to generate A first key frame image, the first key frame image includes a blind spot under the car; determine a plurality of historical frame images without a blind spot under the car corresponding to the first key frame image; based on the multiple historical frame images, calculate the blind spots in the first key frame image Texture mapping is performed on the blind area under the vehicle to obtain the second key frame image without the blind area under the vehicle; based on the initial pose of the target vehicle, the second key frame image is converted into a local semantic map; multiple images of the target vehicle in the target environment are The local semantic maps at each moment are spliced together to generate a global map of the target environment.

Combined with the first aspect, in some implementations of the first aspect, texture mapping is performed on the blind area under the vehicle in the first key frame image based on multiple historical frame images to obtain a second key frame image without the blind area under the vehicle, Including: determining the reference frame image used to map the blind spot under the vehicle in the first key frame image among multiple historical frame images; determining the wheel speed pulse count value of the target vehicle at the current moment, and the corresponding history of the target vehicle in the reference frame image. The wheel speed pulse count value of the target vehicle at the current moment; based on the wheel speed pulse count value of the target vehicle at the current moment and the wheel speed pulse count value of the historical moment, determine the rotation angle and deflection of the target vehicle in the first key frame image and the reference frame image. Shift amount; based on the rotation angle and offset, determine the effective mapping area in the reference frame image; based on the effective mapping area, texture map the blind area under the car in the first key frame image to obtain the second key frame image.

In conjunction with the first aspect, in some implementations of the first aspect, determining a reference frame image for mapping the vehicle bottom blind spot in the first key frame image among multiple historical frame images includes: based on the driving parameters of the target vehicle , respectively determine the transformation matrices of multiple historical frame images and the first key frame image; based on the target sampling error and using the feature information in the first key frame image as the standard, randomly sample the corresponding feature information of the multiple historical frame images , determine the corresponding sampling values of multiple historical frame images, and the sampling values represent the matching degree of the feature information in the historical frame image and the feature information in the first key frame image; based on the multiple historical frame images and the first key frame image The transformation matrix and corresponding sampling values of multiple historical frame images are used to determine the reference frame image from multiple historical frame images.

Combined with the first aspect, in some implementations of the first aspect, texture mapping is performed on the blind area under the vehicle in the first key frame image based on the effective mapping area to obtain a second key frame image, including: determining the blind area under the vehicle Contour line; along the contour line, expand the target distance away from the blind area under the car to obtain the outer edge line, and use the area composed of the contour line and the outer edge line as the transition area; based on the effective mapping area, map the car in the first key frame image Perform texture mapping on the bottom blind area to obtain the image to be calibrated; determine the gray gradient value of the effective mapping area in the image to be calibrated; based on the gray gradient value, calibrate the gray value of the transition area in the image to be calibrated to obtain the second key frame image.

Combined with the first aspect, in some implementations of the first aspect, based on the initial pose of the target vehicle, converting the second key frame image into a local semantic map includes: extracting semantic features from the second key frame image to obtain The semantic features of the target object contained in the second key frame image. The semantic features include position features, direction features and category features; determine the pose of the target vehicle at the current moment; based on the semantic features of the target object contained in the second key frame image. , the pose and initial pose of the target vehicle at the current moment, and convert the second key frame image into a local semantic map.

Combined with the first aspect, in some implementations of the first aspect, local semantic maps of the target vehicle at multiple moments in the target environment are spliced to generate a global map of the target environment, including: local semantics at multiple moments Determine at least two local semantic maps to be matched in the map; determine the target point cloud and source point cloud in at least two local semantic maps to be matched; determine each point cloud in the target point cloud based on the geometric characteristics of the target point cloud point to the nearest point of the source point cloud; based on the nearest point of each point in the target point cloud to the source point cloud, the local semantic maps of the target vehicle at multiple moments in the target environment are spliced to generate a map of the target environment. Global map.

Combined with the first aspect, in some implementations of the first aspect, the mapping method based on visual semantics also includes: matching features in the global map based on the initial pose of the target vehicle, and determining the current pose of the target vehicle. .

In the second aspect, an embodiment of the present application provides a mapping device based on visual semantics, including: an acquisition module for acquiring multi-channel surround images of the target vehicle at the current moment collected in the target environment; a first splicing module, Used to perform bird's-eye splicing of multi-channel surround-view images to generate a first key frame image, which includes a blind spot under the car; a determination module used to determine multiple histories of no blind spots under the car corresponding to the first key frame image Frame image; mapping module, used to texture map the vehicle bottom blind area in the first key frame image based on multiple historical frame images, to obtain a second key frame image without vehicle bottom blind area; conversion module, used to perform texture mapping based on the target vehicle The initial pose converts the second key frame image into a local semantic map; the second splicing module is used to splice the local semantic maps of the target vehicle at multiple moments in the target environment to generate a global map of the target environment.

In a third aspect, an embodiment of the present application provides a computer-readable storage medium that stores a computer program, and the computer program is used to execute the method described in the first aspect.

In a fourth aspect, an embodiment of the present application provides an electronic device. The electronic device includes: a processor; a memory used to store instructions executable by the processor; and the processor is used to execute the method described in the first aspect.

In this embodiment, through motion compensation, the historical frame image is mapped to the blind area of the car bottom of the first key frame image according to specific transformation conditions, eliminating the feature occlusion and truncation effects caused by the visual blind area, and ensuring the second The continuity and accuracy of the semantic features of key frame images avoid feature truncation or loss during the mapping process, provide assistance to users in driving, and improve the user experience. In addition, this application does not require the installation of additional camera modules and is low cost.

Description of drawings

The above and other objects, features and advantages of the present application will become more apparent by describing the embodiments of the present application in more detail in conjunction with the accompanying drawings. The drawings are used to provide further understanding of the embodiments of the present application, and constitute a part of the specification. They are used to explain the present application together with the embodiments of the present application, and do not constitute a limitation of the present application. In the drawings, like reference numbers generally represent like components or steps.

Figure 1 shows a schematic flow chart of mapping provided by an exemplary embodiment of the present application.

Figure 2 shows a schematic flowchart of obtaining a second key frame image provided by an exemplary embodiment of the present application.

Figure 3 shows a schematic flowchart of conversion into a local semantic map provided by an exemplary embodiment of the present application.

Figure 4 shows a schematic flowchart of generating a global map provided by an exemplary embodiment of the present application.

Figure 5 shows a schematic structural diagram of a semantic map-based mapping device provided by an embodiment of the present application.

FIG. 6 shows a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

Traditional SLAM (Simultaneous Localization and Mapping, instant positioning and map construction) technology only contains some low-level information and cannot meet the development of modern computer vision. With the rise of the concept of artificial intelligence, neural network technology is used to achieve image classification, detection, segmentation, etc. It outperforms traditional image processing in many aspects, and has initially shown huge advantages in industries such as autonomous driving, robots, drones, and medical care. Compared with traditional vSLAM (visual Simultaneous Localization and Mapping, visual simultaneous localization and mapping), semantic vSLAM can not only obtain geometric structure information in the environment, but also extract semantic information of independent objects. In mapping, semantic information provides rich object information to build different types of semantic maps, such as pixel-level maps and object-level maps. In positioning, semantic vSLAM improves the accuracy and robustness of positioning with the help of semantic constraints. Therefore, semantic vSLAM can help robots improve their accurate perception and adaptability to unknown complex environments to perform more complex tasks. However, when there is a blind spot under the vehicle in the local semantic map, the relevant features will be truncated or completely blocked, thereby increasing the error rate in the feature matching process, causing the mapping error to increase or even fail.

In view of this, this application provides a mapping method based on visual semantics. Motion compensation is performed based on the acquired first key frame image and vehicle body motion information, thereby filling the semantic features into the blind area of the vehicle bottom of the first key frame image, ultimately making the semantic information more comprehensive and robust, and the success rate of feature matching higher. high.

Figure 1 shows a schematic flow chart of mapping provided by an exemplary embodiment of the present application. As shown in Figure 1, in the embodiment of the present application, the mapping method based on visual semantics includes the following steps.

Step S110: Obtain the multi-channel surround image of the target vehicle at the current moment collected in the target environment.

For example, the panoramic surround view image system of the target vehicle is used to generate the panoramic surround view top view. For example, the video stream obtained through multiple cameras installed around the target vehicle is input into the panoramic surround view imaging system for preprocessing to obtain a multichannel surround view image; or, camera a and camera b installed around the target vehicle are , the multiple images captured by camera c and camera d are used as multi-channel surround images. The multiple images can be two-dimensional images or three-dimensional images. This is not limited in the embodiment of the present application.

Step S120: perform bird's-eye view splicing of multi-channel surround images to generate a first key frame image.

The first key frame image includes the blind area under the car. For example, first, the bird's-eye view of the cameras in different directions and positions of the target vehicle is changed, and then spliced to obtain the first key frame image of the bird's-eye view of the target vehicle and the surrounding area. More specifically, the cameras in the target vehicle are generally installed at an oblique downward angle. Therefore, the original surround image output by the camera is not a top view. In order to achieve the effect of a bird's eye view, the surround images output by different cameras need to be projected onto the new top view. For example, a matrix can be used to represent the change of a point in the original surround image output by the camera. Specifically, its expression is as follows. After that, the surround images of different paths are spliced through calibration to obtain the first key frame image.

Step S130: Determine multiple historical frame images without vehicle bottom blind spots corresponding to the first key frame image.

The above-mentioned first key frame image and the multiple historical frame images may be continuous in time sequence, or may be discontinuous. When the first key frame image and multiple historical frame images are discontinuous, the time interval between each two frames needs to be within the preset range to prevent the difference between the two frames of point cloud data from being too large and improve subsequent inheritance. Timeliness of target object information of multiple historical frame images. It should be noted that the first key frame image is an image taken at the current time, and the multiple historical frame images are images taken before the current time.

Step S140: Based on multiple historical frame images, perform texture mapping on the blind area under the vehicle in the first key frame image to obtain a second key frame image without the blind area under the vehicle.

For example, the historical frame image with the highest correlation with the first key frame image is selected from multiple historical frame images, and an area used for texture mapping is determined in the historical frame image. Obtain the three-dimensional projection model corresponding to the panoramic surround-view imaging system, obtain the world coordinates of the model points of the three-dimensional projection model in the world coordinate system, calculate the world coordinates corresponding to the aforementioned area, and use texture mapping to paste the first key frame image and the aforementioned area to On the three-dimensional projection model, a three-dimensional panorama is obtained after splicing. Further, the three-dimensional panoramic image is converted into a two-dimensional second key frame image according to the internal and external parameters of the camera of the panoramic surround view system. The second key frame image is the same as the first key frame image, and both are images from a bird's eye view.

Step S150: Convert the second key frame image into a local semantic map based on the initial pose of the target vehicle.

For example, the sensor installed on the target vehicle is used to obtain the initial pose of the target vehicle when it starts to start in the target environment. The initial pose includes initial position data and initial attitude data. The initial position data can be absolute position data (for example, longitude and latitude directly obtained by GPS (Global Positioning System)), or relative position data (for example, obtained by Accumulated motion parameters obtained from the number of rotations and speed of the target vehicle's wheels, such as distance, etc.). The initial attitude data may be an absolute orientation angle obtained through differential GPS, or a relative orientation angle (for example, accumulated motion parameters, such as direction, obtained from the rotation angle of the target vehicle's wheels, etc.).

For example, the local posture of the target vehicle can be determined based on the initial posture when the target vehicle is started and parameters such as the accumulated movement distance and direction obtained by the IMU (Inertial Measurement Unit). Since this pose is obtained only by motion changes, it is a relatively local pose.

Further, detection, tracking and recognition are performed on the second key frame image, and the semantic entity (equal to the target object) in the target environment is determined according to the result of the detection, tracking and recognition. According to the initial pose of the target vehicle, the semantic entities in the target environment and the local pose of the target vehicle, the second key frame image is converted into a local semantic map.

Step S160: Splice local semantic maps of the target vehicle at multiple times in the target environment to generate a global map of the target environment.

Match the semantic entities in the local semantic maps at different times to achieve the splicing of the same semantic entities at the same position, and then obtain the global map of the target environment.

In this embodiment, through motion compensation, the historical frame image is mapped to the blind area of the car bottom of the first key frame image according to specific transformation conditions, eliminating the occlusion and truncation effects caused by the visual blind area, and ensuring that the second key frame The continuity and accuracy of the semantic features of the image avoid feature truncation or loss during the mapping process, provide assistance to users in driving, and improve the user experience. Moreover, this application does not require the installation of additional camera modules and is low cost.

Based on the embodiment shown in FIG. 1 , features in the global map can also be matched based on the initial pose of the target vehicle to determine the current pose of the target vehicle.

Specifically, in this embodiment, the initial pose of the target vehicle refers to the pose of the target vehicle at the current moment collected by the odometer while the target vehicle is traveling.

Further, based on the global map, a feature true value database is obtained. While the target vehicle is driving, the semantic features of the target object contained in the image are determined based on the multi-channel surround images collected by the target vehicle. According to the semantic features of the target object, the location feature position is found in the feature truth value database. For example, the semantic features of the target object are converted to the global coordinate system corresponding to the global map, and then the semantic features of the target object are combined with those in the global map. Feature true values perform feature type matching. After that, the positioning feature position is solved for pose to obtain the solved pose. For example, by positioning the static feature point information in the positioning feature position, using triangulation positioning to calculate the current global position of the static feature point information and the relative distance to the lidar. position, thereby solving the current global position of the lidar, and obtaining the current position of the vehicle by locating the feature position, and then obtaining the solved pose.

In addition, the global position of the target vehicle collected by the satellite positioning system, the calculated pose of the target vehicle, and the initial pose of the target vehicle are fused, and the fusion result is used as the current pose of the target vehicle.

In this embodiment, the characteristics of the target environment are obtained through multi-channel surround images, and then the characteristics are registered in the global map. Because the global information is known, the search speed is fast. After obtaining the static global position, the global position information of the target vehicle is deduced based on the geometric relationship between the target vehicle and the static global position, thereby improving the positioning accuracy.

Figure 2 shows a schematic flowchart of obtaining a second key frame image provided by an exemplary embodiment of the present application. The embodiment shown in FIG. 2 is extended based on the embodiment shown in FIG. 1. The following focuses on the differences between the embodiment shown in FIG. 2 and the embodiment shown in FIG. 1, and the similarities will not be described again.

As shown in Figure 2, in the embodiment of the present application, based on multiple historical frame images, texture mapping is performed on the blind area under the vehicle in the first key frame image to obtain a second key frame image without the blind area under the vehicle, including the following steps .

Step S210: Determine a reference frame image for mapping the blind area under the vehicle in the first key frame image among multiple historical frame images.

In one implementation, for the blind area under the vehicle of the first key frame image, multiple historical frame images are sorted in chronological order, and each can provide effective mapping areas corresponding to part or all of the blind area under the vehicle. Determine the historical frame image that provides the largest effective mapping area and use it as a reference frame image to avoid the problem of poor display effects caused by combining mapping of too many historical frame images to the blind area under the car.

If multiple historical frame images cannot provide all the effective mapping areas, you can also select the historical frame image with the most pixels corresponding to the vehicle bottom blind area of the first key frame image from the multiple historical frame images as the reference frame image. , to obtain the effective map area. This method of selecting a larger area of historical frame images to fill the blind spots under the car can reduce the splicing times and the number of splicing in the blind spots under the car as much as possible, effectively avoid the problems of image dislocation and brightness inconsistency caused by multi-image splicing, and improve The display effect of the blind area under the car is improved.

In another implementation, the transformation matrices of multiple historical frame images and the first key frame image can also be determined based on the driving parameters of the target vehicle; based on the target sampling error, the feature information in the first key frame image is Standard, randomly sample the corresponding feature information of multiple historical frame images to determine the corresponding sampling values of multiple historical frame images; based on the transformation matrix of multiple historical frame images and the first key frame image, multiple historical frame images The respective corresponding sampling values determine the reference frame image from multiple historical frame images.

Driving parameters include steering wheel angle information and vehicle speed information, and the sampling values represent the matching degree between the feature information in the historical frame image and the feature information in the first key frame image. Further, for each historical frame image in the plurality of historical frame images, the computing resources (expressed in numerical values) consumed by the historical frame image using its corresponding transformation matrix under the sampling value are determined. The weights corresponding to the computing resources and sampling values are determined, and based on the weights, the score of the historical frame image is obtained, and the historical frame image with the highest score is determined as the reference frame image. The higher the score, the higher the feature similarity between the corresponding historical frame image and the first key frame image. When using it for texture mapping later, less computing resources are consumed, ensuring that the reference frame image is consistent with the first key frame image. It not only improves the feature similarity of frame images, but also reduces the resources consumed in subsequent texture mapping.

Step S220: Determine the wheel speed pulse count value of the target vehicle at the current time and the wheel speed pulse count value of the target vehicle at the historical time corresponding to the reference frame image.

The wheel pulse count value refers to the number of pulses for one revolution of the wheel. For example, one revolution of the wheel can obtain 1080 pulses.

Step S230: Determine the rotation angle and offset of the target vehicle in the first key frame image and the reference frame image based on the wheel speed pulse count value of the target vehicle at the current time and the wheel speed pulse count value at the historical time.

For example, the inertial navigation principle is used to calculate the relative offset and rotation angle of the target vehicle on the reference frame image and the second key frame image. Inertial navigation, or inertial navigation system, is an autonomous navigation system that does not rely on external information and does not radiate energy to the outside. The basic working principle of inertial navigation is based on Newton's laws of mechanics. By measuring the acceleration of the carrier in the inertial reference system, integrating it over time, and transforming it into the navigation coordinate system, we can obtain the acceleration in the navigation coordinate system. Information such as speed, yaw angle and position.

Step S240: Determine the effective mapping area in the reference frame image based on the rotation angle and offset.

The reference frame image is overlaid on the first key frame image, and the target vehicle in the reference frame image is rotated by the above rotation angle relative to the target vehicle in the first key frame image, and is offset by the above offset amount, so that Obtain the effective mapping area in the reference frame image of the vehicle bottom blind area mapped on the first key frame image. For example, image B is the reference frame image, and image A is the first key frame image. According to the relative offset and rotation angle, with the target vehicle in image B as the base point, image B is offset and rotated, so that image B is mapped to image A, and the effective mapping area is obtained.

Step S250: Based on the effective mapping area, perform texture mapping on the blind area under the vehicle in the first key frame image to obtain a second key frame image.

In one implementation, the contour line of the vehicle bottom blind area of the first key frame image is determined; along the contour line, the target distance is expanded in a direction away from the vehicle bottom blind area to obtain the outer edge line, and the area composed of the contour line and the outer edge line is As a transition area; based on the effective mapping area, texture map the blind area under the car in the first key frame image to obtain the image to be calibrated; determine the gray gradient value of the effective mapping area in the image to be calibrated; based on the gray gradient value, Calibrate the gray value of the transition area in the image to be calibrated to obtain the second key frame image.

It can be understood that the shape of the transition area is not limited here, as long as the inner frame line of the transition area can overlap with the outline line of the blind area under the vehicle.

By setting the transition area and adjusting the gray gradient of the pixels in the transition area, the visual effect of the fusion boundary between the effective map area and other areas in the second key frame image can be eliminated.

In this embodiment, the reference frame image is determined from multiple historical frame images, and an image that is similar to the first key frame image to a certain extent can be selected to ensure that the characteristics of the subsequent texture mapping of the blind area under the car are consistent. sex. According to the rotation angle and offset, the effective mapping area in the reference frame image is determined, which further ensures the continuity of semantic features in the texture mapping process and improves the visual effect of the second key frame image.

Figure 3 shows a schematic flowchart of conversion into a local semantic map provided by an exemplary embodiment of the present application. The embodiment shown in FIG. 3 is extended based on the embodiment shown in FIG. 1. The following focuses on the differences between the embodiment shown in FIG. 3 and the embodiment shown in FIG. 1, and the similarities will not be described again.

As shown in Figure 3, in this embodiment of the present application, converting the second key frame image into a local semantic map based on the initial pose of the target vehicle includes the following steps.

Step S310: Perform semantic feature extraction on the second key frame image to obtain the semantic features of the target object contained in the second key frame image.

Semantic features include position features, direction features and category features. For example, deep neural networks are used to extract semantic features, including parking space lines, parking space corners, speed bumps, lane lines, ground markings, buildings, wheel blocks and other semantic entities (i.e. target objects). Further, the attribute information of the semantic entity is determined. For example, the attribute information may indicate the physical characteristics of the semantic entity, or the semantic entity may affect the attributes of the target vehicle's own movement. For example, the attribute information can be spatial attribute information such as position, shape, size, orientation, etc. of each semantic entity, or it can be category attribute information of each semantic entity (such as whether each semantic entity is a feasible road, curb, lane, etc. Which of lane markings, traffic signs, pavement markings, traffic lights, stop lines, crosswalks, roadside trees or pillars, etc.). The relative position relationship between the semantic entity and the target vehicle is determined based on the second key frame image, and the spatial attribute information of the semantic entity is determined based on the local pose information and the relative position relationship. For example, the spatial attributes may include various attributes related to spatial characteristics such as the size, shape, orientation, height, and occupation of the semantic mark. In addition to the spatial attribute information, for example, the category of each semantic entity can also be further determined based on the second key frame image.

Step S320: Determine the position and orientation of the target vehicle at the current moment.

In this embodiment, the pose of the target vehicle at the current moment is equivalent to the local pose information in the embodiment shown in FIG. 1 .

Step S330: Convert the second key frame image into a local semantic map based on the semantic features of the target object contained in the second key frame image, the pose and initial pose of the target vehicle at the current moment.

After determining each semantic entity included in the second key frame image and its attribute information, the information can be synthesized to construct a local semantic map based on the second key frame image. That is, the semantic mark result of the second key frame image is reconstructed and attributes such as position and size are added to obtain a semantic mark map with absolute attributes.

In this embodiment, the second key frame image without the vehicle bottom blind spot is used to obtain the local semantic map, which ensures the integrity, continuity and accuracy of the semantic features in the local semantic map, making the subsequent mapping positioning more accurate. .

Figure 4 shows a schematic flowchart of generating a global map provided by an exemplary embodiment of the present application. The embodiment shown in FIG. 4 is extended based on the embodiment shown in FIG. 1. The following focuses on the differences between the embodiment shown in FIG. 4 and the embodiment shown in FIG. 1, and the similarities will not be described again.

As shown in Figure 4, in this embodiment of the present application, the local semantic maps of the target vehicle at multiple moments in the target environment are spliced to generate a global map of the target environment, which includes the following steps.

Step S410: Determine at least two local semantic maps to be matched among the local semantic maps at multiple times.

At least two local semantic maps to be matched are sequential in time sequence.

Step S420: Determine the target point cloud and the source point cloud in at least two local semantic maps to be matched.

If there are two local semantic maps to be matched, the target point cloud and the source point cloud are the point clouds in the two local semantic maps respectively. For example, the local semantic map containing the source point cloud can be used as the benchmark, and the target point cloud can be used as the reference point cloud. The semantic features of the local semantic maps of the point cloud are spliced. If there are more than two local semantic maps, you can select one of the local semantic maps, use the point cloud in it as the source point cloud, and splice the remaining local semantic maps to be matched with it.

Step S430: Based on the geometric characteristics of the target point cloud, determine the closest point from each point in the target point cloud to the source point cloud.

Step S440: Based on the nearest point from each point in the target point cloud to the source point cloud, the local semantic maps of the target vehicle at multiple moments in the target environment are spliced to generate a global map of the target environment.

The source point cloud is recorded as P and the target point cloud is recorded as Q. For each point cloud in Q, the corresponding nearest point cloud in P is found to form a matching point pair. The sum of the Euclidean distances of all matching point pairs is used as the objective function to be solved, and the rotation matrix R and the translation matrix t are obtained using singular value decomposition to minimize the objective function. Transform Q (including rotation and translation) according to R and t to obtain a new Q ^′ , and find the corresponding point pair again, and so on until the error is minimal. Using the rotation matrix R and translation matrix t obtained when the error is minimized, the local semantic maps are spliced to generate a global map of the target environment.

In this embodiment, by obtaining the matching point pairs between the source point cloud and the target point cloud, a rotation matrix and a translation matrix are constructed based on the matching point pairs, and the source point cloud is transformed using the obtained rotation matrix and translation matrix. to the coordinate system of the target point cloud. Estimate the error function of the transformed source point cloud and target point cloud. If the error function value is greater than the threshold, continue iteration until the error value after splicing based on the rotation matrix and translation matrix meets the given error requirement. The method in this embodiment is simple and has good accuracy.

The above describes in detail the embodiment of the mapping method based on visual semantics of the present application with reference to Figures 1 to 4. The following describes the embodiment of the mapping device based on visual semantics of the present application in detail with reference to Figure 5. It should be understood that the description of the embodiments of the mapping method based on visual semantics corresponds to the description of the embodiments of the mapping device based on visual semantics. Therefore, the parts not described in detail can be referred to the previous method embodiments.

Figure 5 shows a schematic structural diagram of a mapping device based on visual semantics provided by an exemplary embodiment of the present application. As shown in Figure 5, the visual semantics-based mapping device 50 provided by the embodiment of the present application includes:

The acquisition module 510 is used to acquire the multi-channel surround image of the target vehicle at the current moment collected in the target environment;

The first splicing module 520 is used to perform bird's-eye splicing of multi-channel surround-view images to generate a first key frame image, where the first key frame image includes the blind spot under the vehicle;

The determination module 530 is used to determine multiple historical frame images without vehicle bottom blind spots corresponding to the first key frame image;

The mapping module 540 is used to perform texture mapping on the vehicle bottom blind area in the first key frame image based on multiple historical frame images, and obtain a second key frame image without the vehicle bottom blind area;

The conversion module 550 is used to convert the second key frame image into a local semantic map based on the initial pose of the target vehicle;

The second splicing module 560 is used to splice local semantic maps of the target vehicle at multiple moments in the target environment to generate a global map of the target environment.

In an embodiment of the present application, the mapping module 540 is also used to determine a reference frame image for mapping the blind spot under the vehicle in the first key frame image among multiple historical frame images; and determine the wheel speed of the target vehicle at the current moment. The pulse count value and the wheel speed pulse count value of the target vehicle at the historical moment corresponding to the reference frame image; based on the wheel speed pulse count value of the target vehicle at the current moment and the wheel speed pulse count value at the historical moment, it is determined that the target vehicle is in the first The rotation angle and offset in the key frame image and the reference frame image; based on the rotation angle and offset, determine the effective mapping area in the reference frame image; based on the effective mapping area, determine the blind area under the car in the first key frame image Perform texture mapping to obtain the second keyframe image.

In an embodiment of the present application, the mapping module 540 is also used to determine the transformation matrices of multiple historical frame images and the first key frame image based on the driving parameters of the target vehicle; based on the target sampling error, use the first key frame image to The feature information in the image is used as the standard. The feature information corresponding to multiple historical frame images is randomly sampled to determine the corresponding sampling values of the multiple historical frame images. The sampling values represent the feature information in the historical frame image and the first key frame image. The matching degree of the feature information in the image; based on the transformation matrix of the multiple historical frame images and the first key frame image, and the corresponding sampling values of the multiple historical frame images, the reference frame image is determined from the multiple historical frame images.

In one embodiment of the present application, the mapping module 540 is also used to determine the contour line of the blind area under the vehicle; along the contour line, expand the target distance away from the blind area under the vehicle to obtain the outer edge line, and combine the contour line and the outer edge line area as the transition area; based on the effective mapping area, texture map the blind area under the car in the first key frame image to obtain the image to be calibrated; determine the gray gradient value of the effective mapping area in the image to be calibrated; based on the gray gradient value, calibrate the gray value of the transition area in the image to be calibrated, and obtain the second key frame image.

In an embodiment of the present application, the conversion module 550 is also used to extract semantic features from the second key frame image to obtain the semantic features of the target object contained in the second key frame image. The semantic features include position features, direction features and Category features; determine the pose of the target vehicle at the current moment; convert the second key frame image into a local Semantic map.

In an embodiment of the present application, the second splicing module 560 is also used to determine at least two local semantic maps to be matched among the local semantic maps at multiple moments; Target point cloud and source point cloud; based on the geometric characteristics of the target point cloud, determine the closest point from each point in the target point cloud to the source point cloud; based on the closest point from each point in the target point cloud to the source point cloud Point, the local semantic maps of the target vehicle at multiple moments in the target environment are spliced to generate a global map of the target environment.

In an embodiment of the present application, a positioning module is also included. The positioning module is used to match features in the global map based on the initial pose of the target vehicle and determine the current pose of the target vehicle.

Next, an electronic device according to an embodiment of the present application is described with reference to FIG. 6 . Figure 6 shows a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application.

As shown in FIG. 6 , electronic device 60 includes one or more processors 601 and memory 602 .

The processor 601 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 60 to perform desired functions.

Memory 602 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and/or cache memory (cache). The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 601 may execute the program instructions to implement the methods of various embodiments of the application described above and/or other desired Function. Various contents such as multi-channel surround images, first key frame images, second key frame images, multiple historical frame images, local semantic maps, global maps, etc. can also be stored in the computer-readable storage medium.

In one example, the electronic device 60 may also include an input device 603 and an output device 604, and these components are interconnected through a bus system and/or other forms of connection mechanisms (not shown).

The input device 603 may include, for example, a keyboard, a mouse, and the like.

The output device 604 can output various information to the outside, including multi-channel surround images, first key frame images, second key frame images, multiple historical frame images, local semantic maps, global maps, etc. The output device 604 may include, for example, a display, a speaker, a printer, a communication network and its connected remote output devices, and the like.

Of course, for simplicity, only some of the components in the electronic device 60 related to the present application are shown in FIG. 6 , and components such as buses, input/output interfaces, etc. are omitted. In addition, the electronic device 60 may also include any other suitable components depending on the specific application.

In addition to the above-mentioned methods and devices, embodiments of the present application may also be computer program products, which include computer program instructions. When the computer program instructions are run by a processor, the computer program instructions cause the processor to execute the above-described method of the present application according to the present specification. Steps in the methods of various embodiments.

The computer program product can be used to write program codes for performing the operations of the embodiments of the present application in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc. , also includes conventional procedural programming languages, such as the "C" language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on.

In addition, embodiments of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are run by a processor, the computer program instructions cause the processor to perform the steps described above in this specification according to the present application. Steps in the method of an embodiment.

The computer-readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: electrical connection with one or more conductors, portable disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, advantages, effects, etc. mentioned in this application are only examples and not limitations. These advantages, advantages, effects, etc. cannot be considered to be Each embodiment of this application must have. In addition, the specific details disclosed above are only for the purpose of illustration and to facilitate understanding, and are not limiting. The above details do not limit the application to be implemented using the above specific details.

The block diagrams of the devices, devices, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, devices, equipment, and systems may be connected, arranged, and configured in any manner. Words such as "includes," "includes," "having," etc. are open-ended terms that mean "including, but not limited to," and may be used interchangeably therewith. As used herein, the words "or" and "and" refer to the words "and/or" and are used interchangeably therewith unless the context clearly dictates otherwise. As used herein, the word "such as" refers to the phrase "such as, but not limited to," and may be used interchangeably therewith.

It should also be pointed out that in the device, equipment and method of the present application, each component or each step can be decomposed and/or recombined. These decompositions and/or recombinations shall be considered equivalent versions of this application.

The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, this application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for the purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the form disclosed herein. Although various example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims (10)

1. A visual semantic based mapping method, comprising:

acquiring a plurality of paths of looking-around images of a target vehicle at the current moment acquired in a target environment;

Performing aerial view stitching on the multipath surrounding images to generate a first key frame image, wherein the first key frame image comprises a vehicle bottom blind area;

determining a plurality of historical frame images without a vehicle bottom blind area corresponding to the first key frame image;

based on the plurality of historical frame images, performing texture mapping on the blind areas of the vehicle bottom in the first key frame image to obtain a second key frame image without the blind areas of the vehicle bottom;

converting the second key frame image into a local semantic map based on the initial pose of the target vehicle;

and splicing the local semantic maps of the target vehicle at a plurality of moments in the target environment to generate a global map of the target environment.

2. The method of claim 1, wherein the performing texture mapping on the blind areas of the vehicle bottom in the first key frame image based on the plurality of historical frame images to obtain a second key frame image without blind areas of the vehicle bottom comprises:

determining a reference frame image for mapping a blind area of the vehicle bottom in the first key frame image in the plurality of historical frame images;

determining a wheel speed pulse count value of the target vehicle at the current moment and a wheel speed pulse count value of the target vehicle at a historical moment corresponding to the reference frame image;

Determining a rotation angle and an offset of the target vehicle in the first key frame image and the reference frame image based on a wheel speed pulse count value of the target vehicle at the current moment and a wheel speed pulse count value of the historical moment;

determining an effective map area in the reference frame image based on the rotation angle and the offset;

and carrying out texture mapping on the blind areas of the vehicle bottom in the first key frame image based on the effective mapping area to obtain the second key frame image.

3. The method of claim 2, wherein the determining a reference frame image from the plurality of historical frame images for mapping the under-floor dead zone in the first key frame image comprises:

based on the running parameters of the target vehicle, respectively determining transformation matrixes of the plurality of historical frame images and the first key frame image;

based on a target sampling error, taking characteristic information in the first key frame image as a standard, randomly sampling the characteristic information corresponding to each of the plurality of historical frame images, and determining sampling values corresponding to each of the plurality of historical frame images, wherein the sampling values represent the matching degree of the characteristic information in the historical frame image and the characteristic information in the first key frame image;

And determining the reference frame image from the plurality of historical frame images based on the transformation matrix of the plurality of historical frame images and the first key frame image and sampling values corresponding to the plurality of historical frame images respectively.

4. The method of claim 2, wherein the texture mapping the bottom dead zone in the first key frame image based on the valid mapping region to obtain the second key frame image comprises:

determining the contour line of the blind area of the vehicle bottom;

along the contour line, expanding the target distance outwards in a direction away from the blind area of the vehicle bottom to obtain an outer edge line, and taking a region formed by the contour line and the outer edge line as a transition region;

based on the effective mapping area, performing texture mapping on the vehicle bottom blind area in the first key frame image to obtain an image to be calibrated;

determining a gray gradient value of an effective mapping area in the image to be calibrated;

and calibrating the gray level value of the transition region in the image to be calibrated based on the gray level gradient value to obtain the second key frame image.

5. The method of any one of claims 1 to 4, wherein the converting the second keyframe image into a local semantic map based on the initial pose of the target vehicle comprises:

Extracting semantic features of the second key frame image to obtain semantic features of a target object contained in the second key frame image, wherein the semantic features comprise position features, direction features and category features;

determining the pose of the target vehicle at the current moment;

and converting the second key frame image into the local semantic map based on semantic features of a target object contained in the second key frame image, and the pose and initial pose of the target vehicle at the current moment.

6. The method of any one of claims 1 to 4, wherein the stitching together the local semantic maps of the target vehicle at a plurality of moments in the target environment to generate a global map of the target environment comprises:

determining at least two local semantic maps to be matched in the local semantic maps at a plurality of moments;

determining a target point cloud and a source point cloud in the at least two local semantic maps to be matched respectively;

determining, based on geometric features of the target point cloud, a nearest neighbor point of each point in the target point cloud to the source point cloud;

and based on each point in the target point cloud to the nearest point of the source point cloud, splicing local semantic maps of the target vehicle at a plurality of moments in the target environment, and generating a global map of the target environment.

7. The method according to any one of claims 1 to 4, further comprising:

and matching the features in the global map based on the initial pose of the target vehicle, and determining the current pose of the target vehicle.

8. A visual semantic based mapping device, comprising:

the acquisition module is used for acquiring a plurality of paths of looking-around images of the target vehicle at the current moment acquired in the target environment;

the first splicing module is used for performing aerial view splicing on the multipath surrounding images to generate a first key frame image, wherein the first key frame image comprises a vehicle bottom blind area;

the determining module is used for determining a plurality of historical frame images without a vehicle bottom blind area corresponding to the first key frame image;

the mapping module is used for performing texture mapping on the blind areas of the vehicle bottom in the first key frame image based on the historical frame images to obtain a second key frame image without the blind areas of the vehicle bottom;

the conversion module is used for converting the second key frame image into a local semantic map based on the initial pose of the target vehicle;

and the second splicing module is used for splicing the local semantic maps of the target vehicle at a plurality of moments in the target environment to generate a global map of the target environment.

9. A computer readable storage medium, characterized in that the storage medium stores a computer program for executing the method of any of the preceding claims 1 to 7.

10. An electronic device, comprising:

a processor;

a memory for storing the processor-executable instructions;

the processor being adapted to perform the method of any of the preceding claims 1 to 7.