CN110807782A - A map representation system for visual robot and its construction method - Google Patents

Abstract

The invention belongs to the field of mobile robot environment representation, planning and positioning, and more particularly relates to a map representation system of a visual robot and a construction method thereof. The map representation system is composed of data structures of multi-information voxel layer, map element layer and topology layer, which respectively cover three aspects of spatial information, scene instance and connectivity. The map is constructed by six modules: semantic information extraction module, geometric information extraction module, scene semantic extraction module, multi-information voxel integration module, extraction spatial topology module and topology integration module. The invention is only based on the visual sensor, has the advantages of clear construction process, comprehensive information, close hierarchical relationship and easy visualization, and is suitable for the planning, positioning and navigation of indoor and outdoor scenes of mobile robots.

Description

A map representation system for visual robot and its construction method

technical field

The invention belongs to the field of mobile robot environment representation, planning and positioning, and more particularly, relates to a map representation system of a visual robot and a construction method thereof.

Background technique

In the field of robotics, how to design a map representing the environment is an important and critical problem. For the planning module, the map needs to provide support for fast collision detection and obstacle detection; for the positioning module, the map needs to provide high-quality environment modeling; in addition, for the human manipulating the robot, the map needs to provide friendly and Intuitive visual representation. Today, vision sensors are widely used in the field of robotics, such as RGB-D cameras, binocular stereo cameras, etc., or ranging sensors, such as lidar, for map construction. Generally, vision sensors are low-cost, easy to install, and can obtain high-frequency data; while ranging sensors are inherently capable of high-precision depth measurement, they are expensive.

At present, during map construction, different environments (such as ruins, shopping malls, open spaces, etc.) will affect the performance of the sensor, so that the obtained data contains noise, which affects the quality of the map, and further affects the results of planning and positioning. In recent studies, researchers have additionally introduced environmental semantic information into the map to effectively reduce the impact of the environment on the algorithm; in addition, the incremental update and patch optimization of the map is also known as the current focus of research. But at present, there is no map representation method for robots, which closely combines topological, geometric, and semantic information, and supports properties such as rapid construction, incremental update, efficient data storage and reading, and intuitive visualization.

SUMMARY OF THE INVENTION

In order to overcome the above-mentioned defects in the prior art, the present invention provides a map representation system of a visual robot and a construction method thereof, which closely integrates three types of information of topology, geometry and semantics, and supports rapid construction, incremental update, efficient data storage and Read, provide support for fast collision detection and obstacle detection, provide high-quality environment modeling.

In order to solve the above technical problems, the technical solution adopted in the present invention is: a map representation system for visual robots, including a semantic information extraction module, a geometric information extraction module, a scene semantic extraction module, a multi-information voxel integration module, and an extraction space topology module. , a topology integration module, and a map element layer, a multi-information voxel layer, and a topology layer; wherein,

The semantic information extraction module is used to collect images in an environment where a map needs to be constructed by using a visual sensor, and extract semantic information of images to obtain image segmentation results; then perform other specific semantic extraction to obtain specific semantic information, specific semantic information The result of the information part will eventually be stored in the map element layer;

The geometric information extraction module is used to collect images in an environment where a map needs to be constructed by using a visual sensor, obtain a depth map, a set of vertices and a normal vector corresponding to the vertices through calculation, and then calculate several distances and normals based on the depth map. Geometric feature segmentation depth map;

The scene semantic extraction module is used to collect images in an environment where a map needs to be constructed by using a visual sensor, and perform scene semantic extraction to obtain scene classification information, and the scene classification information is an artificially defined scene with a certain degree of identification. title;

The multi-information voxel integration module is used to fuse the image segmentation results obtained by the semantic information extraction module and the depth segmentation results obtained by the geometric information extraction module to obtain a three-dimensional semantic component, which describes a certain scene in the scene by a human. Predetermined objects with a certain degree of identity and significance for planning and positioning; then, combined with the camera pose obtained by the visual SLAM method, the multi-information voxels calculated by using the 3D semantic components and the vertex set obtained by the geometric information extraction module , updated to the multi-information voxel layer of the map;

The described extraction space topology module is used for the multi-information voxel layer obtained based on the described multi-information voxel integration module, and extracts the spatial convex hull comprising scene semantics and the topology information expressing its connection relationship;

The topology integration module is used to associate the three-dimensional space components in the multi-information voxel integration module and the three-dimensional space convex hull obtained by the extraction space topology module based on the adjacent relationship in space, and calculate the relationship between the space nodes and the component nodes. topology information, and combine this information with the topology information from space nodes to space nodes to obtain a complete topology layer;

The map element layer, the multi-information voxel layer and the topology layer together constitute a map, and the topology map connects the spatial convex hull in the map element layer with the three-dimensional component information.

Further, the multi-information voxel layer uses a cube with attached spatial information and semantic information, which does not actually exist in the objective world, but is a data abstract object used to approximate a cubic area of the objective world; The multi-information voxel layer is based on the requirements of robot planning and positioning, and a variety of predefined environments represent objects, which are generally composed of fields describing their position, semantics, geometry, etc. Its center position and category are obstacles and the convex hull surrounding it.

The present invention also provides a map construction method for a visual robot, comprising the following steps:

S1. Use the semantic information extraction module, use the visual sensor to collect images in the environment where the map needs to be constructed, and extract the semantic information of the components to obtain the image segmentation results (including but not limited to doors and windows, tables and chairs), and extract other specific semantics to obtain: Specific semantic information (including but not limited to exercisable areas); the results of specific semantic information parts will eventually be stored in the map element layer;

S2. Use the geometric information extraction module, use the visual sensor to collect images in the environment where the map needs to be constructed, obtain the depth map, the vertex set and the normal corresponding to the vertex, and then calculate the depth segmentation result from the depth map;

S3. Using the scene semantic extraction module, the visual sensor is used to collect images in the environment where the map needs to be constructed, and the scene semantic extraction is performed to obtain the classification information of the scene;

S4. Use the multi-information voxel integration module to fuse the image segmentation results extracted in step S1 and the depth segmentation results extracted in step S2 to obtain three-dimensional semantic components; then, based on the visual sensor posture obtained by the visual SLAM method, the three-dimensional The semantic component and the vertex set extracted in step S2 are calculated to obtain multi-information voxels to form a multi-information voxel layer;

S5. use the extraction space topology module, based on the multi-information voxel layer in step S4, extract the space convex hull comprising scene semantics and the topology information expressing its connection relationship;

S6. Using the topology integration module, based on the adjacent relationship of the space, associate the three-dimensional space component in step S4 with the three-dimensional space convex hull in step S5, obtain the topology information from the space node to the component node, and associate this information with the space node to the space The topology information of nodes is merged to obtain a complete topology map, which constitutes a topology layer;

S7. The map of the present invention is formed together by the map element layer in step S1, the multi-information voxel layer in step S4, and the topology layer in step S6.

Further, the step S1 specifically includes:

S11. Obtain an RGB image from a vision sensor;

S12. Infer each pixel in the RGB image based on step S11, and determine the category of its subordinate objects (including but not limited to doors and windows, desks and chairs), the process is completed based on a deep neural network, and finally the A collection of binary mask images;

S13. The rest of the specific semantic extraction is determined by the actual application. Based on step S11, the RGB image is extracted by using the feature detection and inference module, and then the specific semantic extraction is performed to obtain the specific semantic. The specific semantic includes but is not limited to: exercise area.

Further, the step S2 specifically includes:

S21. Obtain a depth map from a vision sensor;

S22. From the depth map in step S21, calculate the surface normal corresponding to this pixel based on a pixel and its adjacent points, then calculate the normal angle difference between adjacent pixels, and finally calculate the depth difference between adjacent pixels. A total of three features are obtained ;

S23. Based on the three kinds of features obtained in the step S22, clustering and cutting are carried out on the depth map, and a plurality of different depth segmentation regions are obtained by division, which are called as depth segmentation results.

Further, the step S3 specifically includes:

S31. Obtain an RGB image from a vision sensor;

S32. Calculate the high-dimensional features from the pixels of the RGB image and the relationship between the pixels, and then infer the classification information of the scene; the process is implemented based on deep neural networks, including but not limited to CNN networks, and scenes include but not limited to Shopping malls, kitchens, warehouses, corridors.

Further, the step S4 specifically includes:

S41. Denote the depth segmentation result extracted in the step S2 as a set S, where s _i ∈ S is a geometric component segmentation block, consisting of a certain number of pixels and their depth information, denote the image segmentation extracted in the step S1 The result is a set R, where r _j ∈ R is an area on an image, which consists of a certain number of pixels and their classification information; for each s _i , calculate the r _j with the highest overlapping area, and the calculation formula is as follows:

S42. For each s _i in the step S41, assign the category of the highest r _j to the overlapping area, determine the best category of each geometric component segment, and assign adjacent objects with common instance objects and categories to Geometric segmentation and fusion into 3D semantic components;

S43. Compare the three-dimensional semantic components and the three-dimensional semantic components existing in the map one by one. If the current moment and the map have the same instance, that is, an object in the same objective environment, perform instance tracking. Add a new three-dimensional semantic component to the map, and then perform instance tracking; the instance tracking is a method to ensure that the detection result of each instance object is kept temporally incoherent in multiple frames of images when the map is constructed;

S44. fuse the three-dimensional semantic component maintained in the step S43 with the vertex set extracted in the step S2; first divide the space into voxels, and only consider the voxels within the truncation distance t; suppose x is the current voxel The center of , p is the three-dimensional position of a vertex, s is the origin of the sensor, at this time there are:

d(x,p,s)＝||p-x||sign((p-x)·(p-s))

∈=4v

W _i+1 (x,p,s)=min(W _i (x)+w(x,p),W _max )

In the formula, v is the size of the voxel, z is the depth from s to p, and W _max limits the maximum weight of the update; from the above formula, the TSDF value of the i+1th update voxel x can be calculated. D _i+1 (x, p, s) and weight W _i+1 (x, p, s); initial TSDF value D (x, p, s) and weight W (x, p, s) are both is initialized to 0;

S45. Calculate the value of ESDF based on the TSDF value in step S44; the process specifically includes: Starting from the TSDF surface, by means of 26-neighbor search, the area in the set truncation area is directly set as the value of the TSDF. The value of ESDF, the value outside the truncation area is calculated by the water ripple propagation algorithm, and finally the ESDF is obtained;

Among them, the described water ripple algorithm is carried out according to the following steps: S451. The wave starts from a voxel denoted as v, propagates to its 26-neighborhood, and updates the voxels whose ESDF distances have not been updated with their ESDF distances as voxels The ESDF value of v is added to the unit distance, and the newly updated voxel is put into the ripple diffusion queue; for each voxel in the ripple diffusion queue, step S451 is performed recursively until all voxels have updated the ESDF distance. ;

S46. Based on the three-dimensional semantic component information obtained in steps S42 and S43 and the ESDF in step S45, form a multi-information voxel;

S47. Obtain the current sensor pose through the visual SLAM method, convert the multi-information voxels in the sensor coordinate system into the representation in the map coordinate system, and then update them to the information voxel layer of the map; The method stores multiple multi-information voxels, and each voxel contains three-dimensional position information, semantic encoding information, whether it is occupied or not, and a mixed intercept/Euclidean distance field composition.

Further, the step S5 specifically includes:

S51. Randomly collect convex hull growing points on the data collection route;

S52. On each convex hull generation point of the step S51, add height and volume restrictions, and continuously expand outward to form a spatial convex hull; and based on the coordinate position and the scene semantic information given in step S3, obtain a space with semantics convex hull set;

S53. based on the spatial convex hull set with semantics in the described step S52, utilize their mutual space adjacent relationship to obtain the undirected graph representing the spatial connection relationship;

S54. Set a certain threshold that allows the non-convex part of the obstacle to be accommodated, and merge the convex hull, so that the obtained convex hull is more in line with the original spatial shape of the environment based on human intuition, and a larger elliptical-like shape with semantic information is obtained. The spatial convex hull set of ; the process does not merge the convex hull when the semantics conflict.

Further, the step S6 specifically includes:

S61. For each three-dimensional semantic component obtained in step S4, find an ellipse-like spatial convex hull with semantic information obtained in step S54 to which it belongs according to its own position (such as a three-dimensional semantic component of a cup that matches the room. On a space convex hull), get the connection relationship between space nodes and component nodes;

S62. Integrate all the connection relationships between the space nodes obtained in step S61 to the component nodes on the topology map obtained in step S5 to obtain a complete topology map.

Compared with the prior art, the beneficial effects are:

1. the present invention only uses the visual sensor to complete the construction of the robot map, and does not need to use the data of the ranging scanner to construct, which reduces the hardware cost;

2. The structure of this map contains geometric, semantic and topological information, and contains more environmental information; and uses the unique correlation properties of each level to construct the map, the mapping process is clear, and the efficiency of mapping is improved. The relationship between the layers is closely combined, and the data and topology are separated, which is easy to carry out the subsequent incremental update and repair optimization process;

3. The present invention stores and manages the geometric data of the environment in the form of a hash table, reduces the overhead of data search and access, and supports the dynamic and irregular shape expansion of the map, and improves the insertion, update and speed of data. The structure of the grid enables the constructed map to maximize the use of storage space, which is conducive to the maximum coverage of the environment;

4. The robot map in the present invention uses a multi-information voxel structure that includes a combination of geometric and semantic information. This structure is not only beneficial for the planning module to detect obstacles and collisions, but also optimizes the local planning efficiency and facilitates the positioning module to use abundant resources. The geometric information and semantic information can improve the accuracy and efficiency of registration, and provide intuitive and vivid visualization results, which are helpful for humans to issue intuitive navigation instructions.

Description of drawings

FIG. 1 is a schematic diagram of the overall framework of the map representation system of the present invention.

FIG. 2 is a schematic diagram of the connection between the spatial representation and the data representation in the multi-information voxel layer of the present invention.

FIG. 3 is a schematic diagram of the topology layer structure of the present invention.

FIG. 4 is an overall flow chart of the map construction method of the present invention.

Detailed ways

The accompanying drawings are for illustrative purposes only, and should not be construed as limiting the present invention; in order to better illustrate the present embodiment, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art It is understandable to the artisan that certain well-known structures and descriptions thereof may be omitted from the drawings. The positional relationships described in the drawings are only for exemplary illustration, and should not be construed as limiting the present invention.

Example 1:

As shown in FIG. 1, the present invention provides a map representation system of a visual robot, including a semantic information extraction module, a geometric information extraction module, a scene semantic extraction module, a multi-information voxel integration module, a spatial topology extraction module, a topology integration module, And map element layer, multi-information voxel layer and topology layer; first use three modules: semantic information extraction module, geometric information extraction module, scene semantic extraction module to extract image segmentation information, depth segmentation information and vertex and Normal, scene semantic information. Next, the image segmentation information and depth segmentation information will be fused into time-independent 3D spatial component information, and the 3D spatial component information and vertex information will be constructed using the multi-information voxel integration module to obtain multi-information voxels, which are obtained by using the visual SLAM method. Camera pose, updating voxels into the map's informative voxel layer. After obtaining the multi-information voxel layer, using the extracted spatial topology module and scene semantic information, the environment space is divided and merged into multiple 3D space convex hull sets with semantics, and the spatial adjacent relationship between them is used to represent the space node to space. Node topology information. So far, part of the 3D space component information, the semantic information of the semantic information extraction module, and the 3D space convex hull set with semantics have been integrated into the map element layer. In addition, based on the spatial position of the three-dimensional space components, the topology integration module is used to first associate the three-dimensional space components and the three-dimensional space convex hull to obtain the topology information from the space nodes to the component nodes, and then combine this information with the topological information from the space nodes to the space nodes. , to get the complete topology map. Finally, the multi-information voxel layer, the set of map elements and the topological map together constitute the map of the present invention.

The application scenario of this system is indoor or outdoor areas where various robots work. It requires a visual sensor with high pixels and a suitable focal length to clearly capture scenes within 0-20 meters. This system only needs an RGB-D camera or a binocular camera, which can be placed in front of the robot.

In order to better illustrate the specific embodiments of the present invention, the map construction method provided by the present invention will be described in detail below with reference to Figures 1-4 and the specific embodiments.

As shown in Figure 4, a map construction method for a visual robot includes the following steps:

Step 1: Connect the vision sensor, use the internal and external parameters obtained from the previous calibration to calibrate the RGB image and depth map, and obtain the calibrated RGB image and depth map. For the depth map, if the visual sensor is an RGB-D camera, it has depth information; if the visual sensor is a binocular stereo camera, the depth of each pixel is calculated by the "parallax method" to obtain a large depth map.

Step 2: Use the Mask-RCNN method for image segmentation to obtain the image cut area and the corresponding label.

Step 3: Use DCNN to classify the environment based on RGB images to obtain scene category information.

Step 4: Calculate the surface normal corresponding to the pixel based on the pixel and its adjacent points, then calculate the normal angle difference between adjacent pixels, and finally calculate the depth difference between adjacent pixels, and obtain three features in total.

Step 5: Based on the three features obtained in Step 4, perform clustering and cutting on the depth map to obtain multiple different depth segmentation regions.

Step 6: Denote the depth segmentation result extracted in the step 2 as a set S, and _si ∈ S is a geometric component segmentation block, which consists of a certain number of pixels and their depth information, and denote the image extracted in the step 1. The segmentation result is a set R, r _j ∈ R is an area on an image, which consists of a certain number of pixels and their classification information; for each s _i , calculate the r _j with the highest overlapping area, and the calculation formula is as follows:

Step 7: For each s _i in the step 6, assign the category of the highest r _j to the overlapping area, determine the best category of each geometric component segment, and classify the adjacent objects with common instances and categories. The geometric segmentation blocks are fused into 3D semantic components.

Step 8: Compare the three-dimensional semantic components with the three-dimensional semantic components existing in the map one by one. If the same instance (object in the same objective environment) exists in the current moment and the map, perform instance tracking. If there is no such instance in the map , add new 3D semantic parts to the map, and then perform instance tracking. The instance tracking is a method to ensure that the detection result of each instance object is kept temporally incoherent in multiple frames of images when the map is constructed.

Step 9: fuse the three-dimensional semantic components obtained in the step 7 and maintained in the step 8 with the vertex set extracted in the step 2. The space is first divided into voxels, and only voxels within the truncation distance are considered. Assuming that x is the center of the current voxel, p is the three-dimensional position of a vertex, and s is the origin of the sensor, then we have:

d(x,p,s)＝||p-x||sign((p-x)·(p-s))

∈=4v

W _i+1 (x,p,s)=min(W _i (x)+w(x,p),W _max )

In the formula, v is the size of the voxel, z is the depth from s to p, and W _max limits the maximum weight of the update; from the above formula, the TSDF value of the i+1th update voxel x can be calculated. D _i+1 (x, p, s) and weight W _i+1 (x, p, s); initial TSDF value D (x, p, s) and weight W (x, p, s) are both is initialized to 0.

Step 10: Calculate the ESDF value based on the TSDF value in the step 9. The process specifically includes: starting from the TSDF surface, by means of 26-neighbor search, directly using the TSDF value as the ESDF value in the area within the set cut-off area, and the value outside the cut-off area is propagated through water ripples. The algorithm calculates and finally obtains the ESDF.

Among them, the water ripple algorithm is carried out according to the following steps:

Step 10.1: The wave starts at a voxel (denoted v), propagates to its 26-neighborhood, updates the voxels whose ESDF distance has not been updated with their ESDF distance equal to the ESDF value of voxel v plus the unit distance, and Puts the newly updated voxels into the ripple diffusion queue. For each voxel in the ripple diffusion queue, step 10.1 is performed recursively in turn, until all voxels have updated ESDF distances.

Step 11: Based on the three-dimensional semantic component information obtained in the step 7 and maintained in the step 8 and the ESDF in the step 10, a multi-information voxel is formed through coordinate association.

Step 12: Obtain the current sensor pose through the visual SLAM method, convert the multi-information voxels in the sensor coordinate system to the representation in the map coordinate system, and then update them to the information voxel layer of the map. The information voxel layer stores multiple multi-information voxels by hashing. Each voxel contains three-dimensional position information, semantic coding information, whether it is occupied or not, and a mixed intercept/Euclidean distance field composition. Figure 2 shows Storage form, spatial representation and data structure of multi-information voxels.

Step 13: Randomly set the convex hull growth point on the route of collecting data, impose height and volume restrictions on each point, and continue to expand outward to form a spatial convex hull; and based on the coordinate position and the scene semantic information given in step 3, Get the set of spatial convex hulls with semantics.

Step 14: Based on the set of spatial convex hulls with semantics in the step 13, using the spatial adjacent relationship between them, obtain an undirected graph representing the spatial connection relationship.

Step 15: Set a certain threshold that allows obstacles (non-convex parts) to be accommodated, and merge the convex hulls, so that the obtained convex hulls are more in line with the original spatial shape of the environment based on human intuition, and a larger and more semantic information is obtained. A collection of ellipse-like spatial convex hulls. The process does not merge convex hulls when semantics conflict.

Step 16: For each three-dimensional semantic component obtained in step 7 and maintained in step 8, find an ellipse-like spatial convex hull (such as a cup) with semantic information obtained in step 13 to which it belongs according to its own position. This three-dimensional semantic component is matched to the space convex hull of the room), and the connection relationship between the space node and the component node is obtained.

Step 17: Integrate all the connection relationships between space nodes and component nodes obtained in step 17 into the undirected graph obtained in step 5 to obtain a complete topology map.

Step 18: The map of the present invention is formed by the multi-information voxel layer in step 12, the specific semantic information in step 2, the three-dimensional semantic component in step 8, and the topology map in step 17.

Obviously, the above-mentioned embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those of ordinary skill in the art, changes or modifications in other different forms can also be made on the basis of the above description. There is no need and cannot be exhaustive of all implementations here. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the claims of the present invention.

Claims (10)

1. a map representation system of a visual robot, is characterized in that, comprises semantic information extraction module, geometric information extraction module, scene semantic extraction module, multi-information voxel integration module, extraction space topology module, topology integration module and map element layers, informative voxel layers, and topology layers; where, The semantic information extraction module is used to collect images in an environment where a map needs to be constructed by using a visual sensor, and extract semantic information of images to obtain image segmentation results; then perform other specific semantic extraction to obtain specific semantic information, specific semantic information The result of the information part will eventually be stored in the map element layer; The geometric information extraction module is used to collect images in an environment where a map needs to be constructed by using a visual sensor, obtain a depth map, a set of vertices and a normal vector corresponding to the vertices through calculation, and then calculate several distances and normals based on the depth map. Geometric feature segmentation depth map; The scene semantic extraction module is used to collect images in an environment where a map needs to be constructed by using a visual sensor, and perform scene semantic extraction to obtain scene classification information, and the scene classification information is an artificially defined scene with a certain degree of identification. title; The multi-information voxel integration module is used to fuse the image segmentation results obtained by the semantic information extraction module and the depth segmentation results obtained by the geometric information extraction module to obtain a three-dimensional semantic component, which describes a certain scene in the scene by a human. Predetermined objects with a certain degree of identity and significance for planning and positioning; then, combined with the camera pose obtained by the visual SLAM method, the multi-information voxels calculated by using the 3D semantic components and the vertex set obtained by the geometric information extraction module , updated to the multi-information voxel layer of the map; The extraction spatial topology module is used for extracting the spatial convex hull containing scene semantics and the topology information expressing its connection relationship based on the multi-information voxel layer obtained by the multi-information voxel integration module; The topology integration module is used to associate the three-dimensional space components in the multi-information voxel integration module and the three-dimensional space convex hull obtained by the extraction space topology module based on the adjacent relationship in space, and calculate the relationship between the space nodes and the component nodes. topology information, and combine this information with the topology information from space nodes to space nodes to obtain a complete topology layer; The map element layer, the multi-information voxel layer and the topology layer together constitute a map, and the topology map connects the spatial convex hull in the map element layer with the three-dimensional component information. 2. The map representation system of a visual robot according to claim 1, wherein the multi-information voxel layer uses a cube with attached spatial information and semantic information, which does not actually exist in the objective world, It is a data abstract object used to approximate a cubic area of the objective world; the multi-information voxel layer is based on the needs of robot planning and positioning, and a variety of predefined environments represent objects, which are described by their location, semantics , Geometry field composition. 3. a map construction method of visual robot, is characterized in that, comprises the following steps: S1. Use the semantic information extraction module, use the visual sensor to collect images in the environment where the map needs to be constructed, and extract the semantic information of the components to obtain the image segmentation result, and perform the remaining specific semantic extraction to obtain the specific semantic information; the specific semantic information part The result will eventually be stored in the map element layer; S2. Use the geometric information extraction module, use the visual sensor to collect images in the environment where the map needs to be constructed, obtain the depth map, the vertex set and the normal corresponding to the vertex, and then calculate the depth segmentation result from the depth map; S3. Using the scene semantic extraction module, the visual sensor is used to collect images in the environment where the map needs to be constructed, and the scene semantic extraction is performed to obtain the classification information of the scene; S4. Use the multi-information voxel integration module to fuse the image segmentation results extracted in step S1 and the depth segmentation results extracted in step S2 to obtain three-dimensional semantic components; then, based on the visual sensor posture obtained by the visual SLAM method, the three-dimensional The semantic component and the vertex set extracted in step S2 are calculated to obtain multi-information voxels to form a multi-information voxel layer; S5. Using the extraction spatial topology module, based on the multi-information voxel layer in step S4, extract the spatial convex hull containing scene semantics and the topology information expressing its connection relationship; S6. Using the topology integration module, based on the adjacent relationship of the space, associate the three-dimensional space component in step S4 with the three-dimensional space convex hull in step S5, obtain the topology information from the space node to the component node, and associate this information with the space node to the space The topology information of nodes is merged to obtain a complete topology map, which constitutes a topology layer; S7. The map of the present invention is formed by the map element layer in step S1, the multi-information voxel layer in step S4, and the topology layer in step S6. 4. the map construction method of visual robot according to claim 3, is characterized in that, described step S1 specifically comprises: S11. Obtain an RGB image from a vision sensor; S12. Infer each pixel in the RGB image based on step S11, determine the category of its subordinate object, the process is completed based on the deep neural network, and finally obtain the binary mask image set of each instance of each category; S13. The rest of the specific semantic extraction is determined by the actual application. Based on step S11, the RGB image is extracted using the feature detection and inference module, and then the specific semantic extraction is performed to obtain the specific semantic. The specific semantics include but are not limited to available exercise area. 5. The map construction method of visual robot according to claim 3, is characterized in that, described step S2 specifically comprises: S21. Obtain a depth map from a vision sensor; S22. Based on step S21, from the depth map, calculate the surface normal corresponding to this pixel based on a pixel and its adjacent points, then calculate the normal angle difference between adjacent pixels, and finally calculate the depth difference between adjacent pixels to obtain three kinds of features. ; S23. Based on the three features obtained in the step S22, cluster and cut the depth map to obtain a plurality of different depth segmentation regions, which are called depth segmentation results. 6. The map construction method of visual robot according to claim 3, is characterized in that, described step S3 specifically comprises: S31. Obtain an RGB image from a vision sensor; S32. Calculate the high-dimensional features from the pixels of the RGB image and the relationship between the pixels, and then infer the classification information of the scene; the process is implemented based on deep neural networks, including but not limited to CNN networks, and scenes include but not limited to Shopping malls, kitchens, warehouses, corridors. 7. The map construction method of visual robot according to claim 3, is characterized in that, described step S4 specifically comprises: S41. Denote the depth segmentation result extracted in the step S2 as a set S, where s _i ∈ S is a geometric component segmentation block, consisting of a certain number of pixels and their depth information, denote the image segmentation extracted in the step S1 The result is a set R, where r _j ∈ R is an area on an image, which consists of a certain number of pixels and their classification information; for each s _i , calculate the r _j with the highest overlapping area, and the calculation formula is as follows:

S42. For each s _i in the step S41, assign the category of the highest r _j to the overlapping area, determine the best category for each geometric component segment, and assign adjacent objects with common instance objects and categories to Geometric segmentation and fusion into 3D semantic components; S43. Compare the three-dimensional semantic components and the three-dimensional semantic components existing in the map one by one. If the same instance exists in the current moment and the map, that is, an object in the same objective environment, perform instance tracking. If it does not exist in the map, Add a new three-dimensional semantic component to the map, and then perform instance tracking; the instance tracking is a method to ensure that the detection result of each instance object is kept temporally incoherent in multiple frames of images when the map is constructed; S44. fuse the three-dimensional semantic component maintained in the step S43 with the vertex set extracted in the step S2; first divide the space into voxels, and only consider the voxels within the truncation distance t; suppose x is the current voxel The center of , p is the three-dimensional position of a vertex, s is the origin of the sensor, at this time there are: d(x, p, s)=||p-x||sign((p-x)·(p-s))

∈=4v W _i+1 (x, p, s)=min(W _i (x)+w(x, p), W _max ) In the formula, v is the size of the voxel, z is the depth from s to p, and W _max limits the maximum weight of the update; from the above formula, the TSDF value of the i+1th update voxel x can be calculated. D _i+1 (x, p, s) and weight Wi ₊₁ (x, p, s); initial TSDF value D (x, p, s) and weight W (x, p, s) are both is initialized to 0; S45. Calculate the value of ESDF based on the TSDF value in step S44; the process specifically includes: starting from the TSDF surface, by means of 26-neighbor search, the area in the set truncation area is directly set as the value of TSDF as The value of ESDF, the value outside the truncation area is calculated by the water ripple propagation algorithm, and finally the ESDF is obtained; S46. Based on the three-dimensional semantic component information obtained in steps S42 and S43 and the ESDF in step S45, form a multi-information voxel; S47. Obtain the current sensor pose through the visual SLAM method, convert the multi-information voxels in the sensor coordinate system into the representation in the map coordinate system, and then update them to the information voxel layer of the map; The method stores multiple multi-information voxels, and each voxel contains three-dimensional position information, semantic encoding information, whether it is occupied or not, and a mixed intercept/Euclidean distance field composition. 8. the map construction method of visual robot according to claim 7 is characterized in that, described water ripple algorithm is carried out as follows: wave starts from a voxel is marked as v, propagates to its 26-neighborhood, For voxels whose ESDF distance has not been updated, update their ESDF distance to the ESDF value of voxel v plus the unit distance, and put the newly updated voxel into the ripple diffusion queue; for each voxel in the ripple diffusion queue, recurse in turn Repeat the above steps until all voxels have updated ESDF distances. 9. The map construction method of visual robot according to claim 3, is characterized in that, described step S5 specifically comprises: S51. Randomly collect convex hull growing points on the data collection route; S52. On each convex hull generation point of the step S51, add height and volume restrictions, and continuously expand outward to form a spatial convex hull; and based on the coordinate position and the scene semantic information given in step S3, obtain a space with semantics convex hull set; S53. Based on the set of spatial convex hulls with semantics in the step S52, utilize the spatial adjacent relationship between them to obtain an undirected graph representing the spatial connection relationship; S54. Set a certain threshold that allows the non-convex part of the obstacle to be accommodated, and merge the convex hull, so that the obtained convex hull is more in line with the original spatial shape of the environment intuitionally, and a larger elliptical-like shape with semantic information is obtained. The spatial convex hull set of ; the process does not merge the convex hull when the semantics conflict. 10. The map construction method of a visual robot according to claim 3, wherein the step S6 specifically comprises: S61. to the three-dimensional semantic component obtained in each step S4, find an ellipse-like space convex hull with semantic information obtained according to its own position in the step S54 to which it belongs, and obtain the connection relationship between the space node and the component node; S62. Integrate all the connection relationships between the space nodes and the component nodes obtained in step S61 into the topology map obtained in step S5 to obtain a complete topology map.

Images