WO2018119889A1 - Three-dimensional scene positioning method and device - Google Patents

Abstract

The present invention is related to the technical field of mixed reality. Disclosed is a three-dimensional scene positioning method and device for sharing three-dimensional map data. The three-dimensional scene positioning method comprises: generating three-dimensional map data according to first visual information of a current scene and depth information corresponding to the first visual information, the three-dimensional map data including a three-dimensional scene model (S101); performing feature extraction on second visual information of the current scene to obtain a second visual feature (S102); and applying, to the second visual information, and according to a mapping relationship between the second visual feature and the three-dimensional map data, a corresponding three-dimensional scene model, so as to realize positioning of the second visual information in the three-dimensional scene model (S103). The solution is applied to the three-dimensional map data sharing during three-dimensional scene positioning.

Description

Three-dimensional scene positioning method and device Technical field

The present application relates to the field of mixed reality technologies, and in particular, to a method and an apparatus for positioning a three-dimensional scene.

Background technique

Mixed reality (MR) is a technique for calculating the position and angle of a camera image in real time and superimposing images, videos, and 3D models on the image. The goal of this technology is to superimpose the virtual world on the screen. The world interacts. At present, the mature MR technology products on the market include Microsoft's holographic glasses HoloLens, Intel's VR (virtual reality) / AR (augmented reality), and the like.

Microsoft's HoloLens uses four cameras, two sets of binocular cameras as sensors, and combines the TOF (time of flight) sensor in the center of the helmet to achieve three-dimensional reconstruction of the current scene. Intel's Alloy works similarly to HoloLens, but it uses the RealSense 3D structured real-life camera developed by Intel itself as a sensor. In addition to a pair of binocular cameras, two sets of structured light depth cameras are built in, which can directly capture depth images to restore 3D scenes.

The above-mentioned MR head-display devices capture real-world image and depth data by dedicated color and depth sensors integrated on the helmet, without the use of a more integrated monocular, binocular vision sensor due to the existing single Mesh and binocular sensors are not capable of dense and high-precision 3D reconstruction, and are computationally resource intensive and cannot be integrated into mobile devices. In addition, the three-dimensional map data cannot be shared between the above two devices, which means that the 3D scene built by HoloLens cannot be experienced by the Alloy device, and vice versa. In addition, in the field of high-precision 3D map navigation applications, high-precision 3D map street view scanning devices are often complicated, bulky and expensive, and ordinary consumers do not carry these professional devices to locate and navigate.

Application content

Embodiments of the present application provide a three-dimensional scene positioning method and apparatus for sharing three-dimensional map data.

To achieve the above objective, the embodiment of the present application adopts the following technical solutions:

In a first aspect, a method for positioning a three-dimensional scene is provided, including:

Generating three-dimensional map data according to the first visual information of the current scene and the depth information corresponding to the first visual information, where the three-dimensional map data includes a three-dimensional scene model; and extracting the second visual information of the current scene to obtain a second feature a visual feature; loading a corresponding three-dimensional scene model on the second visual information according to the mapping relationship between the second visual feature and the three-dimensional map data, and implementing the second visual information in the three-dimensional scene model Positioning.

In a second aspect, a three-dimensional scene positioning apparatus is provided, including:

a generating unit, configured to generate three-dimensional map data according to the first visual information of the current scene and the depth information corresponding to the first visual information, where the three-dimensional map data includes a three-dimensional scene model;

An extracting unit, configured to perform feature extraction on the second visual information of the current scene to obtain a second visual feature;

a positioning unit, configured to load a corresponding three-dimensional scene model on the second visual information according to a mapping relationship between the second visual feature and the three-dimensional map data, to implement the second visual information in the three-dimensional scene model Positioning in .

In a third aspect, a computer storage medium is provided for storing computer software instructions for use in a three-dimensional scene location device, comprising program code designed to perform the three-dimensional scene location method of the first aspect.

In a fourth aspect, a computer program product is provided that can be directly loaded into an internal memory of a computer and includes software code, and the computer program can be loaded and executed by a computer to implement the three-dimensional scene positioning method according to the first aspect. .

A fifth aspect provides a three-dimensional scene positioning apparatus, including: a memory, a communication interface, and a processor, the memory is configured to store computer execution code, and the processor is configured to execute the computer to perform code control execution. The three-dimensional scene positioning method, the communication interface is used for the number of the three-dimensional scene positioning device and the external device According to transmission.

The method and device for positioning a three-dimensional scene provided by the embodiment of the present application generates three-dimensional map data according to the first visual information and depth information corresponding to the first visual information, and performs feature extraction on the second visual information to obtain a second visual feature. According to the mapping relationship between the second visual feature and the three-dimensional map data, the corresponding three-dimensional scene model is loaded on the second visual information to realize the positioning of the second visual information in the three-dimensional scene model. The three-dimensional map data generated according to the first visual information and the corresponding depth information can be used for the positioning of the second visual information, and the sharing of the three-dimensional map data is realized.

DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings to be used in the embodiments or the prior art description will be briefly described below. Obviously, the drawings in the following description are only It is a certain embodiment of the present application, and other drawings can be obtained according to the drawings without any creative work for those skilled in the art.

FIG. 1 is a schematic structural diagram of a three-dimensional scene positioning system according to an embodiment of the present application;

2 is a schematic structural diagram of a first visual collection device according to an embodiment of the present application;

FIG. 3 is a schematic structural diagram of a server according to an embodiment of the present application;

4 is a schematic structural diagram of a second visual collection device according to an embodiment of the present application;

FIG. 5 is a schematic flowchart of a method for positioning a three-dimensional scene according to an embodiment of the present disclosure;

FIG. 6 is a schematic flowchart diagram of another method for positioning a three-dimensional scene according to an embodiment of the present disclosure;

FIG. 7 is a schematic flowchart of updating a visual feature database and performing a three-dimensional scene reconstruction to obtain a three-dimensional scene model according to an embodiment of the present application;

FIG. 8 is a schematic diagram of a result of a three-dimensional scene model according to an embodiment of the present application;

FIG. 9 is a schematic flowchart of obtaining a real-time pose of a second visual collection device according to an embodiment of the present application;

FIG. 10 is a schematic diagram of a result of performing feature matching on an ORB feature according to an embodiment of the present application; FIG.

FIG. 11 is a schematic diagram of a result of performing feature matching on an overall grayscale distribution feature according to an embodiment of the present application; FIG.

FIG. 12 is a schematic diagram of a result of performing three-dimensional scene positioning on an ORB feature according to an embodiment of the present application;

FIG. 13 is a schematic diagram of a result of performing three-dimensional scene positioning on an overall grayscale distribution feature according to an embodiment of the present disclosure;

FIG. 14 is a schematic flowchart diagram of another method for positioning a three-dimensional scene according to an embodiment of the present disclosure;

FIG. 15 is a schematic flowchart diagram of aligning a three-dimensional dense point cloud with a vSLAM sparse point cloud registration to reconstruct a three-dimensional scene model according to an embodiment of the present application;

16 is a schematic diagram of a vSLAM sparse point cloud and a three-dimensional dense point cloud fused to form a three-dimensional point cloud according to an embodiment of the present disclosure;

FIG. 17 is a schematic diagram of positioning a second visual collection device according to a real-time pose and a three-dimensional scene model according to an embodiment of the present application;

FIG. 18 is a schematic structural diagram of a three-dimensional scene positioning apparatus according to an embodiment of the present disclosure;

FIG. 19 is a schematic structural diagram of still another three-dimensional scene positioning apparatus according to an embodiment of the present application;

FIG. 20 is a schematic structural diagram of another three-dimensional scene positioning apparatus according to an embodiment of the present disclosure.

detailed description

Embodiments of the present application will be described below with reference to the accompanying drawings.

The embodiment of the present application provides a three-dimensional scene positioning system, as shown in FIG. 1, comprising: a first visual collection device 11, a server 12, and a second visual collection device 13.

The first visual acquisition device 11 includes a mobile device with dedicated color and depth sensor devices, such as RGB-D (red green blue-depth) device, RGB-TOF (red green blue-time of flight) , red, green and blue - flight time equipment, Microsoft's Kinect or Intel's RealSense, 3D laser radar equipment, etc., can simultaneously capture the visual information and depth information of the current scene. Refer to Figure 2 As shown, the first visual acquisition device 11 can include a color and depth sensor 1101, a processor 1102, a memory 1103, and a communication interface 1104 that are connected by a bus. The memory 1103 is configured to store code and data and the like for execution by the processor 1102. The processor 1102 controls the color and depth sensor 1101 to collect visual information and depth information of the current location, and then undergoes preliminary processing, and then passes through the communication interface 1104 to be wired or The method is wirelessly transmitted to the server 12.

The server 12 is configured to store a three-dimensional scene model to implement a shared three-dimensional scene model between different devices. Referring to FIG. 3, the server 12 includes a processor 1202, a memory 1203, and a communication interface 1204, which are connected by a bus. The memory 1203 is configured to store code and data and the like for execution by the processor 1202, and the communication interface 1204 is configured to communicate with the first visual collection device 11 and the second visual collection device 13 in a wired or wireless manner.

The second visual collection device 13 includes any mobile device with a common visual sensor, such as a mobile phone with a common monocular or binocular camera, a tablet computer, AR/MR glasses, etc., and can collect visual information of the current scene. Referring to FIG. 4, the second visual acquisition device 13 can include a visual sensor 1301, a processor 1302, a memory 1303, and a communication interface 1304 that are connected by a bus. The memory 1303 is configured to store code and data and the like for execution by the processor 1302, and the processor 1302 controls the visual sensor 1101 to collect visual information of the current location, and controls to receive the three-dimensional scene from the server 12 by the communication interface 1104 in a wired or wireless manner. model.

The visual information described in the embodiment of the present application includes an RGB image frame or a gray scale image frame of the current image frame, and the depth information includes a depth of field. The pose described in the embodiment of the present application refers to the position and posture of the visual acquisition device. The three-dimensional scene model described in the embodiments of the present application includes a point cloud having three-dimensional coordinate points. The point cloud described in the embodiment of the present application refers to a point data set of an object surface in a current scene obtained by a measuring instrument. A point cloud having a relatively small number of points and a large point-to-point spacing is called a sparse point cloud. A relatively large and dense point cloud is called a dense point cloud.

The embodiment of the present application provides a three-dimensional scene positioning method, which is referred to in FIG. 5 . The indication includes: steps S101-S103.

S101. Generate three-dimensional map data according to first visual information of the current scene and depth information corresponding to the first visual information, where the three-dimensional map data includes a three-dimensional scene model.

The first visual information and the corresponding depth information may be acquired on one device or may be acquired on different devices, but to ensure a one-to-one correspondence of the data, it is preferably obtained by using the first visual collection device as described above, so that The three-dimensional map data is formed based on the visual information and the depth information.

The process of generating the three-dimensional map data may be implemented on the first visual collection device or the server. The embodiment of the present application is not limited herein.

The three-dimensional map data can be stored on the server such that other devices for positioning can share the three-dimensional map data.

S102. Perform feature extraction on the second visual information of the current scene to obtain a second visual feature.

The device for acquiring the second visual information may be different from the device for acquiring the first visual information. Since the first visual information is used to generate three-dimensional map data, the accuracy requirement thereof is high, and may be performed by a dedicated visual acquisition device, for example, as described above. A visual acquisition device to obtain. The second visual information is only used for positioning according to the above three-dimensional map data, so the accuracy requirement is low, and can be acquired by the second visual collection device as described above.

A variety of feature extractions can be performed on the visual information, such as extracting corner features, line features, shape features, or vSLAM features. The process of performing feature extraction on the second visual information may be implemented on the second visual collection device or the server. The embodiment of the present application is not limited herein.

S103. Load a corresponding three-dimensional scene model on the second visual information according to the mapping relationship between the second visual feature and the three-dimensional map data, and implement positioning of the second visual information in the three-dimensional scene model.

The second visual feature may be mapped and matched with different three-dimensional scene models in the three-dimensional map data, and the corresponding three-dimensional scene model is simultaneously displayed when the second visual information is displayed, the mixed reality is realized, and the second visual information is realized in the three-dimensional scene model. Positioning.

The three-dimensional scene positioning method provided by the embodiment of the present application generates three-dimensional map data according to the first visual information and the depth information corresponding to the first visual information, and performs feature extraction on the second visual information to obtain a second visual feature; The mapping relationship between the second visual feature and the three-dimensional map data loads the corresponding three-dimensional scene model on the second visual information to realize the positioning of the second visual information in the three-dimensional scene model. The three-dimensional map data generated by the second visual information according to the first visual information and the corresponding depth information may be used for positioning of the second visual information, and the sharing of the three-dimensional map data is realized.

An embodiment of the present application provides another method for positioning a three-dimensional scene. Referring to FIG. 6, the method includes: steps S201-S114.

S201. The first visual collection device collects first visual information and corresponding depth information of the current scene.

Each time the first visual acquisition device acquires the current frame image, the corresponding depth information, that is, the depth of field, is simultaneously acquired.

S202. The first visual collection device performs feature extraction on the first visual information to obtain a first visual feature.

Depending on how the image features are described, different methods can be selected to extract the visual features and then aggregated into a visual feature library. The visual feature library can include corner features, line features, shape features, and the like.

For the feature description of the corner feature, the visual feature library may include a feature binary robust independent elementary feature (ORB) feature library, and the first visual acquisition device extracts the ORB feature from the first visual information. The ORB feature of the first visual information.

For the linear description method, the overall gray scale distribution information can be extracted for the first visual information, and the commonly used feature extraction and positioning methods include LSD-SLAM (large scale direct simultaneous localization and mapping). The embodiments of the present application are not described herein.

S203. The first visual acquiring device performs feature matching on the first visual feature and the visual feature database to obtain a rotation matrix and a displacement matrix of the first visual information, and calculates the first visual information according to the rotation matrix and the displacement matrix of the first visual information. Real-time pose, And updating the visual feature library with the first visual feature.

The ORB feature of the first visual information is matched with the existing key frame ORB feature library, and the rotation matrix and the displacement matrix of the first visual information are calculated, thereby calculating the first vision according to the rotation matrix and the displacement matrix of the first visual information. The real-time pose of the acquisition device when acquiring the first visual information. At the same time, it is determined whether the key frame ORB feature database needs to be updated with the first visual information. That is, whether the first visual information is added to the key frame ORB feature library, or whether the first visual information is replaced with a certain frame in the key frame ORB feature library. For example, whether the number of common feature points of the first visual information and the key frame ORB feature database can be compared is less than a preset threshold, and if not, the first visual information is added to the key frame ORB feature library.

S204. Perform a three-dimensional scene reconstruction according to the depth information and the real-time pose of the first visual information to obtain a three-dimensional scene model.

For the description of the corner feature, referring to FIG. 7, the three-dimensional scene reconstruction of the real-time pose and depth information of the first visual acquisition device to obtain the three-dimensional scene model specifically includes:

S2041: Perform three-dimensional scene reconstruction using the calculated real-time pose and the collected depth information to generate a three-dimensional point cloud of the first visual information.

The 3D point cloud effect obtained from the 3D scene reconstruction is shown in Figure 8 (the color information is removed in the figure).

S2042: Perform a fusion, denoising, and deleting a repeated point on the three-dimensional point cloud of the first visual information to finally obtain a three-dimensional scene model.

Steps S202-S204 correspond to step S101.

S205. The first visual collection device sends the updated visual feature database and the three-dimensional scene model to the server for storage.

At this time, the three-dimensional map data includes a visual feature library and a three-dimensional scene model.

S206. The server receives the visual feature library and the three-dimensional scene model from the first visual collection device.

S207. The second visual acquisition device calibrates the visual sensor of the second visual acquisition device to obtain a camera parameter of the visual sensor of the second visual acquisition device, and according to The camera parameters correct the visual sensor of the second visual acquisition device.

Before using the second visual acquisition device to collect visual information, the user needs to calibrate the device to determine camera parameters of the visual sensor. The camera parameters include parameter information such as a principal point, a focal length, and a distortion correction parameter, where the distortion correction parameter is used to identify the The visual sensor of the second visual acquisition device is distorted when the image is taken. Parameters such as the main point and focal length are used to focus the camera. First select the camera type (depth, binocular or monocular) of the second visual acquisition device, and then use the second visual acquisition device to take multiple (eg 20 to 30) different angle photos of the camera calibration pattern. By detecting and registering the calibration pattern, the camera parameters of the vision sensor used by the second vision acquisition device are calculated, and then the camera parameters are corrected corresponding to the vision sensor to obtain the corrected second vision acquisition device.

S208. The server sends the visual feature database and the three-dimensional scene model to the second visual collection device.

S209. The second visual collection device receives the visual feature library and the three-dimensional scene model from the server.

S210. The second visual collection device collects second visual image information of the current scene.

S211. The second visual collection device performs feature extraction on the second visual image information to obtain a second visual feature.

For the corner feature description manner, the ORB feature is extracted for the second visual information to obtain the ORB feature of the second visual information as the second visual feature. For the linear feature description mode, the overall grayscale distribution information is extracted for the second visual information to obtain the overall grayscale distribution feature of the second visual information as the second visual feature.

Step S211 corresponds to step S102.

S212. Obtain a real-time pose when the second visual information collection device acquires the second visual information according to the second visual feature and the visual feature database.

Referring to FIG. 9, step S212 specifically includes:

S2121. Perform feature matching on the second visual feature according to the visual feature library.

Exemplarily, for the corner feature description manner, the ORB feature of the second visual information is feature-matched according to the key frame ORB feature library. Exemplarily, referring to Figure 10 An example of feature matching for an ORB feature is shown, where the small box is an ORB feature point. An example of feature matching for the overall grayscale distribution feature is shown in FIG.

S2122: performing a PnP (perspective n point problem) pose calculation on the feature-matched second visual feature to obtain a real-time pose of the second visual information.

Exemplarily, for the corner feature description manner, the PnP pose calculation is performed on the ORB feature of the feature-matched second visual information to obtain the real-time pose of the second visual acquisition device.

S213. Load a three-dimensional scene model on the second visual information according to a mapping relationship between the real-time pose of the second visual information and the corresponding three-dimensional scene model in the three-dimensional map data, so as to realize positioning of the second visual information in the three-dimensional scene model.

Exemplarily, referring to FIG. 12 is a schematic diagram showing the result of performing three-dimensional scene positioning on the ORB feature, and the star symbol in the figure indicates the positioning position. Referring to Fig. 13, there is shown a schematic diagram of the result of three-dimensional scene localization of the overall grayscale distribution feature, the white arrows in the figure indicating the positioning position and direction. Steps S212-S213 correspond to step S103.

In the three-dimensional scene positioning method provided by the embodiment of the present application, the first visual collection device updates the visual feature database with the first visual information of the current scene and acquires the real-time pose of the first visual acquisition device, and the real-time pose of the first visual acquisition device. And reconstructing the three-dimensional scene with the depth information corresponding to the first visual information to obtain a three-dimensional scene model, and then transmitting the visual feature database and the three-dimensional scene model to the server; the second visual acquiring device acquires the visual feature database and the three-dimensional scene model from the server, and then The second visual information extracts the visual feature information, obtains the real-time pose according to the visual feature information and the visual feature database, realizes the positioning according to the real-time pose and the three-dimensional scene model, and realizes sharing the three-dimensional between the first visual acquiring device and the second visual collecting device. The scene model and visual feature library achieve the purpose of sharing 3D map data.

Moreover, the embodiment of the present application can realize high-precision three-dimensional map data sharing, and realizes dense three-dimensional reconstruction by using a visual acquisition device with dedicated color and depth sensors in an offline three-dimensional scene reconstruction stage, and even can be performed by a high-speed computing unit. Intelligently combine, denoise, and remove duplicate points to optimize the number of generated 3D point clouds, making the 3D reconstruction point cloud more accurate to meet the graphics rendering requirements in MR applications. At the same time, the visual feature information of these high-precision point clouds is extracted as shared data. In the positioning stage of the online visual acquisition device, the visual sensor (including monocular, binocular camera, etc.) of any visual acquisition device can be used for positioning, which greatly expands the application platform of the MR, and the cost is greatly reduced, and the popularity is more popular.

When the first visual information includes the third visual information and the fourth visual information, the embodiment of the present application provides another three-dimensional scene positioning. Referring to FIG. 14 , the method includes:

S301. The first visual collection device collects third visual information and corresponding depth information of the current scene.

S302. The first visual collection device performs three-dimensional scene reconstruction on the third visual information and the corresponding depth information to generate a three-dimensional dense point cloud, or performs three-dimensional scene reconstruction on the depth information to generate the three-dimensional dense point cloud.

For example, indoors can use a mobile device with dedicated color and depth sensor devices (such as RGB-D, Microsoft's Kinect or Intel's RealSense) to capture RGB images and their corresponding depth image data to form a three-dimensional dense point cloud (x, y, z) _RGBD , which can use 3D lidar to scan depth information to form a 3D dense point cloud (x, y, z) _Radar .

S303. The first visual collection device sends the three-dimensional dense point cloud to the server.

S304. The server receives the three-dimensional dense point cloud from the first visual collection device.

S305. The second visual collection device collects fourth visual information of the current scene.

S306. The second visual collection device performs a three-dimensional scene reconstruction on the fourth visual information of the current scene to generate a vSLAM (visual simultaneous localization and mapping) sparse point cloud.

It should be noted that the fourth visual information may also be the third visual information. At this time, the third visual information is reconstructed by the first visual acquisition device to generate a vSLAM sparse point cloud.

Each three-dimensional point (x, y, z) _vSLAM in the vSLAM sparse point cloud corresponds to the feature information F _{vSLAM of the} visual information (eg, corner feature, line feature or shape feature, etc.).

S307. The second visual collection device sends the vSLAM sparse point cloud to the server.

S308. The server receives the vSLAM sparse point cloud from the second visual collection device.

S309. The server aligns the three-dimensional dense point cloud with the vSLAM sparse point cloud to reconstruct the three-dimensional scene model. Referring to FIG. 15, specifically, steps S3091-S3093 are included:

S3091, the vSLAM sparse point cloud extracts the point cloud feature to obtain the vSLAM sparse point cloud feature, and the three-dimensional dense point cloud extracts the point cloud feature to obtain the three-dimensional dense point cloud feature.

Exemplarily, the point cloud feature can adopt the common point feature histograms (PFH), fast point feature histograms (FPFH), and viewpoint feature histogram (VFH). Signature of histograms of orientations (SHOT) and so on.

S3092: Performing a three-dimensional point cloud registration calculation on the vSLAM sparse point cloud feature and the three-dimensional dense point cloud feature to obtain a conversion relationship RT between the vSLAM sparse point cloud and the three-dimensional dense point cloud.

S3093: Align the vSLAM sparse point cloud with the three-dimensional dense point cloud three-dimensional dense point registration according to the conversion relationship between the vSLAM sparse point cloud and the three-dimensional dense point cloud to reconstruct the three-dimensional scene model. At this time, the three-dimensional map data is a three-dimensional scene model.

Finally, the vSLAM sparse point cloud and the three-dimensional dense point cloud in different coordinate systems are finally merged to form a three-dimensional point cloud. As shown in Fig. 16, the vSLAM sparse point cloud 1601 and the three-dimensional dense point cloud 1602 are merged into one. New 3D point cloud 1603. Steps S302, S306, and S309 correspond to step S101.

S310. The server sends the three-dimensional scene model to the second visual collection device.

S311. The second visual collection device receives the three-dimensional scene model from the server.

S312. The second visual collection device collects second visual information.

S313. The second visual collection device extracts a vSLAM visual feature for the second visual information, where the vSLAM visual feature includes a real-time pose when the second visual acquisition device acquires the second visual information. Step S313 corresponds to step S102.

S314. The second visual acquisition device is based on the real-time pose and the three-dimensional of the second visual information. The mapping relationship of the corresponding three-dimensional scene model in the map data loads the three-dimensional scene model on the second visual information to realize the positioning of the second visual information in the three-dimensional scene model.

For example, the perspective of the current device is mapped into the three-dimensional scene model, and the high-precision three-dimensional map environment information can be presented on the second visual acquisition device. Exemplarily, referring to FIG. 17, the fifth visual information 1701 collected in real time may be displayed in the same interface as the corresponding virtual three-dimensional scene model 1702.

In addition, according to the result of the positioning, path planning and 3D navigation applications can also be performed, and finally presented to the user or the customer service. Step S314 corresponds to step S103.

In the three-dimensional scene positioning method provided by the embodiment of the present application, the first visual acquiring device performs three-dimensional scene reconstruction on the third visual information and the corresponding depth information to generate a three-dimensional dense point cloud, and sends the three-dimensional dense point cloud to the server; The acquiring device performs three-dimensional scene reconstruction on the fourth visual information to generate a vSLAM sparse point cloud, and sends the vSLAM sparse point cloud to the server; the server aligns the three-dimensional dense point cloud with the vSLAM sparse point cloud to reconstruct the three-dimensional scene model; The second visual acquisition device acquires a three-dimensional scene model from the server, extracts a vSLAM visual feature from the fifth visual information, and the vSLAM visual feature includes a real-time pose of the second visual acquisition device, and then according to the real-time pose and the three-dimensional scene of the second visual acquisition device. The model locates the second visual acquisition device, realizes sharing a three-dimensional scene model between the first visual acquisition device and the second visual acquisition device, and achieves the purpose of sharing the three-dimensional map data.

The embodiments of the present application may divide the functional modules of each device according to the foregoing method example. For example, each functional module may be divided according to each function, or two or more functions may be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of the module in the embodiment of the present application is schematic, and is only a logical function division, and the actual implementation may have another division manner.

FIG. 18 is a schematic diagram showing a possible structure of the three-dimensional scene locating device involved in the above embodiment. The three-dimensional scene locating device 10 includes: a generating unit 1011 and an extracting unit. 1012. The positioning unit 1013 and the calibration unit 1014. The generating unit 1011 is configured to support a three-dimensional scene The positioning device performs the process S101 in FIG. 5, the processes S202-S204 in FIG. 6, the processes S2041 and S2042 in FIG. 7, the processes S302, S306, S309 in FIG. 14, the processes S3091-S3093 in FIG. 15, and the extraction unit 1012 is used to support the three-dimensional scene locating device to perform the process S102 in FIG. 5, the process S211 in FIG. 6, the process S313 in FIG. 14; the positioning unit 1013 is configured to support the three-dimensional scene locating device to execute the process S103 in FIG. 5, FIG. Processes S212 and S213, processes S2121 and S2122 in FIG. 9, process S314 in FIG. 14; calibration unit 1014 is configured to support the three-dimensional scene location device to perform process S207 in FIG. All the related content of the steps involved in the foregoing method embodiments may be referred to the functional descriptions of the corresponding functional modules, and details are not described herein again.

In the case of employing an integrated unit, FIG. 19 shows a possible structural diagram of the three-dimensional scene locating device involved in the above embodiment. The three-dimensional scene positioning device 10 includes a processing module 1022 and a communication module 1023. The processing module 1022 is configured to control and manage the actions of the three-dimensional scene positioning device. For example, the processing module 1022 is configured to support the three-dimensional scene positioning device to perform the processes S101-S103 in FIG. 5. The communication module 1013 is for supporting communication of the three-dimensional scene location device with other entities, such as communication with the functional modules or network entities shown in FIG. The three-dimensional scene location device 10 may further include a storage module 1021 for storing program codes and data of the three-dimensional scene location device.

The processing module 1022 can be a processor or a controller, for example, a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), and an application-specific integrated circuit (application-specific). Integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It is possible to implement or carry out the various illustrative logical blocks, modules and circuits described in connection with the present disclosure. The processor may also be a combination of computing functions, for example, including one or more microprocessor combinations, a combination of a DSP and a microprocessor, and the like. The communication module 1023 may be a transceiver, a transceiver circuit, a communication interface, or the like. The storage module 1021 can be a memory.

When the processing module 1022 is a processor, the communication module 1023 is a transceiver, and the storage module 1021 is a memory, the three-dimensional scene positioning device according to the embodiment of the present application may be the three-dimensional scene positioning device shown in FIG.

Referring to FIG. 20, the three-dimensional scene positioning apparatus 10 includes a processor 1032, a transceiver 1033, a memory 1031, and a bus 1034. The transceiver 1033, the processor 1032, and the memory 1031 are connected to each other through a bus 1034. The bus 1034 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. Wait. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

The steps of a method or algorithm described in connection with the present disclosure may be implemented in a hardware or may be implemented by a processor executing software instructions. The embodiment of the present application further provides a storage medium, which may include a memory 1031 for storing computer software instructions used by the three-dimensional scene location device, including program code designed to execute a three-dimensional scene location method. Specifically, the software instructions may be composed of corresponding software modules, and the software modules may be stored in a random access memory (RAM), a flash memory, a read only memory (ROM), and an erasable programmable only. An erasable programmable ROM (EPROM), an electrically erasable programmable read only memory (EEPROM), or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor to enable the processor to read information from, and write information to, the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. Additionally, the ASIC can be located in a three dimensional scene location device. Of course, the processor and the storage medium can also exist as discrete components in the three-dimensional scene location device.

The embodiment of the present application further provides a computer program, which can be directly loaded into the memory 1031 and contains software code, and the computer program can be loaded and executed by a computer to implement the above-described three-dimensional scene localization method.

Claims (23)

A three-dimensional scene positioning method, comprising: Generating three-dimensional map data according to first visual information of the current scene and depth information corresponding to the first visual information, where the three-dimensional map data includes a three-dimensional scene model; Performing feature extraction on the second visual information of the current scene to obtain a second visual feature; And corresponding to the mapping relationship between the second visual feature and the three-dimensional map data, loading a corresponding three-dimensional scene model on the second visual information, and realizing positioning of the second visual information in the three-dimensional scene model. The method according to claim 1, wherein before the feature extraction of the second visual information of the current scene to obtain the second visual feature, the method further comprises: Aligning a visual sensor that acquires the second visual information to obtain a camera parameter of the visual sensor, and correcting the visual sensor according to the camera parameter, the camera parameter including a principal point, a focal length, and a distortion correction parameter . The method of claim 1 wherein said three-dimensional map data further comprises a library of visual features; And generating the three-dimensional map data according to the first visual information of the current scene and the depth information corresponding to the first visual information, including: Performing feature extraction on the first visual information to obtain a first visual feature; Performing feature matching on the first visual feature and the visual feature database to obtain a rotation matrix and a displacement matrix of the first visual information, and calculating the first according to a rotation matrix and a displacement matrix of the first visual information. Real-time pose of visual information, and updating the visual feature library with the first visual feature; Performing the three-dimensional scene reconstruction according to the depth information and the real-time pose of the first visual information to obtain the three-dimensional scene model. The method according to claim 3, wherein the performing the three-dimensional scene reconstruction according to the real-time pose and the depth information to obtain the three-dimensional scene model comprises: Performing a three-dimensional scene reconstruction on the real-time pose and the depth information to generate the a three-dimensional point cloud of the first visual information; And merging, denoising, and deleting the repeated points of the three-dimensional point cloud of the first visual information to obtain the three-dimensional scene model. The method according to claim 3, wherein the loading the corresponding three-dimensional scene model on the second visual information according to the mapping relationship between the second visual feature and the three-dimensional map data comprises: Obtaining a real-time pose of the second visual information according to the second visual feature and the visual feature database; And loading the three-dimensional scene model on the second visual information according to a mapping relationship between a real-time pose of the second visual information and a corresponding three-dimensional scene model in the three-dimensional map data. The method according to claim 5, wherein the obtaining the real-time pose of the second visual information according to the second visual feature and the visual feature library comprises: Performing feature matching on the second visual feature according to the visual feature library; Performing a perspective N-point problem PnP pose calculation on the feature-matched second visual feature to obtain a real-time pose of the second visual information. The method of claim 1, wherein the first visual information comprises third visual information and fourth visual information; The generating the three-dimensional map data according to the first visual information of the current scene and the depth information corresponding to the first visual information includes: Performing a three-dimensional scene reconstruction on the third visual information and the corresponding depth information to generate a three-dimensional dense point cloud, or performing three-dimensional scene reconstruction on the depth information to generate the three-dimensional dense point cloud; Performing a three-dimensional scene reconstruction on the fourth visual information to generate a visual instant location and a map construction vSLAM sparse point cloud; Aligning the three-dimensional dense point cloud with the vSLAM sparse point cloud to reconstruct the three-dimensional scene model. The method of claim 7 wherein said pair of said three dimensions A dense point cloud is aligned with the vSLAM sparse point cloud to reconstruct a three-dimensional scene model, including: Extracting a point cloud feature from the vSLAM sparse point cloud to obtain a vSLAM sparse point cloud feature, and extracting a point cloud feature from the three-dimensional dense point cloud to obtain a three-dimensional dense point cloud feature; Performing a three-dimensional point cloud registration calculation on the vSLAM sparse point cloud feature and the three-dimensional dense point cloud feature to obtain a conversion relationship between the vSLAM sparse point cloud and the three-dimensional dense point cloud; And aligning the vSLAM sparse point cloud with the three-dimensional dense point cloud by three-dimensional dense point registration according to the conversion relationship between the vSLAM sparse point cloud and the three-dimensional dense point cloud to generate a new three-dimensional point cloud as the three-dimensional scene model. The method according to claim 7, wherein the extracting the second visual information of the current scene to obtain the second visual feature comprises: Feature extraction is performed on the second visual information to obtain a vSLAM visual feature. The method according to claim 9, wherein the vSLAM visual feature includes a real-time pose of the second visual information; And loading the corresponding three-dimensional scene model according to the mapping relationship between the second visual feature and the three-dimensional map data, including: And loading the three-dimensional scene model on the second visual information according to a mapping relationship between a real-time pose of the second visual information and a corresponding three-dimensional scene model in the three-dimensional map data. A three-dimensional scene positioning device, comprising: a generating unit, configured to generate three-dimensional map data according to the first visual information of the current scene and the depth information corresponding to the first visual information, where the three-dimensional map data includes a three-dimensional scene model; An extracting unit, configured to perform feature extraction on the second visual information of the current scene to obtain a second visual feature; a positioning unit, configured to load a corresponding three-dimensional scene model on the second visual information according to a mapping relationship between the second visual feature and the three-dimensional map data, to implement the The positioning of the second visual information in the three-dimensional scene model. The device according to claim 11, wherein the device further comprises: a calibration unit, configured to: after performing feature extraction on the second visual information of the current scene to obtain the second visual feature, calibrating the visual sensor that collects the second visual information to obtain a camera parameter of the visual sensor, and The vision sensor is corrected according to the camera parameters, the camera parameters including a principal point, a focal length, and a distortion correction parameter. The apparatus according to claim 11, wherein said three-dimensional map data further comprises a visual feature library; The generating unit is specifically configured to: Performing feature extraction on the first visual information to obtain a first visual feature; Performing feature matching on the first visual feature and the visual feature database to obtain a rotation matrix and a displacement matrix of the first visual information, and calculating the first according to a rotation matrix and a displacement matrix of the first visual information. Real-time pose of visual information, and updating the visual feature library with the first visual feature; Performing the three-dimensional scene reconstruction according to the depth information and the real-time pose of the first visual information to obtain the three-dimensional scene model. The device according to claim 13, wherein the generating unit is specifically configured to: Performing a three-dimensional scene reconstruction on the real-time pose and the depth information to generate a three-dimensional point cloud of the first visual information; And merging, denoising, and deleting the repeated points of the three-dimensional point cloud of the first visual information to obtain the three-dimensional scene model. The device according to claim 13, wherein the positioning unit is specifically configured to: Obtaining a real-time pose of the second visual information according to the second visual feature and the visual feature database; Loading the three-dimensional field on the second visual information according to a mapping relationship between a real-time pose of the second visual information and a corresponding three-dimensional scene model in the three-dimensional map data Scene model. The device according to claim 15, wherein the positioning unit is specifically configured to: Performing feature matching on the second visual feature according to the visual feature library; Performing a perspective N-point problem PnP pose calculation on the feature-matched second visual feature to obtain a real-time pose of the second visual information. The apparatus according to claim 11, wherein said first visual information comprises third visual information and fourth visual information; The generating unit is specifically configured to: Performing a three-dimensional scene reconstruction on the third visual information and the corresponding depth information to generate a three-dimensional dense point cloud, or performing three-dimensional scene reconstruction on the depth information to generate the three-dimensional dense point cloud; Performing a three-dimensional scene reconstruction on the fourth visual information to generate a visual instant location and a map construction vSLAM sparse point cloud; Aligning the three-dimensional dense point cloud with the vSLAM sparse point cloud to reconstruct the three-dimensional scene model. The device according to claim 17, wherein the generating unit is specifically configured to: Extracting a point cloud feature from the vSLAM sparse point cloud to obtain a vSLAM sparse point cloud feature, and extracting a point cloud feature from the three-dimensional dense point cloud to obtain a three-dimensional dense point cloud feature; Performing a three-dimensional point cloud registration calculation on the vSLAM sparse point cloud feature and the three-dimensional dense point cloud feature to obtain a conversion relationship between the vSLAM sparse point cloud and the three-dimensional dense point cloud; And aligning the vSLAM sparse point cloud with the three-dimensional dense point cloud by three-dimensional dense point registration according to the conversion relationship between the vSLAM sparse point cloud and the three-dimensional dense point cloud to generate a new three-dimensional point cloud as the three-dimensional scene model. The device according to claim 17, wherein the extracting unit is specifically configured to: Feature extraction is performed on the second visual information to obtain a vSLAM visual feature. The device according to claim 19, wherein the vSLAM visual feature includes a real-time pose of the second visual information; The positioning unit is specifically configured to: And loading the three-dimensional scene model on the second visual information according to a mapping relationship between a real-time pose of the second visual information and a corresponding three-dimensional scene model in the three-dimensional map data. A computer storage medium for storing computer software instructions for use in a three-dimensional scene location device, comprising program code designed to perform the three-dimensional scene location method of any of claims 1-10. A computer program product, which can be directly loaded into an internal memory of a computer and containing software code, which can be loaded and executed by a computer to implement the method of any one of claims 1-10 3D scene positioning method. A three-dimensional scene positioning apparatus, comprising: a memory, a communication interface, and a processor, wherein the memory is used to store computer execution code, and the processor is configured to execute the computer to execute code control execution in claims 1-10 In any one of the three-dimensional scene positioning methods, the communication interface is used for data transmission between a three-dimensional scene positioning device and an external device.

Images