WO2020034086A1

WO2020034086A1 - Three-dimensional reconstruction method and apparatus for scene, and electronic device and storage medium

Info

Publication number: WO2020034086A1
Application number: PCT/CN2018/100390
Authority: WO
Inventors: 王恺; 廉士国
Original assignee: 深圳前海达闼云端智能科技有限公司
Priority date: 2018-08-14
Filing date: 2018-08-14
Publication date: 2020-02-20
Also published as: CN109155846A; CN109155846B

Abstract

A three-dimensional reconstruction method and apparatus for a scene, and an electronic device and a storage medium. The method is applied to an electronic device or a cloud, and comprises: acquiring image data of a currently photographed region (101); according to the image data, determining first position information of the currently photographed region (102); according to the first position information and position information of a first sub-space corresponding to a previously photographed region, dynamically adjusting data stored in a first memory (103); according to the adjusted data stored in the first memory, reconstructing a three-dimensional model of a second sub-space corresponding to the currently photographed region (104); and performing rendering according to the three-dimensional reconstruction data of the second sub-space, wherein the first sub-space and second sub-space are respectively one of N sub-spaces obtained by dividing the scene, and N is an integer greater than zero (105). The three-dimensional reconstruction method for a scene can increase the speed of three-dimensional reconstruction and rendering, and enlarge the scene scope of three-dimensional reconstruction and rendering.

Description

Three-dimensional reconstruction method, device, electronic device and storage medium of scene

Technical field

The present application relates to the field of computer vision, and in particular, to a method, a device, an electronic device, and a storage medium for three-dimensional reconstruction of a scene.

Background technique

The robot needs to fully understand the scene it is in when performing operations such as navigation and obstacle avoidance. The three-dimensional reconstruction of the scene is one of the core technologies for the robot to fully understand the scene it is in. In order to ensure that the reconstructed area is an effective area and the accuracy of the reconstruction, the reconstruction work often needs to be performed manually or under the supervision of a person. This requires that the reconstruction of the scene can be completed in real time and can be rendered in real time for reference. .

technical problem

The inventors discovered during the research of the prior art that during the 3D reconstruction and rendering of the scene, the amount of data involved is constantly increasing, the consumption of memory and video memory is also increasing, and related computing resources are constantly occupied, resulting in the speed of 3D reconstruction Slower and slower, scene reconstruction and rendering delay, real-time reconstruction and rendering can not be achieved; and after the reconstruction reaches a certain range, such as: the scope of a room, reconstruction and rendering can not continue, limiting the scope of the scene to be reconstructed .

It can be seen that how to increase the speed of 3D reconstruction and rendering, and expand the scope of the reconstructed and rendered scenes, are problems that need to be solved.

Technical solutions

A technical problem to be solved in some embodiments of the present application is to provide a method for 3D reconstruction and rendering of a scene, so that the speed of 3D reconstruction and rendering can be improved, and the scope of the scene for 3D reconstruction and rendering can be expanded.

An embodiment of the present application provides a method for three-dimensional reconstruction of a scene, including: acquiring image data of a current shooting area; determining first position information of the current shooting area according to the image data; according to the first position information and a previous shooting area According to the position information of the corresponding first subspace, the data stored in the first memory is dynamically adjusted; according to the data stored in the adjusted first memory, a three-dimensional model reconstruction is performed on the second subspace corresponding to the current shooting area, and according to the first The 3D reconstruction data of the two subspaces are rendered; wherein the first subspace and the second subspace are N obtained by dividing the scene, respectively. One of the subspaces, N Is greater than 0 Integer.

An embodiment of the present application further provides a scene scanning device, including: an acquisition module, a first position information determination module, an adjustment module, a 3D model reconstruction module, and a 3D model rendering module; an acquisition module configured to acquire a current shooting area Image data; a first position information determining module for determining the first position information of the current shooting area based on the image data; an adjusting module for determining the position of the first subspace corresponding to the previous shooting area based on the first position information Information to dynamically adjust the data stored in the first memory; a three-dimensional model reconstruction module for reconstructing a three-dimensional model of the second subspace corresponding to the current shooting area according to the data stored in the adjusted first memory; For rendering according to the 3D reconstruction data of the second subspace, where the first subspace and the second subspace are N obtained by dividing the scene One of the subspaces, N Is greater than 0 Integer.

An embodiment of the present application further provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are at least A processor executes to enable at least one processor to perform the three-dimensional reconstruction method of the scene described above.

The embodiment of the present application further provides a computer-readable storage medium storing a computer program, and the computer program is executed by a processor to implement the foregoing three-dimensional reconstruction method of a scene.

Beneficial effect

Compared with the prior art, in some embodiments of the present application, the data in the first memory is dynamically adjusted according to the first position information of the current shooting area and the position information of the first subspace corresponding to the previous shooting area to ensure that The first memory has enough space to calculate the current shooting area, and also ensures that the first memory has enough space to calculate the image data of the next shooting area without the problem of calculation delay caused by the large amount of data. It will not cause the problem that the reconstruction calculation and rendering of the scene cannot be continued due to the large scope of the scene, and is suitable for the reconstruction and rendering of scenes of various sizes. In addition, because the scene is divided into subspaces, and the scene is 3D reconstructed and rendered in the form of subspaces, each subspace can be independently 3D reconstructed and rendered, avoiding the large amount of data that is reconstructed and rendered each time. This leads to the problem of slow 3D reconstruction and rendering of the scene.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments are exemplified by the pictures in the accompanying drawings. These exemplary descriptions do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the drawings in the drawings do not constitute a limitation on scale.

figure 1 It is a schematic flowchart of a three-dimensional reconstruction method of a scene in the first embodiment of the present application;

figure 2 It is a schematic diagram of scene division in the first embodiment of the present application;

image 3 Is the calculation subspace in the first embodiment of the present application A With subspace D Diagram of distance

Figure 4 It is a schematic flowchart of a three-dimensional reconstruction method of a scene in the second embodiment of the present application;

Figure 5 It is a schematic flowchart of a three-dimensional reconstruction method of a scene in the third embodiment of the present application;

Figure 6 Is a schematic diagram between adjacent subspaces in a three-dimensional reconstruction method of a scene in a third embodiment of the present application

Figure 7 Is a schematic structural diagram of a three-dimensional reconstruction device for a scene in a fourth embodiment of the present application;

Figure 8 It is a schematic structural diagram of an electronic device in the fifth embodiment of the present application.

Embodiments of the invention

In order to make the purpose, technical solution, and advantages of the present application clearer, some embodiments of the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application. However, a person of ordinary skill in the art can understand that in the embodiments of the present application, many technical details are provided in order to make the reader better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solution claimed in this application can be implemented.

The first embodiment of the present application relates to a method for three-dimensional reconstruction of a scene. The method for three-dimensional reconstruction of a scene may be applied to an electronic device or a cloud. The electronic device may be a smart robot, a driverless car, or the like. The cloud communicates with electronic devices to provide the terminal with scan results of the scene. This embodiment is described by using an electronic device to perform a three-dimensional reconstruction method of the scene as an example. For a process of performing the three-dimensional reconstruction method of the scene in the cloud, refer to the content of the embodiment of the present application. The specific process of the 3D reconstruction method of this scene is shown in Figure 1. As shown.

The scene in this embodiment takes a large-scale scene as an example, such as several rooms, a floor, a gymnasium, and the like.

Step 101 : Get the image data of the current shooting area.

Specifically, the terminal can collect image data of the current shooting area through one sensor or multiple sensors. For example, red (blue green) , RGB ) Camera and depth camera, current RGB The camera and depth camera need to be aligned before capturing image data, which can capture the color and depth of the current shooting area red green blue depth , RGBD )data.

It should be noted that, before acquiring the image data of the current shooting area, the electronic device may first divide the scene in which the current shooting area is located into multiple subspaces, and the process of dividing the subspace is as follows: obtaining volume data of the scene in which the current shooting area is located; Divide the scene into N based on volume data Subspaces, N Is greater than 0 Integer.

Specifically, the volume of the scene in which the current shooting area is located can be obtained by manual input, or the volume data can be obtained through the cloud. In this embodiment, the manner of obtaining the volume data of the scene is not limited. The number of subspaces can be determined in advance, or any number of subspaces can be used. The size and shape of each subspace can be the same. Of course, N is divided After each subspace, the size of the subspace can also be dynamically adjusted according to the data in each subspace, and the size and shape of each subspace can be different after the adjustment.

It can be understood that if each subspace is the same size and is a cuboid, then each subspace is at most 6 The subspaces are adjacent.

In a specific implementation, the scene can be divided according to the data structure of the octree. The division process is as follows: According to the volume data of the scene where the current shooting area is located, and according to the preset maximum recursion depth of the octree, the current shooting area is located. The scene is divided into N Subspaces, and N Each subspace corresponds to each child node in each level of recursion depth.

Specifically, the maximum recursion depth of the octree can be set in advance according to the obtained volume data of the scene. The scene is divided into spaces using the data structure of the octree, and each of the divided subspaces corresponds to a node in the octree. The following uses a specific example to illustrate the division of the scene's subspace and the process corresponding to the nodes in the octree:

Take a room as an example, as shown in Figure 2 As shown, the scene is divided into a cube S Indicates that the preset recursion depth is 2 , The process of dividing according to a preset recursion depth is: the cube S be divided into 8 Subspace, graph 2 The parent node of the octree is A , The first-level node B0 ~ B7 With child cubes S0 ~ S7 Correspondence; respectively S0 to S7 Divide, such as a subcube S0 Can be divided into 8 Subspaces ( S01 ~ S08 ), And S01 ~ S07 Respectively with the child nodes in the second level C0 ~ C7 Correspondingly, the remaining child nodes of the second level ( S1 ~ S7 The corresponding subspace can be divided in this way. In this example, the volume of the subspace corresponding to each level of child nodes of the octree is the same. Of course, in practical applications, the subspace corresponding to each level of child nodes The volume can be different.

Corresponds the subnodes of the octree with the divided subspaces, so that during the three-dimensional reconstruction of the scene according to the subspaces, the corresponding relationship between the nodes and the subspaces can be used to obtain the phase corresponding to the current shooting area. Data in neighboring subspaces.

It should be noted that in the process of dividing the subspace of the scene, each time a subspace is divided, the corresponding space position is marked for the subspace. After the subspace division of the scene is completed, each subspace can be determined. Location information and size of the space.

Step 102 : Determine the first position information of the current shooting area based on the image data.

In a specific implementation, the electronic device constructs the point cloud or grid data corresponding to the current shooting area according to the image data; obtains the position information of the point cloud or grid data, and determines based on the position information of the point cloud or grid data. The first position information of the current shooting area.

Specifically, the electronic device can be based on the RGBD in the image data For the data, the point cloud data or grid data corresponding to the image data is calculated through a matrix transformation formula. Since the point cloud data contains multiple points, the position information in the middle of the point cloud can be used as the first position information of the point cloud data, and the center position of the grid can be used as the first position information of the grid data.

Step 103 : Dynamically adjust the data stored in the first memory according to the first position information and the position information of the first subspace corresponding to the previous shooting area.

In a specific implementation, the electronic device determines an adjustment mode according to the first position information and the position information of the first subspace corresponding to the previous shooting area; according to the adjustment mode, the data stored in the first memory is adjusted. The adjustment mode is a first adjustment mode, a second adjustment mode, or a third adjustment mode. The first adjustment mode is to save and delete data in the first memory, read data in the second subspace corresponding to the current shooting area from the second memory, and add the addition determined by the image data in the first memory. Data; the second adjustment mode is to read the data of the second subspace corresponding to the current shooting area from the second memory, and add the addition data determined by the image data in the first memory; the third adjustment mode is to Addition data determined by image data is added to the memory. The added data may be point cloud or grid data corresponding to the current shooting area.

Those skilled in the art can understand that the adjustment modes may also be other modes, which are not listed here one by one.

It should be noted that the first subspace and the second subspace are N obtained by dividing the scene, respectively. One of the subspaces, N Is greater than 0 Integer.

Specifically, according to the first position information and the position information of the first subspace corresponding to the previous shooting area, or according to the first position information and the position information of all the subspaces, determining the second subspace corresponding to the current shooting area; Determine the positional relationship between the first subspace and the second subspace according to the position information of the first subspace and the position information of the second subspace; and determine the adjustment mode according to the position relationship.

The following describes the two methods of determining the second subspace corresponding to the current shooting area:

method 1 : Determine whether the first position information is within the range of the position information of the first subspace, if yes, use the first subspace as the second subspace corresponding to the current shooting area; if not, use the Position information of other subspaces determines a second subspace corresponding to the current shooting area.

Specifically, the first subspace is the subspace corresponding to the previous shooting area. The position information of the first subspace includes the positions of the vertices of the first subspace. The range formed by the vertices of the first subspace is taken as The range of the position information of the first subspace. Determine whether the first position information is within the range of the position information of the first subspace, and if so, the current shooting area is located in the first subspace, and the first subspace is used as the second subspace corresponding to the current shooting area, that is, the previous The first subspace corresponding to the shooting area and the second subspace corresponding to the current shooting area are the same subspace. If the first position information is not within the range of the position of the first subspace, it indicates that the second subspace corresponding to the current shooting area and the first subspace corresponding to the previous shooting area are not the same subspace. Whether the position information of each subspace except the first subspace in the scene includes the first position information, and the subspace containing the first position information is selected as the second subspace.

Method 2 : Determine whether the first position information is included in the range of the position information of each subspace, and determine the second subspace corresponding to the current shooting area according to the determination result.

Specifically, this method and method 1 Similarly, the difference lies only in the method 2 According to the position information of all the subspaces, a subspace containing the first position information is searched, and the subspace containing the first position information is used as the second subspace.

The method for determining the positional relationship between the first subspace and the second subspace based on the position information of the first subspace and the position information of the second subspace is described below as an example.

In a specific implementation, after the second subspace is determined, the distance between the first subspace and the second subspace is calculated according to the position information of the first subspace and the position information of the second subspace; if it is determined that the distance is greater than If the preset distance threshold is determined, the positional relationship between the first subspace and the second subspace is determined to be non-adjacent; if the determined distance is less than the preset distance threshold and the distance is not zero, the relationship between the first subspace and the second subspace is determined The positional relationship is adjacent; if the determined distance is zero, it is determined that the positional relationship of the first subspace and the second subspace is the same position.

In this embodiment, the distance between the geometric center point of the first subspace and the geometric center point of the second subspace is used as the distance between the first subspace and the second subspace. The preset distance threshold can be set according to the actual application, where the preset distance threshold is related to the number of reconstruction points in the point cloud data (the number of grids in the grid data), the size of the subspace, and The size is related. For example, if the capacity of the first memory is 1G If the capacity is small, the preset distance threshold can be set smaller.

There are many ways to calculate the distance between the first subspace and the second subspace. This embodiment specifically introduces a method for calculating the distance between the first subspace and the second subspace by using the geodesic distance.

Specifically, the geometric center points of adjacent subspaces are connected by straight lines to form a connection graph of all subspaces, that is, the geometric centerline point of each subspace is a connection point in the connection graph. The first subspace and The distance between the second subspaces is the shortest distance between the two corresponding connection points in the connection graph (that is, the number of edges on the shortest path connecting the connection points).

For example, there are 4 Subspaces are A , B , C with D ,among them, A Is the first subspace, D Is the second subspace, AB Adjacent, BC Adjacent, CD Adjacent, AC Adjacent. Connect the geometric centers of adjacent subspaces with straight lines to form a connection graph of all subspaces, as shown in the figure 3 As shown, the hollow circles in the figure represent the connection points (Figure 3 In 4 Connection points A ,Junction B ,Junction C And connection points D ), Then the distance between the first subspace and the second subspace is the connection point A And connection points D Shortest distance between (shortest distance is A-C-D Connection point A And connection points D The number of edges on the shortest path of 2 ), Where the shortest distance is the geodesic distance.

The method for determining the adjustment mode according to the positional relationship between the first subspace and the second subspace is described below as an example.

In a specific implementation, if it is determined that the positional relationship between the first subspace and the second subspace is not adjacent, then the adjustment mode is determined as the first adjustment mode; if the positional relationship between the first subspace and the second subspace is If they are adjacent, the adjustment mode is determined to be the second adjustment mode; if the positional relationship between the first subspace and the second subspace is determined to be the same position, the adjustment mode is determined to be the third adjustment mode.

Specifically, if it is determined that the first subspace and the second subspace are not in the same position, it indicates that the first subspace and the second subspace where the current shooting area is located are not the same subspace. In order to ensure the computing capacity of the first memory, To adjust the data stored in the first memory.

Specifically, if it is determined that the positions of the first subspace and the second subspace are adjacent, the data in the first subspace in the first memory is not processed, and according to the position information of the second subspace, The memory searches for data in the second subspace, reads the data into the first memory, and adds point cloud or grid data to the data contained in the second subspace in the first memory. If it is determined that the positions of the first subspace and the second subspace are not adjacent, first save the data in the first subspace to the second memory, and simultaneously save the position information of the first subspace, and then save the first subspace. The data in the subspace is deleted from the first memory, the data in the second subspace that is queried is loaded, and the point cloud or grid data is added to the data contained in the second subspace in the first memory.

It should be noted that after adding the added data (point cloud data or grid data) to the first storage, the subspace corresponding to the nodes in the octree can be updated. Specifically, when new data is added to the second subspace, the second subspace is divided again according to the principle of the octree data structure, and the correspondence between the octree nodes and the newly divided subspace is adjusted. relationship.

Dividing the second subspace through the added data can reduce the volume of the subspace, reduce the amount of data in the divided subspace, and further adjust the computing capacity of the first memory for the subspace. In addition, the corresponding relationship between the octree nodes is adjusted so that in the subsequent 3D reconstruction and rendering of the scene, the data in the adjacent nodes can be quickly queried through the octree nodes.

Step 104 : Perform a three-dimensional model reconstruction on the second subspace corresponding to the current shooting area according to the data stored in the adjusted first memory.

Specifically, after the data is added, a three-dimensional model of the second subspace is constructed according to point cloud data or grid data in the second subspace.

Step 105 : Rendering based on the 3D reconstruction data of the second subspace.

It should be noted that the 3D reconstruction data of the second subspace may be the 3D grid data of the second subspace or the 3D point cloud data of the second subspace, or the 3D grid data of the second subspace and the second subspace. 3D point cloud data in space. The first memory in this embodiment includes a memory for performing three-dimensional reconstruction, and a video memory for rendering.

Compared with the prior art, in some embodiments of the present application, the data in the first memory is dynamically adjusted according to the first position information of the current shooting area and the position information of the first subspace corresponding to the previous shooting area to ensure that The first memory has enough space to calculate the current shooting area, and also ensures that the first memory has enough space to perform three-dimensional reconstruction of the image data of the next shooting area without the problem of calculation delay caused by the large amount of data. At the same time, the reconstruction and calculation of the scene cannot be continued due to the large scope of the scene, which is suitable for the reconstruction and rendering of scenes of various sizes. In addition, because the scene is divided into subspaces, and the scene is 3D reconstructed and rendered in the form of subspaces, each subspace can be independently 3D reconstructed and rendered, avoiding the large amount of data that is reconstructed and rendered each time. As a result, the 3D reconstruction and rendering of the scene are slow.

The second embodiment of the present application relates to a method for three-dimensional reconstruction of a scene. The second embodiment is a further improvement on the first embodiment. The main improvement is that, after the three-dimensional reconstruction and rendering of all subspaces are completed in this embodiment , Stitch all the subspace data in the scene.

This embodiment includes step 401 To step 407 . Where steps 401 To step 405 Separate from the steps in the first embodiment 101 To step 105 Roughly the same, no longer detailed here, the following mainly introduces the differences:

Step 406 : Detect whether the data in all the subspaces in the scene are contained in the second memory; if so, execute the step 407 , Otherwise return to execute step 401 .

Specifically, the number of subspaces divided by the scene can be used to determine whether the number of subspaces contained in the second memory is the same as the number of subspaces divided by the scene. If they are the same, it is determined whether the second memory contains Data in all subspaces in the scene; otherwise, it is determined that data in all subspaces in the scene are not included in the second memory. It can be understood that there may be other detection methods, for example, a method of determining whether the second memory includes the position information of all the subspaces by using the position information of each subspace divided by the scene is not provided here one by one. List.

Step 407 : According to the position information of each subspace, the data in all the subspaces in the second memory are stitched to form the 3D reconstructed data of the scene, and the 3D reconstructed data of the scene is rendered.

Specifically, the second memory may be a read-only memory, and the data in all subspaces is the point cloud or grid data of the scene after splicing, forming the 3D reconstruction data of the scene, and performing the 3D reconstruction data of the scene. A 3D model of the scene can be obtained by rendering. It can be understood that the stitching is performed in the first memory.

Compared with the prior art, the three-dimensional rendering method of the scene provided by this embodiment, because the data of each subspace is stored in the second memory, and each subspace is relatively independent, the data in each subspace can be simply processed. Fusion, small amount of calculation, and fast speed. Through stitching, the entire scene can be quickly rendered, which is suitable for 3D reconstruction and rendering of scenes of various sizes.

The third embodiment of the present application relates to a three-dimensional rendering method of a scene. The third embodiment is a further improvement on the second embodiment. The main improvement lies in that this embodiment is based on the three-dimensional reconstruction of the second subspace. Before rendering the data, the volume of the second subspace is adjusted according to the number of point cloud data or grid data in the second subspace. The specific process is shown in Figure 5 As shown.

This embodiment includes step 501 To step 508 . Where steps 501 To step 504 ,step 506 To step 508 With the steps in the second embodiment 401 To step 404 ,step 405 To step 407 Roughly the same, no longer detailed here, the following mainly introduces the differences:

Step 505 : Adjust the volume of the second subspace according to the number of point cloud data or grid data in the second subspace.

In a specific implementation, it is determined whether the point cloud or grid data in the second subspace exceeds the first preset value, and if so, the second subspace is divided into at least one subspace.

It should be noted that the division manner of the second subspace is the same as that in the first embodiment. For example, divide the second subspace into 8 Subspace, and then use the subspace where the first position information of the current shooting area is located as a new second subspace.

It is worth mentioning that when there is too much point cloud data in the second subspace, the volume of the second subspace is adjusted, which speeds up the calculation of the second subspace in the first memory.

In another specific implementation, it is determined whether the point cloud or grid data in the second subspace and the subspace adjacent to the second subspace are both smaller than the second preset value, and if so, will be adjacent to the second subspace. And the second subspace are merged.

For example, subspace A Is the second subspace, subspace A With subspace B Adjacent when the second subspace and subspace are detected B The point cloud data or grid data in the second subspace and the subspace B Merge to form a new second subspace.

The first preset threshold and the second preset threshold are set according to the computing capability of the first memory in practical applications.

It should be noted that there is a partially overlapping space between two adjacent subspaces. Take a specific example to illustrate, for example, as shown in Figure 6 Shown, subspace A And subspace AB Adjacent space AB With subspace B Adjacent space AB With subspace A There are partial overlaps at the same time AB With subspace B There is also some overlap. Since the subspace AB Subspace A Part of the subspace B Part of the data to ensure the coherence of the subspace data. It can be understood that when performing a merge operation of multiple adjacent subspaces, redundant data may appear in overlapping portions of adjacent subspaces, and the redundant data may be deleted. For example, as shown in the figure 6 As shown, if the subspace A And subspace B Merge, subspace AB Delete the data inside.

Compared with the prior art, the method for 3D reconstruction of a scene provided by this embodiment can automatically adjust the volume of a subspace according to the amount of data in the subspace, thereby ensuring the speed of 3D reconstruction and rendering of the subspace, while also avoiding Because of the small amount of data, the computing resources are wasted.

The fourth embodiment of the present application relates to a three-dimensional reconstruction device 70 for a scene , Including: Get module 701 First location information determination module 702 Adjustment module 703 3D model reconstruction module 704 And 3D model rendering module 705 ; The specific structure is shown in Figure 7 As shown.

Acquisition module 701 For acquiring image data of a current shooting area; a first position information determining module 702 For determining the first position information of the current shooting area according to the image data; the adjustment module 703 For dynamically adjusting data stored in the first memory according to the first position information and the position information of the first subspace corresponding to the previous shooting area; a three-dimensional model reconstruction module 704 For reconstructing a three-dimensional model of a second subspace corresponding to the current shooting area according to the data stored in the adjusted first memory; a three-dimensional model rendering module 705 For rendering according to the 3D reconstruction data of the second subspace; wherein the first subspace and the second subspace are obtained by dividing the scene respectively N One of the subspaces, N Is greater than 0 Integer.

This embodiment is an embodiment of a virtual device corresponding to the three-dimensional reconstruction method of the foregoing scene. The technical details in the foregoing method embodiments are still applicable in this embodiment, and are not described herein again.

It should be noted that the device embodiments described above are only schematic and do not limit the scope of protection of this application. In practical applications, those skilled in the art may select some or all of the modules according to actual needs. To achieve the purpose of the solution of this embodiment, there is no limitation here.

The fifth embodiment of the present application relates to an electronic device, whose structure is shown in FIG. 8 As shown. Includes: at least one processor 801 ; And, with at least one processor 801 Communication connected storage 802 . Memory 802 Stored by at least one processor 801 Instruction executed. Instruction by at least one processor 801 Execute to make at least one processor 801 The three-dimensional reconstruction method of the scene described above can be performed.

In this embodiment, the processor is a central processing unit (Central Processing Unit). , CPU ) As an example, the memory is a readable and writable memory ( Random Access Memory , RAM ) As an example. The processor and memory can be connected through the bus or other methods. 8 Take the connection via the bus as an example. As a non-volatile computer-readable storage medium, the memory can be used to store non-volatile software programs, non-volatile computer executable programs, and modules. The processor executes various functional applications and data processing of the device by running non-volatile software programs, instructions, and modules stored in the memory, that is, a three-dimensional reconstruction method of the foregoing scene.

The memory may include a storage program area and a storage data area, wherein the storage program area may store an operating system and an application program required for at least one function; the storage data area may store a list of options and the like. In addition, the memory may include a high-speed random access memory, and may further include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely set with respect to the processor, and these remote memories may be connected to an external device through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

One or more modules are stored in the memory, and when executed by one or more processors, execute a method for generating a set of sample images in any of the foregoing method embodiments.

The above-mentioned products can execute the three-dimensional reconstruction method of the scene provided by the embodiment of the present application, and have corresponding functional modules and beneficial effects of the execution method. For technical details not described in this embodiment, refer to the scene provided by the embodiment of the present application. 3D reconstruction method.

The sixth embodiment of the present application relates to a computer-readable storage medium. The readable storage medium is a computer-readable storage medium. The computer-readable storage medium stores computer instructions that enable a computer to execute the first application of the present application. A three-dimensional reconstruction method of a scene involved in any one of the first to third method embodiments.

That is, those skilled in the art can understand that all or part of the steps in the method of the above embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions for making a device Can be a microcontroller, chip, etc.) or a processor (processor ) Perform all or part of the steps of the method described in each embodiment of this application. The foregoing storage media include: U Disk, mobile hard disk, read-only memory ( ROM , Read-Only Memory ), Random access memory ( RAM , Random Access Memory ), Magnetic disks, or compact discs, which can store program code.

Those of ordinary skill in the art can understand that the foregoing embodiments are specific embodiments for implementing the present application, and in practical applications, various changes can be made in form and details without departing from the spirit and range.

Claims

A three-dimensional reconstruction method of a scene, including:

Obtain image data of the current shooting area;

Determining, according to the image data, first position information of the current shooting area;

Dynamically adjusting data stored in the first memory according to the first position information and the position information of the first subspace corresponding to the previous shooting area;

Performing a three-dimensional model reconstruction on the second subspace corresponding to the current shooting area according to the adjusted data stored in the first memory, and rendering according to the three-dimensional reconstruction data of the second subspace;

The first subspace and the second subspace are respectively one of N subspaces obtained by dividing the scene, and N is an integer greater than 0.
The method for three-dimensional reconstruction of a scene according to claim 1, wherein dynamically adjusting data stored in the first memory according to the first position information and position information of a first subspace corresponding to a previous shooting area specifically includes:

Determining an adjustment mode according to the first position information and position information of a first subspace corresponding to a previous shooting area;

Adjusting data stored in the first memory according to the adjustment mode;

The adjustment mode is a first adjustment mode, a second adjustment mode, or a third adjustment mode;

The first adjustment mode is to save and delete data in the first memory, read data in a second subspace corresponding to the current shooting area from a second memory, and save the data in the first memory. Adding additional data determined by the image data in a memory;

The second adjustment mode is to read data in a second subspace corresponding to the current shooting area from a second memory, and add additional data determined by the image data in the first memory;

The third adjustment mode is to add addition data determined by the image data to the first memory.
The method for three-dimensional reconstruction of a scene according to claim 1 or 2, wherein before acquiring the image data of the current shooting area, the method for three-dimensional reconstruction of the scene further comprises:

Acquiring volume data of a scene in which the current shooting area is located;

According to the volume data, the scene is divided into the N subspaces.
The method for three-dimensional reconstruction of a scene according to claim 3, wherein dividing the scene into the N subspaces according to the volume data specifically includes:

Divide the scene in which the current shooting area is located into the N subspaces according to the preset maximum recursive depth of the octree and the volume data, and the N subspaces are each associated with each level of recursive depth Corresponds to child nodes.
The method for three-dimensional reconstruction of a scene according to claim 2, wherein determining the first position information of the current shooting area based on the image data specifically includes:

Constructing point cloud or grid data corresponding to the current shooting area according to the image data;

Acquiring position information of the point cloud or grid data, and determining first position information of the current shooting area according to the position information of the point cloud or grid data.
The method for three-dimensional reconstruction of a scene according to claim 2, wherein determining an adjustment mode according to the first position information and position information of a first subspace corresponding to a previous shooting area specifically includes:

Determine the second sub-corresponding to the current shooting area according to the first position information and the position information of the first sub-space corresponding to the previous shooting area, or according to the first position information and the position information of all sub-spaces space;

Determining a positional relationship between the first subspace and the second subspace according to the position information of the first subspace and the position information of the second subspace;

Determining the adjustment mode according to the position relationship.
The method for three-dimensional reconstruction of a scene according to claim 6, wherein according to the first position information and the position information of the first subspace corresponding to the previous shooting area, or according to the first position information and all the subspaces The position information of the second subspace corresponding to the current shooting area specifically includes:

Determine whether the first position information is within the range of the position information of the first subspace, if yes, use the first subspace as the second subspace corresponding to the current shooting area; if not, use the first subspace Position information of other subspaces outside the space, and determining a second subspace corresponding to the current shooting area;

or,

It is respectively determined whether the position information of each subspace contains the first position information, and the second subspace corresponding to the current shooting area is determined according to the determination result.
The method for three-dimensional reconstruction of a scene according to claim 6 or 7, wherein the first subspace and the second subspace are determined according to position information of the first subspace and position information of the second subspace. Location relationship, including:

Calculating the distance between the first subspace and the second subspace according to the position information of the first subspace and the position information of the second subspace;

If it is determined that the distance is greater than a preset distance threshold, determining that the positional relationship between the first subspace and the second subspace is not adjacent;

If it is determined that the distance is less than the preset distance threshold and the distance is not zero, determining that the positional relationship between the first subspace and the second subspace is adjacent;

If it is determined that the distance is zero, it is determined that the positional relationship between the first subspace and the second subspace is the same position.
The method for three-dimensional reconstruction of a scene according to claim 8, wherein determining an adjustment mode according to the positional relationship specifically includes:

If it is determined that the positional relationship is not adjacent, determining that the adjustment mode is the first adjustment mode;

If it is determined that the positional relationship is adjacent, determining that the adjustment mode is the second adjustment mode;

If it is determined that the position relationship is the same position, it is determined that the adjustment mode is the third adjustment mode.
The method of three-dimensional reconstruction of a scene according to claim 5, wherein the added data is point cloud or grid data corresponding to the current shooting area.
The method of three-dimensional reconstruction of a scene according to claim 2, wherein after rendering according to the reconstructed three-dimensional reconstruction data, the method of three-dimensional reconstruction of the scene further comprises:

Detecting whether the second memory contains data in all subspaces in the scene;

If so, the data in all the subspaces in the second memory are stitched according to the position information of each subspace to form the three-dimensional reconstruction data of the scene, and the three-dimensional reconstruction data of the scene is rendered.
The method for three-dimensional reconstruction of a scene according to any one of claims 5 to 11, before performing rendering based on the three-dimensional reconstruction data of the second subspace, further comprising:

Judging whether the point cloud or grid data in the second subspace exceeds the first preset value, and if so, dividing the second subspace into at least one subspace;

Judging whether the point cloud or grid data in the second subspace and the subspace adjacent to the second subspace is smaller than the second preset value, and if it is, will be compared with the second subspace Adjacent subspaces are merged with the second subspace.
The method for three-dimensional reconstruction of a scene according to any one of claims 1 to 12, wherein two adjacent subspaces have a partially overlapping space.
A scene 3D reconstruction device, comprising: an acquisition module, a first position information determination module, an adjustment module, a 3D model reconstruction module, and a 3D model rendering module;

The acquisition module is configured to acquire image data of a current shooting area;

The first position information determining module is configured to determine first position information of the current shooting area according to the image data;

The adjusting module is configured to dynamically adjust data stored in the first memory according to the first position information and the position information of the first subspace corresponding to the previous shooting area;

The three-dimensional model reconstruction module is configured to perform three-dimensional model reconstruction on a second subspace corresponding to the current shooting area according to the adjusted data stored in the first memory;

The three-dimensional model rendering module is configured to render according to three-dimensional reconstruction data of the second subspace;

The first subspace and the second subspace are respectively one of N subspaces obtained by dividing the scene, and N is an integer greater than 0.
An electronic device including:

At least one processor; and

A memory connected in communication with the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the method according to any one of claims 1 to 13. 3D reconstruction method of scene.
A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, a three-dimensional reconstruction method for a scene according to any one of claims 1 to 13 is implemented.