CN110223380A - Fusion is taken photo by plane and the scene modeling method of ground multi-view image, system, device - Google Patents

Abstract

The invention belongs to the field of scene modeling, and specifically relates to a scene modeling method, system, and device for integrating aerial photography and ground perspective images. The problem of imprecise fusion. The method of the present invention includes: S100, acquiring an aerial perspective image of an indoor scene to be modeled, and constructing an aerial map; S200, based on the aerial map, obtaining a composite image by synthesizing a ground perspective reference image from the aerial map; S300, Obtain a set of ground perspective images through the ground perspective images collected by the ground camera; S400, based on the synthesized image, fuse the aerial perspective images and the ground perspective images to obtain an indoor scene model. The present invention can generate a complete and accurate indoor scene model, takes into account both acquisition efficiency and reconstruction accuracy, and has strong robustness.

Description

Scene modeling method, system and device for fusing aerial and ground perspective images

technical field

The invention belongs to the field of scene modeling, and in particular relates to a scene modeling method, system and device for fusing aerial photography and ground perspective images.

Background technique

3D reconstruction of indoor scenes plays an important role in many real-world applications, such as indoor navigation, service robots, building information modeling (BIM, building information modeling), etc. Existing indoor scene reconstruction methods can be roughly divided into three categories: (1) methods based on LiDAR (light detection and ranging), (2) methods based on RGB-D cameras, and (3) methods based on images.

Although both the LiDAR-based method and the RGB-D camera-based method have high accuracy, when reconstructing large indoor scenes, the above two methods have problems such as high cost and poor scalability. For LiDAR-based methods, scene occlusion is unavoidable due to the limitation of scanning viewing angles, and multi-view laser scanning is often required to be aligned with the point cloud during scanning. For methods based on RGB-D cameras, due to the limited effective working distance of the sensor, a large amount of data needs to be collected and processed. Therefore, the above methods all have the disadvantages of high cost and low efficiency when performing large-scale indoor scene reconstruction.

Compared with LiDAR-based methods and RGB-D camera-based methods, although image-based methods are cheaper and more flexible, they also have some shortcomings, such as complex scenes, repetitive structures, lack of texture, etc. Incomplete and imprecise reconstruction results. Even with the most advanced structure from motion recovery (SfM, structure from motion) and multi-view stereo technology (MVS, multiple view stereo) technology, the reconstruction effect in large-scale and complex indoor scenes is still unsatisfactory. In addition, some image-based methods utilize some prior assumptions to deal with the indoor scene reconstruction problem, such as the Manhattan world assumption. Although these methods can sometimes achieve better results, they often lead to wrong reconstruction results when the prior assumptions are not met.

Contents of the invention

In order to solve the above-mentioned problems in the prior art, that is, in order to solve the problems of complex indoor scene structure, lack of texture, incomplete and inaccurate fusion of image-based modeling results, the first aspect of the present invention proposes a fusion of aerial photography and A scene modeling method for a ground perspective image, comprising the following steps:

Step S100, obtaining the aerial perspective image of the indoor scene to be modeled, and constructing an aerial map;

Step S200, based on the aerial map, obtain a synthesized image through the method of synthesizing a ground perspective reference image from the aerial map;

Step S300, obtaining a set of ground perspective images through the ground perspective images collected by the ground camera;

Step S400, based on the synthesized image, the aerial perspective image and the ground perspective image are fused to obtain an indoor scene model.

In some preferred implementations, in step S100 "acquire the aerial perspective image of the indoor scene to be modeled, and construct an aerial map", the method is:

For the aerial perspective video of the indoor scene, the image frame is extracted by using the adaptive video frame extraction method based on the bag-of-words model, and the aerial perspective image collection of the indoor scene is obtained;

Based on the set of aerial perspective images, an aerial map is constructed through an image modeling method.

In some preferred implementation manners, in step S200 "the method for synthesizing ground perspective reference images from aerial maps", the method is:

Based on the aerial map, calculate the virtual camera pose;

Through the graph cut algorithm, the aerial map is obtained to obtain the synthetic image of the reference image of the ground perspective;

In some preferred implementations, "acquire aerial maps to obtain synthetic images of ground perspective reference images through the graph cut algorithm", the method is:

Among them, E(l) is the energy function in the graph cut process; is a set of two-dimensional triangles projected from the three-dimensional space grid visible to the virtual camera, and t _i is the i-th triangle among them; is the set of common sides of the triangles in the projected two-dimensional triangle set; l _i is the serial number of the aerial image of t _i ; D _i (l _i ) is the data item; V _i (l _i , l _j ) is the smoothing item;

When the spatial patch corresponding to t _i is visible in the l _i aerial image, the data item Otherwise D _i (l _i )=α, where is the scale median value of local features in the l _i -th aerial image, is the projected area of the spatial patch corresponding to t _i in the l _i aerial image, and α is a relatively large constant;

When l _i =l _j , the smoothing term V _i (l _i , l _j )=0; otherwise V _i (l _j ,l _j )=1.

In some preferred implementation manners, in step S300, "obtain a set of ground perspective images through the ground perspective images collected by the ground camera", the method is:

Based on the planned path, the ground robot continuously collects ground perspective video through the ground camera set on it;

For the ground-view video of the indoor scene, the adaptive video frame extraction method based on the bag-of-words model is used to extract image frames, and the ground-view image collection of the indoor scene is obtained.

In some preferred implementations, in the process of "the ground robot continuously collects the ground perspective video through the ground camera set on it based on the planned path", its positioning method includes initial robot positioning and mobile robot positioning;

The initial robot positioning method is as follows: obtain the first frame of the video collected by the ground camera, obtain the initial position of the robot in the aerial map, and use this position as the starting point of the subsequent movement of the robot;

The positioning of the mobile robot is as follows: based on the initial position and the driving data of the robot at each time, the rough positioning of the robot position is performed, and by matching the video frame image collected at the current time with the composite image, the current position of the robot on the aerial map is obtained. The position in , and use this position to revise the position information of coarse positioning.

In some preferred implementation manners, step S400 "based on the synthesized image, fuse the aerial perspective image with the ground perspective image to obtain an indoor scene model", the method is:

Obtain the position of the ground camera corresponding to each image in the ground perspective image collection in the aerial map;

Connect the matching points of the ground perspective image and the synthetic image into the original aerial photography and ground feature point trajectory to generate cross-view constraints;

Optimize the aerial and ground image poses through bundle adjustment (BA, bundle adjustment);

Dense reconstruction of aerial and ground view images is used to obtain a dense model of the indoor scene.

In the second aspect of the present invention, a scene modeling system that integrates aerial photography and ground perspective images is proposed, the system includes an aerial photography map construction module, a synthetic image acquisition module, a perspective image collection acquisition module, and an indoor scene model acquisition module;

The aerial map construction module is configured to obtain an aerial perspective image of the indoor scene to be modeled, and construct an aerial map;

The synthetic image acquisition module is configured to obtain a synthetic image based on the aerial map by synthesizing a ground perspective reference image from the aerial map;

The viewing angle image set acquisition module is configured to acquire a ground viewing angle image set through ground viewing angle images collected by a ground camera;

The indoor scene model acquisition module is configured to fuse the aerial perspective image with the ground perspective image based on the synthesized image to acquire an indoor scene model.

In the third aspect of the present invention, a storage device is provided, in which a plurality of programs are stored, and the programs are adapted to be loaded and executed by a processor to implement the above-mentioned scene modeling method for fusing aerial photography and ground perspective images.

According to the fourth aspect of the present invention, a processing device is proposed, including a processor and a storage device; the processor is suitable for executing various programs; the storage device is suitable for storing multiple programs; and the program is suitable for being loaded by the processor And execute to realize the above-mentioned scene modeling method of fusing aerial photography and ground perspective images.

Beneficial effects of the present invention:

The invention constructs a three-dimensional aerial map to guide the robot to travel in the indoor scene and collects ground perspective images, then fuses the aerial and ground images, and generates a complete and accurate indoor scene model through the fused images. The indoor scene reconstruction process of the present invention takes both acquisition efficiency and reconstruction accuracy into consideration, and has strong robustness.

Description of drawings

Other characteristics, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

Fig. 1 is a schematic diagram of a flow chart of a scene modeling method for fusing aerial photography and ground perspective images according to an embodiment of the present invention;

Fig. 2 is an example diagram of an aerial map reconstructed through 271 extracted video frames in an embodiment of the present invention;

Fig. 3 is a schematic diagram of grid-based image synthesis in an embodiment of the present invention;

Fig. 4 is an exemplary diagram of the relationship between local feature scales and image clarity in an embodiment of the present invention;

Fig. 5 is an example diagram of image synthesis results based on graph cuts under different configurations in an embodiment of the present invention;

Figure 6 is a comparison of some other image synthesis results and an example diagram of ground images under similar viewing angles;

Fig. 7 is an example diagram of image matching results in an embodiment of the present invention;

Fig. 8 is a schematic diagram of searching for candidate matching composite images during robot movement in an embodiment of the present invention;

Fig. 9 is a schematic diagram of batch camera positioning process in an embodiment of the present invention;

Fig. 10 is an example diagram of batch camera positioning results based on three feature point trajectories in an embodiment of the present invention;

Fig. 11 is an example diagram of batch type camera positioning process in an embodiment of the present invention;

Fig. 12 is a schematic diagram of aerial photography and ground feature point trajectory generation for aerial photography views in an embodiment of the present invention;

Fig. 13 is the data acquisition equipment used in the test of an embodiment of the present invention;

Fig. 14 is an example aerial image in the Hall data set and an example diagram of the generated three-dimensional aerial map in the test of an embodiment of the present invention;

Fig. 15 is an example diagram of comparative experimental results between the frame drawing method of the present invention and the frame drawing method at equal intervals on the aerial video of the Hall data set in the test of an embodiment of the present invention;

Fig. 16 is an example diagram of qualitative comparison results of ground camera positioning in a test of an embodiment of the present invention;

Fig. 17 is an example diagram of qualitative results of indoor scene reconstruction in a test according to an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than Full examples. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

The application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain related inventions, rather than to limit the invention. It should also be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

Due to the complexity of indoor scenes, the following two issues need to be considered for image-based methods to achieve complete scene reconstruction. The first is the image acquisition process, i.e. how to acquire images to completely and efficiently cover the indoor scene. The second is the scene reconstruction algorithm, that is, how to fuse images from different perspectives in the process of SfM and MVS to obtain complete and accurate reconstruction results. In view of the above two problems, the present invention proposes a novel image-based indoor scene acquisition and reconstruction process. The process uses mini-aircraft and ground robots and includes four main steps (as shown in Figure 1): (1) Aerial map construction: a mini-aircraft is used to collect aerial perspective images indoors, and then the aerial perspective images are used to represent indoor scenes (2) Reference image synthesis: perform plane detection in the aerial map, obtain the ground plane and use it for path planning of the ground robot. Then, a number of ground perspective images are synthesized based on the aerial map for the positioning of the ground robot; (3) Ground robot positioning: the ground robot enters the indoor scene to collect the ground perspective images. While the robot is collecting images while moving, the positioning of the robot is realized by matching the collected images with the synthesized ground perspective images; (4) Indoor scene reconstruction: After the ground robot completes the image acquisition, the image-based modeling process The image of the mini-aircraft and the image of the ground robot are fused to realize the complete and accurate modeling of the indoor scene.

In the modeling process of the present invention, only the aerial image acquisition process requires manual operation, and the subsequent ground image acquisition and indoor scene modeling processes are all fully automatic, which means that the process of the present invention is highly expandable and suitable for large Acquisition and reconstruction of large-scale indoor scenes. Aerial images can also be collected automatically through autonomous navigation according to the acquired navigation path, but it increases the complexity of the algorithm, so manual operation is preferred to ensure the flexibility, integrity, and scalability of image acquisition.

Compared with ground images collected by ground robots, aerial images collected by mini-aircraft have a better viewing angle and larger field of view, which means that compared with ground images, the problems of occlusion and mismatch in aerial images will be smaller. Therefore, the maps generated from aerial images can be more reliably used in the subsequent ground robot positioning process.

The aerial images taken by the mini-aircraft complement the ground images taken by the ground robot and can completely cover the indoor scene. Therefore, by fusing aerial and ground images, a more complete and accurate indoor scene model can be obtained.

A scene modeling method for fusing aerial photography and ground perspective images of the present invention comprises the following steps:

Step S100, obtaining the aerial perspective image of the indoor scene to be modeled, and constructing an aerial map;

Step S200, based on the aerial map, obtain a synthesized image through the method of synthesizing a ground perspective reference image from the aerial map;

Step S300, obtaining a set of ground perspective images through the ground perspective images collected by the ground camera;

Step S400, based on the synthesized image, the aerial perspective image and the ground perspective image are fused to obtain an indoor scene model.

In order to more clearly describe the scene modeling method of the present invention that fuses aerial photography and ground perspective images, the steps in an embodiment of the inventive method will be described in detail below in conjunction with the accompanying drawings.

According to an embodiment of the present invention, the scene modeling method for fusing aerial photography and ground perspective images of the present invention includes steps S100-S400.

Step S100, acquiring an aerial perspective image of the indoor scene to be modeled, and constructing an aerial map.

First, a mini-aircraft is used to collect aerial video of indoor scenes, and some images are extracted from the video. Then, the extracted images are reconstructed through an image-based modeling process to obtain an aerial model, which is used as a three-dimensional map for ground robot positioning.

Step S101 , for the aerial view video of the indoor scene, image frames are extracted using an adaptive video frame extraction method based on the bag-of-words model, to obtain an aerial view image set of the indoor scene.

In this embodiment, a top-down aerial perspective video is collected by a mini-aircraft in an indoor scene, and the resolution of the collected video is 1080p, and the frame rate is 25FPS. Due to its small size and high flexibility, the mini drone is very suitable for shooting indoor scenes. As an example, the miniature aircraft used in this example is a DJI Spark equipped with a stabilizer and a 4K camera, and its weight is only 300g. In addition, compared with the ground perspective, shooting indoor scenes from an aerial perspective is not easily affected by scene occlusion, so using a mini aircraft can cover the scene more efficiently and completely.

Given the collected aerial video, an aerial map can be constructed by a SLAM (simultaneous localization and mapping) system. However, in this embodiment, offline SfM technology is used for aerial map construction. This is because: (1) in the embodiment, the aerial map is used for ground robot positioning, so there is no need for online construction; (2) compared with SLAM, which is prone to scene drift, SfM is more suitable for large-scale scene modeling. However, if SfM is used for aerial map construction, it is obviously not necessary to use all the frames in the aerial video. Because aerial video frames contain a lot of redundant information, this will seriously reduce the efficiency of SfM map construction. To solve the above problems, a direct way is to extract a frame at a fixed interval of frames in the video, and then use the extracted video frames for map construction. However, there are still some disadvantages in this approach: (1) It is difficult to achieve stable and constant-speed video acquisition in indoor scenes by manually operating a miniature aircraft, and this problem will become more difficult at the corner of the route; (2) due to Texture richness in indoor scenes is inconsistent, so uniform coverage of the scene is also inappropriate. In order to solve the above-mentioned problems existing in the aerial map construction process, a kind of adaptive video frame extraction method based on the bag of words (BoW, bag of words) model is adopted in the present embodiment, and its process is described in detail as follows:

In the BoW model, an image can be expressed as a normalized vector vi, and the similarity of a pair of images can be expressed by the point product of the corresponding vector express. As known to those skilled in the art, too high similarity between adjacent images will introduce too much redundant information, thereby reducing composition efficiency; and too low similarity between adjacent images will lead to connectivity between images Poor, incomplete composition. Therefore, in this embodiment, a method for adaptively extracting subsets from all video frames is proposed. When extracting frames, the method limits the similarity between each extracted video frame and its adjacent extracted video frames to within within an appropriate range. Specifically, the normalized vector v _i of each frame is first generated through the libvot library, and the first frame is used as a starting point. In the frame extraction process, assuming that the current i-th frame has been extracted, obtain the similarity score between the frame and its subsequent frames: {s _i,j |j=i+1,i+2,...}, where followed by Compared with the preset similarity threshold t, t=0.1 in this embodiment; suppose The first one in {s _{i, j} } satisfies the following inequality: s _{i, j} <t, then the j ^* -1th frame (that is, the first previous frame that satisfies the above inequality) is the next extracted video frame . The above process is performed iteratively until all video frames are verified.

Step S102, based on the set of aerial perspective images, construct an aerial map by using an image modeling method.

Based on the aerial perspective image set obtained in step S101, an aerial map is constructed through a standard image-based modeling process, which includes: (1) SfM, (2) MVS, and (3) surface reconstruction. In addition, since the GPS signal cannot be received indoors, the aerial map can be scaled to its real physical size through the ground control point (GCP, ground control point). Figure 2 is an example of an aerial map reconstructed from 271 extracted video frames. The first three columns in the figure are the sample aerial image and its corresponding 3D aerial map area, the fourth column is the entire 3D aerial map, and the fifth column is the Robot path planning and virtual camera pose calculation results on the aerial map, where the ground plane is marked in light gray, the planned path is marked as a line segment in the figure, and the virtual camera pose is represented by a pyramid.

Step S200, based on the aerial map, obtain a synthesized image by synthesizing a ground perspective reference image from the aerial map.

The aerial map constructed in step S100 of this embodiment plays two roles in the follow-up process: the first is to plan the path for the ground robot and perform positioning during the movement of the robot; the second is to play a role in the indoor scene reconstruction process Contribute to the fusion of aerial photography and ground images. Both of the above two processes need to establish a two-dimensional to three-dimensional point correspondence between the ground image and the aerial map. To obtain the above correspondence points, a possible effective solution is to directly match aerial and ground images. However, it is very difficult to directly match the two images due to the huge difference in perspective. Here, in this embodiment, the above-mentioned problem is solved by synthesizing the ground perspective reference image from the aerial map. The reference images are synthesized through the following two steps: virtual camera pose calculation and image synthesis based on graph cut.

Step S201, calculating the virtual camera pose based on the aerial map.

The virtual camera pose used for reference image synthesis is calculated based on the ground plane of the indoor scene. In this embodiment, the ground plane of the aerial map is detected by a plane detection method based on random sampling consistency (RANSAC, random sample consensus) (see Fig. 2). The virtual camera pose is calculated in two steps, first calculating the position and then calculating the orientation.

Step S2011, calculating the position of the virtual camera.

Find the 2D bounding box of the ground plane and divide it into a square grid. The size of the grid determines the number of virtual cameras. In order to achieve a balance between positioning accuracy and efficiency, the grid side length is set to 1 m in this embodiment. For each grid, when the ground plane area therein accounts for more than 50% of the total area of the grid, the grid is considered to be a valid grid for placing the virtual camera. The virtual camera position is set to the center of the active grid with an elevation offset of height h (see Figure 2). The value of h is determined by the height of the ground camera, and its value is set to 1m in this embodiment.

Step S2012, the virtual camera faces the design.

After obtaining the virtual camera position, in order to realize the omnidirectional observation of the scene, it is necessary to place multiple virtual cameras with the same optical center and different orientations at each virtual camera position. In this embodiment, since the optical axis of the camera installed on the ground robot is approximately parallel to the ground plane, only a horizontally oriented virtual camera is generated here. In addition, in order to eliminate the perspective projection distortion between the ground and the synthetic image, it is necessary to set the field of view (intrinsic parameters) of the virtual camera close to that of the ground camera. In this embodiment, 6 virtual cameras are placed at each virtual camera position, and the yaw angle between the virtual cameras is 60°.

In addition, the path used for ground robot movement is also planned by checking the ground plane. Since this embodiment does not focus on planning the optimal path of the ground robot, the detected skeleton of the ground plane is used as the robot path here, and the skeleton is extracted by the central axis transformation method (see FIG. 2 ).

Step S202, through the graph cut algorithm, obtain an aerial map and obtain a synthesized image of a reference image of a ground perspective.

In this embodiment, image synthesis is carried out by means of spatially continuous grids, as shown in Figure 3, f in the figure is a three-dimensional space surface, and its two-dimensional projection triangle on the aerial camera C _a and the virtual ground camera C _v camera They are denoted as t _a and t _v respectively. The principle of image synthesis is to change t _a to t _v through f. Specifically, the visible grids of each aerial and virtual camera are obtained first. Then, for each virtual camera, its visible mesh is projected onto the camera to form a 2D triangle set. When performing virtual image synthesis, for a specific two-dimensional triangle in the virtual image, it is necessary to determine which aerial image to use for transformation to fill this area based on the following three factors: (1) Visibility, for the two The three-dimensional space surface corresponding to the three-dimensional triangle, the selected aerial image should have a better viewing angle and a closer viewing distance; (2) clarity, because some of the images obtained from indoor aerial video frame extraction have poor clarity, It is necessary to select a sufficiently clear aerial image; (3) Consistency, the adjacent triangles in the virtual image should be synthesized from the same aerial image as much as possible to maintain the consistency of the synthesized image. In this embodiment, the visibility factor is measured by the projected area of the space patch on the aerial image (the larger the better), and the clarity factor is measured by the median value of the local feature scale of the aerial image (the smaller the better), see Fig. 4. The two columns on the left are images with the largest median value of the two local feature scales, the two columns on the right are images with the smallest median value of the two local feature scales, and the second row is the enlarged image of the rectangular area in the first row. Based on the above description, the image synthesis problem in this embodiment can be attributed to a multi-label optimization problem, defined as shown in formula (1):

Among them, E(l) is the energy function in the graph cut process; is a set of two-dimensional triangles projected from the three-dimensional space grid visible to the virtual camera, and t _i is the i-th triangle among them; is the set of common sides of the triangles in the projected two-dimensional triangle set; l _i is the label of t _i , that is, the serial number of the aerial image. When the spatial patch corresponding to t _i is visible in the l _i aerial image, the data item in is the scale median value of local features in the l _i -th aerial image and is the projected area of the spatial patch corresponding to t _i in the l _i aerial image; otherwise, D _i (l _i )=α, where α is a relatively large constant (α=10 ⁴ in this embodiment) and punish the situation. When l _j =l _j , the smoothing term V _i (l _i , l _j )=0; otherwise V _i (l _i ,l _j )=1. The optimization problem defined in formula (1) can be efficiently solved by the graph cut algorithm.

In order to clarify the influence of the clarity factor and the consistency factor, in this embodiment, image synthesis is performed on one of the virtual cameras under four different configurations, and the results are shown in Figure 5, from left to right: neither The sharpness factor is not considered, and the consistency factor is not considered; only the consistency factor is considered; only the sharpness factor is considered; the image synthesis result that considers both the sharpness factor and the consistency factor. The large rectangle in the upper right corner of each figure is a square enlargement of the small rectangle in the figure. It can be seen from Figure 5 that the sharpness factor makes the composite image clearer and the consistency factor makes the composite image less holes and sharp edges. In addition, Figure 6 shows some other image synthesis results and ground images under similar viewing angles. Although there are still some unavoidable synthesis errors, the synthetic image and its corresponding ground image have a relatively large similarity in the common visible area, which verifies the effectiveness of the image synthesis method in this embodiment. The synthesized images in this step will be used as a reference database for ground robot localization.

In step S300, a set of ground perspective images is acquired through the ground perspective images collected by the ground camera.

When the ground robot is placed in the indoor scene, the robot will move along the planned path and automatically collect the ground perspective video. If the robot is only positioned by its built-in sensors, such as wheel encoders and inertial measurement units (IMUs, inertial measurement units), it will not strictly follow the planned path. This is due to the problem of accumulating errors in sensors built into robots, especially for low-cost sensors mounted on consumer-grade robots. Therefore, the pose of the robot needs to be corrected by visual positioning, and in this step, the visual positioning is achieved by matching the synthetic and ground images.

Step S301, based on the planned path, the ground robot continuously collects ground perspective videos through the ground camera set on it.

In this step, the positioning method includes initial robot positioning and mobile robot positioning.

(1) Initial robot positioning

The initial robot positioning method is as follows: obtain the first frame of the video collected by the ground camera, obtain the initial position of the robot in the aerial map, and use this position as the starting point of the subsequent movement of the robot.

By locating the first frame of the video collected by the ground camera, the initial position of the robot in the aerial map can be obtained, and this position can be used as the starting point of the subsequent movement of the robot. The above initial positioning can be realized by matching the first frame image with all synthetic images or the k most similar synthetic images obtained through semantic tree retrieval. In this step, the method based on image retrieval is used, and k=30. It should be noted that although the ground perspective image is synthesized, there is still a big difference between the ground image and the synthetic image in terms of illumination and perspective, and the commonly used scale-invariant feature transform (SIFT, scale-invariant feature transform) feature is not enough to deal with it. The ASIFT (affine-SIFT) feature is used in this step.

In order to verify the effectiveness of the image synthesis method in this step and compare the performance of SIFT features and ASIFT features, this embodiment uses SIFT features and ASIFT features to perform synthesis and ground image matching and aerial photography and ground image matching. Wherein, the ground image is also extracted from the video collected by the ground robot through the adaptive video frame extraction method based on the bag-of-words model in step S100. When performing image matching, by retrieving different numbers of synthetic images and aerial images that are the closest to the current ground image, it is found that when the number of matching points after the basic matrix verification is still greater than 16, the pair of images can be considered to be matched. The image matching results are shown in Figure 7 (the x-axis in the figure is the number of retrieved images, and the y-axis is the logarithm of the number of matching image pairs). It can be seen from Figure 7 that the number of matching pairs obtained by using ASIFT for synthesis and ground image matching is about 6 times that of using ASIFT for aerial photography and ground image matching, using SIFT for synthesis and ground image matching, and using SIFT for aerial photography and ground image matching , 8 times and 19 times.

Given the two-dimensional matching points between the first ground image and the retrieved synthetic image, the corresponding three-dimensional space points can be obtained on the aerial map by ray casting. In this way, a method based on perspective-n-point (PnP, perspective-n-point) can be used to realize the positioning of the first frame of the ground image. Specifically, given the corresponding points from 2D to 3D and the internal parameters of the ground camera, the camera pose is solved by RANSAC using different PnP algorithms. The PnP algorithms adopted include P3P, AP3P and EPnP. When at least one of the interior points corresponding to the above algorithm exceeds 16, it can be considered that this pose estimation is a successful estimation, and the pose of the camera is set as the one with the largest number of interior points in the PnP result. In the RANSAC process of this embodiment, a total of 500 random samplings are performed, and the distance threshold is set to 4px.

(2) Mobile robot positioning

The positioning of the mobile robot is as follows: based on the initial position and the driving data of the robot at each time, the rough positioning of the robot position is performed, and by matching the video frame image collected at the current time with the composite image, the current position of the robot on the aerial map is obtained. The position in , and use this position to revise the position information of coarse positioning.

When the ground robot moves in the indoor scene and collects video, it can be roughly positioned by the wheel odometer. In this step, the ground robot is globally positioned on the aerial map by matching the ground and the synthetic image to correct the rough positioning result of the robot. The pose correction is only performed on the extracted ground video frames, not all video frames. This is because: (1) the ground robot moves relatively slowly indoors, and will not seriously deviate from the planned path in a short period of time; (2) it takes about 0.5s to perform global visual positioning each time, and the time is mainly spent on ASIFT features Extract on. It should be noted that for some extracted video frames, the visual localization is not always successful due to the insufficient number of inliers for PnP.

Assume that the position and orientation of the ground image successfully positioned last time are recorded as c _A and n _A respectively, and the rough position and orientation obtained by the wheel odometer of the current ground image to be positioned are respectively recorded as c _B and n _B . Here, the candidate matching synthetic image of the current ground image is searched based on the rough positioning results, not based on the image retrieval method. The schematic diagram of this method is shown in Figure 8, in which c _A and n _A are the position and orientation of the ground image successfully positioned last time, c _B and n _B are the rough position and orientation of the current ground image, and the circle in the figure Indicates the search range, the center of the circle is c _B , the radius is r _B , the triangle in the figure represents the virtual camera pose, the light gray triangle represents the selected composite image and the dark gray triangle represents the unselected composite image. When the synthesized image satisfies the following two conditions, it will be matched with the current ground image: (1) The synthesized image is located in a circle with center c _B and radius r _B , where r _B =max(∥c _B -c _A ‖, β) and β=2m; (2) The angle between the composite image orientation and n _B is less than 90°. The reason for using a variable radius r _B here is that as the robot moves, the drift of the relative pose acquired by the robot's built-in sensor will become more and more serious. After matching the current ground image with the obtained candidate matching synthetic image, the current ground image adopts a method similar to the method in the initial robot localization, and achieves localization through the method based on PnP RANSAC. If the position and orientation deviations of the positioning results from the rough positioning results are small enough (in this embodiment, the position deviation is less than 5m, and the orientation deviation is less than 30°), then the current ground image positioning is successful. It is confirmed that the pose of the robot has been globally corrected by the ground image of the current localization success, and the pose in the wheel odometer is reset to the current vision-based localization result. Ground images that are not successfully positioned in this step will be relocated in the subsequent indoor scene reconstruction process.

Step S302 , for the ground-view video of the indoor scene, image frames are extracted using an adaptive video frame extraction method based on the bag-of-words model, to obtain a ground-view image set of the indoor scene.

In this step, through the adaptive video frame extraction method based on the bag-of-words model in step S100, image frames are extracted from the obtained ground-view video of the indoor scene to obtain a set of ground-view images of the indoor scene. Since the method is consistent, it is not described here Let me repeat.

Step S400, based on the synthesized image, the aerial perspective image and the ground perspective image are fused to obtain an indoor scene model.

After robot localization and video acquisition, not all frames extracted from the ground video were successfully localized to the aerial map. However, to obtain a complete indoor scene reconstruction result, all images extracted from (aerial and terrestrial) videos need to be localized and fused. Here, a pipeline for batch-wise localization of previously unsuccessfully localized ground images is first proposed. Then, the matching interior points of the ground and the synthetic image are connected to the original feature point trajectory, and the fusion of the aerial photography and the ground point cloud is realized through bundle adjustment (BA, bundle adjustment). Finally, a complete and dense indoor scene reconstruction result is obtained by fusing aerial and ground images.

Step S401, acquiring the position of the ground camera corresponding to each image in the ground perspective image set in the aerial map.

In order to locate the ground images that were not successfully located in step S301, the present invention proposes a batch camera positioning process. In each camera positioning loop, try to position as many cameras as possible. The 3D spatial points in the 2D to 3D corresponding points used for camera positioning here include not only the spatial points reconstructed in the SfM process, but also the spatial points obtained by intersecting the aerial map (3D mesh) through ray casting. Each batch camera positioning cycle includes three steps: (1) camera positioning, (2) scene expansion and bundle adjustment (BA, bundle adjustment), (3) camera filtering, and its flowchart is shown in Figure 9 . Before performing batch camera positioning, the images obtained by extracting frames from the ground video are first matched and the matching points are connected into feature point trajectories. For the feature point trajectories of at least two visible images that have been successfully positioned, their spatial coordinates are calculated by means of triangulation.

Step S4011, camera positioning.

There are two ways to obtain 2D and 3D corresponding points to locate the ground image that is currently not successfully positioned: (1) Aerial map, for the 2D feature points in the ground image that is currently not successfully positioned, you can obtain its position in the successfully positioned image matching point. Then a ray is projected from the optical center of the successfully positioned camera to these matching points, and the intersection point of the projected ray and the aerial map is the 3D space point corresponding to the 2D feature point in the currently unsuccessfully positioned ground image. (2) Ground feature point trajectories. Given the current ground feature point trajectories obtained through triangulation, the corresponding two-dimensional feature points in the current unsuccessful ground image can be obtained through the matching results between previous ground images. Ground cameras that have not been successfully positioned at present can use the above two 2D and 3D corresponding points to achieve positioning through the PnP-based RANSAC method, and the positioning result uses the one with the most internal points among the two results. The method of realizing camera positioning through two 2D and 3D corresponding points is compared with the method using only one of them. point trajectory, (2) only based on the aerial map, (3) the batch camera positioning results based only on the ground feature point trajectory, the x-axis in the figure is the number of batch camera positioning cycles, and the y-axis is the number of successfully positioned cameras; when x =0, the corresponding y value is the number of cameras successfully positioned in step S300. It can be seen from Figure 10 that after several iterative cycles, the three methods can locate the same number of cameras. However, in this embodiment, the method for realizing camera positioning through two 2D and 3D corresponding points requires the least number of iterative cycles (only 5 times, while the other two methods need 6 times and 8 times respectively).

Step S4012, scene extension and BA.

After camera localization, ground feature point trajectories are triangulated according to the newly localized camera to achieve scene expansion. In order to improve the accuracy of the camera pose and scene points, after triangulation, the localized ground camera pose and the spatial position of the ground feature point trajectory obtained by triangulation are optimized by BA.

Step S4013, camera filtering.

Considering the robustness of the method, a camera filtering operation is added to the successfully located cameras after BA. If the position or orientation of the newly positioned camera in this iterative cycle is greatly deviated from its rough positioning result (the positioning result obtained by the wheel odometer) after BA optimization (the position deviation is greater than 5m or the orientation deviation is greater than 30°). , the positioning result is determined to be unreliable and filtered out. It should be noted in this step that the cameras filtered out in the current iterative cycle can still be successfully located in the subsequent iterative cycle.

The above three steps are performed iteratively until all cameras are successfully located or no more cameras can be successfully located. The process of batch camera positioning is shown in Figure 11. The pyramid in the figure represents the pose of the successfully positioned camera. The 0th iteration represents the result of camera positioning in step S300.

Step S402, connecting the matching points of the ground perspective image and the synthetic image into the original aerial photography and ground feature point trajectory to generate cross-view constraints.

In order to fuse aerial photography and ground point clouds through BA, it is necessary to introduce constraints between aerial photography and ground images. Here, the above-mentioned cross-view constraint can be provided by the aerial photography and ground feature point trajectories generated by the image matching points obtained by matching the ground and the synthetic image in step S300. The matched ground image feature points can be easily connected to the original ground feature point trajectory by querying their serial numbers. However, although the synthetic image is generated from the aerial image, it is not so easy to connect the matching synthetic image feature points to the original aerial feature point trajectory. This is because the synthetic image feature points used for matching with the ground image are re-extracted on the synthetic image. In this step, the ground and synthetic image matching points are extended to the _aerial view by means of ray projection and point projection. camera, X _j (j=1,2,3) is the spatial point corresponding to the feature point of the matched synthetic image, t _ij is the projection of point X _j on camera C _i , t _1j -t _2j -t _2j (j =1,2) is the feature point trajectory of the jth inter-aerial view. Specifically, the spatial points corresponding to the matching synthetic image feature points are obtained on the aerial map by ray projection, and then the acquired spatial points are projected onto the visible aerial image to produce aerial and ground feature point trajectories.

Step S403, optimize the point cloud of the aerial map and the ground perspective image through BA.

In this step, the Ceres library is used to minimize the back-projection error to connect the generated aerial and ground feature point trajectories, the original (aerial and ground) feature point trajectories, and the internal and external parameters of all (aerial and ground) cameras. optimization.

Step S404, using the aerial photography and ground camera poses optimized in step S403, fusing aerial photography and ground images to perform dense reconstruction to obtain a dense model of the indoor scene.

Due to the introduction of cross-aerial and ground view constraints in the optimization process, and the fusion of aerial and ground images in the dense reconstruction process, the reconstructed model is more complete and accurate than the model reconstructed with only a single source of image.

In order to verify the scene modeling method for the fusion of aerial photography and ground perspective images in the embodiment of the present invention, based on the experimental equipment for collecting aerial photography and ground metadata as shown in Figure 13, and the collected two sets of indoor scene data sets, in The method of this embodiment is tested on these two sets of data sets.

1. Dataset

Since there are almost no public datasets of aerial photography and ground images for indoor scenes, two sets of indoor scene datasets were collected for method evaluation in this test. Specifically, the DJI Spark mini-aircraft is used for aerial perspective scene collection, and the GoPro HERO4 installed on the TurtleBot is used for ground perspective scene collection. The metadata collection equipment is shown in Figure 13, from left to right are TurtleBot on the ground, DJISpark in the air, DJI Spark on the desktop. The collected aerial photography and ground metadata are in the form of videos with a resolution of 1080p and a frame rate of 25FPS. The collected two indoor scene data sets are called Room and Hall respectively. Some information about the Room and Hall datasets is shown in Table 1. The sample aerial images and the generated 3D aerial maps in the Room and Hall datasets are shown in Figure 2 and Figure 14, respectively. It can be seen from Figure 2 and Figure 14 that the aerial map of the Hall dataset is worse in quality and larger in scale than the aerial map of the Room dataset. However, it can be seen from the subsequent method evaluation content that the method of the present invention can achieve the expected results on the above two data sets, which shows that the method of the present invention has better robustness and scalability.

Table 1

data set Room Hall Aerial video length/s 218 494 Ground video length/s 61 113 Coverage area/m² 30 130

In addition, the virtual camera pose calculation and robot path planning results on the Room and Hall datasets are shown on the far right of Figure 2 and Figure 14, respectively. As shown in the figure, through the method of the present invention, the ground plane used for virtual camera pose calculation and robot path planning can be successfully detected, and the generated virtual camera and robot path evenly cover the indoor scene. 60 and 384 virtual cameras are respectively generated on the Room and Hall data sets through the virtual camera pose calculation method of the present invention. The first three columns in Fig. 14 are example aerial images and their corresponding three-dimensional aerial map areas. The fourth column is the entire 3D aerial map. The fifth column is the robot path planning and virtual camera pose calculation results on the aerial map, where the ground plane is marked in light gray, the planned path is marked as a line segment, and the virtual camera pose is represented by a pyramid.

2. Adaptive frame drawing result

Through the adaptive frame extraction method in the present invention, 271 and 112 frames of images are respectively extracted from the aerial photography and ground videos of the Room dataset, and 721 and 250 frames of images are extracted from the aerial photography and ground videos of the Hall dataset. In order to verify the effectiveness of the frame extraction method in the present invention, a comparative experiment was carried out between the method of the present invention and the frame extraction method at equal intervals on the aerial video of the Hall dataset. The self-adaptive frame extraction method of the present invention is used to extract 721 frames of images on a video with a length of 494s and a frame rate of 25FPS. For the equal interval frame extraction method, an image is extracted every 17 frames (494×25/721≈ 17), a total of 730 frames of images were extracted. Then, the video frames obtained by the two different frame extraction methods are calibrated through the open source SfM system COLMAP, and the result is shown in Figure 15. Left: the COLMAP result of the adaptively extracted video frames, where The video frames of were successfully calibrated; the middle and right images: the COLMAP results of video frames extracted at equal intervals, where The video frame is successfully calibrated, but is broken into two parts. The middle image and the right image respectively correspond to the two rectangular areas in the left image. The circled parts in the left and right images show the comparison results at the same corner. It can be seen from Fig. 15 that, compared with the equal interval method, the video frames extracted by the method of the present invention are more connected, so by reconstructing them, a consistent aerial map can be obtained. In addition, the black circles in Figure 15 indicate that in order to obtain a more complete aerial map, it is necessary to perform more intensive frame extraction operations on the video at the corners.

3. Ground camera positioning results

In order to verify the batch camera positioning and aerial photography and ground image fusion method of the present invention, the qualitative and quantitative comparisons of batch camera positioning, aerial photography and ground image fusion camera positioning results and COLMAP results are carried out here. It should be noted that for COLMAP, the ground camera pose is not initialized, and only the image itself is calibrated, that is, the camera positioning result of the aerial map in step S300 is not provided to COLMAP as a priori.

The qualitative comparison results are shown in Figure 16. The first row: the results of the Room dataset; the second row: the results of the Hall dataset; from left to right: the results of the fusion of aerial photography and ground images, the results of batch positioning of ground cameras, COLMAP calibration result; the rectangle in the figure indicates the wrong camera pose. It can be seen from Figure 16 that for the Room dataset, the camera poses obtained by the three comparison methods are relatively similar, because the scene structure of the Room dataset is relatively simple. For the Hall dataset, the camera trajectory calculated by COLMAP has obvious errors in the left part of the scene. This is due to the fact that the matching results between ground images contain more matching outliers due to repeated textures and weak textures, which will lead to obvious scene drift in the incremental SfM system. In contrast, for batch camera positioning, since some ground images have been initially positioned on the aerial map, there are only some slight scene drifts in the positioning results. Moreover, the wrong camera poses mentioned above are all corrected in the subsequent stage of aerial photography and ground image fusion. This is due to the introduction of connection generated aerial photography and ground feature point trajectories in the global optimization of image fusion. The above results show that localization of ground cameras by fusing aerial and ground images is more robust than using ground images alone.

4. Indoor scene reconstruction results

Finally, qualitative and quantitative evaluations are carried out on the indoor scene reconstruction algorithm in the present invention. In this test, the indoor reconstruction results in the present invention are compared with the reconstruction results using only aerial photography or ground images. The qualitative comparison results are shown in Figure 17, the first column: the results of the Room dataset; the second column: the results in the first column Magnified view of the rectangular area; third column: Hall dataset results; fourth column: enlarged view of the rectangular area in the third column. From top to bottom: Ground imagery only, aerial imagery only, results using fused aerial and ground imagery. It should be noted that: (1) For the indoor reconstruction algorithm in the present invention, the camera pose used is the camera pose after fusing aerial photography and ground images; (2) For the method using only ground images, the camera pose used is the camera pose after batch camera positioning; (3) For the method that only uses aerial cameras, the camera pose used is the camera pose estimated by SfM. It can be seen from Fig. 17 that although due to the existence of occlusion and weak texture, some areas are inevitably missing in the reconstruction results. Compared with only using a single image for reconstruction, the indoor reconstruction results by fusing aerial photography and ground images are more accurate. whole.

According to the second embodiment of the present invention, a scene modeling system that integrates aerial photography and ground perspective images, the system includes an aerial photography map construction module, a synthetic image acquisition module, a perspective image collection acquisition module, and an indoor scene model acquisition module;

The aerial map construction module is configured to obtain an aerial perspective image of the indoor scene to be modeled, and construct an aerial map;

The synthetic image acquisition module is configured to obtain a synthetic image based on the aerial map by synthesizing a ground perspective reference image from the aerial map;

The viewing angle image set acquisition module is configured to acquire a ground viewing angle image set through ground viewing angle images collected by a ground camera;

The indoor scene model acquisition module is configured to fuse the aerial perspective image with the ground perspective image based on the synthesized image to acquire an indoor scene model. ,include

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process and related descriptions of the above-described system can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

It should be noted that the scene modeling system for fusing aerial photography and ground perspective images provided by the above-mentioned embodiments is only illustrated by the division of the above-mentioned functional modules. In practical applications, the above-mentioned functions can be assigned to different function modules, that is, to decompose or combine the modules or steps in the embodiments of the present invention. For example, the modules in the above embodiments can be combined into one module, or can be further split into multiple sub-modules, so as to complete all or the steps described above. Some functions. The names of the modules and steps involved in the embodiments of the present invention are only used to distinguish each module or step, and are not regarded as improperly limiting the present invention.

A storage device according to the third embodiment of the present invention stores a plurality of programs, and the programs are adapted to be loaded and executed by a processor to implement the above-mentioned scene modeling method for fusing aerial photography and ground perspective images.

A processing device according to the fourth embodiment of the present invention includes a processor and a storage device; the processor is suitable for executing various programs; the storage device is suitable for storing multiple programs; the program is suitable for being loaded and executed by the processor In order to realize the above-mentioned scene modeling method of fusing aerial photography and ground perspective images.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process and related descriptions of the storage device and the processing device described above can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here. repeat.

Those skilled in the art should be able to realize that the modules and method steps described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two, and that the programs corresponding to the software modules and method steps Can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or known in the technical field any other form of storage medium. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in terms of functions in the above description. Whether these functions are performed by electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may implement the described functionality using different methods for each particular application, but such implementation should not be considered as exceeding the scope of the present invention.

The terms "first", "second", etc. are used to distinguish similar items, and are not used to describe or represent a specific order or sequence.

The term "comprising" or any other similar term is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus/apparatus comprising a set of elements includes not only those elements but also other elements not expressly listed, or Also included are elements inherent in these processes, methods, articles, or devices/devices.

So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings, but those skilled in the art will easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to relevant technical features, and the technical solutions after these changes or substitutions will all fall within the protection scope of the present invention.

Claims (10)

1. A scene modeling method for merging aerial photography and ground perspective image, is characterized in that, comprises the following steps: Step S100, obtaining the aerial perspective image of the indoor scene to be modeled, and constructing an aerial map; Step S200, based on the aerial map, obtain a synthesized image through the method of synthesizing a ground perspective reference image from the aerial map; Step S300, obtaining a set of ground perspective images through the ground perspective images collected by the ground camera; Step S400, based on the synthesized image, the aerial perspective image and the ground perspective image are fused to obtain an indoor scene model. 2. The scene modeling method for fusing aerial photography and ground perspective images according to claim 1, characterized in that, in step S100, "acquire the aerial perspective images of the indoor scene to be modeled, and construct an aerial map", the method is : For the aerial perspective video of the indoor scene, the image frame is extracted by using the adaptive video frame extraction method based on the bag-of-words model, and the aerial perspective image collection of the indoor scene is obtained; Based on the set of aerial perspective images, an aerial map is constructed through an image modeling method. 3. The scene modeling method for merging aerial photography and ground perspective images according to claim 1, characterized in that, in step S200, "the method for synthesizing ground perspective reference images by aerial photography maps" is as follows: Based on the aerial map, calculate the virtual camera pose; Through the graph cut algorithm, the aerial map is obtained to obtain the synthetic image of the reference image of the ground perspective. 4. the scene modeling method of fusion aerial photography and ground perspective image according to claim 3, is characterized in that, "by graph cut algorithm, obtain aerial photography map and obtain the synthetic image of ground perspective reference image", its method is: Among them, E(l) is the energy function in the graph cut process; is a set of two-dimensional triangles projected from the three-dimensional space grid visible to the virtual camera, and t _i is the i-th triangle among them; is the set of common sides of the triangles in the projected two-dimensional triangle set; l _i is the serial number of the aerial image of t _i ; D _i (l _i ) is the data item; V _i (l _i , l _j ) is the smoothing item; When the spatial patch corresponding to t _i is visible in the l _i aerial image, the data item Otherwise D _i (l _i )=α, where is the scale median value of local features in the l _i -th aerial image, is the projected area of the spatial patch corresponding to t _i in the l _i aerial image, and α is a relatively large constant; When l _i =l _j , the smoothing term V _i (l _i , l _j )=0; otherwise V _i (l _i ,l _j )=1. 5. The scene modeling method for fusing aerial photography and ground perspective images according to claim 1, characterized in that, in step S300, "obtain a ground perspective image collection through the ground perspective images collected by the ground camera", the method is: Based on the planned path, the ground robot continuously collects ground perspective video through the ground camera set on it; For the ground-view video of the indoor scene, the adaptive video frame extraction method based on the bag-of-words model is used to extract image frames, and the ground-view image collection of the indoor scene is obtained. 6. The scene modeling method of fusion aerial photography and ground perspective images according to claim 5, characterized in that, in the process of "the ground robot continuously collects ground perspective videos through the ground camera provided on it" based on the planned path, its positioning The method includes initial robot positioning, mobile robot positioning; The initial robot positioning method is as follows: obtain the first frame of the video collected by the ground camera, obtain the initial position of the robot in the aerial map, and use this position as the starting point of the subsequent movement of the robot; The positioning of the mobile robot is as follows: based on the initial position and the driving data of the robot at each time, the rough positioning of the robot position is performed, and by matching the video frame image collected at the current time with the composite image, the current position of the robot on the aerial map is obtained. The position in , and use this position to revise the position information of coarse positioning. 7. The scene modeling method for fusing aerial photography and ground perspective images according to claim 1, characterized in that step S400 "based on the composite image, fuses the aerial photography perspective image with the ground perspective image to obtain Indoor Scene Model", the method is: Obtain the position of the ground camera corresponding to each image in the ground perspective image collection in the aerial map; Connect the matching points of the ground perspective image and the synthetic image into the original aerial photography and ground feature point trajectory to generate cross-view constraints; Optimize aerial maps and point clouds of ground perspective images through bundled adjustments; Dense reconstruction of aerial maps and ground perspective images is used to obtain dense models of indoor scenes. 8. A scene modeling system that fuses aerial photography and ground perspective images, characterized in that the system includes an aerial photography map building module, a synthetic image acquisition module, a perspective image collection acquisition module, and an indoor scene model acquisition module; The aerial map construction module is configured to obtain an aerial perspective image of the indoor scene to be modeled, and construct an aerial map; The synthetic image acquisition module is configured to obtain a synthetic image based on the aerial map by synthesizing a ground perspective reference image from the aerial map; The viewing angle image set acquisition module is configured to acquire a ground viewing angle image set through ground viewing angle images collected by a ground camera; The indoor scene model acquisition module is configured to fuse the aerial perspective image with the ground perspective image based on the synthesized image to acquire an indoor scene model. 9. A storage device, wherein a plurality of programs are stored, wherein the programs are adapted to be loaded and executed by a processor to realize the scene of merging aerial photography and ground perspective images according to any one of claims 1-7 modeling method. 10. A processing device, comprising a processor and a storage device; the processor is suitable for executing various programs; the storage device is suitable for storing a plurality of programs; it is characterized in that the program is suitable for being loaded and executed by the processor to The scene modeling method for fusing aerial photography and ground perspective images described in any one of claims 1-7 is realized.