CN112560137B - Multi-model fusion method and system based on smart city - Google Patents

Abstract

The present invention provides a multi-model fusion method, system, electronic device and storage medium based on smart city, the method includes: generating BIM model and oblique photography model according to aerial photography images; rendering BIM model and oblique photography model by B/S end mode and GPU rendering mode; fusing the rendered BIM model and oblique photography model to obtain a fused real-life three-dimensional model; constructing a real-time perception model of a mobile target according to the coordinate points, movement trajectory and attribute information of indoor and outdoor mobile targets collected in real time, and fusing the constructed real-time perception model of the mobile target into the real-life three-dimensional model and displaying it. The present invention can fuse multiple models, browse and access oblique photography data, BIM data model and real-time perception model of mobile targets on the browser side through B/S end and based on GPU and server rendering technology, so as to improve the browsing experience of users.

Description

Multi-model fusion method and system based on smart city

Technical Field

The present invention relates to the field of model fusion, and more specifically, to a multi-model fusion method and system based on a smart city.

Background Art

In recent years, with the rapid development of high-tech technologies such as 5G, cloud computing, the Internet of Things and AI (artificial intelligence), GIS (Geographic Information System) has gradually been given new vitality. Currently, real-scene three-dimensional technology plays an important role in the construction of smart cities.

Among them, drone aerial photography technology is currently the main means of obtaining real-life 3D data due to its convenient and fast data acquisition method and highly automated modeling method. At the same time, as people's demand for indoor and outdoor scenes increases, the combination of oblique photography and BIM models in large scenes can effectively make up for the problem of insufficient indoor scene display. In addition, in terms of IOT and AI recognition results, the combination of IoT sensing devices and AI real-time sensing technology has enhanced the authenticity of 3D scenes to a certain extent.

Summary of the invention

The embodiments of the present invention provide a multi-model fusion method and system based on a smart city that overcomes the above-mentioned problems or at least partially solves the above-mentioned problems.

According to a first aspect of the present invention, a multi-model fusion method based on a smart city is provided, comprising: mapping a BIM model according to an aerial photographic image to generate a BIM model of an indoor or outdoor scene; rendering the BIM model and the oblique photography model of the indoor or outdoor scene through a B/S terminal mode and using a GUP rendering mode; fusing the rendered BIM model and the oblique photography model to obtain a fused real-scene three-dimensional model; constructing a real-time perception model of a mobile target according to the coordinate points, movement trajectories and attribute information of the indoor or outdoor mobile target collected in real time, fusing the constructed real-time perception model of the mobile target into the real-scene three-dimensional model, and displaying it on the front end.

Based on the above technical solution, the embodiment of the present invention can also make the following improvements.

Furthermore, the method of mapping the BIM model according to the aerial photographic images to generate the BIM model of the indoor and outdoor scenes includes: parsing the IFC format used by the BIM model and splitting the nodes of the BIM model; establishing a corresponding relationship between each node of the BIM model and the aerial photographic images; importing the aerial photographic images corresponding to each node of the BIM model into the BIM model, batch mapping the BIM model, and generating the BIM model of the indoor and outdoor scenes.

Furthermore, the rendering of the BIM model and oblique photography model of indoor and outdoor scenes through the B/S end and using the GUP rendering method includes: converting the BIM model data structure into a 3D Tiles data format on the B/S end, and using the Cesium map engine to load and render the BIM model data.

Furthermore, it also includes: reducing the screen error rate and maintaining the precision of the BIM model by setting the maximum screen space error parameter in the Cesium map engine; and increasing the loading speed of the BIM model data by configuring the parameters for prioritizing the loading of the central block of the screen.

Furthermore, the rendering of the BIM model and the oblique photography model is fused to obtain a fused real-scene 3D model, including:

For the overlapping parts of the BIM model and the oblique photography model, the oblique photography model is cropped based on the Cesium 3D earth framework;

The BIM model is integrated into the cropped oblique photography model to generate a real-life 3D model.

Furthermore, for the overlapping part of the BIM model and the oblique photography model, clipping the oblique photography model based on the Cesium three-dimensional earth framework includes:

For the oblique photography model, the BIM model boundary is used as the initialization polygon clipping area, and the longitude and latitude coordinate values of each edge of the polygon clipping area are recorded;

Taking the coordinate origin of the oblique photography model as a reference, matrix operation of coordinate values is performed to correct the coordinates of the polygonal clipping area;

Taking the surface where each edge of the corrected polygonal clipping area is located as the clipping surface, a plurality of clipping surfaces are obtained;

Reconstructing the clipping surface according to the normal vector of any clipping surface and the shortest distance from the coordinate origin of the oblique photography model to any clipping surface, and finally obtaining a plurality of reconstructed clipping surfaces;

The oblique photography model is cropped based on the reconstructed multiple cropping planes.

Furthermore, the construction of a real-time perception model of a mobile target based on the coordinate points, movement trajectory and attribute information of indoor and outdoor mobile targets collected in real time includes: constructing a real-time perception model of a mobile target based on the coordinate points, movement trajectory and attribute information of indoor and outdoor mobile targets sent by the monitoring device through the established WebSocket real-time transmission channel; after integrating the constructed real-time perception model of the mobile target into the real-scene three-dimensional model and displaying it on the front end, it also includes: updating the position and orientation of the real-time perception model of the mobile target based on the coordinate points and movement trajectory of the mobile target updated in real time by the monitoring device, and displaying the updated real-time perception model of the mobile target on the front end.

Furthermore, the real-time perception model of the mobile target includes a human and a vehicle model, and also includes: when receiving the coordinate points, movement trajectories and attribute information of indoor and outdoor mobile targets sent by the monitoring device through the established WebSocket real-time transmission channel, establishing a correspondence between the real-time perception model ID of each mobile target and the time, position and orientation; updating the position and orientation of the real-time perception model of the mobile target according to the coordinate points and movement trajectories of the mobile target updated in real time by the monitoring device includes: inputting the real-time perception model ID, time, position and orientation of the mobile target into the update orientation function headingPitchRollQuaternion to realize the update of the position and orientation of the real-time perception model of the corresponding mobile target at the corresponding time.

According to a second aspect of an embodiment of the present invention, a multi-model fusion system is provided, including:

A mapping module is used to map the BIM model according to the aerial photography images to generate the BIM model of the indoor and outdoor scenes; a rendering module is used to render the BIM model and the oblique photography model of the indoor and outdoor scenes through the B/S terminal mode and the GPU rendering mode; a fusion module is used to fuse the rendered BIM model and the oblique photography model to obtain a fused real-scene three-dimensional model; a display module is used to construct a real-time perception model of the mobile target according to the coordinate points, movement trajectory and attribute information of the indoor and outdoor mobile targets collected in real time, fuse the constructed real-time perception model of the mobile target into the real-scene three-dimensional model, and display it on the front end.

According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein the processor is used to implement the steps of a multi-model fusion method when executing a computer management program stored in the memory.

According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer management program is stored, and when the computer management program is executed by a processor, the steps of the multi-model fusion method are implemented.

The embodiment of the present invention provides a multi-model fusion method and system based on smart city, which generates BIM models and oblique photography models of indoor and outdoor scenes according to aerial photography images; renders the BIM model and oblique photography model through the B/S end mode and adopts GPU rendering mode; fuses the rendered BIM model and oblique photography model to obtain a fused real-life three-dimensional model; constructs a real-time perception model of the mobile target according to the coordinate points, movement trajectory and attribute information of the indoor and outdoor mobile targets collected in real time, fuses the constructed real-time perception model of the mobile target into the real-life three-dimensional model, and displays it on the front end. The embodiment of the present invention can fuse multiple models such as oblique photography models and BIM models, browse and access oblique photography data and BIM data models on the browser side through the B/S end and based on GPU and server rendering technology, so as to improve the user's browsing experience.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a multi-model fusion method provided by an embodiment of the present invention;

FIG2 is a schematic diagram showing the correspondence between each node of the BIM model and the aerial image;

FIG3 is an overall flow chart of a multi-model fusion method provided by an embodiment of the present invention;

FIG4 is a structural diagram of a multi-model fusion system provided by an embodiment of the present invention;

FIG5 is a flow chart of a method for cropping an oblique photography model provided by an embodiment of the present invention;

FIG6 is a schematic diagram of a possible hardware structure of an electronic device provided by an embodiment of the present invention;

FIG. 7 is a schematic diagram of a hardware structure of a possible computer-readable storage medium provided in an embodiment of the present invention.

DETAILED DESCRIPTION

The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

Figure 1 is a flow chart of a multi-model fusion method provided by an embodiment of the present invention. As shown in Figure 1, the method includes: 101. According to the aerial photography image, the BIM model is mapped to generate the BIM model of the indoor and outdoor scenes; 102. The BIM model and the oblique photography model of the indoor and outdoor scenes are rendered through the B/S end mode and the GUP rendering mode; 103. The rendered BIM model and the oblique photography model are fused to obtain a fused real-scene three-dimensional model; 104. According to the coordinate points, movement trajectories and attribute information of the indoor and outdoor mobile targets collected in real time, a real-time perception model of the mobile target is constructed, and the constructed real-time perception model of the mobile target is fused into the real-scene three-dimensional model, and displayed on the front end.

It is understandable that based on the existing research on the integration methods of real-life 3D technology and artificial intelligence technology, there are several research methods: 1. Realize multi-source real-life data fusion by processing data sources; 2. Load large-scene real-life 3D in a C/S manner; 3. Integrate GIS with AI real-time perception technology.

According to the above introduction, the traditional fusion methods have the following shortcomings and areas that need to be improved: 1. Data source level: The pictures collected by drones are not effectively applied to the BIM model texture material. 2. Multi-source data fusion usually uses the method of processing source data, which reduces the flexibility and reusability of source data operations; 3. The C/S end software has poor scalability and is difficult to meet the current user customization needs; 4. The fusion method of AI and real-life 3D is single, and the visualization effect is not good.

In view of the singleness or insufficiency of traditional fusion methods, the embodiments of the present invention comprehensively consider the fusion of multi-source data and AI real-time perception technology, mainly including: making full use of aerial photography images to map the BIM model, and generating BIM models and oblique photography models of indoor and outdoor scenes. Among them, the BIM model includes various micro models, for example, a specific building and various components inside the building, and the oblique photography model includes the location coordinates of each small model. Generally speaking, the degree of refinement of the oblique photography model is higher than that of the BIM model.

After generating the BIM model and oblique photography model of the indoor and outdoor scenes, the BIM model and oblique photography model of the indoor and outdoor scenes are rendered through the B/S end and the GPU rendering method, and the rendered BIM model and oblique photography model are fused to generate a fused real-life 3D model. Based on the coordinate points, movement trajectory and attribute information of the indoor and outdoor moving targets collected in real time, a real-time perception model of the moving targets is constructed, and the constructed real-time perception model of the moving targets is fused into the real-life 3D model and displayed on the front end.

The embodiment of the present invention can integrate multiple models such as oblique photography models and BIM models, and browse and access the oblique photography data and BIM data models on the browser side through the B/S side and based on GPU and server rendering technology, thereby improving the user's browsing experience.

In a possible implementation manner, mapping is performed on a BIM model according to an aerial photographic image to generate a BIM model of indoor and outdoor scenes, including: parsing the IFC format used by the BIM model and splitting the nodes of the BIM model; establishing a correspondence between each node of the BIM model and the aerial photographic image; importing the aerial photographic image corresponding to each node of the BIM model into the BIM model, batch mapping the BIM model, and generating a BIM model of the indoor and outdoor scenes.

It can be understood that, as shown in Figure 2, for the drone aerial image, the aerial image is identified by the artificial recognition algorithm (AI). Usually when building a BIM model, it is necessary to specify the material for the BIM model, but generally speaking, the material of the modeling software cannot reflect the real texture characteristics of the BIM model. By studying the IFC format used by the BIM model and parsing it, splitting the nodes, obtaining the graphic data and spatial structure information of the BIM model, and establishing the correspondence between the base point reference in IfcMaterialResource and the AI recognition image result, it can be understood as establishing the correspondence between each node of the BIM model and the aerial image.

After establishing the correspondence between each node in the BIM model and the aerial image, the aerial image corresponding to all nodes of the BIM model are imported into the BIM model. If the aerial image is an indoor or outdoor scene on the ground, the generated BIM model is an indoor or outdoor scene BIM model, so that the BIM model will automatically be mapped when it is sliced. In this way, the aerial image resources are effectively utilized, and the nodes of the BIM model are used as calculation references to achieve the batch fast mapping effect of the BIM model.

In a possible implementation manner, rendering the BIM model and oblique photography model of indoor and outdoor scenes through the B/S end and using the GUP rendering method includes: converting the BIM model data structure or the oblique photography model data structure into a 3D Tiles data format on the B/S end, and using the Cesium map engine to load and render the BIM model data or the oblique photography model data.

It is understandable that at the client rendering level, in order to meet the 3D rendering requirements of the browser, the B/S end GPU rendering method is used for scene fusion. In addition, model slicing optimization and parameter tuning can be performed to achieve the purpose of improving the loading rate.

Specifically, the loading of real-life 3D data on the B/S side usually requires optimization of its data structure. The 3DTiles data format can be used to effectively stream and render large amounts of 3D geographic data. The Cesium map engine supports the 3DTiles data format and loads it according to the LOD level, which can effectively solve the problem of rendering massive data on the browser side.

Specifically, on the B/S side, the data structure of the BIM model and the data structure of the oblique photography model are converted into the 3DTiles data format, and the Cesium map engine is used to load and render the BIM model data and the oblique photography model data.

Among them, in order to further optimize the loading efficiency of 3dtiles, the embodiment of the present invention provides a set of optimization configuration parameters. For example, when loading a Cesium3DTileset object, the performance can be controlled by adding the maximum screen space error parameter in the Cesium map engine. The higher the value, the better the performance, but there will be low visual quality. After repeated testing, controlling the parameter value between 10-16 can effectively reduce the screen error rate and maintain the model refinement, ensuring the stable operation of the three-dimensional scene; in addition, the parameter of preferentially loading the central tile of the screen can be configured at the same time to improve the loading rate.

In a possible implementation manner, the fusion of the rendered BIM model and the oblique photography model to obtain a fused real-scene three-dimensional model includes: for the overlapping part of the BIM model and the oblique photography model, cropping the oblique photography model based on the Cesium three-dimensional earth framework; and fusing the BIM model into the cropped oblique photography model to generate a real-scene three-dimensional model.

It can be understood that, referring to FIG3, in order to solve the problem that the indoor scene cannot be browsed due to the overlap of the BIM model and the oblique photography model, the oblique photography model needs to be cropped, specifically by cropping the model based on the Cesium three-dimensional earth framework. Specifically, for the oblique photography model, the BIM model boundary is used as the initial polygonal cropping area, and the longitude and latitude coordinate values of each edge of the polygonal cropping area are recorded; the coordinates of the polygonal cropping area are corrected by matrix operation of the coordinate values based on the coordinate origin of the oblique photography model; the surface where each edge of the corrected polygonal cropping area is located is used as the cropping surface to obtain multiple cropping surfaces; the cropping surface is reconstructed according to the normal vector of any cropping surface and the shortest distance from the coordinate origin of the oblique photography model to any cropping surface, and finally multiple reconstructed cropping surfaces are obtained; based on the reconstructed multiple cropping surfaces, the oblique photography model is cropped.

The specific process of clipping the oblique photography model is as follows: in order to solve the problem that the indoor scene cannot be browsed due to the overlap between the BIM model and the oblique photography model, the oblique photography model needs to be clipped, and the model is clipped by building a ClippingPlane collection based on the Cesium earth framework.

Take the BIM model boundary as the initial polygon clipping area, _and record the latitude and longitude coordinate values of each vertex of the initial polygon clipping area, i = 0, 1, 2, ..., where They are The longitude, latitude and altitude coordinate values of the coordinates:

.

The inverse matrix of the oblique photography model origin matrix is used as the reference, and the recorded polygon vertices are calculated to generate the correction point coordinates based on the inverse matrix of the oblique photography origin. The rotated matrix ,in is the local coordinate value, which can be calculated by Cesium:

;

The converted coordinates ,in , , is the coordinate point The longitude, latitude, and altitude of , and so on, generate the corrected vertex coordinates:

;

;

.

Define an arbitrary vertical ground vector The vector formed by the corrected coordinate points Perform cross multiplication to generate the final required clipping surface normal vector , where i=0,1,2…

The ClippingPlane collection can be directly constructed through the parameter normal vector and the clipping plane coordinate point to achieve the clipping effect.

In a possible implementation manner, constructing a real-time perception model of a mobile target based on the coordinate points, movement trajectory and attribute information of indoor and outdoor mobile targets collected in real time includes: constructing a real-time perception model of the mobile target based on the coordinate points, movement trajectory and attribute information of the indoor and outdoor mobile targets sent by the monitoring device through the established WebSocket real-time transmission channel; integrating the constructed real-time perception model of the mobile target into the real-scene three-dimensional model, and displaying it on the front end also includes: updating the position and orientation of the real-time perception model of the mobile target based on the coordinate points and movement trajectory of the mobile target updated in real time by the monitoring device, and displaying the updated real-time perception model of the mobile target on the front end.

It is understandable that after the BIM model and the oblique photography model are integrated to generate a real-life 3D model of the indoor and outdoor scenes, it is necessary to construct a detailed model of each moving target therein. At this time, the actions of each moving target in the indoor and outdoor scenes are collected according to the monitoring devices set at various locations, for example, the coordinate points, movement trajectories and their own attribute information of each moving target are collected, wherein each moving target mainly refers to people and cars on the road, and the own attribute information of people and cars mainly includes the color, size, license plate, etc. of people and cars. According to the coordinate points, movement trajectories and their own attribute information of the moving targets, a real-time perception model of the moving targets is constructed. The constructed real-time perception models of each moving target are integrated into the real-life 3D model and displayed on the front end to present the people and car models and the real-life 3D model of the entire indoor and outdoor scenes.

Among them, the mobile target moves in real time, so the real-time perception model of the mobile target needs to be updated in real time. Specifically, the coordinate points and movement trajectory of the mobile target are updated in real time according to the monitoring devices set at various locations indoors and outdoors, and the position and orientation of the real-time perception model of the mobile target are updated, and the updated real-time perception model of the mobile target is displayed on the front end.

In a possible implementation manner, the real-time perception model of the mobile target includes a human and a vehicle model, and also includes: when the coordinate points, movement trajectories and attribute information of indoor and outdoor mobile targets are sent by the monitoring device through the established WebSocket real-time transmission channel, a correspondence between the real-time perception model ID of each mobile target and the time, position and orientation is established; the position and orientation of the real-time perception model of the mobile target is updated according to the coordinate points and movement trajectories of the mobile target updated in real time by the monitoring device, including: inputting the real-time perception model ID, time, position and orientation of the mobile target into the update orientation function headingPitchRollQuaternion to realize the update of the position and orientation of the real-time perception model of the corresponding mobile target at the corresponding time.

It can be understood that the process of updating the real-time perception model of each moving target in the real-scene three-dimensional model is as follows: monitoring devices can be set up at various locations indoors and outdoors, and each monitoring device and the front end realize data stream transmission by establishing a WebSocket real-time network transmission channel.

The monitoring device sends the collected coordinate points, movement trajectories and attribute information of indoor and outdoor mobile targets to the front end through the established WebSocket real-time transmission channel. When the front end receives the coordinate points, movement trajectories and attribute information of indoor and outdoor mobile targets sent by the monitoring device, a correspondence between the real-time perception model ID of each mobile target and the time, position and orientation is established.

As time goes by, when each mobile target moves, the position and orientation of the real-time perception model of the mobile target are updated according to the coordinate points and movement trajectory of the mobile target updated in real time by the monitoring device. Specifically, the real-time perception model ID, time, position, and orientation of the mobile target are input into the updated orientation function headingPitchRollQuaternion to realize the update of the position and orientation of the real-time perception model of the corresponding mobile target at the corresponding time, which can accurately present the movement of people and vehicles in real time.

Referring to FIG. 4 , the multi-model fusion method provided by the embodiment of the present invention is described. First, according to the aerial image, the BIM model is sliced and mapped, and the oblique photography model is sliced and mapped, and the BIM model and the oblique photography effect are rendered and fused to generate a rendered real-life 3D model. According to the moving trajectory of the mobile target collected in real time, the human and vehicle models in the real-life 3D model are updated in real time, and the updated real-life 3D model is displayed in real time.

Among them, the scenes used in the embodiment of the present invention are oblique photography near the company and the company's internal BIM scenes. The area of the three-dimensional real-life model after fusion of BIM is 1 square kilometer, and there are about 80 test models of people and vehicles after fusion and recognition; the test machine is an i5-8250U processor with integrated graphics card, and the loading of the fused scene model is not stuck. Since the real-time data transmission is only related to the network bandwidth and is not affected by the machine performance, the real-time transmission of AI recognition data is not affected by the machine. The loading of the real-life three-dimensional model has certain requirements for GPU rendering and is related to the amount of real-life three-dimensional data loaded. It is recommended to use a high-performance independent graphics card machine. Therefore, the machine performance needs to be improved when the real-life model of a large scene and the real-life model of the people and vehicles are fused in real time.

The embodiment of the present invention generates BIM models and oblique photography models of indoor and outdoor scenes based on aerial photography images; renders the BIM model and oblique photography model through the B/S end mode and adopts the GPU rendering mode; fuses the rendered BIM model and oblique photography model to obtain a fused real-scene three-dimensional model; constructs a real-time perception model of the mobile target based on the coordinate points, movement trajectory and attribute information of the indoor and outdoor mobile targets collected in real time, fuses the constructed real-time perception model of the mobile target into the real-scene three-dimensional model, and displays it on the front end. The embodiment of the present invention can fuse multiple models such as oblique photography models and BIM models, establish the corresponding relationship between each node of the model and the aerial photography image to realize batch mapping of the model, browse and access the oblique photography data and BIM data model on the browser side through the B/S end and based on GPU and server rendering technology, so as to improve the user's browsing experience; establish WebSocket network communication, establish a one-to-one correspondence between time and position, and smoothly, fluently and accurately realize the dynamic update of the coordinate trajectory points of the human and vehicle models.

FIG5 is a structural diagram of a multi-model fusion system provided by an embodiment of the present invention. As shown in FIG5 , a multi-model fusion system includes a mapping module 501, a rendering module 502, a fusion module 503 and a display module 504, wherein:

A mapping module 501 is used to perform mapping processing on the BIM model according to the aerial photography image to generate a BIM model and an oblique photography model of the indoor and outdoor scenes;

A rendering module 502 is used to render the BIM model and the oblique photography model of the indoor and outdoor scenes by using a B/S terminal mode and a GPU rendering mode;

A fusion module 503 is used to fuse the rendered BIM model and the oblique photography model to obtain a fused real-scene 3D model;

The display module 504 is used to construct a real-time perception model of the mobile target based on the coordinate points, movement trajectory and attribute information of the indoor and outdoor mobile targets collected in real time, integrate the constructed real-time perception model of the mobile target into the real-scene three-dimensional model, and display it on the front end.

The multi-model fusion system provided in an embodiment of the present invention corresponds to the multi-model fusion method provided in the aforementioned embodiments. The relevant technical features of the multi-model fusion system can refer to the relevant technical features of the multi-model fusion method, which will not be repeated here.

Please refer to Figure 6, which is a schematic diagram of an embodiment of an electronic device provided by an embodiment of the present application. As shown in Figure 6, an embodiment of the present application provides an electronic device, including a memory 610, a processor 620, and a computer program 611 stored in the memory 610 and executable on the processor 620. When the processor 620 executes the computer program 611, the following steps are implemented: according to the aerial photography image, the BIM model is mapped to generate the BIM model and the oblique photography model of the indoor and outdoor scenes; the BIM model and the oblique photography model of the indoor and outdoor scenes are rendered by the B/S end mode and the GUP rendering mode; the rendered BIM model and the oblique photography model are fused to obtain a fused real-life three-dimensional model; according to the coordinate points, movement trajectories and attribute information of the indoor and outdoor mobile targets collected in real time, a real-time perception model of the mobile target is constructed, the constructed real-time perception model of the mobile target is integrated into the real-life three-dimensional model, and displayed on the front end.

Please refer to Figure 7, which is a schematic diagram of an embodiment of a computer-readable storage medium provided by an embodiment of the present application. As shown in Figure 7, this embodiment provides a computer-readable storage medium 700, on which a computer program 711 is stored, and when the computer program 711 is executed by a processor, the following steps are implemented: according to the aerial photography image, the BIM model is mapped to generate the BIM model and the oblique photography model of the indoor and outdoor scenes; the BIM model and the oblique photography model of the indoor and outdoor scenes are rendered through the B/S end mode and the GUP rendering mode; the rendered BIM model and the oblique photography model are fused to obtain a fused real-life three-dimensional model; according to the coordinate points, movement trajectories and attribute information of the indoor and outdoor mobile targets collected in real time, a real-time perception model of the mobile target is constructed, and the constructed real-time perception model of the mobile target is integrated into the real-life three-dimensional model, and displayed on the front end.

It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

The present application is described with reference to the flowchart and/or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each process and/or box in the flowchart and/or block diagram, as well as the combination of the process and/or box in the flowchart and/or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate a device for implementing the functions specified in one process or multiple processes in the flowchart and/or one box or multiple boxes in the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims (8)

1. A multi-model fusion method based on smart city, characterized by comprising: According to the aerial photography images, the BIM model is mapped to generate the BIM model of indoor and outdoor scenes; Render the BIM model and oblique photography model of indoor and outdoor scenes through B/S end and GPU rendering; The rendered BIM model and the oblique photography model are fused to obtain a fused real-scene 3D model; According to the coordinate points, movement trajectory and attribute information of indoor and outdoor moving targets collected in real time, a real-time perception model of the moving target is constructed, and the constructed real-time perception model of the moving target is integrated into the real-scene 3D model and displayed on the front end; The fusion of the rendered BIM model and the oblique photography model to obtain a fused real-scene 3D model includes: For the overlapping parts of the BIM model and the oblique photography model, the oblique photography model is cropped based on the Cesium 3D earth framework; The BIM model is integrated into the cropped oblique photography model to generate a real-life 3D model; For the overlapping part of the BIM model and the oblique photography model, the oblique photography model is clipped based on the Cesium three-dimensional earth framework, including: For the oblique photography model, the BIM model boundary is used as the initial polygonal clipping area, and the longitude and latitude coordinate values of each edge of the polygonal clipping area are recorded; Taking the coordinate origin of the oblique photography model as a reference, matrix operation of coordinate values is performed to correct the coordinates of the polygonal clipping area; Taking the surface where each edge of the corrected polygonal clipping area is located as the clipping surface, a plurality of clipping surfaces are obtained; Reconstructing the clipping surface according to the normal vector of any clipping surface and the shortest distance from the coordinate origin of the oblique photography model to any clipping surface, and finally obtaining a plurality of reconstructed clipping surfaces; The oblique photography model is cropped based on the reconstructed multiple cropping planes. 2. The multi-model fusion method according to claim 1 is characterized in that the step of mapping the BIM model according to the aerial photography image to generate the BIM model of the indoor and outdoor scenes comprises: Parsing the IFC format used by the BIM model and splitting the nodes of the BIM model; Establish the corresponding relationship between each node of the BIM model and the aerial photography image; The aerial photography images corresponding to each node of the BIM model are imported into the BIM model, and the BIM model is batch mapped to generate the BIM model of indoor and outdoor scenes. 3. The multi-model fusion method according to claim 1 or 2, characterized in that the rendering of the BIM model and the oblique photography model of the indoor and outdoor scenes by means of a B/S terminal and a GPU rendering method comprises: On the B/S side, the BIM model data structure is converted into 3D Tiles data format, and the Cesium map engine is used to load and render the BIM model data. 4. The multi-model fusion method according to claim 3, further comprising: By setting the maximum screen space error parameter in the Cesium map engine, the screen error rate can be reduced and the level of detail of the BIM model can be maintained; Improve the loading speed of BIM model data by configuring the parameters of loading the central block of the screen first. 5. The multi-model fusion method according to claim 1, characterized in that the step of constructing a real-time perception model of a moving target based on coordinate points, moving trajectories and attribute information of indoor and outdoor moving targets collected in real time comprises: According to the coordinate points, movement trajectories and attribute information of indoor and outdoor mobile targets sent by the monitoring device through the established WebSocket real-time transmission channel, a real-time perception model of the mobile target is constructed; The method further includes integrating the constructed real-time perception model of the moving target into the real-scene three-dimensional model and displaying it on the front end: According to the coordinate points and movement trajectory of the moving target updated in real time by the monitoring device, the position and orientation of the real-time perception model of the moving target are updated, and the updated real-time perception model of the moving target is displayed on the front end. 6. The multi-model fusion method according to claim 5, characterized in that the real-time perception model of the mobile target includes a human and a vehicle model, and further includes: When receiving the coordinate points, movement trajectories and attribute information of indoor and outdoor mobile targets sent by the monitoring device through the established WebSocket real-time transmission channel, a corresponding relationship between the real-time perception model ID of each mobile target and the time, position and orientation is established; The updating of the position and orientation of the real-time perception model of the moving target according to the coordinate points and movement trajectory of the moving target updated in real time by the monitoring device comprises: The real-time perception model ID, time, position, and orientation of the mobile target are input into the heading update function headingPitchRollQuaternion to update the position and orientation of the real-time perception model of the corresponding mobile target at the corresponding time. 7. A multi-model fusion system, characterized by comprising: The mapping module is used to map the BIM model based on the aerial photography images to generate the BIM model of the indoor and outdoor scenes; A rendering module is used to render the BIM model and oblique photography model of indoor and outdoor scenes through the B/S end and the GPU rendering method; A fusion module is used to fuse the rendered BIM model and the oblique photography model to obtain a fused real-scene 3D model; The display module is used to construct a real-time perception model of the moving target based on the coordinate points, movement trajectory and attribute information of the indoor and outdoor moving targets collected in real time, integrate the constructed real-time perception model of the moving target into the real-scene three-dimensional model, and display it on the front end; The fusion of the rendered BIM model and the oblique photography model to obtain a fused real-scene 3D model includes: For the overlapping parts of the BIM model and the oblique photography model, the oblique photography model is cropped based on the Cesium 3D earth framework; The BIM model is integrated into the cropped oblique photography model to generate a real-life 3D model; For the overlapping part of the BIM model and the oblique photography model, the oblique photography model is clipped based on the Cesium three-dimensional earth framework, including: For the oblique photography model, the BIM model boundary is used as the initial polygonal clipping area, and the longitude and latitude coordinate values of each edge of the polygonal clipping area are recorded; Taking the coordinate origin of the oblique photography model as a reference, matrix operation of coordinate values is performed to correct the coordinates of the polygonal clipping area; Taking the surface where each edge of the corrected polygonal clipping area is located as the clipping surface, a plurality of clipping surfaces are obtained; Reconstructing the clipping surface according to the normal vector of any clipping surface and the shortest distance from the coordinate origin of the oblique photography model to any clipping surface, and finally obtaining a plurality of reconstructed clipping surfaces; The oblique photography model is cropped based on the reconstructed multiple cropping planes. 8. An electronic device, characterized in that it comprises a memory and a processor, wherein the processor is used to implement the steps of the multi-model fusion method as described in any one of claims 1 to 6 when executing a computer management program stored in the memory.