CN114913105A - Laser point cloud fusion method, device, server and computer-readable storage medium - Google Patents

Abstract

The present application is applicable to the technical field of image processing, and provides a laser point cloud fusion method, device, server, and computer-readable storage medium. The method includes: acquiring a point cloud to be processed; wherein, the point cloud to be processed includes a first to-be-processed point cloud. point cloud and the second point cloud to be processed; fusing the first point cloud to be processed and the second point cloud to be processed to obtain the target point cloud. It can be seen that the present application can improve the accuracy of laser point cloud fusion.

Description

Laser point cloud fusion method, device, server and computer-readable storage medium

technical field

The present application belongs to the technical field of point clouds, and in particular, relates to a laser point cloud fusion method, device, server, and computer-readable storage medium.

Background technique

With the continuous development of sensing technology and surveying and mapping equipment, there are various data sources. In the field of 3D reconstruction, it is necessary to obtain comprehensive point cloud data to make the reconstruction effect better. However, the fusion accuracy between existing different types of point cloud data is low.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a laser point cloud fusion method, device, server, and computer-readable storage medium, which can solve the technical problem of low point cloud fusion accuracy in the prior art.

In a first aspect, an embodiment of the present application provides a laser point cloud fusion method, including:

Obtain a point cloud to be processed; wherein, the point cloud to be processed includes a first point cloud to be processed and a second point cloud to be processed;

The target point cloud is obtained by fusing the first point cloud to be processed and the second point cloud to be processed.

In a possible implementation of the first aspect, acquiring the point cloud to be processed includes:

acquiring the first image to be processed and the first laser data;

acquiring the second to-be-processed image and the second laser data;

performing coordinate system calibration on a first coordinate system where the first laser data is located and a second coordinate system where the second laser data is located;

matching the first image to be processed and the second image to be processed;

generating a first point cloud to be processed according to the matched first image to be processed and the first laser data after coordinate system calibration;

A second point cloud to be processed is generated according to the matched second image to be processed and the second laser data after coordinate system calibration.

In a possible implementation manner of the first aspect, matching the first image to be processed and the second image to be processed includes:

extracting the first feature point of the first image to be processed;

extracting second feature points of the second image to be processed;

The target feature points common to the first feature point and the second feature point are screened out.

In a possible implementation manner of the first aspect, the first point cloud to be processed and the second point cloud to be processed are fused to obtain a target point cloud, including:

Obtain pose information according to the target feature point;

According to the pose information, the first point cloud to be processed and the second point cloud to be processed are fused to obtain a target point cloud.

In a possible implementation manner of the first aspect, after fusing the first point cloud to be processed and the second point cloud to be processed to obtain the target point cloud, the method further includes:

Optimize the target point cloud.

In a second aspect, an embodiment of the present application provides a laser point cloud fusion device, including:

an acquisition module, configured to acquire a point cloud to be processed; wherein, the point cloud to be processed includes a first point cloud to be processed and a second point cloud to be processed;

A fusion module, configured to fuse the first point cloud to be processed and the second point cloud to be processed to obtain a target point cloud.

In an optional implementation manner of the second aspect, the obtaining module includes:

a first acquisition sub-module for acquiring the first image to be processed and the first laser data;

a second acquisition sub-module for acquiring the second to-be-processed image and the second laser data;

a calibration sub-module for performing coordinate system calibration on the first coordinate system where the first laser data is located and the second coordinate system where the second laser data is located;

a matching submodule for matching the first image to be processed and the second image to be processed;

a first generation submodule, configured to generate a first point cloud to be processed according to the matched first image to be processed and the first laser data after coordinate system calibration;

The second generation sub-module is configured to generate a second point cloud to be processed according to the matched second image to be processed and the second laser data after coordinate system calibration.

In an optional implementation manner of the second aspect, the matching submodule includes:

a first extraction unit, configured to extract the first feature point of the first image to be processed;

a second extraction unit, configured to extract the second feature point of the second to-be-processed image;

A screening unit, configured to screen out target feature points common to the first feature point and the second feature point.

In an optional implementation manner of the second aspect, the fusion module includes:

a third acquisition sub-module, configured to acquire pose information according to the target feature point;

The fusion sub-module is configured to fuse the first point cloud to be processed and the second point cloud to be processed according to the pose information to obtain a target point cloud.

In an optional implementation manner of the second aspect, the device further includes:

An optimization module for optimizing the target point cloud.

In a third aspect, an embodiment of the present application provides a server, including a memory, a processor, and a computer program stored in the memory and executable on the processor, which is implemented when the processor executes the computer program The method as described in the first aspect above.

In a fourth aspect, an embodiment of the present application provides a readable computer-readable storage medium, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

The beneficial effects that the embodiments of the present application have compared with the prior art are:

The embodiment of the present application obtains the point cloud to be processed; wherein the point cloud to be processed includes a first point cloud to be processed and a second point cloud to be processed; and the first point cloud to be processed and the second point cloud to be processed are obtained by fusing the first point cloud to be processed and the second point cloud to be processed Target point cloud to improve the accuracy of laser point cloud fusion.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only for the present application. In some embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort.

1 is a schematic flowchart of a laser point cloud fusion method provided by an embodiment of the present application;

2 is a structural block diagram of a laser point cloud fusion device provided by an embodiment of the present application;

FIG. 3 is a schematic structural diagram of a server provided by an embodiment of the present application.

Detailed ways

In the following description, for the purpose of illustration rather than limitation, specific details such as a specific system structure and technology are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It is to be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described feature, integer, step, operation, element and/or component, but does not exclude one or more other The presence or addition of features, integers, steps, operations, elements, components and/or sets thereof.

It will also be understood that, as used in this specification and the appended claims, the term "and/or" refers to and including any and all possible combinations of one or more of the associated listed items.

As used in the specification of this application and the appended claims, the term "if" may be contextually interpreted as "when" or "once" or "in response to determining" or "in response to detecting ". Similarly, the phrases "if it is determined" or "if the [described condition or event] is detected" may be interpreted, depending on the context, to mean "once it is determined" or "in response to the determination" or "once the [described condition or event] is detected. ]" or "in response to detection of the [described condition or event]".

In addition, in the description of the specification of the present application and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the description, and should not be construed as indicating or implying relative importance.

References in this specification to "one embodiment" or "some embodiments" and the like mean that a particular feature, structure or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in other embodiments," etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean "one or more but not all embodiments" unless specifically emphasized otherwise. The terms "including", "including", "having" and their variants mean "including but not limited to" unless specifically emphasized otherwise.

The technical solutions provided by the embodiments of the present application will be introduced below through specific embodiments.

Referring to FIG. 1, which is a schematic flowchart of a laser point cloud fusion method provided by an embodiment of the present application, as an example and not a limitation, the method can be applied to a server, and the method can include the following steps:

Step S101, acquiring a point cloud to be processed.

The point cloud to be processed includes a first point cloud to be processed and a second point cloud to be processed.

It should be noted that the first point cloud to be processed and the second point cloud to be processed are generated by photographing the same target object by different types of laser scanning devices.

The first point cloud to be processed and the second point cloud to be processed correspond to the same photographing object, the first point cloud to be processed may be photographed by a ground laser scanner, and the second point cloud to be processed may be photographed by a drone.

In specific applications, the point cloud to be processed is obtained, including:

Step S101-1, acquiring the first image to be processed and the first laser data.

Step S101-2, acquiring a second image to be processed and second laser data.

The first image to be processed and the first laser data may be captured by a terrestrial laser scanner, the first image to be processed may be a panoramic image, and the first laser data includes the depth of each pixel in the first image to be processed value; the second image to be processed and the second laser data can be obtained from drone aerial photography, the second image to be processed can be an ordinary image, and the second laser data contains the depth value of each pixel in the second image to be processed . In addition, the ground laser scanner can also obtain the first GPS data of the target object, and the drone can also obtain the second GPS data of the target object.

Step S101-3, performing coordinate system calibration on the first coordinate system where the first laser data is located and the second coordinate system where the second laser data is located.

Specifically, the first GPS data is located in the first coordinate system, the second GPS data is located in the second coordinate system, and the coordinates of the first GPS data and the second GPS data are aligned, that is, the coordinate alignment of the first GPS data and the second GPS data is performed. Coordinate system calibration is performed on the obtained first coordinate system and the second coordinate system where the second laser data is located.

Step S101-4, matching the first image to be processed and the second image to be processed.

Exemplarily, matching the first image to be processed and the second image to be processed includes:

Step S101-4-1, extract the first feature point of the first image to be processed.

Exemplarily, a feature extraction algorithm is used to extract the first feature point of the first image to be processed.

Step S101-4-2, extract the second feature point of the second image to be processed.

Exemplarily, a feature extraction algorithm is used to extract the first feature point of the first image to be processed.

It should be noted that the feature extraction algorithm is the AKAZE algorithm. The AKAZE feature algorithm is an improved version of the SIFT feature algorithm, but does not use Gaussian blur to construct the scale space, because the Gaussian blur has the disadvantage of losing edge information, and then uses nonlinear diffusion filtering to construct the scale space, so as to retain more of the image. edge features.

Step S101-4-3, screening out target feature points common to the first feature point and the second feature point.

Exemplarily, the first feature point and the second feature point are input to a pre-trained neural network, and a common target feature point is output.

Optionally, before filtering out the common target feature points in the first feature point and the second feature point, the method further includes: training a neural network.

It is understandable that the existing algorithm is used to extract the rough match first, the sfm algorithm is used for dense reconstruction, the effective points are screened, and a neural network is trained to realize the matching between the UAV shooting and the panoramic image.

The training process of a neural network includes an initial step and an iterative step. Among them, the initial step is the rough matching between the ordinary image and the panoramic image. The panoramic image already in the model can be sliced, and the ordinary image after slicing the panoramic image can be matched with the feature points of the ordinary image. The initial step can also convert the image of the UAV into a part of the panoramic image, so as to realize the matching of the panorama to the panorama. The iteration step directly uses the matching of the UAV image and the panoramic image to iterate. The model of the neural network may be a neural network model that performs feature point matching on an ordinary image and a panoramic image, such as a SuperGlue model, a variant of the SuperGlue model, or an OANet model.

Step S101-5: Generate a first point cloud to be processed according to the matched first image to be processed and the first laser data after coordinate system calibration.

Exemplarily, the three-dimensional coordinates of the first point cloud to be processed are obtained according to the following formula:

，

,

Among them, (u, v) is the pixel coordinate of each feature point in the first image to be processed, d is the depth value of each pixel point in the first image to be processed, K is the internal parameter of the ground laser scanner, (X, Y, Z) are the three-dimensional coordinates of the first point cloud to be processed.

Step S101-6: Generate a second point cloud to be processed according to the matched second image to be processed and the second laser data after coordinate system calibration.

Exemplarily, the three-dimensional coordinates of the second point cloud to be processed are obtained according to the following formula:

，

,

Among them, (u, v) is the pixel coordinate of each feature point in the second image to be processed, d is the depth value of each pixel in the second image to be processed, K is the internal parameter of the drone, (X, Y , Z) are the three-dimensional coordinates of the second point cloud to be processed.

Step S102, fusing the first point cloud to be processed and the second point cloud to be processed to obtain a target point cloud.

In a specific application, the first point cloud to be processed and the second point cloud to be processed are fused to obtain the target point cloud, including:

Step S102-1, obtaining pose information according to the target feature point.

Among them, the target feature points are matched according to the SFM algorithm to obtain the pose information.

Step S102-2, according to the pose information, the first point cloud to be processed and the second point cloud to be processed are fused to obtain a target point cloud.

Exemplarily, according to the pose information, the ICP algorithm is used to fuse the first point cloud to be processed and the second point cloud to be processed to obtain the target point cloud.

In an optional implementation manner, after fusing the first point cloud to be processed and the second point cloud to be processed to obtain the target point cloud, the method further includes:

Optimize the target point cloud.

It can be understood that after the first point cloud to be processed and the second point cloud to be processed are fused, after obtaining the target point cloud, some point clouds will overlap. First, the synthesized point cloud is rasterized, and each grid is rasterized. Calculate the density A and visibility B of the two parts of the point cloud (the angle between the point cloud normal and the camera position and the line connecting the point cloud). The credibility C = A * (cosB). Which part of the point cloud is more credible, and which part is selected for the final synthesized point cloud, so as to achieve the purpose of optimizing the target point cloud.

In the embodiment of the present application, the point cloud to be processed is obtained by acquiring the point cloud to be processed; wherein the point cloud to be processed includes a first point cloud to be processed and a second point cloud to be processed; the first point cloud to be processed and the second point cloud to be processed are fused , get the target point cloud and improve the accuracy of laser point cloud fusion.

It should be understood that the size of the sequence numbers of the steps in the above embodiments does not mean the sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

Corresponding to the methods described in the above embodiments, FIG. 2 shows a structural block diagram of the laser point cloud fusion apparatus provided by the embodiments of the present application. For convenience of description, only the parts related to the embodiments of the present application are shown.

Referring to Figure 2, the device includes:

The acquiring module 21 is used for acquiring the point cloud to be processed; wherein, the point cloud to be processed includes a first point cloud to be processed and a second point cloud to be processed;

The fusion module 22 is configured to fuse the first point cloud to be processed and the second point cloud to be processed to obtain a target point cloud.

In an optional implementation manner, the acquisition module includes:

a first acquisition sub-module for acquiring the first image to be processed and the first laser data;

a second acquisition sub-module for acquiring the second to-be-processed image and the second laser data;

a calibration sub-module for performing coordinate system calibration on the first coordinate system where the first laser data is located and the second coordinate system where the second laser data is located;

a matching submodule for matching the first image to be processed and the second image to be processed;

a first generation submodule, configured to generate a first point cloud to be processed according to the matched first image to be processed and the first laser data after coordinate system calibration;

The second generation sub-module is configured to generate a second point cloud to be processed according to the matched second image to be processed and the second laser data after coordinate system calibration.

In an optional embodiment, the matching submodule includes:

a first extraction unit, configured to extract the first feature point of the first image to be processed;

a second extraction unit, configured to extract the second feature point of the second to-be-processed image;

A screening unit, configured to screen out target feature points common to the first feature point and the second feature point.

In an optional embodiment, the fusion module includes:

a third acquisition sub-module, configured to acquire pose information according to the target feature point;

The fusion sub-module is configured to fuse the first point cloud to be processed and the second point cloud to be processed according to the pose information to obtain a target point cloud.

In an optional embodiment, the device further includes:

An optimization module for optimizing the target point cloud.

It should be noted that the information exchange, execution process and other contents between the above-mentioned devices/units are based on the same concept as the method embodiments of the present application. For specific functions and technical effects, please refer to the method embodiments section. It is not repeated here.

FIG. 3 is a schematic structural diagram of a server provided by an embodiment of the present application. As shown in FIG. 3, the server 3 of this embodiment includes: at least one processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the at least one processor 30, the processing When the computer 30 executes the computer program 32, the steps in any of the foregoing method embodiments are implemented.

The server 3 may be a computing device such as a cloud server. The server may include, but is not limited to, the processor 30 and the memory 31 . Those skilled in the art can understand that FIG. 3 is only an example of the server 3, and does not constitute a limitation on the server 3. It may include more or less components than the one shown in the figure, or combine some components, or different components, such as It may also include input and output devices, network access devices, and the like.

The so-called processor 30 may be a central processing unit (Central Processing Unit, CPU), and the processor 30 may also be other general-purpose processors, digital signal processors (Digital Signal Processors, DSP), application specific integrated circuits (Application Specific Integrated Circuits) , ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 31 may be an internal storage unit of the server 3 in some embodiments, such as a hard disk or a memory of the server 3 . In other embodiments, the memory 31 may also be an external storage device of the server 3, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure) Digital, SD) card, flash card (Flash Card), etc. Further, the memory 31 may also include both an internal storage unit of the server 3 and an external storage device. The memory 31 is used to store an operating system, an application program, a boot loader (Boot Loader), data, and other programs, such as program codes of the computer program. The memory 31 can also be used to temporarily store data that has been output or will be output.

Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned functions can be allocated to different functional units, Module completion, that is, dividing the internal structure of the device into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated in one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit, and the above-mentioned integrated units may adopt hardware. It can also be realized in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing from each other, and are not used to limit the protection scope of the present application. For the specific working processes of the units and modules in the above-mentioned system, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

Embodiments of the present application further provide a computer-readable storage medium, where the computer-readable storage medium is preferably a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and the computer program When executed by the processor, the implementation can implement the steps in the above-mentioned various method embodiments.

The embodiments of the present application provide a computer program product, when the computer program product runs on a mobile terminal, the steps in the foregoing method embodiments can be implemented when the mobile terminal executes the computer program product.

The integrated units, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable computer-readable storage medium. Based on this understanding, the present application realizes all or part of the processes in the methods of the above embodiments, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When executed by a processor, the steps of each of the above method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form, and the like. The computer-readable medium may include at least: any entity or device capable of carrying the computer program code to the photographing device/terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signals, telecommunication signals, and software distribution media. For example, U disk, mobile hard disk, disk or CD, etc. In some jurisdictions, under legislation and patent practice, computer readable media may not be electrical carrier signals and telecommunications signals.

In the foregoing embodiments, the description of each embodiment has its own emphasis. For parts that are not described or described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

In the embodiments provided in this application, it should be understood that the disclosed apparatus/network device and method may be implemented in other manners. For example, the apparatus/network device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units. Or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the above-mentioned embodiments, those of ordinary skill in the art should understand that: it can still be used for the above-mentioned implementations. The technical solutions described in the examples are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the application, and should be included in the within the scope of protection of this application.

Claims (10)

1. A laser point cloud fusion method is characterized by comprising the following steps: a

Acquiring point clouds to be processed; the point clouds to be processed comprise a first point cloud to be processed and a second point cloud to be processed;

and fusing the first point cloud to be processed and the second point cloud to be processed to obtain a target point cloud.

2. The laser point cloud fusion method of claim 1, wherein obtaining the point cloud to be processed comprises:

acquiring a first image to be processed and first laser data;

acquiring a second image to be processed and second laser data;

calibrating a coordinate system of a first coordinate system where the first laser data are located and a second coordinate system where the second laser data are located;

matching the first image to be processed and the second image to be processed;

generating a first point cloud to be processed according to the matched first image to be processed and the first laser data after the coordinate system is calibrated;

and generating a second point cloud to be processed according to the matched second image to be processed and the second laser data after the coordinate system is calibrated.

3. The laser point cloud fusion method of claim 2, wherein matching the first to-be-processed image and the second to-be-processed image comprises:

extracting a first feature point of the first image to be processed;

extracting a second feature point of the second image to be processed;

and screening out target characteristic points which are common to the first characteristic points and the second characteristic points.

4. The laser point cloud fusion method of claim 3, wherein fusing the first point cloud to be processed and the second point cloud to be processed to obtain a target point cloud comprises:

acquiring pose information according to the target feature points;

and fusing the first point cloud to be processed and the second point cloud to be processed according to the pose information to obtain a target point cloud.

5. The laser point cloud fusion method of claim 1, wherein after fusing the first point cloud to be processed and the second point cloud to be processed to obtain a target point cloud, the method further comprises:

optimizing the target point cloud.

6. A laser point cloud fusion device, comprising:

the acquisition module is used for acquiring point clouds to be processed; the point clouds to be processed comprise a first point cloud to be processed and a second point cloud to be processed;

and the fusion module is used for fusing the first point cloud to be processed and the second point cloud to be processed to obtain a target point cloud.

7. The laser point cloud fusion apparatus of claim 6, wherein the acquisition module comprises:

the first acquisition submodule is used for acquiring a first image to be processed and first laser data;

the second acquisition submodule is used for acquiring a second image to be processed and second laser data;

the calibration sub-module is used for calibrating a coordinate system of a first coordinate system where the first laser data are located and a second coordinate system where the second laser data are located;

the matching submodule is used for matching the first image to be processed with the second image to be processed;

the first generation submodule is used for generating a first point cloud to be processed according to the matched first image to be processed and the first laser data after the coordinate system is calibrated;

and the second generation submodule is used for generating a second point cloud to be processed according to the matched second image to be processed and the second laser data after the coordinate system is calibrated.

8. The laser point cloud fusion apparatus of claim 7, wherein the matching sub-module comprises:

a first extraction unit, configured to extract a first feature point of the first image to be processed;

the second extraction unit is used for extracting second feature points of the second image to be processed;

and the screening unit is used for screening out the target characteristic points which are common to the first characteristic points and the second characteristic points.

9. A server comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 5 when executing the computer program.

10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 5.

Images