CN114862964A - Sensor automatic calibration method, electronic device and storage medium - Google Patents

Abstract

The embodiment of the invention provides an automatic calibration method of a sensor, electronic equipment and a storage medium. The method comprises the following steps: acquiring laser point cloud data acquired by laser equipment and image data acquired by a camera; according to the driving state of the mobile device, identifying a scene target related to the driving state in the laser point cloud data to obtain a first coordinate of the scene target under a laser point cloud coordinate system; according to the driving state of the mobile device, identifying a scene target related to the driving state in the image data to obtain a second coordinate of the scene target in an image coordinate system; re-projecting the first coordinate into a third coordinate under an image coordinate system; matching the second coordinate with the third coordinate under an image coordinate system to obtain a matching result; and calibrating external parameters between the laser equipment and the camera based on the matching result. The embodiment of the invention does not depend on the traditional calibration scene, and can realize more accurate automatic calibration of external parameters between the laser equipment and the camera.

Description

Sensor automatic calibration method, electronic device and storage medium

technical field

The invention relates to the field of automatic driving, in particular to a sensor automatic calibration method, an electronic device and a storage medium.

Background technique

Autonomous driving is usually composed of a perception system, a decision-making system, an execution system, and a communication system. Due to the high safety requirements of autonomous driving, it is necessary for sensors to collect more sufficient information about the surrounding environment to make reliable inferences.

However, there is no single sensor that can perceive the features that include the distance and the object information is comprehensive. Therefore, in order to obtain more comprehensive features, reliable sensor perception integration solutions are lidar and camera. Lidar can better perceive the position information of surrounding objects, and the accuracy is high, but the data of Lidar is relatively sparse, and it cannot give clear type characteristics to some obstacles encountered in automatic driving. At the same time, it may fail in the case of rain, snow, light dust, etc. Although such applications can be optimized through the scheme of multiple echoes, the back end still needs to filter abnormal targets. The camera can better identify the semantic information in the scene and identify traffic obstacles, but the segmentation is easy to fail in some scenes, such as objects with unclear boundaries, which may make the overall detection abnormal.

Therefore, combining the features of lidar and camera with relevant information can better assist autonomous driving perception operations. Before the fusion is used, the parameters need to be calibrated, so that there is no obvious deviation between the corresponding target position in the laser and the target position of the camera. So as to better complete the fusion perception of the target.

In the process of realizing the present invention, the inventor found that there are at least the following problems in the related art:

The integration of the camera and the lidar has been fixed, and the relative position of the integrated and fixed camera and lidar may change due to bumps, sudden braking, etc. during the driving process of the autonomous vehicle. At this time, the calibration of the camera and lidar during integration The parameters will no longer be applicable, and the external parameters between the camera and the lidar need to be re-calibrated. However, for the integrated and fixed sensors, it is difficult to find a calibration environment in a suitable scene, and the specific scene parameters require special targets. Objects, it is more difficult to re-calibrate. Therefore, how to realize automatic calibration of external parameters of cameras and lidars has become a technical problem that needs to be solved urgently.

SUMMARY OF THE INVENTION

The technical solution of the present invention solves the technical problem in the prior art that the external parameters of the camera and the laser radar cannot be automatically calibrated when the relative position of the camera and the laser radar changes during the driving process of the automatic driving vehicle. In a first aspect, an embodiment of the present invention provides an automatic sensor calibration method, applied to a mobile device, including:

Obtain laser point cloud data collected by laser equipment;

Obtain the image data collected by the camera;

According to the driving state of the mobile device, identify the scene target related to the driving state in the laser point cloud data, and obtain the first coordinates of the scene target in the laser point cloud coordinate system;

According to the driving state of the mobile device, identify the scene object related to the driving state in the image data, and obtain the second coordinate of the scene object in the image coordinate system;

reprojecting the first coordinate into a third coordinate in the image coordinate system;

Matching the second coordinate and the third coordinate in the image coordinate system to obtain a matching result;

Based on the matching result, the external parameters between the laser device and the camera are calibrated.

In a second aspect, an embodiment of the present invention provides an automatic sensor calibration device, characterized in that it includes:

Point cloud acquisition module, used to acquire laser point cloud data collected by laser equipment;

The image acquisition module is used to acquire the image data collected by the camera;

The first coordinate determination module is used to identify the scene target related to the driving state in the laser point cloud data according to the driving state of the mobile device, and obtain the scene target in the laser point cloud coordinate system. first coordinate;

The second coordinate determination module is configured to identify the scene object related to the driving state in the image data according to the driving state of the mobile device, and obtain the second coordinate of the scene object in the image coordinate system. coordinate;

A third coordinate determination module, configured to reproject the first coordinate to a third coordinate in the image coordinate system;

a matching module, configured to match the second coordinate and the third coordinate in the image coordinate system to obtain a matching result;

A calibration module, configured to calibrate an external parameter between the laser device and the camera based on the matching result.

In a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, The instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the automatic sensor calibration method of any embodiment of the present invention.

In a fourth aspect, an embodiment of the present invention provides a mobile device, including a main body and the electronic device according to any embodiment of the present invention installed on the main body.

In a fifth aspect, an embodiment of the present invention provides a storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the steps of the automatic sensor calibration method of any embodiment of the present invention are implemented.

In a sixth aspect, an embodiment of the present invention further provides a computer program product, which, when the computer program product runs on a computer, enables the computer to execute the automatic sensor calibration method described in any one of the embodiments of the present invention.

The beneficial effects of the embodiments of the present invention are as follows: the technical scheme of the present invention selects a corresponding scene target based on the driving state of the mobile device, matches the laser point cloud coordinates of the scene target and the image coordinates, and realizes the matching between the camera and the lidar according to the matching result. The automatic calibration of the external parameters between the laser equipment and the camera does not depend on the traditional calibration scene, and can realize more accurate automatic calibration of the calibration between the laser equipment and the camera.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1 is a flowchart of a method for automatic calibration of a sensor provided by an embodiment of the present invention;

2 is a schematic diagram of a signboard point cloud feature filtered by height and reflectivity in point cloud data of a sensor automatic calibration method provided by an embodiment of the present invention;

3 is a schematic diagram of a road sign detection effect in image data of an automatic sensor calibration method provided by an embodiment of the present invention;

4 is a schematic diagram of a traffic sign point cloud and image fusion display diagram of a sensor automatic calibration method provided by an embodiment of the present invention;

5 is a schematic diagram of pedestrian and vehicle projection matching between an image and a point cloud of an automatic sensor calibration method provided by an embodiment of the present invention;

6 is a flow chart of laser point cloud and image fusion calibration of a sensor automatic calibration method provided by an embodiment of the present invention;

7 is a schematic structural diagram of a sensor automatic calibration device provided by an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of an embodiment of an electronic device for automatic sensor calibration provided by an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Those skilled in the art know that the embodiments of the present application can be implemented as a system, apparatus, device, method or computer program product. Accordingly, the present disclosure may be embodied in entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

For ease of understanding, the following technical terms involved in this application are explained:

The "mobile device" referred to in this application can be any equipment with mobile capabilities, including but not limited to automobiles, ships, submarines, airplanes, aircrafts and other equipment, wherein automobiles include the Society of Automotive Engineers International (SAE International, SAE International) ) or vehicles with six automatic driving technology levels from L0 to L5 formulated by the Chinese national standard "Automotive Driving Automation Classification", hereinafter referred to as ADV (Auto-Driving Vehicle).

The "autonomous driving vehicle ADV" referred to in this application may be a vehicle device or a robot device with the following various functions:

(1) Manned functions, such as family cars, buses, etc.;

(2) Cargo-carrying functions, such as ordinary trucks, vans, trailers, enclosed trucks, tank trucks, flatbed trucks, container trucks, dump trucks, special structure trucks, etc.;

(3) Tool functions, such as logistics distribution vehicles, automatic guided transport vehicles AGV, patrol vehicles, cranes, cranes, excavators, bulldozers, forklifts, road rollers, loaders, off-road construction vehicles, armored construction vehicles, sewage treatment vehicles, Sanitation vehicles, vacuum cleaners, floor washing vehicles, sprinklers, sweeping robots, food delivery robots, shopping guide robots, lawn mowers, golf carts, etc.;

(4) Entertainment functions, such as recreational vehicles, playground automatic driving devices, balance cars, etc.;

(5) Special rescue functions, such as fire trucks, ambulances, electric repair vehicles, engineering rescue vehicles, etc.

FIG. 1 is a flowchart of a method for automatic calibration of a sensor provided by an embodiment of the present invention, including the following steps:

S11: Obtain the laser point cloud data collected by the laser device;

S12: obtain image data collected by the camera;

S13: According to the driving state of the mobile device, identify the scene target related to the driving state in the laser point cloud data, and obtain the first coordinates of the scene target in the laser point cloud coordinate system;

S14: According to the driving state of the mobile device, identify the scene object related to the driving state in the image data, and obtain the second coordinate of the scene object in the image coordinate system;

S15: Reproject the first coordinate to a third coordinate in the image coordinate system;

S16: Match the second coordinate and the third coordinate in the image coordinate system to obtain a matching result;

S17: Based on the matching result, calibrate the external parameters between the laser device and the camera.

The laser device of the present application can be, for example, a lidar.

In order to improve the universality of the technical solution of the present application, the scene target in the embodiment of the present invention can be set as the target that can be easily collected around the vehicle, for example, the dynamic target includes pedestrians, vehicles, etc., and the static target includes lane lines, signs, etc. .

For steps S11 and S12, the laser point cloud data and the camera collected by the laser equipment mounted on the vehicle are used to collect image data.

In the embodiment of the present invention, the correspondence between the driving state and the corresponding scene target is preset. For example, the driving state of the vehicle includes the vehicle driving, the vehicle starting, the vehicle stopping, etc.; the scene target corresponding to the vehicle driving includes pedestrians and/or vehicles and other dynamic Targets, scene targets corresponding to vehicle start and vehicle stop include static targets such as lane lines and/or signs.

In the aforementioned step S13, the scene target related to the driving state in the laser point cloud data is identified. In one embodiment, a target detection model can be used to identify the scene target. For example, the target detection model can use cnn-seg segmentation. The network, cnn-seg segmentation network uses semantic segmentation for target detection, and performs regression layer processing while performing semantic segmentation, and then aggregates according to the results of center offset and semantic segmentation to obtain the detection results of a single target, thus obtaining The laser point cloud data corresponding to each scene object. The point cloud coordinates of the scene object can be obtained from the laser point cloud data through a point cloud-based segmentation detection network, for example, the ground is culled from the laser point cloud data, the foreground point cloud is extracted, and the foreground point cloud is sorted according to a specific point cloud. The clustering feature obtains the specific clustering target feature, and the target information corresponding to the scene target is obtained according to the clustering target feature detection, and the target information includes the bounding box, center point and height information of the scene target. According to the clustering target features, the grid clustering information is obtained, and according to the grid features, the target category is obtained, so as to determine the target, and obtain several features such as the height, length, width, orientation, and center point of the target. Map to their respective coordinate systems, so as to obtain the first coordinates of the scene target in the laser point cloud coordinate system.

For step S14, the scene target is identified from the image collected by the camera, and the scene target can be identified by the preset target detection model. Optionally, the target detection model can use the visual detection model of yolov5 to obtain the scene target in the image coordinate system. the second coordinate of . The target image coordinates of the scene object can be obtained from the image through the image-based detection and segmentation model. The image-based detection and segmentation model is an end-to-end learning network, which is used for feature extraction through the first N layers, and after K Each layer classifies the mentioned features, and obtains the target information of the scene target, the target information includes detection frame, direction, center point, etc.

For step S15, since the point cloud coordinates are the three-dimensional coordinates in the lidar coordinate system, and the image coordinates are the two-dimensional coordinates in the camera coordinate system, the two cannot be directly compared. It is necessary to first project the point cloud coordinates into the camera coordinate system and convert them into image coordinates, that is, reproject the first coordinates in step S13 to the third coordinates in the image coordinate system, and then convert them to the first coordinates of the image coordinate system. The third coordinate and the second coordinate in the image coordinate system can be compared.

For step S16, the third coordinate converted from the point cloud data is matched with the second coordinate of the image data to obtain a matching result. The matching methods of the coordinates in the scene objects related to different driving states are different.

As an embodiment, when the driving state is a stopped state after starting, the scene target is a predetermined static target. The static objects include lane markings and/or signage.

When the driving state is the stationary state after starting, the second coordinate and the third coordinate are matched in the image coordinate system, and the matching result obtained includes:

After reprojecting the first coordinate to a third coordinate in the image coordinate system, determining a point cloud detection frame of the static target based on the third coordinate;

determining a visual detection frame of the static target based on the second coordinates;

When the intersection ratio of the point cloud detection frame and the visual detection frame is greater than a set ratio, determine the laser device and the image coordinate system based on the reprojection relationship between the laser point cloud coordinate system and the image coordinate system. The relative pose between the cameras, as the matching result.

In the present embodiment, the extensiveness of the application and the scene on the actual road are taken into consideration. When the vehicle is started, a preliminary parameter estimation is performed first. Here, the calibration needs to collect the reflectivity of the lidar in the scene where the driving state is the stationary state after startup.

In this case, first of all, the selection of the scene target is more inclined to consider the ground area and the suspended area. For example, in the actual environment, lane lines and signs can be selected. Under normal circumstances, the reflectivity of road signs in the laser is close to the maximum reflectivity. Therefore, as shown in Figure 2, lidar can use the reflectivity feature to screen out signs. In addition, the reflectivity of the ground lane line is significantly different from other non-lane line areas on the road. Therefore, with the lane lines and dangling sign features, the lane lines and dangling sign can be separated from the point cloud.

In the visual image, using the trained sign detection model, as shown in Figure 3, the detection frame of the unobstructed sign can be easily obtained. Here, the yolov3 traffic sign detection network model can be used to detect the traffic signs in the upper half of the image for deep separation, and obtain the location of the sign in the image. (Fig. 2 and Fig. 3 are used for illustration, in order to show that Fig. 2 and Fig. 3 are obtained by different reflectivity of laser light, and the real images are distinguished based on the difference of gray scale).

Using the re-projection relationship between the laser point cloud and the signboard in the image, the point cloud is transformed into spatial coordinates, and the rotational position relationship of the point cloud signboard projected to the image is roughly obtained. The matching projection example is shown in Figure 4, and the image and The signs and lane lines in the point cloud can achieve preliminary matching and achieve the purpose of parameter initialization iteration.

Specifically, the projection of the signboard segmented from the point cloud is converted using a projection matrix to obtain the projection of the point cloud on the image. The point cloud detection frame and the visual detection frame calculate the IOU (Intersection over Union, intersection ratio) to find the position with the largest proportion of the overlapping frame. If the cumulative proportion of the overlapping frame cannot reach the preset ratio threshold P0 (for example, the ratio threshold is set to 75%, those skilled in the art can also flexibly set the proportional threshold according to actual needs, which is not strictly limited in this application), then it can be considered that the preliminary registration effect cannot meet the requirements, and no parameter update is performed, and no detailed parameter optimization is performed.

In view of the sparseness of the point cloud, there may be point cloud rotation features in the process of detection frame matching, so consider adding ground information for initial parameter optimization, such as the lane lines on the ground. In the display of the point cloud, the ground lane lines are extracted from the background points, that is, the ground information. Using the reflectivity features in the ground information, the lane lines are extracted to complete the initialization matching of the laser equipment and the vision, and realize the calibration of the laser equipment. Preliminary calibration of parameters. It can be seen from this embodiment that, for the stationary state after the vehicle is started, based on the initialization calibration parameters, the calibration method based on the initialization of the laser point cloud reflectivity parameter is realized, which can be applied to a variety of current multi-line autonomous driving lidars, and the applicability is wider.

As another implementation manner, when the driving state is motion, the scene target is a predetermined dynamic target. The dynamic targets include: pedestrians and/or vehicles.

When the driving state is motion, the second coordinate and the third coordinate are matched in the image coordinate system, and the matching result obtained includes:

After the first coordinate is reprojected to the third coordinate in the image coordinate system, the first number of dynamic objects in the third coordinate and the second number of dynamic objects in the second coordinate are multiplied. target model matching;

Multi-point perspective imaging projection is performed on the matching result, and the relative pose between the laser device and the camera is determined as the matching result.

In this implementation manner, as described in the defects in the prior art, after a period of driving of the autonomous vehicle, unavoidable shaking and other situations exist, which may cause slight changes in the relative positions of the camera and the laser. After driving for a period of time, the relative positions of the camera and the lidar will change. At this time, the driving state is motion. In this scenario, the laser device and the camera need to be calibrated with high precision.

In this case, the choice of dynamic targets at first favors pedestrians and vehicles, as these are the easiest to get in driving. Similarly, the point cloud coordinates are the three-dimensional coordinates in the lidar coordinate system, and the image coordinates are the two-dimensional coordinates in the camera coordinate system, and the two cannot be directly compared. It is still necessary to use reprojection to project the first coordinate in the point cloud coordinate system to the third coordinate in the image coordinate system. When the difference between the converted third coordinates (the converted point cloud coordinates) and the second coordinates (image coordinates) is greater than or equal to the set threshold, that is to say, the relative position of the lidar and the camera changes.

Matching is performed based on the targets obtained by image detection and point cloud segmentation detection. For example, gradient iterations can be used to obtain matched projection images, such as pedestrians and vehicles as shown in Figure 5. The optimization objective of the overall matching includes the second coordinates (x _i , y _i ) and the third coordinates (m _k , n _k ) of the ith scene objective. For the multi-objective model, according to the detected scene object, the following optimization objective function can be obtained:

min sum((x _i -m _k ) ² +(y _i -n _k ) ² ), i=0, 1, 2, 3, ..., s; k=0, 1, 2, 3, .. ., s.

For the above objective function, pnp (pespective-n-point, multi-point perspective imaging) can be used to solve the problem, and the external parameters between the camera and the lidar (ie, the rotation and translation matrix, the even rotation matrix and the translation matrix) can be obtained.

For step S17, use the matching result of the above steps to optimize the objective function to calibrate the external parameters between the laser device and the camera.

In some application scenarios, the vehicle is in a stationary state at the beginning, and the vehicle is in a moving state after starting the vehicle. In order to further improve the accuracy of the external parameter calibration between the laser device and the camera, as shown in Figure 6, when starting the vehicle Determine the scene target (such as static target, lane line and/or sign, etc.), identify the matching static target (such as lane line, sign) from the image collected by the camera and the laser point cloud data collected by the laser device, according to Determine whether the external parameter calibration between the camera and the laser device needs to be performed based on the matching result between the second coordinate and the third coordinate corresponding to the static target, so as to achieve preliminary parameter estimation (ie, complete the first calibration); When moving, determine that the scene target is a dynamic target (such as pedestrians and/or vehicles), and identify the matching dynamic target from the image collected by the camera and the laser point cloud data collected by the laser device. The matching result between the third coordinates calibrates the external parameters between the camera and the laser device to realize the second calibration. In terms of specific implementation, as shown in Figure 6, using the point cloud and the scene target (high sign, lane line) of scene 1 in the stationary state of the driving state to perform preliminary image and point cloud matching, by rotating the point cloud, Perform front projection, complete the parameter search calculation, and determine the initial position parameters. On the basis of the initialization parameters, the data modeling of scene 2 in which the driving state is motion is carried out, and scene 2 depends on the information of pedestrians and cars ubiquitous in the road. The pedestrians and vehicles are extracted through the deep network, and the center point features are obtained, and then multi-objective PNP matching is performed, and finally the appropriate external parameters, that is, the rotation matrix and the offset matrix, are obtained by iteration. The matching result of the detected pedestrian/vehicle and the vehicle and pedestrian (after conversion to the image coordinate system) obtained from the laser point cloud, T0 is the preset threshold for comparison, which is similar to the P0 in the above example, and will not be repeated here. .

It can be seen from this embodiment that, in the process of each vehicle startup, the calibration parameter verification process is started, and when the effect of the verification parameters meets the expected threshold, that is: the camera target (x _i , y _i ) and the point cloud target (m If the deviation of _i , n _i ) is less than the set threshold, it can be considered that the laser equipment and the external inclination of the camera do not shake greatly. When the parameters are too large or the position does not meet the requirements, the parameter self-checking is realized by judging the difference between the coordinates of the target point cloud and the coordinates of the target image; for automatic driving shaking, the resulting parameters are not applicable, the system automatically collects data and updates The closed-loop process of parameters optimizes the idea of algorithm calibration, broadens the usability, and does not rely on traditional calibration scenarios, which can realize more accurate automatic calibration of external parameter calibration between laser equipment and cameras.

After the calibration and optimization of the laser point cloud, this method also considers the extensiveness of application and the special circumstances in the actual road, and can be adjusted in a targeted manner, such as:

Due to the unclear division of road targets during the use of vision, the corresponding features (such as ground patterns, etc.) cannot be clearly obtained for whether some scene targets are foreground targets, so the laser point cloud data can be used. is the real target.

Obstacles in the actual operation process, according to the target detection model, it is difficult to categorize them into specific marked categories, but in automatic driving detection, such targets must be installed and avoided. Therefore, through the extraction of indeterminate types of point clouds in the laser point cloud, it is determined that the scene target really exists and the security is ensured.

In scenes with weak vision such as at night, the reliability of the laser point cloud is higher. Therefore, reducing the weight of the visual target through the image can improve the night driving performance to a certain extent.

Similarly, after the optimization of image calibration, the laser point cloud will have many abnormal points in abnormal weather such as rainy days. With the help of visual image filtering, it can be determined as noise in the lane, and the filtering can be completed to ensure abnormality. Obstacles are removed to avoid misoperations such as emergency braking and improve the driving experience.

According to different situations, it realizes the screening of different abnormal targets in laser and vision, provides ideas of elimination and screening, facilitates the extraction of obstacles, and improves the optimization effect.

FIG. 7 is a schematic structural diagram of a sensor automatic calibration device according to an embodiment of the present invention. The system can execute the sensor automatic calibration method described in any of the above embodiments, and is configured in a terminal.

A sensor automatic calibration device 10 provided in this embodiment includes: a point cloud acquisition module 11, an image acquisition module 12, a first coordinate determination module 13, a second coordinate determination module 14, a third coordinate determination module 15, a matching module 16 and Calibration module 17.

Among them, the point cloud acquisition module 11 is used to acquire laser point cloud data collected by the laser equipment; the image acquisition module 12 is used to acquire the image data collected by the camera; Identify the scene target related to the driving state in the laser point cloud data, and obtain the first coordinates of the scene target in the laser point cloud coordinate system; the second coordinate determination module 14 is used for determining according to the mobile device. Driving state, identify the scene object related to the driving state in the image data, and obtain the second coordinate of the scene object in the image coordinate system; the third coordinate determination module 15 is used for A coordinate is reprojected to a third coordinate in the image coordinate system; the matching module 16 is used for matching the second coordinate and the third coordinate in the image coordinate system to obtain a matching result; the calibration module 17 is used for Based on the matching result, the external parameters between the laser device and the camera are calibrated.

Preferably, when the driving state is a static state after starting, the scene target is a predetermined static target.

Preferably, when the driving state is the stationary state after starting, the matching module 16 matches the second coordinate and the third coordinate in the image coordinate system, and obtaining a matching result includes: After the first coordinate is reprojected to the third coordinate in the image coordinate system, the point cloud detection frame of the static target is determined based on the third coordinate; the visual detection frame of the static target is determined based on the second coordinate; When the intersection ratio of the point cloud detection frame and the visual detection frame is greater than a set ratio, determine the laser device and the image coordinate system based on the reprojection relationship between the laser point cloud coordinate system and the image coordinate system. The relative pose between the cameras, as the matching result.

Preferably, the static objects include lane markings and/or signage.

Preferably, when the driving state is motion, the scene target is a predetermined dynamic target.

Preferably, when the driving state is motion, the matching module 16 matches the second coordinate and the third coordinate in the image coordinate system, and obtaining a matching result includes: After the coordinates are reprojected to the third coordinates in the image coordinate system, the multi-object model matching is performed on the first number of dynamic objects in the third coordinates and the second number of dynamic objects in the second coordinates; The matching result is subjected to multi-point perspective imaging projection, and the relative pose between the laser device and the camera is determined as the matching result.

Preferably, the dynamic targets include pedestrians and/or vehicles.

Preferably, the laser device and the camera are triggered to collect laser point cloud data and image data in response to activation of the mobile device.

Embodiments of the present invention further provide a non-volatile computer storage medium, where the computer storage medium stores computer-executable instructions, and the computer-executable instructions can execute the automatic sensor calibration method in any of the above method embodiments;

As an embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are set to:

Obtain laser point cloud data collected by laser equipment;

Obtain the image data collected by the camera;

According to the driving state of the mobile device, identify the scene target related to the driving state in the laser point cloud data, and obtain the first coordinates of the scene target in the laser point cloud coordinate system;

According to the driving state of the mobile device, identify the scene object related to the driving state in the image data, and obtain the second coordinate of the scene object in the image coordinate system;

reprojecting the first coordinate into a third coordinate in the image coordinate system;

Matching the second coordinate and the third coordinate in the image coordinate system to obtain a matching result;

Based on the matching result, the external parameters between the laser device and the camera are calibrated.

As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the methods in the embodiments of the present invention. One or more program instructions are stored in a non-volatile computer-readable storage medium, and when executed by a processor, perform the automatic sensor calibration method in any of the above method embodiments.

An embodiment of the present invention further provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor , the instructions are executed by the at least one processor to enable the at least one processor to perform a sensor automatic calibration method.

In some embodiments, embodiments of the present invention further provide a mobile device, comprising a body and the electronic device according to any of the foregoing embodiments mounted on the body. The mobile device may be an unmanned vehicle, such as an unmanned sweeper, an unmanned floor washing vehicle, an unmanned logistics vehicle, an unmanned passenger car, an unmanned sanitation vehicle, and an unmanned minibus /Bus, truck, minecart, etc., also robots, etc.

In some embodiments, the embodiments of the present invention further provide a computer program product, which, when the computer program product runs on a computer, enables the computer to execute the automatic sensor calibration method described in any one of the embodiments of the present invention .

FIG. 8 is a schematic diagram of the hardware structure of an electronic device for an automatic sensor calibration method provided by another embodiment of the present application. As shown in FIG. 8 , the device includes:

One or more processors 810 and a memory 820, one processor 810 is taken as an example in FIG. 8 . The apparatus of the sensor automatic calibration method may further include: an input device 830 and an output device 840 .

The processor 810, the memory 820, the input device 830, and the output device 840 may be connected through a bus or in other ways, and the connection through a bus is taken as an example in FIG. 8 .

The memory 820, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs and modules, such as programs corresponding to the sensor automatic calibration method in the embodiments of the present application Directive/Module. The processor 810 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 820, that is, to implement the automatic sensor calibration method of the above method embodiment.

The memory 820 may include a stored program area and a stored data area, wherein the stored program area can store an operating system and an application program required by at least one function; the stored data area can store data and the like. Additionally, memory 820 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 820 may optionally include memory located remotely from processor 810, which may be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The input device 830 may receive input numerical or character information. The output device 840 may include a display device such as a display screen.

The one or more modules are stored in the memory 820, and when executed by the one or more processors 810, perform the automatic sensor calibration method in any of the above method embodiments.

The above product can execute the method provided by the embodiments of the present application, and has functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, reference may be made to the methods provided in the embodiments of this application.

The non-volatile computer-readable storage medium may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the device. data etc. In addition, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory located remotely from the processor, which may be connected to the device through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

An embodiment of the present invention further provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor , the instructions are executed by the at least one processor, so that the at least one processor can execute the steps of the automatic sensor calibration method according to any embodiment of the present invention.

The electronic devices in the embodiments of the present application exist in various forms, including but not limited to:

(1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions, and its main goal is to provide voice and data communication. Such terminals include: smart phones, multimedia phones, feature phones, and low-end phones.

(2) Ultra-mobile personal computer equipment: This type of equipment belongs to the category of personal computers, has computing and processing functions, and generally has the characteristics of mobile Internet access. Such terminals include: PDAs, MIDs, and UMPC devices, such as tablet computers.

(3) Portable entertainment equipment: This type of equipment can display and play multimedia content. Such devices include: audio and video players, handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

(4) Other mobile devices with data processing functions.

As used herein, the terms "comprising" and "comprising" include not only those elements, but also other elements not expressly listed, or elements inherent to such a process, method, article or apparatus. Without further limitation, an element defined by the phrase "comprises" does not preclude the presence of additional identical elements in a process, method, article, or device that includes the element.

The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic A disc, an optical disc, etc., includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in various embodiments or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (13)

1. A sensor automatic calibration method, applied to a mobile device, the sensor comprises a laser device and a camera, and the method comprises: Obtain laser point cloud data collected by laser equipment; Obtain the image data collected by the camera; According to the driving state of the mobile device, identify the scene target related to the driving state in the laser point cloud data, and obtain the first coordinates of the scene target in the laser point cloud coordinate system; According to the driving state of the mobile device, identify the scene object related to the driving state in the image data, and obtain the second coordinate of the scene object in the image coordinate system; reprojecting the first coordinate into a third coordinate in the image coordinate system; Matching the second coordinate and the third coordinate in the image coordinate system to obtain a matching result; Based on the matching result, the external parameters between the laser device and the camera are calibrated. 2 . The method according to claim 1 , wherein when the driving state is a static state after starting, the scene target is a predetermined static target. 3 . 3 . The method according to claim 2 , wherein when the driving state is a stationary state after starting, the second coordinate and the third coordinate are matched in the image coordinate system, 3 . The matching results obtained include: After reprojecting the first coordinate to a third coordinate in the image coordinate system, determining a point cloud detection frame of the static target based on the third coordinate; determining a visual detection frame of the static target based on the second coordinates; When the intersection ratio of the point cloud detection frame and the visual detection frame is greater than a set ratio, determine the laser device and the image coordinate system based on the reprojection relationship between the laser point cloud coordinate system and the image coordinate system. The relative pose between the cameras, as the matching result. 4. The method of claim 2, wherein the static target comprises lane markings and/or signage. 5 . The method according to claim 1 , wherein when the driving state is motion, the scene target is a predetermined dynamic target. 6 . 6 . The method according to claim 5 , wherein when the driving state is motion, matching the second coordinate and the third coordinate in the image coordinate system, and obtaining a matching result comprises the following steps: 6 . : After the first coordinate is reprojected to the third coordinate in the image coordinate system, the first number of dynamic objects in the third coordinate and the second number of dynamic objects in the second coordinate are multiplied. target model matching; Multi-point perspective imaging projection is performed on the matching result, and the relative pose between the laser device and the camera is determined as the matching result. 7. The method according to claim 5, wherein the dynamic target comprises: pedestrians and/or vehicles. 8 . The method of claim 1 , wherein the laser device and the camera are triggered to collect laser point cloud data and image data in response to activation of the mobile device. 9 . 9. A sensor automatic calibration device, characterized in that, comprising: Point cloud acquisition module, used to acquire laser point cloud data collected by laser equipment; The image acquisition module is used to acquire the image data collected by the camera; The first coordinate determination module is used to identify the scene target related to the driving state in the laser point cloud data according to the driving state of the mobile device, and obtain the scene target in the laser point cloud coordinate system. first coordinate; The second coordinate determination module is configured to identify the scene object related to the driving state in the image data according to the driving state of the mobile device, and obtain the second coordinate of the scene object in the image coordinate system. coordinate; A third coordinate determination module, configured to reproject the first coordinate to a third coordinate in the image coordinate system; a matching module, configured to match the second coordinate and the third coordinate in the image coordinate system to obtain a matching result; A calibration module, configured to calibrate an external parameter between the laser device and the camera based on the matching result. 10. An electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, The instructions are executed by the at least one processor to enable the at least one processor to perform the automatic sensor calibration method of any of claims 1-8. 11. A mobile device, characterized by comprising a body and the electronic device according to claim 10 mounted on the body. 12. A storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the automatic sensor calibration method according to any one of claims 1-8 is implemented. 13. A computer program product, characterized in that, when the computer program product runs on a computer, the computer is caused to execute the automatic sensor calibration method according to any one of claims 1-8.

Images