CN117213515A - Visual SLAM path planning method and device, electronic equipment and storage medium - Google Patents

Abstract

This application relates to a visual SLAM path planning method, device, electronic equipment and storage medium. The method includes: obtaining an offline map of the current scene, which is pre-constructed by a SLAM algorithm; obtaining the current image frame through a camera, extracting ORB feature points on the current image frame, and comparing the ORB feature points with the current scene Match the offline map to obtain the pose information of the camera; perform target point detection on the current image frame through a deep learning model, and use pose transformation to calculate the position coordinates of the target point; use the pose of the camera The information is used as the starting point of path planning, the location coordinates of the target point are used as the end point of path planning, and a path planning algorithm is used to perform path planning on the offline map. The embodiments of this application make full use of offline map information, can more accurately match feature points and map points, and obtain more accurate pose information.

Description

Visual SLAM path planning method, device, electronic equipment and storage medium

Technical field

This application belongs to the technical field of simultaneous positioning and mapping, and particularly relates to a visual SLAM path planning method, device, electronic equipment and storage medium.

Background technique

For the visually impaired, once they go out, they may face situations that are more powerless than physical disabilities, such as weird blind roads, traffic lights without prompts, missing guide dogs, or transportation and restaurants that refuse to admit guide dogs... . Therefore, up to 30% of visually impaired people choose not to step out of their homes. How to help blind people establish their perception of the outside world and guide them in their daily lives is an urgent problem that needs to be solved.

In the field of existing guide technology for the blind, visual SLAM (Simultaneous Localization and Mapping) technology is usually used to guide visually impaired people to move. Visual SLAM can help visually impaired people locate their own position and establish an understanding of the surrounding environment. Perception, the visual SLAM method used determines the effect of guiding the blind. SLAM refers to a technology that uses sensors mounted on a robot to sense the surrounding environment without prior information, build an environment model during movement, and estimate its own positioning at the same time. However, the existing visual SLAM technology does not make use of the offline map information in the scene, resulting in inaccurate matching of feature points and map points, and the resulting pose information is not highly accurate, ultimately affecting the blind guidance effect.

Contents of the invention

This application provides a visual SLAM path planning method, device, electronic device, and storage medium, aiming to solve one of the above technical problems in the prior art at least to a certain extent.

In order to solve the above problems, this application provides the following technical solutions:

A visual SLAM path planning method, including:

Obtain the offline map of the current scene, which is pre-constructed by the SLAM algorithm;

Obtain the current image frame through the camera, use the ORB algorithm to extract ORB feature points from the current image frame, and use the relocation algorithm to match the ORB feature points with the offline map of the current scene to obtain the pose information of the camera ;

Perform target point detection on the current image frame through a deep learning model, and use pose transformation to calculate the position coordinates of the target point;

The pose information of the camera is used as the starting point of path planning, the position coordinates of the target point are used as the end point of path planning, and a path planning algorithm is used to perform path planning on the offline map.

The technical solution adopted by the embodiment of this application also includes: the offline map includes a point cloud map and an octree map of the current scene. The specific steps of obtaining the offline map of the current scene are:

Use the SLAM algorithm to map the current scene and obtain a point cloud map;

Use a hierarchical clustering algorithm to form the map point descriptors in the point cloud map into a K-tree format, and convert the point cloud map into an octree map;

Save the point cloud map and octree map into a file format.

The technical solution adopted by the embodiment of the present application also includes: the ORB feature points include key points and descriptors, and obtaining the current image frame through a camera and using the ORB algorithm to extract ORB feature points from the current image frame includes:

Use the FAST algorithm to find pixels with high intensity changes in the current image frame as key points;

Calculate the gray gradient direction around the key point, determine the direction of the key point based on the gray gradient direction, and use it as the direction of the ORB feature point;

For each keypoint, a fixed-length binary descriptor is generated using the BRIEF algorithm, which compares pairs of pixels at specific locations to generate a binary code that corresponds to the image area around the keypoint.

The technical solution adopted by the embodiment of the present application also includes: matching the ORB feature points with the offline map of the current scene through a relocation algorithm to obtain the pose information of the camera as follows:

Match the descriptors of the ORB feature points with the map point descriptors of the offline map to obtain several pairs of matching points;

An N-point perspective pose solving algorithm is used to solve the pose of the matching points.

The technical solution adopted by the embodiment of the present application also includes: matching the ORB feature points with the offline map of the current scene through a relocation algorithm, and after obtaining the pose information of the camera, it also includes:

The beam adjustment method is used to optimize the pose information of the current camera and the IMU data to obtain optimized pose information.

The technical solution adopted by the embodiment of the present application also includes: performing target point detection on the current image frame through a deep learning model, and using pose transformation to calculate the position coordinates of the target point, specifically:

Use a deep learning model to perform target point detection on the current image frame, and obtain the image detection frame of the target point and the image coordinates of the image detection frame;

The world coordinates of the target point are calculated through pose transformation based on the image coordinates.

The technical solution adopted by the embodiment of the present application also includes: after using the path planning algorithm to perform path planning on the offline map, it also includes:

Based on the path planning results, the route information is converted into voice playback.

Another technical solution adopted by the embodiment of the present application is: a visual SLAM path planning device, including:

Offline building module: used to obtain the offline map of the current scene, which is pre-constructed by the SLAM algorithm;

Online matching module: used to obtain the current image frame through the camera, use the ORB algorithm to extract ORB feature points from the current image frame, and match the ORB feature points with the offline map of the current scene through the relocation algorithm to obtain the Describes the camera’s pose information;

Target detection module: used to perform target point detection on the current image frame through a deep learning model, and use pose transformation to calculate the position coordinates of the target point;

Path planning module: used to use the pose information of the camera as the starting point of path planning, use the position coordinates of the target point as the end point of path planning, and use a path planning algorithm to perform path planning on the offline map.

Another technical solution adopted by the embodiment of the present application is: an electronic device. The electronic device includes a processor, a memory, a computer program stored in the memory and executable on the processor, a communication interface, and an external Equipment; the processor implements the above visual SLAM path planning method when executing the computer program.

Another technical solution adopted by the embodiments of the present application is: a storage medium that stores a computer program executable by a processor, and the computer program is used to execute the above visual SLAM path planning method.

Compared with the existing technology, the beneficial effects produced by the embodiments of the present application are: the visual SLAM path planning method, device, electronic device and storage medium of the embodiments of the present application pre-construct an offline map of the current scene, and use the relocation algorithm and the offline map Match and calculate the current pose information, use the deep learning model to detect the target point in the current image frame, obtain the coordinates of the target point detection frame in the current image frame and transform them into world coordinates, and perform path based on the current pose information and the world coordinates of the target point planning. The embodiments of this application make full use of offline map information, can more accurately match feature points and map points, obtain more accurate pose information, have better adaptability to the relocation task in visual SLAM, and can better Help visually impaired people to establish their perception and guidance of the outside world in daily life.

Description of drawings

Figure 1 is a flow chart of the visual SLAM path planning method according to the first embodiment of the present application;

Figure 2 is a flow chart of the visual SLAM path planning method according to the second embodiment of the present application;

Figure 3 is a schematic structural diagram of the visual SLAM path planning device according to the embodiment of the present application;

Figure 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

The terms “first”, “second” and “third” in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Thus, features defined as "first", "second", and "third" may explicitly or implicitly include at least one of these features. In the description of this application, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically limited. All directional indications (such as up, down, left, right, front, back...) in the embodiments of this application are only used to explain the relative positional relationship between components in a specific posture (as shown in the drawings). , sports conditions, etc., if the specific posture changes, the directional indication will also change accordingly. Furthermore, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or electronic device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes unlisted steps or units, or optionally also Includes other steps or units inherent to such processes, methods, products, or electronic devices.

Reference herein to "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of this phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those skilled in the art understand, both explicitly and implicitly, that the embodiments described herein may be combined with other embodiments.

Please refer to Figure 1, which is a flow chart of the visual SLAM path planning method according to the first embodiment of the present application. The visual SLAM path planning method can be applied to smartphones, augmented reality (AR)/virtual reality (VR) devices, tablet computers and other electronic devices. The embodiments of this application do not place any restrictions on the specific types of electronic devices.

Specifically, the visual SLAM path planning method in the first embodiment of this application includes the following steps:

S100: Obtain the offline map of the current scene, which is pre-constructed by the SLAM algorithm;

In this step, the offline map includes the point cloud map and the octree map of the current scene. The construction method of the offline map specifically includes:

S101: Use the SLAM algorithm to map the current scene and obtain a point cloud map;

Among them, point cloud map is a three-dimensional map representation method, which consists of a large number of discrete points. Each point has spatial coordinates and other attributes such as color and normal. Point cloud maps obtain point cloud data through lidar or other sensors to describe the three-dimensional structure and characteristics of the environment. Point cloud maps can provide more accurate and detailed environmental models to support applications such as autonomous driving and robot navigation.

S102: Use hierarchical clustering algorithm to form the map point descriptors in the point cloud map into K-ary tree format, and convert the point cloud map into an octree map;

Among them, octree map is a common data structure for point cloud maps. It organizes point cloud data into a tree by dividing the space into eight equal-sized sub-regions and recursively dividing in each sub-region. Structure, each node represents a sub-region, containing point cloud data or other attributes within the region. This layered structure enables octree maps to efficiently store and access point cloud data, and is better suited to application scenarios such as dynamic environments, local map updates, and fast searches.

S103: Save the point cloud map and octree map into file format;

S110: Based on the offline map of the current scene, use the camera to obtain the current image frame, and use the ORB (OrientedFAST and Rotated BRIEF) algorithm to extract ORB feature points of the current image frame;

In this step, the cameras used include but are not limited to binocular cameras or RGB-D cameras. The ORB algorithm is a manually designed feature point descriptor algorithm in computer vision. It combines the advantages of the FAST (Features from Accelerated Segment Test) key point detection algorithm and the BRIEF (Binary Robust Independent Elementary Features) descriptor algorithm, and can Robustly detect and describe image feature points under , rotation and lighting conditions.

In the embodiment of this application, ORB feature points include two parts: key-point and descriptor. The ORB feature point extraction process using the ORB algorithm is specifically: first, use the FAST algorithm to find high-resolution features in the current image frame. The pixel points with changing intensity are used as key points; then, the gray gradient direction around the key points is calculated, and the gray gradient direction is used as the direction of the ORB feature point, so that the ORB feature point can be invariant under different rotation angles, thus improving the accuracy of the algorithm. Stickiness and matching performance. For each keypoint, a fixed-length binary descriptor is generated using the BRIEF algorithm, which compares pairs of pixels at specific locations to generate a binary code that corresponds to the image area around the keypoint. Due to the use of binary descriptors, ORB feature points have the characteristics of fast matching and efficient storage.

S120: Match the ORB feature points with the offline map of the current scene through the relocation algorithm to obtain the pose information of the current camera, and use the pose information of the current camera as the starting point of path planning;

In the embodiment of this application, the matching process of ORB feature points and offline maps is specifically: matching the descriptors of ORB feature points of the current image frame with the map point descriptors of the offline map to obtain several pairs of matching points, and use PnP ( Perspective-n-Point, N-point perspective pose solution) algorithm solves the pose of matching points. Among them, the PnP algorithm is an algorithm used in computer vision to solve the camera pose. It estimates the camera pose through known three-dimensional points and two-dimensional points in the corresponding image. The basic principle of the PnP algorithm is to use at least three non-collinear three-dimensional points and their corresponding two-dimensional points in the image to calculate the camera pose by solving the perspective transformation matrix. The input of the PnP algorithm includes the coordinates of the three-dimensional point and the coordinates of the two-dimensional point of the corresponding image, and the output is the rotation matrix and translation vector of the camera.

Furthermore, PnP algorithms include EPnP (Efficient PnP), UPnP (Uncalibrated PnP) and Direct Linear Transform (DLT) and other solving algorithms. Among them: EPnP algorithm solves the camera pose by minimizing the reprojection error, using linear least square Multiplication method, and using SVD decomposition to calculate the camera pose, has high computational efficiency and accuracy. The UPnP algorithm estimates the camera's pose by minimizing the ray projection error when the camera is not internally calibrated. The UPnP algorithm requires known constraints such as camera angle of view, camera plane and depth information of three-dimensional points. The DLT algorithm is a classic method for solving the PnP problem. It estimates the camera's pose by linearly solving the perspective transformation matrix. DLT algorithms may be affected by noise and errors in terms of calculation accuracy, so nonlinear optimization algorithms are usually required for further accurate estimation.

S130: Based on the current image frame, perform target point detection on the current image frame through the deep learning model, use the target point detection result as the end point of path planning, and use pose transformation to calculate the position coordinates of the end point of path planning;

In this step, the deep learning models used include but are not limited to YOLO or SSD (Single Shot MultiBox Detector, single-step multi-frame target detection). In the embodiment of this application, taking the use of the yolov5 model to detect target points in the current image frame as an example, the specific detection process includes:

S131: Use the yolov5 model to detect the target point in the current image frame, and obtain the image detection frame of the target point and the image coordinates of the image detection frame;

S132: Calculate the world coordinates of the target point through pose transformation based on the image coordinates;

Among them, the calculation process of world coordinates specifically includes:

The first step: obtain the depth z _i of each image point;

Among them, the depth z _i is obtained by: using an RGB-D depth camera to obtain the depth z _i of each image point from the depth map, or using a binocular camera to recover the depth z _i from the disparity map: the binocular camera is known Focal length f and baseline b, use OpenCV to obtain the disparity d of each image point, and further obtain the depth z _i of each image point:

Step 2: Calculate the average depth of all image points average the depth/> As the distance between the current camera and the target point:

Optionally, you can also use an external ultrasonic rangefinder to obtain the distance between the current camera and the target point.

Step 3: According to distance Calibrated camera internal parameter matrix K and camera pose R _cw and t _cw calculate the world coordinate P _w of the target point as:

In formula (3), the subscript cw represents the transformation from the world coordinate system to the camera coordinate system, and the subscript w represents the coordinates in the world coordinate system.

S140: Use the pose information of the current camera as the starting point of path planning, use the position coordinates of the target point as the end point of path planning, and use the path planning algorithm to perform path planning on the octree map;

In this step, the path planning algorithm can select the bidirectional rapidly expanding random tree (RRT-Connect) algorithm or the rapid exploring random tree (Rapid-exploring random tree, RRT) algorithm. This application does not limit the specific implementation method of path planning. Taking the bidirectional rapidly expanding random tree (RRT-Connect) algorithm as an example, the path planning process is as follows: growing two expansion trees from the starting point and the target point respectively, and in each iteration process, one of the trees is expanded, and Try to connect the nearest node of another tree to extend the new node.

S150: Based on the path planning results, convert the route information into voice playback;

In the embodiment of this application, the TextToSpeech engine can be used for voice broadcast. Voice broadcast content includes but is not limited to: distance information to the target point As well as travel route information based on path planning, etc.

Based on the above, the visual SLAM path planning method of the first embodiment of the present application pre-constructs an offline map of the current scene, calculates the current pose information by matching the relocation algorithm with the offline map, and uses a deep learning model to perform target point detection on the current image frame. , obtain the coordinates of the target point detection frame in the current image frame and transform them into world coordinates, and perform path planning and voice broadcasting based on the current pose information and the world coordinates of the target point. The embodiments of this application make full use of offline map information, can more accurately match feature points and map points, obtain more accurate pose information, have better adaptability to the relocation task in visual SLAM, and can better Help visually impaired people to establish their perception and guidance of the outside world in daily life.

Please refer to Figure 2, which is a flow chart of the visual SLAM path planning method according to the second embodiment of the present application. The visual SLAM path planning method in the second embodiment of this application includes the following steps:

S200: Obtain the offline map of the current scene, which is pre-constructed by the SLAM algorithm;

This step is the same as S100 in the first embodiment. For details, please refer to the relevant description of S100 and will not be described again here.

S210: Based on the offline map of the current scene, use the camera to obtain the current image frame, and use the ORB (OrientedFAST and Rotated BRIEF) algorithm to extract feature points of the current image frame;

This step is the same as S110 in the first embodiment. For details, please refer to the relevant description of S110 and will not be described again here.

S220: Match the ORB feature points with the offline map of the current scene through the relocation algorithm to obtain the pose information of the current camera, and use the pose information of the current camera as the starting point of path planning;

This step is the same as S120 in the first embodiment. For details, please refer to the relevant description of S120 and will not be described again here.

S230: Use the beam adjustment method to optimize the current camera's pose information and IMU data to obtain the optimized pose information;

In this step, IMU (Inertial Measurement Unit, inertial sensor) is a device that integrates multiple inertial sensors and is used to measure the acceleration and angular velocity of the object. IMU usually consists of two main sensors: accelerometer and gyroscope. Accelerometer: The accelerometer is used to measure the acceleration of an object in three axes and can detect the linear motion of the object and the acceleration of gravity. The output unit of the accelerometer is m/s ² , and velocity and position can be calculated by integrating the acceleration data. Gyroscope: Gyroscope is used to measure the angular velocity of an object around three axes and can detect the rotation and angle changes of the object. The gyroscope's output units are degrees/second or radians/second, and attitude and orientation can be calculated by integrating angular velocity data. IMU is usually used in fields such as inertial navigation, attitude estimation and motion tracking. By combining data from accelerometers and gyroscopes, the attitude and motion of an object can be estimated. In visual inertial SLAM, it is necessary to measure the pose T _i and velocity v _i of the i-th frame, as well as the offsets of the gyroscope and accelerometer. and/> specifically:

T _i =[R _i , p _i ]∈SE(3) (4)

It is necessary to obtain IMU pre-integrated measurements between consecutive visual frames i and i+1 for ΔR _i,i+1 , Δv _i,i+1 , Δp _i,i+1 and the covariance moment of the entire measurement vector and inertia residual/>

Where Log: SO(3)→R ₃ represents the mapping from Lie group to Lie algebra. Together with the inertial residual, the reprojection error r _ij between frame i and 3D point j at position x _j is used:

In the formula, Π: R ₃ → R _n is the projection function of the corresponding camera model, u _ij is the observation of point j on image frame i, with a covariance matrix ∑ _ij , T _CB ∈SE (3) represents the IMU from the ontology Rigid transformation to camera (left or right), known from calibration, is the transformation operation of the SE(3) group on R ₃ elements.

In the embodiment of this application, the camera can be approximated as the position of the IMU, and T _CB can be approximated as the unit matrix I; define the state vector:

Given a set of k+1 keyframes and their states and a set of l three-dimensional points and their states The visual inertia optimization problem can be formulated as follows:

where κ _j is the set of key frames for observing three-dimensional point j. For reprojection errors, the robust Huber kernel ρ _Hub is used to reduce the impact of mismatching.

S240: Based on the current image frame, perform target point detection on the current image frame through the deep learning model, use the target point detection result as the end point of path planning, and use pose transformation to calculate the position coordinates of the end point of path planning;

This step is the same as S130 in the first embodiment. For details, please refer to the relevant description of S130, which will not be described again here.

S250: Use the pose information of the current camera as the starting point of path planning, use the position coordinates of the target point as the end point of path planning, and use the path planning algorithm to perform path planning on the octree map;

This step is the same as S140 in the first embodiment. For details, please refer to the relevant description of S140 and will not be described again here.

S260: Based on the path planning results, convert the route information into voice playback;

This step is the same as S150 in the first embodiment. For details, please refer to the relevant description of S150 and will not be described again here.

Based on the above, the visual SLAM path planning method in the second embodiment of the present application, based on the first embodiment, simultaneously optimizes the IMU data and camera pose information through the beam adjustment method, which can further improve the accuracy and robustness, thereby obtaining More accurate pose information.

Please refer to Figure 3, which is a schematic structural diagram of a visual SLAM path planning device according to an embodiment of the present application. The visual SLAM path planning device in this embodiment of the present application includes:

Offline construction module 31: used to obtain the offline map of the current scene, which is pre-constructed by the SLAM algorithm;

The above-mentioned offline building module 31 includes:

The scene reconstruction unit is used to map the current scene using the SLAM algorithm to obtain a point cloud map;

Map conversion unit, used to convert point cloud maps into octree maps;

The map saving unit is used to use the hierarchical clustering algorithm to form the feature point descriptors corresponding to the map points in the point cloud map into a K-tree format, and save the point cloud map and the octree map into a file format.

The online matching module 32 is used for the offline map based on the current scene, using the camera to obtain the current image frame, and using the ORB algorithm to extract feature points of the current image frame, and matching the ORB feature points with the offline map of the current scene through the relocation algorithm. , obtain the pose information of the current camera, and use the pose information of the current camera as the starting point of path planning;

The above-mentioned online matching module 32 includes:

Feature point extraction unit, used to extract feature points of the current image frame;

The relocation unit is used to match feature points with map points of the offline map and calculate the pose information of the current camera;

The pose optimization unit uses the beam adjustment method to optimize pose information and IMU data to obtain more accurate pose information.

The target detection module 33 is used to perform target point detection on the current image frame based on the current image frame through a deep learning model, use the target point detection result as the end point of the path planning, and use pose transformation to calculate the position coordinates of the end point of the path planning;

The above-mentioned target detection module 33 includes:

The target detection unit is used to detect the target point of the current image frame using a deep learning model to obtain the image detection frame of the target point and the image coordinates of the image detection frame;

Distance estimation unit, used to calculate the distance between the current camera and the target point;

The coordinate calculation unit is used to calculate the world coordinates of the target point based on the distance, the calibrated camera internal parameter matrix, and the camera pose.

Path planning module 34: used to use the pose information of the current camera as the starting point of path planning, use the position coordinates of the target point as the end point of path planning, and use the path planning algorithm to perform path planning on the octree map;

Voice broadcast module 35: used to convert route information into voice broadcast based on the path planning results.

It should be noted that the information interaction, execution process, etc. between the above-mentioned devices/units are based on the same concept as the method embodiments of the present application. For details of their specific functions and technical effects, please refer to the method embodiments section. No further details will be given.

The devices provided by the embodiments of the present application can be applied in the foregoing method embodiments. For details, please refer to the description of the foregoing method embodiments, which will not be described again here.

Please refer to FIG. 4 , which is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 4 includes: one or more processors 41 (only one is shown in the figure), a memory 42, a computer program 43 stored in the memory 42 and executable on the processor 41, and a communication interface. 44 and external devices 45. When the processor 41 executes the computer program 43, the steps in each of the above visual SLAM path planning method embodiments are implemented.

Those skilled in the art can understand that FIG. 4 is only an example of the electronic device 4 and does not constitute a limitation of the electronic device 4. It may include more or fewer components than shown in the figure, or some components may be combined, or different components may be used. , for example, the electronic device may also include input and output devices, network access devices, buses, etc.

The so-called processor 41 may be a central processing unit (CPU), or other general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), Ready-made programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor.

The memory 42 may be an internal storage unit of the electronic device 4 , such as a hard disk or memory of the electronic device 4 . The memory 42 may also be an external storage device of the electronic device 4, such as a plug-in hard disk, a smart media card (SMC), or a secure digital (SD) equipped on the electronic device 4. card, flash card, etc. Further, the memory 42 may also include both an internal storage unit of the electronic device 4 and an external storage device. The memory 42 is used to store the computer program and other programs and data required by the electronic device. The memory 42 can also be used to temporarily store data that has been output or is to be output.

The external device may be a binocular camera and an IMU.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above functional units and modules is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs. Module completion means dividing the internal structure of the device into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be hardware-based. It can also be implemented in the form of software functional units. In addition, the specific names of each functional unit and module are only for the convenience of distinguishing each other and are not used to limit the scope of protection of the present application. For the specific working processes of the units and modules in the above device, reference can be made to the corresponding processes in the foregoing method embodiments, which will not be described again here.

Embodiments of the present application also provide a computer-readable storage medium. The storage medium stores a computer program. When the computer program is executed by a processor, the steps in each of the above method embodiments can be implemented.

Embodiments of the present application also provide a computer program product. When the computer program product is run on an electronic device, the steps in each of the above method embodiments can be implemented when the electronic device is executed.

In the above embodiments, each embodiment is described with its own emphasis. For parts that are not detailed or documented in a certain embodiment, please refer to the relevant descriptions of other embodiments.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

In the embodiments provided in this application, it should be understood that the disclosed devices/electronic devices and methods can be implemented in other ways. For example, the apparatus/electronic equipment embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple divisions. Units or components may be combined or integrated into another system, or some features may be omitted, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

If the integrated module/unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the processes in the methods of the above embodiments, which can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer can When the program is executed by the processor, the steps of each of the above method embodiments can be implemented. Wherein, the computer program includes computer program code, which may be in the form of source code, object code, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (Read-Only Memory, ROM) , random access memory (Random Access Memory, RAM), electrical carrier signals, telecommunications signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately added or deleted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium Excludes electrical carrier signals and telecommunications signals.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still implement the above-mentioned implementations. The technical solutions described in the examples are modified, or some of the technical features are equivalently replaced; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions in the embodiments of this application, and should be included in within the protection scope of this application.

Claims (10)

1. A visual SLAM path planning method, characterized by including: Obtain the offline map of the current scene, which is pre-constructed by the SLAM algorithm; Obtain the current image frame through the camera, use the ORB algorithm to extract ORB feature points from the current image frame, and use the relocation algorithm to match the ORB feature points with the offline map of the current scene to obtain the pose information of the camera ; Perform target point detection on the current image frame through a deep learning model, and use pose transformation to calculate the position coordinates of the target point; The pose information of the camera is used as the starting point of path planning, the position coordinates of the target point are used as the end point of path planning, and a path planning algorithm is used to perform path planning on the offline map. 2. The visual SLAM path planning method according to claim 1, characterized in that the offline map includes a point cloud map and an octree map of the current scene, and the obtaining the offline map of the current scene is specifically: Use the SLAM algorithm to map the current scene and obtain a point cloud map; Use a hierarchical clustering algorithm to form the map point descriptors in the point cloud map into a K-tree format, and convert the point cloud map into an octree map; Save the point cloud map and octree map into a file format. 3. The visual SLAM path planning method according to claim 2, characterized in that the ORB feature points include key points and descriptors, the current image frame is obtained through a camera, and the ORB algorithm is used to perform processing on the current image frame. ORB feature point extraction includes: Use the FAST algorithm to find pixels with high intensity changes in the current image frame as key points; Calculate the gray gradient direction around the key point, determine the direction of the key point based on the gray gradient direction, and use it as the direction of the ORB feature point; For each keypoint, a fixed-length binary descriptor is generated using the BRIEF algorithm, which compares pairs of pixels at specific locations to generate a binary code that corresponds to the image area around the keypoint. 4. The visual SLAM path planning method according to claim 3, characterized in that, by matching the ORB feature points with the offline map of the current scene through a relocation algorithm, the pose information of the camera is obtained. : Match the descriptors of the ORB feature points with the map point descriptors of the offline map to obtain several pairs of matching points; An N-point perspective pose solving algorithm is used to solve the pose of the matching points. 5. The visual SLAM path planning method according to claim 4, characterized in that, after matching the ORB feature points with the offline map of the current scene through a relocation algorithm to obtain the pose information of the camera, Also includes: The beam adjustment method is used to optimize the pose information of the current camera and the IMU data to obtain optimized pose information. 6. The visual SLAM path planning method according to claim 1, characterized in that the target point is detected on the current image frame through a deep learning model, and the position coordinates of the target point are calculated using pose transformation, specifically: : Use a deep learning model to perform target point detection on the current image frame, and obtain the image detection frame of the target point and the image coordinates of the image detection frame; The world coordinates of the target point are calculated through pose transformation based on the image coordinates. 7. The visual SLAM path planning method according to any one of claims 1 to 6, characterized in that after using the path planning algorithm to perform path planning on the offline map, it also includes: Based on the path planning results, the route information is converted into voice playback. 8. A visual SLAM path planning device, characterized by including: Offline building module: used to obtain the offline map of the current scene, which is pre-constructed by the SLAM algorithm; Online matching module: used to obtain the current image frame through the camera, use the ORB algorithm to extract ORB feature points from the current image frame, and match the ORB feature points with the offline map of the current scene through the relocation algorithm to obtain the Describes the camera’s pose information; Target detection module: used to perform target point detection on the current image frame through a deep learning model, and use pose transformation to calculate the position coordinates of the target point; Path planning module: used to use the pose information of the camera as the starting point of path planning, use the position coordinates of the target point as the end point of path planning, and use a path planning algorithm to perform path planning on the offline map. 9. An electronic device, characterized in that the electronic device includes a processor, a memory, a computer program stored in the memory and executable on the processor, a communication interface, and an external device; the processor When the computer program is executed, the visual SLAM path planning method described in any one of claims 1 to 7 is implemented. 10. A storage medium, characterized in that the storage medium stores a computer program executable by a processor, and the computer program is used to execute the visual SLAM path planning method according to any one of claims 1 to 7.