CN109919825B - An ORB-SLAM Hardware Accelerator - Google Patents

Abstract

The invention discloses an ORB-SLAM hardware accelerator, comprising an FPGA hardware acceleration module for accelerating feature extraction and feature matching; a sensor module for capturing images; a processor system for controlling the FPGA hardware acceleration module and the sensor module as a host , and is responsible for running pose estimation, pose optimization, and map updates. The invention uses the FPGA hardware acceleration module to accelerate the process with the largest amount of calculation and the most time-consuming in the ORB-SLAM process, which can effectively improve the running speed of the ORB-SLAM and reduce the power consumption, greatly improve the energy consumption ratio, and reduce the ORB-SLAM in the process. Difficulty deploying on power-constrained platforms.

Description

An ORB-SLAM Hardware Accelerator

technical field

The invention relates to the field of autonomous navigation, in particular to an ORB-SLAM hardware accelerator.

Background technique

SLAM (simultaneous localization and mapping) is one of the most critical technologies in the field of autonomous navigation. It enables the autonomous navigation system to build an incremental map of the surrounding environment based on the information captured by sensors in an unknown environment. At the same time determine its own position in the environment. SLAM is a critical technology widely used in autonomous vehicles, autonomous navigation robots, virtual reality, and augmented reality.

ORB-SLAM is a feature point-based visual SLAM based on ORB descriptors. It is a very efficient and robust SLAM system that has received extensive research attention. Existing ORB-SLAM systems can only run on traditional computing platforms (CPU, GPU), which results in their relatively low performance-to-power ratio. Limited by the performance and power consumption of the computing platform, the low-power real-time operation of ORB-SLAM has always been a difficult problem to solve. ORB-SLAM consumes a lot of computing resources in the process of feature extraction and matching. In order to ensure real-time operation, high-performance CPU and GPU are often required, which brings a lot of power consumption overhead. However, if an embedded CPU or GPU is used in order to reduce power consumption, the frame rate will be too low to run in real time.

SUMMARY OF THE INVENTION

The present invention provides an ORB-SLAM hardware accelerator, which is used to solve the problem that the ORB-SLAM is difficult to run in real time with low power consumption.

An ORB-SLAM hardware accelerator, which includes a sensor module, an FPGA hardware acceleration module and a processor system;

the sensor module is used for collecting image data;

The FPGA hardware acceleration module is used for extracting ORB features on the images collected by the sensor module, and matching the extracted features with map points in the global map;

The processor system is configured to calculate the camera pose and maintain the global map according to the ORB feature extracted by the FPGA hardware acceleration module and the matching result between the ORB feature and the map points in the global map.

Preferably, the FPGA hardware acceleration module includes: an image downsampling module, a feature extraction module and a feature matching module;

The image downsampling module is used to generate an image pyramid;

The feature extraction module is used to extract ORB features on each layer of the image pyramid;

The feature matching module is used to match the ORB features extracted by the feature extraction module with map points in the global map.

Preferably, the feature extraction module includes: a key point extraction unit, a non-maximum value suppression unit, a Gaussian filter unit, a feature direction calculation unit, a descriptor calculation unit, and a first cache unit;

The key point extraction unit is used to extract the FAST corner point on the input image and calculate the Harris response value of the FAST corner point;

The non-maximum suppression unit is used to perform non-maximum suppression on the Harris response value of the FAST corner point, and retain a FAST corner point with the largest Harris response value in the neighborhood;

The Gaussian filtering unit is used to perform Gaussian blurring on the input image, remove noise in the image, and generate a smooth image;

The feature direction calculation unit is used to calculate the feature direction on the Gaussian filtered image;

The descriptor calculating unit is used to calculate the Brief descriptor of the feature on the Gaussian filtered image according to the feature direction;

The first cache unit is used to temporarily store input data, intermediate calculation results and final results.

Preferably, the work flow of the feature direction calculation unit is as follows: first, calculate the pixel gray centroid coordinates of the neighborhood where the feature is located; then, calculate the ratio of the horizontal and vertical coordinates of the gray centroid, and obtain the feature direction according to the lookup table.

Preferably, the first cache unit includes: an input image cache, a smooth image cache, a response value cache, and a feature cache. Among them, the input image cache, the smooth image cache and the response value cache adopt a ping-pong-like architecture, which consists of multiple identical caches, which can process the input and output of data at the same time. The feature cache adopts a max-heap architecture, which is used to filter features while saving features, and only retain some features with large Harris response values.

Preferably, each unit of the feature extraction module adopts a streaming computing architecture, all computing units run in parallel, and only part of the data is stored in the cache, and the data is discarded immediately after use.

Preferably, the feature matching module includes: a matching unit and a second cache unit;

The matching unit is used for matching the features extracted by the feature extraction module with map points in the global map. The workflow is as follows: First, calculate the Hamming distance between any two descriptors in the two sets of descriptors (descriptors of map points in the global map and descriptors of features currently extracted from the previous frame); after that, use brute force The search method matches two sets of descriptors according to the Hamming distance;

The second cache unit includes a descriptor cache and a result cache, which are respectively used to temporarily store two sets of descriptors to be matched and matching results.

The processor system includes a general purpose processor, memory and a memory controller. The general-purpose processor is used to calculate the camera pose according to the extracted image features and the matching relationship between the features and map points in the global map, and to update and maintain the global map.

The FPGA hardware acceleration module and the processor system communicate through the AXI bus. The general-purpose processor in the processor system can directly configure the instruction registers in the FPGA hardware acceleration module through the AXI bus; the FPGA hardware acceleration module can also directly read data from the memory in the processor system through the AXI bus, or store the calculation results in the FPGA hardware acceleration module. Memory.

The FPGA hardware acceleration module and the processor system run in parallel in a pipeline manner.

An ORB-SLAM hardware accelerator of the present invention has the advantages and efficacy of: using a dedicated hardware acceleration module to accelerate the most intensive calculation process in ORB-SLAM, reducing power consumption while increasing the frame rate of the entire system, enabling ORB-SLAM Low-power real-time operation of SLAM becomes possible.

Description of drawings

FIG. 1 is a schematic structural diagram of an ORB-SLAM hardware accelerator according to an embodiment of the present invention.

FIG. 2 is a schematic structural diagram of a feature extraction module of an ORB-SLAM hardware accelerator according to an embodiment of the present invention.

FIG. 3 is a schematic structural diagram of a feature matching module of an ORB-SLAM hardware accelerator according to an embodiment of the present invention.

4 is a schematic diagram of a pipeline when an FPGA hardware acceleration module of an ORB-SLAM hardware accelerator and a processor system run in parallel according to an embodiment of the present invention.

Detailed ways

Aiming at the problem that the existing ORB-SLAM is difficult to run in real time on a low power consumption platform, the present invention proposes an ORB-SLAM hardware accelerator. The acceleration module accelerates the most computationally intensive process in the ORB-SLAM process through a dedicated hardware acceleration module, improves the overall performance-to-power ratio of the system, and enables low-power real-time operation. Next, the present invention will be further described in detail with reference to specific embodiments and accompanying drawings.

FIG. 1 is a schematic structural diagram of an ORB-SLAM hardware accelerator according to an embodiment of the present invention. As shown in Figure 1, this accelerator consists of three parts: FPGA hardware acceleration module, processor system and sensor module. The processor system is the host computer, the sensor module is connected to the processor system through the universal serial bus, and the FPGA hardware acceleration module is connected to the processor system through the AXI bus. The processor system includes a general-purpose processor and memory; the FPGA hardware acceleration module includes an image downsampling module, a feature extraction module and a feature matching module.

The sensor module captures images at a certain frequency, and each time an image is captured, it is transferred to memory in the processor system for temporary storage. Next, the general-purpose processor notifies the feature extraction module to start running. After receiving the notification, the feature extraction module initiates data transfer without the participation of the general-purpose processor, stores the image stored in the memory into the input image cache, and begins to extract features on the image. At the same time, the feature extraction module notifies the image downsampling module to downsample the images stored in the memory to generate image pyramids for further feature extraction. After the feature extraction is completed, the feature extraction module stores the extracted features in the memory and the descriptor cache of the feature matching module, and notifies the general-purpose processor through an interrupt. Then, the general-purpose processor notifies the feature matching module to start running. The feature matching module reads the map points in the global map from the memory, matches the features obtained from the feature extraction module, stores the results in the memory and informs the general-purpose processor after completion. After that, the general-purpose processor performs the following steps: pose estimation, pose optimization, and key frame judgment. If it is a keyframe, the global map is updated.

FIG. 2 is a schematic structural diagram of a feature extraction module of an ORB-SLAM hardware accelerator according to an embodiment of the present invention, where NMS represents a non-maximum value suppression unit. The feature extraction module is used to extract ORB features on the input image. It reads the picture from the memory through the AXI interface, extracts the ORB feature on the picture, and finally writes the result back to the memory through the AXI interface, and sends the feature descriptor to the feature matching module. As shown in Figure 2, the feature extraction module includes:

AXI interface: used to communicate with the processor system. The AXI interface includes a master interface and a slave interface. The slave interface is used to receive instructions from the general-purpose processor in the processor system, and the master interface is used to read and write the memory in the processor system. It should be pointed out that the feature extraction module does not need the participation of the general-purpose processor when reading and writing the memory through the AXI main interface, and the general-purpose processor can process other tasks at the same time.

Keypoint extraction unit: used to extract FAST keypoints on the image and calculate the Harris response value of the keypoints. The key point extraction unit takes a 7x7 pixel area as input, and determines whether the pixel is a FAST key point by comparing the gray value of the central pixel with the gray value of the pixels on the circumference of the circle with a radius of 3. And calculate the response value by calculating the difference between the gray value of the center pixel and the pixel on the circumference.

Gaussian filtering unit: used to blur the image with Gaussian. A 7x7 Gaussian kernel is stored in the Gaussian filtering unit, and a smooth image is generated by the convolution of the Gaussian kernel and the pixels of the input image.

Non-maximum suppression unit: used for non-maximum suppression of key points. The non-maximum suppression unit filters the key points according to the Harris response value of the key point, and only retains one key point with the largest response value in any 3x3 neighborhood.

Feature Orientation Calculation Unit: Used to calculate feature orientations on smoothed images. The feature direction calculation unit takes a circular area with a radius of 15 pixels around the feature position as input, first calculates the gray centroid of the area, and then uses the vector from the feature position to the gray centroid position as the feature direction.

Descriptor Computation Unit: Descriptors for computing features on smooth images. 256 pairs of test positions are stored in the descriptor calculation unit, and a 256-bit descriptor is obtained by comparing the pixel gray values of these 256 pairs of test positions in a 31x31 neighborhood around the feature. When the descriptor calculation unit calculates the descriptor, the test position will be rotated to be consistent with the direction of the feature point to ensure the rotation invariance of the feature.

The first cache unit: consists of an input image cache, a smooth image cache, a response value cache and a feature cache, wherein:

Input image cache, smooth image cache, and response value cache: used to cache the input image, smooth image, and response values of key points, respectively. The three cache areas are designed with a ping-pong architecture and consist of multiple identical caches. When a part of the cache cannot be input due to the output data being occupied, the rest of the cache can receive input data. It should be pointed out that since the feature extraction module adopts a pipeline computing architecture, the data stored in the cache can be discarded after use, so only a small amount of data needs to be stored in the three caches. Taking the input image cache as an example, only 16 lines of pixels of the image need to be stored in the input image cache, and there is no need to store the entire image.

Feature cache: used to save the extracted features and filter the features. The feature cache adopts the maximum heap architecture. While inputting features, the features are sorted according to their response values, and only the features with larger response values are retained.

The process of extracting ORB features from the image by the feature extraction module is as follows: First, a part of the pixels of the image are stored in the input image buffer through the AXI interface. The key point extraction unit and the Gaussian filtering unit start to extract FAST key points and perform Gaussian blurring on this part of the pixels, respectively. In the process of extracting FAST key points, the key point extraction unit calculates the Harris response value of the key point at the same time, and stores the response value in the response value cache (it should be pointed out that the data stored in the response value cache not only represents the key point The response value also indicates whether the pixel is a key point: if the response value is 0, it means that the pixel is not a key point, otherwise it is a key point). The Gaussian filtering unit stores the generated smoothed image in the smoothed image buffer. Then, a non-maximum suppression (NMS) unit performs non-maximum suppression on the extracted keypoints, and a feature orientation calculation unit calculates the feature orientations of the keypoints on the smoothed image. Next, the descriptor computing unit calculates the descriptor of the feature according to the feature direction, and stores the result in the feature cache. After all computations are over, the results saved in the feature cache are sent back to the memory and sent to the feature matching module. It should be pointed out that the various computing units and caches in the feature extraction module are not run in series in the actual operation process, but run in parallel in a pipeline manner.

FIG. 3 is a schematic structural diagram of a feature matching module of an ORB-SLAM hardware accelerator according to an embodiment of the present invention. The feature matching module is used to match the features extracted from the image by the feature extraction module with map points in the global map, including:

AXI interface: consistent with the AXI interface in the feature extraction module.

Matching unit: used to match two sets of descriptors (descriptors of map points in the global map and descriptors of features currently extracted from the current frame). The matching unit includes a plurality of Hamming distance calculation units and comparators. It first calculates the Hamming distance between the descriptors, which represents the similarity between the descriptors. Next, match the two sets of descriptors with the highest similarity.

Descriptor cache and result cache: used to cache descriptors and matching results respectively. The two sets of descriptors stored in the descriptor cache (descriptors of map points in the global map and descriptors of features currently extracted from the previous frame) come from memory and feature extraction modules, respectively. It should be pointed out that, in order to reduce the data transmission overhead, the descriptors of map points stored in the descriptor cache will be updated if and only when the global map is updated.

4 is a schematic diagram of a pipeline when an FPGA hardware acceleration module of an ORB-SLAM hardware accelerator and a processor system run in parallel according to an embodiment of the present invention, wherein the rectangle represents each process in the SLAM workflow, and PE represents pose estimation, PO stands for pose optimization, FE for feature extraction, FM for feature matching, and MU for global map update. Also, PS stands for Processor System.

The ORB-SLAM hardware accelerator runs feature extraction, feature matching, pose estimation, and pose optimization in sequence when processing ordinary frames. Among them, feature extraction and feature matching run in the FPGA hardware acceleration module, while pose estimation and pose optimization run in the processor system. In order to enable the FPGA hardware acceleration module and the processor system to run in parallel and improve the throughput speed, while the processor system performs pose estimation and pose optimization, the FPGA hardware acceleration module starts to perform feature extraction and feature matching on the next frame of image.

When the ORB-SLAM hardware accelerator processes key frames, the difference from the process of processing ordinary frames is that the processor system needs to perform global map update after the pose estimation and pose optimization. Since the updated global map is required for the feature matching of the next frame, when the processor system is running, the FPGA hardware acceleration module will only run the feature extraction of the next frame at the same time, and wait for the global map to be updated before starting to run the feature. match.

The present invention is further explained above by using specific embodiments. It should be noted that the above contents are only specific embodiments of the present invention, and should not be used to limit the present invention. Any modifications, substitutions, improvements, etc. within the spirit of the present invention should fall within the protection scope of the present invention.

Claims (6)

1. An ORB-SLAM hardware accelerator, comprising: the accelerator comprises a sensor module, an FPGA hardware acceleration module and a processor system;

the sensor module is used for acquiring image data;

the FPGA hardware acceleration module is used for extracting ORB features from the image acquired by the sensor module and matching the extracted features with map points in the global map;

the processor system is used for calculating the camera pose and maintaining the global map according to the ORB features extracted by the FPGA hardware acceleration module and the matching result of the ORB features and map points in the global map;

the FPGA hardware acceleration module comprises: the device comprises an image down-sampling module, a feature extraction module and a feature matching module;

the image down-sampling module is used for generating an image pyramid;

the feature extraction module is used for extracting ORB features on each layer of the image pyramid;

the feature matching module is used for matching the ORB features extracted by the feature extraction module with map points in the global map;

the feature extraction module includes: the device comprises a key point extracting unit, a non-maximum value inhibiting unit, a Gaussian filtering unit, a characteristic direction calculating unit, a descriptor calculating unit and a first cache unit;

the key point extracting unit is used for extracting a FAST corner on an input image and calculating a Harris response value of the FAST corner;

the non-maximum suppression unit is used for performing non-maximum suppression on the FAST corner according to the Harris response value of the FAST corner, and reserving the FAST corner with the maximum Harris response value in the neighborhood;

the Gaussian filtering unit is used for carrying out Gaussian blur processing on the input image, removing noise in the image and generating a smooth image;

the characteristic direction calculating unit is used for calculating a characteristic direction on the image after Gaussian filtering;

the descriptor calculation unit is used for calculating a BRIEF descriptor of the feature on the image after Gaussian filtering according to the feature direction;

the first cache unit is used for temporarily storing input data, intermediate calculation results and final results;

the work flow of the characteristic direction calculating unit is as follows: firstly, calculating the pixel gray scale centroid coordinates of the neighborhood where the features are located; then, calculating the ratio of the horizontal coordinate to the vertical coordinate of the gray centroid, and obtaining the direction of the features according to a lookup table;

the first cache unit includes: inputting an image cache, a smooth image cache, a response value cache and a characteristic cache; the input image cache, the smooth image cache and the response value cache adopt a ping-pong-like architecture, are composed of a plurality of same caches and can simultaneously process the input and the output of data; the feature cache adopts a maximum heap architecture, is used for screening the features while preserving the features, and only reserves partial features with large Harris response values.

2. The ORB-SLAM hardware accelerator of claim 1, wherein: each unit of the feature extraction module adopts a pipeline type computing architecture, all computing units run in parallel, only partial data is stored in a cache, and the data is discarded immediately after being used.

3. The ORB-SLAM hardware accelerator of claim 1, wherein: the feature matching module includes: the matching unit and the second cache unit;

the matching unit is used for matching the features extracted by the feature extraction module with map points in the global map; the working process is as follows: firstly, calculating the Hamming distance between any two descriptors in two sets of descriptors; then, matching the two groups of descriptors according to the Hamming distance by using a violent searching method; the two groups of descriptors refer to descriptors of map points in the global map and descriptors of features extracted from a previous frame;

the second cache unit comprises a descriptor cache and a result cache, and is used for temporarily storing two groups of descriptors to be matched and matching results.

4. The ORB-SLAM hardware accelerator of claim 1, wherein: the processor system comprises a general processor, a memory and a memory controller; and the general processor is used for calculating the camera pose according to the extracted image features and the matching relation between the features and map points in the global map, and updating and maintaining the global map.

5. The ORB-SLAM hardware accelerator of claim 4, wherein: the processor system and the FPGA hardware acceleration module communicate via an AXI bus; a general processor in the processor system can directly configure an instruction register in the FPGA hardware acceleration module through an AXI bus; the FPGA hardware acceleration module may also directly read data from the memory in the processor system through the AXI bus, or store the calculation result in the memory.

6. The ORB-SLAM hardware accelerator of claim 1, wherein: the FPGA hardware acceleration module and the processor system run in parallel in a pipeline mode.

Images