research-article

Eventor: an efficient event-based monocular multi-view stereo accelerator on FPGA platform

Authors:

Bei Yu, Weisheng ZhaoAuthors Info & Claims

DAC '22: Proceedings of the 59th ACM/IEEE Design Automation Conference

Pages 331 - 336

https://doi.org/10.1145/3489517.3530452

Published: 23 August 2022 Publication History

Abstract

Event cameras are bio-inspired vision sensors that asynchronously represent pixel-level brightness changes as event streams. Event-based monocular multi-view stereo (EMVS) is a technique that exploits the event streams to estimate semi-dense 3D structure with known trajectory. It is a critical task for event-based monocular SLAM. However, the required intensive computation workloads make it challenging for real-time deployment on embedded platforms. In this paper, Eventor is proposed as a fast and efficient EMVS accelerator by realizing the most critical and time-consuming stages including event back-projection and volumetric ray-counting on FPGA. Highly paralleled and fully pipelined processing elements are specially designed via FPGA and integrated with the embedded ARM as a heterogeneous system to improve the throughput and reduce the memory footprint. Meanwhile, the EMVS algorithm is reformulated to a more hardware-friendly manner by rescheduling, approximate computing and hybrid data quantization. Evaluation results on DAVIS dataset show that Eventor achieves up to 24X improvement in energy efficiency compared with Intel i5 CPU platform.

References

[1]

Christian Brandli, Raphael Berner, Minhao Yang, Shih-Chii Liu, and Tobi Delbruck. A 240X180 130dB 3μs latency global shutter spatiotemporal vision sensor. IEEE JSSC, 49(10):2333--2341, 2014.

[2]

Bongki Son, Yunjae Suh, Sungho Kim, et al. A 640X 480 dynamic vision sensor with a 9μm pixel and 300meps address-event representation. In IEEE ISSCC, pages 66--67. IEEE, 2017.

[3]

Cesar Cadena, Luca Carlone, et al. Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transactions on Robotics, 32(6):1309--1332, 2016.

Digital Library

[4]

Yi Zhou, Guillermo Gallego, and Shaojie Shen. Event-based stereo visual odometry. IEEE Transactions on Robotics, 37(5):1433--1450, 2021.

[5]

Yi Zhou, Guillermo Gallego, Henri Rebecq, Laurent Kneip, Hongdong Li, and Davide Scaramuzza. Semi-dense 3D reconstruction with a stereo event camera. In Proceedings of ECCV, pages 235--251, 2018.

[6]

Guillermo Gallego, Tobi Delbrück, et al. Event-based vision: A survey. IEEE TPAMI, 44(1):154--180, 2020.

Digital Library

[7]

Henri Rebecq, Guillermo Gallego, et al. EMVS: Event-based multi-view stereo---3D reconstruction with an event camera in real-time. IJCV, 126(12):1394--1414, 2018.

Digital Library

[8]

Hanme Kim et al. Real-time 3D reconstruction and 6-DOF tracking with an event camera. In Proceedings of ECCV, pages 349--364. Springer, 2016.

[9]

Guillermo Gallego, Henri Rebecq, and Davide Scaramuzza. A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation. In Proceedings of CVPR, pages 3867--3876, 2018.

[10]

Robert T Collins. A space-sweep approach to true multi-image matching. In Proceedings of CVPR, pages 358--363. IEEE, 1996.

[11]

Henri Rebecq, Timo Horstschäfer, et al. EVO: A geometric approach to event-based 6-DOF parallel tracking and mapping in real time. IEEE RAL, 2(2):593--600, 2016.

[12]

Elias Mueggler, Henri Rebecq, Guillermo Gallego, Tobi Delbruck, and Davide Scaramuzza. The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAM. IJRR, 36(2):142--149, 2017.

Digital Library

[13]

Runze Liu, Jianlei Yang, Yiran Chen, and Weisheng Zhao. eSLAM: An energy-efficient accelerator for real-time ORB-SLAM on FPGA platform. In Proceedings of DAC, pages 1--6, 2019.

[14]

Wenzhi Fu, Jianlei Yang, Pengcheng Dai, Yiran Chen, and Weisheng Zhao. A scalable pipelined dataflow accelerator for object region proposals on FPGA platform. In Proceedings of FPT, pages 346--349. IEEE, 2018.

[15]

Xilinx. Zynq-7000 SoC. https://www.xilinx.com/products/silicondevices/soc/zynq-7000.html, 2018.

[16]

Intel Chip's Specifications. Intel Core i5-7300HQ Processor. https://ark.intel.com/content/www/us/en/ark/products/97456/intel-core-i57300hq-processor-6m-cache-up-to-3-50-ghz.html, 2017.

Cited By

Li DFeng XShen CChen QYang LQiu SJin XLiu M(2023)Vector-Based Dedicated Processor Architecture for Efficient Tracking in VSLAM SystemsIEEE Embedded Systems Letters10.1109/LES.2023.329890015:4(182-185)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1109/LES.2023.3298900
Kalra S(2023)Effective Co-Segmentation and Data Augmentation for Unsupervised Multi-View Stereo2023 2nd International Conference on Futuristic Technologies (INCOFT)10.1109/INCOFT60753.2023.10425573(1-6)Online publication date: 24-Nov-2023
https://doi.org/10.1109/INCOFT60753.2023.10425573
Yan GLiu XWang HHa Y(2023)Fast FPGA Accelerator of Graph Cut Algorithm with Out-of-order Parallel Execution in Folding Grid Architecture2023 60th ACM/IEEE Design Automation Conference (DAC)10.1109/DAC56929.2023.10247784(1-6)Online publication date: 9-Jul-2023
https://doi.org/10.1109/DAC56929.2023.10247784

Recommendations

EMVS: Event-Based Multi-View Stereo--3D Reconstruction with an Event Camera in Real-Time

Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. They offer significant advantages over standard cameras, namely a very high dynamic range, no motion blur, and a latency in ...
Stereo fusion

A stereo fusion system that combines binocular and refractive stereo is presented.Our stereo fusion outperforms traditional binocular and refractive stereo.An efficient calibration method for refractive stereo is proposed. Display Omitted The ...
Portable and scalable FPGA-based acceleration of a direct linear system solver

FPGAs have the potential to serve as a platform for accelerating many computations including scientific applications. However, the large development cost and short life span for FPGA designs have limited their adoption by the scientific computing ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

DAC '22: Proceedings of the 59th ACM/IEEE Design Automation Conference

July 2022

1462 pages

ISBN:9781450391429

DOI:10.1145/3489517

General Chair:
Rob Oshana
NXP

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGDA: ACM Special Interest Group on Design Automation
IEEE CEDA

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 August 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSFC

Conference

DAC '22

Sponsor:

SIGDA

DAC '22: 59th ACM/IEEE Design Automation Conference

July 10 - 14, 2022

California, San Francisco

Acceptance Rates

Overall Acceptance Rate 1,770 of 5,499 submissions, 32%

Upcoming Conference

DAC '25

Sponsor:
sigda

62nd ACM/IEEE Design Automation Conference

June 22 - 26, 2025

San Francisco , CA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
208
Total Downloads

Downloads (Last 12 months)70
Downloads (Last 6 weeks)6

Reflects downloads up to 14 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li DFeng XShen CChen QYang LQiu SJin XLiu M(2023)Vector-Based Dedicated Processor Architecture for Efficient Tracking in VSLAM SystemsIEEE Embedded Systems Letters10.1109/LES.2023.329890015:4(182-185)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1109/LES.2023.3298900
Kalra S(2023)Effective Co-Segmentation and Data Augmentation for Unsupervised Multi-View Stereo2023 2nd International Conference on Futuristic Technologies (INCOFT)10.1109/INCOFT60753.2023.10425573(1-6)Online publication date: 24-Nov-2023
https://doi.org/10.1109/INCOFT60753.2023.10425573
Yan GLiu XWang HHa Y(2023)Fast FPGA Accelerator of Graph Cut Algorithm with Out-of-order Parallel Execution in Folding Grid Architecture2023 60th ACM/IEEE Design Automation Conference (DAC)10.1109/DAC56929.2023.10247784(1-6)Online publication date: 9-Jul-2023
https://doi.org/10.1109/DAC56929.2023.10247784

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents