Correcting motion distortions in time-of-flight imaging
Article No.: 8, Pages 1 - 7
Abstract
Time-of-flight point cloud acquisition systems have grown in precision and robustness over the past few years. However, even subtle motion can induce significant distortions due to the long acquisition time. In contrast, there exists sensors that produce depth maps at a higher frame rate, but they suffer from low resolution and accuracy. In this paper, we correct distortions produced by small motions in time-of-flight acquisitions and even output a corrected animated sequence by combining a slow but high-resolution time-of-flight LiDAR system and a fast but low-resolution consumer depth sensor. We cast the problem as a curve-to-volume registration, by seeing a LiDAR point cloud as a curve in a 4-dimensional spacetime and the captured low-resolution depth video as a 4-dimensional spacetime volume. Our approach starts by registering both captured sequences in 4D, in a coarse-to-fine approach. It then computes an optical flow between the low-resolution frames and finally transfers high-resolution details by advecting along the flow. We demonstrate the efficiency of our approach on both synthetic data, on which we can compute registration errors, and real data.
Supplementary Material
MP4 File (a8-fulop-balogh.mp4)
- Download
- 64.76 MB
References
[1]
C. Ancuti, C. O. Ancuti, and P. Bekaert. 2010. Video super-resolution using high quality photographs. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing. 862--865.
[2]
S. Berkiten, M. Halber, J. Solomon, C. Ma, H. Li, and S. Rusinkiewicz. 2017. Learning Detail Transfer based on Geometric Features. In Computer Graphics Forum, Proceedings Eurographics 2017.
[3]
P. J. Besl and N. D. McKay. 1992. A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 2 (Feb 1992), 239--256.
[4]
Pravin Bhat, C. Lawrence Zitnick, Noah Snavely, Aseem Agarwala, Maneesh Agrawala, Michael Cohen, Brian Curless, and Sing Bing Kang. 2007. Using Photographs to Enhance Videos of a Static Scene. In Proceedings of the 18th Eurographics Conference on Rendering Techniques (EGSR'07). Eurographics Association, 327--338.
[5]
Derek Bradley, Wolfgang Heidrich, Tiberiu Popa, and Alla Sheffer. 2010. High Resolution Passive Facial Performance Capture. ACM Trans. on Graphics (Proc. SIGGRAPH) 29, 3 (2010).
[6]
D. Herrera C., J. Kannala, and J. Heikkilä. 2012. Joint Depth and Color Camera Calibration with Distortion Correction. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, 10 (Oct 2012), 2058--2064.
[7]
Yaron Caspi and Michal Irani. 2002. Spatio-Temporal Alignment of Sequences. IEEE Trans. Pattern Anal. Mach. Intell. 24, 11 (Nov. 2002), 1409--1424.
[8]
Sean Clarkson, Jonathan Wheat, Ben Heller, J Webster, and Simon Choppin. 2013. Distortion correction of depth data from consumer depth cameras. 3D Body Scanning Technologies, Long Beach, California, Hometrica Consulting (2013), 426--437.
[9]
James Davis, Diego Nehab, Ravi Ramamoorthi, and Szymon Rusinkiewicz. 2005. Spacetime Stereo: A Unifying Framework for Depth from Triangulation. IEEE Trans. Pattern Anal. Mach. Intell. 27, 2 (Feb. 2005), 296--302.
[10]
D. Fleet and Y. Weiss. 2006. Optical Flow Estimation. Springer US, Boston, MA, 237--257.
[11]
Ankit Gupta, Pravin Bhat, Mira Dontcheva, Oliver Deussen, Brian Curless, and Michael Cohen. 2008. Enhancing and Experiencing Spacetime Resolution with Videos and Stills, In International Conference on Computational Photography.
[12]
Bjoern Haefner, Yvain Quéau, Thomas Möllenhof, and Daniel Cremers. 2018. Fight Ill-Posedness With Ill-Posedness: Single-Shot Variational Depth Super-Resolution From Shading. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[13]
A. Hamdi-Cherif, J. Digne, and Chaine R. 2017. Super-resolution of Point Set Surfaces using Local Similarities. to appear in Computer Graphics Forum (2017).
[14]
E. Herbst, X. Ren, and D. Fox. 2013. RGB-D flow: Dense 3-D motion estimation using color and depth. In 2013 IEEE International Conference on Robotics and Automation. 2276--2282.
[15]
F. Huguet and F. Devernay. 2007. A Variational Method for Scene Flow Estimation from Stereo Sequences. In 2007 IEEE 11th International Conference on Computer Vision. 1--7.
[16]
Yong Joo Kil, Boris Mederos, and Nina Amenta. 2006. Laser Scanner Super-resolution. In Proc. Point-Based Graphics (SPBG'06). 9--16.
[17]
T. P. Koninckx and L. Van Gool. 2006. Real-time range acquisition by adaptive structured light. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 3 (March 2006), 432--445.
[18]
D. Liu, J. Gu, Y. Hitomi, M. Gupta, T. Mitsunaga, and S. K. Nayar. 2014. Efficient Space-Time Sampling with Pixel-Wise Coded Exposure for High-Speed Imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 2 (Feb 2014), 248--260.
[19]
Niloy J. Mitra, Simon Flöry, Maks Ovsjanikov, Natasha Gelfand, Leonidas Guibas, and Helmut Pottmann. 2007. Dynamic Geometry Registration. In Proc. Symposium on Geometry Processing (SGP '07). Eurographics, 173--182.
[20]
OpenNI organization. 2017. OpenNI User Guide.
[21]
J. Quiroga, T. Brox, F. Devernay, and J. Crowley. 2014. Dense semi-rigid scene flow estimation from RGBD images. In European Conference on Computer Vision (ECCV). http://lmb.informatik.uni-freiburg.de/Publications/2014/Bro14
[22]
Damien Rohmer, Tiberiu Popa, Marie-Paule Cani, Stefanie Hahmann, and Alla Sheffer. 2010. Animation Wrinkling: Augmenting Coarse Cloth Simulations with Realistic-looking Wrinkles. In ACM SIGGRAPH Asia 2010 Papers. ACM, New York, NY, USA, Article 157, 8 pages.
[23]
E. Shechtman, Y. Caspi, and M. Irani. 2005. Space-time super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence 27, 4 (April 2005), 531--545.
[24]
Thibaut Weise, Sofien Bouaziz, Hao Li, and Mark Pauly. 2011. Realtime Performance-based Facial Animation. ACM Trans. Graph. 30, 4, Article 77 (July 2011), 10 pages.
[25]
T. Weise, B. Leibe, and L. Van Gool. 2007. Fast 3D Scanning with Automatic Motion Compensation. In 2007 IEEE Conference on Computer Vision and Pattern Recognition. 1--8.
[26]
Li Zhang, Brian Curless, and Steven M. Seitz. 2003. Spacetime Stereo: Shape Recovery for Dynamic Scenes. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 367--374.
[27]
Li Zhang, Noah Snavely, Brian Curless, and Steven M. Seitz. 2004. Spacetime Faces: High Resolution Capture for Modeling and Animation. ACM Trans. Graph. 23, 3 (Aug. 2004), 548--558.
[28]
Song Zhang. 2010. Recent progresses on real-time 3D shape measurement using digital fringe projection techniques. Optics and Lasers in Engineering 48, 2 (2010), 149 -- 158. Fringe Projection Techniques.
Index Terms
- Correcting motion distortions in time-of-flight imaging
Recommendations
Doppler time-of-flight imaging
Over the last few years, depth cameras have become increasingly popular for a range of applications, including human-computer interaction and gaming, augmented reality, machine vision, and medical imaging. Many of the commercially-available devices use ...
Doppler time-of-flight imaging
SIGGRAPH '15: ACM SIGGRAPH 2015 Emerging TechnologiesOver the last few years, depth cameras have become increasingly popular for a range of applications, including human-computer interaction and gaming, augmented reality, machine vision, and medical imaging. Many of the commercially-available devices use ...
Frame Rate Up-Conversion for Depth-Based 3D Video
ICME '12: Proceedings of the 2012 IEEE International Conference on Multimedia and ExpoA novel frame rate up-conversion (FRUC) scheme for depth-based 3D video is proposed. Differing from the existing conventional FRUC methods which are designed for two dimensional (2D) video only, the proposed method is designed for depth-based 3D video, ...
Comments
Information & Contributors
Information
Published In
November 2018
185 pages
ISBN:9781450360159
DOI:10.1145/3274247
- Conference Chairs:
- Panayiotis Charalambous,
- Yiorgos Chrysanthou,
- Program Chairs:
- Ben Jones,
- Jehee Lee
Copyright © 2018 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 08 November 2018
Check for updates
Author Tags
Qualifiers
- Short-paper
Funding Sources
- Agence Nationale de la Recherche
Conference
MIG '18
Sponsor:
Acceptance Rates
Overall Acceptance Rate -9 of -9 submissions, 100%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 94Total Downloads
- Downloads (Last 12 months)7
- Downloads (Last 6 weeks)0
Reflects downloads up to 09 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in