A Robust Indoor Localization System Integrating Visual Localization Aided by CNN-Based Image Retrieval with Monte Carlo Localization
<p>Overview of the proposed localization system.</p> "> Figure 2
<p>Architecture of the proposed image retrieval scheme.</p> "> Figure 3
<p>Architecture of the proposed LSTM network.</p> "> Figure 4
<p>The mobile robot used to validate the proposed localization approach.</p> "> Figure 5
<p>Classes for object detection in dataset.</p> "> Figure 6
<p>The grid map in the experimental site.</p> "> Figure 7
<p>Three image retrieval results in the stage of coarse place recognition. The first column is the query image with a blue contour and the others are retrieved images. The images with red contour are mismatching.</p> "> Figure 8
<p>CDF of localization accuracy with and without image-based localization.</p> "> Figure 9
<p>(<b>a</b>) Typical evolution of number of samples of two methods in global localization. (<b>b</b>) CDF of localization accuracy of two methods in global localization.</p> "> Figure 10
<p>Localization error of three methods in global localization.</p> "> Figure 11
<p>The motion path of the robot when it is kidnapped.</p> "> Figure 12
<p>Typical localization error of three methods when the robot is kidnapped.</p> ">
Abstract
:1. Introduction
2. Related Work
2.1. Image Retrieval
2.2. Monte Carlo Localization
3. Proposed Multi-Sensor-Based Indoor Localization System
3.1. Image-Based Localization
3.1.1. Offline Phase
3.1.2. Online Phase
3.2. Laser-Based Localization
3.2.1. Prediction Phase
3.2.2. Update Phase
Algorithm 1: Adaptive Monte Carlo localization |
Input: observation information , control information ; Output: ;
|
3.3. Image Sequence Correlation with LSTM
4. Experiments
4.1. Experimental Setup
4.2. Coarse Place Recognition
4.3. Accurate Localization
4.3.1. Accurate Localization Estimation
4.3.2. Impact of the Image-Based Localization
4.3.3. Impact of the Laser-Based Localization
4.3.4. Evaluation of Proposed Localization System
4.4. Re-Localization from Kidnapping
5. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Thrun, S.; Burgard, W.; Fox, D. Probabilistic Robotics; MIT Press: Cambridge, MA, USA, 2005. [Google Scholar]
- Thrun, S.; Fox, D.; Burgard, W.; Dallaert, F. Robust Monte Carlo localization for mobile robots. Artif. Intell. 2001, 128, 99–141. [Google Scholar] [CrossRef] [Green Version]
- Thrun, S.; Fox, D.; Burgard, W. Monte Carlo Localization with Mixture Proposal Distribution. In Proceedings of the National Conference on Artificial Intelligence (AAAI), Austin, TX, USA, 30 July–3 August 2000; pp. 859–865. [Google Scholar]
- Liu, H.; Darabi, H.; Banerjee, P.; Liu, J. Survey of Wireless Indoor Positioning Techniques and Systems. IEEE Trans. Syst. Man Cybern. C. 2007, 37, 1067–1080. [Google Scholar] [CrossRef]
- Jiang, Y.; Pan, X.; Li, K.; Lv, Q.; Dick, RP.; Hannigan, M.; Shang, L. ARIEL: Automatic wi-fi based room fingerprinting for indoor localization. In Proceedings of the ACM International Conference on Ubiquitous Computing (UbiComp), Pittsburgh, PA, USA, 5–8 September 2012; pp. 441–450. [Google Scholar]
- Dellaert, F.; Fox, D.; Burgard, W.; Thrun, S. Monte Carlo localization for mobile robots. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Detroit, MI, USA, 10–15 May 1999; pp. 1322–1328. [Google Scholar]
- Fox, D.; Burgard, W.; Thrun, S. Markov localization for mobile robots in dynamic environments. J. Artif. Intell. Res. 1999, 2, 327–391. [Google Scholar] [CrossRef]
- Roumeliotis, S.I.; Bekey, G.A.; Burgard, W.; Thrun, S. Bayesian estimation and Kalman filtering: A unified framework for mobile robot localization. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), San Francisco, CA, USA, 24–28 April 2000; pp. 2985–2992. [Google Scholar]
- Park, S.; Roh, K.S. Coarse-to-Fine Localization for a Mobile Robot Based on Place Learning With a 2-D Range Scan. IEEE Trans. Robot. 2016, 32, 528–544. [Google Scholar] [CrossRef]
- Wang, J.; Zha, H.; Cipolla, R. Coarse-to-fine vision-based localization by indexing scale-Invariant features. IEEE Trans. Syst. Man Cybern. B. 2006, 36, 413–422. [Google Scholar] [CrossRef]
- Sattler, T.; Leibe, B.; Kobbelt, L. Image retrieval for image-based localization revisited. In Proceedings of the British Machine Vision Conference (BMVC), Guildford, Surrey, UK, 3–7 September 2012; pp. 1–12. [Google Scholar]
- Sattler, T.; Havlena, M.; Scjindler, K.; Pollefeys, M. Large-Scale Location Recognition and the Geometric Burstiness Problem. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 26 June–1 July 2016; pp. 1582–1590. [Google Scholar]
- Sunderhaulf, N.; Dayoub, F.; McMahon, S.; Talbot, B.; Schulz, R.; Corke, P.; Wyeth, G.; Upcroft, B.; Milford, M. Place categorization and semantic mapping on a mobile robot. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden, 16–21 May 2016; pp. 5729–5736. [Google Scholar]
- Zeisl, B.; Sattler, T.; Pollefeys, M. Camera Pose Voting for Large-Scale Image-Based Localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 13–16 December 2015; pp. 2704–2712. [Google Scholar]
- Zamir, A.R.; Shah, M. Image Geo-Localization Based on MultipleNearest Neighbor Feature Matching UsingGeneralized Graphs. IEEE Trans. Pattern Anal. Mach. Intell. 2014, 36, 1546–1558. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Philbin, J.; Chum, O.; Isard, M.; Sivic, J.; Zisserman, A. Object retrieval with large vocabularies and fast spatial matching. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Minneapolis, MN, USA, 18–23 June 2007; pp. 1–8. [Google Scholar]
- Biswas, J.; Veloso, M. Multi-sensor Mobile Robot Localization for Diverse Environments. In Proceedings of the Robot Soccer World Cup (RobotCup), Eindhoven, The Netherlands, 24 June–1 July 2013; pp. 468–479. [Google Scholar]
- Srinivasan, K.; Gu, J. Multiple Sensor Fusion in Mobile Robot Localization. In Proceedings of the Canadian Conference on Electrical and Computer Engineering (CCECE), Vancouver, BC, Canada, 22–26 April 2007; pp. 1207–1210. [Google Scholar]
- Duan, P.; Tian, G.; Wu, H. A multi-sensor-based mobile robot localization framework. In Proceedings of the IEEE International Conference on Robotics and Biomimetics (ROBIO), Zhuhai, China, 6–9 December 2015; pp. 642–647. [Google Scholar]
- Wolf, J.; Burgard, W.; Burkhardt, H. Robust vision-based localization by combining an image-retrieval system with Monte Carlo localization. IEEE Trans. Robot 2005, 21, 208–216. [Google Scholar] [CrossRef]
- Irschara, A.; Zach, C.; Frahm, J.M.; Bischof, H. From structure-from-motion point clouds to fast location recognition. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Kyoto, Japan, 27 September–4 October 2009; pp. 2599–2606. [Google Scholar]
- Sattler, T.; Leibe, B.; Kobbelt, L. Efficient & Effective Prioritized Matching for Large-Scale Image-Based Localization. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 1744–1756. [Google Scholar] [PubMed]
- Zamir, A.R.; Shah, M. Accurate Image Localization Based on Google Maps Street View. In Proceedings of the IEEE International Conference on Computer Vision (ECCV), Heraklion, Crete, Greece, 5–11 September 2010; pp. 255–268. [Google Scholar]
- Kim, H.; Lee, D.; Oh, T.; Myung, H. A Probabilistic Feature Map-Based Localization System Using a Monocular Camera. Sensors 2015, 15, 21636–21659. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Smeulders, A.W.M.; Worring, M.; Santini, S.; Gupta, A.; Jain, R. Content-Based Image Retrieval at the End of the Early Years. IEEE Trans. Pattern Anal. Mach. Intell. 2000, 22, 1349–1380. [Google Scholar] [CrossRef]
- Jing, F.; Li, M.; Zhang, H.J.; Zhang, B. A unified framework for image retrieval using keyword and visual features. IEEE Trans. Image Process. 2005, 14, 979–989. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zhou, X.S.; Huang, T.S. Relevance feedback in image retrieval: A comprehensive review. Multimedia Syst. 2003, 8, 536–544. [Google Scholar] [CrossRef]
- Smith, J.R.; Chang, S.F. VisualSEEk: A fully automated content-based image query system. In Proceedings of the Acm International Conference on Multimedia (ACMMM), Boston, MA, USA, 18–22 November 1996; pp. 87–98. [Google Scholar]
- Wu, V.; Manmatha, R.; Riseman, E.M. TextFinder: An Automatic System to Detect and Recognize Text In Images. IEEE Trans. Pattern Anal. Mach. Intell. 1999, 21, 1224–1229. [Google Scholar] [CrossRef]
- Jung, K.; Kim, K.I.; Jain, A.K. Text information extraction in images and video: A survey. Pattern Recognit. 2004, 37, 977–997. [Google Scholar] [CrossRef]
- Er, N.; Stewennius, H. Scalable recognition with a vocabulary tree. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), New York, NY, USA, 17–22 June 2006; pp. 2161–2168. [Google Scholar]
- Zhu, C.Z.; Satoh, S. Large vocabulary quantization for searching instances from videos. In Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR), Hong Kong, China, 5–8 June 2012; pp. 1–8. [Google Scholar]
- Jegou, H.; Douze, M.; Schmid, C. Hamming Embedding and Weak Geometry Consistency for Large Scale Image Search. In Proceedings of the IEEE International Conference on Computer Vision (ECCV), Marseille, France, 12–18 October 2008; pp. 304–317. [Google Scholar]
- Jegou, H.; Douze, M.; Schmid, C.; Perez, P. Aggregating local descriptors into a compact image representation. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, CA, USA, 13–18 June 2010; pp. 3304–3311. [Google Scholar]
- Gong, Y.; Wang, L.; Guo, R.; Lazebnik, S. Multi-scale Orderless Pooling of Deep Convolutional Activation Features. In Proceedings of the IEEE International Conference on Computer Vision (ECCV), Zurich, Switzerland, 6–12 September 2014; pp. 392–407. [Google Scholar]
- Arandjelovic, R.; Gronat, P.; Torii, A.; Pajdla, T.; Sivic, J. NetVLAD: CNN Architecture for Weakly Supervised Place Recognition. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 5297–5307. [Google Scholar]
- Radenović, F.; Tolias, G.; Chum, O. CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples. In Proceedings of the IEEE International Conference on Computer Vision (ECCV), Amsterdam, The Netherlands, 11–14 October 2016; pp. 3–20. [Google Scholar]
- Liu, H.; Tian, Y.; Wang, Y.; Pang, L.; Huang, T. Deep Relative Distance Learning: Tell the Difference between Similar Vehicles. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 2167–2175. [Google Scholar]
- Babenko, A.; Slesarev, A.; Chigorin, A.; Lempitsky, V. Neural Codes for Image Retrieval. In Proceedings of the IEEE International Conference on Computer Vision (ECCV), Zurich, Switzerland, 6–12 September 2014; pp. 584–599. [Google Scholar]
- Ng, J.Y.H.; Yang, F.; Davis, L.S. Exploiting local features from deep networks for image retrieval. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Boston, MA, USA, 7–12 June 2015; pp. 53–61. [Google Scholar]
- Gordo, A.; Almazán, J.; Revaud, J.; Larlus, D. Deep Image Retrieval: Learning Global Representations for Image Search. In Proceedings of the IEEE International Conference on Computer Vision (ECCV), Amsterdam, The Netherlands, 11–14 October 2016; pp. 241–257. [Google Scholar]
- Salvador, A.; Giroinieto, X.; Marques, F.; Satoh, S. Faster R-CNN Features for Instance Search. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 26 June–1 July 2016; pp. 394–401. [Google Scholar]
- Merwe, R.V.D.; Doucet, A.; Freitas, N.D. The unscented particle filter. In Proceedings of the 13th International Conference on Neural Information Processing Systems (NIPS), Denver, CO, USA, 1–2 December 2000; pp. 563–569. [Google Scholar]
- Kim, J.; Lin, Z.; Kweon, I.S. Rao-Blackwellized particle filtering with Gaussian mixture models for robust visual tracking. Comput. Vis. Image Underst. 2014, 125, 128–137. [Google Scholar] [CrossRef]
- Zhang, L.; Zapata, R.; Lépinay, P. Self-adaptive Monte Carlo localization for mobile robots using range finders. Robot 2012, 30, 229–244. [Google Scholar] [CrossRef]
- Sattler, T.; Torii, A.; Sivic, J.; Pollefeys, M.; Taira, H.; Okutomi, M.; Pajdla, T. Are Large-Scale 3D Models Really Necessary for Accurate Visual Localization? In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 1637–1646. [Google Scholar]
- Hao, J.D.; Dong, J.; Wang, W.; Tan, T.N. What Is the Best Practice for CNNs Applied to Visual Instance Retrieval? In Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France, 24–26 April 2017; pp. 1–13. [Google Scholar]
- Ullah, A.; Ahmad, J.; Muhammad, K.; Sajjad, M.; Baik, S.W. Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features. IEEE Access 2018, 6, 1155–1166. [Google Scholar] [CrossRef]
- Zaremba, W.; Sutskever, I.; Vinyals, O. Recurrent Neural Network Regularization. arXiv, 2014; arXiv:1409.2329. [Google Scholar]
- Luong, M.T.; Sutskever, I.; Le, Q.V.; Vinyals, O.; Zaremba, W. Addressing the Rare Word Problem in Neural Machine Translation. arXiv, 2014; arXiv:1410.8206. [Google Scholar]
- Varior, R.R.; Shuai, B.; Lu, J.; Xu, D.; Wang, G. A Siamese Long Short-Term Memory Architecture for Human Re-identification. In Proceedings of the IEEE International Conference on Computer Vision (ECCV), Amsterdam, The Netherlands, 11–14 October 2016; pp. 135–153. [Google Scholar]
- Donahue, J.; Hendricks, L.A.; Marcus, R.; Venugopalan, S.; Guadarrama, S.; Saenko, K.; Darrell, T. Long-term Recurrent Convolutional Networks for Visual Recognition and Description. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015; pp. 2625–2634. [Google Scholar]
X (cm) | Y (cm) | () | |
---|---|---|---|
Candidate node 1 | 1592.87 | 3720.63 | −172.11 |
Candidate node 4 | 3681.13 | 304.76 | 94.25 |
Candidate node 5 | −1548.34 | 3728.05 | −169.68 |
Fine pose | 1572.23 | 3723.51 | −179.64 |
Ground truth | 1561.45 | 3730.72 | −173.42 |
Error | −10.78 | 7.21 | 6.22 |
© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Xu, S.; Chou, W.; Dong, H. A Robust Indoor Localization System Integrating Visual Localization Aided by CNN-Based Image Retrieval with Monte Carlo Localization. Sensors 2019, 19, 249. https://doi.org/10.3390/s19020249
Xu S, Chou W, Dong H. A Robust Indoor Localization System Integrating Visual Localization Aided by CNN-Based Image Retrieval with Monte Carlo Localization. Sensors. 2019; 19(2):249. https://doi.org/10.3390/s19020249
Chicago/Turabian StyleXu, Song, Wusheng Chou, and Hongyi Dong. 2019. "A Robust Indoor Localization System Integrating Visual Localization Aided by CNN-Based Image Retrieval with Monte Carlo Localization" Sensors 19, no. 2: 249. https://doi.org/10.3390/s19020249