Efficient and Accurate Hierarchical SfM Based on Adaptive Track Selection for Large-Scale Oblique Images
"> Figure 1
<p>Workflow of the proposed methodology.</p> "> Figure 2
<p>Illustration of image grouping.</p> "> Figure 3
<p>Illustration of track georeferencing.</p> "> Figure 4
<p>Illustration of adaptive track selection: (<b>a</b>) first iteration; (<b>b</b>) second iteration.</p> "> Figure 5
<p>Orthophoto of survey area.</p> "> Figure 6
<p>Position of exposures.</p> "> Figure 7
<p>Sample images: (<b>a</b>) oblique image; (<b>b</b>) nadir image.</p> "> Figure 8
<p>Visualization of tracks with lengths in the following ranges: (<b>a</b>) 1–10; (<b>b</b>) 11–35; (<b>c</b>) 36–60; (<b>d</b>) 61–130.</p> "> Figure 9
<p>Relationship between the MNO and number of selected tracks.</p> "> Figure 10
<p>Histogram of image observations: (<b>a</b>) all tracks; (<b>b</b>) adaptively selected tracks (MNO = 50).</p> "> Figure 11
<p>Visualization of adaptively selected tracks (MNO = 50) with lengths in the following ranges: (<b>a</b>) 1–10; (<b>b</b>) 11–35; (<b>c</b>) 36–60; and (<b>d</b>) 61–130.</p> "> Figure 12
<p>Reconstructed submaps: (<b>a</b>) Submap 1; (<b>b</b>) Submap 2; (<b>c</b>) Submap 3; (<b>d</b>) Submap 4; and (<b>e</b>) Submap 5.</p> "> Figure 13
<p>Positional difference histograms of common tracks between Submaps 1 and 5: (<b>a</b>) <span class="html-italic">X</span> axis; (<b>b</b>) <span class="html-italic">Y</span> axis; and (<b>c</b>) <span class="html-italic">Z</span> axis.</p> "> Figure 14
<p>Common tracks between submaps: (<b>a</b>) 5-1; (<b>b</b>) 5-2; (<b>c</b>) 5-3; and (<b>d</b>) 5-4.</p> "> Figure 15
<p>Image observations of an outlier detected in common tracks between Submaps 1 and 5: (<b>a</b>) cam1_0009.jpg; (<b>b</b>) cam1_0010.jpg; (<b>c</b>) cam5_0004.jpg; and (<b>d</b>) cam5_0005.jpg.</p> "> Figure 16
<p>Incremental submap merging: (<b>a</b>) 5-1; (<b>b</b>) 5-1-2; (<b>c</b>) 5-1-2-3; and (<b>d</b>) 5-1-2-3-4.</p> ">
Abstract
:1. Introduction
2. Methodology
2.1. Image Grouping Based on Traversal of Match Pairs
Algorithm 1 Image grouping by traversal of match pairs |
Input: match pairs ; each pair specifies the match relationship between image i and j |
Output: image groups |
Initialization: |
1: for each pair in |
2: if is not processed |
3: initialize a new group |
4: initialize a stack , add and to |
5: while is not empty |
6: pop top element from , add to |
7: for each match pair in |
8: if is not processed |
9: push into |
10: end if |
11: end for |
12: end while |
13: add to |
14: end if |
15: end for |
16: return |
2.2. Adaptive Track Selection
Algorithm 2 Adaptive track selection |
Input: tracks , where each track stores its image observations; the value of ; the number of observations operator |
Output: selected tracks |
Initilization: |
1: for each track in |
2: georeference |
3: end for |
4: calculate |
5: initialize a set |
6: add all the images into |
7: initialize the ground sample distance |
8: while is not empty |
9: initialize 2D grid with |
10: for each track in |
11: if is in |
12: continue |
13: end if |
14: find the images to which is visible |
15: if |
16: continue |
17: end if |
18: find the cell in which lies |
19: if is occupied by another track and |
20: continue |
21: else |
22: stores in |
23: end if |
24: end for |
25: for each cell in grid |
26: add the track stored in to |
27: end for |
28: update according to |
29: |
30: end while |
31: return |
2.3. Parallel Submap Reconstruction and Incremental Submap Merging
3. Experimental Results
3.1. Survey Area and Data Specification
3.2. Pairwise Matching and Image Grouping
3.3. Track Selection
3.4. Submap Reconstruction and Merging
3.5. Comparison with Software Packages
4. Discussion
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Colomina, I.; Molina, P. Unmanned aerial systems for photogrammetry and remote sensing: A review. ISPRS J. Photogramm. Remote Sens. 2014, 92, 79–97. [Google Scholar] [CrossRef] [Green Version]
- Nex, F.; Armenakis, C.; Cramer, M.; Cucci, D.A.; Gerke, M.; Honkavaara, E.; Kukko, A.; Persello, C.; Skaloud, J. UAV in the advent of the twenties: Where we stand and what is next. ISPRS J. Photogramm. Remote Sens. 2022, 184, 215–242. [Google Scholar] [CrossRef]
- Haala, N.; Kada, M. An update on automatic 3D building reconstruction. ISPRS J. Photogramm. Remote Sens. 2010, 65, 570–580. [Google Scholar] [CrossRef]
- Duarte, D.; Nex, F.; Kerle, N.; Vosselman, G. Detection of seismic façade damages with multi-temporal oblique aerial imagery. GIScience Remote Sens. 2020, 57, 670–686. [Google Scholar] [CrossRef]
- Giordan, D.; Hayakawa, Y.; Nex, F.; Remondino, F.; Tarolli, P. Review article: The use of remotely piloted aircraft systems (RPASs) for natural hazards monitoring and management. Nat. Hazards Earth Syst. Sci. 2018, 18, 1079–1096. [Google Scholar] [CrossRef] [Green Version]
- Vetrivel, A.; Gerke, M.; Kerle, N.; Nex, F.; Vosselman, G. Disaster damage detection through synergistic use of deep learning and 3D point cloud features derived from very high resolution oblique aerial images, and multiple-kernel-learning. ISPRS J. Photogramm. Remote Sens. 2018, 140, 45–59. [Google Scholar] [CrossRef]
- Fernández-Hernandez, J.; González-Aguilera, D.; Rodríguez-Gonzálvez, P.; Mancera-Taboada, J. Image-based modelling from unmanned aerial vehicle (uav) photogrammetry: An effective, low-cost tool for archaeological applications. Archaeometry 2015, 57, 128–145. [Google Scholar] [CrossRef]
- Nex, F.; Remondino, F. UAV for 3D mapping applications: A review. Appl. Geomat. 2014, 6, 1–15. [Google Scholar] [CrossRef]
- Lowe, D.G. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 2004, 60, 91–110. [Google Scholar] [CrossRef]
- Hartley, R.; Zisserman, A. Multiple View Geometry in Computer Vision, 2nd ed.; Cambridge University Press: Cambridge, UK, 2004. [Google Scholar]
- Gerke, M.; Nex, F.; Remondino, F.; Jacobsen, K.; Kremer, J.; Karel, W.; Hu, H.; Ostrowski, W. Orientation of oblique airborne image sets-experiences from the ISPRS/EUROSDR benchmark on multi-platform photogrammetry. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2016, XLI-B1, 185–191. [Google Scholar] [CrossRef] [Green Version]
- Hirschmüller, H. Stereo Processing by Semiglobal Matching and Mutual Information. IEEE Trans. Pattern Anal. Mach. Intell. 2008, 30, 328–341. [Google Scholar] [CrossRef] [PubMed]
- Jiang, S.; Jiang, C.; Jiang, W. Efficient structure from motion for large-scale UAV images: A review and a comparison of SfM tools. ISPRS J. Photogramm. Remote Sens. 2020, 167, 230–251. [Google Scholar] [CrossRef]
- Snavely, N.; Seitz, S.M.; Szeliski, R. Photo tourism: Exploring photo collections in 3D. ACM Trans. Graph. 2006, 25, 835–846. [Google Scholar] [CrossRef] [Green Version]
- Snavely, N.; Seitz, S.M.; Szeliski, R. Modeling the world from internet photo collections. Int. J. Comput. Vis. 2008, 80, 189–210. [Google Scholar] [CrossRef] [Green Version]
- Agarwal, S.; Snavely, N.; Seitz, S.M.; Szeliski, R. Bundle Adjustment in the Large. In Proceedings of the ECCV 2010, Crete, Greece, 5–11 September 2010; pp. 29–42. [Google Scholar]
- Agarwal, S.; Furukawa, Y.; Snavely, N.; Simon, I.; Curless, B.; Seitz, S.M.; Szeliski, R. Building rome in a day. Commun. ACM 2011, 54, 105–112. [Google Scholar] [CrossRef]
- Wu, C. Towards linear-time incremental structure from motion. In Proceedings of the International Conference on 3D Vision (3DV), Seattle, DC, USA, 29 June–1 July 2013; pp. 127–134. [Google Scholar]
- Wu, C.; Agarwal, S.; Curless, B.; Seitz, S.M. Multicore bundle adjustment. In Proceedings of the CVPR 2011, Colorado Springs, CO, USA, 20–25 June 2011; pp. 3057–3064. [Google Scholar]
- Ni, K.; Steedly, D.; Dellaert, F. Out-of-Core bundle adjustment for large-scale 3D reconstruction. In Proceedings of the ICCV 2007, Rio de Janeiro, Brazil, 14–21 October 2007; pp. 1–8. [Google Scholar]
- Snavely, N.; Seitz, S.M.; Szeliski, R. Skeletal graphs for efficient structure from motion. In Proceedings of the CVPR 2008, Anchorage, AK, USA, 23–28 June 2008; pp. 1–8. [Google Scholar]
- Farenzena, M.; Fusiello, A.; Gherardi, R. Structure-and-motion pipeline on a hierarchical cluster tree. In Proceedings of the ICCV 2009, Kyoto, Japan, 27 September–4 October 2009; pp. 1489–1496. [Google Scholar]
- Gherardi, R.; Farenzena, M.; Fusiello, A. Improving the efficiency of hierarchical structure-and-motion. In Proceedings of the CVPR 2010, San Francisco, CA, USA, 13–18 June 2010; pp. 1594–1600. [Google Scholar]
- Toldo, R.; Gherardi, R.; Farenzena, M.; Fusiello, A. Hierarchical structure-and-motion recovery from uncalibrated images. Comput. Vis. Image Underst. 2015, 140, 127–143. [Google Scholar] [CrossRef] [Green Version]
- Shah, R.; Deshpande, A.; Narayanan, P.J. Multistage SFM: Revisiting incremental structure from motion. In Proceedings of the International Conference on 3D Vision 2014, Tokyo, Japan, 8–11 December 2014; pp. 417–424. [Google Scholar]
- Schönberger, J.L.; Frahm, J.-M. Structure-from-motion revisited. In Proceedings of the CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016; pp. 4104–4113. [Google Scholar]
- Bhowmick, B.; Patra, S.; Chatterjee, A.; Madhav Govindu, V.; Banerjee, S. Divide and conquer: A hierarchical approach to large-scale structure-from-motion. Comput. Vis. Image Underst. 2017, 157, 190–205. [Google Scholar] [CrossRef]
- Zhu, S.; Shen, T.; Zhou, L.; Zhang, R.; Wang, J.; Fang, T.; Quan, L. Parallel structure from motion from local increment to global averaging. arXiv 2017, arXiv:1702.08601. [Google Scholar]
- Xu, B.; Zhang, L.; Liu, Y.; Ai, H.; Wang, B.; Sun, Y.; Fan, Z. Robust hierarchical structure from motion for large-scale unstructured image sets. ISPRS J. Photogramm. Remote Sens. 2021, 181, 367–384. [Google Scholar] [CrossRef]
- Jiang, S.; Li, Q.; Jiang, W.; Chen, W. Parallel Structure From Motion for UAV Images via Weighted Connected Dominating Set. IEEE Trans. Geosci. Remote Sens. 2022, 60, 5413013. [Google Scholar] [CrossRef]
- Frahm, J.-M.; Fite-Georgel, P.; Gallup, D.; Johnson, T.; Raguram, R.; Wu, C.; Jen, Y.-H.; Dunn, E.; Clipp, B.; Lazebnik, S.; et al. Building Rome on a cloudless day. In Proceedings of the ECCV 2010, Crete, Greece, 5–11 September 2010; pp. 368–381. [Google Scholar]
- Sun, Y.; Sun, H.; Yan, L.; Fan, S.; Chen, R. RBA: Reduced bundle adjustment for oblique aerial photogrammetry. ISPRS J. Photogramm. Remote Sens. 2016, 121, 128–142. [Google Scholar] [CrossRef]
- Cui, H.; Shen, S.; Hu, Z. Tracks selection for robust, efficient and scalable large-scale structure from motion. Pattern Recognit. 2017, 72, 341–354. [Google Scholar] [CrossRef]
- Zhou, G.; Bao, X.; Ye, S.; Wang, H.; Yan, H. Selection of optimal building facade texture images from uav-based multiple oblique image flows. IEEE Trans. Geosci. Remote Sens. 2021, 59, 1534–1552. [Google Scholar] [CrossRef]
- Liang, Y.; Fan, X.; Yang, Y.; Li, D.; Cui, T. Oblique view selection for efficient and accurate building reconstruction in rural areas using large-scale UAV images. Drones 2022, 6, 175. [Google Scholar] [CrossRef]
- Gehrke, S.; Müller, M.; Kukla, M.; Beshah, B. HIERARCHICAL AERIAL TRIANGULATION OF OBLIQUE IMAGE DATA. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2022, XLIII-B2-2022, 45–50. [Google Scholar] [CrossRef]
- Liang, Y.; Li, D.; Feng, C.; Mao, J.; Wang, Q.; Cui, T. Efficient match pair selection for matching large-scale oblique UAV images using spatial priors. Int. J. Remote Sens. 2021, 42, 8878–8905. [Google Scholar] [CrossRef]
- Moulon, P.; Monasse, P.; Perrot, R.; Marlet, R. OpenMVG: Open Multiple View Geometry. In Proceedings of the International Workshop on Reproducible Research in Pattern Recognition 2016, Cancún, Mexico, 4 December 2016; pp. 60–74. [Google Scholar]
- Ceres Solver Official Web Site. Available online: https://http://ceres-solver.org/ (accessed on 21 January 2023).
- Tachikawa, T.; Hato, M.; Kaku, M.; Iwasaki, A. Characteristics of ASTER GDEM version 2. In Proceedings of the 2011 IEEE International Geoscience and Remote Sensing Symposium, Vancouver, Canada, 24–29 July 2011; pp. 3657–3660. [Google Scholar]
- Metashape Official Web Site. Available online: https://www.agisoft.com/ (accessed on 21 January 2023).
Item | Specification |
---|---|
Number of cameras in the oblique imaging system | 5 |
Camera model | SONY ILCE-5100 |
Image resolution (pixel) | 6000 by 4000 |
Focal length of nadir and oblique cameras (mm) | 20, 35 |
Forward and side overlap ratio | 80%, 70% |
Flight height (m) | 460 |
Ground sample distance (GSD) (cm) | 7 |
Number of images | 9775 |
POS observations | Latitude, longitude, altitude, omega, phi, and kappa |
Observation direction of camera (1, 2, 3, 4, and 5) | Backward, forward, right, left, and down |
Area covered (km2) | 9.1 |
Camera | 1 | 2 | 3 | 4 | 5 |
---|---|---|---|---|---|
1 | 1918/1922 | 6834/7195 | - | - | 1764/1865 |
2 | - | 1921/1922 | - | - | 1744/1849 |
3 | - | - | 1921/1922 | 7229/7452 | 1521/1710 |
4 | - | - | - | 1921/1922 | 1510/1733 |
5 | - | - | - | - | 3767/3805 |
Min | Max | Mean | Median | STD | |
---|---|---|---|---|---|
Track length | 3 | 130 | 6.4 | 4 | 7.5 |
Image observations | 0 | 1572 | 747.9 | 763 | 271.7 |
MNO | Number of Selected Tracks | Mean | Median | STD |
---|---|---|---|---|
30 | 61,329 | 17.2 | 9 | 18.9 |
40 | 82,708 | 16.0 | 9 | 17.7 |
50 | 103,918 | 15.0 | 8 | 16.8 |
60 | 124,469 | 14.3 | 8 | 16.0 |
70 | 144,095 | 13.7 | 8 | 15.4 |
80 | 165,252 | 13.2 | 8 | 14.8 |
90 | 186,390 | 12.7 | 8 | 14.3 |
100 | 207,924 | 12.3 | 7 | 13.9 |
MNO | Max | Mean | Median | STD |
---|---|---|---|---|
30 | 383 | 107.7 | 95 | 46.9 |
40 | 465 | 135.3 | 120 | 56.7 |
50 | 561 | 159.5 | 143 | 64.8 |
60 | 593 | 181.7 | 166 | 71.3 |
70 | 676 | 202.0 | 184 | 76.8 |
80 | 700 | 222.6 | 205 | 81.4 |
90 | 741 | 242.8 | 225 | 85.2 |
100 | 764 | 261.8 | 246 | 88.4 |
Submap | Registered Images | Reconstructed Tracks | Track Length Mean | Image Observations Mean | RMSE (Pixels) |
---|---|---|---|---|---|
1 | 1947/1955 | 36,405 | 7 | 132.3 | 0.40 |
2 | 1955/1955 | 29,864 | 8 | 123.6 | 0.52 |
3 | 1955/1955 | 33,536 | 8 | 151.3 | 0.45 |
4 | 1955/1955 | 34,875 | 8 | 149.2 | 0.44 |
5 | 1955/1955 | 33,103 | 12 | 210.8 | 0.47 |
Submaps | Number of Common Tracks | Min | Max | Mean | STD | |
---|---|---|---|---|---|---|
1-5 | 18,358 | X | −228.34 | 223.76 | −1.84 | 6.33 |
Y | −136.57 | 165.65 | 3.76 | 4.37 | ||
Z | −266.05 | 400.56 | 3.12 | 7.24 | ||
2–5 | 18,049 | X | −206.69 | 361.10 | −1.01 | 6.54 |
Y | −113.63 | 206.02 | 5.30 | 3.62 | ||
Z | −221.60 | 207.53 | 3.08 | 5.27 | ||
3–5 | 14,405 | X | −278.84 | 283.24 | −1.58 | 7.66 |
Y | −248.83 | 258.23 | 1.51 | 7.35 | ||
Z | −465.37 | 370.04 | 3.06 | 12.15 | ||
4–5 | 14,461 | X | −269.90 | 212.66 | −0.73 | 7.24 |
Y | −205.99 | 283.89 | 5.26 | 7.41 | ||
Z | −311.22 | 399.73 | 0.31 | 11.73 |
Merging | Reconstructed Tracks | Registered Images | RMSE (Pixels) |
---|---|---|---|
5-1 | 51,055 | 3902 | 0.43 |
5-1-2 | 62,779 | 5857 | 0.47 |
5-1-2-3 | 81,831 | 7812 | 0.46 |
5-1-2-3-4 | 102,148 | 9767 | 0.45 |
Software Package | Match Pair Selection | Image Matching | SfM Strategy | Version | Source |
---|---|---|---|---|---|
Metashape | Position and visual similarity | Highest accuracy, maximum features: 40,000, maximum tie points: 4000 | Hierarchical | 1.8.4 build 14,856 | https://www.agisoft.com/ (accessed on 21 January 2023) |
COLMAP | Position | Maximum resolution: 2000 px | Hierarchical | 3.7 | https://github.com/colmap/colmap (accessed on 21 January 2023) |
Software Package | Registered Images | Reconstructed Tracks | RMSE (Pixel) | Time Efficiency |
---|---|---|---|---|
Metashape | 9775 | 8,970,391 | 1.00 | 3 h 10 min |
COLMAP | 9775 | 4,612,725 | 1.72 | 20 h 8 min |
Proposed | 9767 | 102,148 | 0.45 | 54 min |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Liang, Y.; Yang, Y.; Fan, X.; Cui, T. Efficient and Accurate Hierarchical SfM Based on Adaptive Track Selection for Large-Scale Oblique Images. Remote Sens. 2023, 15, 1374. https://doi.org/10.3390/rs15051374
Liang Y, Yang Y, Fan X, Cui T. Efficient and Accurate Hierarchical SfM Based on Adaptive Track Selection for Large-Scale Oblique Images. Remote Sensing. 2023; 15(5):1374. https://doi.org/10.3390/rs15051374
Chicago/Turabian StyleLiang, Yubin, Yang Yang, Xiaochang Fan, and Tiejun Cui. 2023. "Efficient and Accurate Hierarchical SfM Based on Adaptive Track Selection for Large-Scale Oblique Images" Remote Sensing 15, no. 5: 1374. https://doi.org/10.3390/rs15051374