High-Magnification Object Tracking with Ultra-Fast View Adjustment and Continuous Autofocus Based on Dynamic-Range Focal Sweep
<p>Overview of the proposed high-magnification object tracking system: The combination of (1) C-AF based on dynamic-range focal sweep and (2) 500 fps frame-by-frame object tracking achieves high-speed and precise high-magnification tracking of small objects moving in a wide scene.</p> "> Figure 2
<p>From time <math display="inline"><semantics> <msub> <mi>t</mi> <mn>0</mn> </msub> </semantics></math> to <math display="inline"><semantics> <msub> <mi>t</mi> <mi>k</mi> </msub> </semantics></math>, the focus of the camera system changes with the variation of the diopter of the liquid lens. During each period, the high-speed camera captures multiple images with different focuses. Using the focus measure algorithm, the image with the best focus can be extracted. Simultaneously, through the correlation between the focal length and the distance of the subject plane, the object’s depth can be calculated.</p> "> Figure 3
<p>Schematic of depth measurement with focal sweep.</p> "> Figure 4
<p>Diagram of the adjustment of the dynamic-range focal sweep: The first forward focal sweep measures the object’s depth. Then, at the backward focal sweep, the range of the focal sweep can be adjusted. At the second forward focal sweep, the system can finish the range adjustment and measure the object’s depth again.</p> "> Figure 5
<p>Diagram of the adjustment of the view direction: (<b>a</b>) Two Galvano mirrors are used to adjust the horizontal and vertical viewpoints. (<b>b</b>) Schematic representation of the horizontal viewpoint adjustment.</p> "> Figure 6
<p>The Pipeline of the object tracking method in our system: The pipeline consists of three main threads: (1) Thread 1: 500 fps Focus Measure, (2) Thread 2: Object Main-Color Updating, and (3) Thread 3: 500 fps Frame-by-Frame Object Tracking. The focus measure algorithm is implemented to extract 50 fps well-focused images and to determine the object’s depth, adjusting the focal sweep range. The object detection algorithm operates at 25 fps, providing color updating at the same rate. Meanwhile, the object tracking algorithm achieves 500 fps frame-by-frame object tracking.</p> "> Figure 7
<p>System configuration of proposed high-magnification autofocus tracking system.</p> "> Figure 8
<p>The environment of Experiment 1.</p> "> Figure 9
<p>The focal sweep adjustment and the depth measurement results using the proposed C-AF with butterfly model’s movements.</p> "> Figure 10
<p>The focal sweep adjustment and the depth measurement results using the proposed C-AF with qr code’s movements.</p> "> Figure 11
<p>The focal sweep adjustment and the depth measurement results using the proposed C-AF with screw’s movements.</p> "> Figure 12
<p>Variation of size with butterfly model’s movements.</p> "> Figure 13
<p>Variation of size with QR code’s movements.</p> "> Figure 14
<p>Variation of size with screw’s movements.</p> "> Figure 15
<p>Output images’ detection results with object movements at 3 m/s.</p> "> Figure 16
<p>Results of proposed HFR object tracking method for multiple objects at different distances: These figures were captured at 500 fps sequentially during one process of the focal sweep. The first, the second, the third columns shows the original images, color-filtered maps, and the results.</p> "> Figure 17
<p>Environment of Experiment 4.</p> "> Figure 18
<p>The focal sweep adjustment and the depth measurement results using the proposed C-AF with the object’s movements.</p> "> Figure 19
<p>Viewpoint’s variation during the object movement.</p> "> Figure 20
<p>Some results of HFR high-magnification object tracking.</p> ">
Abstract
:1. Introduction
- C-AF Method Based on Dynamic-Range Focal Sweep: We introduce a novel C-AF method that utilizes a 500 fps high-speed camera in conjunction with a liquid lens for dynamic-range focal sweep at 50 Hz. The optimal focused image is extracted using the focus measure method based on the Canny sharpness detector [28]. The adjustment of the focal sweep range is achieved by measuring the depth of the object through the Depth-from-Focus (DFF) technique [29,30]. This results in a narrowed focal sweep range around the object’s true focus, leading to stable 50 fps well-focused images. Moreover, due to the effective shortening of the focal sweep range, the blur in the remaining images, which are less focused, is minimal. Thus, all 500 fps images can be utilized for object tracking.
- 500 fps Frame-by-Frame High-Magnification Tracking Pipeline: We propose a 500 fps frame-by-frame tracking method specifically designed for high-speed high-magnification tracking. The proposed HFR tracking pipeline utilizes the main color of the object, continuously updated by applying the K-means clustering method [31] on the ROI obtained by YOLOv8 [32]. Leveraging the ultra-fast pan-tilt mechanism based on the Galvano mirror, our system achieves 500 Hz view direction adjustment.
2. Related Works
2.1. Development of Autofocus
2.2. Development of Object Tracking Algorithms
2.3. Development of High-Speed Active Vision Systems
3. High-Speed Continuous Autofocus High-Magnification Object Tracking
3.1. Overview of the Proposed System
3.2. Continuous Autofocus Based on Dynamic-Range Focal Sweep
3.2.1. Optical Geometry of Focal Sweep
3.2.2. Dynamic-Range Focal Sweep
3.3. Optical Geometry of Ultra-Fast Pan-Tilt Adjustment Based on Galvano Mirror
3.4. High-Frame-Rate Frame-by-Frame Object Tracking
3.4.1. Pipeline of the Proposed High-Magnification Autofocus Tracking
3.4.2. Thread 1: 500 FPS Focus Measure
Algorithm 1: The Proposed C-AF with 500 FPS Focus Measure. |
3.4.3. Thread 2: Object Main-Color Updating
Algorithm 2: Main Color Updating and Error Distance Calculation. |
Input: Well-focused Image Parameter: Color-filtering Thresholds and Output: Main color Error Distance d YOLO-based ROI Step 1: YOLO Detection Run YOLO on image to obtain Calculate central position based on YOLO Step 2: Main Color Obtaining Calculate the core ROI Obtain background pixels’ array P Run k-means to obtain object’s main colors Q and background’s main colors R Determine the object’s main-color Step 3: Color Filtering Convert and pixels in to HSV format Calculate the central position based on color filtering Step 4: Error Distance Calculation Calculate Error distance d |
3.4.4. Thread 3: 500 fps Frame-by-Frame Object Tracking
Algorithm 3: HFR Object Tracking based on Color Filtering. |
Input: 500 fps Image YOLO-based ROI Object’s Main-Color Error Distance d Parameter: Additional Expansion Columns k Output: Pan and Tilt Angles and Step 1: Object Central Position Calculation with Color Filtering Calculate by expanding with k Convert pixels in to HSV format Calculate the central position Obtain the object center by Equation (21) Step 2: Pan-tilt Control Calculate the pan and tilt angles and |
4. System Configuration
5. Experiments
5.1. Experiment 1: Continuous Autofocus Based on Dynamic-Range Focal Sweep
- Over-Narrow DoF: When the focused subject plane was too close, the DoF became excessively narrow, possibly causing only part of Object 2 to be in focus. To ensure that the entire object is focused in the output, we believe that our previous work on a real-time DoF extension algorithm can be utilized effectively [48].
- Insufficiently Sensitive Focus Measure Algorithm: The strength of the edge features of the QR code is too homogeneous, making it difficult to distinguish the best-focused image precisely, unlike Object 1 and Object 3, where edges with different strengths are contained. To address this issue, we plan to use a more sensitive focus measure algorithm in future iterations.
5.2. Experiment 2: Effectiveness and Robustness Analysis of C-AF Based on Dynamic-Range Focal Sweep
- Lighting Condition (i): Fluorescent Lamps + LED Light Illuminator;
- Lighting Condition (ii): Indoor natural lighting with fluorescent lamps;
- Lighting Condition (iii): Indoor natural lighting.
5.3. Experiment 3: Feasibility Demonstration of the Proposed HFR Object Tracking Method
5.4. Experiment 4: HFR Frame-by-Frame High-Magnification Autofocus Tracking
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Gad, A.; Basmaji, T.; Yaghi, M.; Alheeh, H.; Alkhedher, M.; Ghazal, M. Multiple Object Tracking in Robotic Applications: Trends and Challenges. Appl. Sci. 2021, 12, 9408. [Google Scholar] [CrossRef]
- Shahzad, A.; Gao, X.; Yasin, A.; Javed, K.; Anwar, M.S. A Vision-Based Path Planning and Object Tracking Framework for 6-DOF Robotic Manipulator. IEEE Access 2020, 8, 203158–203167. [Google Scholar] [CrossRef]
- Sukkar, M.; Shukla, M.; Kumar, D.; Gerogiannis, V.C.; Kanavos, A.; Acharya, B. Enhancing Pedestrian Tracking in Autonomous Vehicles by Using Advanced Deep Learning Techniques. Information 2024, 15, 104. [Google Scholar] [CrossRef]
- Premachandra, C.; Ueda, S.; Suzuki, Y. Detection and Tracking of Moving Objects at Road Intersections Using a 360-Degree Camera for Driver Assistance and Automated Driving. IEEE Access 2020, 8, 135652–135660. [Google Scholar] [CrossRef]
- Li, C.; Wang, Y.; Liu, X. A Multi-Pedestrian Tracking Algorithm for Dense Scenes Based on an Attention Mechanism and Dual Data Association. Appl. Sci. 2021, 12, 9597. [Google Scholar] [CrossRef]
- Sriram, V.K.; Havaldar, H.R. Human detection and tracking in video surveillance system. In Proceedings of the 2016 IEEE International Conference on Computational Intelligence and Computing Research (ICCIC), Chennai, India, 15–17 December 2016; pp. 1–3. [Google Scholar]
- Mirzaei, B.; Raoof, A.; Derakhshani, R. Small Object Detection and Tracking: A Comprehensive Review. Sensors 2023, 23, 6887. [Google Scholar] [CrossRef]
- Behrendt, K.; Novak, L.; Botros, R. A deep learning approach to traffic lights: Detection, tracking, and classification. In Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, 29 May–3 June 2017; pp. 1370–1377. [Google Scholar]
- Canepa, A.; Ragusa, E.; Zunino, R.; Gastaldo, P. T-RexNet—A Hardware-Aware Neural Network for Real-Time Detection of Small Moving Objects. Sensors 2020, 21, 1252. [Google Scholar] [CrossRef] [PubMed]
- Kashiyama, T.; Sobue, H.; Sekimoto, Y. Sky monitoring system for flying object detection using 4K resolution camera. Sensors 2020, 20, 7071. [Google Scholar] [CrossRef]
- Ruzicka, V.; Franchetti, F. Fast and accurate object detection in high resolution 4K and 8K video using GPUs. In Proceedings of the 2018 IEEE High Performance extreme Computing Conference (HPEC), Waltham, MA, USA, 25–27 September 2018; pp. 1–7. [Google Scholar]
- Chen, S.; Xu, T.; Li, D.; Zhang, J.; Jiang, S. Moving Object Detection Using Scanning Camera on a High-Precision Intelligent Holder. Sensors 2016, 16, 1758. [Google Scholar] [CrossRef] [PubMed]
- Liu, Y.; Sun, P.; Namiki, A. Target tracking of moving and rotating object by high-speed monocular active vision. IEEE Sens. J. 2020, 20, 6727–6744. [Google Scholar] [CrossRef]
- Tilmon, B.; Jain, E.; Ferrari, S.; Koppal, S. FoveaCam: A MEMS Mirror-Enabled Foveating Camera. In Proceedings of the 2020 IEEE International Conference on Computational Photography (ICCP), St. Louis, MO, USA, 24–26 April 2020; pp. 1–11. [Google Scholar]
- Jiang, M.; Shimasaki, K.; Hu, S.; Senoo, T.; Ishii, I. A 500-Fps Pan-Tilt Tracking System with Deep-Learning-Based Object Detection. IEEE Robot. Autom. Lett. 2021, 6, 691–698. [Google Scholar] [CrossRef]
- Shih, L. Autofocus survey: A comparison of algorithms. Digit. Photogr. III SPIE 2007, 6502, 90–100. [Google Scholar]
- Zhang, Y.; Liu, L.; Gong, W.; Yu, H.; Wang, W.; Zhao, C.; Wang, P.; Ueda, T. Autofocus System and Evaluation Methodologies: A Literature Review. Sens. Mater. 2018, 30, 1165–1174. [Google Scholar]
- Xu, X.; Wang, Y.; Tang, J.; Zhang, X.; Liu, X. Robust Automatic Focus Algorithm for Low Contrast Images Using a New Contrast Measure. Sensors 2011, 11, 8281–8294. [Google Scholar] [CrossRef] [PubMed]
- Yang, C.; Chen, M.; Zhou, F.; Li, W.; Peng, Z. Accurate and Rapid Auto-Focus Methods Based on Image Quality Assessment for Telescope Observation. Appl. Sci. 2020, 10, 658. [Google Scholar] [CrossRef]
- Chan, C.C.; Huang, S.K.; Chen, H.H. Enhancement of phase detection for autofocus. In Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China, 17–20 September 2017; pp. 41–45. [Google Scholar]
- Anikina, A.; Rogov, Y.O.; Dylov, V.D. Detect to Focus: Latent-Space Autofocusing System with Decentralized Hierarchical Multi-Agent Reinforcement Learning. IEEE Access 2023, 11, 85214–85223. [Google Scholar] [CrossRef]
- Hua, Z.; Zhang, X.; Tu, D. Autofocus methods based on laser illumination. Opt. Express 2023, 31, 29465–29479. [Google Scholar] [CrossRef] [PubMed]
- Zhang, X.; Fan, F.; Gheisari, M.; Srivastava, G. A Novel Auto-Focus Method for Image Processing Using Laser Triangulation. IEEE Access 2019, 7, 64837–64843. [Google Scholar] [CrossRef]
- Hasinoff, W.S.; Kutulakos, N.K.; Durand, F.; Freeman, T.W. Time-constrained Photography. In Proceedings of the IEEE 12th International Conference on Computer Vision, Kyoto, Japan, 29 September–2 October 2009; pp. 333–340. [Google Scholar]
- Vaquero, D.; Gelfand, N.; Tico, M.; Pulli, K.; Turk, M. Generalized Autofocus. In Proceedings of the 2011 IEEE Workshop on Applications of Computer Vision (WACV), Kona, HI, USA, 5–7 January 2011; pp. 511–518. [Google Scholar]
- Yamato, K.; Chiba, H.; Oku, H. High Speed Three Dimensional Tracking of Swimming Cell by Synchronous Modulation Between TeCE Camera and TAG Lens. IEEE Robot. Autom. Lett. 2020, 5, 1907–1914. [Google Scholar] [CrossRef]
- Zhang, T.; Hu, S.; Shimasaki, K.; Ishii, I.; Namiki, A. Dual-camera High Magnification Surveillance System with Non-delay Gaze Control and Always-in-focus Function in Indoor Scenes. In Proceedings of the 2022 IEEE/RSJ International Conference on Intelligent Robotics and Systems (IROS), Kyoto, Japan, 23–27 October 2022; pp. 6637–6642. [Google Scholar]
- Canny, J. A Computational Approach to Edge Detection. IEEE Trans. Pattern Anal. Mach. Intell. 1986, 8, 679–698. [Google Scholar] [CrossRef]
- Grossmann, P. Depth From Focus. Pattern Recognit. Lett. 1987, 5, 63–69. [Google Scholar] [CrossRef]
- Martel, J.; Muller, L.; Carey, S.; Muller, J. Real-time Depth From Focus on a Programmable Focal Plane Processor. IEEE Trans. Circuits Syst. I Regul. Pap. 2018, 65, 925–934. [Google Scholar] [CrossRef]
- Ahmed, M.; Seraj, R.; Islam, S.M.S. The k-means Algorithm: A Comprehensive Survey and Performance Evaluation. Electronics 2020, 9, 1295. [Google Scholar] [CrossRef]
- Jocher, G.; Chaurasia, A.; Qiu, J. Ultralytics YOLOv8 (ver. 8.0.0). Available online: https://github.com/ultralytics/ultralytics (accessed on 15 September 2023).
- Abele, R.; Fronte, D.; Liardet, P.; Boi, J.; Damoiseaux, J.; Merad, D. Autofocus in infrared microscopy. In Proceedings of the 2018 IEEE 23rd International Conference on Emerging Technologies and Factory Automation (ETFA), Turin, Italy, 4–7 September 2018; pp. 631–637. [Google Scholar]
- Liang, H.; Lu, K.; Liu, X.; Xue, J. The Auto-focus Method for Scanning Acoustic Microscopy by Sparse Representation. Sens. Imaging 2019, 20, 33. [Google Scholar] [CrossRef]
- Hsu, W.; Yaung, D.; Hung, F.; Chou, K. Phase Detection Autofocus Techniques. U.S. Patent 9804357B2, 31 October 2017. [Google Scholar]
- Ho, C.; Chan, C.; Chen, H. AF-Net: A Convolutional Neural Network Approach to Phase Detection Autofocus. IEEE Trans. Image Process. 2020, 29, 6386–6395. [Google Scholar] [CrossRef]
- Zhang, X.; Liu, Z.; Jiang, M.; Chang, M. Fast and Accurate Auto-focusing Algorithm Based on the Combination of Depth from Focus and Improved Depth from Defocus. Opt. Express 2014, 22, 31237–31247. [Google Scholar] [CrossRef]
- Yasugi, S.; Nguyen, K.; Ezawa, K.; Kawamura, T. Depth from Defocus applied to Auto Focus. In Proceedings of the 2014 IEEE 3rd Global Conference on Consumer Electronics (GCCE), Tokyo, Japan, 7–10 October 2014; pp. 171–173. [Google Scholar]
- Tang, H.; Cohen, S.; Price, B.; Schiller, S.; Kutulakos, N.K. Depth from Defocus in the Wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 4773–4781. [Google Scholar]
- Herrmann, C.; Bown, R.; Wadhwa, N.; Garg, R.; He, Q.; Barron, J.; Zabih, R. Learning to Autofocus. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–19 June 2020; pp. 2230–2239. [Google Scholar]
- Wang, C.; Huang, Q.; Cheng, M.; Ma, Z.; Brady, J.D. Deep Learning for Camera Autofocus. IEEE Trans. Comput. Imaging 2021, 7, 258–271. [Google Scholar] [CrossRef]
- Kehtarnavaz, N.; Oh, J.H. Development and Real-Time Implementation of a Rule-based Auto-Focus Algorithm. Real-Time Imaging 2003, 9, 197–203. [Google Scholar] [CrossRef]
- He, J.; Zhou, R.; Hong, Z. Modified Fast Climbing Search Auto-focus Algorithm with Adaptive Step Size Searching Technique for Digital Camera. IEEE Trans. Consum. Electron. 2003, 49, 257–262. [Google Scholar] [CrossRef]
- Park, C.; Cha, S.; Lee, Y.; Kwon, O.; Park, D.; Kwon, K.; Lee, J. A highly accurate piezoelectric actuator driver IC for auto-focus in camera module of mobile phone. In Proceedings of the 2010 IEEE International Symposium on Circuits and Systems, Paris, France, 30 May–2 June 2010; pp. 1284–1287. [Google Scholar]
- Hsieh, L.C.; Liu, S.C. Design of a Voice Coil Motor Actuator With L-Shape Coils for Optical Zooming Smartphone Cameras. IEEE Access 2020, 8, 20884–20891. [Google Scholar] [CrossRef]
- Wang, Z.; Lei, M.; Yao, B.; Cai, Y.; Liang, Y.; Yang, Y.; Yang, X.; Li, H.; Xiong, D. Compact Multi-band Fluorescent Microscope with an Electrically Tunable lens for Autofocusing. Biomed. Opt. Express 2015, 6, 4353–4364. [Google Scholar] [CrossRef] [PubMed]
- Oku, H.; Ishikawa, M. High-speed liquid lens for computer vision. In Proceedings of the 2010 IEEE International Conference on Robotics and Automation, Anchorage, AK, USA, 3–7 May 2010; pp. 2643–2648. [Google Scholar]
- Zhang, T.; Li, Z.; Wang, Q.; Shimasaki, K.; Ishii, I.; Namiki, A. DoF-Extended Zoomed-In Monitoring System With High-Frame-Rate Focus Stacking and High-Speed Pan-Tilt Adjustment. IEEE Sens. J. 2024, 24, 6765–6776. [Google Scholar] [CrossRef]
- Xiong, T.; Debrunnerr, C. Stochastic Car Tracking with Line- and Color-Based Features. IEEE Trans. Intell. Transp. Syst. 2005, 5, 324–328. [Google Scholar] [CrossRef]
- Doi, M.; Matsumoto, T.; Kimachi, A.; Nishi, S.; Ikoma, N. Robust color object tracking method against illumination color Change. In Proceedings of the 2014 Joint 7th International Conference on Soft Computing and Intelligent Systems (SCIS) and 15th International Symposium on Advanced Intelligent Systems (ISIS), Kitakyushu, Japan, 3–6 December 2014; pp. 716–722. [Google Scholar]
- Dalal, N.; Triggs, B. Histograms of oriented gradients for human detection. In Proceedings of the IEEE Computer Society Conference on Computer Vision & Pattern Recognition, San Diego, CA, USA, 20–25 June 2005; pp. 886–893. [Google Scholar]
- Viola, P.; Jones, M.J. Robust real-time face detection. Int. J. Comput. Vis. 2004, 57, 137–154. [Google Scholar] [CrossRef]
- Henriques, J.; Caseiro, R. High-speed tracking with kernelized correlation filters. IEEE Trans. Pattern Anal. Mach. Intell. 2015, 37, 583–596. [Google Scholar] [CrossRef] [PubMed]
- Njuguna, J.; Alabay, A.; Celebi, A.; Celebi, T.A.; Gullu, K.M. Efficient Hardware Implementation of Real-Time Object Tracking. In Proceedings of the 2022 30th Signal Processing and Communications Applications Conference, Safranbolu, Turkey, 15–18 May 2022; pp. 1–4. [Google Scholar]
- Du, K.; Ju, Y.; Jin, Y.; Li, G.; Qian, S.; Li, Y. MeanShift tracking algorithm with adaptive block color histogram. In Proceedings of the 2012 2nd International Conference on Consumer Electronics, Communications and Networks (CECNet), Yichang, China, 21–23 April 2012; pp. 2692–2695. [Google Scholar]
- Xiu, C.; Wang, R. Hybrid tracking based on camshift and template matching. In Proceedings of the 2017 29th Chinese Control And Decision Conference (CCDC), Chongqing, China, 28–30 May 2017; pp. 5753–5756. [Google Scholar]
- Meng, W.; Yuan, Y. SGN-YOLO: Detecting Wood Defects with Improved YOLOv5 Based on Semi-Global Network. Sensors 2023, 23, 8705. [Google Scholar] [CrossRef] [PubMed]
- Berwo, M.A.; Khan, A.; Fang, Y.; Fahim, H.; Javaid, S.; Mahmood, J.; Abideen, Z.U.; MS, S. Deep Learning Techniques for Vehicle Detection and Classification from Images/Videos: A Survey. Sensors 2023, 23, 4832. [Google Scholar] [CrossRef]
- Pistellato, M.; Bergamasco, F.; Bigaglia, G.; Gasparetto, A.; Albarelli, A.; Boschetti, M.; Passerone, R. Quantization-Aware NN Layers with High-throughput FPGA Implementation for Edge AI. Sensors 2023, 23, 4667. [Google Scholar] [CrossRef]
- Gu, Q.; Ishii, I. Review of some advances and applications in real-time high-speed vision: Our views and experiences. Int. J. Autom. Comput. 2016, 13, 305–318. [Google Scholar] [CrossRef]
- Park, M.; Kim, H. A Real-Time Edge-Detection CMOS Image Sensor for Machine Vision Applications. IEEE Sens. J. 2021, 23, 9254–9261. [Google Scholar] [CrossRef]
- Lyu, C.; Li, P.; Wang, D.; Yang, S.; Lai, Y.; Sui, C. High-Speed Optical 3D Measurement Sensor for Industrial Application. IEEE Sens. J. 2021, 21, 11253–11261. [Google Scholar] [CrossRef]
- Noda, A.; Hirano, M.; Yamakawa, Y.; Ishikawa, M. A networked high-speed vision system for vehicle tracking. In Proceedings of the 2014 IEEE Sensors Applications Symposium (SAS), Queenstown, New Zealand, 18–20 February 2014; pp. 343–348. [Google Scholar]
- Lukezic, A.; Vojir, T.; Zajc, L.; Matas, J.; Kristan, M. Discriminative correlation filter with channel and spatial reliability. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 6309–6318. [Google Scholar]
- Jiang, S.; Li, S.; Zhu, C. Kernelized Correlation Filter Tracking with Scale Adaptive Filter and Feature Integration. In Proceedings of the 2018 IEEE 4th International Conference on Computer and Communications (ICCC), Chengdu, China, 7–10 December 2018; pp. 201–206. [Google Scholar]
- Cai, Y.; Medioni, G. Persistent people tracking and face capture using a PTZ camera. Mach. Vis. Appl. 2013, 27, 397–413. [Google Scholar] [CrossRef]
- Liu, R.; Jin, S.; Wu, X. Real time auto-focus algorithm for eye gaze tracking system. In Proceedings of the 2007 International Symposium on Intelligent Signal Processing and Communication Systems, Xiamen, China, 28 November–1 December 2007; pp. 742–745. [Google Scholar]
- Okumura, K.; Oku, H.; Ishikawa, M. High-speed gaze controller for millisecond-order pan/tilt camera. In Proceedings of the 2011 IEEE International Conference on Robotics and Automation, Shanghai, China, 9–13 May 2011; pp. 6186–6191. [Google Scholar]
- Kobayashi-Kirschvink, K.; Oku, H. Design Principles of a High-Speed Omni-Scannable Gaze Controller. IEEE Robot. Autom. Lett. 2016, 1, 836–843. [Google Scholar] [CrossRef]
- Aoyama, T.; Kaneishi, M.; Takaki, T.; Ishii, I. View expansion system for microscope photography based on viewpoint movement using Galvano mirror. In Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada, 24–28 September 2017; pp. 1140–1145. [Google Scholar]
- Sakamoto, K.; Aoyama, T.; Takeuchi, M.; Hasegawa, Y. Intuitive Cell Manipulation Microscope System with Haptic Device for Intracytoplasmic Sperm Injection Simplification. Sensors 2024, 24, 711. [Google Scholar] [CrossRef] [PubMed]
- Hu, S.; Matsumoto, Y.; Takaki, T.; Ishii, I. Monocular Stereo Measurement Using High-Speed Catadioptric Tracking. Sensors 2017, 17, 1839. [Google Scholar] [CrossRef] [PubMed]
- Hu, S.; Shimasaki, K.; Jiang, M.; Senoo, T.; Ishii, I. A Simultaneous Multi-Object Zooming System Using an Ultrafast Pan-Tilt Camera. IEEE Sens. J. 2021, 21, 9436–9448. [Google Scholar] [CrossRef]
- Hu, S.; Shimasaki, K.; Jiang, M.; Senoo, T.; Ishii, I. Omnidirectional Panoramic Video System with Frame-by-Frame Ultrafast Viewpoint ControlA Simultaneous Multi-Object Zooming System Using an Ultrafast Pan-Tilt Camera. IEEE Robot. Autom. Lett. 2022, 7, 4086–4093. [Google Scholar] [CrossRef]
- Li, Q.; Hu, S.; Shimasaki, K.; Ishii, I. An Active Multi-Object Ultrafast Tracking System with CNN-Based Hybrid Object Detection. Sensors 2023, 23, 4150. [Google Scholar] [CrossRef]
Method | Necessity of Additional Sensors | Capability of Depth Measurement | Frame Rate of Well-Focused Images |
---|---|---|---|
(i) Contrast-Based AF [37] | No | No | Not Stable |
(ii) Phase-Based AF [20,35,36] | Yes | No | Not Stable |
(iii) Active-Based AF [22,23,33,34] | Yes | Yes | Stable |
(iv) AF based on Global-Range Focal Sweep [26,27] | No | Yes | Stable but Relatively Low |
(v) AF based on Dynamic-Range Focal Sweep (This Study) | No | Yes | Stable and Relatively High |
Active Vision System | Speed of Pan-Tilt Adjustment | Applicability to High-Magnification Tracking in Pan-Tilt Directions | Applicability to High-Magnification Tracking in Depth Direction |
---|---|---|---|
(i) Camera-Rotation-Based Fixed Focus AVS [66] | Slow. (Pan: 170 deg./s, Tilt: 77 deg./ms) | Only for Low-Speed Objects | No |
(ii) Camera-Rotation-Based Autofocus AVS [67] | Slow (Not Explicitly Stated in the Paper) | Only for Low-Speed Objects | Only for Low-Speed Objects |
(iii) Galvano-Mirror-Based Fixed Focus AVS [68] | Fast (Pan/Tilt: 5.7 deg./ms, Frequency: 500 Hz) | Yes | No |
(iv) Galvano-Mirror-Based AVS with Autofocus Based on Global-Range Focal Sweep [27] | Fast But Low Feedback Frequency (Pan/Tilt: 10 deg./ms, Frequency: 10–15 Hz) | Only for Low-Speed Objects | Yes |
(v) Galvano-Mirror-Based AVS with Autofocus Based on Dynamic-Range Focal Sweep (This Study) | Fast (Pan/Tilt: 10 deg./ms, Frequency: 500 Hz) | Yes | Yes |
Algorithm Item | The Utilized PC | Cost Time per Image (Unit: ms) |
---|---|---|
Focus Measure Algorithm | PC 1 | 1.7 |
YOLOv8 Detection | PC 2 | 11.8 |
K-means Clustering | PC 2 | 27.0 |
Tracking Based on Color-Filtering | PC 1 | 1.8 |
Object Item | Width (cm) | Length (cm) |
---|---|---|
Butterfly (Object 1) | 1.6 | 1.2 |
QR Code (Object 2) | 1.5 | 1.5 |
Screw (Object 3) | 1.9 | 0.7 |
Object | Lighting Condition (i) | Lighting Condition (ii) | Lighting Condition (iii) | |||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0.6 m | 0.9 m | 1.2 m | 1.5 m | 1.8 m | 2.1 m | 0.6 m | 0.9 m | 1.2 m | 1.5 m | 1.8 m | 2.1 m | 0.6 m | 0.9 m | 1.2 m | 1.5 m | 1.8 m | 2.1 m | |
1. | 0.024 | 0.082 | 0.131 | 0.191 | 0.156 | 0.307 | 0.021 | 0.172 | 0.132 | 0.165 | 0.181 | 0.207 | 0.064 | 0.054 | 0.155 | 0.226 | 0.318 | 0.582 |
2. | 0.035 | 0.081 | 0.159 | 0.184 | 0.170 | 0.239 | 0.051 | 0.089 | 0.180 | 0.192 | 0.149 | 0.296 | 0.039 | 0.077 | 0.160 | 0.140 | 0.327 | 0.454 |
3. | 0.020 | 0.070 | 0.181 | 0.160 | 0.186 | 0.215 | 0.054 | 0.119 | 0.186 | 0.213 | 0.179 | 0.510 | 0.079 | 0.117 | 0.167 | 0.299 | 0.199 | 0.524 |
Avg. | 0.026 | 0.078 | 0.157 | 0.178 | 0.171 | 0.254 | 0.044 | 0.127 | 0.166 | 0.190 | 0.170 | 0.338 | 0.061 | 0.083 | 0.161 | 0.222 | 0.281 | 0.520 |
Object | Lighting Condition (i) | Lighting Condition (ii) | Lighting Condition (iii) | |||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0.6 m | 0.9 m | 1.2 m | 1.5 m | 1.8 m | 2.1 m | 0.6 m | 0.9 m | 1.2 m | 1.5 m | 1.8 m | 2.1 m | 0.6 m | 0.9 m | 1.2 m | 1.5 m | 1.8 m | 2.1 m | |
1. | 6.50% | 12.56% | 23.21% | 17.03% | 15.42% | 31.54% | 6.08% | 20.73% | 25.50% | 21.32% | 24.46% | 31.03% | 35.36% | 17.98% | 23.48% | 20.18% | 20.67% | 20.66% |
2. | 2.01% | 2.58% | 2.82% | 2.44% | 14.66% | 25.49% | 3.97% | 3.71% | 3.86% | 8.53% | 12.02% | 18.10% | 1.87% | 2.15% | 3.71% | 5.06% | 16.77% | 11.70% |
3. | 9.01% | 18.02% | 23.38% | 44.98% | 48.67% | 35.88% | 18.05% | 31.33% | 23.95% | 30.50% | 42.57% | 38.50% | 8.56% | 17.56% | 28.19% | 49.9% | 32.32% | 28.93% |
Avg. | 5.84% | 11.05% | 16.47% | 21.48% | 25.26% | 30.97% | 9.37% | 18.59% | 17.77% | 20.12% | 26.68% | 29.21% | 15.26% | 12.56% | 18.46% | 25.05% | 23.25% | 20.43% |
Object Item | 0.75 m | 1.25 m | 1.75 m |
---|---|---|---|
Butterfly (Object 1) | 4.93 | 0.89 | 2.66 |
QR Code (Object 2) | 14.81 | 12.23 | 3.11 |
Screw (Object 3) | 2.72 | 3.36 | 3.83 |
Avg. | 7.49 | 5.49 | 3.20 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zhang, T.; Shimasaki, K.; Ishii, I.; Namiki, A. High-Magnification Object Tracking with Ultra-Fast View Adjustment and Continuous Autofocus Based on Dynamic-Range Focal Sweep. Sensors 2024, 24, 4019. https://doi.org/10.3390/s24124019
Zhang T, Shimasaki K, Ishii I, Namiki A. High-Magnification Object Tracking with Ultra-Fast View Adjustment and Continuous Autofocus Based on Dynamic-Range Focal Sweep. Sensors. 2024; 24(12):4019. https://doi.org/10.3390/s24124019
Chicago/Turabian StyleZhang, Tianyi, Kohei Shimasaki, Idaku Ishii, and Akio Namiki. 2024. "High-Magnification Object Tracking with Ultra-Fast View Adjustment and Continuous Autofocus Based on Dynamic-Range Focal Sweep" Sensors 24, no. 12: 4019. https://doi.org/10.3390/s24124019