A High-Speed Imaging Method Based on Compressive Sensing for Sound Extraction Using a Low-Speed Camera
<p>The schematic diagram of a compressive camera for sound detection. The forward sensing model includes a light spot video (<math display="inline"> <semantics> <mi>x</mi> </semantics> </math>) consisting of T frames, with <math display="inline"> <semantics> <mrow> <mi mathvariant="normal">H</mi> <mo>×</mo> <mi mathvariant="normal">W</mi> </mrow> </semantics> </math> pixels each, which is multiplied by T sampling patterns embedded within the sensing matrix <math display="inline"> <semantics> <mrow> <mi mathvariant="normal">Φ</mi> </mrow> </semantics> </math>. The CCD (charge-coupled device) camera integrates over the exposure time, producing a single modulated image <math display="inline"> <semantics> <mi>y</mi> </semantics> </math> consisting of <math display="inline"> <semantics> <mrow> <mi mathvariant="normal">H</mi> <mo>×</mo> <mi mathvariant="normal">W</mi> </mrow> </semantics> </math> pixels. In the backward model, the algorithms based on compressive sensing are used to reconstruct the fast light spot video from the modulated images. Finally, the centroids of the reconstructed light spots are used to recover the sound.</p> "> Figure 2
<p>The simulation setup. The red dashed arrow line indicates the light path.</p> "> Figure 3
<p>The simulation results for a sinusoidal audio signal with a frequency of 50 Hz. (<b>a</b>) The reconstruction quality of the light spot video images as a function of the compression rate for both the GAP (Generalized Alternating Projection) and TwIST (Two-step Iterative-Shrinkage Thresholding) algorithms; (<b>b</b>) the quality of the sound signal recovered by the reconstructed sound vibrating light spot video as a function of the compression rate.</p> "> Figure 4
<p>The simulation results for an audio signal with increasing frequencies from 50 Hz to 300 Hz with an interval of 50 Hz. Each segment lasts 2 s. (<b>a</b>) The original audio spectrogram. (<b>b</b>) The spectrogram of audio reconstructed directly from the high-speed video and (<b>c</b>) its frequency spectrum. The spectrogram of audio reconstructed from the high-speed video reconstructed using the compressive camera with (<b>d</b>) 8×, (<b>e</b>) 16×, and (<b>f</b>) 24× compression. (<b>g</b>) The frequency spectrum of the signal in (<b>f</b>). The red text of 50 to 300 Hz in (<b>c</b>) and (<b>g</b>) indicate the expected signal, while f1 to f11 in (<b>g</b>) indicate the undesired harmonic noise related to the frame rate (2000/24 Hz) of the compressive camera.</p> "> Figure 5
<p>(<b>a</b>) The original audio waveform; (<b>b</b>) the audio waveform recovered directly from the high-speed video captured by the high-speed camera; (<b>c</b>) the audio waveform recovered from the reconstructed high-speed video using the compressive camera with 8× compression.</p> "> Figure 6
<p>The simulation results for complex sound (speech containing counting from one to four in Chinese). (<b>a</b>) The original audio (see <a href="#app1-sensors-18-01524" class="html-app">supplementary, Media1</a>) spectrogram; (<b>b</b>) the audio (see <a href="#app1-sensors-18-01524" class="html-app">supplementary, Media 2</a>) spectrogram reconstructed directly from the high-speed video, reconstructed audio spectrogram from the high-speed video reconstructed using the compressive camera with 8× (<b>c</b>) (see <a href="#app1-sensors-18-01524" class="html-app">supplementary, Media 3</a>), 16× (<b>d</b>) (see <a href="#app1-sensors-18-01524" class="html-app">supplementary, Media 4</a>) and 24× (<b>e</b>) (see <a href="#app1-sensors-18-01524" class="html-app">supplementary, Media 5</a>) compression and (<b>f</b>) intelligibility values (0–1) that evaluate the similarity between the reconstructed sounds and the test one.</p> "> Figure 7
<p>The comparison of the reconstructed high-speed video with the original one at the case of 16× compression. (<b>Top row</b>): the first four frames of the original high-speed video. (<b>Bottom row</b>): the corresponding reconstructed video and the coded image. The high-speed video in the bottom row are reconstructed from the coded image.</p> "> Figure 8
<p>The experimental setup. The red dashed arrow line indicates the light path.</p> "> Figure 9
<p>The comparison of the designed sampling patterns with the used image patterns.</p> "> Figure 10
<p>The dependence of the SNR (signal-to-noise ratio) of the recovered sinusoidal sound signal on the delay between the signals that trigger the CCD camera recording and DMD (digital micro-mirror device) modulation. The frequency of the sinusoidal voltage applied to the loudspeaker is 150 Hz, the rate of CCD image recording is 100 Hz, and the rate of DMD modulation is 800 Hz.</p> "> Figure 11
<p>The dependence of the SNRs of the recovered sinusoidal sound signal on the ratio of the number of zeros to ones in each designed sampling pattern at different compressions of 8×, 16×, and 20×. The frequency of the sinusoidal voltage applied to the loudspeaker is 150 Hz and the rate of CCD image recording is 100 Hz.</p> "> Figure 12
<p>The dependence of the SNR (circle symbol) and THD (total harmonic distortion) (star symbol) of the recovered sinusoidal sound signal on the amplitude of the sinusoidal voltage applied to the loudspeaker at the different compressions of 8× (red points), 16× (green points), and 20× (blue points). The frequency of the sinusoidal voltage applied to the loudspeaker is 150 Hz and the rate of CCD image recording is 100 Hz.</p> "> Figure 13
<p>The experimental results for an audio signal consisting of six sinusoidal sound segments with increasing frequencies from 70 Hz to 320 Hz with an interval of 50 Hz. Each segment lasted one second. (<b>a</b>) The test sound frequency spectrum; (<b>b</b>) the recovered sound frequency spectrum without filtering and (<b>c</b>) with filtering; (<b>d</b>) the test audio spectrogram; (<b>e</b>) the recovered audio spectrogram without filtering and (<b>f</b>) with filtering.</p> "> Figure 14
<p>The experimental result for speech (counting from one to four in Chinese). (<b>a</b>) The test sound (see <a href="#app1-sensors-18-01524" class="html-app">supplementary, Media 6</a>) frequency spectrum; (<b>b</b>) the recovered sound frequency spectrum without filtering and (<b>c</b>) with filtering (see <a href="#app1-sensors-18-01524" class="html-app">supplementary, Media 7</a>); (<b>d</b>) the test audio spectrogram; (<b>e</b>) the recovered audio spectrogram without filtering and (<b>f</b>) with filtering.</p> "> Figure 15
<p>(<b>Top row</b>): the left is a static light spot image; the right is a coded image at the case of 20× compression for sound recovery. (<b>Bottom row</b>): the first four frames of the high-speed video reconstructed from the coded image in the (<b>Top row</b>).</p> "> Figure 16
<p>The comparison of the performance of the white light source and the laser light source for sound recovery. The sound vibration was generated by the loudspeaker, which was driven by a 2 Hz sinusoidal sound signal. The reflected light spot video was captured by the CCD camera with a frame rate of 30 Hz. (<b>Top row</b>): the white light was illuminated on the loudspeaker membrane. (<b>Middle row</b>): laser light was illuminated on the loudspeaker membrane. (<b>Bottom row</b>): laser light was illuminated on a reflecting mirror that was taped onto the loudspeaker membrane. From left to right, (<b>First column</b>): one typical frame captured by the CCD camera. (<b>Second column</b>): recovered audio waveform directly from the captured video by the CCD camera. (<b>Third column</b>): one typical reconstructed frame using the GAP algorithm, and eight frames were reconstructed from each coded frame. (<b>Fourth column</b>): the quality of the reconstruction in terms of PSNR. (<b>Fifth column</b>): the recovered audio waveform from the reconstructed video.</p> ">
Abstract
:1. Introduction
2. Methods
3. Simulations
3.1. Simulation Setup
3.2. Simulation Results
4. Experiments
4.1. Experimental Setup
4.2. Experimental Results
5. Discussions
6. Conclusions
Supplementary Materials
Author Contributions
Funding
Conflicts of Interest
References
- Zieger, C.; Brutti, A.; Svaizer, P. Acoustic Based Surveillance System for Intrusion Detection. In Proceedings of the 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, Genova, Italy, 2–4 September 2009; pp. 314–319. [Google Scholar]
- Clavel, C.; Ehrette, T.; Richard, G. Events Detection for an Audio-Based Surveillance System. In Proceedings of the 2005 IEEE International Conference on Multimedia and Expo, Amsterdam, The Netherlands, 6 July 2005; pp. 1306–1309. [Google Scholar]
- Zhang, L.; Zhang, X.; Tang, W. Amplitude Measurement of Weak Sinusoidal Water Surface Acoustic Wave Using Laser Interferometer. Chin. Opt. Lett. 2015, 13, 091202. [Google Scholar] [CrossRef]
- Li, L.; Zeng, H.; Zhang, Y.; Kong, Q.; Zhou, Y.; Liu, Y. Analysis of Backscattering Characteristics of Objects for Remote Laser Voice Acquisition. Appl. Opt. 2014, 53, 971–978. [Google Scholar] [CrossRef] [PubMed]
- Shang, J.; He, Y.; Liu, D.; Zang, H.; Chen, W.B. Laser Doppler Vibrometer for Real-Time Speech-Signal Acquirement. Chin. Opt. Lett. 2009, 7, 732–733. [Google Scholar] [CrossRef]
- Li, R.; Wang, T.; Zhu, Z.G.; Xiao, W. Vibration Characteristics of Various Surfaces Using an LDV for Long-Range Voice Acquisition. IEEE Sens. J. 2011, 11, 1415–1422. [Google Scholar] [CrossRef]
- Qu, Y.; Wang, T.; Zhu, Z.G. Vision-Aided Laser Doppler Vibrometry for Remote Automatic Voice Detection. IEEE/ASME Trans. Mech. 2011, 16, 1110–1119. [Google Scholar] [CrossRef]
- Qu, Y.; Wang, T.; Zhu, Z.G. An Active Multimodal Sensing Platform for Remote Voice Detection. In Proceedings of the 2010 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, Montreal, QC, Canada, 6–9 July 2010; pp. 627–632. [Google Scholar]
- Veber, A.A.; Lyashedko, A.; Sholokhov, E.; Trikshev, A.; Kurkov, A.; Pyrkov, Y.; Veber, A.E.; Seregin, V.; Tsvetkov, V. Laser Vibrometry Based on Analysis of the Speckle Pattern from a Remote Object. Appl. Phys. B 2011, 105, 613–617. [Google Scholar] [CrossRef]
- Akutsu, M.; Oikawa, Y.; Yamasaki, Y. Extract Voice Information Using High-Speed Camera. J. Acoust. Soc. Am. 2013, 133, 3297. [Google Scholar] [CrossRef]
- Davis, A.; Rubinstein, M.; Wadhwa, N.; Mysore, G.J.; Durand, F.; Freeman, W.T. The Visual Microphone: Passive Recovery of Sound from Video. ACM Trans. Graph. 2014, 33, 79. [Google Scholar] [CrossRef]
- Wang, Z.Y.; Nguyen, H.; Quisberth, J. Audio Extraction from Silent High-Speed Video Using an Optical Technique. Opt. Eng. 2014, 53, 110502. [Google Scholar] [CrossRef]
- Zhang, D.; Guo, J.; Lei, X.; Zhu, C. Note: Sound Recovery from Video Using SVD-Based Information Extraction. Rev. Sci. Instrum. 2016, 87, 086111. [Google Scholar] [CrossRef] [PubMed]
- Zhang, D.S.; Guo, J.; Jin, Y.; Zhu, C.A. Efficient Subtle Motion Detection from High-Speed Video for Sound Recovery and Vibration Analysis Using Singular Value Decomposition-Based Approach. Opt. Eng. 2017, 56, 094105. [Google Scholar] [CrossRef]
- Zalevsky, Z.; Beiderman, Y.; Margalit, I.; Gingold, S.; Teicher, M.; Mico, V.; Garcia, J. Simultaneous Remote Extraction of Multiple Speech Sources and Heart Beats from Secondary Speckles Pattern. Opt. Express 2009, 17, 21566–21580. [Google Scholar] [CrossRef] [PubMed]
- Chen, Z.; Wang, C.; Huang, C.; Fu, H.; Luo, H.; Wang, H. Audio Signal Reconstruction Based on Adaptively Selected Seed Points from Laser Speckle Images. Opt. Commun. 2014, 331, 6–13. [Google Scholar] [CrossRef]
- Zhu, G.; Yao, X.R.; Qiu, P.; Mahmood, W.; Yu, W.K.; Sun, Z.B.; Zhai, G.J.; Zhao, Q. Sound Recovery via Intensity Variations of Speckle Pattern Pixels Selected with Variance-Based Method. Opt. Eng. 2018, 57, 026117. [Google Scholar] [CrossRef]
- El-Desouki, M.; Deen, M.J.; Fang, Q.; Liu, L.; Tse, F.; Armstrong, D. CMOS Image Sensors for High Speed Applications. Sensors 2009, 9, 430–444. [Google Scholar] [CrossRef] [PubMed]
- Pournaghi, R.; Wu, X.L. Coded Acquisition of High Frame Rate Video. IEEE Trans. Image Process. 2014, 23, 5670–5682. [Google Scholar] [CrossRef] [PubMed]
- Bub, G.; Tecza, M.; Helmes, M.; Lee, P.; Kohl, P. Temporal Pixel Multiplexing for Simultaneous High-Speed, High-Resolution Imaging. Nat. Methods 2010, 7, 209–211. [Google Scholar] [CrossRef] [PubMed]
- Feng, W.; Zhang, F.; Qu, X.; Zheng, S. Per-Pixel Coded Exposure for High-Speed and High-Resolution Imaging Using a Digital Micromirror Device Camera. Sensors 2016, 16, 331. [Google Scholar] [CrossRef] [PubMed]
- Donoho, D.L. Compressed sensing. IEEE Trans. Inf. Theory 2006, 52, 1289–1306. [Google Scholar] [CrossRef]
- Candes, E.J.; Wakin, M.B. An introduction to compressive sampling. IEEE Signal Process. Mag. 2008, 25, 21–30. [Google Scholar] [CrossRef]
- Rombers, J. Imaging via compressive sampling [introduction to compressive sampling and recovery via convex programming]. IEEE Signal Process. Mag. 2008, 25, 14–20. [Google Scholar]
- Baraniuk, R.G. Compressive sensing [lecture notes]. IEEE Signal Process. Mag. 2007, 24, 118–124. [Google Scholar] [CrossRef]
- Dadkhah, M.; Deen, M.J.; Shirani, S. Compressive Sensing Image Sensors-Hardware Implementation. Sensors 2013, 13, 4961–4978. [Google Scholar] [CrossRef] [PubMed]
- Sankaranarayanan, A.C.; Studer, C.; Baraniuk, R.G. CS-MUVI: Video Compressive Sensing for Spatial-Multiplexing Cameras. In Proceedings of the IEEE International Conference on Computational Photography, Seattle, WA, USA, 28–29 April 2012; pp. 1–10. [Google Scholar]
- Reddy, D.; Veeraraghavan, A.; Chellappa, R. P2C2: Programmable Pixel Compressive Camera for High Speed Imaging. In Proceedings of the CVPR 2011, Colorado Springs, CO, USA, 20–25 June 2011; pp. 329–336. [Google Scholar]
- Hitomi, Y.; Gu, J.; Gupta, M.; Mitsunaga, T.; Nayar, S.K. Video from a Single Coded Exposure Photograph Using a Learned over-Complete Dictionary. In Proceedings of the International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011; pp. 287–294. [Google Scholar]
- Liu, D.; Gu, J.; Hitomi, Y.; Gupta, M.; Mitsunaga, T.; Nayar, S.K. Efficient Space-Time Sampling with Pixel-Wise Coded Exposure for High-Speed Imaging. IEEE Trans. Pattern Anal. Mach. Intell. 2014, 36, 248–260. [Google Scholar] [PubMed]
- Llull, P.; Liao, X.; Yuan, X.; Yang, J.; Kittle, D.; Carin, L.; Sapiro, G.; Brady, D.J. Coded Aperture Compressive Temporal Imaging. Opt. Express 2013, 21, 10526–10545. [Google Scholar] [CrossRef] [PubMed]
- Koller, R.; Schmid, L.; Matsuda, N.; Niederberger, T.; Spinoulas, L.; Cossairt, O.; Schuster, G.; Katsaggelos, A.K. High Spatio-Temporal Resolution Video with Compressed Sensing. Opt. Express 2015, 23, 15992–16007. [Google Scholar] [CrossRef] [PubMed]
- Veeraraghavan, A.; Reddy, D.; Raskar, R. Coded Strobing Photography: Compressive Sensing of High Speed Periodic Videos. IEEE Trans. Pattern Anal. Mach. Intell. 2011, 33, 671–686. [Google Scholar] [CrossRef] [PubMed]
- Holloway, J.; Sankaranarayanan, A.C.; Veeraraghavan, A.; Tambe, S. Flutter Shutter Video Camera for Compressive Sensing of Videos. In Proceedings of the IEEE International Conference on Computational Photography, Seattle, WA, USA, 28–29 April 2012; pp. 1–9. [Google Scholar]
- Serrano, A.; Gutierrez, D.; Masia, B. Compressive High Speed Video Acquisition. In Proceedings of the Spanish Computer Graphics Conference (CEIG), Benicàssim, Spain, 1–3 July 2015; p. 1. [Google Scholar]
- Yuan, X. Generalized Alternating Projection Based Total Variation Minimization for Compressive Sensing. In Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 25–28 September 2016; pp. 2539–2543. [Google Scholar]
- Bioucas-Dias, J.M.; Figueiredo, M.A.T. A New TwIST: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration. IEEE Trans. Image Process. 2007, 16, 2992–3004. [Google Scholar] [CrossRef] [PubMed]
- Boldt, J.B.; Ellis, D.P.W. A Simple Correlation-Based Model of Intelligibility for Nonlinear Speech Enhancement and Separation. In Proceedings of the 2009 17th European Signal Processing Conference, Glasgow, UK, 24–28 August 2009; pp. 1849–1853. [Google Scholar]
- 0.7 XGA 12° DDR DMD Discovery. Available online: http://www.loreti.it/Download/PDF/DMD/7XGAdlp.pdf (accessed on 30 August 2005).
- MantaG-145-30fps. Available online: https://www.alliedvision.com/en/products/cameras/detail/Manta/G-145-30fps/action/pdf.html (accessed on 8 July 2016).
- Bloshkina, A.I.; Li, L.; Gubarev, F.A.; Klenovskii, M.S. Investigation of Extracting Information from Vibrating Objects by Digital Speckle Correlation. In Proceedings of the 2016 17th International Conference of Young Specialists on Micro/Nanotechnologies and Electron Devices (EDM), Erlagol, Russia, 30 June–4 July 2016; pp. 637–641. [Google Scholar]
- Ri, S.E.; Matsunaga, Y.; Fujigaki, M.; Matui, T.; Morimoto, Y. Development of DMD Reflection-Type CCD Camera for Phase Analysis and Shape Measurement. Optomech. Sens. Instrum. 2005, 6049. [Google Scholar] [CrossRef]
© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zhu, G.; Yao, X.-R.; Sun, Z.-B.; Qiu, P.; Wang, C.; Zhai, G.-J.; Zhao, Q. A High-Speed Imaging Method Based on Compressive Sensing for Sound Extraction Using a Low-Speed Camera. Sensors 2018, 18, 1524. https://doi.org/10.3390/s18051524
Zhu G, Yao X-R, Sun Z-B, Qiu P, Wang C, Zhai G-J, Zhao Q. A High-Speed Imaging Method Based on Compressive Sensing for Sound Extraction Using a Low-Speed Camera. Sensors. 2018; 18(5):1524. https://doi.org/10.3390/s18051524
Chicago/Turabian StyleZhu, Ge, Xu-Ri Yao, Zhi-Bin Sun, Peng Qiu, Chao Wang, Guang-Jie Zhai, and Qing Zhao. 2018. "A High-Speed Imaging Method Based on Compressive Sensing for Sound Extraction Using a Low-Speed Camera" Sensors 18, no. 5: 1524. https://doi.org/10.3390/s18051524