A Fine-Grained Ship-Radiated Noise Recognition System Using Deep Hybrid Neural Networks with Multi-Scale Features
"> Figure 1
<p>Selected 12 sound recordings: (<b>a</b>) Minho uno; (<b>b</b>) Arroios; (<b>c</b>) Motorboat “Duda”; (<b>d</b>) Pirata de Cies; (<b>e</b>) Mussel boat2; (<b>f</b>) Mar de Onza; (<b>g</b>) Mussel boat4; (<b>h</b>) Pirata de Salvora; (<b>i</b>) Sailboat; (<b>j</b>) Noise 1; (<b>k</b>) Noise 2; (<b>l</b>) Mar de Cangas.</p> "> Figure 2
<p>The flowchart of the proposed ship-radiated noise recognition system.</p> "> Figure 3
<p>The process of multi-scale feature extraction.</p> "> Figure 4
<p>The architecture of the proposed MFAGNet model.</p> "> Figure 5
<p>Original sample signals and MEEMD decomposition results: (<b>a</b>) Bateero, (<b>b</b>) PirataSalvora.</p> "> Figure 6
<p>Hilbert-Huang spectrum and marginal spectrum: (<b>a</b>) Bateero, (<b>b</b>) PirataSalvora.</p> "> Figure 7
<p>The correlation coefficient between IMFs and original sampled signal and normalized energy of IMFs: (<b>a</b>) Bateero, (<b>b</b>) PirataSalvora.</p> "> Figure 8
<p>The loss and accuracy by using combination A during the training process: (<b>a</b>) MFAGNet, (<b>b</b>) LSTMs.</p> "> Figure 9
<p>Confusion matrix of 12 different specific noises.</p> "> Figure 10
<p>Raw radiated noise from four different types of ships: (<b>a</b>) cargo; (<b>b</b>) passenger ship; (<b>c</b>) tanker; (<b>d</b>) tug.</p> "> Figure 11
<p>MEEMD decomposition results, Hilbert-Huang spectrum and marginal spectrum of four different types of ship samples: (<b>a</b>) cargo; (<b>b</b>) passenger ship; (<b>c</b>) tanker; (<b>d</b>) tug.</p> "> Figure 12
<p>Multi-scale features 3D display by using T-SNE.</p> "> Figure 13
<p>The loss (<b>a</b>) and accuracy (<b>b</b>) of MFAGNet with the new dataset by using combination A during the training process.</p> ">
Abstract
:1. Introduction
- The special multi-scale and multi-dimensional features are extracted by the improved HHT-based method, including six characteristics of the ship-radiated noise from the energy and time–frequency domain.
- A form of the feature matrix is proposed for the first time, which is a six-order narrow band sub-signal obtained from original ship noise through the improved HHT method, and includes six amplitude–frequency–time-based features from each sub-signal, respectively.
- An innovative multi-scale feature adaptive generalized neural network (MFAGNet) for ship recognition from the extracted feature matrix is designed for ship recognition. The MFAGNet model adopts 1D CNN and LSTM architecture to obtain regional high-level information and aggregate timing correlation characteristics, which can efficiently utilize multi-scale and multi-dimensional feature matrices.
- Unlike other methods that only recognize ship types, this study provides robust and flexible fine-grained identification of different specific ships. To improve the performance of the MFAGNet model, 1D CNN is utilized instead of 2D CNN to learn features, and the pooling layer is removed to maximize the retention of feature information. These modifications provide better insights into the performance of the model in achieving accurate ship recognition.
2. Materials
3. Methods
3.1. The Fine-Grained Ship-Radiated Noise Recognition System
- Step 1: Underwater acoustic signals are obtained from sonar devices.
- Step 2: The obtained noise signal is intercepted to form data samples, and each sample has approximately 3000 sampling points.
- Step 3: Aiming at the high randomness of noise data, a modified ensemble empirical mode decomposition (MEEMD) method based on permutation entropy is employed to adaptively decompose sample data into a series of sub-signals with different frequencies.
- Step 4: Spectral analysis is conducted on the decomposed sub-signals.
- Step 5: Amplitude–time–frequency features are calculated by using decomposed sub-signals and the spectral analysis results. Thus, multi-scale and multi-dimensional feature extraction can be realized.
- Step 6: Multi-scale features are reshaped to sequences in a specific form.
- Step 7: The 1D convolutional module is constructed to extract high-dimensional features.
- Step 8: The designed LSTM module is applied to identify temporal features.
- Step 9: The fully connected module is built as a classifier to achieve the fine-grained classification of 12 specific ship-radiated noises and ambient noises, rather than just the type of ships.
- Step 10: Accurate recognition of specific ship-radiated noises is achieved.
3.1.1. Multi-Scale Feature Extraction Method
- 1.
- Improved Hilbert-Huang transform algorithm
- 2.
- Amplitude–time–frequency domain features
3.1.2. The Proposed MFAGNet Model
- 1.
- The architecture of MFAGNet
- (1)
- Convolutional Layers:
- (2)
- LSTM Layers:
- (3)
- Fully connected layers:
Layer Type | Configuration |
---|---|
Convolutional Layers | |
Convolution 1D + BatchNorm | Filters: 2 × 72, RReLU |
Convolution 1D + BatchNorm | Filters: 2 × 72, RReLU |
Convolution 1D + BatchNorm | Filters: 2 × 72, RReLU |
Convolution 1D + BatchNorm | Filters: 2 × 72, RReLU |
LSTM Layers | |
LSTM + Dropout | Filters: 128, Tanh, 0.5 |
LSTM + Dropout | Filters: 128, Tanh, 0.5 |
LSTM + Dropout | Filters: 128, Tanh, 0.5 |
LSTM + Dropout | Filters: 128, Tanh, 0.5 |
Fully Connected Layers | |
Fully Connected + Dropout | Filters: 512, RReLU, 0.5 |
Output Layer | Filters: 12, RReLU |
3.2. Evaluation Metrics
4. Results and Discussion
4.1. Analysis of Ship-Radiated Noise and Multi-Scale Features
4.2. Comparison between MFAGNet and Other Models
4.2.1. Effective Evaluation of MFAGNet
4.2.2. Robustness Test and Application
5. Conclusions
- A form of feature matrix is proposed for the first time, which adopts the improved HHT-based method to extract the energy, frequency and time domain amplitude features from the original ship-radiated noise.
- MFAGNet, a proposed spatio-temporal model, combines the significant advantages of LSTM and the spatial learning capability of the CNN to compensate for the inadequacies of the feature extraction of higher dimensional spaces. It obtains the best recognition accuracy and generalization for the different specific ships instead of just the type of ships.
- The proposed model is compared with other deep learning and machine learning methods by utilizing six different multi-scale feature combinations, achieving the highest accuracy for 12 specific noises. A total of 48 different models have been generated for comparison. In order to verify the universality and robustness with other datasets, experiments have been conducted, which also demonstrate excellent accuracy.
- The proposed method could automatically learn and update variables based on the data, eliminating the need for researchers to repeatedly tune various parameters. This end-to-end architecture can reduce the time required for deployment and decreases the probability of module docking issues, making actual deployment less troublesome.
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Salerno, E. Using Low-Resolution SAR Scattering Features for Ship Classification. IEEE Geosci. Remote Sens. Lett. 2022, 19, 1–4. [Google Scholar] [CrossRef]
- Tollefsen, D.; Hodgkiss, W.S.; Dosso, S.E.; Bonnel, J.; Knobles, D.P. Probabilistic Estimation of Merchant Ship Source Levels in an Uncertain Shallow-Water Environment. IEEE J. Oceanic Eng. 2022, 47, 647–656. [Google Scholar] [CrossRef]
- Zhu, C.Y.; Seri, S.G.; Mohebbi-Kalkhoran, H.; Ratilal, P. Long-Range Automatic Detection, Acoustic Signature Characterization and Bearing-Time Estimation of Multiple Ships with Coherent Hydrophone Array. Remote Sens. 2020, 12, 3731. [Google Scholar] [CrossRef]
- Chen, X.; Liu, Y.; Achuthan, K.; Zhang, X. A ship movement classification based on Automatic Identification System (AIS) data using Convolutional Neural Network. Ocean Eng. 2020, 218, 108182–108194. [Google Scholar] [CrossRef]
- Parsons, M.J.G.; Lin, T.-H.; Mooney, T.A.; Erbe, C.; Juanes, F.; Lammers, M.; Li, S.; Linke, S.; Looby, A.; Nedelec, S.L.; et al. Sounding the Call for a Global Library of Underwater Biological Sounds. Front. Ecol. Evol. 2022, 10, 810156–810175. [Google Scholar] [CrossRef]
- Miglianti, L.; Cipollini, F.; Oneto, L.; Tani, G.; Gaggero, S.; Coraddu, A.; Viviani, M. Predicting the cavitating marine propeller noise at design stage: A deep learning based approach. Ocean Eng. 2020, 209, 107481–107506. [Google Scholar] [CrossRef]
- Dini, G.; Lo Duca, A. A secure communication suite for underwater acoustic sensor networks. Sensors 2012, 12, 15133–15158. [Google Scholar] [CrossRef] [Green Version]
- Jin, S.-Y.; Su, Y.; Guo, C.-J.; Fan, Y.-X.; Tao, Z.-Y. Offshore ship recognition based on center frequency projection of improved EMD and KNN algorithm. Mech. Syst. Sig. Process. 2023, 189, 110076–110086. [Google Scholar] [CrossRef]
- Zhu, S.; Zhang, G.J.; Wu, D.Y.; Jia, L.; Zhang, Y.F.; Geng, Y.N.; Liu, Y.; Ren, W.R.; Zhang, W.D. High Signal-to-Noise Ratio MEMS Noise Listener for Ship Noise Detection. Remote Sens. 2023, 15, 777. [Google Scholar] [CrossRef]
- Syrjal, J.; Kalliola, R.; Pajala, J. Underwater Acoustic Environment of Coastal Sea With Heavy Shipping Traffic: NE Baltic Sea During Wintertime. Front. Mar. Sci. 2020, 7, 589141–589152. [Google Scholar] [CrossRef]
- Yi, Y.; Li, Y.; Wu, J. Multi-scale permutation Lempel-Ziv complexity and its application in feature extraction for Ship-radiated noise. Front. Mar. Sci. 2022, 9, 1047332. [Google Scholar] [CrossRef]
- Li, Y.; Li, Y.; Chen, X.; Yu, J. Denoising and feature extraction algorithms using NPE combined with VMD and their applications in ship-radiated noise. Symmetry 2017, 9, 256. [Google Scholar] [CrossRef] [Green Version]
- Ke, X.; Yuan, F.; Cheng, E. Integrated optimization of underwater acoustic ship-radiated noise recognition based on two-dimensional feature fusion. Appl. Acoust. 2020, 159, 107057–107069. [Google Scholar] [CrossRef]
- Filho, W.S.; de Seixas, J.M.; de Moura, N.N. Preprocessing passive sonar signals for neural classification. Iet Radar Sonar Navig. 2011, 5, 605–612. [Google Scholar] [CrossRef]
- Wang, S.; Zeng, X. Robust underwater noise targets classification using auditory inspired time-frequency analysis. Appl. Acoust. 2014, 78, 68–76. [Google Scholar] [CrossRef]
- Choi, J.; Choo, Y.; Lee, K. Acoustic Classification of Surface and Underwater Vessels in the Ocean Using Supervised Machine Learning. Sensors 2019, 19, 3492. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Li, Y.; Gao, P.; Tang, B.; Yi, Y.; Zhang, J. Double Feature Extraction Method of Ship-Radiated Noise Signal Based on Slope Entropy and Permutation Entropy. Entropy 2021, 24, 22. [Google Scholar] [CrossRef]
- Li, Y.; Geng, B.; Jiao, S. Dispersion entropy-based Lempel-Ziv complexity: A new metric for signal analysis. Chaos Solitons Fractals 2022, 161, 112400–112409. [Google Scholar] [CrossRef]
- Li, Y.; Tang, B.; Yi, Y. A novel complexity-based mode feature representation for feature extraction of ship-radiated noise using VMD and slope entropy. Appl. Acoust. 2022, 196, 108899–108913. [Google Scholar] [CrossRef]
- Liu, F.; Li, G.; Yang, H. A new feature extraction method of ship radiated noise based on variational mode decomposition, weighted fluctuation-based dispersion entropy and relevance vector machine. Ocean Eng. 2022, 266, 113143. [Google Scholar] [CrossRef]
- Santos-Dominguez, D.; Torres-Guijarro, S.; Cardenal-Lopez, A.; Pena-Gimenez, A. ShipsEar: An underwater vessel noise database. Appl. Acoust. 2016, 113, 64–69. [Google Scholar] [CrossRef]
- Irfan, M.; Zheng, J.; Ali, S.; Iqbal, M.; Masood, Z.; Hamid, U. DeepShip: An underwater acoustic benchmark dataset and a separable convolution based autoencoder for classification. Expert Syst. Appl. 2021, 183, 115270–115281. [Google Scholar] [CrossRef]
- Yang, H.; Li, L.-L.; Li, G.-H.; Guan, Q.-R. A novel feature extraction method for ship-radiated noise. Def. Technol. 2022, 18, 604–617. [Google Scholar] [CrossRef]
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature 2015, 521, 436–444. [Google Scholar] [CrossRef] [PubMed]
- Yue, H.; Zhang, L.; Wang, D.; Wang, Y.; Lu, Z. The classification of underwater acoustic targets based on deep learning methods. In Proceedings of the 2017 2nd International Conference on Control, Automation and Artificial Intelligence (CAAI 2017), Sanya, China, 25–26 June 2017; pp. 526–529. [Google Scholar]
- Zheng, Y.; Gong, Q.; Zhang, S. Time-Frequency Feature-Based Underwater Target Detection with Deep Neural Network in Shallow Sea. J. Phys. Conf. Ser. 2021, 1756, 012006. [Google Scholar] [CrossRef]
- Zhang, Q.; Da, L.; Zhang, Y.; Hu, Y. Integrated neural networks based on feature fusion for underwater target recognition. Appl. Acoust. 2021, 182, 108261–108269. [Google Scholar] [CrossRef]
- Liu, F.; Shen, T.; Luo, Z.; Zhao, D.; Guo, S. Underwater target recognition using convolutional recurrent neural networks with 3-D Mel-spectrogram and data augmentation. Appl. Acoust. 2021, 178, 107989–107995. [Google Scholar] [CrossRef]
- Xie, Y.; Ren, J.; Xu, J. Adaptive ship-radiated noise recognition with learnable fine-grained wavelet transform. Ocean Eng. 2022, 265, 112626–112634. [Google Scholar] [CrossRef]
- Planakis, N.; Papalambrou, G.; Kyrtatos, N. Ship energy management system development and experimental evaluation utilizing marine loading cycles based on machine learning techniques. Appl. Energy 2022, 307, 118085–118103. [Google Scholar] [CrossRef]
- Zhu, Y.; Wang, B.; Zhang, Y.; Li, J.; Wu, C. Convolutional neural network based filter bank multicarrier system for underwater acoustic communications. Appl. Acoust. 2021, 177, 107920–107925. [Google Scholar] [CrossRef]
- Xie, Y.; Xiao, Y.; Liu, X.; Liu, G.; Jiang, W.; Qin, J. Time-Frequency Distribution Map-Based Convolutional Neural Network (CNN) Model for Underwater Pipeline Leakage Detection Using Acoustic Signals. Sensors 2020, 20, 5040. [Google Scholar] [CrossRef]
- Zare, M.; Nouri, N.M. A novel hybrid feature extraction approach of marine vessel signal via improved empirical mode decomposition and measuring complexity. Ocean Eng. 2023, 271, 113727–113743. [Google Scholar] [CrossRef]
- Ying, W.; Zheng, J.; Pan, H.; Liu, Q. Permutation entropy-based improved uniform phase empirical mode decomposition for mechanical fault diagnosis. Digital Signal Process. 2021, 117, 103167–103182. [Google Scholar] [CrossRef]
- Xie, Y.; Wang, S.; Zhang, G.; Fan, Y.; Fernandez, C.; Blaabjerg, F. Optimized multi-hidden layer long short-term memory modeling and suboptimal fading extended Kalman filtering strategies for the synthetic state of charge estimation of lithium-ion batteries. Appl. Energy 2023, 336, 120866–120882. [Google Scholar] [CrossRef]
- Kim, J.; Kim, E.; Kim, D. A Black Ice Detection Method Based on 1-Dimensional CNN Using mmWave Sensor Backscattering. Remote Sens. 2022, 14, 5252. [Google Scholar] [CrossRef]
- Li, J.R.; Chen, L.B.; Shen, J.; Xiao, X.W.; Liu, X.S.; Sun, X.; Wang, X.; Li, D.R. Improved Neural Network with Spatial Pyramid Pooling and Online Datasets Preprocessing for Underwater Target Detection Based on Side Scan Sonar Imagery. Remote Sens. 2023, 15, 440. [Google Scholar] [CrossRef]
- Bao, F.; Wang, X.; Tao, Z.; Wang, Q.; Du, S. EMD-based extraction of modulated cavitation noise. Mech. Syst. Sig. Process. 2010, 24, 2124–2136. [Google Scholar] [CrossRef]
- Flandrin, P.; Rilling, G.; Goncalves, P. Empirical mode decomposition as a filter bank. IEEE Signal Process Lett. 2004, 11, 112–114. [Google Scholar] [CrossRef] [Green Version]
- Khayer, K.; Roshandel Kahoo, A.; Soleimani Monfared, M.; Tokhmechi, B.; Kavousi, K. Target-Oriented Fusion of Attributes in Data Level for Salt Dome Geobody Delineation in Seismic Data. Nat. Resour. Res. 2022, 31, 2461–2481. [Google Scholar] [CrossRef]
- Al-qaness, M.A.A.; Ewees, A.A.; Fan, H.; Abualigah, L.; Elaziz, M.A. Boosted ANFIS model using augmented marine predator algorithm with mutation operators for wind power forecasting. Appl. Energy 2022, 314, 118851–118862. [Google Scholar] [CrossRef]
- Velasco-Gallego, C.; Lazakis, I. RADIS: A real-time anomaly detection intelligent system for fault diagnosis of marine machinery. Expert Syst. Appl. 2022, 204, 117634–117646. [Google Scholar] [CrossRef]
- Gers, F.A.; Schmidhuber, J.; Cummins, F. Learning to forget: Continual prediction with LSTM. Neural Comput. 2000, 12, 2451–2471. [Google Scholar] [CrossRef]
- Van Houdt, G.; Mosquera, C.; Nápoles, G. A review on the long short-term memory model. Artif. Intell. Rev. 2020, 53, 5929–5955. [Google Scholar] [CrossRef]
- Duan, Y.; Liu, C.; Li, S.; Guo, X.; Yang, C. An automatic affinity propagation clustering based on improved equilibrium optimizer and t-SNE for high-dimensional data. Inf. Sci. 2023, 623, 434–454. [Google Scholar] [CrossRef]
Name | ID | Type | Duration(s) | Recording Time |
---|---|---|---|---|
Minho uno | 36 | Passengers | 101.78 | 19 July 2013 12:22:00 |
Arroios | 38 | Passengers | 89.41 | 19 July 2013 00:05:00 |
Motorboat “Duda” | 39 | Motorboat | 61.85 | 19 July 2013 00:30:00 |
Pirata de Cies | 43 | Passengers | 43.09 | 19 July 2013 00:08:00 |
Mussel boat2 | 47 | Mussel boat | 314.08 | 23 July 2013 12:25:00 |
Mar de Onza | 10 | Passengers | 314.00 | 10 July 2013 00:00:00 |
Mussel boat4 | 49 | Mussel boat | 167.00 | 23 July 2013 13:00:00 |
Pirata de Salvora | 53 | Passengers | 161.00 | 23 July 2013 12:40:00 |
Sailboat | 56 | Sailboat | 49.00 | 23 July 2013 12:20:00 |
Mar de Cangas | 62 | Passengers | 154.86 | 23 July 2013 14:00:00 |
Layer Type | Configuration |
---|---|
Convolutional Layers | |
Convolution 1D + BatchNorm | Filters: 2 × 72, RReLU |
Convolution 1D + BatchNorm | Filters: 2 × 72, RReLU |
Convolution 1D + BatchNorm | Filters: 2 × 72, RReLU |
Convolution 1D + BatchNorm | Filters: 2 × 72, RReLU |
Fully Connected Layers | |
Fully Connected + Dropout | Filters: 512, RReLU, 0.5 |
Output Layer | Filters: 12, RReLU |
Layer Type | Configuration |
---|---|
LSTM Layers | |
LSTM + Dropout | Filters: 128, Tanh, 0.5 |
LSTM + Dropout | Filters: 128, Tanh, 0.5 |
LSTM + Dropout | Filters: 128, Tanh, 0.5 |
LSTM + Dropout | Filters: 128, Tanh, 0.5 |
Fully Connected Layers | |
Fully Connected + Dropout | Filters: 512, RReLU, 0.5 |
Output Layer | Filters: 12, RReLU |
Layer Type | Configuration |
---|---|
Fully Connected Layers | |
Fully Connected + Dropout | Filters: 256, RReLU, 0.5 |
Fully Connected + Dropout | Filters: 512, RReLU, 0.5 |
Fully Connected + Dropout | Filters: 512, RReLU, 0.5 |
Fully Connected + Dropout | Filters: 256, RReLU, 0.5 |
Output Layer | Filters: 12, RReLU |
Method | Accuracy | Precision | Recall | F1-Score |
---|---|---|---|---|
MFAGNet | 98.89 | 98.90 | 98.91 | 98.90 |
CNNs | 92.68 | 92.67 | 93.45 | 92.76 |
LSTMs | 93.19 | 93.21 | 94.28 | 93.12 |
DNNs | 84.18 | 84.19 | 84.35 | 83.96 |
KNN | 82.68 | 82.93 | 83.45 | 82.92 |
SVM | 85.69 | 86.00 | 86.11 | 85.96 |
NB | 79.49 | 79.88 | 79.33 | 79.35 |
RF | 79.66 | 79.85 | 79.82 | 79.68 |
Type of Noise | Accuracy | Precision | Recall | F1-Score |
---|---|---|---|---|
Cargo | 98.90 | 99.32 | 97.35 | 98.33 |
Passenger Ship | 98.98 | 100 | 99.49 | |
Tanker | 98.31 | 100 | 99.15 | |
Tug | 98.96 | 98.29 | 98.63 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Liu, S.; Fu, X.; Xu, H.; Zhang, J.; Zhang, A.; Zhou, Q.; Zhang, H. A Fine-Grained Ship-Radiated Noise Recognition System Using Deep Hybrid Neural Networks with Multi-Scale Features. Remote Sens. 2023, 15, 2068. https://doi.org/10.3390/rs15082068
Liu S, Fu X, Xu H, Zhang J, Zhang A, Zhou Q, Zhang H. A Fine-Grained Ship-Radiated Noise Recognition System Using Deep Hybrid Neural Networks with Multi-Scale Features. Remote Sensing. 2023; 15(8):2068. https://doi.org/10.3390/rs15082068
Chicago/Turabian StyleLiu, Shuai, Xiaomei Fu, Hong Xu, Jiali Zhang, Anmin Zhang, Qingji Zhou, and Hao Zhang. 2023. "A Fine-Grained Ship-Radiated Noise Recognition System Using Deep Hybrid Neural Networks with Multi-Scale Features" Remote Sensing 15, no. 8: 2068. https://doi.org/10.3390/rs15082068