Nearest Unitary and Toeplitz matrix techniques for adaptation of Deep Learning models in photonic FPGA
DOI:
https://doi.org/10.7557/18.6825Keywords:
deep learning, unitary , photonics, toeplitz, neural networks, quantum computingAbstract
Photonic circuits pave the way to extremely quick computation and real-time inference in critical applications, such as imaging flow cytometry (IFC). Nevertheless, current photonic FPGA implementations display intrinsic limitations that restrict the complexity of Deep Learning (DL) models that could be sustained. One of these restrictions implies the weight matrices to be unitary. Thus, machine learning mechanisms to transform weight matrices to their nearest unitary one, are essential for the effective deployment of such demanding tasks. Furthermore, DL models that perform convolutions, require special handling so as to fit in the photonic system. In this work, several methods have been investigated for conversion of non-unitary matrices to unitary ones, as well as, linear algebra techniques for the transformation of Convolutional Neural Networks (CNNs) to Feed-Forward models, under the prism of discovery of the best candidate for the photonic FPGA in terms of accuracy and restrictions. Experimental results proved that post-training or iterative techniques to find the nearest unitary weight matrix can be applied for photonic chips with the minimum loss in accuracy, while CNNs adapted well in a photonic configuration employing a Toeplitz matrix implementation. The proposed approach envisions efficient tackling of DL models limitations for deployment in photonic FPGAs.
References
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Man´e, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Vi´egas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https://www.tensorflow.org/. Software available from tensorflow.org.
M. Arjovsky, A. Shah, and Y. Bengio. Unitary evolution recurrent neural networks. In International conference on machine learning, pages 1120–1128. PMLR, 2016. doi: 10.48550/arXiv.1511.06464.
H. Bagherian, S. Skirlo, Y. Shen, H. Meng, V. Ceperic, and M. Soljacic. On-chip optical convolutional neural networks. arXiv preprint arXiv:1808.03303, 2018. doi: 10.48550/arXiv.1808.03303.
B. Bai, H. Shu, X. Wang, and W. Zou. Towards silicon photonic neural networks for artificial intelligence. Science China Information Sciences, 63(6):1–14, 2020. doi: 10.1007/ s11432-020-2872-3.
A. Baldominos, Y. Saez, and P. Isasi. A survey of handwritten character recognition with mnist and emnist. Applied Sciences, 9(15): 3169, 2019. doi: 10.3390/app9153169.
J. R. Basani, M. Heuck, D. R. Englund, and S. Krastanov. All-photonic artificial neural network processor via non-linear optics. arXiv preprint arXiv:2205.08608, 2022. doi: doi.org/ 10.48550/arXiv.2205.08608.
D. Brunner, B. Penkovsky, B. A. Marquez,M. Jacquot, I. Fischer, and L. Larger. Tutorial: Photonic neural networks in delay systems. Journal of Applied Physics, 124(15): 152004, 2018. doi: 10.1063/1.5042342.
J. Cheng, H. Zhou, and J. Dong. Photonic matrix computing: from fundamentals to applications. Nanomaterials, 11(7):1683, 2021. doi: 10.3390/nano11071683.
D. Dang, S. V. R. Chittamuru, S. Pasricha, R. N. Mahapatra, and D. Sahoo. Bplight-cnn: A photonics-based backpropagation accelerator for deep learning. CoRR, abs/2102.10140, 2021. doi: 10.1145/3446212. URL https://arxiv.org/abs/2102.10140.
M. Doan, I. Vorobjev, P. Rees, A. Filby, O. Wolkenhauer, A. E. Goldfeld, J. Lieberman, N. Barteneva, A. E. Carpenter, and H. Hennig. Diagnostic potential of imaging flow cytometry. Trends in biotechnology, 36 (7):649–652, 2018. doi: 10.1016/2017.12.008.
S. K. Esser, R. Appuswamy, P. Merolla, J. V. Arthur, and D. S. Modha. Backpropagation for energy-efficient neuromorphic computing. Advances in neural information processing systems, 28, 2015.
P. Eulenberg, N. K¨ohler, T. Blasi, A. Filby, A. E. Carpenter, P. Rees, F. J. Theis, and F. A. Wolf. Reconstructing cell cycle and disease progression using deep learning. Nature communications, 8(1):1–6, 2017. doi: 10.1038/s41467-017-00623-3.
X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315–323. JMLR Workshop and Conference Proceedings, 2011.
I. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press, 2016.
B. K. Horn, H. M. Hilden, and S. Negahdaripour. Closed-form solution of absolute orientation using orthonormal matrices. JOSA A, 5(7):1127–1135, 1988. doi: 10.1364/JOSAA.5.001127.
S. K. Kumar. On weight initialization in deep neural networks. arXiv preprint arXiv:1704.08863, 2017. doi: 10.48550/arXiv.1704.08863.7
H. Li, B. Wu, W. Tong, J. Dong, and X. Zhang. All-optical nonlinear activation function based on germanium silicon hybrid asymmetric coupler. IEEE Journal of Selected Topics in Quantum Electronics, 2022. doi: 10.1109/JSTQE.2022.3166510.
S. Lloyd and R. Maity. Efficient implementation of unitary transformations. arXiv preprint arXiv:1901.03431, 2019. doi: 10.48550/arXiv.1901.03431.
D. P. L´opez. Programmable integrated silicon photonics waveguide meshes: optimized designs and control algorithms. IEEE Journal of Selected Topics in Quantum Electronics, 26(2):1–12, 2019. doi: 10.1109/JSTQE.2019.2948048.
A. Macho-Ortiz, D. P´erez-L´opez, and J. Capmany. Optical implementation of 2×2 universal unitary matrix transformations. Laser & Photonics Reviews, 15(7):2000473, 2021. doi: 10.1002/lpor.202000473.
L. S. Madsen, F. Laudenbach, M. F. Askarani, F. Rortais, T. Vincent, J. F. Bulmer, F. M. Miatto, L. Neuhaus, L. G. Helt, M. J. Collins, et al. Quantum computational advantage with a programmable photonic processor. Nature, 606(7912):75–81, 2022. doi: 10.1038/s41586-022-04725-x.
M. Miscuglio, A. Mehrabian, Z. Hu, S. I. Azzam, J. George, A. V. Kildishev, M. Pelton, and V. J. Sorger. All-optical nonlinear activation function for photonic neural networks. Optical Materials Express, 8(12):3851–3863, 2018. doi: 10.1364/OME.8.003851.
M. V. Narkhede, P. P. Bartakke, and M. S. Sutaone. A review on weight initialization strategies for neural networks. Artificial intelligence review, 55(1):291–322, 2022. doi:10.1007/s10462-021-10033-z.
N. Passalis, G. Mourgias-Alexandris, A. Tsakyridis, N. Pleros, and A. Tefas.
Training deep photonic convolutional neural networks with sinusoidal activations. IEEE Transactions on Emerging Topics in Computational Intelligence, 5(3):384–393, 2019. doi: 10.1109/TETCI.2019.2923001.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´e-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019. doi:10.48550/arXiv.1912.01703.
F. Pavosevic and J. Flick. Polaritonic unitary coupled cluster for quantum computations. The Journal of Physical Chemistry Letters, 12(37):9100–9107, 2021. doi: 10.1021/acs.jpclett.1c02659.
S. Pitris, C. Mitsolidou, T. Alexoudi, D. P´erez-Galacho, L. Vivien, C. Baudot, P. De Heyn, J. Van Campenhout, D. Marris-Morini, and N. Pleros. O-band energy-efficient broadcastfriendly interconnection scheme with sipho mach-zehnder modulator (mzm) & arrayed waveguide grating router (awgr). In 2018 Optical Fiber Communications Conference and Exposition (OFC), pages 1–3. IEEE, 2018.
M. Reck, A. Zeilinger, H. J. Bernstein, and P. Bertani. Experimental realization of any discrete unitary operator. Physical review letters, 73(1):58, 1994. doi: 10.1103/PhysRevLett.73.58.
A. Salehi. My Research Software. 2022. doi: 10.5281/zenodo.1234. URL https://github.com/alisaaalehi/convolution_as_multiplication.
B. Shashni, S. Ariyasu, R. Takeda, T. Suzuki, S. Shiina, K. Akimoto, T. Maeda, N. Aikawa, R. Abe, T. Osaki, et al. Size-based differentiation of cancer and normal cells by a particle size analyzer assisted by a cell-recognition pc software. Biological and Pharmaceutical Bulletin, 41(4):487–503, 2018. doi: 10.1248/bpb.b17-00776.
Y. Shen, N. C. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, et al. Deep learning with coherent nanophotonic circuits. Nature Photonics, 11(7):441–446, 2017. doi: 10.1038/nphoton.2017.93.
A. N. Tait, T. F. De Lima, E. Zhou, A. X. Wu, M. A. Nahmias, B. J. Shastri, and P. R. Prucnal. Neuromorphic photonic networks using silicon photonic weight banks. Scientific reports, 7(1):1–10, 2017. doi: doi.org/10.1038/s41598-017-07754-z.
K. Vandoorne, P. Mechet, T. Van Vaerenbergh, M. Fiers, G. Morthier, D. Verstraeten, B. Schrauwen, J. Dambre, and P. Bienstman. Experimental demonstration of reservoir computing on a silicon photonics chip. Nature communications, 5(1):1–6, 2014. doi: 10.1038/ncomms4541.
K. Wood, G. Bianchin, and E. Dall’Anese. Online projected gradient descent for stochastic optimization with decision-dependent distributions. IEEE Control Systems Letters, 6: 1646–1651, 2021. doi: 10.1109/LCSYS.2021.3124187.
X. Xu, M. Tan, B. Corcoran, J. Wu, A. Boes, T. G. Nguyen, S. T. Chu, B. E. Little, D. G. Hicks, R. Morandotti, et al. 11 tops photonic convolutional accelerator for optical neural networks. Nature, 589(7840):44–51, 2021.doi: 10.1038/s41586-020-03063-0.
C. Yakopcic, R. Hasan, and T. M. Taha. Memristor based neuromorphic circuit for ex-situ training of multi-layer neural network algorithms. In 2015 International Joint Conference on Neural Networks (IJCNN), pages 1–7. IEEE, 2015. doi: 10.1109/IJCNN.2015. 7280813.
G.-Y. Yang, X.-L. Li, R. R. Martin, and S.-M. Hu. Sampling equivariant self-attention networks for object detection in aerial images. arXiv preprint arXiv:2111.03420, 2021. doi: 10.48550/arXiv.2111.03420.
C. Zhang, D. Wu, J. Sun, G. Sun, G. Luo, and J. Cong. Energy-efficient cnn implementation on a deeply pipelined fpga cluster:. pages 326–331, 08 2016. doi: 10.1145/2934583.2934644.
J. Zhu and P. Sutton. Fpga implementations of neural networks–a survey of a decade of progress. In International conference on field programmable logic and applications, pages 1062–1066. Springer, 2003. doi: 10.1007/978-3-540-45234-8_120.
Downloads
Published
Issue
Section
License
Copyright (c) 2023 Georgios Agrafiotis, Eftychia Makri, Ilias Kalamaras, Antonios Lalas, Konstantinos Votis, Dimitrios Tzovaras
This work is licensed under a Creative Commons Attribution 4.0 International License.
