Towards accurate and high-speed spiking neuromorphic systems with data quantization-aware deep networks
Authors: 
Fuqiang Liu, Chenchen LiuAuthors Info & Claims
Fuqiang Liu, Chenchen LiuAuthors Info & ClaimsArticle No.: 104, Pages 1 - 6
Abstract
Deep Neural Networks (DNNs) have gained immense success in cognitive applications and greatly pushed today's artificial intelligence forward. The biggest challenge in executing DNNs is their extremely data-extensive computations. The computing efficiency in speed and energy is constrained when traditional computing platforms are employed in such computational hungry executions. Spiking neuromorphic computing (SNC) has been widely investigated in deep networks implementation own to their high efficiency in computation and communication. However, weights and signals of DNNs are required to be quantized when deploying the DNNs on the SNC, which results in unacceptable accuracy loss. Previous works mainly focus on weights discretize while inter-layer signals are mainly neglected. In this work, we propose to represent DNNs with fixed integer inter-layer signals and fixed-point weights while holding good accuracy. We implement the proposed DNNs on the memristor-based SNC system as a deployment example. With 4-bit data representation, our results show that the accuracy loss can be controlled within 0.02% (2.3%) on MNIST (CIFAR-10). Compared with the 8-bit dynamic fixed-point DNNs, our system can achieve more than 9.8× speedup, 89.1% energy saving, and 30% area saving.
References
[1]
A. Krizhevsky et al., "Imagenet classification with deep convolutional neural networks," in NIPS, pp. 1097--1105, 2012.
[2]
K. Simonyan et al., "Very deep convolutional networks for large-scale image recognition," in ICLR, 2015.
[3]
K. He et al., "Deep residual learning for image recognition," in CVPR, pp. 770--778, 2016.
[4]
Girshick et al., "Rich feature hierarchies for accurate object detection and semantic segmentation," in CVPR, pp. 580--587, 2014.
[5]
S. Ren et al., "Faster R-CNN: Towards real-time object detection with region proposal networks," in NIPS, pp. 1137--1149, 2015.
[6]
D. Li et al., "Recent advances in deep learning for speech research at microsoft," in ICASSP, pp. 8604--8608, 2013.
[7]
M. Rastegari et al., "Xnor-net: Imagenet classification using binary convolutional neural networks," in ECCV, 2016.
[8]
P. Chi et al., "Prime: A novel processing-in-memory architecture for neural network computation in reram-based main memory," in ISCA, pp. 27--39, 2016.
[9]
E. S. K. et al., "Convolutional networks for fast, energy efficient neuromorphic computing," in PNAS, 2016.
[10]
S. Han et al., "EIE: efficient inference engine on compressed deep neural network," in ISCA, pp. 243--254, 2016.
[11]
P. Merolla et al., "A digital neurosynaptic core using embedded crossbar memory with 45pj per spike in 45nm," in CICC, pp. 1--4, 2011.
[12]
C. Liu et al., "A spiking neuromorphic design with resistive crossbar," in DAC, pp. 1--6, 2015.
[13]
S. A. G. et al., "Effective calculations on neuromorphic hardware based on spiking neural network approaches," in Lobachevskii Journal of Mathematics, pp. 964--996, 2017.
[14]
A. Aayush et al., "Resparc: A reconfigurable and energy-efficient architecture with memristive crossbars for deep spiking neural networks," in DAC, pp. 1--6, 2017.
[15]
Merolla et al., "A million spiking-neuron integrated circuit with a scalable communication network and interface," Science, pp. 668--673, 2014.
[16]
C. Liu et al., "Rescuing memristor-based neuromorphic design with high defects," in DAC, pp. 1--6, 2017.
[17]
Y. Wang et al., "Classification accuracy improvement for neuromorphic computing systems with one-level precision synapses," in ASP-DAC, pp. 776--781, 2017.
[18]
Hubara et al., "Binarized neural networks," in NIPS, pp. 4107--4115, 2016.
[19]
O. Russakovsky et al., "Imagenet large scale visual recognition challenge," in IJCV, pp. 211--252, 2015.
[20]
L. S. et al., "Pipelayer: A pipelined reram-based accelerator for deep learning," in HPCA, pp. 541--552, 2017.
[21]
D. D. Lin et al., "Fixed point quantization of deep convolutional networks," in ICML, pp. 2849--2858, 2016.
[22]
H. Song et al., "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding," in ICLR, 2016.
[23]
P. Gysel et al., "Hardware-oriented approximation of convolutional neural networks," in ICLR, 2016.
[24]
T. Hokchhay et al., "Hardware-software codesign of accurate, multiplier-free deep neural networks," in DAC, pp. 1--6, 2017.
[25]
T. W. Lee et al., "Memristor resistance modulation for analog applications," EDL, pp. 1456--1458, 2012.
[26]
H. Jo et al., "Nanoscale memristor device as synapse in neuromorphic systems," in Nano letters, pp. 1297--1301, 2010.
[27]
G. S. S., "Spike-timing-dependent learning in memristive nanodevices," in NANOARCH, pp. 85--92, 2008.
[28]
Y. Wang et al., "Group scissor: Scaling neuromorphic computing design to large neural networks," in DAC, pp. 1--6, 2017.
Recommendations
Towards Accurate and High-Speed Spiking Neuromorphic Systems with Data Quantization-Aware Deep Networks
2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC)Deep Neural Networks (DNNs) have gained immense success in cognitive applications and greatly pushed today’s artificial intelligence forward. The biggest challenge in executing DNNs is their extremely data-extensive computations. The computing ...
An adaptive threshold mechanism for accurate and efficient deep spiking convolutional neural networks
AbstractSpiking neural networks(SNNs) can potentially offer an efficient way of performing inference because the neurons in the networks are sparsely activated and computations are event-driven. SNNs with higher accuracy can be obtained by ...
Deep residual learning in spiking neural networks
NIPS '21: Proceedings of the 35th International Conference on Neural Information Processing SystemsDeep Spiking Neural Networks (SNNs) present optimization difficulties for gradient-based approaches due to discrete binary activation and complex spatial-temporal dynamics. Considering the huge success of ResNet in deep learning, it would be natural to ...
Comments
Information & Contributors
Information
Published In
June 2018
1089 pages
ISBN:9781450357005
DOI:10.1145/3195970
Copyright © 2018 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
- EDAC: Electronic Design Automation Consortium
- SIGDA: ACM Special Interest Group on Design Automation
- IEEE Council on Electronic Design Automation (CEDA)
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 24 June 2018
Check for updates
Qualifiers
- Research-article
Conference
DAC '18
Sponsor:
- EDAC
- SIGDA
DAC '18: The 55th Annual Design Automation Conference 2018
June 24 - 29, 2018
California, San Francisco
Acceptance Rates
Overall Acceptance Rate 1,770 of 5,499 submissions, 32%
Upcoming Conference
DAC '25
- Sponsor:
- sigda
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 225Total Downloads
- Downloads (Last 12 months)12
- Downloads (Last 6 weeks)0
Reflects downloads up to 09 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in