CN114241245B - Image classification system based on residual capsule neural network - Google Patents
Image classification system based on residual capsule neural network Download PDFInfo
- Publication number
- CN114241245B CN114241245B CN202111587100.5A CN202111587100A CN114241245B CN 114241245 B CN114241245 B CN 114241245B CN 202111587100 A CN202111587100 A CN 202111587100A CN 114241245 B CN114241245 B CN 114241245B
- Authority
- CN
- China
- Prior art keywords
- capsule
- residual
- layer
- network
- convolution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000002775 capsule Substances 0.000 title claims abstract description 453
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 47
- 238000004364 calculation method Methods 0.000 claims abstract description 28
- 239000013598 vector Substances 0.000 claims description 50
- 238000003491 array Methods 0.000 claims description 38
- 238000010586 diagram Methods 0.000 claims description 18
- 230000004913 activation Effects 0.000 claims description 12
- 239000011159 matrix material Substances 0.000 claims description 12
- 238000012360 testing method Methods 0.000 claims description 12
- 230000009466 transformation Effects 0.000 claims description 9
- 238000013461 design Methods 0.000 claims description 8
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 238000012938 design process Methods 0.000 claims description 3
- 238000002789 length control Methods 0.000 claims description 3
- 239000013589 supplement Substances 0.000 claims description 3
- 230000017105 transposition Effects 0.000 claims description 3
- 238000001125 extrusion Methods 0.000 claims description 2
- 230000009469 supplementation Effects 0.000 claims description 2
- 230000008034 disappearance Effects 0.000 abstract description 3
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000013527 convolutional neural network Methods 0.000 description 19
- 238000000034 method Methods 0.000 description 17
- 238000012549 training Methods 0.000 description 15
- 238000002474 experimental method Methods 0.000 description 13
- 230000008569 process Effects 0.000 description 10
- 230000006870 function Effects 0.000 description 9
- 238000011160 research Methods 0.000 description 6
- 210000002569 neuron Anatomy 0.000 description 5
- 238000013507 mapping Methods 0.000 description 3
- 235000009854 Cucurbita moschata Nutrition 0.000 description 2
- 240000001980 Cucurbita pepo Species 0.000 description 2
- 235000009852 Cucurbita pepo Nutrition 0.000 description 2
- 238000002679 ablation Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 210000004556 brain Anatomy 0.000 description 2
- 210000004027 cell Anatomy 0.000 description 2
- 230000007423 decrease Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 235000019580 granularity Nutrition 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 235000020354 squash Nutrition 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 1
- 206010058467 Lung neoplasm malignant Diseases 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 201000005202 lung cancer Diseases 0.000 description 1
- 208000020816 lung neoplasm Diseases 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000000653 nervous system Anatomy 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000000513 principal component analysis Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000005549 size reduction Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 210000000225 synapse Anatomy 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to the technical field of image classification, and particularly discloses an image classification system based on a residual capsule neural network, wherein the system is provided with the residual capsule neural network, and the network comprises a first residual capsule module, a second residual capsule module, a third residual capsule module and a digital capsule layer; the first residual capsule module comprises a first residual convolution sub-network and a first initial capsule layer, the second residual capsule module comprises a second residual convolution sub-network and a second initial capsule layer, and the third residual capsule module comprises a third residual convolution sub-network and a third initial capsule layer. The invention realizes multi-stage information multiplexing by a mode of connecting multi-stage residual capsule modules in series based on the thought of residual learning, solves the problem of gradient disappearance after network deepening, introduces cavity convolution, and is beneficial to extracting more features; the transposed convolution is used for constructing a reconstruction network, so that the network performance is improved while the network parameter number is reduced; hardware deployment of the capsule network is realized based on the memristor cross array, so that the problem of large calculation amount is solved.
Description
Technical Field
The invention relates to the technical field of image classification, in particular to an image classification system based on a residual capsule neural network.
Background
Computer vision based on convolutional neural networks (Convolutional neural network, CNN) has evolved rapidly in recent years, and the impact of this sudden surge has spread to almost all industries and has been applied in the application fields of assisted driving, image processing, object recognition, etc. With the deep research of convolutional neural networks, more and more CNN new structures are proposed, so that the performance of the CNN new structures is also improved obviously. However, there are still some drawbacks to the CNN at this stage, firstly, the CNN is not robust to affine transformation of the object, i.e. slight rotation and scaling of the object may lead to erroneous results. While the occurrence of such problems can be reduced by using data enhancement methods during the model training phase, this does not make the network robust to any unknown new changes. Second, CNN uses regularly sliding receptive fields to learn the features of the picture, which determines that it will only make decisions based on local features in the input image, and does not take into account the relative relationship between the features. In addition, in order to reduce the parameters of the depth CNN, a pooling operation is introduced in the CNN, which is sensitive only to important features in the image, so this disadvantage of the CNN is exacerbated.
A new Network architecture called Capsule Network (CapsNet) in which data is transformed from scalar to vector form in a simple neural Network, these neurons consisting of multidimensional vectors are called capsules, overcomes the above-mentioned drawbacks. In order to coordinate the transmission and operation of data in capsule neurons among layers of a network, a routing protocol among capsules is proposed, and experiments prove that CapsNet has excellent performance on MNIST data sets. The experimental capsule network also proves that the method can resist black box attacks and white box attacks better than a convolutional network, and has stronger robustness. However, the baseline capsule network model has a simple structure, so the performance under a complex background data set is not very good, and in recent years, with the continuous and deep research on the capsule network, a plurality of researchers optimize and improve the capsule network in terms of network structure, routing algorithm and the like. For example, in order to solve the problem of poor performance caused by the simple CapsNet structure, researchers replace the convolution layer of the capsule network with three densely connected sub-networks, so that the extraction capability of the convolution layer on the image is enhanced, the iteration times of the new network are reduced, and good results are obtained on the problem of image classification; researchers deepen the capsule network, propose the concept of a deep capsule network, and put forward a dynamic routing method suitable for 3D convolution to reduce network parameters, optimize the network from the dimension of the capsule; researchers have improved on capsule routing algorithms, have proposed a dynamic routing algorithm based on the attention mechanism, and have implemented similar activation functions as Relu in the capsule network. Meanwhile, researchers have expanded the application scene of the capsule network, for example, have proposed a capsule network suitable for optical remote sensing image processing, can process the picture of different visual angles, obtain the satisfactory performance under the smaller training set; researchers have proposed a self-care capsule network (SACN) that can significantly improve the performance of neural networks on medical datasets; capsule networks have been used by researchers in lung cancer screening in the medical field and demonstrated that capsule networks perform better than CNN under small data sets. Thus, research on end-side deployment schemes for high-performance capsule networks has positive implications for pushing capsule networks into application in realistic scenarios.
Compared with CNN, the capsule network has larger parameters and calculation amount, but almost all deep learning algorithms are realized by a von Neumann structure-based computer at present, the von Neumann structure computer at present has the bottleneck that a calculation unit and a storage unit are mutually separated, and in the running process of the computer, data transmission between the calculation unit and the storage unit needs to consume great power and bring a certain time delay, and high-efficiency and low-power consumption real-time information processing similar to biological brain is difficult to realize, so that a deployment scheme of the capsule network on the end side needs to be further studied.
The capsule network has natural white-box attack resistance, is a potential emerging research direction in the field of deep learning, but a baseline capsule network model has poor performance under a complex background data set, and the network has large calculation amount and is unfavorable for deployment on end-side equipment.
Disclosure of Invention
The invention provides an image classification system based on a residual capsule neural network, which solves the technical problems that: how to realize high-precision image classification based on a capsule network and how to further realize high efficiency and low power consumption of image classification.
In order to solve the technical problems, the invention provides an image classification system based on a residual capsule neural network, which is provided with the residual capsule neural network; the residual capsule neural network comprises a first residual capsule module, a second residual capsule module, a third residual capsule module and a digital capsule layer; the first residual capsule module comprises a first residual convolution sub-network and a first initial capsule layer, the second residual capsule module comprises a second residual convolution sub-network and a second initial capsule layer, and the third residual capsule module comprises a third residual convolution sub-network and a third initial capsule layer;
The first residual convolution sub-network carries out convolution calculation on an input original image and then outputs a first group of feature images to the second residual convolution sub-network and the first initial capsule layer, the first initial capsule layer converts the first group of feature images into a first initial capsule group, and the digital capsule layer converts the first initial capsule group into a first digital capsule group;
the second residual convolution sub-network carries out convolution calculation on the input first group of feature images and then outputs a second group of feature images to the third residual convolution sub-network and the second initial capsule layer, the second initial capsule layer converts the second group of feature images into a second initial capsule group, and the digital capsule layer converts the second initial capsule group into a second digital capsule group;
The third residual convolution sub-network carries out convolution calculation on the input second group of feature images and then outputs a third group of feature images to the third initial capsule layer, the third initial capsule layer converts the third group of feature images into a third initial capsule group, and the digital capsule layer converts the third initial capsule group into a third digital capsule group;
the digital capsule layer splices the first initial capsule group, the second initial capsule group and the third initial capsule group to obtain a fourth initial capsule group and converts the fourth initial capsule group into a fourth digital capsule group;
and the digital capsule layer is used for outputting M new digital capsules for target classification and image reconstruction after splicing and fusing the four groups of digital capsules in the digital capsule layer, wherein M is equal to the total category number of image classification.
Preferably, the residual capsule neural network further comprises a reconstruction module, wherein the reconstruction module is used for generating a reconstructed image with the same size as the original image according to M new digital capsules.
Preferably, the loss function of the residual capsule neural network is:
Wherein L total represents total loss ;Lk＝Tkmax(0,m+-||vk||)2+λ(1-Tk)max(0,||vk||-m-)2 represents edge loss of capsule k in the last capsule layer, T k represents matching parameters of a kth class classification target corresponding to capsule k, T k =1 if and only if classification is correct, otherwise T k＝0;vk represents activation vector of capsule k, m +、m-, λ are capsule vector modular length control parameters, m +＝0.9,m-＝0.1,λ＝0.5;Lr represents reconstruction loss, which is equal to mean square error between pixels of reconstructed image and original image; η represents a weight coefficient of the reconstruction loss at the total loss.
Preferably, the first residual convolution sub-network, the second residual convolution sub-network and the third residual convolution sub-network each comprise two convolution layers and a jump connection layer, and each convolution layer adopts the same convolution with the introduced hole convolution; the digital capsule layer converts the initial capsule into a digital capsule through a dynamic routing algorithm and an extrusion activation function.
Preferably, in a convolution layer of the first residual convolution sub-network, the second residual convolution sub-network, or the third residual convolution sub-network, for a matrix-vector multiplication operation having h inputs and o outputs, the matrix-vector multiplication operation is represented by the following formula:
In the formula (8), x n represents an nth element in the input vector, w mn represents a weight of an mth row and an nth column in the weight matrix, and y m represents an mth output in the output vector;
In the hardware design corresponding to the convolution layer, matrix-vector multiplication operation with h inputs and o outputs is realized by adopting a first 2D memristor cross array;
The first 2D memristor cross array comprises h rows, o columns and columns of memristors, x 1～xh is converted into row input voltage V x1～Vxh of the first 2D memristor cross array, a weight w mn is mapped into conductance values of m rows and n columns of memristors in the first 2D memristor cross array, input voltage V xn is multiplied by the conductance value w mn of the corresponding memristor, output current flowing through each memristor is superposed through a wire to obtain column output current, and the output current of each column is converted into voltage V ym through a current-voltage conversion circuit and then output.
Preferably, for any one of the first initial capsule layer, the second initial capsule layer and the third initial capsule layer, the initial capsule layer has R capsule units, each capsule unit includes X convolution units, the convolution kernel size is u×u, and on the corresponding hardware design, the operation of the initial capsule layer is implemented by adopting a first 3D memristive cross array;
The first 3D memristor cross array comprises T layers of second 2D memristor cross arrays with similar structures as the first 2D memristor cross arrays, T layers of second 2D memristor cross arrays correspond to T input feature maps, the memristor specification of each layer of second 2D memristor cross arrays is (R X) column (U X U) row, each layer of second 2D memristor cross arrays operate independently, the outputs of different layers of second 2D memristor cross arrays are connected in the same column, the output voltage of each layer of second 2D memristor cross arrays is summed with the outputs of other layers in the direction of the vertical column, and the T feature maps are converted into one-dimensional voltage output through the first 3D memristor cross arrays;
after a voltage sequence, the first 3D memristive crossover array outputs R X one-dimensional voltage signals representing an X-dimensional vector of R capsule units; after all voltage sequences are passed through by using serial input mode, the initial capsule layer outputs Y X-dimensional vectors, i.e. Y X R initial capsules, Y X R represents the size of output characteristic diagram.
Preferably, the digital capsule layer has four sublayers for converting four sets of initial capsules into corresponding four sets of digital capsules; any sub-layer comprises M capsule units, Y X dimension vectors output by the initial capsule layer and M capsule units of the sub-layer share Y X weight transformation matrixes, and the weight transformation matrixes are mapped to Y X independent second 3D memristor cross arrays for parallel operation in the hardware design process;
Each second 3D memristor cross array comprises M layers of third 2D memristor cross arrays similar to the first 2D memristor cross arrays in structure, the memristor specification of each layer of third 2D memristor cross arrays is V rows by W columns, V rows correspond to the input of V-order tensors, W columns correspond to the output of W-order tensors, and finally the sub-layers output M W-order tensors, namely M digital capsules with W dimensions.
Preferably, the dimensions of the digital capsules output by the four sub-layers of the digital capsule layer are W 1、W2、W3、W4 respectively, and after the digital capsule layer splices the four groups of digital capsules, M new digital capsules with dimensions of W 1+W2+W3+W4 are obtained for target classification and image reconstruction.
Preferably, the reconstruction module comprises a full connection layer and four transposed convolution layers, wherein each transposed convolution layer firstly complements 0 on an input feature map according to rules, and then carries out convolution operation on the feature map after the complementation of 0;
Defining the size of an input feature diagram of a transposed convolution layer as h×h×n, the convolution step size as S, and the size of each channel of the input feature diagram after 0 is complemented as H out*Hout, wherein the calculation method is as follows:
Hout＝H+(H-1)×(S-1)+2P+O (11)
The filling rule is as follows: s-1 0 is inserted between every two pixels of the input feature map, then P circles of 0 values are supplemented around the feature map, finally O rows of 0 values are supplemented below the feature map, and O columns of 0 values are supplemented to the right of the feature map.
Preferably, the dataset used for training and testing the residual capsule neural network comprises more than ten thousand m=10 gray-scale images or RGB color images derived from MNIST, cifar-10 or SVHN datasets.
The image classification system based on the residual capsule neural network has the beneficial effects that:
1. Based on the thought of residual learning, multi-level information multiplexing is realized by connecting multi-level residual capsule modules in series, the problem of gradient disappearance after network deepening is solved, and cavity convolution is introduced into the residual capsule modules of higher layers, so that receptive fields are enlarged, more characteristics are extracted by a capsule network, and classification precision is improved;
2. the reconstruction network constructed by using the transposed convolution is provided, so that the network performance is improved while the network parameter is reduced;
3. The high-efficiency and low-power-consumption hardware deployment scheme for the capsule network for image classification solves the problem of large calculation amount of the residual capsule neural network by utilizing the advantages of low power consumption and support of parallel matrix calculation of the memristor cross array.
Drawings
FIG. 1 is a block diagram of a baseline capsule network model provided by an embodiment of the present invention;
fig. 2 is a block diagram of an image classification system based on a residual capsule neural network according to an embodiment of the present invention;
Fig. 3 is a block diagram of a residual capsule module 1 in a residual capsule neural network according to an embodiment of the present invention;
FIG. 4 is a schematic representation of hole convolution provided by an embodiment of the present invention;
FIG. 5 is a block diagram of a transpose convolution based reconstruction network provided by an embodiment of the present invention;
FIG. 6 is a block diagram of a first 2D memristor crossbar array provided by an embodiment of the present disclosure;
FIG. 7 is a block diagram of a first 3D memristor crossbar array provided by an embodiment of the present disclosure;
FIG. 8 is a block diagram of a second 3D memristor crossbar array provided by an embodiment of the present disclosure;
FIG. 9 is a graph comparing losses of MRCAPSNET residual capsule modules provided by an embodiment of the present invention;
fig. 10 is a graph comparing MRCAPSNET network reconstruction errors with CapsNet network reconstruction errors under CIFAR-10 data sets provided by an embodiment of the present invention.
Detailed Description
The following examples are given for the purpose of illustration only and are not to be construed as limiting the invention, including the drawings for reference and description only, and are not to be construed as limiting the scope of the invention as many variations thereof are possible without departing from the spirit and scope of the invention.
In the capsule neural network, a capsule is a basic element forming the network, and is composed of a group of vector neurons, and the input and the output of the capsule neural network are vectors. Wherein the modular length of the vector representing the capsule represents the probability of the classified entity, and each element in the vector represents the characteristic information of the entity. Similar to CNN, capsule networks can only detect lower levels of attributes when the capsule network's receptive field is small, while CapsNet can obtain more and more complex characteristic attributes when the receptive field is large. Figure 1 shows a simple structure of CapsNet (baseline capsule network model) consisting of four parts, a convolutional layer, an initial capsule (PRIMARYCAPS) layer, a digital capsule (DIGITCAPS) layer, and a fully-connected layer. The data flows in the form of vectors in the capsule layer according to the dynamic routing algorithm rules. During training, each low-level capsule outputs a prediction vector to the next-level capsule, and then by comparing actual labels, if the prediction structure and the actual labels match, the coupling coefficient between the two capsules increases.
In the capsule dynamic routing algorithm, u i is defined to represent the activation vector of capsule i located in the lower capsule layer, u j|i is defined to represent the prediction vector of capsule i in the lower capsule layer for capsule j in the next capsule layer, then:
ui|j＝Wijui (1)
where W ij denotes a weight transform matrix for matrix transforms between parts and whole. Also:
bij＝bij+uj|i·vj (4)
c ij represents the weight parameters of the capsule i in the lower capsule layer (containing m capsules) for the capsule j in the next capsule layer, calculated from the softmax function; the input s j of capsule j is derived from the weighted summation of c ij and u j|i, which is passed through a nonlinear activation function squash to ensure that the vector length is between 0, 1), for representing the probability of existence; v j denotes the activation vector of capsule j located in the upper capsule layer; b ij denotes the degree of matching between capsule i and capsule j, j=1, 2,3, …, m. At the first iteration, the value of b ij is initialized to 0, so the probability weight for c ij being equal is calculated. The values of all parameters are then updated iteratively by the above formula. After 3 iterations, the weight distribution values tend to converge.
For capsule k of the last layer of the capsule network, edge loss L k is calculated by the following formula:
Lk＝Tkmax(0,m+-||vk||)2+λ(1-Tk)max(0,||vk||-m-)2 (6)
In formula (6), v k represents the activation vector of capsule k; t k denotes the matching parameters of the kth class classification target corresponding to capsule k, if and only if the classification is correct, T k = 1, otherwise T k＝0,vk denotes the activation vector of capsule k, m +、m-, λ are both capsule vector modular length control parameters, m +＝0.9,m- = 0.1, λ = 0.5.λ is set to 0.5, so that the length of all capsules can be prevented from being reduced due to erroneous classification results.
In order to achieve high-precision image classification based on a capsule network and enable the classification network to have good expressive force under a complex background data set, the embodiment provides an image classification system based on a residual capsule neural network, wherein the system is provided with the residual capsule neural network, and the residual capsule neural network is also called as a memristor-based residual capsule network (MEMRISTIVE RESIDUAL CapsNet, MRCapsNet) because the residual capsule neural network is realized based on memristors on the hardware design described below, and the model structure of the residual capsule neural network is as shown in fig. 1 and comprises a first residual capsule module (residual capsule module 1), a second residual capsule module (residual capsule module 2), a third residual capsule module (residual capsule module 3), a digital capsule layer (Digit capsule layer) and a reconstruction module (reconstruction network); the first residual capsule module comprises a first residual convolution sub-network and a first initial capsule layer, the second residual capsule module comprises a second residual convolution sub-network and a second initial capsule layer, and the third residual capsule module comprises a third residual convolution sub-network and a third initial capsule layer.
The first residual convolution sub-network carries out convolution calculation on an input original image and then outputs a first group of feature images to the second residual convolution sub-network and a first initial capsule layer, the first initial capsule layer converts the first group of feature images into a first initial capsule group, and the digital capsule layer converts the first initial capsule group into a first digital capsule group.
The second residual convolution sub-network carries out convolution calculation on the input first group of feature images and then outputs a second group of feature images to the third residual convolution sub-network and a second initial capsule layer, the second initial capsule layer converts the second group of feature images into a second initial capsule group, and the digital capsule layer converts the second initial capsule group into a second digital capsule group.
The third residual convolution sub-network carries out convolution calculation on the input second group of feature images and then outputs a third group of feature images to a third initial capsule layer, the third initial capsule layer converts the third group of feature images into a third initial capsule group, and the digital capsule layer converts the third initial capsule group into a third digital capsule group.
The digital capsule layer splices the first initial capsule group, the second initial capsule group and the third initial capsule group to obtain a fourth initial capsule group and converts the fourth initial capsule group into a fourth digital capsule group.
The digital capsule layer is used for outputting M new digital capsules for target classification and image reconstruction after splicing and fusing four groups of digital capsules in the digital capsule layer, wherein M is equal to the total category number of image classification.
The reconstruction module is used for generating a reconstructed image with the same size as the original image according to the M new digital capsules.
The memristive residual capsule network has a multi-layer multiplexed neural network structure, containing 3 residual capsule modules ResCapsBlock, the output of the first stage ResCapsBlock (first residual capsule module/residual capsule module 1) is used to create a first stage Digit capsule (first digital capsule group) while also serving as input to the second stage ResCapsBlock (second residual capsule module/residual capsule module 2), similarly, the second and third stages ResCapsBlock (third residual capsule module/residual capsule module 3) are also created in the manner described above. Three serially connected residual capsule modules can realize multi-level feature reuse, and each level of residual capsule module can also output Digit capsules in parallel, wherein the three serially connected residual capsule modules contain features with different granularities learned from different levels, and the capsules can be spliced and fused in a Digit capsule layer.
Taking the picture (32×32×3) in the CIFAR-10 dataset as an example of the input image, the output feature map size of each residual capsule module under the CIFAR-10 dataset is shown in table 1.
TABLE 1
The memristive residual error capsule network takes images as input, takes capsules as units of data flow in the network, and trains the network based on multi-level feature reuse and capsule splicing, so that the network is more suitable for classification of complex images.
Taking the residual capsule module 1 as an example, the structure of the residual capsule module is shown in fig. 3, the module adopts the idea of residual learning proposed by ResNet (residual network), and the module comprises two convolution layers, a jump connection layer and an initial (Primary) capsule layer, unlike ResNet, and in order to prevent spatial information loss, the module does not comprise a pooling layer. In the residual capsule module 1, in order to increase the receptive field of the deep network, the principle of the present example is shown in fig. 4 below, the receptive field is enlarged by adding a hole in the hole convolution, the output of each convolution operation contains a larger range of information under the same parameter quantity and calculation quantity, and the hole convolution introduces a super parameter called "expansion rate", and the parameter defines the interval quantity of each point in the convolution kernel. In the residual capsule module 1, the input image (32×32×3) first passes through the first residual convolution sub-network (Res Convlotion Subnetwok), after two times of same convolution calculation, 256 Feature maps (Feature-map, 32×32×256) are output, then a primry capsule layer is input, the layer contains 8 capsule units, each capsule unit contains 12 convolution units with convolution kernel size (5*5), the Feature map (third-order tensor) is converted into Feature capsules (fourth-order tensor) through convolution and Reshape, squash operation, and after the primry capsule layer, the data of the network are transmitted and calculated in a capsule form. The residual capsule modules 2, 3 are constructed based on a principle similar to the residual capsule module 1.
The Digit capsule layer is used for identifying the classification result, and the test data sets in this example are all 10 classification problems, so the digital capsule layer outputs 10 capsule units. As shown in fig. 2, the memristive residual capsule network includes three residual capsule modules, which respectively output PRIMARYCAPS (first initial capsule group), PRIMARYCAPS (second initial capsule group), PRIMARYCAPS (third initial capsule group), and each primay capsule forms a corresponding digital capsule (Digit capsule) through a dynamic routing algorithm and a squeeze activation function, including a first digital capsule group (DIGITCAPS 1), a second digital capsule group (DIGITCAPS 2), and a third digital capsule group (DIGITCAPS 3). As the information extracted by the residual convolution sub-network per stage gradually decreases, the size of the Digit capsule also gradually decreases, namely DIGITCAPS1 (10×16d, d represents dimension), DIGITCAPS2 (10×12d), DIGITCAPS (10×10d). In addition, in order to realize information fusion with different granularity extracted by different residual convolution sub-networks, three independent initial capsules are spliced at the capsule level to form a combined initial PRIMARYCAPS (fourth initial capsule group), and the capsule is also subjected to a dynamic routing algorithm and a squeeze activation function to form a fourth digital capsule group DIGITCAPS (10×16d). Finally, the four digital capsule groups output 10 digital capsules (the fourth digital capsule group, 10×54d) for classification and image reconstruction after the Digit capsule layer completes the capsule-level splicing.
As shown in fig. 5, the reconstruction module/network consists of one fully connected layer and four transposed convolutional layers, unlike the fully connected reconstruction network in the baseline capsule network model, the MRCAPSNET reconstruction network creates more spatial relationships between features of interest when reconstructing an image. The input of the reconstruction network is the output of the Digit capsule layer, and the output is the reconstructed image with the same size as the original image.
MRCAPSNET is composed of two parts, edge loss and reconstruction loss.
The length of the vector used by the capsule network indicates the probability that the capsule exists in the classification entity, e.g., if and only if there is a class k classification object in the image, the class k capsule in the highest-level digital capsule layer ideally has the longest vector modulo length. For multi-object classification in one image, the edge loss function uses a separate margin loss L k for each object:
Lk＝Tkmax(0,m+-||vk||)2+λ(1-Tk)max(0,||vk||-m-)2 (6)
if and only if a classification target k is present in the image, T k＝1,m+＝0.9,m- = 0.1, λ = 0.5.
The present example uses a reconstruction penalty to assist in capsule network training, with the reconstruction penalty L r representing the mean square error between the pixels of the reconstructed image and the original image.
In order to balance the ratio of the edge loss and the reconstruction loss, the weight coefficient of the reconstruction loss in the total loss is eta, and eta takes 0.512 in the scheme. The total loss L total is calculated as follows:
in the experiment, the loss is calculated after the multi-layer residual capsule blocks are spliced, and the combination mode strengthens the fusion of the features learned by the deep capsule and the shallow capsule.
In the training and reasoning process of the capsule network, a large number of matrix multiplication operations are required, and the operations consume a large amount of calculation time and power consumption. The traditional von neumann architecture-based computer memory unit and the computing unit are separated from each other, and a computing bottleneck exists. In 1990, the concept of "neuromorphic computation" was first proposed, the idea being to combine analog circuitry with very large scale integrated circuitry for simulating biological nervous systems, thereby obtaining neuromorphic systems approaching the level of human brain intelligence. Memristors are commonly used in the current research of neuromorphic computing devices to simulate biological neurons and to simulate building biological synapses. The memristor is proposed by Leon Chua in 1971, and a nano memristor element is prepared by a Hewlett packard laboratory for the first time in 2008, so that a foundation is laid for the application of the subsequent memristor element. The nanoscale memristive element supports nonvolatile storage, has low power consumption, and is compatible with a CMOS system. In recent years, many researchers have proposed neuromorphic computing architectures built based on memristors. For example, researchers design a memristor-CMOS chip, can realize high-speed multiply-add operation, realize experiments such as principal component analysis, sparse coding and the like on the chip, and provide a hardware solution for low-power-consumption deployment of an algorithm at an end side; researchers construct a full hardware computational integration system of a non-von neumann architecture, which consists of a plurality of memristive cross arrays, has energy efficiency which is two orders of magnitude higher than that of a graphic processor, and obtains the accuracy equivalent to a software algorithm in an image recognition experiment, which further proves the potential of the memristive neuromorphic computing system in the aspect of side-deployment of a neural network.
Based on the memristor cross array, the scheme for realizing the capsule neural network by using memristor cross array hardware is provided, any matrix can be stored by mapping to the conductance value of the memristor in the memristor cross array, parallel operation of the matrix is realized by the current and voltage accumulation characteristics of the memristor cross array, the storage and operation processes are completed in the memristor cross array, data do not need to be transmitted between an operation unit and a storage unit, and the integration of storage and calculation is realized.
The mixed cross array circuit formed by memristive elements and CMOS elements is constructed, convolution operation is equivalently converted into matrix-vector multiplication operation, and parallel operation of the matrix-vector multiplication operation in the memristive cross array is realized.
In each residual convolution sub-network, in order to avoid the size reduction of the output feature map and the loss of image edge information, the embodiment adopts a 'same' convolution mode, and the implementation method is to supplement 0 to the edge of the input image, and when the convolution step length is 1, the feature map output through convolution operation has the same size as the input image. When the "same" convolution is implemented using a memristive crossover array, this can be achieved by asserting 0 on the corresponding input signal.
In convolutional neural networks, for a matrix-vector multiplication operation with h inputs and o outputs, the following formula can be used:
In the formula (8), x n represents the nth element in the input vector, w mn represents the weight of the mth row and n column in the weight matrix, and y m represents the mth output in the output vector.
In the hardware design corresponding to the convolution layer, the matrix-vector multiplication operation with h inputs and o outputs is realized by adopting a first 2D memristor cross array in the embodiment. As shown in fig. 6, the circuit includes h rows of o columns of first wires which are crisscrossed but not connected, a second wire which is crisscrossed but not connected with the input ends of the h rows of first wires, and a third wire which is crisscrossed but not connected with the input ends of the o columns of first wires, wherein the second wire and the third wire are connected through an amplifying circuit;
each of the two first wires is connected with two memristors at the staggered position, each group of the first wires and the second wires and each group of the first wires and the third wires are connected with two wires by a first auxiliary constant value resistor (R a), and the output ends of the o columns of the first wires are connected in series with a current-voltage conversion circuit;
The amplifying circuit comprises a first auxiliary constant value resistor and a first operational amplifier, wherein the inverting input end and the output end of the first operational amplifier are respectively connected with a second wire and a third wire, the non-inverting input end of the first operational amplifier is grounded, and the first auxiliary constant value resistor is connected between the inverting input end and the output end of the operational amplifier in parallel;
The current-voltage conversion circuit comprises a second auxiliary constant value resistor (R c) and a second operational amplifier, wherein the second auxiliary constant value resistor is connected in parallel between the inverting input end and the output end of the second operational amplifier, the output end of the second operational amplifier is used as a final voltage output end, and the non-inverting input end of the second operational amplifier is grounded.
The wires and memristors used in the first 2D memristive crossover array are both nanoscale.
Of course, other circuit structures besides the circuit structure shown in fig. 6 may be used for the first 2D memristive cross array, so long as the same function is achieved.
When the first 2D memristor cross array works, an input vector x 1～xh in the formula (8) is converted into a row input voltage V x1～Vxh of the first 2D memristor cross array, a weight w mn is mapped into a conductance value of a memristor in an mth row and an n-th column in the first 2D memristor cross array, the input voltage V xn is multiplied by a conductance value w mn of a corresponding memristor, an output current flowing through each memristor is superposed through a wire to obtain a column output current, and the output current of each column is converted into a voltage V ym through a current-voltage conversion circuit and then is output. According to kirchhoff's law, the output V ym of the first 2D memristive crossover array is represented as equation (9), and the weight w mn is represented as equation (10):
wmn＝(Ga-Gmn)·Rc,m＝1,2,…,o；n＝1,2,…,h (10)
wherein, R a is a first auxiliary constant value resistor, G a is the conductance of R a, R c is a second auxiliary constant value resistor, and G mn is the conductance of the m-th row n-th column memristor in the first 2D memristor cross array.
The weights may be written into the first 2D memristor crossover array by adjusting the conductance values G mn of the respective position memristors. Since the conductance G mn of the memristor is constant to be positive and cannot represent negative weight, a constant value resistor R a is introduced, the conductance values of the constant value resistor G a,Gon and the constant value resistor G off respectively represent the conductance values of the lowest resistance state and the highest resistance state of the memristor, so that the weight range of the memristor in the memristor cross array is [ - (G on-Goff)*Rc/2,(Gon-Goff)*Rc/2 ].
Taking the first layer of convolution layer in the first residual convolution sub-network as an example, it uses a convolution kernel with a size of 3*3, the number of input channels is 3, the number of output channels is 64, and in hardware, the convolution kernel will map into 1 column vector with a length of 27, h=27, o=64 in the 2D memristive crossover array in fig. 6. In the MRCAPSNET deployment process, 3×3×64 weights are mapped to conductance values of memristors in the 27×64 memristor cross array. In the network reasoning stage, the test set of the input network is converted into a read voltage input cross array within a voltage threshold, and the output voltages at the V y1～Vy64 end respectively represent the output values of the corresponding channels of the capsule network. Similarly, the 2D memristive crossover array described above is equally applicable to hardware deployment of the first residual convolution sub-network and other convolution layers in the second and third residual convolution sub-networks.
Compared with the traditional convolutional neural network, the input and output of the capsule neural network are expanded by one dimension on the basis of the traditional convolutional neural network, so that the weight calculation among layers in the capsule neural network is correspondingly expanded by one dimension, the matrix-vector calculation in the traditional convolutional network is changed into matrix-matrix calculation, and a 3D memristor cross array is needed to be adopted. The embodiment constructs a 3D memristor-based cross array for realizing hardware deployment of a capsule layer.
For any one of the first initial capsule layer, the second initial capsule layer and the third initial capsule layer, the initial capsule layer is provided with R capsule units, each capsule unit comprises X convolution units, the convolution kernel size is U, and on the corresponding hardware design, the operation of the initial capsule layer is realized by adopting a first 3D memristor cross array.
The first 3D memristor cross array comprises T layers of second 2D memristor cross arrays similar to the first 2D memristor cross array in structure, the T layers of second 2D memristor cross arrays correspond to T input feature patterns, the memristor specification of each layer of second 2D memristor cross arrays is (R X) column (U X) row, each layer of second 2D memristor cross arrays operates independently, the outputs of the different layers of second 2D memristor cross arrays are connected in the same column, the output voltage of each layer of second 2D memristor cross arrays is summed with the outputs of other layers in the vertical column direction, and the T feature patterns are converted into one-dimensional voltage output through the first 3D memristor cross arrays.
After a voltage sequence, the first 3D memristor cross array outputs r×x one-dimensional voltage signals, representing an X-dimensional vector of R capsule units; after all voltage sequences are passed through by using serial input mode, the initial capsule layer outputs Y X-dimensional vectors, i.e. Y X R initial capsules, Y X R represents the size of output characteristic diagram.
Taking the first residual capsule module (ResCapsBlock 1) as an example, the initial capsule layer has r=8 capsule units, each capsule unit contains x=12 convolution units, the convolution kernel size is u×u= 5*5, so that the initial capsule layer can be equivalent to 12 parallel common convolution layers, the input of each convolution layer is t=256 channels, and the output of each convolution layer is 8 channels, and thus, the initial capsule layer can be deduced to contain 12×8×256 convolution kernels in total.
The structure of the first 3D memristor cross array adopted by the first initial capsule layer is shown in fig. 7, each layer structure is similar to the implementation mode of the first 2D memristor cross array, dark gray ports are input and output ports, auxiliary resistors and operational amplifiers are included in the dark gray ports, light gray strip-shaped elements represent nanoscale wires, elements at intersections between each wire and other wires are nano memristors, and different layers are connected through longitudinal buses. The first 3D memristive cross array has a total of t=256 layers of first 2D memristive cross arrays, the convolution kernel of each 5*5 is mapped into one column in a cross array, 96 (8×12) columns represent 12 convolution kernels in 8 capsule units, so that the specification of each layer of first 2D memristive cross array is 96×25 (8×12×5×5), each layer of array is independently operated, and the outputs of different layers are connected in the same column. 256 layers of the first 3D memristor cross array correspond to 256 feature maps of the input initial capsule, output voltage of each layer is summed with output of other layers in the vertical column direction, and the 256 feature maps are converted into one-dimensional voltage output through the first 3D memristor cross array. After a voltage sequence, the first 3D memristive crossover array outputs 96 (8×12) one-dimensional voltage signals representing a 12-dimensional vector of 8 capsule cells. After all voltage sequences, the initial capsule layer output size is 14×14×8 12-dimensional vectors, i.e., 14×14×8 initial capsules.
The second initial capsule layer and the third initial capsule layer have the same specification, and the hardware deployment scheme is also realized in the manner described above.
For the digital capsule layer, the digital capsule layer has four sublayers for converting four sets of initial capsules into corresponding four sets of digital capsules; any sub-layer comprises M capsule units, Y X dimension vectors output by the initial capsule layer and M capsule units of the sub-layer share Y X weight transformation matrixes, and the weight transformation matrixes are mapped to Y X independent second 3D memristor cross arrays for parallel operation in the hardware design process;
Each second 3D memristor cross array comprises M layers of third 2D memristor cross arrays similar to the first 2D memristor cross arrays in structure, the memristor specification of each layer of third 2D memristor cross arrays is V rows x W columns, V rows correspond to the input of V-order tensors, W columns correspond to the output of W-order tensors, and finally the sub-layers output M W-order tensors, namely M digital capsules with W dimensions.
Taking Cifar data sets as an example, MRCAPSNET needs to solve the 10 classification problem, i.e. m=10, so the digital capsule layer of the network contains 10 capsule units corresponding to 10 pictures to be classified. Each capsule in the initial capsule layer and each capsule in the digital capsule layer have a corresponding weight transformation matrix, taking the first sublayer DIGITCAPSLAYER1 of the digital capsule layer as an example, the output of the layer is 10 16-order tensors, so that 14×14×8 12-order tensors output by the initial capsule layer PRIMARYCAPSLAYER1 and 10 capsule units of the digital capsule layer1 share 14×14×12×10 weight transformation matrices, these matrices can be mapped to 14×14×8×12 independent second 3D memristive cross arrays in a reasoning process, the structure of each second 3D memristive cross array is shown in fig. 8, the second 3D memristive cross array is composed of a third 2D memristive cross array of 10 layers 12×16 (similar to the structure of the first 2D memristive cross array), v=12 rows input represents the input of the digital capsule layer is 12-order tensors, and w=16 columns output represents the output of the digital capsule layer is 16-order tensors. In the digital capsule layers, the 10 layers of the second 3D memristive crossbar array all have the same 12 th order tensor input from the initial capsule, with each layer, i.e., each capsule cell, independently outputting one 16 th order tensor. The first sublayer DIGITCAPSLAYER of the final digital capsule layer outputs 10 16-order tensors for object classification and image reconstruction. Similarly, hardware implementations of other sublayers are also implemented as described above.
And the dimensions of the digital capsules output by the four sublayers of the digital capsule layer are W 1、W2、W3、W4 respectively, so that the digital capsule layer splices the four groups of digital capsules to obtain M new digital capsules with the dimensions W 1+W2+W3+W4 for target classification and image reconstruction. As in fig. 2, w 1、W2、W3、W4 is equal to 16, 12, 10, 16, w 1+W2+W3+W4 =54, m=10, respectively, the digital capsule layer outputs 10 new digital capsules of dimension 54.
The 3D memristor cross array improves packaging density, is more compact in structure, and is additionally provided with one-dimensional parallel operation on the basis of the 2D memristor cross array, so that more efficient parallel operation is supported.
The transposed convolution layer up-samples the input tensor, and a reconstruction module formed by the multi-layer superimposed transposed convolution restores the feature map to the original size. The first step of transpose convolution implementation is to supplement 0 to the input feature map according to rules, and the second step is to execute normal convolution operation to the feature map after 0 supplementation. The size of the transposed convolution input feature diagram is defined as H.times.H.times.N, the convolution step length is defined as S, the size of each channel of the feature diagram after 0 is complemented is defined as H out*Hout, and the calculation mode is as follows:
Hout＝H+(H-1)×(S-1)+2P+O (11)
The filling rule is as follows: s-1 0 is inserted between every two pixels of the input feature map, then P circles of 0 values are supplemented around the feature map, finally O rows of 0 values are supplemented below the feature map, and O columns of 0 values are supplemented to the right of the feature map. In the process of realizing transposed convolution by hardware, firstly, setting 0 for a corresponding input signal according to the rule, and then, mapping the weight of the input signal into a memristor cross array according to the circuit implementation scheme of the convolution network to finish deployment.
Experiments and analyses were performed as follows.
The experiment tested memristive residual capsule network MRCAPSNET on three datasets of MNIST, cifar-10, SVHN, and compared with other capsule models. In addition, an ablation experiment is performed in the embodiment to prove the effectiveness of the multi-level residual capsule module provided by the scheme. The test is carried out on software, the test environments are Ubuntu operating system, keras2.1.6, tensorflow gpu1.15.4 and Python3.6, and the hardware environment for model training is Geforce RTX3090 and Intel Xeon E3 processor.
To fully test MRCAPSNET the performance on both simple and complex datasets, the following three datasets were selected for testing in this example.
1. MNIS: the dataset consisted of 70K 10-class gray scale handwritten digital images, with 60K images for training and 10K images for testing, with image sizes of 28 x 28.
2. CIFAR-10: the dataset contained 60K color images in total of 10 categories, 32 x 3 in size, 50K images for training and 10K images for testing. The image in CIFAR-10 data set has complex characteristic information and background information, and can well detect the performance of the network under the condition of strong interference.
3. STREET VIEW House Numbers (SVHN): the data set comprises 10 color images of 32 x 3 in category, 73K training sets and 26K test sets, wherein the images come from house numbers in google street view, and the performance of the neural network in a real natural scene can be detected.
In the training process, an Adam optimizer is used in the example, the initial learning rate is 0.001, the attenuation of each epoch is 0.9 times of the original attenuation, the batch size is 128, and the training iteration number is 100. The parameters remain unchanged in all datasets during the MRCAPSNET training process.
The accuracy of this experiment was compared MRCAPSNET with capsule network CapsNet and the results are shown in table 2. As can be seen from table 2, compared with the capsule network CapsNet shown in fig. 1, the performance of the MRCAPSNET single model on the CIFAR-10 dataset and the SVHN dataset is improved by 21.6% and 0.9%, respectively, and the performance of MRCAPSNET for the MNIST dataset is slightly lower than CapsNet, but is also maintained at a higher level, the parameter amount of MRCAPSNET is 16.01M, and the parameter amount of CapsNet is 22.48M, which indicates that the multi-level residual module multiplexing structure proposed by the present example can effectively learn multi-level features in an image.
TABLE 2
| Model | MNIST | SVHN | CIFAR-10 |
| CapsNet | 99.75％ | 95.70％ | 68.74％ |
| MRCapsNet | 99.70％ | 96.60％ | 90.34％ |
In the following, the embodiment proves the superiority of the reconstruction network of multilevel ResBlock, cavity convolution and transposition convolution through an ablation experiment. In the CNN field, the research result about the residual network shows that the multi-stage residual block can improve the performance of the network to a certain extent, in order to understand how the number of residual capsule modules has an effect on the performance of MRCaps, a series of experiments are performed in this embodiment, and the experimental result is shown in table 3, when the number of residual capsule modules is increased from 1 to 3, the accuracy of the network is increased from 80.58% to 89.65%, and when a hole convolution method is added to the residual capsule modules and three capsule modules of the capsule are spliced, the accuracy is increased to 90.34%. As can be seen from the training Loss graph in fig. 9, there is a certain drop in Loss value after each stage ResCapsBlock, indicating that each stage ResCapsBlock learns new features.
In the comparison experiment of the number of the capsule modules, the comparison experiment of the cavity convolution is added, and the experimental result table 3 shows that after the cavity convolution is used, the precision of each stage of residual capsule module is improved, the overall network precision is improved from 89.95% to 90.34%, and the improvement effect is obvious. The shallow residual capsule module has extracted more small characteristic information by using a convolution kernel of 3 multiplied by 3, and the method of cavity convolution in the deep residual capsule module can increase the receptive field and extract the characteristic information in a larger range on the premise of not increasing the calculated amount.
TABLE 3 Table 3
| Model | Test accuracy (%) |
| ResCapsBlock 1 (without hole convolution) | 80.58 |
| ResCapsBlock 2 (without hole convolution) | 88.32 |
| ResCapsBlock 3 (without hole convolution) | 89.65 |
| Merging ResCapsBlock A1, 2,3 (without hole convolution) | 89.95 |
| ResCapsBlock 1 (using hole convolution with expansion ratio (1, 1)) | 80.59 |
| ResCapsBlock 2 (using hole convolution with expansion ratio (1, 1)) | 88.71 |
| ResCapsBlock 3 (using hole convolution with expansion ratio (2, 2)) | 89.8 |
| Merging ResCapsBlock, 1,2, 3 (with hole convolution) | 90.34 |
In this example, reconstruction experiments are performed on CIFAR-10 datasets by using the reconstruction network proposed by MRCAPSNET and the reconstruction network of CapsNet base line networks, as can be derived from fig. 10, the reconstruction network consisting of the multi-layer transposed convolution proposed in this example has smaller reconstruction errors in the training phase, and as can be seen from table 4, the number of reconstruction sub-networks formed by the transposed convolution is only 16.39% of that of the CapsNet base line networks, and has better performance.
TABLE 4 Table 4
| Reconstruction module | Quantity of parameters | ACC |
| Transpose convolution (MRCAPSNET) | 303497 | 90.34％ |
| Full connection (CapsNet) | 1851904 | 89.77％ |
The power consumption of the core computing unit MRCAPSNET mainly comprises two parts, one part is the power consumption required by mapping the neural network weight to the cross array, and the other part is the power consumption of the memristive cross array in the reasoning process. The memristor has a maximum resistance R on and a minimum resistance R off of 100kΩ and 100mΩ, respectively. Memristor write voltage is 6.5V, and maximum write power consumption of a single memristor is 0.5 μw. In the memristor cross array reasoning stage, the written weight of each memristor is different, so that the power consumption is different, and therefore, the maximum power consumption of the memristor under the limit condition is estimated to be 6.9 mu W according to the above method, and the same calculated CMOS circuit power consumption is realized to be close to 60 mu W. Therefore, the hardware implementation scheme provided by the embodiment can greatly reduce the side power consumption of the capsule network side.
In summary, in terms of algorithm, the embodiment of the invention provides a novel capsule network structure MRCAPSNET, which constructs a ResCapsBlock module based on the residual learning idea, solves the problem of gradient disappearance after network deepening, realizes multi-stage information multiplexing by a multi-stage ResCapsBlock serial connection mode, introduces cavity convolution in a ResCapsBlock of a higher layer, expands receptive fields, and is helpful for extracting more features of the capsule network. The embodiment of the invention also provides a reconstruction network constructed by using the transposed convolution, which improves the network performance while reducing the network parameter quantity. MRCASPNET achieved more advanced performance than the current stage capsule network model.
The embodiment of the invention also provides a capsule network hardware implementation scheme based on the memristor cross array, and the memristor cross array has the advantages of low power consumption and support of parallel matrix calculation, so that the problem of large calculation amount of the capsule network can be solved. The power consumption of neurons in the circuit is analyzed and calculated, and the result shows that the power consumption of the circuit based on the memristor cross array is far smaller than that of a CMOS circuit.
The above examples are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above examples, and any other changes, modifications, substitutions, combinations, and simplifications that do not depart from the spirit and principle of the present invention should be made in the equivalent manner, and the embodiments are included in the protection scope of the present invention.
Claims (10)
1. An image classification system based on a residual capsule neural network is characterized in that the residual capsule neural network is carried; the residual capsule neural network comprises a first residual capsule module, a second residual capsule module, a third residual capsule module and a digital capsule layer; the first residual capsule module comprises a first residual convolution sub-network and a first initial capsule layer, the second residual capsule module comprises a second residual convolution sub-network and a second initial capsule layer, and the third residual capsule module comprises a third residual convolution sub-network and a third initial capsule layer;
The first residual convolution sub-network carries out convolution calculation on an input original image and then outputs a first group of feature images to the second residual convolution sub-network and the first initial capsule layer, the first initial capsule layer converts the first group of feature images into a first initial capsule group, and the digital capsule layer converts the first initial capsule group into a first digital capsule group;
the second residual convolution sub-network carries out convolution calculation on the input first group of feature images and then outputs a second group of feature images to the third residual convolution sub-network and the second initial capsule layer, the second initial capsule layer converts the second group of feature images into a second initial capsule group, and the digital capsule layer converts the second initial capsule group into a second digital capsule group;
The third residual convolution sub-network carries out convolution calculation on the input second group of feature images and then outputs a third group of feature images to the third initial capsule layer, the third initial capsule layer converts the third group of feature images into a third initial capsule group, and the digital capsule layer converts the third initial capsule group into a third digital capsule group;
the digital capsule layer splices the first initial capsule group, the second initial capsule group and the third initial capsule group to obtain a fourth initial capsule group and converts the fourth initial capsule group into a fourth digital capsule group;
and the digital capsule layer is used for outputting M new digital capsules for target classification and image reconstruction after splicing and fusing the four groups of digital capsules in the digital capsule layer, wherein M is equal to the total category number of image classification.
2. An image classification system based on a residual capsule neural network according to claim 1, characterized in that: the residual capsule neural network further comprises a reconstruction module, wherein the reconstruction module is used for generating a reconstruction image with the same size as the original image according to M new digital capsules.
3. The image classification system based on residual capsule neural network of claim 2, wherein the residual capsule neural network has a loss function of:
Wherein L total represents total loss ;Lk＝Tkmax(0,m+-||vk||)2+λ(1-Tk)max(0,||vk||-m-)2 represents edge loss of capsule k in the last capsule layer, T k represents matching parameters of a kth class classification target corresponding to capsule k, T k =1 if and only if classification is correct, otherwise T k＝0,vk represents activation vector of capsule k, m +、m-, λ are capsule vector modular length control parameters, m +＝0.9,m-＝0.1,λ＝0.5;Lr represents reconstruction loss, which is equal to mean square error between pixels of reconstructed image and original image; η represents a weight coefficient of the reconstruction loss at the total loss.
4. An image classification system based on a residual capsule neural network according to claim 3, characterized in that: the first residual convolution sub-network, the second residual convolution sub-network and the third residual convolution sub-network comprise two convolution layers and a jump connection layer, and each convolution layer adopts the same convolution with the introduced hole convolution; the digital capsule layer converts the initial capsule into a digital capsule through a dynamic routing algorithm and an extrusion activation function.
5. The residual capsule neural network-based image classification system of claim 4, wherein: in a convolution layer of the first residual convolution sub-network, the second residual convolution sub-network, or the third residual convolution sub-network, for a matrix-vector multiplication operation having h inputs and o outputs, the operation is expressed by the following formula:
In the formula (8), x n represents an nth element in the input vector, w mn represents a weight of an mth row and an nth column in the weight matrix, and y m represents an mth output in the output vector;
In the hardware design corresponding to the convolution layer, matrix-vector multiplication operation with h inputs and o outputs is realized by adopting a first 2D memristor cross array;
The first 2D memristor cross array comprises h rows, o columns and columns of memristors, x 1～xh is converted into row input voltage V x1～Vxh of the first 2D memristor cross array, a weight w mn is mapped into conductance values of m rows and n columns of memristors in the first 2D memristor cross array, input voltage V xn is multiplied by the conductance value w mn of the corresponding memristor, output current flowing through each memristor is superposed through a wire to obtain column output current, and the output current of each column is converted into voltage V ym through a current-voltage conversion circuit and then output.
6. The residual capsule neural network-based image classification system of claim 5, wherein: for any one of the first initial capsule layer, the second initial capsule layer and the third initial capsule layer, the initial capsule layer is provided with R capsule units, each capsule unit comprises X convolution units, the convolution kernel size is U, and on the corresponding hardware design, the operation of the initial capsule layer is realized by adopting a first 3D memristor cross array;
The first 3D memristor cross array comprises T layers of second 2D memristor cross arrays with similar structures as the first 2D memristor cross arrays, T layers of second 2D memristor cross arrays correspond to T input feature maps, the memristor specification of each layer of second 2D memristor cross arrays is (R X) column (U X U) row, each layer of second 2D memristor cross arrays operate independently, the outputs of different layers of second 2D memristor cross arrays are connected in the same column, the output voltage of each layer of second 2D memristor cross arrays is summed with the outputs of other layers in the direction of the vertical column, and the T feature maps are converted into one-dimensional voltage output through the first 3D memristor cross arrays;
after a voltage sequence, the first 3D memristive crossover array outputs R X one-dimensional voltage signals representing an X-dimensional vector of R capsule units; after all voltage sequences are passed through by using serial input mode, the initial capsule layer outputs Y X-dimensional vectors, i.e. Y X R initial capsules, Y X R represents the size of output characteristic diagram.
7. The residual capsule neural network-based image classification system of claim 6, wherein: the digital capsule layer is provided with four sublayers and is used for converting four groups of initial capsules into four corresponding groups of digital capsules; any sub-layer comprises M capsule units, Y X dimension vectors output by the initial capsule layer and M capsule units of the sub-layer share Y X weight transformation matrixes, and the weight transformation matrixes are mapped to Y X independent second 3D memristor cross arrays for parallel operation in the hardware design process;
Each second 3D memristor cross array comprises M layers of third 2D memristor cross arrays similar to the first 2D memristor cross arrays in structure, the memristor specification of each layer of third 2D memristor cross arrays is V rows by W columns, V rows correspond to the input of V-order tensors, W columns correspond to the output of W-order tensors, and finally the sub-layers output M W-order tensors, namely M digital capsules with W dimensions.
8. The residual capsule neural network-based image classification system of claim 7, wherein: and the dimensions of the digital capsules output by the four sublayers of the digital capsule layer are W 1、W2、W3、W4 respectively, so that the digital capsule layer can splice the four groups of digital capsules to obtain M new digital capsules with the dimensions W 1+W2+W3+W4 for target classification and image reconstruction.
9. An image classification system based on a residual capsule neural network according to claim 3, characterized in that: the reconstruction module comprises a full-connection layer and four transposition convolution layers, wherein each transposition convolution layer firstly supplements 0 to an input characteristic diagram according to rules and then carries out convolution operation on the characteristic diagram after 0 supplementation;
Defining the size of an input feature diagram of a transposed convolution layer as h×h×n, the convolution step size as S, and the size of each channel of the input feature diagram after 0 is complemented as H out*Hout, wherein the calculation method is as follows:
Hout＝H+(H-1)×(S-1)+2P+O (11)
The filling rule is as follows: s-1 0 is inserted between every two pixels of the input feature map, then P circles of 0 values are supplemented around the feature map, finally O rows of 0 values are supplemented below the feature map, and O columns of 0 values are supplemented to the right of the feature map.
10. An image classification system based on a residual capsule neural network according to any one of claims 1-9, characterized in that: the dataset used to train and test the residual capsule neural network includes more than ten thousand m=10 gray-scale images or RGB color images derived from MNIST, cifar-10 or SVHN datasets.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111587100.5A CN114241245B (en) | 2021-12-23 | 2021-12-23 | Image classification system based on residual capsule neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111587100.5A CN114241245B (en) | 2021-12-23 | 2021-12-23 | Image classification system based on residual capsule neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114241245A CN114241245A (en) | 2022-03-25 |
| CN114241245B true CN114241245B (en) | 2024-05-31 |
Family
ID=80761930
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111587100.5A Active CN114241245B (en) | 2021-12-23 | 2021-12-23 | Image classification system based on residual capsule neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114241245B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114781628A (en) * | 2022-03-29 | 2022-07-22 | 清华大学 | Memristor noise-based data enhancement method and device, electronic equipment and medium |
| CN114463161B (en) * | 2022-04-12 | 2022-09-13 | 之江实验室 | Method and device for processing continuous images by neural network based on memristor |
| CN115545110B (en) * | 2022-10-12 | 2024-02-02 | 中国电信股份有限公司 | High resolution data reconstruction method for generating an antagonism network and related method and apparatus |
| WO2024108505A1 (en) * | 2022-11-24 | 2024-05-30 | 深圳先进技术研究院 | Myocardial fibrosis classification method based on residual capsule network |
| CN118351118B (en) * | 2024-06-18 | 2024-11-12 | 湖南科技大学 | Printer casing defect detection method and device based on deep convolutional neural network |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018102748A1 (en) * | 2016-12-01 | 2018-06-07 | Berkeley Lights, Inc. | Automated detection and repositioning of micro-objects in microfluidic devices |
| CN110009097A (en) * | 2019-04-17 | 2019-07-12 | 电子科技大学 | Capsule Residual Neural Network, Image Classification Method of Capsule Residual Neural Network |
| CN110032925A (en) * | 2019-02-22 | 2019-07-19 | 广西师范大学 | A kind of images of gestures segmentation and recognition methods based on improvement capsule network and algorithm |
| CN110443867A (en) * | 2019-08-01 | 2019-11-12 | 太原科技大学 | Based on the CT image super-resolution reconstructing method for generating confrontation network |
| CN110657984A (en) * | 2019-09-27 | 2020-01-07 | 重庆大学 | A Fault Diagnosis Method of Planetary Gearbox Based on Enhanced Capsule Network |
| CN110991633A (en) * | 2019-12-04 | 2020-04-10 | 电子科技大学 | A residual neural network model based on memristive network and its application method |
| CN111046916A (en) * | 2019-11-20 | 2020-04-21 | 上海电机学院 | A method and system for motor fault diagnosis based on hollow convolutional capsule network |
| CN111241958A (en) * | 2020-01-06 | 2020-06-05 | 电子科技大学 | Video image identification method based on residual error-capsule network |
| CN112070174A (en) * | 2020-09-11 | 2020-12-11 | 上海海事大学 | Text detection method in natural scene based on deep learning |
| CN112233106A (en) * | 2020-10-29 | 2021-01-15 | 电子科技大学中山学院 | Thyroid cancer ultrasonic image analysis method based on residual capsule network |
| CN112446357A (en) * | 2020-12-15 | 2021-03-05 | 电子科技大学 | SAR automatic target recognition method based on capsule network |
| CN112766340A (en) * | 2021-01-11 | 2021-05-07 | 中山大学 | Depth capsule network image classification method and system based on adaptive spatial mode |
| CN113012714A (en) * | 2021-02-22 | 2021-06-22 | 哈尔滨工程大学 | Acoustic event detection method based on pixel attention mechanism capsule network model |
| CN113034414A (en) * | 2021-03-22 | 2021-06-25 | 上海交通大学 | Image reconstruction method, system, device and storage medium |
| JP2021525359A (en) * | 2018-05-31 | 2021-09-24 | バークレー ライツ，インコーポレイテッド | Automatic detection and characterization of micro-objects with microfluidic devices |
| CN113538616A (en) * | 2021-07-09 | 2021-10-22 | 浙江理工大学 | Magnetic resonance image reconstruction method combining PUGAN and improved U-net |
| CN113780140A (en) * | 2021-08-31 | 2021-12-10 | 河北大学 | Gesture image segmentation and recognition method and device based on deep learning |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11010902B2 (en) * | 2018-06-04 | 2021-05-18 | University Of Central Florida Research Foundation, Inc. | Capsules for image analysis |
| US11514579B2 (en) * | 2018-06-04 | 2022-11-29 | University Of Central Florida Research Foundation, Inc. | Deformable capsules for object detection |
-
2021
- 2021-12-23 CN CN202111587100.5A patent/CN114241245B/en active Active
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018102748A1 (en) * | 2016-12-01 | 2018-06-07 | Berkeley Lights, Inc. | Automated detection and repositioning of micro-objects in microfluidic devices |
| JP2021525359A (en) * | 2018-05-31 | 2021-09-24 | バークレー ライツ，インコーポレイテッド | Automatic detection and characterization of micro-objects with microfluidic devices |
| CN110032925A (en) * | 2019-02-22 | 2019-07-19 | 广西师范大学 | A kind of images of gestures segmentation and recognition methods based on improvement capsule network and algorithm |
| CN110009097A (en) * | 2019-04-17 | 2019-07-12 | 电子科技大学 | Capsule Residual Neural Network, Image Classification Method of Capsule Residual Neural Network |
| CN110443867A (en) * | 2019-08-01 | 2019-11-12 | 太原科技大学 | Based on the CT image super-resolution reconstructing method for generating confrontation network |
| CN110657984A (en) * | 2019-09-27 | 2020-01-07 | 重庆大学 | A Fault Diagnosis Method of Planetary Gearbox Based on Enhanced Capsule Network |
| CN111046916A (en) * | 2019-11-20 | 2020-04-21 | 上海电机学院 | A method and system for motor fault diagnosis based on hollow convolutional capsule network |
| CN110991633A (en) * | 2019-12-04 | 2020-04-10 | 电子科技大学 | A residual neural network model based on memristive network and its application method |
| CN111241958A (en) * | 2020-01-06 | 2020-06-05 | 电子科技大学 | Video image identification method based on residual error-capsule network |
| CN112070174A (en) * | 2020-09-11 | 2020-12-11 | 上海海事大学 | Text detection method in natural scene based on deep learning |
| CN112233106A (en) * | 2020-10-29 | 2021-01-15 | 电子科技大学中山学院 | Thyroid cancer ultrasonic image analysis method based on residual capsule network |
| CN112446357A (en) * | 2020-12-15 | 2021-03-05 | 电子科技大学 | SAR automatic target recognition method based on capsule network |
| CN112766340A (en) * | 2021-01-11 | 2021-05-07 | 中山大学 | Depth capsule network image classification method and system based on adaptive spatial mode |
| CN113012714A (en) * | 2021-02-22 | 2021-06-22 | 哈尔滨工程大学 | Acoustic event detection method based on pixel attention mechanism capsule network model |
| CN113034414A (en) * | 2021-03-22 | 2021-06-25 | 上海交通大学 | Image reconstruction method, system, device and storage medium |
| CN113538616A (en) * | 2021-07-09 | 2021-10-22 | 浙江理工大学 | Magnetic resonance image reconstruction method combining PUGAN and improved U-net |
| CN113780140A (en) * | 2021-08-31 | 2021-12-10 | 河北大学 | Gesture image segmentation and recognition method and device based on deep learning |
Non-Patent Citations (2)
| Title |
|---|
| Lightweight multi-dimensional memristive CapsNet;Dan, S等;《2021 International Joint Conference on Neural Networks (IJCNN)》;20210920;1-8 * |
| 改进深度胶囊网络的分类模型研究及其应用;周松;《中国优秀硕士学位论文全文数据库 信息科技辑》;20210115(第1期);138-1054 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114241245A (en) | 2022-03-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114241245B (en) | Image classification system based on residual capsule neural network | |
| CN113011499B (en) | Hyperspectral remote sensing image classification method based on double-attention machine system | |
| CN109460817B (en) | A Convolutional Neural Network On-Chip Learning System Based on Nonvolatile Memory | |
| CN113361664B (en) | Image recognition system and method based on quantum convolution neural network | |
| Yu et al. | An overview of neuromorphic computing for artificial intelligence enabled hardware-based hopfield neural network | |
| US9646243B1 (en) | Convolutional neural networks using resistive processing unit array | |
| US11087204B2 (en) | Resistive processing unit with multiple weight readers | |
| CN107506740A (en) | A kind of Human bodys' response method based on Three dimensional convolution neutral net and transfer learning model | |
| KR102744306B1 (en) | A method for slicing a neural network and a neuromorphic apparatus | |
| CN108681539A (en) | A kind of illiteracy Chinese nerve interpretation method based on convolutional neural networks | |
| CN107657204A (en) | The construction method and facial expression recognizing method and system of deep layer network model | |
| EP3923201A1 (en) | Apparatus and method with neural network | |
| CN109086802A (en) | A kind of image classification method based on biquaternion convolutional neural networks | |
| CN111027619A (en) | A K-means classifier based on memristor array and its classification method | |
| CN114626500A (en) | Neural network computing method and related equipment | |
| CN115699028A (en) | Efficient Tile Mapping for Progressive Convolutional Neural Network Mapping to Simulate AI Network Inference | |
| Gao et al. | Natural scene recognition based on convolutional neural networks and deep Boltzmannn machines | |
| Shi et al. | A pooling-based feature pyramid network for salient object detection | |
| CN111461308B (en) | Memristive neural network and weight training method | |
| Goswami et al. | Automatic object recognition from satellite images using artificial neural network | |
| KR20210143614A (en) | Neuromorphic device for implementing neural network and method for thereof | |
| Turay et al. | SSP framework: A new approach to designing lightweight convolutional neural networks | |
| EP4089585A1 (en) | Device and method with multidimensional vector neural network | |
| Kendall et al. | Deep learning in memristive nanowire networks | |
| Ji et al. | Remote Sensing Scene Classification via Pseudo Category-Relation and Orthogonal Feature Learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |