JPH05101031A - Neural network device - Google Patents

Abstract

PURPOSE:To accelerate a propagation arithmetic operation and to save memory capacity when local coupling is used by inserting a delay element to a ring register bus, and applying mapping by concentrating a matrix element with value 0 in a specific column. CONSTITUTION:The delay elements (variable delay first-in/first-out memory) 50-52, 61, and 62 to the ring register buses 1, 2. For example, when the product of transposed matrix and a vector is computed, the ring register 1 and internal registers in the delay elements 50-52 are used as accumulators. In such a way. the mapping in which the matrix element with value O is concentrated in the specific column can be performed. Thereby, the number of times of operations required for computation can be reduced. Also, the capacity of memory devices 41-43 for coupling load can be saved. Furthermore, the setting of input data can be performed by using another ring register bus 1 or 2 while a sum of products arithmetic operation is being performed by using a certain ring register bus 1 or 2. Therefore. a time required for such setting can be ignored.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a neural network device used for image recognition, voice recognition and the like.

[0002]

2. Description of the Related Art A ring register bus is known as a parallel architecture for performing high-speed product-sum calculation such as back propagation of a neural network, and is described in Technical Report NC89-76. The structure and operation of a conventional ring register bus type neural network device will be briefly described.

FIG. 10 is a block diagram showing the configuration of a conventional ring register bus type neural network device. As shown in FIG. 10, a plurality of ring registers 11 to 11
15, processing units (PE) 31 to 33, and storage device 4
1 to 43. The ring registers 11 to 15 are registers having a transfer function, and are connected to the ring registers on both sides to form the ring register bus 1 as a whole, and collect and supply data with each arithmetic unit. Ring register 1
The arithmetic units 31 to 33 are connected to some of the units 1 to 15, and a multiplier and an adder are built therein.
Further, the arithmetic units 31 to 33 have storage devices 41 to 4 respectively.
3 is connected. Basically, one arithmetic unit is assigned to one neuron, and a plurality of layers are realized by time-division sequential execution by software. The ring register to which the arithmetic unit is not connected can be replaced with a first-in-first-out (FIFO) memory.

The operation of this architecture will be described with reference to the drawings. FIG. 11 is a timing chart showing the operation at the time of calculating the product WX of a matrix and a vector, and FIG. 12 is a timing chart showing the operation at the time of calculating the product W ^T δ of a transposed matrix and a vector. However, the matrix W is a 4 × 3 matrix, and the vector X and the vector δ are vectors of 4 elements and 3 elements, respectively. The matrix W is mapped in the storage devices 41 to 43 connected to the respective arithmetic devices 31 to 33 such that each diagonal element of W is at the head.

When calculating WX, as shown in FIG.
Each element of X is placed in the ring registers 11 to 14 and moves on the ring register bus 1. A register inside the processor _i is used as an accumulator acc _i . At time T ₁ , PE _i fetches X _i from ring registers 11 to 13 and w _i and _i from storage devices 41 to 43 and multiplies them, and adds the product to accumulator acc _i . Immediately after this, the input data X _i rotates counterclockwise by one ring register. At the next time T ₂ , PE _i becomes X.
_{i + 1} is taken out from the ring registers 11 to 13, w _i and _{i + 1} are taken out from the storage devices 41 to 43, respectively, and multiplied, and the product is added to the accumulator. After that, in the same manner, after T ₄ when the input data X _i makes one round on the ring register bus 1, the product sum Σw _ij X _j is obtained in acc _i . In general, mx
When performing the n-matrix product-sum operation, the product-sum is obtained after T _m by using n arithmetic units. When the number of arithmetic units is less than the number of rows, the input data X _i is rotated on the ring register bus 1 for two or more rounds. If the number of arithmetic units is only 1 / k of the number of rows, it is sufficient to rotate k rounds.
Therefore, sum-of-products is obtained after T _{k. M.}

When calculating W ^T δ, the ring register 11
The internal registers of -14 accumulator acc ₁ ~a
Used as cc ₄ . As shown in FIG. 12, the ring registers 11 to 14 are placed and moved on the ring register bus 1. In this case as well, after T _{4 when} δ _j makes one round on the ring register bus 1 at the input, the sum of products Σ is added to acc _i.
w _IJ δ _j is obtained. In the case of multiply-add operation of m × n matrix, n
By using this number of arithmetic units, the sum of products can be obtained after T _m , and if the number of arithmetic units is only 1 / k of the number of rows, the sum of products can be obtained after T _km rotated k times. , WX
The same as when calculating At this time, each element of W ^T is WX
It is a feature of this network device that it is stored in the same position as when the calculation was performed.

Next, a second conventional example of the ring register type neural network device will be described. Similar to the second conventional example, a network device is configured using a plurality of ring registers, an arithmetic unit, and a memory device, and each diagonal element of the transpose matrix W ^T is stored in the memory device connected to each arithmetic device. Even if the matrix W is mapped so that is at the top, both the matrix-vector product WX and the transposed matrix-vector product W ^T δ can be calculated, as described in IEICE Technical Report NM88-134. There is. However, the actual calculation time when applied to backpropagation becomes long. In backpropagation, 5 cycles of product-sum operation are performed for each learning, and 1 cycle is performed for the transposed matrix.
In the mapping shown in FIG. 10, the matrix calculation in which the accumulator is placed in the ring register is one cycle, whereas in the mapping in which each diagonal element of the transposed matrix W ^T is at the head, the matrix calculation in which the accumulator is placed in the ring register is performed. It becomes 4 cycles. The operation of placing the accumulator in the ring register requires about three times the operation time per step as compared with the operation of placing the accumulator in the arithmetic unit. Therefore, the mapping in which the number of the operations placing the accumulator in the ring register is small is advantageous.

When images and the like are used as input data, the network is a two-dimensional neural network that handles two-dimensional input data. When the ring register type neural network device is applied to the two-dimensional neural network device, the input / output vector and the connection weight are one-dimensionally expanded and used.

[0009]

In a neural network that handles images and the like, it is possible to obtain a sufficient effect even with local connection in which only neighboring neurons are connected instead of complete connection in which all neurons between layers are connected. Are known. By using the locally connected network, the calculation amount of the product-sum calculation is significantly reduced.

However, in the conventional ring register type neural network device, even if a locally connected network with a small amount of product-sum operation is used as the network for the product-sum operation, instead of the fully-connected network, the operation is performed. There was a drawback that the time was not shortened so much.

FIG. 5 is a schematic diagram of a locally connected neural network using one-dimensional data as input data. Figure 5
Indicates a connection state between the input layer 5a and the intermediate layer 6a, the number of neurons in the input layer 5a is N, and the intermediate layer 6a is
The number of neurons in is L. Only n neurons in the input layer 5a are connected to each neuron in the intermediate layer 6a. These n areas are called local areas. The overlap between adjacent local regions is also an important parameter indicating the state of local coupling, and is represented by v in FIG.

FIG. 7 is a schematic diagram of the connection weight matrix W of the one-dimensional locally connected neural network shown in FIG. 5, and this matrix is a sparse matrix. FIG. 8 is a mapping diagram showing a mapping state of this matrix to the storage device of the arithmetic unit. Here, N is the number of neurons in the input layer 5a, and L is the intermediate layer 6a.
, N is the size of the local region. In FIG. 7 and FIG. 8, there are finite-valued matrix elements indicating the coupling load in the shaded area, and in other parts, there is no coupling, that is, the matrix element value is always 0. In the ring register type neural network device, the product-sum operation is performed in parallel for each column of the mapped matrix. When the number of arithmetic units is equal to the number of rows of the matrix W, the minimum number of steps required for the product-sum operation is the quotient of the total number of nonzero matrix elements and the number of rows. However, in the mapping shown in FIG. 8, there is a column in which matrix elements each having a value of 0 and matrix elements each having a finite value coexist, so that a matrix in which the number of steps required for the product-sum operation is not 0 is used. It had the drawback that it would greatly exceed the quotient of the total number of elements and the number of lines. Further, since the number of elements of the connection weight matrix mapped to the storage device greatly exceeds the number of actual connections, there is a disadvantage that the storage device requires a huge storage capacity.

FIG. 6 is a schematic diagram of a neural network when two-dimensional data is used as input data. In FIG. 6, the number of neurons in the input layer 5b is M × N, the number of neurons in the intermediate layer 6b is K × L, and the number of neurons in the local region is m × n. The overlap is u in the X-axis direction and v in the y-axis direction.

When the ring register is applied to the two-dimensional neural network as shown in FIG. 6, the input / output vector and the connection weight are expanded one-dimensionally and used. FIG. 9 is a schematic diagram of the one-dimensionally expanded two-dimensional locally combined connection weight matrix W. This matrix is similar to the matrix W of the one-dimensional locally connected neural network as shown in FIG. That is, the matrix is an sparse matrix in which the N × L matrix indicating the coupling in the X-axis direction in FIG. In the case of two dimensions, since the matrix elements with the value 0 are distributed more than in the case of the one-dimensional locally connected neural network, the number of steps required for the product-sum calculation is the quotient of the total number of matrix elements and the number of rows The degree to exceed is even more remarkable.

Further, in the ring register type neural network device, the input data is normally supplied from one end of the ring register bus while rotating on the ring register bus step by step. Therefore, there is a drawback that it takes a lot of time to set the input data.

It is an object of the present invention to provide a neural network device capable of performing backpropagation operation of a locally connected neural network at extremely high speed and with a small memory capacity.

[0017]

In order to achieve the above object, at least a ring register bus constituted by connecting a plurality of ring registers having a transfer function in a ring shape and at least some of the ring registers. In a neural network device including a plurality of arithmetic devices connected to each one and a plurality of storage devices connected to each of the arithmetic devices, the ring register bus is connected to the arithmetic device. A ring register,
A neural network device is configured including a delay element inserted between the ring registers connected to the arithmetic unit.

To achieve the above object, the network to be operated is a one-dimensional local connection, the number of neurons in an arbitrary first layer is N, and the number of neurons in the second layer of the first next stage is L. When the number of neurons in the local region of the first layer connected to one neuron of the second layer is N and the number of neurons overlapping between adjacent local regions is u, the delay amount of the delay element Is the size of the local region n
And a value obtained by subtracting 1 from the difference in the overlap v between the adjacent local areas and the time T required for the data arranged on the ring register bus to rotate one step on the ring register bus, (Nv-1) T to set the neural network device.

To achieve the above object, the network to be operated is a two-dimensional local connection, the number of neurons in the x-axis direction is N, the number of neurons in the y-axis direction is M, and the first layer is arbitrary in the first layer. The number of neurons in the x-axis direction of the second layer next to the layer of L is L, the number of neurons in the y-axis direction is K, and the second layer
The number of neurons in the x-axis direction in the local region of the first layer connected to one neuron in the layer of n is n, the number of neurons in the y-axis direction is m, and x overlaps between adjacent local regions.
When the number of neurons in the axial direction is v and the number of neurons in the y-axis direction is u, L ring registers to which the arithmetic unit is connected are alternately connected to (L-1) first delay elements. Form K ring register groups, insert a second delay element between the ring register groups, and make the delay amount of the first delay element adjacent to the size n of the local region in the x-axis direction. The product of a value obtained by subtracting 1 from the difference in the overlap v in the x-axis direction between the local regions and the time T required for the data arranged on the ring register bus to rotate one step on the ring register bus. , (N−v−1) T, and the delay amount of the second delay element is the difference between the size m of the local region in the y-axis direction and the overlap u in the y-axis direction between adjacent local regions. The value obtained by subtracting 1 from N and the value multiplied by N are distributed on the ring register bus. Time T data is required for one step rotation of said ring register on the bus
And the delay amount in the first delay element group, {N (m−u−1) + (n−v−1)} T, to construct a neural network device.

To achieve the above object, at least one ring register bus is constructed by connecting a plurality of ring registers having a transfer function in a ring shape, and at least one ring register bus is connected to each of the ring registers. In a neural network device including a plurality of arithmetic units and a plurality of storage devices connected to each of the arithmetic units, a plurality of the ring register buses are arranged to form a neural network unit.

In order to achieve the above-mentioned object, the data held in the ring register is collectively stored in each ring register bus at any rotation position on the ring register bus and stored in at least one other ring register bus. It is possible to transfer to each ring register.

[0022]

By inserting the delay element into the ring register bus, it becomes possible to perform mapping in which the matrix elements having the value 0 are aggregated in a specific column. Therefore, the number of times required for calculation is shortened. In addition, the capacity of the storage device that stores the coupling load can be saved.

While performing a sum of products operation using one ring register bus, input data can be set using another ring register bus. Therefore, the time required to set the input data can be ignored.

[0024]

Embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the first embodiment of the present invention. In the present embodiment, the product-sum calculation of a locally connected network having 7 neurons in the input layer, 3 neurons in the intermediate layer, and 3 local regions is targeted.

The ring register bus 1 and the ring register bus 2 are provided with arithmetic units 11 to 13 and variable delay units first-in-first-out memory (FIFO) 51 and 52, arithmetic units 21 to 23 and variable delay unit, respectively. First-in first-out memories 61 and 62 are alternately arranged, and a variable delay device first-in-first
A double ring register bus is constructed with the out memory 50. The delay in the variable delay device first-in-first-out memory 51, 52, 61, 62 is a value obtained by subtracting 1 from the difference between the size n of the local area and the overlap u between adjacent local areas (n- u-1), that is, the product of 1 and the time T required for the data arranged on the ring register buses 1 and 2 to rotate one step on the ring register buses 1T. The arithmetic units 31 to 33 are connected to the ring registers 11 to 13 on the ring register bus 1, respectively.
The ~ 33 stores the connection weights W _ij to keep the storage device 41
~ 43 are connected. Data rotating on the ring register bus 2 can be collectively transferred onto the ring register bus 1 at an arbitrary rotation position.

The operation for performing the following calculation will be described with reference to FIGS. 3 and 4.

[0027]

[Equation 1]

FIG. 3 is a timing chart showing the operation in the calculation of the product WX of the matrix and vector, and FIG. 4 is a timing chart showing the operation in the calculation of the product W ^T δ of the transposed matrix and vector.

When calculating WX, as shown in FIG.
Each element of the above moves on the ring register bus 1. Registers in the arithmetic units 11 to 13 are used as accumulators acc.
Used from ₁ to acc ₃ . At time T ₁ , the arithmetic unit 31 fetches x ₁ from the ring register 11 and w ₁₁ from the storage unit 41 and multiplies them, and adds the product to the accumulator acc ₁ . Similarly, the arithmetic unit 32 takes x ₃ from the ring register 12 and w ₂₃ from the storage unit 42 and multiplies them, adds the product to the accumulator acc ₂ , and the arithmetic unit 33 puts x ₅ from the ring register 13 into w.
Each ₃₅ is taken out from the storage device 43 and multiplied, and the product is added to the accumulator acc ₃ . Immediately after this, the ring rotates counterclockwise by one ring register. At the next time T ₂ , the arithmetic unit 31 sets x ₂ to the ring register 11
, W ₁₂ are taken out from the storage device 41 and multiplied, and the product is added to the accumulator acc ₁ . Similarly, the product sum Σw _ij x _j is obtained for acc _i after T ₃ .

When calculating W ^T δ, the ring register 11
~ 14 and variable delay first in first
Registers inside the out memories 51 to 54 are used as accumulators acc ₁ to acc ₇ . As shown in FIG. 4, the accumulators acc _{1 to} acc ₄ are moved on the ring register bus 1. In this case as well, T ₃
After, the sum of products Σ w _ij δ _j is obtained in acc _i .

By rotating the input data on the ring register bus 2 while performing the multiply-accumulate operation using the ring register bus 1, the input data is transferred from the first-in-first-out memory 50 to the ring register. Set on bus 2. When the product-sum calculation is completed and the next product-sum calculation is to be performed, the input data on the ring register bus 2 is transferred to the ring register bus 1 all at once.

Next, a second embodiment will be shown. FIG. 2 is a block diagram showing the second configuration of the present invention. In the case of two-dimensional local coupling, a ring register bus (but not circular connection) corresponding to the coupling in the x-axis direction in FIG. 6 is configured, and this is cascaded by the dimension number M of the input signal vector in the y-axis direction. Connected.

First In First Out
The memory delay is the first first-in-first-out memory 511, 512, ..., 5M (R-
1), 611, 612, ..., 6M (R-1), a value obtained by subtracting 1 from the difference between the size n of the local region in the x-axis direction and the overlap v in the x-axis direction between adjacent local regions. And the time T required for the data arranged on the ring register bus 1 to rotate one step on the ring register bus 1 are set to (n−v−1) T. Further, the second first-in-first-out memories 51, 52, ..., 5 (M-
1), 61, 62, ..., 6 (M−1), a value obtained by subtracting 1 from the difference between the size m of the local region in the y-axis direction and the overlap u of the adjacent local regions in the y-axis direction. Is multiplied by N and the time T required for the data arranged on the ring register bus 1 to rotate on the ring register bus 1 by one step, and the first first-in-first-out memory 511. , 512, ..., 5M (R-1), 6
The sum of the delay amounts of 11, 612, ..., 6M (R-1), {N (mu-1) + (nv-1)} T, is set.

The second example was applied to the alphanumeric recognition of 9 × 11 pixels. In this neural network, both the input layer and the intermediate layer are two-dimensional, and the number of neurons is
9 × 11 and 3 × 4. The input layer and the intermediate layer are locally coupled, and the size of the local region is 5 × 5. The middle layer and the output layer are completely connected, and the number of neurons in the output layer is 5. A 32-bit floating point arithmetic DSP: TMS320C30 manufactured by TI Co. is used as an arithmetic unit.
This DSP is equipped with one adder and one multiplier, and the product-sum operation can be executed in one machine cycle. The capacity of the internal RAM is 1 kW × 2. Machine cycle and clock cycle are 60ns and 3 respectively.
It is 0 ns. In the conventional ring register type neural network device configured by using the same DSP, the learning speed is 15.4 MCUPS, whereas when the second embodiment is used, it becomes 28.9 MCUPS, and the learning speed is about 1.9 times higher.

The neural network device of the present invention is particularly effective for a two-dimensional net having a large number of hidden neurons, with a small overlap between local regions, and it is possible to increase the speed by 10 times or more and reduce the storage device capacity. is there. Therefore, the present invention is applicable not only to a normal three-layered neural network, but also to Biological Cybernetics.
It is particularly effective when there is little overlap between local regions in a multistage net such as neocognitron described in Vol. 36 (1980), pages 193 to 202.

[0036]

As described above, according to the present invention, there is provided a neural network apparatus capable of performing backpropagation operation of a locally connected neural network at an extremely high speed and with a small memory capacity. It is possible and extremely effective.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a first embodiment.

FIG. 2 is a block diagram showing a configuration of a second exemplary embodiment.

FIG. 3 shows the product W of a matrix and a vector in the first embodiment.
FIG. 6 is a timing chart showing an operation when X is calculated.

FIG. 4 is a timing chart showing an operation in calculating a product W ^T δ of a transposed matrix and a vector in the first embodiment.

FIG. 5 is a schematic diagram of a one-dimensional locally connected network.

FIG. 6 is a schematic diagram of a two-dimensional locally connected network.

FIG. 7 is a schematic diagram of a connection weight matrix W of a one-dimensional locally connected neural network.

FIG. 8 is a mapping diagram showing a mapping state of a connection weight matrix W of a one-dimensional locally connected neural network onto a storage device.

FIG. 9 is a schematic diagram of a connection weight matrix W of a one-dimensional expanded two-dimensional locally connected neural network.

FIG. 10 is a block diagram showing a configuration of a conventional example.

FIG. 11 is a timing chart showing an operation when a product WX of a matrix and a vector is calculated in a conventional example.

FIG. 12 is a product W of a transposed matrix and a vector in the conventional example.
FIG. 6 is a timing chart showing an operation when calculating ^T δ.

[Explanation of symbols]

1, 2 Ring register bus 5a, 5b Input layer 6a, 6b Intermediate layer 11-15, 111-111MR, 21-23, 211
-2MR ring register 31-33, 311-3MR arithmetic device 41-43, 411-4MR storage device 50, 51-5 (M-1), 61-6 (M-1), 51
1-5M (R-1), 611-6M (R-1) variable delay first-in-first-out memory

Claims (5)

[Claims] 1. A ring register path configured by connecting a plurality of ring registers having a transfer function in a ring shape, and a plurality of arithmetic units connected to at least one of the ring registers. A neural network device including a plurality of storage devices connected to each of the arithmetic devices, wherein the ring register path includes the ring register connected to the arithmetic device, and the ring register connected to the arithmetic device. A neural network device having a delay element inserted between them. 2. The neural network device according to claim 1, wherein the network to be operated is one-dimensional local connection, the number of neurons in an arbitrary first layer is N, and the number of neurons in the second stage next to the first layer is N. The number of neurons in the layer is L, the number of neurons in the local region of the first layer connected to one neuron in the second layer is n, and the number of neurons overlapping between adjacent local regions is u. , The delay amount of the delay element is a value obtained by subtracting 1 from the difference between the size n of the local area and the overlap v between the adjacent local areas, and the data arranged on the ring register path is A product of (n−v−1) T and a time T required to rotate the upper part by one step, which is (n−v−1) T. 3. The neural network device according to claim 1, wherein the network to be operated is a two-dimensional local connection, and the number of neurons in the x-axis direction of an arbitrary first layer is N and the number of neurons in the y-axis direction is M, the number of neurons in the x-axis direction of the second layer next to the first layer is L, the number of neurons in the y-axis direction is K, and the first neuron connected to one neuron of the second layer The number of neurons in the x-axis direction in the local region of the layer is n,
When the number of neurons in the y-axis direction is m, the number of neurons in the x-axis direction overlapping between the adjacent local regions is v, and the number of neurons in the y-axis direction is u, the ring register connected to the arithmetic unit is set to L. Each of them is alternately connected to the (L-1) first delay elements and K
Each ring register group is configured, and the second delay element is inserted between the ring register groups, and the delay amount of the first delay element is adjacent to the size n of the local region in the x-axis direction. The product of a value obtained by subtracting 1 from the difference in the overlap v in the x-axis direction between the local regions and the time T required for the data arranged on the ring register bus to rotate one step on the ring register bus. , (N−v−1) T, and the delay amount of the second delay element is calculated from the difference between the size m of the local region in the y-axis direction and the overlap u in the y-axis direction between adjacent local regions. A value obtained by multiplying the value obtained by subtracting 1 by N and the time T required for the data arranged on the ring register bus to rotate one step on the ring register bus, and the delay in the first delay element group. Sum with quantity, {N (mu-1) + ( -v-1)} neural network system, characterized in that the T. 4. A ring register bus configured by connecting a plurality of ring registers having a transfer function in a ring shape, and a plurality of arithmetic units connected to at least one of the ring registers. A neural network device including a plurality of storage devices connected to each of the arithmetic units, wherein the neural network device has a plurality of ring register buses. 5. The neural network device according to claim 4, wherein the data held in the ring register is collectively stored for each ring register bus at any rotation position on the ring register bus. A neural network device capable of transferring to each ring register of at least one ring register bus.