CN108647773A - A kind of hardwired interconnections framework of restructural convolutional neural networks - Google Patents

Abstract

The invention belongs to the hardware design technique field of image processing algorithm, specially a kind of hardwired interconnections framework of restructural convolutional neural networks.The present invention interconnection architecture include：Data and the outer cache module of parameter piece, for caching, the pixel data in the pending picture inputted is gentle to deposit into the parameter inputted when row convolutional neural networks calculate；Basic calculation cell array module, for realizing the core calculations of convolutional neural networks；Arithmetic logic unit computing module, the result of calculation for handling the basic calculation cell array are realized to down-sampling layer, activation primitive and part and add up.Wherein, basic calculation cell array module interconnects in the way of two-dimensional array, in the row direction, shares input data, and parallel computation is realized by using different supplemental characteristics；In a column direction, result of calculation is transmitted line by line, and the input as next line participates in operation.While the present invention interconnects promotion data-reusing ability by structure, the demand of bandwidth can be reduced.

Description

A Hardware Interconnect Architecture for Reconfigurable Convolutional Neural Networks

technical field

The invention belongs to the technical field of hardware design of image processing algorithms, and in particular relates to a hardware interconnection architecture of a reconfigurable convolutional neural network.

Background technique

With the rise of artificial intelligence, deep learning has been widely used in computer vision, speech recognition and other big data applications, and has been more and more widely used. As an important algorithm model in deep learning, convolutional neural network has been widely used, such as in image classification, face recognition, video detection, speech recognition, etc.

As the accuracy of image recognition improves, convolutional neural networks become more complex and require more computation. This makes traditional general-purpose processors with a large number of redundant resources have low performance when processing large-scale convolutional neural networks, which cannot meet the actual needs in many scenarios. Therefore, using hardware platforms to accelerate convolutional neural networks in engineering has gradually become a mainstream method. If a parallel pipeline hardware architecture is used to implement the algorithm, a good acceleration effect can be obtained to achieve real-time processing.

Although the performance of the hardware platform is much higher than that of traditional general-purpose processors, as the network structure of the convolutional neural network becomes increasingly complex and needs to face convolution kernels of various sizes, the low flexibility of the hardware structure will make the hardware processing The controller only improves the acceleration effect of some specific networks, but the acceleration efficiency of other networks is not high. Therefore, the demand for a reconfigurable convolutional neural network hardware architecture that can be applied to convolution kernels of any size and various network structures is becoming more and more urgent.

For the reconfigurable convolutional neural network hardware implementation, the current main research difficulties have two aspects, one is how to use a fixed hardware structure to configure convolution kernels of different sizes, and the other is how to make the hardware structure for any convolution kernel can achieve high efficiency. In the existing technology, first design a hardware structure that implements the most common 3×3 convolution template as the basic unit, and then transfer data through the interconnection between the basic units, and use multiple basic units to realize other types of convolutions. The product kernel, for example, requires 5 basic units to realize three 5×5 convolution templates. In this scheme, since the number of basic units in the hardware is fixed but the number of basic units required to implement a convolution kernel is indeterminate, some basic units will not work when implementing a convolution of a certain size. 100% utilization cannot be guaranteed. And under this scheme, not all basic units can generate effective output, and all output needs to be cached first, and then effective data is selected, resulting in a waste of a large amount of additional storage resources.

In the prior art, many image processing hardware accelerators design an optimal hardware structure for a certain algorithm, and then support other algorithms through interconnection. Due to the diversity of algorithms and the limitation of fixed hardware scale, when the network model changes or the type of calculation changes or the type of calculation changes, the utilization rate of resources and computing performance will be reduced. Moreover, in the field of image processing, especially in neural network algorithms, in addition to the utilization of computing units, bandwidth is also a key factor affecting computing performance.

There is an urgent need for a method that can reduce bandwidth requirements while improving data multiplexing capabilities through structural interconnection.

Contents of the invention

The object of the present invention is to provide a hardware interconnection architecture of a reconfigurable convolutional neural network capable of reducing bandwidth requirements while improving data multiplexing capability through structural interconnection.

The hardware interconnection architecture of the reconfigurable convolutional neural network provided by the present invention is applied in the field of image processing, and the hardware interconnection architecture includes:

The data and parameter off-chip cache module is used to cache the input pixel data in the picture to be processed and the parameters input when the convolutional neural network is calculated;

The basic computing unit array module is respectively connected with the data off-chip cache module and the parameter off-chip cache module; the basic computing unit array module is used to realize the core calculation of the convolutional neural network; the basic computing unit array module Interconnected in the form of a two-dimensional array, in the row direction, the basic computing unit array shares input data, and achieves parallel computing by using different parameter data; in the column direction, the calculation results of the basic computing unit array are passed row by row as the next row The input participates in the operation;

An arithmetic logic unit calculation module connected to the basic calculation unit array module, the arithmetic logic unit calculation module is used to process the calculation results of the basic calculation unit array, and realize the downsampling layer, activation function and partial sum accumulation;

When calculating the convolutional neural network, when each column of the basic computing unit array module independently calculates an output feature map, the convolution kernels of different sizes are mapped by dividing the input feature map into multiple inputs according to the convolution window position;

The control module is respectively connected with the basic calculation unit array module and the arithmetic logic unit calculation module, and the control module is used to realize a convolution kernel of any size and various calculation modes according to different parameters.

Optionally, when the arithmetic logic unit calculation module and the basic calculation unit array module calculate the convolutional neural network, only one parameter of the convolution kernel is required for each input feature map after segmentation.

Optionally, the basic computing unit array module is internally composed of interconnections, which can be used to support different types of calculations with different bit widths by changing the internal interconnection mode and data path of the basic computing unit array module.

Optionally, the hardware interconnection architecture is realized by any one of a field programmable gate array and an integrated circuit.

According to the specific embodiments provided by the present invention, the present invention has the following technical effects: the present invention uses a new mapping method to map the convolutional neural network, and converts the degrees of freedom of different neural network models from the spatial level to the time level, so that the fixed hardware The interconnect architecture is capable of maintaining 100 percent resource utilization for any convolutional neural network. At the same time, the basic calculation unit module fully utilizes the digital signal processing unit, and adopts a flexible interconnection structure to support different calculation modes and variable data bit width calculation requirements. In addition, the parallel calculation of the pipeline between rows and the multiplexing of input data between columns reduce the demand for input data and output bandwidth.

Description of drawings

FIG. 1 is a schematic structural diagram of the hardware interconnection architecture of the convolutional neural network provided by the present invention.

FIG. 2 is a schematic diagram of specific mapping in the mapping manner provided by the present invention.

Fig. 3 is a schematic diagram of the internal structure of the basic computing module provided by the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Apparently, the drawings are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort. The described embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments described in the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

The purpose of the present invention is to provide a hardware interconnection architecture of a convolutional neural network capable of reducing bandwidth requirements while improving data multiplexing capability through structural interconnection.

A hardware interconnection architecture of a reconfigurable convolutional neural network, the interconnection architecture is applied in the field of image processing, and the interconnection architecture includes:

The data and parameter off-chip cache module is used to cache the pixel data in the input picture to be processed and the parameters input during the convolutional neural network calculation;

The basic computing unit array module is respectively connected with the data off-chip cache module and the parameter off-chip cache module, and the basic computing unit array module is used to realize the image processing core - the calculation of the convolutional neural network. The modules are interconnected in a two-dimensional array. In the row direction, the basic computing unit array modules share input data, and achieve parallel computing by using different parameter data; in the column direction, the calculation results of the basic computing modules are row by row Pass, participate in the operation as the input of the next line;

When calculating the convolutional neural network, when each column of the basic computing unit array module independently calculates an output feature map, the convolution kernels of different sizes are mapped by dividing the input feature map into multiple inputs according to the convolution window position;

The arithmetic logic unit calculation module is connected to the basic calculation module, and the arithmetic logic unit calculation module is used to process the calculation results of the basic calculation unit array, and realize the downsampling layer, activation function and partial sum accumulation;

The control module is respectively connected with the basic calculation module and the arithmetic logic unit calculation module, and the control module is used to realize convolution kernels of any size and multiple calculation modes according to different parameters, and can be used for different image processing methods. Design efficient and flexible computing units and interconnection structures for network-structured convolutional neural networks, distance calculations, vector calculations, and digital signal processing calculations.

Optionally, when the ALU calculation module and the basic calculation module calculate the convolutional neural network, only one parameter of the convolution kernel is required for each segmented input feature map.

Optionally, the basic computing modules are internally composed of interconnections, which can be used to support different types of computing with different bit widths by changing the internal interconnection methods and data paths of the basic computing modules.

Optionally, the architecture is realized by any one of a field programmable gate array and an integrated circuit.

As shown in Figure 1, the data off-chip cache module and the parameter off-chip cache module 1 are configured such that the externally input image data and parameters are first cached in the module and wait to be read out by the basic computing unit array module ; and the calculated results are cached in the module, waiting to be called again by the arithmetic logic unit calculation module or waiting for output. The data and parameter off-chip cache module is equivalent to a larger field programmable gate array, which is used to deal with the situation that the processing speed of the basic computing unit array module is greatly different from the external input speed, otherwise if the external directly sends data to the internal The basic computing unit array module will need more complex handshake signals to ensure that there is no data leakage or retransmission when the speeds of the two do not match.

The basic computing unit array module 2 connected to the data parameter off-chip cache module 1 is used to realize the core calculation of the convolutional neural network, and the interconnection mode between the basic computing unit arrays is shown in FIG. 1 . The interconnection architecture consists of multi-stage pipelines, and the function of each pipeline is the same, which is used to calculate the multiplication of the input feature map and the parameters, and add the partial sum to the result obtained by the previous pipeline. The input data is calculated from the first stage of the pipeline at the front, and it goes backward step by step until the last stage of the pipeline is stored in the off-chip cache module. When the number of input feature maps is greater than the number of rows of the calculation array, these calculations can be The results are sent back from the off-chip cache module to the arithmetic logic unit calculation module to continue to complete the accumulation to obtain the final output feature map. When the first feature map starts to be calculated for one cycle in the first-stage pipeline, the second feature map will start to be calculated in the second-stage pipeline, which makes the result of the first-stage pipeline just enough to be sent to the second-stage pipeline Participating in the addition operation, when the pipeline is fully filled, a higher degree of parallelism and higher processing performance can be achieved. In order to make the pipeline calculation more efficient, the number of rows/columns of the calculation array should be a factor of the number of input feature maps/number of output feature maps as much as possible.

The arithmetic logic unit calculation module 3 connected to the basic calculation unit array module 2 is used to realize the partial sum and accumulation of the convolution, the down-sampling layer and the activation function layer. The specific configuration is that in the convolutional layer, after the calculation result obtained by the last line of the calculation array is stored in the off-chip cache module, if it is still a partial sum, these partial sums will be read from the off-chip cache module and sent to the off-chip cache module. In the arithmetic logic unit calculation module and the calculation results of the last row of the next group of calculation arrays are further accumulated, and then stored in the off-chip cache module to realize the accumulation of the calculation results of multiple sets of input feature maps; in the down sampling layer, the arithmetic logic unit The calculation module will read the output feature map of the previous convolutional layer from the off-chip cache module, and select its function as a comparator or an adder according to whether the downsampling layer is the maximum value or the average value to realize the downsampling operation; in the activation function In the layer, in this example, it is a Relu function, and the arithmetic logic unit calculation module will use a data selector Mux to realize the activation function.

The hardware interconnection architecture of the reconfigurable convolutional neural network of the present invention can effectively support distance calculation, linear algebra calculation and image signal processing calculation in addition to being used for calculating the convolutional neural network.

The control module 4 connected with the above-mentioned modules is used to generate control signals of other modules. Its specific configuration is to accept and decode external instructions, generate enable signals and addresses for the off-chip cache module to read and write data off-chip, and for the computing array to read and write data to the off-chip cache module, and generate computing units and the arithmetic logic Data and function strobe signals in the cell computing module.

In the above-mentioned basic computing unit array 2, it is composed of multi-stage pipelines. When mapping, it extracts the image data in the same position of the convolution template (that is, the same corresponding parameters) in the convolution sliding window to form a new input feature map, which is equivalent to dividing the convolution template into multiple 1x1 volumes The original convolution result can be obtained by accumulating the results calculated by using the segmented convolution template. Taking the 5x5 convolution template as an example, its mapping method is shown in Figure 2. In this way, convolution templates of any size can be divided into 1x1 convolution templates, so this structure can maintain high efficiency for convolution kernels of any size. Secondly, the pipeline structure between the rows reduces the number of output ports from the number of computing units to the number of columns of the computing array while efficiently utilizing the DSP function, which makes the calculation results of the computing units except the last row not require additional caching space, reducing the need for on-chip storage resources.

In the arithmetic logic unit calculation module 3 above, it connects the downsampling layer, the activation function layer and the convolution layer together. Since the downsampling layer and the activation function layer do not require parameters and the operation is simple, in order to reduce the external data bandwidth requirements and improve computing efficiency, this architecture does not map these two layers into the computing array, but sends them before the output of the convolutional layer results. into the calculation module of the arithmetic logic unit to complete the operation of comparison or addition or data selection. The specific method is to configure the arithmetic logic unit calculation module as a comparator when realizing the maximum value; to remove the denominator from the weight in the upper convolutional layer in advance when realizing the average value, so that the average value only needs to be The ALU calculation module can be configured as an adder; when the activation function Relu is implemented, the ALU calculation module can be configured as a data selector Mux.

In the arithmetic and logic unit calculation module 4 mentioned above, the control signals generated by it for controlling the calculation array are also transmitted layer by layer along with the data so as to match the calculation timings of the pipelines at all levels. And after the multi-stage delay of the control signal, the number of computing units that need to be controlled by the same control signal is reduced from the number of computing units to the number of columns, which reduces the fan-out of the control signal and can increase the operating frequency of the chip.

The invention adopts a new interconnection structure and a new mapping method to realize a reconfigurable convolutional neural network hardware architecture. It can achieve 100% resource utilization for convolution kernels of any size, achieving higher performance and less hardware resource consumption.

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other.

In this paper, specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; meanwhile, for those of ordinary skill in the art, according to the present invention Thoughts, there will be changes in specific implementation methods and application ranges. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims (7)

1. A hardware interconnection architecture of a reconfigurable convolutional neural network, applied in the field of image processing, characterized in that, the hardware interconnection architecture includes: The data and parameter off-chip cache module is used to cache the pixel data in the input picture to be processed and the parameters input during the convolutional neural network calculation; The basic computing unit array module is respectively connected to the data and parameter off-chip cache modules; the basic computing unit array module is used to realize the core calculation of the convolutional neural network; the basic computing unit array module is in the form of a two-dimensional array Interconnection, in the row direction, the basic computing unit array shares input data, and achieves parallel computing by using different parameter data; in the column direction, the calculation results of the basic computing unit array are passed row by row, and participate in the operation as the input of the next row; An arithmetic logic unit calculation module connected to the basic calculation unit array module, the arithmetic logic unit calculation module is used to process the calculation results of the basic calculation unit array, and realize the downsampling layer, activation function and partial sum accumulation; When calculating the convolutional neural network, when each column of the basic computing unit array module independently calculates an output feature map, the convolution kernels of different sizes are mapped according to the convolution window position to divide the input feature map into multiple inputs; The control module is respectively connected with the basic calculation unit array module and the arithmetic logic unit calculation module, and the control module is used to realize a convolution kernel of any size and various calculation modes according to different parameters. 2. The hardware interconnection architecture of the reconfigurable convolutional neural network according to claim 1, wherein when the arithmetic logic unit computing module and the basic computing unit array module are computing the convolutional neural network, they are divided into After each input feature map, only one parameter of the convolution kernel is required. 3. The hardware interconnection architecture of the reconfigurable convolutional neural network according to claim 1, wherein the interconnection mode is adopted inside the basic computing unit array module, and by changing the internal The interconnection methods and data paths are used to support calculations of different types and different bit widths. 4. The hardware interconnection architecture of the reconfigurable convolutional neural network according to claim 1, characterized in that, it is realized by any one of a field programmable gate array and an integrated circuit. 5. The hardware interconnection architecture of the reconfigurable convolutional neural network according to claim 1, wherein the basic computing unit array modules are interconnected in the form of a two-dimensional array, and its interconnection architecture consists of a multi-stage pipeline Composition, the function of each stage of the pipeline is the same, it is used to calculate the multiplication of the input feature map and the parameter, and add the partial sum to the result obtained by the previous pipeline; the input data is calculated from the front first pipeline , step by step backward until the last stage of the pipeline is stored in the off-chip cache module; when the number of input feature maps is greater than the number of rows of the calculation array, these calculation results are sent back from the off-chip cache module to the arithmetic logic unit calculation module Continue to complete the accumulation in order to obtain the final output feature map; when the first feature map starts to be calculated for one cycle in the first-stage pipeline, the second feature map starts to be calculated in the second-stage pipeline, so that the first-stage pipeline gets The result can just be sent to the second-stage pipeline to participate in the addition operation; when the pipeline is fully filled, a higher degree of parallelism and higher processing performance can be achieved. 6. The hardware interconnection architecture of the reconfigurable convolutional neural network according to claim 1, wherein the arithmetic logic unit calculation module is used to realize the convolutional part and accumulation, down-sampling layer and activation function layer ; Its specific configuration is as follows: In the convolutional layer, after the calculation result obtained by the last line of the calculation array is stored in the off-chip cache module, if it is still a partial sum, these partial sums are read from the off-chip cache module and sent to the off-chip cache module. In the arithmetic logic unit calculation module and the calculation results of the last row of the next group of calculation arrays are further accumulated, and then stored in the off-chip cache module to realize the accumulation of the calculation results of multiple sets of input feature maps; in the down sampling layer, the arithmetic logic unit The calculation module reads the output feature map of the previous convolutional layer from the off-chip cache module, and selects its function as a comparator or an adder according to whether the downsampling layer is the maximum value or the average value to realize the downsampling operation; in the activation function layer In, the arithmetic logic unit calculation module uses a data selector Mux to realize the activation function. 7. The hardware interconnection architecture of the reconfigurable convolutional neural network according to claim 1, wherein the control module is used to generate control signals of the remaining modules; its specific configuration is: accepting external instructions and translating code, generate the off-chip cache module to read and write data off-chip and the enable signal and address of the calculation array to read and write data to the off-chip cache module, and generate the data and function strobe signals in the calculation unit and the arithmetic logic unit calculation module .