CN111353591B - Computing device and related product - Google Patents

Abstract

The application provides a computing device and a related product, wherein the computing device comprises a compression unit, an operation unit and a controller unit; the controller unit is used for acquiring a compression request for the first input data and indicating the compression unit to compress the first input data according to the compression request; wherein the first input data comprises a first weight matrix; the compression unit is used for compressing the first weight matrix into a second weight matrix; and the controller unit is also used for executing the neural network calculation according to the second input data and the calculation instruction. According to the application, in the compression process of the neural network, the topology structure of the neural network model can be kept unchanged, so that the occurrence of irregularity of the topology structure of the neural network model is avoided, and the operation amount of the neural network is reduced.

Description

A computing device and related products

Technical Field

The present application relates to the field of information processing technology, and in particular to a computing device and related products.

Background Art

A neural network is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. This network is composed of a large number of nodes (or neurons) that are connected to each other. It simulates the information processing process of the human brain by adjusting the interconnected relationships between a large number of internal nodes and using input neuron data and weights to generate output data to process information and generate results after pattern recognition.

At present, neural networks are widely used in various fields of computer vision, such as image recognition, object detection, image segmentation, etc. However, in practical applications, neural network models often have a large number of model parameters (for example, ultra-large-scale weights). In this case, this means that the neural network requires a large amount of computing resources and storage resources. The overhead of a large amount of computing resources and storage resources will reduce the computing speed of the neural network, and the requirements for the transmission bandwidth and operator of the hardware will also be greatly increased. Therefore, how to reduce the computational complexity of the neural network while reducing the parameters of the neural network model becomes very important.

In the prior art, the parameters of the neural network model are adjusted by pruning methods to reduce the parameters of the neural network model and reduce the amount of calculation of the neural network. Taking the pruning of the weights of the neural network as an example, as shown in FIG1A , before the weights of the neural network are pruned, the topological structure of the neural network is regular. However, after the weights of the neural network are pruned, the original regular topological structure in the neural network model is easily caused to become irregular. Therefore, how to prevent the topological structure in the neural network model from becoming irregular is a technical problem that needs to be solved urgently.

Summary of the invention

The embodiments of the present application provide a computing device and related products, which can ensure that the topological structure of the neural network model remains unchanged during the neural network compression process, thereby avoiding the irregularity of the topological structure of the neural network model and reducing the amount of computation of the neural network.

In a first aspect, a computing device is provided, the computing device being used to perform machine learning calculations of a machine learning model, the computing device comprising: a compression unit, an operation unit, and a controller unit;

The controller unit is configured to obtain a compression request for first input data, and instruct the compression unit to compress the first input data according to the compression request; wherein the first input data includes a first weight matrix;

The compression unit is used to compress the first weight matrix into a second weight matrix;

The controller unit is further used to obtain second input data and calculation instructions; the second input data includes the second weight matrix and input neuron data;

The controller unit is further configured to parse the calculation instruction to obtain a plurality of operation instructions, and send the plurality of operation instructions and the second input data to the operation unit;

The operation unit obtains the operation instruction and performs neural network calculation according to the operation instruction and the second input data.

Through the present application, the first weight matrix can be compressed by a compression unit to obtain a second weight matrix, and then the neural network calculation can be performed according to the second weight matrix and the input neuron data, which solves the problem that the topological structure of the neural network is easily irregular when the neural network pruning algorithm is used in the prior art. The neural network can be deeply compressed, the calculation amount of the neural network can be reduced, and the operation speed can be improved.

In a second aspect, an embodiment of the present application provides a machine learning computing device, which includes one or more computing devices described in the first aspect. The machine learning computing device is used to obtain data to be computed and control information from other processing devices, and perform specified machine learning operations, and transmit the execution results to other processing devices through an I/O interface;

When the machine learning computing device includes a plurality of computing devices, the plurality of computing devices may be linked and transmit data through a specific structure;

Among them, multiple computing devices are interconnected and transmit data through a PCIE bus to support larger-scale machine learning operations; multiple computing devices share the same control system or have their own control systems; multiple computing devices share memory or have their own memory; the interconnection method of multiple computing devices is any interconnection topology.

In a third aspect, an embodiment of the present application provides a combined processing device, which includes a machine learning processing device, a universal interconnection interface, and other processing devices as described in the third aspect. The machine learning operation device interacts with the above-mentioned other processing devices to jointly complete the operation specified by the user. The combined processing device may also include a storage device, which is respectively connected to the machine learning operation device and the other processing device, and is used to save data of the machine learning operation device and the other processing device.

In a fourth aspect, an embodiment of the present application provides a neural network chip, which includes the computing device described in the first aspect, the machine learning computing device described in the second aspect, or the combined processing device described in the third aspect.

In a fifth aspect, an embodiment of the present application provides a neural network chip packaging structure, which includes the neural network chip described in the fourth aspect above.

In a sixth aspect, an embodiment of the present application provides a board card, which includes the neural network chip packaging structure described in the fifth aspect above.

In a seventh aspect, an embodiment of the present application provides an electronic device, which includes the neural network chip described in the sixth aspect or the board described in the sixth aspect.

In an eighth aspect, an embodiment of the present application further provides a calculation method for executing a machine learning model, wherein the calculation method is applied to a computing device, and the computing device is used to perform machine learning calculations; the computing device includes: a compression unit, a computing unit, and a controller unit; the method includes:

The controller unit obtains a compression request for first input data, and instructs the compression unit to compress the first input data according to the compression request; wherein the first input data includes a first weight matrix;

The compression unit is used to compress the first weight matrix into a second weight matrix;

The controller unit is further used to obtain second input data and calculation instructions; the second input data includes the second weight matrix and input neuron data;

The controller unit is further configured to parse the calculation instruction to obtain a plurality of operation instructions, and send the plurality of operation instructions and the second input data to the operation unit;

The operation unit is used to obtain the operation instruction and perform neural network calculation according to the operation instruction and the second input data.

In some embodiments, the electronic device includes a data processing device, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a mobile phone, a driving recorder, a navigator, a sensor, a camera, a server, a cloud server, a camera, a camcorder, a projector, a watch, a headset, a mobile storage, a wearable device, a vehicle, a household appliance, and/or a medical device.

In some embodiments, the means of transportation include airplanes, ships and/or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes magnetic resonance imaging, ultrasound equipment and/or electrocardiographs.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1A is a schematic diagram of an operation of pruning a neural network provided by an embodiment of the present application;

FIG1B is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

FIG2 is a schematic diagram of the structure of a control unit provided in an embodiment of the present application;

FIG3 is a flow chart of a neural network operation method provided in an embodiment of the present application;

FIG4 is a schematic diagram of a flow chart of a neural network compression method provided in an embodiment of the present application;

FIG5A is a schematic diagram of a neural network architecture provided in an embodiment of the present application;

FIG5B is a schematic diagram of a fully connected layer weight matrix provided in an embodiment of the present application;

FIG5C is a schematic diagram of an operation of compressing a fully connected layer weight matrix provided in an embodiment of the present application;

FIG5D is a schematic diagram of the structure of a convolution kernel in a convolution layer provided in an embodiment of the present application;

FIG5E is a schematic diagram of a fully connected layer weight matrix provided in another embodiment of the present application;

FIG5F is a schematic diagram of an operation of compressing an LSTM layer provided in an embodiment of the present application;

FIG6 is a schematic diagram of the structure of another computing device provided in an embodiment of the present application;

7 is a schematic diagram of the structure of a main processing circuit provided in an embodiment of the present application;

FIG8 is a schematic diagram of the structure of another computing device provided in an embodiment of the present application;

FIG9 is a schematic diagram of the structure of a tree module provided in an embodiment of the present application;

FIG10 is a structural diagram of another computing device provided in an embodiment of the present application;

FIG11 is a structural diagram of another computing device provided in an embodiment of the present application;

FIG12 is a structural diagram of another computing device provided in an embodiment of the present application;

FIG13 is a structural diagram of a combined processing device provided in an embodiment of the present application;

FIG14 is a structural diagram of another combined processing device provided in an embodiment of the present application;

FIG. 15 is a schematic diagram of the structure of a board provided in an embodiment of the present application.

FIG16 is a flow chart of a neural network compression method provided in an embodiment of the present application;

FIG17A is a schematic diagram of the structure of a neural network compression device provided in an embodiment of the present application;

FIG17B is a schematic diagram of the structure of a compression unit provided in an embodiment of the present application;

FIG. 18 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

The present application provides a compression element for compressing a first weight matrix into a second weight matrix, which solves the problem that the topological structure of the neural network is easily irregular when the neural network pruning algorithm is used in the prior art. In practical applications, the above compression unit can be used in neural network calculations, specifically, in a computing device for performing neural network calculations. The present invention is introduced below in conjunction with the computing device shown in FIG1B.

Referring to FIG. 1B , FIG. 1B is a schematic diagram of the structure of a computing device provided by an embodiment of the present invention, the computing device is used to perform machine learning calculations, and the computing device includes: a controller unit 11, a computing unit 12, and a compression unit 13, wherein the controller unit 11 is connected to the computing unit 12 and the compression unit 13 respectively;

The controller unit 11 is used to obtain a compression request for first input data, and instruct the compression unit to compress the first input data according to the compression request; wherein the first input data includes a first weight matrix; in an optional solution, the compression request can be triggered by a data input and output unit, and the data input and output unit can specifically be one or more data I/O interfaces or I/O pins;

The compression unit 13 is used to compress the first weight matrix into a second weight matrix; wherein the second weight matrix includes at least two sub-matrices;

In a specific implementation, the compression unit 13 includes a decomposition unit 131, a solution unit 132, and a training unit 133. The decomposition unit 131 is used to decompose the first weight matrix into a third weight matrix; wherein the third weight matrix includes at least two sub-matrices; the solution unit 132 is used to determine the size of each sub-matrix in the at least two sub-matrices according to a first formula, and the first formula is Q≈Q ₁ *Q ₂ *......*Q _n ; wherein Q represents the first weight matrix; Q ₁ represents the first sub-matrix in the at least two sub-matrices; Q ₂ represents the second sub-matrix in the at least two sub-matrices; and Q _n represents the nth sub-matrix in the at least two sub-matrices; the training unit 133 is used to adjust the size of each sub-matrix in the at least two sub-matrices, and obtain a second weight matrix that meets the preset accuracy by training the compressed machine learning model.

The controller unit 11 is also used to obtain second input data and calculation instructions; the second input data includes a second weight matrix and input neuron data; in an optional scheme, specifically, the second input data and calculation instructions can be obtained through a data input and output unit, which can specifically be one or more data I/O interfaces or I/O pins.

The controller unit 11 is further used to parse the calculation instruction to obtain multiple operation instructions, and send the multiple operation instructions and the second input data to the operation unit;

The computing unit 12 is used to obtain the computing instruction and perform neural network calculation according to the computing instruction and the second input data.

In one implementation, considering that a "compression instruction" is provided in the above-mentioned computing device, in this case, the controller unit 11 is used to obtain first input data and a compression instruction; wherein, the first input data includes a first weight matrix; in an optional scheme, specifically, the first input data and the compression instruction method can be obtained through a data input-output unit, and the data input-output unit can specifically be one or more data I/O interfaces or I/O pins.

The controller unit 11 is further configured to parse the compression instruction to obtain a plurality of operation instructions, and send the plurality of operation instructions and the first weight matrix to the compression unit;

The compression unit 13 is used to compress the first weight matrix into a second weight matrix according to the multiple operation instructions;

The controller unit 11 is also used to obtain second input data and calculation instructions; the second input data includes the second weight matrix and input neuron data; in an optional scheme, specifically, the second input data and calculation instruction method can be obtained through a data input and output unit, and the data input and output unit can specifically be one or more data I/O interfaces or I/O pins.

The controller unit 11 is further used to parse the calculation instruction to obtain multiple operation instructions, and send the multiple operation instructions and the second input data to the operation unit;

The computing unit 12 is used to obtain the computing instruction and perform neural network calculation according to the computing instruction and the second input data.

In a specific implementation, the operation unit 12 includes a master processing circuit 101 and a plurality of slave processing circuits 102, wherein the master processing circuit 101 is configured to perform pre-order processing on the second input data and transmit data and operation instructions between the plurality of slave processing circuits;

A plurality of slave processing circuits 102, configured to perform intermediate operations in parallel according to data and operation instructions transmitted from the master processing circuit to obtain a plurality of intermediate results, and transmit the plurality of intermediate results to the master processing circuit;

The main processing circuit 101 is used to perform subsequent processing on the multiple intermediate results to obtain the calculation result of the calculation instruction.

Optionally, the second input data may specifically include: a second weight matrix and input neuron data. The calculation result may specifically be: the result of the neural network operation, that is, the output neuron data.

In one embodiment, the computing device may further include: the storage unit 10 and a direct memory access unit 50. The storage unit 10 may include: one or any combination of a register and a cache. Specifically, the cache is used to store the computing instructions; the register is used to store the input data and scalars; the cache is a high-speed temporary cache. The direct memory access unit 50 is used to read or store data from the storage unit 10.

In the embodiment of the present application, as shown in FIG2 , the controller unit 11 includes: an instruction cache unit 110 , an instruction processing unit 111 , a dependency processing unit 112 , and a storage queue unit 113 ;

The instruction cache unit 110 is used to store the computing instructions associated with the artificial neural network operation. When the zeroth computing instruction is being executed, other instructions that have not been submitted for execution are cached in the instruction cache unit 110. After the zeroth computing instruction is executed, if the first computing instruction is the earliest instruction among the instructions that have not been submitted in the instruction cache unit 110, the first computing instruction will be submitted. Once submitted, the change of the device state caused by the operation performed by the instruction cannot be undone.

The instruction processing unit 111 is used to obtain the calculation instruction from the instruction cache unit, and parse the calculation instruction to obtain multiple operation instructions;

The dependency processing unit 112 is used to determine whether a first operation instruction has an association relationship with a zeroth operation instruction before the first operation instruction when there are multiple operation instructions. If the first operation instruction has an association relationship with the zeroth operation instruction, the first operation instruction is stored in the storage queue unit 113. After the zeroth operation instruction is executed, the association relationship between the first operation instruction and the zeroth operation instruction is released, and the first operation instruction is extracted from the storage queue unit 113 and transmitted to the operation unit.

The determining whether the first operation instruction has an association relationship with the zeroth operation instruction before the first operation instruction comprises:

The first storage address interval of the data (e.g., a matrix) required in the first operation instruction is extracted according to the first operation instruction, and the zeroth storage address interval of the matrix required in the zeroth operation instruction is extracted according to the zeroth operation instruction. If the first storage address interval and the zeroth storage address interval have an overlapping area, it is determined that the first operation instruction and the zeroth operation instruction have an associated relationship. If the first storage address interval and the zeroth storage address interval do not have an overlapping area, it is determined that the first operation instruction and the zeroth operation instruction have no associated relationship.

The storage queue unit 113 is used to store an instruction queue, which includes: a plurality of operation instructions or calculation instructions to be executed in the order of the queue.

In an embodiment of the present application, as shown in FIG2 , the instruction processing unit 111 includes an instruction fetch module, a decoding module and an instruction queue, wherein the instruction fetch module is used to obtain computing instructions of the neural network from the instruction cache unit 110; the decoding module is used to decode the computing instructions obtained by the instruction fetch module to obtain operating instructions of the neural network; the instruction queue is used to sequentially store the operating instructions obtained after decoding in the order to be executed.

For example, in an optional technical solution, the main operation processing circuit may also include a controller unit, which may include a main instruction processing unit, specifically for decoding instructions into microinstructions. Of course, in another optional solution, the slave operation processing circuit may also include another controller unit, which includes a slave instruction processing unit, specifically for receiving and processing microinstructions. The above-mentioned microinstructions may be the next level instructions of the instructions, which may be obtained by splitting or decoding the instructions, and can be further decoded into control signals for each component, each unit or each processing circuit.

In an optional solution, the structure of the calculation instruction may be as shown in the following table.

Table 1

Operation Code Register or immediate value Register/Immediate ...

The ellipsis in the above table indicates that multiple registers or immediate values can be included.

In another optional solution, the computing instruction may include: one or more operation domains and an operation code. The computing instruction may include a neural network operation instruction, and may also include the compression instruction involved above. Taking the neural network operation instruction as an example, as shown in Table 1, register number 0, register number 1, register number 2, register number 3, and register number 4 may be operation domains. Each register number 0, register number 1, register number 2, register number 3, and register number 4 may be the number of one or more registers.

Table 2

The above-mentioned registers can be off-chip memories, and of course in practical applications, they can also be on-chip memories for storing data. The data can specifically be n-dimensional data, where n is an integer greater than or equal to 1. For example, when n=1, it is 1-dimensional data, i.e., a vector; when n=2, it is 2-dimensional data, i.e., a matrix; and when n=3 or more, it is a multi-dimensional tensor.

In the embodiment of the present invention, the process of the computing device executing the neural network operation is shown in FIG3 , and includes:

Step S1: The controller unit receives a compression instruction, decodes the compression instruction into a plurality of operation instructions, and sends the plurality of operation instructions to the compression unit.

After the controller unit reads the compression instruction from the storage unit, it parses the compression instruction into an operation instruction and sends the operation instruction to the compression unit. Specifically, the instruction fetch module of the instruction processing unit 111 in the controller unit 11 obtains the compression instruction from the instruction cache unit 110 and transmits the instruction to the decoding module. The decoding module decodes the compression instruction to obtain the operation instruction, and splits the operation instruction into an operation code and various operation domains according to the preset instruction rule, wherein the composition and function of the operation code and the operation domain can be referred to the foregoing, and will not be repeated here. The decoding module transmits the operation instruction obtained after decoding to the instruction queue for sequential storage. In the instruction queue, the data address of the to-be-processed data corresponding to the instruction is obtained according to the operation code and operation of the operation instruction, and the data address is transmitted to the dependency processing unit 112. The dependency processing unit analyzes whether there is an association relationship between the instruction and the instruction being executed. If there is, the operation instruction is stored in the storage queue unit 113 until the association relationship is resolved. If there is no association relationship, the operation instruction is sent to the compression unit to perform the corresponding operation.

S2. The compression unit receives the operation instruction sent by the control unit, and performs compression processing according to the first weight matrix read from the storage unit to obtain a second weight matrix that meets a preset accuracy.

In conjunction with the flowchart of the neural network compression method provided by the embodiment of the present invention shown in FIG. 4 , the following specifically describes how the embodiment of the present invention implements compression of the first weight matrix to obtain the second weight matrix, which may include but is not limited to the following steps:

Step S21: decompose the first weight matrix into a third weight matrix; wherein the third weight matrix includes at least two sub-matrices.

In a specific implementation, the weight data in the first weight matrix can be any real number. Here, the weight data refers to the connection value between the neural network layers, that is, the information transmission strength between neurons.

In one embodiment, the third weight matrix includes two sub-matrices, and each of the two sub-matrices includes a compression parameter K. Here, the compression parameter K is an unknown number, that is, when the first weight matrix is decomposed, it can be determined that the first weight matrix can be decomposed into two sub-matrices, but the size of each of the two sub-matrices is not determined.

In another embodiment, the number of sub-matrices in the third weight matrix is n, where n is a positive integer greater than 2. The number of compression parameters K included in the n sub-matrices is (n-1). Taking the example of dividing the first weight matrix into three sub-matrices, the compression parameters K to be solved may include K1 and K2.

Step S22: determine the size of each submatrix in the at least two submatrices according to a first formula, wherein the first formula is Q≈Q ₁ *Q ₂ *......*Q _n ; wherein Q represents a first weight matrix; Q ₁ represents a first submatrix in the at least two submatrices; Q ₂ represents a second submatrix in the at least two submatrices; and Q _n represents an nth submatrix in the at least two submatrices.

In a specific implementation, the operation symbol “*” in the first formula represents a matrix multiplication operation.

In one implementation manner, when the third weight matrix includes two sub-matrices, the first formula can be expressed as:

Q≈Q ₁ *Q ₂ (1.1)

In another embodiment, when the third weight matrix includes at least two sub-matrices, the first formula can be expressed as:

Q≈Q ₁ *Q ₂ *......*Q _n (1.2)

In the above formula (1.2), n is a positive integer greater than 2.

In a specific implementation, the size of each submatrix in the at least two submatrices is determined according to the first formula and the second formula, and the second formula is ||QQ ₁ *Q ₂ *......*Q _n ||≤T, where T represents a preset error threshold.

In a specific implementation, the preset error threshold involved here may be between 5% and 10%. It is understandable that the smaller the preset error threshold is, the better the at least two sub-matrices determined according to the first formula and the second formula can represent the attribute characteristics of the first weight matrix.

Step S23: adjust the size of each submatrix of the at least two submatrices, and obtain a second weight matrix that meets a preset accuracy by training the compressed machine learning model.

In a specific implementation, the process of adjusting the size of each of the at least two sub-matrices is actually a dynamic change process of the compression parameter K value to find the optimal compression parameter K. As the compression parameter K value changes, the compression ratio between the first weight matrix and the second weight matrix will also change.

Taking the application scenario of speech recognition as an example, in a certain sequence of words, some words may be inserted, deleted or replaced by mistake. For example, for a segment of initial recognition text containing N words, if I words are inserted, D words are deleted and E words are replaced, then the word error rate WER (Word Error Rate, WER) is:

WER＝(I+D+E)/N (1.3)

Among them, the error rate WER is usually expressed as a percentage.

When the neural network model is used to recognize the word sequence, the detection accuracy of the word error rate of the word sequence can be obtained. In the embodiment of the present invention, the preset accuracy involved here is the detection accuracy of the neural network model for the word error rate WER before compression. For example, the preset accuracy is 70%. In general, the error rate WER of the compressed neural network will increase, which means that the accuracy of the compressed neural network will deteriorate.

In an embodiment of the invention, the detection accuracy of the word error rate of the neural network model corresponding to different compression ratios (different values of the compression parameter K) is measured to obtain a second weight matrix that meets the preset accuracy.

In a preferred embodiment, the training unit is used to adjust the size of each submatrix of the at least two submatrices, and obtain a second weight matrix that meets a preset accuracy by training the compressed machine learning model, including:

The training unit is specifically used to adjust the size of each submatrix of the at least two submatrices, and to obtain a second weight matrix that meets a preset accuracy and has a compression ratio that meets a preset compression ratio with the first weight matrix by training the compressed machine learning model.

It can be understood that, in this embodiment, the compression parameter K value in the current state can not only enable the neural network model to obtain the best compression effect, but also enable the compressed neural network model to meet the preset accuracy when detecting the word error rate WER. When the neural network model is in the best compression effect, the amount of computation of the neural network can be further reduced.

Taking the fully connected layer of a neural network as an example, the fully connected layer means that for the n-1 layer and the n layer, any node in the n-1 layer is connected to all the nodes in the n layer. Specifically, referring to FIG5A, it is a schematic diagram of the structure of a one-dimensional fully connected layer of a neural network provided in an embodiment of the present invention. As shown in FIG5A, the neural network includes an input layer, a hidden layer, and an output layer, wherein the two-dimensional parameter matrix of the fully connected layer between the input layer and the hidden layer is (3,4), and the two-dimensional parameter matrix (3,4) indicates that in the fully connected layer structure between the input layer and the hidden layer, the number of input neurons is 3, the number of output neurons is 4, and the number of weights is 12. In a specific implementation, these 12 weights can be represented as a weight matrix of 4 rows and 3 columns, and the expression of the weight matrix can be shown in FIG5B.

In a fully connected layer neural network, the first formula includes: M≈M ₁ *M ₂ ; the two sub-matrices refer to a first sub-matrix M1 and a second sub-matrix M2, the M1 is an N _in *K matrix, and the M2 is a K*N _out matrix; wherein K is a compression parameter, N _in is the number of input neurons of the neural network, and N _out is the number of output neurons of the neural network; the compression parameter is used to characterize the number of output neurons of the M1 and the number of input neurons of the M2, and K is a positive integer greater than 0 and less than or equal to min(N _in , N _out ).

As mentioned above, the process of adjusting the size of each of the two sub-matrices is actually a dynamic change process of the compression parameter K value to find the optimal compression parameter K. In practical applications, a binary search method can be used to determine the compression parameter K value in the fully connected layer neural network, so as to obtain a second weight matrix that meets the preset accuracy. In one embodiment, the compression parameter K determined by the binary search method can make the second weight matrix meet the preset accuracy. In another embodiment, the compression parameter K determined by the binary search method can make the second weight matrix meet the preset accuracy while the compression ratio of the first weight matrix to the second weight matrix meets the preset compression ratio, that is, a better compression effect is obtained for the compression of the neural network model.

In the specific implementation, the compression parameter K has different values, that is, the first weight matrix is compressed based on multiple different compression ratios. Here, in the fully connected layer neural network, the compression ratio is

Next, we will explain in detail how to use binary search to determine the compression parameter K value. First, set two parameters KL and KR. In the initialization, let KL = 1, KR = min (N _in , N _out ). In the process of adjusting the parameters, K = (KL + KR) / 2. If the second weight matrix represented by _M1 * _M2 causes the accuracy of the compressed neural network model to decrease by X% (here, X = 1 to 10, etc.), adjust the parameter KL so that KL = K. If the second weight matrix represented by _M1 * _M2 causes the compressed neural network model to meet the preset accuracy, then adjust KR so that KR = K, and repeat the above steps until the end condition K = KL or K = KR is met.

Taking the fully connected layer between the input layer and the hidden layer in FIG5A as an example, the compression parameter K value is a positive integer greater than 0 and less than or equal to 3. The compression parameter K=2 is determined by the above binary search method, that is, the first submatrix M1 in the second weight matrix that meets the preset accuracy is a (3,2) matrix, and the second submatrix M2 is a (2,4) matrix. Specifically, the compression of the fully connected layer between the input layer and the hidden layer in FIG5A can be shown in FIG5C.

In one embodiment, when the first formula is expressed as shown in formula (1.2), that is, the number of sub-matrices in the third weight matrix is n, where n is a positive integer greater than 2. At this time, the number of compression parameters K is (n-1), which can be expressed as K1, K2, ..., Kn _-1 . In practical applications, an adaptive algorithm (for example, a genetic algorithm) can be used to determine the (n-1) compression parameter K values in the fully connected layer neural network, so as to obtain a second weight matrix that meets the preset accuracy and/or meets the compression effect. The following specifically describes how to use a genetic algorithm to determine the (n-1) compression parameter K values in the fully connected layer neural network:

Step 1: Randomly generate a population: Set the size of the population to P, set the maximum number of iterations T _max , for example, T _max = 100. In the initial state, set the iteration counter t = 0; crossover probability Pc = A (for example, A = 0.4), mutation probability Pm = B (for example, B = 0.6), each row of the population matrix represents a gene string individual, and each column represents the number of individuals; here, each individual is a set of solutions about the value of the compression parameter K (for example, K _j );

Step 2: Calculate the fitness of each individual in the population; here, fitness refers to the compression ratio and/or accuracy of the first weight matrix and the second weight matrix corresponding to the individual, where the compression ratio is used to characterize the compression effect on the neural network.

Step 3: Apply the selection operator to the population and directly pass the optimized individuals to the next generation;

Step 4: Apply the crossover operator to the population. For any two individuals, randomly generate several gene string position points and exchange the values of the two individuals at that position.

Step 5: Apply the mutation operator to the population. For any individual, randomly generate the positions of several gene strings and then change the values at these positions. Here, mutation refers to randomly changing the value of _Kj .

Step 6: Keep the individuals with the highest fitness in each generation and enter the next generation;

Step 7: Determine whether the maximum number of iterations T _max is reached. If t=T _max , output the individual with the maximum fitness and terminate the calculation; otherwise, jump to step 2 to continue.

Therefore, the (n-1) compression parameter K values in the fully connected layer neural network can be determined according to the above genetic algorithm.

Taking the convolution layer of a neural network as an example, as shown in Figure 5D, the convolution layer can be considered as a four-dimensional matrix (N _fin , N _fout , K _x , _Ky ), where N _fin is the number of input feature images, N _fout is the number of output feature images, and (K _x , _Ky ) is the size of the convolution kernel in the convolution layer.

In a convolutional layer neural network, the convolutional layer neural network includes N _fin *N _fout convolution kernels; the first formula includes: F≈F ₁ *F ₂ ; wherein F represents any one of the N _fin *N _fout convolution kernels; the F1 is the first sub-convolution kernel; the F2 is the second sub-convolution kernel; the first sub-convolution kernel F1 is (K _x , R), the second sub-convolution kernel F2 is (R, _Ky ), (K _x , _Ky ) represents the size of the convolution kernel, R is a compression parameter, and R is a positive integer greater than 0 and less than or equal to min(K _x , _Ky ).

As mentioned above, the process of adjusting the size of each of the two sub-matrices is actually a dynamic change process of the compression parameter R value to find the best compression parameter R. In practical applications, a binary search method can be used to determine the compression parameter R value in the convolutional layer neural network, so as to obtain a second weight matrix that meets the preset accuracy.

In the specific implementation, the compression parameter R value is different, that is, the first weight matrix is compressed based on multiple different compression ratios. Here, in the convolutional layer neural network, the compression ratio

In the embodiment of the present invention, the implementation process of determining the value of the compression parameter R by using a binary search method is described in the foregoing text and will not be elaborated herein.

For example, in the convolutional layer neural network structure shown in FIG5D , the convolutional layer includes 4 convolution kernels, and the convolution kernel size is 3*3, wherein, in the first convolution kernel, N _fin =4, N _fout =6, and the compression parameter R value is a positive integer greater than 0 and less than or equal to 4. The compression parameter R=4 is determined by the above-mentioned binary search method, that is, the first sub-convolution kernel F1 in the first convolution kernel that meets the preset accuracy is a (3,4) matrix, and the second sub-convolution kernel F2 is a (4,3) matrix. In one embodiment, for other convolution kernels shown in FIG5D , the same compression method as the first convolution kernel can be used, or a compression method different from the first convolution kernel can be used, which is not specifically limited in the embodiment of the present invention.

In one embodiment, when the first formula is expressed as shown in formula (1.2), an adaptive algorithm (for example, a genetic algorithm) can be used to determine the value of the compression parameter R in the convolutional layer neural network. Please refer to the above description for the specific implementation process, which will not be elaborated here.

Taking the LSTM layer (Long Short-term Memory) of the neural network as an example, the weights of the LSTM layer are composed of multiple fully connected layer weights. Assume that the weights of the LSTM layer are composed of t fully connected layer weights, and t is a positive integer greater than 0. For example, the weights of the j-th fully connected layer are (N _{in_j} ,N _{out_j} ), where N _{in_j} represents the number of input neurons of the j-th fully connected layer, N _{out_j} represents the number of output neurons of the j-th fully connected layer, and the number of weights of the j-th fully connected layer is N _{in_j} *N _{out_j} .

In an LSTM layer neural network, the LSTM layer includes N fully connected layers, where N is a positive integer greater than 0; for the j-th fully connected layer, the first formula includes: M _j ≈M _{j_1} *M _{j_2} ; the two sub-matrices in the j-th fully connected layer include a first sub-matrix M _{j_1} and a second sub-matrix M _{j_2} , where M _{j_1} is an N _{in_j} *S matrix, and M _{j_2} is an S*N _{out_j} matrix; wherein S is a compression parameter, N _{in_j} is the number of input neurons of the j-th fully connected layer of the neural network, and N _{out_j} is the number of output neurons of the j-th fully connected layer of the neural network; the compression parameter is used to characterize the number of output neurons of the M _{j_1} and the number of input neurons of the M _{j_2} , and S is a positive integer greater than 0 and less than or equal to min(N _{in_j} , N _{out_j} ).

As mentioned above, the process of adjusting the size of each of the two sub-matrices is actually a dynamic change process of the compression parameter S value to find the best compression parameter S. In practical applications, a binary search method can be used to determine the compression parameter S value in the convolutional layer neural network, thereby obtaining a second weight matrix that meets the preset accuracy.

In the specific implementation, for the jth fully connected layer, the compression parameter S value is different, that is, the first weight matrix is compressed based on multiple different compression ratios. Here, in the jth fully connected layer, the compression ratio

In the embodiment of the present invention, the implementation process of determining the value of the compression parameter S in the j-th fully connected layer by using a binary search method is described in the above text and will not be repeated here.

Taking the neural network architecture shown in FIG5A as an example, the neural network includes an input layer, a hidden layer, and an output layer, wherein the first fully connected layer is between the input layer and the hidden layer, and the second fully connected layer is between the hidden layer and the output layer. For the specific elaboration of this fully connected layer structure between the input layer and the hidden layer (that is, the first fully connected layer), please refer to the above description, and no further elaboration is given here. As can be seen from FIG5A, the two-dimensional parameter matrix of this fully connected layer between the hidden layer and the output layer is (4,2), and the two-dimensional parameter matrix (4,2) indicates that in the fully connected layer structure between the hidden layer and the output layer, the number of input neurons is 4, the number of output neurons is 2, and the number of weights is 8. In a specific implementation, these 8 weights can be represented as a 2-row 4-column weight matrix, and the expression of the weight matrix can be shown in FIG5E. Then, in this case, the value of the compression parameter S is a positive integer greater than 0 and less than or equal to 2. The compression parameter S=2 is determined by binary search, that is, in the second fully connected layer, the first submatrix M ₂ _ ₁ in the second weight matrix that meets the preset accuracy is a (4,2) matrix, and the second submatrix M ₂ _ ₂ is a (2,2) matrix. Specifically, the compression of the fully connected layer between the input layer and the hidden layer in FIG5A and the fully connected layer between the hidden layer and the output layer can be shown in FIG5F.

In one embodiment, when the first formula is expressed as shown in formula (1.2), an adaptive algorithm (for example, a genetic algorithm) can be used to determine the value of the compression parameter S in the LSTM layer neural network. For the specific implementation process, please refer to the above description and no further details will be given here.

Through the embodiment of the present invention, after obtaining the compression instruction, the controller unit can parse it to obtain multiple operation instructions, and then send these multiple operation instructions and the first weight matrix to the compression unit, and then the compression unit can obtain the second weight matrix by decomposing the first weight matrix. In a specific implementation, the second weight matrix includes at least two sub-matrices, and then by adjusting the size of each of the at least two sub-matrices, and combining the machine learning model after training compression, a second weight matrix that meets the preset accuracy is obtained, which solves the problem of irregular topological structure of the neural network that is easily caused by the use of neural network pruning algorithms in the prior art. In addition, compressing the neural network can reduce the amount of calculation of the neural network, thereby improving the calculation speed.

S3. The controller unit obtains second input data and calculation instructions, wherein the second input data includes a second weight matrix and input neuron data.

S4. The controller unit parses the calculation instruction into an operation instruction, and sends the operation instruction and the second input data to the operation unit.

In a specific implementation, the controller unit obtains calculation instructions and parses the calculation instructions to obtain implementation methods of multiple operation instructions. Please refer to the text description of the controller unit obtaining compression instructions mentioned above, and no further details will be given here.

S5. The operation unit receives the operation instruction sent by the controller unit, and performs neural network calculation according to the operation instruction and the second input data.

In practical applications, the neural network calculations involved here may include artificial neural network operations, convolutional neural network operations, and so on.

Taking artificial neural network operation as an example, for artificial neural network operation, if the artificial neural network operation has multi-layer operation, the input neurons and output neurons of the multi-layer operation do not refer to the neurons in the input layer and the neurons in the output layer of the entire neural network, but for any two adjacent layers in the network, the neurons in the lower layer of the network forward operation are the input neurons, and the neurons in the upper layer of the network forward operation are the output neurons. Taking convolutional neural network as an example, suppose a convolutional neural network has L layers, K＝1,2,...,L-1, for the Kth layer and the K+1th layer, we call the Kth layer the input layer, the neurons therein are the input neurons, and the K+1th layer is called the output layer, the neurons therein are the output neurons. That is, except for the top layer, each layer can be used as an input layer, and the next layer is the corresponding output layer.

In a specific implementation, the operation in the neural network can be the operation of a layer in the neural network. For a multi-layer neural network, the implementation process is that in the forward operation, after the execution of the previous layer of artificial neural network is completed, the operation instruction of the next layer will use the output neuron calculated in the operation unit as the input neuron of the next layer for operation (or perform certain operations on the output neuron and then use it as the input neuron of the next layer), and at the same time, the weights are also replaced by the weights of the next layer; in the reverse operation, when the reverse operation of the previous layer of artificial neural network is completed, the operation instruction of the next layer will use the input neuron gradient calculated in the operation unit as the output neuron gradient of the next layer for operation (or perform certain operations on the input neuron gradient and then use it as the output neuron gradient of the next layer), and at the same time, the weights are replaced by the weights of the next layer.

Taking the forward operation process of the neural network as an example, first, the operation unit reads the second input data from the storage unit, wherein the second input data includes a second weight matrix and input neuron data.

Secondly, the main processing circuit reads the corresponding neuron data and broadcasts the neuron data to each slave processing circuit in the specified order. In practical applications, the neuron data can be broadcast only once, and the slave processing circuit receives the data and temporarily stores it in a cache or register to facilitate reuse. In addition, the neuron data can also be broadcast multiple times, and the slave processing circuit uses the data directly after receiving it without multiplexing. In one possible implementation, the main processing circuit directly broadcasts the neuron data after reading it.

Afterwards, each slave processing circuit performs an inner product operation on the read neuron data and the second weight matrix according to the operation instruction, and then transmits the inner product result back to the master processing circuit.

In one of the embodiments, the slave processing circuit may transmit the partial sum obtained from each inner product operation back to the main processing circuit for accumulation; in one of the embodiments, the partial sum obtained from each inner product operation performed by the slave processing circuit may be stored in the register and/or on-chip cache of the slave processing circuit, and transmitted back to the main processing circuit after the accumulation is completed; in one of the embodiments, the partial sum obtained from each inner product operation performed by the slave processing circuit may be stored in the register and/or on-chip cache of the slave processing circuit for accumulation in some cases, and transmitted to the main processing circuit for accumulation in some cases, and transmitted back to the main processing circuit after the accumulation is completed.

Finally, the main processing circuit accumulates and activates the results of each slave processing circuit until the forward operation process of the neural network is completed, and the error value between the predicted result and the actual result, that is, the neuron gradient data of the last layer, is obtained and saved in the storage unit.

In an embodiment of the present invention, the operation unit 12 can be set to a one-master-multiple-slave structure. In an optional embodiment, the operation unit 12, as shown in FIG6 , may include a master processing circuit 101 and multiple slave processing circuits 102. In one embodiment, as shown in FIG6 , multiple slave processing circuits are distributed in an array; each slave processing circuit is connected to other adjacent slave processing circuits, and the master processing circuit is connected to k slave processing circuits among the multiple slave processing circuits, and the k slave processing circuits are: n slave processing circuits in the first row, n slave processing circuits in the mth row, and m slave processing circuits in the first column. It should be noted that the K slave processing circuits shown in FIG6 only include n slave processing circuits in the first row, n slave processing circuits in the mth row, and m slave processing circuits in the first column, that is, the k slave processing circuits are slave processing circuits directly connected to the master processing circuit among the multiple slave processing circuits.

K slave processing circuits are used to forward data and instructions between the master processing circuit and multiple slave processing circuits.

Optionally, as shown in FIG7 , the main processing circuit may further include: one or any combination of a conversion processing circuit 120 , an activation processing circuit 121 , and an addition processing circuit 122 ;

The conversion processing circuit 120 is used to perform an exchange between a first data structure and a second data structure (e.g., conversion between continuous data and discrete data) on the data block or intermediate result received by the main processing circuit; or perform an exchange between a first data type and a second data type (e.g., conversion between a fixed-point type and a floating-point type) on the data block or intermediate result received by the main processing circuit;

An activation processing circuit 121, used to perform activation operations on data in the main processing circuit;

The addition processing circuit 122 is used to perform addition operations or accumulation operations.

The master processing circuit is used to determine that the input neuron is broadcast data, the weight is distribution data, distribute the distribution data into multiple data blocks, and send at least one data block of the multiple data blocks and at least one operation instruction of the multiple operation instructions to the slave processing circuit;

The multiple slave processing circuits are used to perform operations on the received data blocks according to the operation instructions to obtain intermediate results, and transmit the operation results to the master processing circuit;

The main processing circuit is used to process the intermediate results sent by the multiple slave processing circuits to obtain the result of the calculation instruction, and send the result of the calculation instruction to the controller unit.

The slave processing circuit includes: a multiplication processing circuit;

The multiplication processing circuit is used to perform a product operation on the received data block to obtain a product result;

A forwarding processing circuit (optional) is used to forward the received data block or product result.

The accumulation processing circuit is used to perform an accumulation operation on the product result to obtain the intermediate result.

In another embodiment, the operation instruction is a matrix multiplication instruction, an accumulation instruction, an activation instruction, or the like.

The specific calculation method of the calculation device shown in FIG1B is explained below by using a neural network operation instruction. For the neural network operation instruction, the formula that actually needs to be executed may be: s=s( _∑wxi +b), wherein the weight w is multiplied by the input _dataxi , the sum is taken, and then the bias b is added and the activation operation s(h) is performed to obtain the final output result s.

In an optional implementation, as shown in Figure 8, the operation unit includes: a tree module 40, the tree module includes: a root port 401 and multiple branch ports 404, the root port of the tree module is connected to the main processing circuit, and the multiple branch ports of the tree module are respectively connected to one of the multiple slave processing circuits; the above-mentioned tree module has a transceiver function, and the tree module has a transceiver function for forwarding data blocks, weights and operation instructions between the main processing circuit and the multiple slave processing circuits, that is, the data of the main processing circuit can be transmitted to each slave processing circuit, and the data of each slave processing circuit can also be transmitted to the main processing circuit.

Optionally, the tree module is an optional result of the computing device, which may include at least one layer of nodes, the nodes are line structures with forwarding functions, and the nodes themselves may not have computing functions. If the tree module has zero-layer nodes, the tree module is not needed.

Optionally, the tree module may be an n-ary tree structure, for example, a binary tree structure as shown in FIG9 , or a ternary tree structure, and n may be an integer greater than or equal to 2. The specific implementation of the present application does not limit the specific value of n, and the number of layers may also be 2, and the slave processing circuit may be connected to nodes of other layers except the penultimate layer nodes, for example, the penultimate layer nodes as shown in FIG9 may be connected.

Optionally, the above-mentioned operation unit may carry a separate cache, as shown in FIG10 , and may include: a neuron cache unit, wherein the neuron cache unit 63 caches the input neuron vector data and output neuron value data of the slave processing circuit.

As shown in FIG. 11 , the operation unit may further include: a weight cache unit 64 for caching weight data required by the slave processing circuit during the calculation process.

In an optional embodiment, the operation unit 12 is shown in FIG. 12 and may include a branch processing circuit 103; its specific connection structure is shown in FIG. 12, wherein:

The master processing circuit 101 is connected to the branch processing circuit 103 (one or more), and the branch processing circuit 103 is connected to one or more slave processing circuits 102;

The branch processing circuit 103 is used to forward data or instructions between the main processing circuit 101 and the slave processing circuit 102 .

In an optional embodiment, taking the full connection operation in the neural network operation as an example, the process can be: y = f (wx + b), where x is the input neuron matrix, w is the weight matrix, b is the bias scalar, and f is the activation function, which can be: sigmoid function, any one of tanh, relu, and softmax functions. Here, it is assumed that the binary tree structure has 8 slave processing circuits, and the implementation method can be:

The controller unit obtains the input neuron matrix x, the weight matrix w and the full connection operation instruction from the storage unit, and transmits the input neuron matrix x, the weight matrix w and the full connection operation instruction to the main processing circuit;

The master processing circuit determines that the input neuron matrix x is broadcast data, determines that the weight matrix w is distributed data, splits the weight matrix w into 8 sub-matrices, and then distributes the 8 sub-matrices to 8 slave processing circuits through the tree module, and broadcasts the input neuron matrix x to the 8 slave processing circuits.

The slave processing circuit performs multiplication and accumulation operations of the eight sub-matrices and the input neuron matrix x in parallel to obtain eight intermediate results, and sends the eight intermediate results to the master processing circuit;

The main processing circuit is used to sort the 8 intermediate results to obtain the operation result of wx, perform the operation of bias b on the operation result and then perform the activation operation to obtain the final result y, and send the final result y to the controller unit, and the controller unit outputs or stores the final result y in the storage unit.

The method for the computing device shown in FIG. 1B to execute the neural network forward operation instruction may specifically be:

The controller unit extracts the neural network forward operation instruction, the operation domain corresponding to the neural network operation instruction and at least one operation code from the instruction storage unit, transmits the operation domain to the data access unit, and sends the at least one operation code to the operation unit.

The controller unit extracts the weight w and bias b corresponding to the operation domain from the storage unit (when b is 0, there is no need to extract bias b), and transmits the weight w and bias b to the main processing circuit of the operation unit. The controller unit extracts the input data Xi from the storage unit and sends the input data Xi to the main processing circuit.

The main processing circuit determines that the at least one operation code is a multiplication operation, determines that the input data Xi is broadcast data, determines that the weight data is distribution data, and splits the weight w into n data blocks;

The instruction processing unit of the controller unit determines the multiplication instruction, the bias instruction and the accumulation instruction according to the at least one operation code, and sends the multiplication instruction, the bias instruction and the accumulation instruction to the main processing circuit, the main processing circuit sends the multiplication instruction and the input data Xi to multiple slave processing circuits in a broadcast manner, and distributes the n data blocks to the multiple slave processing circuits (for example, if there are n slave processing circuits, then each slave processing circuit sends a data block); multiple slave processing circuits are used to perform multiplication operations on the input data Xi and the received data blocks according to the multiplication instruction to obtain an intermediate result, and send the intermediate result to the main processing circuit, the main processing circuit performs accumulation operations on the intermediate results sent by the multiple slave processing circuits according to the accumulation instruction to obtain an accumulation result, adds the bias b to the accumulation result according to the bias instruction to obtain a final result, and sends the final result to the controller unit.

In addition, the order of addition and multiplication operations can be reversed.

The technical solution provided in the present application implements the multiplication operation and bias operation of the neural network through one instruction, namely the neural network operation instruction. The intermediate results of the neural network calculation do not need to be stored or extracted, which reduces the storage and extraction operations of the intermediate data. Therefore, it has the advantages of reducing the corresponding operation steps and improving the calculation effect of the neural network.

The present application also discloses a machine learning computing device, which includes one or more computing devices mentioned in the present application, and is used to obtain data to be calculated and control information from other processing devices, perform specified machine learning operations, and transmit the execution results to peripheral devices through I/O interfaces. Peripheral devices include cameras, displays, mice, keyboards, network cards, wifi interfaces, and servers. When more than one computing device is included, the computing devices can be linked and data can be transmitted through a specific structure, for example, interconnected and data can be transmitted through a PCIE bus to support larger-scale machine learning operations. At this time, the same control system can be shared, or each independent control system can be provided; memory can be shared, or each accelerator can have its own memory. In addition, the interconnection method can be any interconnection topology.

The machine learning computing device has high compatibility and can be connected to various types of servers through a PCIE interface.

The present application also discloses a combined processing device, which includes the above-mentioned machine learning computing device, a universal interconnection interface, and other processing devices. The machine learning computing device interacts with other processing devices to jointly complete the operation specified by the user. FIG13 is a schematic diagram of the combined processing device.

Other processing devices include one or more types of processors such as central processing unit (CPU), graphics processing unit (GPU), neural network processor, and other general/special processors. There is no limit on the number of processors included in other processing devices. Other processing devices serve as interfaces between the machine learning computing device and external data and control, including data handling, to complete basic control of the machine learning computing device such as starting and stopping; other processing devices can also collaborate with the machine learning computing device to complete computing tasks.

A universal interconnection interface is used to transmit data and control instructions between the machine learning computing device and other processing devices. The machine learning computing device can obtain the required input data from other processing devices and write it into the storage device on the machine learning computing device chip; it can obtain control instructions from other processing devices and write them into the control cache on the machine learning computing device chip; it can also read data in the storage module of the machine learning computing device and transmit it to other processing devices.

Optionally, as shown in FIG14 , the structure may further include a storage device, which is connected to the machine learning operation device and the other processing device, respectively. The storage device is used to store data in the machine learning operation device and the other processing device, and is particularly suitable for data that cannot be fully stored in the internal storage of the machine learning operation device or other processing devices.

The combined processing device can be used as a SOC chip system for mobile phones, robots, drones, video surveillance equipment and other devices, effectively reducing the core area of the control part, improving the processing speed, and reducing the overall power consumption. In this case, the universal interconnection interface of the combined processing device is connected to certain components of the device. Certain components include cameras, displays, mice, keyboards, network cards, and wifi interfaces.

In some embodiments, a chip is also applied for, which includes the above-mentioned machine learning computing device or combined processing device.

In some embodiments, a chip packaging structure is applied for, which includes the above-mentioned chip.

In some embodiments, a board card is applied, which includes the above chip packaging structure. Referring to FIG. 15 , FIG. 15 provides a board card, which includes, in addition to the above chip 389 , other supporting components, including but not limited to: a storage device 390 , an interface device 391 and a control device 392 ;

The memory device 390 is connected to the chip in the chip package structure via a bus for storing data. The memory device may include multiple groups of memory cells 393. Each group of memory cells is connected to the chip via a bus. It is understood that each group of memory cells may be DDR SDRAM (English: Double Data Rate SDRAM, double rate synchronous dynamic random access memory).

DDR can double the speed of SDRAM without increasing the clock frequency. DDR allows data to be read out on the rising and falling edges of the clock pulse. The speed of DDR is twice that of standard SDRAM. In one embodiment, the storage device may include 4 groups of storage units. Each group of storage units may include multiple DDR4 particles (chips). In one embodiment, the chip may include 4 72-bit DDR4 controllers, 64 bits of the above 72-bit DDR4 controllers are used to transmit data, and 8 bits are used for ECC verification. It can be understood that when DDR4-3200 particles are used in each group of storage units, the theoretical bandwidth of data transmission can reach 25600MB/s.

In one embodiment, each group of the storage units includes a plurality of double rate synchronous dynamic random access memories arranged in parallel. DDR can transmit data twice in one clock cycle. A controller for controlling DDR is arranged in the chip to control the data transmission and data storage of each of the storage units.

The interface device is electrically connected to the chip in the chip packaging structure. The interface device is used to realize data transmission between the chip and an external device (such as a server or a computer). For example, in one embodiment, the interface device can be a standard PCIE interface. For example, the data to be processed is transmitted to the chip by the server through the standard PCIE interface to realize data transfer. Preferably, when the PCIE 3.0X 16 interface is used for transmission, the theoretical bandwidth can reach 16000MB/s. In another embodiment, the interface device can also be other interfaces. This application does not limit the specific forms of expression of the above-mentioned other interfaces. The interface unit can realize the switching function. In addition, the calculation results of the chip are still transmitted back to the external device (such as a server) by the interface device.

The control device is electrically connected to the chip. The control device is used to monitor the state of the chip. Specifically, the chip and the control device can be electrically connected via an SPI interface. The control device may include a single-chip microcomputer (Micro Controller Unit, MCU). For example, the chip may include multiple processing chips, multiple processing cores or multiple processing circuits, which can drive multiple loads. Therefore, the chip can be in different working states such as multi-load and light load. The control device can realize the regulation of the working states of multiple processing chips, multiple processing and/or multiple processing circuits in the chip.

In some embodiments, an electronic device is applied for, which includes the above-mentioned board.

Electronic devices include data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, mobile phones, driving recorders, navigators, sensors, cameras, servers, cloud servers, cameras, camcorders, projectors, watches, headphones, mobile storage, wearable devices, vehicles, household appliances, and/or medical devices.

The transportation means include airplanes, ships and/or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes magnetic resonance imaging, ultrasound machines and/or electrocardiographs.

In the embodiment of the present invention, it is considered that the compression method for the neural network can be applied to the above-mentioned computing device, but can also be applied to other scenarios, for example, to reduce the accuracy loss of the neural network. Based on this, the following is a flowchart of the neural network compression method provided by the embodiment of the present invention shown in FIG16 to specifically explain how the present invention implements compression of the first weight matrix to obtain the second weight matrix, which can include but is not limited to the following steps:

Step S100: Obtain first input data; wherein the first input data includes a first weight matrix.

In a specific implementation, the weight data in the first weight matrix can be any real number. Here, the weight data refers to the connection value between the neural network layers, that is, the information transmission strength between neurons.

Step S102: compress the first weight matrix into a second weight matrix; wherein the second weight matrix includes at least two sub-matrices.

In one implementation manner, adjusting the first weight matrix to a second weight matrix includes:

Decomposing the first weight matrix into a third weight matrix; wherein the third weight matrix includes at least two sub-matrices;

The size of each submatrix in the at least two submatrices is determined according to a first formula, wherein the first formula is Q≈Q ₁ *Q ₂ *......*Q _n ; wherein, Q represents a first weight matrix; Q ₁ represents a first submatrix in the at least two submatrices; Q ₂ represents a second submatrix in the at least two submatrices; and Q _n represents an nth submatrix in the at least two submatrices;

The size of each submatrix of the at least two submatrices is adjusted, and a second weight matrix satisfying a preset accuracy is obtained by training a compressed machine learning model.

In a specific implementation, the operation symbol “*” in the first formula represents a matrix multiplication operation.

In one implementation manner, when the third weight matrix includes two sub-matrices, the first formula can be expressed as:

Q≈Q ₁ *Q ₂ (1.1)

Wherein, Q represents a first weight matrix, _Q1 represents a first submatrix among the at least two submatrices, and _Q2 represents a second submatrix among the at least two submatrices.

In another embodiment, when the third weight matrix includes at least two sub-matrices, the first formula can be expressed as:

Q≈Q ₁ *Q ₂ *......*Q _n (1.2)

In the above formula (1.2), n is a positive integer greater than 2.

In a specific implementation, in the process of compressing the first weight matrix into the second weight matrix, when applied to different neural networks (for example, a fully connected layer neural network, a convolutional layer neural network, an LSTM layer neural network), the above-mentioned decomposition operation on the first weight matrix, solving each of the at least two sub-matrices, and adjusting each of the at least two sub-matrices to obtain a second weight matrix that meets the preset accuracy will be different, which will be specifically explained below:

(1) Fully connected layer neural network:

A fully connected layer means that for the n-1 layer and the n layer, any node in the n-1 layer is connected to all the nodes in the n layer. Specifically, referring to FIG5A, it is a schematic diagram of the structure of a one-dimensional fully connected layer of a neural network provided in an embodiment of the present invention. As shown in FIG5A, the neural network includes an input layer, a hidden layer, and an output layer, wherein the two-dimensional parameter matrix of the fully connected layer between the input layer and the hidden layer is (3,4), and the two-dimensional parameter matrix (3,4) indicates that in the fully connected layer structure between the input layer and the hidden layer, the number of input neurons is 3, the number of output neurons is 4, and the number of weights is 12. In a specific implementation, these 12 weights can be represented as a weight matrix of 4 rows and 3 columns, and the representation of the weight matrix can be shown in FIG5B.

In a fully connected layer neural network, the first formula includes: M≈M ₁ *M ₂ ; the two sub-matrices refer to a first sub-matrix M1 and a second sub-matrix M2, the M1 is an N _in *K matrix, and the M2 is a K*N _out matrix; wherein K is a compression parameter, N _in is the number of input neurons of the neural network, and N _out is the number of output neurons of the neural network; the compression parameter is used to characterize the number of output neurons of the M1 and the number of input neurons of the M2, and K is a positive integer greater than 0 and less than or equal to min(N _in , N _out ).

As mentioned above, the process of adjusting the size of each of the two sub-matrices is actually a dynamic change process of the compression parameter K value to find the optimal compression parameter K. In practical applications, a binary search method can be used to determine the compression parameter K value in the fully connected layer neural network, so as to obtain a second weight matrix that meets the preset accuracy. In one embodiment, the compression parameter K determined by the binary search method can make the second weight matrix meet the preset accuracy. In another embodiment, the compression parameter K determined by the binary search method can make the second weight matrix meet the preset accuracy while the compression ratio of the first weight matrix to the second weight matrix meets the preset compression ratio, that is, a better compression effect is obtained for the compression of the neural network model.

In the specific implementation, the compression parameter K has different values, that is, the first weight matrix is compressed based on multiple different compression ratios. Here, in the fully connected layer neural network, the compression ratio is

Next, we will explain in detail how to use binary search to determine the compression parameter K value. First, set two parameters KL and KR. In the initialization, let KL = 1, KR = min (N _in , N _out ). In the process of adjusting the parameters, K = (KL + KR) / 2. If the second weight matrix represented by _M1 * _M2 causes the accuracy of the compressed neural network model to decrease by X% (here, X = 1 to 10, etc.), adjust the parameter KL so that KL = K. If the second weight matrix represented by _M1 * _M2 causes the compressed neural network model to meet the preset accuracy, then adjust KR so that KR = K, and repeat the above steps until the end condition K = KL or K = KR is met.

Taking the fully connected layer between the input layer and the hidden layer in FIG5A as an example, the compression parameter K value is a positive integer greater than 0 and less than or equal to 3. The compression parameter K=2 is determined by the above binary search method, that is, the first submatrix M1 in the second weight matrix that meets the preset accuracy is a (3,2) matrix, and the second submatrix M2 is a (2,4) matrix. Specifically, the compression of the fully connected layer between the input layer and the hidden layer in FIG5A can be shown in FIG5C.

In one embodiment, when the first formula is expressed as shown in formula (1.2), that is, the number of sub-matrices in the third weight matrix is n, where n is a positive integer greater than 2, and the number of compression parameters K is (n-1). In practical applications, an adaptive algorithm (e.g., a genetic algorithm) can be used to determine the (n-1) compression parameter K values in the fully connected layer neural network, thereby obtaining a second weight matrix that meets the preset accuracy and/or meets the compression effect. The following specifically describes how to use a genetic algorithm to determine the (n-1) compression parameter K values in the fully connected layer neural network:

Step 1: Randomly generate a population: Set the size of the population to P, set the maximum number of iterations T _max , for example, T _max = 100. In the initial state, set the iteration counter t = 0; crossover probability Pc = A (for example, A = 0.4), mutation probability Pm = B (for example, B = 0.6), each row of the population matrix represents a gene string individual, and each column represents the number of individuals; here, each individual is a set of solutions about the value of the compression parameter K (for example, K _j );

Step 2: Calculate the fitness of each individual in the population; here, fitness refers to the compression ratio and/or accuracy of the first weight matrix and the second weight matrix corresponding to the individual, where the compression ratio is used to characterize the compression effect on the neural network.

Step 3: Apply the selection operator to the population and directly pass the optimized individuals to the next generation;

Step 4: Apply the crossover operator to the population. For any two individuals, randomly generate several gene string position points and exchange the values of the two individuals at that position.

Step 5: Apply the mutation operator to the population. For any individual, randomly generate the positions of several gene strings and then change the values at these positions. Here, mutation refers to randomly changing the value of _Kj .

Step 6: Keep the individuals with the highest fitness in each generation and enter the next generation;

Step 7: Determine whether the maximum number of iterations T _max is reached. If t=T _max , output the individual with the maximum fitness and terminate the calculation; otherwise, jump to step 2 to continue.

Therefore, the (n-1) compression parameter K values in the fully connected layer neural network can be determined according to the above genetic algorithm.

(2) Convolutional neural network:

Taking the convolution layer of a neural network as an example, as shown in Figure 5D, the convolution layer can be considered as a four-dimensional matrix (N _fin , N _fout , K _x , _Ky ), where N _fin is the number of input feature images, N _fout is the number of output feature images, and (K _x , _Ky ) is the size of the convolution kernel in the convolution layer.

In a convolutional layer neural network, the convolutional layer neural network includes N _fin *N _fout convolution kernels; the first formula includes: F≈F ₁ *F ₂ ; wherein F represents any one of the N _fin *N _fout convolution kernels; the F1 is the first sub-convolution kernel; the F2 is the second sub-convolution kernel; the first sub-convolution kernel F1 is (K _x , R), the second sub-convolution kernel F2 is (R, _Ky ), (K _x , _Ky ) represents the size of the convolution kernel, R is a compression parameter, and R is a positive integer greater than 0 and less than or equal to min(K _x , _Ky ).

As mentioned above, the process of adjusting the size of each of the two sub-matrices is actually a dynamic change process of the compression parameter R value to find the best compression parameter R. In practical applications, a binary search method can be used to determine the compression parameter R value in the convolutional layer neural network, so as to obtain a second weight matrix that meets the preset accuracy.

In the specific implementation, the compression parameter R value is different, that is, the first weight matrix is compressed based on multiple different compression ratios. Here, in the convolutional layer neural network, the compression ratio

In the embodiment of the present invention, the implementation process of determining the value of the compression parameter R by using a binary search method is described in the foregoing text and will not be elaborated herein.

For example, in the convolutional layer neural network structure shown in FIG5D , the convolutional layer includes 4 convolution kernels, and the convolution kernel size is 3*3, wherein, in the first convolution kernel, N _fin =4, N _fout =6, and the compression parameter R value is a positive integer greater than 0 and less than or equal to 4. The compression parameter R=4 is determined by the above-mentioned binary search method, that is, the first sub-convolution kernel F1 in the first convolution kernel that meets the preset accuracy is a (3,4) matrix, and the second sub-convolution kernel F2 is a (4,3) matrix. In one embodiment, for other convolution kernels shown in FIG5D , the same compression method as the first convolution kernel can be used, or a compression method different from the first convolution kernel can be used, which is not specifically limited in the embodiment of the present invention.

In one embodiment, when the first formula is expressed as shown in formula (1.2), an adaptive algorithm (for example, a genetic algorithm) can be used to determine the value of the compression parameter R in the convolutional layer neural network. Please refer to the above description for the specific implementation process, which will not be elaborated here.

(3) LSTM layer neural network:

Taking the LSTM layer (Long Short-term Memory) of the neural network as an example, the weights of the LSTM layer are composed of multiple fully connected layer weights. Assume that the weights of the LSTM layer are composed of t fully connected layer weights, and t is a positive integer greater than 0. For example, the weights of the j-th fully connected layer are (N _{in_j} ,N _{out_j} ), where N _{in_j} represents the number of input neurons of the j-th fully connected layer, N _{out_j} represents the number of output neurons of the j-th fully connected layer, and the number of weights of the j-th fully connected layer is N _{in_j} *N _{out_j} .

In an LSTM layer neural network, the LSTM layer includes N fully connected layers, where N is a positive integer greater than 0; for the j-th fully connected layer, the first formula includes: M _j ≈M _{j_1} *M _{j_2} ; the two sub-matrices in the j-th fully connected layer include a first sub-matrix M _{j_1} and a second sub-matrix M _{j_2} , where M _{j_1} is an N _{in_j} *S matrix, and M _{j_2} is an S*N _{out_j} matrix; wherein S is a compression parameter, N _{in_j} is the number of input neurons of the j-th fully connected layer of the neural network, and N _{out_j} is the number of output neurons of the j-th fully connected layer of the neural network; the compression parameter is used to characterize the number of output neurons of the M _{j_1} and the number of input neurons of the M _{j_2} , and S is a positive integer greater than 0 and less than or equal to min(N _{in_j} , N _{out_j} ).

As mentioned above, the process of adjusting the size of each of the two sub-matrices is actually a dynamic change process of the compression parameter S value to find the best compression parameter S. In practical applications, a binary search method can be used to determine the compression parameter S value in the convolutional layer neural network, thereby obtaining a second weight matrix that meets the preset accuracy.

In the specific implementation, for the jth fully connected layer, the compression parameter S value is different, that is, the first weight matrix is compressed based on multiple different compression ratios. Here, in the jth fully connected layer, the compression ratio

In the embodiment of the present invention, the implementation process of determining the value of the compression parameter S in the j-th fully connected layer by using a binary search method is described in the above text and will not be repeated here.

Taking the neural network architecture shown in FIG5A as an example, the neural network includes an input layer, a hidden layer, and an output layer, wherein the first fully connected layer is between the input layer and the hidden layer, and the second fully connected layer is between the hidden layer and the output layer. For the specific elaboration of this fully connected layer structure between the input layer and the hidden layer (that is, the first fully connected layer), please refer to the above description, and no further elaboration is given here. As can be seen from FIG5A, the two-dimensional parameter matrix of this fully connected layer between the hidden layer and the output layer is (4,2), and the two-dimensional parameter matrix (4,2) indicates that in the fully connected layer structure between the hidden layer and the output layer, the number of input neurons is 4, the number of output neurons is 2, and the number of weights is 8. In a specific implementation, these 8 weights can be represented as a 2-row 4-column weight matrix, and the expression of the weight matrix can be shown in FIG5E. Then, in this case, the value of the compression parameter S is a positive integer greater than 0 and less than or equal to 2. The compression parameter S=2 is determined by binary search, that is, in the second fully connected layer, the first submatrix M ₂ _ ₁ in the second weight matrix that meets the preset accuracy is a (4,2) matrix, and the second submatrix M ₂ _ ₂ is a (2,2) matrix. Specifically, the compression of the fully connected layer between the input layer and the hidden layer and the fully connected layer between the hidden layer and the output layer in FIG. 5A can be shown in FIG. 5F.

In one embodiment, when the first formula is expressed as shown in formula (1.2), an adaptive algorithm (for example, a genetic algorithm) can be used to determine the value of the compression parameter S in the LSTM layer neural network. For the specific implementation process, please refer to the above description and no further details will be given here.

Step S104: performing neural network calculation according to second input data, wherein the second input data includes a second weight matrix and neuron data.

In practical applications, the neural network calculations involved here may include artificial neural network operations, convolutional neural network operations, and so on.

Taking artificial neural network operation as an example, for artificial neural network operation, if the artificial neural network operation has multi-layer operation, the input neurons and output neurons of the multi-layer operation do not refer to the neurons in the input layer and the neurons in the output layer of the entire neural network, but for any two adjacent layers in the network, the neurons in the lower layer of the network forward operation are the input neurons, and the neurons in the upper layer of the network forward operation are the output neurons. Taking convolutional neural network as an example, suppose a convolutional neural network has L layers, K＝1,2,...,L-1, for the Kth layer and the K+1th layer, we call the Kth layer the input layer, the neurons therein are the input neurons, and the K+1th layer is called the output layer, the neurons therein are the output neurons. That is, except for the top layer, each layer can be used as an input layer, and the next layer is the corresponding output layer.

In a specific implementation, the operation in the neural network can be the operation of a layer in the neural network. For a multi-layer neural network, the implementation process is that in the forward operation, after the execution of the previous layer of artificial neural network is completed, the operation instruction of the next layer will use the output neuron calculated in the operation unit as the input neuron of the next layer for operation (or perform certain operations on the output neuron and then use it as the input neuron of the next layer), and at the same time, the weights are also replaced by the weights of the next layer; in the reverse operation, when the reverse operation of the previous layer of artificial neural network is completed, the operation instruction of the next layer will use the input neuron gradient calculated in the operation unit as the output neuron gradient of the next layer for operation (or perform certain operations on the input neuron gradient and then use it as the output neuron gradient of the next layer), and at the same time, the weights are replaced by the weights of the next layer.

The embodiment of the present invention can obtain at least two sub-matrices containing compression parameters by decomposing the first weight matrix. Then, each of the at least two sub-matrices is solved according to the formula, and the compressed neural network is trained to obtain a second weight matrix that meets the preset accuracy. This solves the problem of irregular topological structure of the neural network that is easily caused by the use of the neural network pruning algorithm in the prior art, and can deeply compress the neural network, reduce the calculation amount of the neural network, and improve the operation speed.

In order to better implement the above-mentioned solution of the embodiment of the present invention, the present invention also provides a neural network compression device, which is described in detail below with reference to the accompanying drawings:

FIG. 17A is a schematic diagram of the structure of a neural network compression device provided by an embodiment of the present invention, wherein the neural network compression device includes: an acquisition unit 300, a compression unit 13, and a calculation unit 304;

Wherein, the acquisition unit 300 is used to acquire first input data; wherein, the first input data includes a first weight matrix;

The compression unit 13 is used to compress the first weight matrix into a second weight matrix; wherein the second weight matrix includes at least two sub-matrices;

The calculation unit 304 is used to perform neural network calculation according to the second input data, wherein the second input data includes the second weight matrix and input neuron data.

In one embodiment, as shown in FIG17B , the compression unit 13 includes a decomposition unit 130 , a solution unit 131 , and a training unit 132 ;

The decomposition unit 130 is used to decompose the first weight matrix into a third weight matrix; wherein the third weight matrix includes at least two sub-matrices;

The solving unit 131 is used to determine the size of each submatrix in the at least two submatrices according to a first formula, wherein the first formula is Q≈Q ₁ *Q ₂ *......*Q _n ; wherein Q represents a first weight matrix; Q ₁ represents a first submatrix in the at least two submatrices; Q ₂ represents a second submatrix in the at least two submatrices; and Q _n represents an nth submatrix in the at least two submatrices;

The training unit 132 is used to adjust the size of each sub-matrix of the at least two sub-matrices, and obtain a second weight matrix that meets a preset accuracy by training the compressed machine learning model.

Optionally, the solving unit 131 is specifically configured to determine the size of each submatrix of the at least two submatrices according to the first formula and a second formula, wherein the second formula is ||QQ ₁ *Q ₂ *......*Q _n ||≤T, where T represents a preset error threshold.

Optionally, the training unit 132 is specifically used to adjust the size of each sub-matrix of the at least two sub-matrices, and to obtain a second weight matrix that meets a preset accuracy and has a compression ratio that meets a preset compression ratio with the first weight matrix by training the compressed machine learning model.

Optionally, the neural network includes a fully connected layer neural network; the first formula includes: M≈M ₁ *M ₂ ; the two sub-matrices refer to a first sub-matrix M1 and a second sub-matrix M2, the M1 is an N _in *K matrix, and the M2 is a K*N _out matrix; wherein K is a compression parameter, N _in is the number of input neurons of the neural network, and N _out is the number of output neurons of the neural network; the compression parameter is used to characterize the number of output neurons of M1 and the number of input neurons of M2, and K is a positive integer greater than 0 and less than or equal to min(N _in , N _out ).

Optionally, the neural network includes a convolutional layer neural network; the convolutional layer neural network includes N _fin *N _fout convolution kernels; the first formula includes: F≈F ₁ *F ₂ ; wherein F represents any one of the N _fin *N _fout convolution kernels; the F1 is the first sub-convolution kernel; the F2 is the second sub-convolution kernel; the first sub-convolution kernel F1 is (K _x , R), the second sub-convolution kernel F2 is (R, _Ky ), (K _x , _Ky ) represents the size of the convolution kernel, R is a compression parameter, and R is a positive integer greater than 0 and less than or equal to min(K _x , _Ky ).

Optionally, the neural network includes an LSTM layer neural network, the LSTM layer includes N fully connected layers, and N is a positive integer greater than 0; for the j-th fully connected layer, the first formula includes: M _j ≈M _{j_1} *M _{j_2} ; the two sub-matrices in the j-th fully connected layer include a first sub-matrix M _{j_1} and a second sub-matrix M _{j_2} , the M _{j_1} is an N _{in_j} *S matrix, and the M _{j_2} is an S*N _{out_j} matrix; wherein S is a compression parameter, N _{in_j} is the number of input neurons of the j-th fully connected layer of the neural network, and N _{out_j} is the number of output neurons of the j-th fully connected layer of the neural network; the compression parameter is used to characterize the number of output neurons of the M _{j_1} and the number of input neurons of the M _{j_2} , and S is a positive integer greater than 0 and less than or equal to min(N _{in_j} , N _{out_j} ).

The embodiment of the present invention can obtain at least two sub-matrices containing compression parameters by decomposing the first weight matrix. Then, each of the at least two sub-matrices is solved according to the formula, and the compressed neural network is trained to obtain a second weight matrix that meets the preset accuracy. This solves the problem of irregular topological structure of the neural network that is easily caused by the use of the neural network pruning algorithm in the prior art. The deep compression of the neural network can reduce the calculation amount of the neural network and improve the operation speed.

In order to better implement the above solution of the embodiment of the present invention, the present invention also provides another electronic device, which is described in detail below with reference to the accompanying drawings:

As shown in FIG18 , a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, the electronic device 40 may include a processor 401, a memory 404 and a communication module 405, and the processor 401, the memory 404 and the communication module 405 may be interconnected via a bus 406. The memory 404 may be a high-speed random access memory (RAM) memory, or a non-volatile memory, such as at least one disk memory. The memory 404 may optionally be at least one storage system located away from the aforementioned processor 401. The memory 404 is used to store application code, and may include an operating system, a network communication module, a user interface module and a data processing program. The communication module 405 is used to interact with external devices for information; the processor 401 is configured to call the program code and execute the following steps:

Acquire first input data; wherein the first input data includes a first weight matrix;

Compressing the first weight matrix into a second weight matrix; wherein the second weight matrix includes at least two sub-matrices;

A neural network calculation is performed according to second input data, wherein the second input data includes the second weight matrix and input neuron data.

The processor 401 compresses the first weight matrix into a second weight matrix; wherein the second weight matrix includes at least two sub-matrices, which may include:

Decomposing the first weight matrix into a third weight matrix; wherein the third weight matrix includes at least two sub-matrices;

Determine the size of each submatrix in the at least two submatrices, the first formula is Q≈Q ₁ *Q ₂ *......*Q _n ; wherein Q represents a first weight matrix; Q ₁ represents a first submatrix in the at least two submatrices; Q ₂ represents a second submatrix in the at least two submatrices; Q _n represents an nth submatrix in the at least two submatrices;

The size of each submatrix of the at least two submatrices is adjusted, and a second weight matrix satisfying a preset accuracy is obtained by training a compressed machine learning model.

The processor 401 determines each sub-matrix of the at least two sub-matrices according to a first formula, where the first formula is Q≈Q ₁ *Q ₂ *...*Q _n , and may include:

The size of each of the two sub-matrices is determined according to the first formula and the second formula, wherein the second formula is ||QQ ₁ *Q ₂ *......*Q _n ||≤T, wherein T represents a preset error threshold.

The processor 401 adjusts the size of each of the at least two sub-matrices and obtains a second weight matrix that meets a preset accuracy by training the compressed machine learning model, which may include:

The size of each submatrix of the at least two submatrices is adjusted, and the compressed machine learning model is trained to obtain a second weight matrix that meets the preset accuracy and has a compression ratio that meets the preset compression ratio with the first weight matrix.

Wherein, the neural network is a fully connected layer neural network; the first formula includes: M≈M ₁ *M ₂ ; the two sub-matrices refer to a first sub-matrix M1 and a second sub-matrix M2, the M1 is an N _in *K matrix, and the M2 is a K*N _out matrix; wherein K is a compression parameter, N _in is the number of input neurons of the neural network, and N _out is the number of output neurons of the neural network; the compression parameter is used to characterize the number of output neurons of the M1 and the number of input neurons of the M2, and K is a positive integer greater than 0 and less than or equal to min(N _in , N _out ).

In which, the neural network is a convolutional layer neural network; the convolutional layer neural network includes N _fin *N _fout convolution kernels; the first formula includes: F≈F ₁ *F ₂ ; wherein F represents any one of the N _fin *N _fout convolution kernels; the F1 is the first sub-convolution kernel; the F2 is the second sub-convolution kernel; the first sub-convolution kernel F1 is (K _x , R), the second sub-convolution kernel F2 is (R, _Ky ), (K _x , _Ky ) represents the size of the convolution kernel, R is a compression parameter, and R is a positive integer greater than 0 and less than or equal to min(K _x , _Ky ).

Wherein, the neural network is an LSTM layer neural network; the LSTM layer neural network includes N fully connected layers, where N is a positive integer greater than 0; for the j-th fully connected layer, the first formula includes: M _j ≈M _{j_1} *M _{j_2} ; the two sub-matrices in the j-th fully connected layer include a first sub-matrix M _{j_1} and a second sub-matrix M _{j_2} , where M _{j_1} is an N _{in_j} *S matrix, and M _{j_2} is an S*N _{out_j} matrix; wherein S is a compression parameter, N _{in_j} is the number of input neurons of the j-th fully connected layer of the neural network, and N _{out_j} is the number of output neurons of the j-th fully connected layer of the neural network; the compression parameter is used to characterize the number of output neurons of the M _{j_1} and the number of input neurons of the M _{j_2} , and S is a positive integer greater than 0 and less than or equal to min(N _{in_j} , N _{out_j} ).

It should be noted that the execution steps of the processor in the electronic device 40 in the embodiment of the present invention can refer to the specific implementation methods of the operation of the electronic device in the embodiment of Figure 16 in the above-mentioned method embodiments, which will not be repeated here.

In practical applications, the processor 401 in the electronic device 40 includes but is not limited to only one. In one embodiment, the electronic device 40 also includes a graphics processor GPU (GPU, Graphic ProcessingUni) for processing images, and may also include an embedded neural network processor (NPU, Neural-network Process Units). In this case, the compression method for the neural network can be integrated into the NPU. In one embodiment, the processor 401 can control the NPU to execute the compression method for the first weight matrix.

In a specific implementation, as mentioned above, the electronic device 40 may include a data processing device, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a mobile phone, a driving recorder, a navigator, a sensor, a camera, a server, a cloud server, a camera, a camcorder, a projector, a watch, a headset, a mobile storage, a wearable device, a vehicle, a household appliance, and/or a medical device, which is not specifically limited in the embodiments of the present invention.

The embodiment of the present invention further provides a computer storage medium for storing computer software instructions used by the electronic device shown in FIG. 16, which includes a program for executing the method embodiment. By executing the stored program, the first weight matrix can be compressed to obtain a second weight matrix that meets the preset accuracy, thereby avoiding irregular topological structure of the neural network model and reducing the amount of computation of the neural network.

It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software program module.

If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.

The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims (19)

1. A computing device for performing machine learning calculations, the computing device comprising: a compression unit, an operation unit and a controller unit;

the controller unit is used for acquiring a compression request for first input data and indicating the compression unit to compress the first input data according to the compression request; wherein the first input data comprises a first weight matrix;

The compression unit is used for compressing the first weight matrix into a second weight matrix;

the controller unit is also used for acquiring second input data and a calculation instruction; the second input data includes the second weight matrix and input neuron data;

the controller unit is further configured to parse the calculation instruction to obtain a plurality of operation instructions, and send the plurality of operation instructions and the second input data to the operation unit;

the operation unit is used for acquiring the operation instruction and executing neural network calculation according to the operation instruction and the second input data; the neural network is a convolutional layer neural network; the compression unit includes: the system comprises a decomposition unit, a solving unit and a training unit;

The decomposing unit is used for decomposing the first weight matrix into a third weight matrix; wherein the third weight matrix comprises at least two sub-matrices;

the solving unit is used for determining the size of each sub-matrix in the at least two sub-matrices according to a first formula,

The training unit is used for adjusting the size of each of the at least two submatrices to obtain a second weight matrix;

The computing device is used for executing convolutional neural network computation; the convolutional layer neural network comprises N _fin*N_fout convolutional kernels; the first formula includes: f is approximately equal to F ₁*F₂; wherein F represents any one of the N _fin*N_fout convolution kernels; the F1 is a first sub convolution kernel; the F2 is a second sub convolution kernel; the first sub convolution kernel F1 is (K _x, R), the second sub convolution kernel F2 is (R, K _y),(K_x,K_y) representing the size of the convolution kernel, R is a compression parameter, and R is a positive integer greater than 0 and less than or equal to min (K _x,K_y);

The arithmetic unit includes: a master processing circuit and a plurality of slave processing circuits;

the main processing circuit executes preamble processing on the second input data and transmits data and operation instructions with the plurality of auxiliary processing circuits;

the plurality of slave processing circuits execute intermediate operation in parallel according to the data and operation instructions transmitted from the master processing circuit to obtain a plurality of intermediate results, and the plurality of intermediate results are transmitted to the master processing circuit;

and the main processing circuit executes subsequent processing on the plurality of intermediate results to obtain a calculation result of the calculation instruction.

2. The computing device of claim 1, wherein the computing device is configured to,

The first formula is Q approximately equal to Q ₁*Q₂*......*Q_n; wherein, Q represents a first weight matrix; the Q ₁ represents a first sub-matrix of the at least two sub-matrices; the Q ₂ represents a second sub-matrix of the at least two sub-matrices; the Q _n represents an nth sub-matrix of the at least two sub-matrices;

the training unit is used for adjusting the size of each of the at least two submatrices and obtaining a second weight matrix meeting preset precision by training the compressed machine learning model.

3. The computing device of claim 2, wherein the solving unit is configured to determine each of the at least two submatrices according to a first formula, where the first formula is q≡q ₁*Q₂*......*Q_n, and includes:

The solving unit is specifically configured to determine a size of each of the at least two submatrices according to the first formula and the second formula, where the second formula is ||q-Q ₁*Q₂*......*Q_n |t, and the T represents a preset error threshold.

4. The computing device of claim 2, wherein the training unit to resize each of the at least two submatrices and to derive the second weight matrix satisfying the preset accuracy by training the compressed machine learning model comprises:

the training unit is specifically configured to adjust a size of each of the at least two submatrices, and obtain a second weight matrix that meets a preset precision and a compression ratio with the first weight matrix meets a preset compression ratio by training a compressed machine learning model.

5. The computing device of any of claims 2 to 4, wherein the computing device is configured to perform full-connection layer neural network computations; the at least two sub-matrices include two sub-matrices; the first formula includes: m is approximately equal to M ₁*M₂; the two submatrices comprise a first submatrix M1 and a second submatrix M2, wherein M1 is an N _in x K matrix, and M2 is a K x N _out matrix; wherein K is a compression parameter, N _in is the number of input neurons of the neural network, and N _out is the number of output neurons of the neural network; the compression parameter is used for representing the number of output neurons of the M1 and the number of input neurons of the M2, and K is a positive integer which is more than 0 and less than or equal to min (N _in,N_out).

6. The computing device of any of claims 2-4, wherein the computing device is configured to perform LSTM layer neural network computations, the LSTM layer comprising N fully connected layers, the N being a positive integer greater than 0; for the j-th fully connected layer, the first formula includes: m _j≈M_{j_1}*M_{j_2}; the two submatrices in the j-th full connection layer include a first submatrix M _j_₁ and a second submatrix M _{j_2}, where M _{j_1} is an N _{in_j} x S matrix, and M _{j_2} is an S x N _{out_j} matrix; wherein S is a compression parameter, N _{in_j} is the number of input neurons of the j-th full-connection layer of the neural network, and N _{out_j} is the number of output neurons of the j-th full-connection layer of the neural network; the compression parameter is used for representing the number of output neurons of the M _{j_1} and the number of input neurons of the M _{j_2}, and S is a positive integer greater than 0 and less than or equal to min (N _{in_j},N_{out_j}).

7. The computing device of claim 1, wherein the computing device further comprises: a storage unit and a direct memory access unit, the storage unit comprising: registers, caches, any combination;

The buffer is used for storing the first input data and the second input data;

The register is used for storing the first input data and the second input data;

The cache includes a scratch pad cache;

The controller unit includes: an instruction storage unit, an instruction processing unit and a storage queue unit;

the instruction storage unit is used for storing calculation instructions related to the artificial neural network operation;

the instruction processing unit is used for analyzing the calculation instructions to obtain a plurality of operation instructions;

The store queue unit is configured to store an instruction queue, where the instruction queue includes: a plurality of operation instructions or calculation instructions to be executed according to the front-back sequence of the queue;

The controller unit includes a main processing circuit including: a dependency relationship processing unit;

The dependency relation processing unit is used for determining whether a first operation instruction and a zeroth operation instruction before the first operation instruction have an association relation, if so, caching the first operation instruction in the instruction storage unit, and after the zeroth operation instruction is executed, extracting the first operation instruction from the instruction storage unit and transmitting the first operation instruction to the operation unit;

the determining whether the association relationship exists between the first operation instruction and the zeroth operation instruction before the first operation instruction includes:

Extracting a first storage address interval of required data in the first operation instruction according to the first operation instruction, extracting a zeroth storage address interval of required data in the zeroth operation instruction according to the zeroth operation instruction, determining that the first operation instruction and the zeroth operation instruction have an association relation if the first storage address interval and the zeroth storage address interval have overlapping areas, and determining that the first operation instruction and the zeroth operation instruction do not have an association relation if the first storage address interval and the zeroth storage address interval do not have overlapping areas.

8. A machine learning computing device, characterized in that the machine learning computing device comprises one or more computing devices according to any one of claims 1-7, and is configured to obtain input data and control information to be computed from other processing devices, perform specified machine learning operations, and transmit the execution results to the other processing devices through I/O interfaces;

when the machine learning computing device comprises a plurality of computing devices, the computing devices are connected through a specific structure and data are transmitted;

the computing devices are interconnected through a PCIE bus of a rapid external equipment interconnection bus and transmit data so as to support larger-scale machine learning operation; a plurality of the computing devices share the same control system or have respective control systems; a plurality of computing devices share memory or have respective memories; the manner in which the plurality of computing devices are interconnected is an arbitrary interconnection topology.

9. A combination processing device, comprising the machine learning computing device of claim 8, a universal interconnect interface, a storage device, and other processing devices;

The machine learning operation device interacts with the other processing devices to jointly complete the calculation operation designated by the user; the storage device is connected with the machine learning operation device and the other processing device respectively and used for storing data of the machine learning operation device and the other processing device.

10. A neural network chip, characterized in that the machine learning chip includes the machine learning arithmetic device according to claim 8 or the combination processing device according to claim 9.

11. An electronic device, characterized in that, the electronic device comprising the chip of claim 10.

12. A board, characterized in that, the board includes: a memory device, an interface device and a control device, and a neural network chip as claimed in claim 10;

The neural network chip is respectively connected with the storage device, the control device and the interface device;

the storage device is used for storing data;

The interface device is used for realizing data transmission between the chip and external equipment;

the control device is used for monitoring the state of the chip.

13. A computing method of executing a machine learning model, characterized in that the computing method is applied to a computing device for executing a machine learning calculation; the computing device includes: a compression unit, an operation unit and a controller unit; the method comprises the following steps:

the controller unit obtains a compression request for first input data and instructs the compression unit to compress the first input data according to the compression request; wherein the first input data comprises a first weight matrix;

The compression unit is used for compressing the first weight matrix into a second weight matrix;

the controller unit acquires second input data and a calculation instruction; the second input data includes the second weight matrix and input neuron data;

the controller unit analyzes the calculation instruction to obtain a plurality of calculation instructions, and sends the plurality of calculation instructions and the second input data to the calculation unit;

The operation unit obtains the operation instruction and executes neural network calculation according to the operation instruction and the second input data; the neural network is a convolutional layer neural network;

the compression unit includes: the device comprises a decomposition unit, a solving unit and a training unit;

The decomposing unit is used for decomposing the first weight matrix into a third weight matrix; wherein the third weight matrix comprises at least two sub-matrices;

the solving unit is used for determining the size of each sub-matrix in the at least two sub-matrices according to a first formula,

The training unit is used for adjusting the size of each of the at least two submatrices to obtain a second weight matrix;

The computing device is used for executing convolutional neural network computation; the convolutional layer neural network comprises N _fin*N_fout convolutional kernels; the first formula includes: f is approximately equal to F ₁*F₂; wherein F represents any one of the N _fin*N_fout convolution kernels; the F1 is a first sub convolution kernel; the F2 is a second sub convolution kernel; the first sub convolution kernel F1 is (K _x, R), the second sub convolution kernel F2 is (R, K _y),(K_x,K_y) representing the size of the convolution kernel, R is a compression parameter, and R is a positive integer greater than 0 and less than or equal to min (K _x,K_y);

The arithmetic unit includes: a master processing circuit and a plurality of slave processing circuits;

the main processing circuit executes preamble processing on the second input data and transmits data and operation instructions with the plurality of auxiliary processing circuits;

the plurality of slave processing circuits execute intermediate operation in parallel according to the data and operation instructions transmitted from the master processing circuit to obtain a plurality of intermediate results, and the plurality of intermediate results are transmitted to the master processing circuit;

and the main processing circuit executes subsequent processing on the plurality of intermediate results to obtain a calculation result of the calculation instruction.

14. The method of claim 13, wherein the step of determining the position of the probe is performed,

The first formula is Q approximately equal to Q ₁*Q₂*......*Q_n; wherein, Q represents a first weight matrix; the Q ₁ represents a first sub-matrix of the at least two sub-matrices; the Q ₂ represents a second sub-matrix of the at least two sub-matrices; the Q _n represents an nth sub-matrix of the at least two sub-matrices;

the training unit is used for adjusting the size of each of the at least two submatrices and obtaining a second weight matrix meeting preset precision by training the compressed machine learning model.

15. The method of claim 14, wherein the solving unit is configured to determine each of the at least two sub-matrices according to a first formula, where the first formula is q≡q ₁*Q₂*......*Q_n, and includes:

The solving unit is specifically configured to determine a size of each of the at least two submatrices according to the first formula and the second formula, where the second formula is ||q-Q ₁*Q₂*......*Q_n |t, and the T represents a preset error threshold.

16. The method according to claim 14, wherein the training unit, configured to adjust the size of each of the at least two sub-matrices, and obtain the second weight matrix satisfying the preset precision by training the compressed machine learning model, includes:

the training unit is specifically configured to adjust a size of each of the at least two submatrices, and obtain a second weight matrix that meets a preset precision and a compression ratio with the first weight matrix meets a preset compression ratio by training a compressed machine learning model.

17. The method of any of claims 14-16, wherein the computing device is configured to perform a full connection layer neural network calculation; the at least two sub-matrices include two sub-matrices; the first formula includes: m is approximately equal to M ₁*M₂; the two submatrices comprise a first submatrix M1 and a second submatrix M2, wherein M1 is an N _in x K matrix, and M2 is a K x N _out matrix; wherein K is a compression parameter, N _in is the number of input neurons of the neural network, and N _out is the number of output neurons of the neural network; the compression parameter is used for representing the number of output neurons of the M1 and the number of input neurons of the M2, and K is a positive integer which is more than 0 and less than or equal to min (N _in,N_out).

18. The method of any of claims 14-16, wherein the computing device is configured to perform LSTM layer neural network computations, the LSTM layer comprising N fully connected layers, the N being a positive integer greater than 0; for the j-th fully connected layer, the first formula includes: m _j≈M_j_₁*M_j_₂; the two submatrices in the j-th full connection layer include a first submatrix M _j_₁ and a second submatrix M _j_₂, where M _j_₁ is an N _{in_j} x S matrix, and M _j_₂ is an S x N _{out_j} matrix; wherein S is a compression parameter, N _{in_j} is the number of input neurons of the j-th full-connection layer of the neural network, and N _{out_j} is the number of output neurons of the j-th full-connection layer of the neural network; the compression parameter is used for representing the number of output neurons of the M _j_₁ and the number of input neurons of the M _j_₂, and S is a positive integer greater than 0 and less than or equal to min (N _{in_j},N_{out_j}).

19. The method of claim 13, wherein the computing device further comprises: a storage unit and a direct memory access unit, the storage unit comprising: registers, caches, any combination;

the cache stores the first input data and the second input data;

The register stores scalar quantities in the first input data and the second input data; the cache includes a scratch pad cache;

The controller unit includes: an instruction storage unit, an instruction processing unit and a storage queue unit;

The instruction storage unit stores calculation instructions related to the artificial neural network operation;

the instruction processing unit analyzes the calculation instructions to obtain a plurality of operation instructions;

The store queue unit stores an instruction queue, the instruction queue comprising: a plurality of operation instructions or calculation instructions to be executed according to the front-back sequence of the queue;

The controller unit includes a main processing circuit including: a dependency relationship processing unit;

The dependency relation processing unit determines whether a first operation instruction and a zeroth operation instruction before the first operation instruction have an association relation, if so, the first operation instruction is cached in the instruction storage unit, and after the execution of the zeroth operation instruction is finished, the first operation instruction is extracted from the instruction storage unit and transmitted to the operation unit;

the determining whether the association relationship exists between the first operation instruction and the zeroth operation instruction before the first operation instruction includes:

Extracting a first storage address interval of required data in the first operation instruction according to the first operation instruction, extracting a zeroth storage address interval of required data in the zeroth operation instruction according to the zeroth operation instruction, determining that the first operation instruction and the zeroth operation instruction have an association relation if the first storage address interval and the zeroth storage address interval have overlapping areas, and determining that the first operation instruction and the zeroth operation instruction do not have an association relation if the first storage address interval and the zeroth storage address interval do not have overlapping areas.