CN106503797A - The data for being received from neural memorizer are arranged the neutral net unit and collective with neural memorizer the neural pe array for being shifted - Google Patents

Abstract

A neural network unit includes a first memory, a second memory and a neural processing unit array. The first memory is loaded with elements of the data matrix. The second memory is loaded with elements of the convolution kernel. Each neural processing unit includes a multitasking register, a register, an accumulator and an arithmetic unit. A multitasking buffer receives elements from the first memory and receives multitasking buffer outputs of adjacent neural processing units. The cache receives elements from the second memory. The arithmetic unit receives the output of the register, the multitasking register and the accumulator and performs a multiply-accumulate operation on it. For each sub-matrix, the arithmetic unit selectively receives elements from the first memory or the multitasking buffer of the adjacent neural processing unit, and performs a series of multiply-accumulate operations to accumulate the result of the convolution operation into the accumulator.

Description

Neural network units with neural memory and collectives will receive from neural memory array of neural processing units for shifting the data columns of the

technical field

The invention relates to a processor, in particular to a processor for improving the computing performance and efficiency of an artificial neural network.

This application claims international priority to the following US Provisional Application. The entire contents of these priority claims are incorporated herein by reference.

This application is related to the following concurrently filed US application. These related applications are hereby incorporated by reference in their entirety.

Background technique

In recent years, artificial neural networks (ANN) have attracted renewed attention. These studies are often called deep learning, computer learning, and similar terms. The increase in computing power of general-purpose processors has also fueled interest in artificial neural networks decades later. Recent applications of artificial neural networks include language and image recognition. The need to increase the computational performance and efficiency of artificial neural networks appears to be increasing.

Contents of the invention

In view of this, the present invention provides a neural network unit. The neural network unit includes a first memory, a second memory and a neural processing unit (NPU) array. The first memory is used for loading elements of a data matrix. The second memory is used for loading elements of a convolution kernel. The neural processing unit array is coupled to the first memory and the second memory. Each neural processing unit includes a multitasking register, a register, an accumulator and an arithmetic unit. Wherein, the multitasking register has an output, and the multitasking register receives a corresponding element from a row of the first memory, and receives an output of the multitasking register of an adjacent neural processing unit. The register has an output, and the register receives a corresponding element from a column of the second memory. The accumulator has an output. The arithmetic unit receives the output of the register, the multitasking register and the accumulator, and performs a multiply-accumulate operation on it. Wherein, for each sub-matrix in the plurality of sub-matrices of the data matrix, each arithmetic unit selectively receives elements from the first memory or elements output from the multi-tasking buffer of the adjacent neural processing unit, and performs a series of multiplications The accumulation operation is to accumulate the result of a convolution operation of the sub-matrix and the convolution kernel into the accumulator.

The invention also provides a method for operating a neural network unit. The neural network unit has a neural processing unit array. Each neural processing unit includes a multitasking register, a register, an accumulator and an arithmetic unit. Wherein, the multitasking register has an output, and the multitasking register receives a corresponding element from a column of a first memory, and receives the multitasking register output of an adjacent neural processing unit. The register has an output, and the register receives a corresponding element from a column of a second memory. The accumulator has an output. The arithmetic unit receives the output of the register, the multitasking register and the accumulator, and performs a multiply-accumulate operation on it. The method includes: using a first memory, loading elements of a data matrix; using a second memory, loading elements of a convolution kernel; for each sub-matrix of a plurality of sub-matrices of the data matrix: using each arithmetic unit, selectively receiving an element from the first memory or an element output from a multitasking register of an adjacent neural processing unit; and performing a series of multiply-accumulate operations, and accumulating a result of a convolution operation of the sub-matrix and the convolution kernel into the accumulator Inside.

The present invention also provides a computer program product encoded on at least one non-transitory computer usable medium for use by a computer device. The computer program product includes computer usable program code embodied on the medium for describing a neural network unit. The computer usable program code includes first program code, second program code and third program code. Wherein, the first program code is used to describe a first memory, and the first memory is loaded with elements of a data matrix. The second program code is used to describe a second memory, and the second memory is loaded with elements of a convolution kernel. The third program code is used to describe a neural processing unit array, and the neural processing unit array is coupled to the first memory and the second memory. Each neural processing unit includes a multitasking register, a register, an accumulator and an arithmetic unit. The multitasking register has an output, and the multitasking register receives a corresponding element from a column of the first memory, and receives a multitasking register output of an adjacent NPU. The register has an output, and the register receives a corresponding element from a column of the second memory. The accumulator has an output. The arithmetic unit receives the output of the register, the multitasking register and the accumulator, and performs a multiply-accumulate operation on it. For each of the plurality of sub-matrices of the data matrix, each arithmetic unit selectively receives elements from the first memory or from the multitasking buffer output of an adjacent neural processing unit, and performs a series of multiply-accumulate operations , and accumulate a convolution operation result of the sub-matrix and the convolution kernel into the accumulator.

The specific embodiments adopted by the present invention will be further described through the following embodiments and drawings.

Description of drawings

FIG. 1 is a schematic block diagram showing a processor including a neural network unit (NNU).

FIG. 2 is a schematic block diagram showing a neural processing unit (NPU) in FIG. 1 .

FIG. 3 is a block diagram showing N multitasking registers of N NPUs utilizing the NNU of FIG. 1 to perform rotation as N literals on a row of data literals retrieved from the data random access memory of FIG. 1 The operation of the rotator or circular shifter.

FIG. 4 is a table showing a program stored in the program memory of the NNU of FIG. 1 and executed by the NNU.

FIG. 5 is a timing diagram showing that the neural network unit executes the program of FIG. 4 .

FIG. 6A is a block diagram showing that the neural network unit of FIG. 1 executes the program of FIG. 4 .

FIG. 6B is a flow chart showing the operation of the processor of FIG. 1 executing an architectural program to perform a typical multiply-accumulate activation function operation associated with neurons of a hidden layer of an artificial neural network using a neural network unit, as performed by the program of FIG. 4 operation.

FIG. 7 is a block diagram showing another embodiment of the neural processing unit of FIG. 1 .

FIG. 8 is a block diagram showing still another embodiment of the neural processing unit of FIG. 1 .

FIG. 9 is a table showing a program stored in the program memory of the NNU of FIG. 1 and executed by the NNU.

FIG. 10 is a timing diagram showing that the neural network unit executes the program of FIG. 9 .

FIG. 11 is a block diagram showing an embodiment of the neural network unit of FIG. 1 . In the embodiment of Fig. 11, a neuron is divided into two parts, i.e. the start function unit part and the ALU part (this part also includes the shift register part), and each start function unit part is composed of a plurality of ALU parts shared.

FIG. 12 is a sequence diagram showing that the neural network unit of FIG. 11 executes the program of FIG. 4 .

FIG. 13 is a sequence diagram showing that the neural network unit of FIG. 11 executes the program of FIG. 4 .

FIG. 14 is a block schematic diagram showing a move to neural network (MTNN) architecture instruction and its operation corresponding to the part of the neural network unit in FIG. 1 .

FIG. 15 is a block schematic diagram showing a move to neural network (MTNN) architecture instruction and its operation corresponding to the part of the neural network unit in FIG. 1 .

FIG. 16 is a block diagram showing an embodiment of the data random access memory of FIG. 1 .

FIG. 17 is a block diagram showing an embodiment of the weight RAM and buffer of FIG. 1 .

FIG. 18 is a schematic block diagram showing the dynamically configurable neural processing unit of FIG. 1 .

Fig. 19 is a schematic block diagram showing that according to the embodiment of Fig. 18, using 2N multitasking registers of N neural processing units of the neural network unit of Fig. 1, for a row of data characters obtained by the data random access memory of Fig. 1 Executes like a rotator.

FIG. 20 is a table showing a program stored in the program memory of the NNU of FIG. 1 and executed by the NNU having the NPU shown in the embodiment of FIG. 18 .

FIG. 21 is a timing diagram showing a neural network unit executing the program of FIG. 20 with the neural processing unit shown in FIG. 18 executing in a narrow configuration.

FIG. 22 is a schematic block diagram showing the neural network unit of FIG. 1 , which has the neural processing unit shown in FIG. 18 to execute the program of FIG. 20 .

FIG. 23 is a block diagram illustrating another embodiment of the dynamically configurable neural processing unit of FIG. 1 .

FIG. 24 is a block diagram showing an example of a data structure used by the neural network unit of FIG. 1 to perform convolution operations.

FIG. 25 is a flow chart showing that the processor of FIG. 1 executes the architecture program to use the neural network unit to perform the convolution operation of the convolution kernel according to the data array of FIG. 24 .

FIG. 26A is a program listing of the neural network unit program, which uses the convolution kernel of FIG. 24 to perform the convolution operation of the data matrix and write it back to the weight random access memory.

FIG. 26B is a block diagram showing an embodiment of some fields of the control register of the NNU of FIG. 1 .

FIG. 27 is a block diagram showing an example of the weighted random access memory of FIG. 1 filled with input data for pooling operation performed by the neural network unit of FIG. 1 .

FIG. 28 is a program listing of a neural network unit program that performs common-source operations on the input data matrix of FIG. 27 and writes it back to the weight random access memory.

FIG. 29A is a block diagram showing an embodiment of the control register of FIG. 1 .

FIG. 29B is a block diagram showing another embodiment of the control register of FIG. 1 .

FIG. 29C is a block schematic diagram showing an embodiment of storing the reciprocal of FIG. 29A in two parts.

FIG. 30 is a block diagram illustrating an embodiment of the activation function unit (AFU) of FIG. 2 .

FIG. 31 shows an example of the operation of the activation function unit of FIG. 30 .

FIG. 32 shows a second example of the operation of the boot function unit of FIG. 30 .

FIG. 33 shows a third example of the operation of the activation functional unit of FIG. 30 .

FIG. 34 is a block diagram showing some details of the processor and the neural network unit of FIG. 1 .

Figure 35 is a block diagram showing a processor with a variable rate neural network unit.

FIG. 36A is a timing diagram showing an example of operation of a processor with an NNU operating in a normal mode, ie, operating at a primary clock rate.

FIG. 36B is a timing diagram showing an example of operation of a processor with a NNU operating in a relaxed mode, where the frequency of operation in the relaxed mode is lower than that of the main clock.

FIG. 37 is a flowchart showing the operation of the processor of FIG. 35 .

Figure 38 is a block diagram detailing the sequence of neural network units.

Figure 39 is a block diagram showing certain fields of the control and state registers of the NNU.

FIG. 40 is a block diagram showing an example of an Elman temporal recurrent neural network (RNN).

FIG. 41 is a block diagram showing an example of data configuration in the data RAM and weight RAM of the NNU when the NNU performs computations associated with the Elman time recurrent neural network of FIG. 40 .

42 is a table showing the program stored in the program memory of the neural network unit, which is executed by the neural network unit and uses data and weights according to the configuration of FIG. 41 to achieve an Elman temporal recurrent neural network.

FIG. 43 is a block diagram showing an example of a Jordan temporal recurrent neural network.

FIG. 44 is a block diagram showing an example of data configuration in the data RAM and weight RAM of the NNU when the NNU performs computations associated with the Jordan Time Recurrent Neural Network of FIG. 43 .

FIG. 45 is a table showing the program stored in the program memory of the neural network unit, which is executed by the neural network unit and uses data and weights according to the configuration of FIG. 44 to achieve a Jordan time recurrent neural network.

FIG. 46 is a block diagram showing an embodiment of a long short term memory (LSTM) cell.

FIG. 47 is a block diagram showing an example of data configuration in the data RAM and weight RAM of the NNU when the NNU performs calculations associated with the LSTM cell layer of FIG. 46 .

Fig. 48 is a table showing the program stored in the program memory of the neural network unit executed by the neural network unit and using the data and weights according to the configuration of Fig. 47 to achieve calculations associated with the LSTM cell layer.

FIG. 49 is a block diagram showing an embodiment of a neural network unit with output buffer masking and feedback capabilities within a group of neural processing units.

50 is a block diagram showing the data RAM, weight RAM and output buffer of the neural network unit of FIG. 49 when the neural network unit performs calculations associated with the LSTM cell layer of FIG. 46 An example of data configuration.

Fig. 51 is a table showing the program stored in the program memory of the neural network unit executed by the neural network unit of Fig. 49 and using data and weights according to the configuration of Fig. 50 to achieve calculations associated with the LSTM cell layer.

FIG. 52 is a block diagram showing an embodiment of a neural network unit. In this embodiment, a group of neural processing units has output buffer masking and feedback capabilities, and a shared activation function unit.

Fig. 53 is a block diagram showing the data RAM, weight RAM and output buffer of the neural network unit of Fig. 49 when the neural network unit performs calculations associated with the LSTM cell layer of Fig. 46 Another embodiment of data configuration.

Fig. 54 is a table showing the program stored in the program memory of the neural network unit executed by the neural network unit of Fig. 49 and using data and weights according to the configuration of Fig. 53 to achieve calculations associated with the LSTM cell layer.

Fig. 55 is a block diagram showing part of a neural processing unit according to another embodiment of the present invention.

56 is a block diagram showing the data RAM and weight RAM of the NNU when the NNU performs computations associated with the Jordan Time Recurrent Neural Network of FIG. 43 and utilizes the embodiment of FIG. 55. An example of data configuration.

Fig. 57 is a table showing the program stored in the program memory of the neural network unit, which is executed by the neural network unit and uses data and weights according to the configuration of Fig. 56 to achieve a Jordan time recurrent neural network.

detailed description

Processor with architectural neural network unit

FIG. 1 is a block diagram showing a processor 100 including a neural network unit (NNU) 121 . As shown in the figure, the processor 100 includes an instruction fetch unit 101, an instruction cache 102, an instruction translator 104, a renaming unit 106, a plurality of reservation stations 108, a plurality of media buffers 118, and a plurality of general purpose registers 116 , multiple execution units 112 and memory subsystems 114 outside the aforementioned neural network unit 121 .

The processor 100 is an electronic device as a central processing unit of an integrated circuit. The processor 100 receives input digital data, processes the data according to instructions fetched from the memory, and generates as its output a processing result of an operation indicated by the instruction. The processor 100 can be used in a desktop computer, a mobile device, or a tablet computer, and is used for applications such as computing, word processing, multimedia display, and web browsing. The processor 100 can also be installed in an embedded system to control various devices including equipment, mobile phones, smart phones, vehicles, and industrial controllers. The central processing unit is an electronic circuit (ie, hardware) that executes instructions of a computer program (or called a computer application program or application program) by performing operations on data, including arithmetic, logic, and input/output. An integrated circuit is a group of electronic circuits fabricated in small semiconductor materials, usually silicon. Integrated circuit is also commonly used to mean a chip, microchip or die.

The instruction fetch unit 101 controls the operation of fetching architectural instructions 103 from the system memory (not shown) to the instruction cache 102 . The instruction fetch unit 101 provides the fetch address to the instruction cache 102 to specify the memory address of the cache line of the architectural instruction bytes that the processor 100 fetches into the cache 102 . The selection of the grab address is based on the current value of the instruction pointer (not shown) or the program counter of the processor 100 . Generally speaking, the program counter will increase sequentially according to the size of the instruction until a control instruction such as branch, call, or return occurs in the instruction stream, or an exceptional condition such as an interrupt, trap, exception, or error occurs. Non-sequential addresses such as branch target addresses, return addresses, or exception vectors update the program counter. In summary, the program counter is updated as the execution units 112/121 execute instructions. The program counter may also be updated when an exceptional condition is detected, such as the instruction translator 104 encountering an instruction 103 not defined in the ISA of the processor 100 .

The instruction cache 102 stores architectural instructions 103 fetched from a system memory coupled to the processor 100 . These architectural instructions 103 include move to neural network (MTNN) instructions and move from neural network (MFNN) instructions, which will be described in detail later. In one embodiment, the architectural instruction 103 is an instruction of the x86 instruction set architecture, and an MTNN instruction and an MFNN instruction are added. In this disclosure, an x86 instruction set architecture processor is understood as executing the same machine language instructions as Processor A processor that produces the same results at the instruction set architecture level. However, other instruction set architectures, such as Advanced Reduced Instruction Set Machine Architecture (ARM), Sun's Scalable Processor Architecture (SPARC), or Enhanced Reduced Instruction Set Performance Computing Performance Optimization Architecture (PowerPC), Other embodiments of the invention may also be used. The instruction cache 102 provides the architectural instructions 103 to the instruction translator 104 to translate the architectural instructions 103 into microinstructions 105 .

The microinstructions 105 are provided to the renaming unit 106 and finally executed by the execution units 112/121. These microinstructions 105 implement architectural instructions. According to a preferred embodiment, the instruction translator 104 includes a first part for translating frequently executed and/or relatively uncomplex architectural instructions 103 into microinstructions 105 . The instruction translator 104 also includes a second part, which has a microcode unit (not shown). The microcode unit has a microcode memory loaded with microcode instructions to execute complex and/or infrequently used instructions in the architectural instruction set. The microcode unit also includes a microsequencer (microsequencer) to provide a non-architectural micro-program counter (micro-PC) to the microcode memory. For a preferred embodiment, these microinstructions are translated into microinstructions 105 by a microtranslator (not shown). The selector selects the microinstructions 105 from the first part or the second part to provide to the renaming unit 106 according to whether the microcode unit currently has the control right.

The renaming unit 106 renames the architectural register specified by the architectural instruction 103 to the physical register of the processor 100 . For a preferred embodiment, the processor 100 includes a reorder buffer (not shown). The renaming unit 106 assigns the items of the reorder buffer to each microinstruction 105 in program order. In this way, the processor 100 can remove the microinstructions 105 and the corresponding architectural instructions 103 according to the program order. In one embodiment, the media register 118 has a width of 256 bits, while the general register 116 has a width of 64 bits. In one embodiment, the media buffer 118 is an x86 media buffer, such as an Advanced Vector Extensions (AVX) buffer.

In one embodiment, each entry of the shuffle buffer has storage space to store the result of the microinstruction 105 . In addition, the processor 100 includes an architectural register file, and the architectural register file has physical registers corresponding to various architectural registers, such as the media register 118 , the general purpose register 116 and other architectural registers. (For a preferred embodiment, for example, the size of the media buffer 118 and the general register 116 are different, and separate register files can be used to correspond to these two registers.) For the specified in the microinstruction 105 For each source operand of an architectural register, the renaming unit will fill the source operand field of the microinstruction 105 with the rearrangement buffer directory of the latest microinstruction in the old microinstruction 105 written into the architectural register. When the execution unit 112 / 121 completes the execution of the microinstruction 105 , the execution unit 112 / 121 writes the result into the reorder buffer entry of the microinstruction 105 . When a microinstruction 105 is retired, a eviction unit (not shown) writes the results from the microinstruction's reorder buffer field into the registers of the physical register file associated with the microinstruction thus removed The architectural destination register specified by 105 .

In another embodiment, the processor 100 includes a physical register file with more physical registers than architectural registers, however, the processor 100 does not include an architectural register file and the reorder buffer The result storage space is not included in the project. (For a preferred embodiment, because the size of the media buffer 118 and the general buffer 116 are different, separate register files can be used to correspond to these two registers.) The processor 100 also includes a pointer table, It has corresponding pointers to each architectural register. For each operand in the microinstruction 105 that specifies an architectural register, the renaming unit uses a pointer to a free register in the physical register file to fill in the destination operand field in the microinstruction 105 . If there is no free register in the physical register file, the renaming unit 106 temporarily suspends the pipeline. For each source operand within microinstruction 105 that specifies an architectural register, the renaming unit utilizes a pointer to a register in the physical register file assigned to the newest microinstruction in the old microinstruction 105 that was written to the architectural register Pointer, fill in the source operand field in the microinstruction 105. When the execution unit 112/121 finishes executing the microinstruction 105, the execution unit 112/121 will write the result into the register pointed to by the destination operand field of the microinstruction 105 in the physical register file. When the microinstruction 105 is retired, the eviction unit copies the destination operand field value of the microinstruction 105 to the pointer table associated with the architectural destination register specified by the microinstruction 105 to be removed.

Reservation station 108 is loaded with microinstructions 105 until they are ready to be issued to execution units 112/121 for execution. A microinstruction 105 is ready to issue when all source operands of a microinstruction 105 are available and execution units 112/121 are also available for execution. The execution unit 112 / 121 receives the register source operand from the rearrangement buffer or the architectural register file described in the aforementioned first embodiment, or from the physical register file described in the aforementioned second embodiment. In addition, the execution unit 112/121 can directly receive the register source operand through the result transfer bus (not shown). In addition, the execution unit 112 / 121 may receive the immediate operand specified by the microinstruction 105 from the reservation station 108 . The MTNN and MFNN architecture instructions 103 include immediate operands to specify the functions to be performed by the neural network unit 121 , and the functions are provided by one or more microinstructions 105 translated from the MTNN and MFNN architecture instructions 103 , as described later.

The execution unit 112 includes one or more load/store units (not shown), which load data from the memory subsystem 114 and store data to the memory subsystem 114 . For a preferred embodiment, the memory subsystem 114 includes a memory management unit (not shown), and the memory management unit may include, for example, a plurality of translation lookaside buffers, a table movement (tablewalk) unit, A level-1 data cache (and instruction cache 102), a level-2 unified cache, and a bus interface unit interface between processor 100 and system memory. In one embodiment, the processor 100 of FIG. 1 is represented as one of a plurality of processing cores of a multi-core processor, and the multi-core processor shares a last-level cache. The execution unit 112 may further include a plurality of integer units, a plurality of media units, a plurality of floating point units and a branch unit.

The neural network unit 121 includes a weight random access memory (RAM) 124, a data random access memory 122, N neural processing units (NPU) 126, a program memory 129, a sequencer 128 and a plurality of control and state registers 127. These neural processing units 126 conceptually function like neurons in a neural network. The weight RAM 124 , the data RAM 122 and the program memory 129 can be written and read through the MTNN and MFNN architecture instructions 103 respectively. The weight random access memory 124 is arranged in W columns with N weight words in each column, and the data random access memory 122 is arranged in D columns with N data words in each column. Each data word and each weight word are a plurality of bits, for a preferred embodiment, may be 8 bits, 9 bits, 12 bits or 16 bits. Each data literal is used as the output value (sometimes expressed as activation value) of the neuron in the previous layer in the network, and each weight literal is used as the weight of the connection in the network associated with the neuron entering the current layer of the network. Although in many applications of the NNU 121, the text or operands loaded into the WRAM 124 are actually the weights associated with the connections entering the neuron, it should be noted that in some NNU 121 In some applications, the words loaded in the weight random access memory 124 are not weights, but because these words are stored in the weight random access memory 124, they are still represented by the term "weight words". For example, in some applications of the neural network unit 121, such as the example of the convolution operation of FIGS. 24-26A or the example of the common-source operation of FIGS. Objects other than , such as elements of a data matrix (such as image pixel data). Similarly, although in many applications of the neural network unit 121, the text or operand loaded in the data random access memory 122 is essentially the output value or activation value of the neuron, it should be noted that in the neural network unit 121 In some applications, the text loaded in the DRAM 122 is not like this, but because these texts are stored in the DRAM 122, they are still represented by the term "data text". For example, in some applications of the neural network unit 121 , such as the example of the convolution operation in FIGS. 24 to 26A , the DRAM 122 will be loaded with non-neuron outputs, such as elements of the convolution kernel.

In one embodiment, the neural processing unit 126 and the sequencer 128 include combinational logic, sequential logic, state machine, or a combination thereof. Architectural instructions (such as MFNN instruction 1500) will load the contents of the state register 127 into one of the general registers 116 to confirm the state of the neural network unit 121, such as the neural network unit 121 has completed a command or a program from the program memory 129 operation, or the NNU 121 is free to receive a new command or start a new NNU program.

The number of neural processing units 126 can be increased according to requirements, and the width and depth of the weight random access memory 124 and the data random access memory 122 can also be adjusted and expanded accordingly. For a preferred embodiment, the weight RAM 124 is larger than the data RAM 122 because there are many connections in a typical neural network layer, thus requiring a larger storage space to store the connections associated with each neuron. the weight of. Many embodiments are disclosed herein regarding the size of data and weight literals, the size of weight RAM 124 and data RAM 122 , and the number of different NPUs 126 . In one embodiment, the neural network unit 121 has a data RAM 122 with a size of 64KB (8192 bits x 64 columns), a weight RAM 124 with a size of 2MB (8192 bits x 2048 columns), and 512 neural processing unit 126 . The neural network unit 121 is manufactured by Taiwan Integrated Circuits (TSMC) with a 16nm process, and occupies an area of about 3.3 square millimeters.

The sequencer 128 fetches and executes instructions from the program memory 129 , and its execution includes generating address and control signals to provide to the data random access memory 122 , the weight random access memory 124 and the neural processing unit 126 . The sequencer 128 generates a memory address 123 and a read command to provide to the DRAM 122 , so as to select one of the N data words in the D column to provide to the N NPUs 126 . The sequencer 128 also generates a memory address 125 and a read command to provide to the weight random access memory 124 , so as to select one of the N weight words in the W column to provide to the N neural processing units 126 . The sequencer 128 generates the sequence of addresses 123, 125 which are also provided to the NPU 126 or determine the "connections" between the neurons. The sequencer 128 also generates a memory address 123 and a write command to provide to the DRAM 122 , so as to select one of the N data words in the D column to be written by the N NPUs 126 . The sequencer 128 also generates a memory address 125 and a write command to provide to the weight random access memory 124 , so as to select one of the N weight words in the W column to be written by the N neural processing units 126 . The sequencer 128 also generates the memory address 131 to the program memory 129 to select the NNU instruction provided to the sequencer 128 , which will be described in subsequent chapters. The memory address 131 corresponds to a program counter (not shown), and the sequencer 128 usually increments the program counter according to the position sequence of the program memory 129, unless the sequencer 128 encounters a control instruction, such as a loop instruction (please refer to FIG. 26A ), in which case sequencer 128 will update the program counter with the target address of this control instruction. The sequencer 128 also generates control signals to the neural processing unit 126, instructing the neural processing unit 126 to perform various operations or functions, such as initialization, arithmetic/logic operations, rotation/shift operations, activation functions, and write The relevant examples will be described in more detail in subsequent chapters (please refer to micro-operation 3418 shown in FIG. 34 ).

The N NPUs 126 will generate N result words 133 which can be written back to a row of the weight RAM 124 or the data RAM 122 . For a preferred embodiment, the weight RAM 124 and the data RAM 122 are directly coupled to the N neural processing units 126 . Furthermore, the weight random access memory 124 and the data random access memory 122 belong to these neural processing units 126, and are not shared with other execution units 112 in the processor 100. These neural processing units 126 can be continuously used in each In a time-frequency cycle, a row is obtained from one or both of the weight RAM 124 and the data RAM 122 and completed. For a preferred embodiment, pipeline processing can be adopted. In one embodiment, each of the data RAM 122 and the weight RAM 124 can provide 8192 bits to the NPU 126 in each clock cycle. The 8192 bits can be treated as 512 16-bytes or 1024 8-bytes, which will be described in detail later.

The size of the data set processed by the NNU 121 is not limited by the size of the weight RAM 124 and the data RAM 122, but only by the size of the system memory, since the data and weights can be stored in System memory is moved between the weight RAM 124 and the data RAM 122 through the use of MTNN and MFNN instructions (eg, through the media register 118 ). In one embodiment, the DRAM 122 is provided with dual ports, enabling data words to be written while reading or writing data words from the DRAM 122 to the DRAM 122 to the data random access memory 122. In addition, the large memory hierarchy of the memory subsystem 114 including caches can provide very large data bandwidth for data transfer between the system memory and the NNU 121 . Additionally, for a preferred embodiment, the memory subsystem 114 includes a hardware data pre-fetcher that tracks memory access patterns, such as neural data and weights loaded from system memory, and performs data pre-fetching on the cache hierarchy. Grabbing facilitates high-bandwidth and low-latency transmission during transmission to the weight RAM 124 and the data RAM 122 .

Although in the embodiment herein, one of the operands provided by the weight memory to each neural processing unit 126 is marked as a weight, this term is often used in neural networks, but it should be understood that these operands can also be other calculation-related type of data, and its calculation speed can be improved by these devices.

FIG. 2 is a block diagram illustrating the neural processing unit 126 of FIG. 1 . As shown in the figure, the NPU 126 operates to perform many functions or operations. In particular, the neural processing unit 126 can operate as neurons or nodes in an artificial neural network to perform typical multiply-accumulate functions or operations. That is, in general, a neural network unit 126 (neuron) is used to: (1) receive an input from each neuron it has a connection to, which typically, but not necessarily, is from a previous neuron in the artificial neural network; layer; (2) multiply each output value by the corresponding weight value associated with its connection to produce a product; (3) sum all the products to produce a sum; (4) execute an activation function on this sum to generate a neuron Output. However, unlike traditional methods that need to perform all multiplication operations associated with all connected inputs and sum their products, each neuron of the present invention can perform a weight multiplication operation associated with one of the connected inputs within a given time-frequency cycle And the product thereof is added (accumulated) to the accumulated value of the product of the link input executed in the time-frequency period before the time point. Assuming that there are M connections connected to this neuron, after M products are summed up (approximately M time-frequency cycles are required), this neuron will execute an activation function on this accumulated number to generate an output or result. The advantage of this approach is that it reduces the number of multipliers required and requires only a smaller, simpler, and faster adder circuit within the neuron (for example, using two-input adders) instead of Use the adders needed to be able to sum the products of all connected inputs, or even a subset of them. This approach is also conducive to the use of a very large number (N) of neurons (neural processing unit 126) in the neural network unit 121, so that after about M time-frequency cycles, the neural network unit 121 can generate this large number (N) N) The output of the neuron. Finally, for a large number of different connection inputs, the neural network unit 121 composed of these neurons can effectively perform as an artificial neural network layer. That is to say, if the number of M in different layers increases or decreases, the number of time-frequency cycles required to generate the output of the memory cell will increase or decrease accordingly, and resources (such as multipliers and accumulators) will be fully utilized. In contrast, for small values of M in the conventional design, some parts of the multipliers and adders are not utilized. Therefore, according to the number of connection outputs of the neural network unit, the embodiments described herein have both the advantages of flexibility and efficiency, and can provide extremely high performance.

The NPU 126 includes a register 205 , a dual-input multitasking register 208 , an arithmetic logic unit (ALU) 204 , an accumulator 202 , and an activation function unit (AFU) 212 . The register 205 receives the weight word 206 from the weight RAM 124 and provides its output 203 in subsequent clock cycles. The multiplexing register 208 selects one of the two inputs 207, 211 to store in its register and provide it to its output 209 in a subsequent clock cycle. Input 207 receives a data word from data RAM 122 . Another input 211 receives the output 209 of the adjacent neural processing unit 126 . The neural processing unit 126 shown in FIG. 2 is marked as a neural processing unit J among the N neural processing units shown in FIG. 1 . That is to say, the neural processing unit J is a representative example of the N neural processing units 126 . In terms of a preferred embodiment, the input 211 of the J-instance multitasking register 208 of the neural processing unit 126 receives the output 209 of the J-1 instance of the multitasking register 208 of the neural processing unit 126, and the neural processing unit J The output 209 of the multitasking register 208 of the NPU 126 is provided to the input 211 of the multitasking register 208 of the J+1 instance of the NPU 126 . In this way, the multitasking registers 208 of the N NPUs 126 can work together, like N word rotators or cyclic shifters. This part will be described in more detail in FIG. 3 . The multiplexing register 208 uses the control input 213 to control which of the two inputs will be selected by the multiplexing register 208 to store in its register and subsequently provided on the output 209 .

The arithmetic logic unit 204 has three inputs. One of the inputs receives weight text 203 from buffer 205 . Another input receives the output 209 of the multitasking buffer 208 . Yet another input receives the output 217 of the accumulator 202 . The ALU 204 performs arithmetic and/or logic operations on its input to generate a result at its output. For a preferred embodiment, the arithmetic and/or logical operations performed by ALU 204 are specified by instructions stored in program memory 129 . For example, the multiply-accumulate instruction in FIG. 4 specifies a multiply-accumulate operation, ie, the result 215 will be the sum of the product of the accumulator 202 value 217 and the product of the weight literal 203 and the output 209 of the multitasking register 208 . However, other operations can also be specified, these operations include but are not limited to: result 215 is the value passed by the multitasking register output 209; result 215 is the value passed by the weight literal 203; result 215 is a zero value; The sum of 217 and weight 203; Result 215 is the sum of accumulator 202 value 217 and multitasking register output 209; Result 215 is the maximum value in accumulator 202 value 217 and weight 203; Result 215 is accumulator 202 value 217 and the maximum value of the multitasking buffer output 209.

The ALU 204 provides its output 215 to the accumulator 202 for storage. The ALU 204 includes a multiplier 242 that multiplies the weight word 203 and the data word output 209 of the multiplexing register 208 to generate a product 246 . In one embodiment, multiplier 242 multiplies two 16-bit operands to produce a 32-bit result. The ALU 204 also includes an adder 244 that adds a product 246 to the output 217 of the accumulator 202 to generate a sum, which is the result 215 of the accumulation operation stored in the accumulator 202 . In one embodiment, the adder 244 adds a 32-bit result of the multiplier 242 to a 41-bit value 217 of the accumulator 202 to generate a 41-bit result. In this way, by using the rotator characteristic of the multitasking register 208 during multiple time-frequency periods, the neural processing unit 126 can achieve the sum of products of neurons required by the neural network. The ALU 204 may also include other circuit components to perform other arithmetic/logic operations as described above. In one embodiment, the second adder subtracts the weight word 203 from the data word 209 of the multiplex register 208 output 209 to generate a difference, and then the adder 244 adds the difference to the output 217 of the accumulator 202 to obtain A result 215 is generated, which is the accumulated result in the accumulator 202 . In this way, the neural processing unit 126 can achieve the operation of summing up the differences during multiple time-frequency periods. For a preferred embodiment, although the weight literal 203 and the data literal 209 have the same size (in bits), they may also have different binary point positions, as described later. For a preferred embodiment, the multiplier 242 and the adder 244 are integer multipliers and adders, and the ALU 204 is low-complexity, small, fast and Advantages of low energy consumption. However, in other embodiments of the present invention, the ALU 204 can also perform floating-point operations.

Although only a multiplier 242 and an adder 244 are shown in the ALU 204 in FIG. 2 , as far as a preferred embodiment is concerned, the ALU 204 also includes other components to perform the aforementioned other different operations. For example, the ALU 204 may include a comparator (not shown) to compare the accumulator 202 with the data/weight literal, and a multiplexer (not shown) to select the larger of the two values specified by the comparator (maximum value) is stored in the accumulator 202 . In another example, the ALU 204 includes selection logic (not shown) to skip the multiplier 242 by using the data/weight word so that the adder 224 adds the data/weight word to the value 217 of the accumulator 202 to A total is generated and stored in the accumulator 202 . These additional operations will be described in more detail in the subsequent chapters as shown in FIG. 18 to FIG. 29A , and these operations also facilitate the execution of convolution operations and common-source operations.

The activation function unit 212 receives the output 217 of the accumulator 202 . The activation function unit 212 executes the activation function on the output of the accumulator 202 to generate the result 133 of FIG. 1 . In general, the activation function in the neurons of the intermediary layer of the artificial neural network can be used to normalize the accumulated sum of the multiplication, especially in a non-linear manner. To "normalize" the accumulated total, the activation function of the current neuron produces a result value within the range of values that other neurons connected to the current neuron are expected to receive as input. (The normalized result is sometimes referred to as a "startup". In this paper, a start is the output of the current node, and the receiving node will multiply this output by the weight associated with the link between the output node and the receiving node to produce a product, and This product will be accumulated with the multiplication of other input connections associated with this receiving node.) For example, in the case where the receiving/connected neuron is expected to receive as input a value between 0 and 1, the output neuron will need Accumulated totals outside the range of 0 and 1 are non-linearly squeezed and/or adjusted (eg, shifted up to convert negative values to positive values) so that they fall within this expected range. Thus, enabling the function unit 212 to perform operations on the value 217 of the accumulator 202 will bring the result 133 into a known range. The results 133 of the N NEPs 126 can all be written back to the data RAM 122 or the weight RAM 124 at the same time. As far as a preferred embodiment is concerned, the activation function unit 212 is used to execute multiple activation functions, and for example, the input from the control register 127 selects one of these activation functions to be executed on the output 217 of the accumulator 202 . These activation functions may include, but are not limited to, step functions, correction functions, sigmoid functions, hyperbolic tangent functions, and soft-add functions (also known as smooth correction functions). The analytical formula of the soft addition function is f(x)=ln(1+e ^x ), that is, the natural logarithm of the sum of 1 and e ^x , where "e" is Euler's number and x is The input to this function is 217. As far as a preferred embodiment is concerned, the startup function may also include a pass-through function, directly passing the value 217 of the accumulator 202 or a part thereof, as described in detail later. In one embodiment, the circuit of the activation function unit 212 executes the activation function in a single clock cycle. In one embodiment, the activation function unit 212 includes a plurality of tables, which receive the accumulated value and output a value. For some activation functions, such as Sigmoid function, hyperbolic tangent function, soft addition function, etc., the value will be approximately The value provided by the actual startup function.

For a preferred embodiment, the width (in bits) of the accumulator 202 is greater than the width of the output 133 of the enable function function 212 . For example, in one embodiment, the width of this accumulator is 41 bits, so as to avoid accumulating to a maximum of 512 32-bit products (this part will be more detailed in subsequent chapters such as corresponding to FIG. 30 ) loses precision, and the result 133 is 16 bits wide. In one embodiment, in subsequent time-frequency cycles, the enable function unit 212 will pass other unprocessed portions of the accumulator 202 output 217 and will write these portions back to the data random access memory 122 or weight random access memory 122 The memory 124, this part will be described in more detail in the subsequent chapters corresponding to FIG. 8 . In this way, the unprocessed value of the accumulator 202 can be loaded back to the media register 118 through the MFNN instruction, whereby the instructions executed by the other execution units 112 of the processor 100 can perform complex activations that cannot be performed by the activation function unit 212 Functions, such as the common softmax (softmax) function, this function is also known as the standardized exponential function. In one embodiment, the instruction set architecture of the processor 100 includes an instruction to perform the exponential function, usually denoted as ex or exp( ^x ), which can be used by other execution units 112 of the processor 100 to enhance soft-maximum activation The execution speed of the function.

In one embodiment, the neural processing unit 126 adopts a pipeline design. For example, the NPU 126 may include a register of the ALU 204, such as a register between a multiplier and an adder and/or other circuits of the ALU 204. The NPU 126 may also include a load A buffer for the output of function 212 is enabled. Other embodiments of the neural processing unit 126 will be described in subsequent chapters.

FIG. 3 is a block diagram showing N multitasking registers 208 utilizing N NPUs 126 of the neural network unit 121 of FIG. The operation of the rotator (rotator) or circular shifter (circular shifter) of N characters. In the embodiment of FIG. 3 , N is 512, therefore, the NNU 121 has 512 multi-tasking registers 208 , marked as 0 to 511 , corresponding to 512 NPUs 126 respectively. Each multiplexing register 208 receives the corresponding data word 207 on one of the D columns of the DRAM 122 . That is, MMR 0 will receive data word 0 from DRAM column 122, MMR 1 will receive data word 1 from DRAM column 122, MMR 2 will receive data word 1 from DRAM column 122, and MMR 2 will receive data word 1 from DRAM column 122. The RAM row 122 receives the data word 2, and so on, the multitasking register 511 receives the data word 511 from the data RAM row 122 . In addition, multitasking register 1 will receive the output 209 of multitasking register 0 as another input 211, multitasking register 2 will receive the output 209 of multitasking register 1 as another input 211, and multitasking register 3 will receive Receives output 209 of multitasking buffer 2 as another input 211, and so on, multitasking buffer 511 receives output 209 of multitasking buffer 510 as another input 211, and multitasking buffer 0 receives multitasking The output 209 of the buffer 511 serves as the other input 211 . Each multiplexing register 208 receives a control input 213 to control it to select a data word 207 or a loop input 211 . In this mode of operation, the control input 213 will control each multiplexing register 208 to select the data word 207 to store in the register during the first clock cycle and provide it to the ALU 204 in a subsequent step, and in a subsequent step In the time-frequency period of (eg, M-1 time-frequency periods mentioned above), the control input 213 controls each multiplex register 208 to select the loop input 211 to store in the register and provide it to the arithmetic logic unit 204 in a subsequent step.

Although in the embodiment described in FIG. 3 (and subsequent FIG. 7 and FIG. 19 ), a plurality of neural processing units 126 can be used to rotate the values of these multitasking registers 208/705 to the right, that is, the neural processing unit J Move toward the neural processing unit J+1, but the present invention is not limited thereto. In other embodiments (for example, corresponding to the embodiment of FIG. 24 to FIG. 26 ), multiple neural processing units 126 can be used to The value of 208/705 rotates to the left, that is, moves from the neural processing unit J to the neural processing unit J-1. In addition, in other embodiments of the present invention, these NPUs 126 can selectively rotate the value of the multitasking register 208/705 to the left or right, for example, this selection can be specified by a NNU instruction .

FIG. 4 is a table showing a program stored in the program memory 129 of the neural network unit 121 of FIG. 1 and executed by the neural network unit 121 . As mentioned earlier, this sample program performs calculations associated with one layer of the artificial neural network. The table shown in Figure 4 has four columns and three rows. Each column corresponds to the address in the program memory 129 indicated in the first row. The second line specifies the corresponding instruction, and the third line indicates the number of clock cycles associated with this instruction. For a preferred embodiment, the aforementioned number of clock cycles represents the effective number of clock cycles per instruction clock cycle value in the embodiment of pipeline execution, rather than instruction latency. As shown in the figure, because the neural network unit 121 has the nature of pipeline execution, each instruction has an associated time-frequency cycle, the instruction at address 2 is an exception, and this instruction will actually repeat itself 511 times, so 511 time-frequency cycles are required, which will be described in detail later.

All NPUs 126 process each instruction in the program in parallel. That is to say, all the N neural processing units 126 will execute the instructions of the first row in the same time-frequency cycle, and all the N neural processing units 126 will execute the instructions of the second row in the same time-frequency cycle, and so on analogy. However, the present invention is not limited thereto. In other embodiments in subsequent chapters, some instructions are executed in a partially parallel partial sequence. For example, as described in the embodiment of FIG. 11 , multiple neural processing units 126 In embodiments that share a boot function unit, the boot function and output instructions at addresses 3 and 4 are executed in this manner. The example in FIG. 4 assumes that one layer has 512 neurons (NPU 126 ), and each neuron has 512 connection inputs from 512 neurons in the previous layer, for a total of 256K connections. Each neuron receives a 16-bit data value from each connection input and multiplies this 16-bit data value by an appropriate 16-bit weight value.

The first column at address 0 (which can also be assigned to other addresses) specifies the instruction to initialize the NPU. This initialization command clears the value of accumulator 202 to zero. In one embodiment, the initialization instruction may also load a row in the data RAM 122 or the weight RAM 124 in the accumulator 202 with the corresponding text specified by the instruction. The initialization command will also load the configuration value into the control register 127, and this part will be described in more detail in FIG. 29A and FIG. 29B. For example, the width of the data word 207 and the weight word 209 can be loaded for use by the ALU 204 to determine the size of the operation performed by the circuit. This width also affects the result 215 stored in the accumulator 202 . In one embodiment, the NPU 126 includes a circuit that fills the output 215 of the ALU 204 before storing it in the accumulator 202, and the initialization command loads the circuit with configuration values that affect the aforementioned filling operation. In one embodiment, it can also be specified in the arithmetic logic unit function instruction (such as the multiplication and accumulation instruction of address 1) or output instruction (such as the write start function unit output instruction of address 4), so that the accumulator 202 is cleared to zero value.

The second column at address 1 specifies a multiply-accumulate instruction that instructs the 512 NPUs 126 to load the corresponding data word from a column of data RAM 122 and the corresponding weight word from a column of weight RAM 124 , and perform the first multiply-accumulate operation on the data text input 207 and the weight text input 206 , that is, add the zero value of the initializing accumulator 202 . Further, the command instructs the sequencer 128 to generate a value at the control input 213 to select the data text input 207 . In the example of FIG. 4, the designated column of the data RAM 122 is column 17, and the designated column of the weight RAM 124 is column 0, so the sequencer will be instructed to output the value 17 as the data RAM address 123 , output the value 0 as the weight random access memory address 125 . Thus, 512 data words from column 17 of data RAM 122 are provided as corresponding data inputs 207 to 512 neural processing units 126, while 512 weight words from column 0 of weight RAM 124 are provided as The corresponding weight inputs 206 of the 512 neural processing units 126 .

The third column at address 2 specifies the multiply-accumulate rotate instruction, which has a count value of 511 to instruct the 512 NPUs 126 to perform 511 multiply-accumulate operations. This instruction instructs the 512 NPUs 126 to input each of the 511 multiply-accumulate operations into the data word 209 of the ALU 204 as the rotation value 211 from the adjacent NPU 126 . That is, the command instructs the sequencer 128 to generate a value at the control input 213 to select the rotation value 211 . In addition, the instruction instructs the 512 NPUs 126 to load the corresponding weight values in each of the 511 multiply-accumulate operations into the “next” column of the weight RAM 124 . That is, the instruction instructs the sequencer 128 to increment the weight RAM address 125 by one from the value of the previous clock cycle. In this example, the first clock cycle of the instruction is column 1, and the next clock cycle The period is column 2, the next time-frequency period is column 3, and so on, the 511th time-frequency period is column 511. In each of the 511 multiply-accumulate operations, the product of the rotation input 211 and the weight text input 206 is added to the previous value of the accumulator 202 . The 512 NPUs 126 will perform the 511 multiply-accumulate operations in 511 time-frequency cycles, and each NPU 126 will respond to a different data word from column 17 of the DRAM 122—that is, the same The data words that the adjacent neural processing unit 126 performs operations in the previous time-frequency period, and the different weight words associated with the data words perform a multiply-accumulate operation are conceptually different connection inputs of neurons. This example assumes that each neural processing unit 126 (neuron) has 512 connection inputs, thus involving the processing of 512 data words and 512 weight words. After the last iteration of the multiply accumulate rotate instruction in column 2, the accumulator 202 will store the sum of the products of the 512 connected inputs. In one embodiment, the instruction set of the NPU 126 includes an "execute" instruction to instruct the ALU 204 to perform the ALU operation specified by the initialize NPU instruction, such as specified by the ALU function 2926 of FIG. 29A , instead of having separate instructions for different types of arithmetic and logic operations (such as the aforementioned multiplication and accumulation, the maximum value of the accumulator and the weight, etc.).

The fourth column at address 3 specifies the start function instruction. The enable function instruction instructs the enable function unit 212 to perform the specified enable function on the accumulator 202 value to generate the result 133 . The implementation of the startup function will be described in more detail in the subsequent chapters.

The fifth column at address 4 designates a write enable function unit output instruction to instruct the 512 NPUs 216 to write their enable function unit 212 output back as result 133 to a column of DRAM 122, in this example Middle is column 16. That is, the command instructs the sequencer 128 to output the value 16 as the data RAM address 123 and the write command (corresponding to the read command specified by the multiply-accumulate command for address 1). In a preferred embodiment, due to the nature of pipeline execution, the write-enable FU output instruction can be executed concurrently with other instructions, so the write-enable FU output instruction can actually be executed in a single clock cycle.

As far as a preferred embodiment is concerned, each neural processing unit 126 is used as a pipeline, and this pipeline has various functional components, such as the multitasking register 208 (and the multitasking register 705 in FIG. 7 ), the arithmetic logic unit 204 , the accumulator 202, the activation function unit 212, the multiplexer 802 (please refer to FIG. 8), the column buffer 1104 and the activation function unit 1112 (please refer to FIG. 11), etc., some of which can be pipelined by themselves. In addition to data words 207 and weight words 206 , the pipeline also receives instructions from program memory 129 . These instructions flow down the pipeline and control various functional units. In another embodiment, the program does not include the startup function instruction, but the startup function specified by the initialization neural processing unit instruction to be executed on the value 217 of the accumulator 202, indicating that the value of the designated startup function is stored in the configuration register, The startup function unit 212 portion of the pipeline is used after the last accumulator 202 value 217 was generated, ie after the last iteration of the multiply accumulate rotate instruction at address 2 was executed. As far as a preferred embodiment is concerned, in order to save energy consumption, the start function unit 212 part of the pipeline will be in an inactive state before the write start function unit output command arrives, and when the command arrives, the start function unit 212 will start and The accumulator 202 output 217 specified by the initialize instruction executes the start function.

FIG. 5 is a sequence diagram showing that the neural network unit 121 executes the procedure of FIG. 4 . Each column of this timing diagram corresponds to a consecutive time-frequency period indicated by the first row. Other rows correspond to different neural processing units 126 among the 512 neural processing units 126 and indicate their operations. In the figure, only the operations of NPUs 0, 1, 511 are shown to simplify the description.

In clock cycle 0, each of the 512 NPUs 126 executes the initialization command shown in FIG. 4 , which is to assign zero to the accumulator 202 in FIG. 5 .

In time-frequency cycle 1, each of the 512 neural processing units 126 executes the multiply-accumulate instruction at address 1 in FIG. 4 . As shown in the figure, the NPU0 will add the accumulator 202 value (i.e. zero) to the product of the literal 0 in column 17 of the data RAM 122 and the literal 0 in column 0 of the weight RAM 124; The NPU 1 will add the value of the accumulator 202 (i.e. zero) to the product of the literal 1 of column 17 of the data RAM 122 and the literal 1 of column 0 of the weight RAM 124; Unit 511 adds the accumulator 202 value (ie zero) to the product of the word 511 in row 17 of DRAM 122 and the word 511 in row 0 of weight RAM 124 .

In time-frequency cycle 2, each of the 512 NPUs 126 performs the first iteration of the multiply-accumulate-rotate instruction at address 2 in FIG. 4 . As shown in the figure, NPU 0 will add the accumulator 202 value to the rotated data word 211 received by the multitasking register 208 output 209 of NPU 511 (i.e., the data word received by DRAM 122 511) with literal 0 of column 1 of weight RAM 124; NPU1 will add accumulator 202 value to rotation data literal 211( That is, the product of the data word (0) received by the data random access memory 122 and the word 1 of column 1 of the weight random access memory 124; and so on, the neural processing unit 511 will add the value of the accumulator 202 to The multiplexing register 208 output 209 at 510 is the product of the received rotated data word 211 (ie, the data word 510 received from the DRAM 122 ) and the word 511 of column 1 of the weighted RAM 124 .

In time-frequency cycle 3, each of the 512 NPUs 126 performs the second iteration of the multiply-accumulate-rotate instruction at address 2 in FIG. 4 . As shown in the figure, NPU 0 will add the accumulator 202 value to the rotated data word 211 received by the multitasking register 208 output 209 of NPU 511 (i.e., the data word received by DRAM 122 510) with literal 0 of column 2 of weight RAM 124; NPU1 will add accumulator 202 value to rotation data literal 211( That is, the product of the data word 511) received by the data random access memory 122 and the word 1 of column 2 of the weight random access memory 124; and so on, the neural processing unit 511 will add the accumulator 202 value to the The multiplexed register 208 output 209 at 510 is the product of the received rotated data word 211 (ie, the data word 509 received from the DRAM 122 ) and the word 511 of column 2 of the weighted RAM 124 . As indicated by the omitted numbers in FIG. 5 , the next 509 time-frequency periods will continue in this way until the time-frequency period 512 .

In the time-frequency cycle 512 , each of the 512 NPUs 126 executes the 511th iteration of the multiply-accumulate-rotate instruction at address 2 in FIG. 4 . As shown in the figure, NPU 0 will add the accumulator 202 value to the rotated data word 211 received by the multitasking register 208 output 209 of NPU 511 (i.e., the data word received by DRAM 122 1) Product with literal 0 of column 511 of weight RAM 124; NPU1 will add accumulator 202 value to rotation data literal 211 received by MTR 208 output 209 of NPU0 ( That is, the product of the data word 2) received by the data random access memory 122 and the word 1 of the column 511 of the weight random access memory 124; and so on, the neural processing unit 511 will add the value of the accumulator 202 to The multiplexing register 208 output 209 at 510 is the product of the received rotated data word 211 (ie, the data word 0 received from the DRAM 122 ) and the word 511 of the column 511 of the weighted RAM 124 . In one embodiment, multiple time-frequency cycles are required to read the data word and the weight word from the data random access memory 122 and the weight random access memory 124 to execute the multiply-accumulate instruction of address 1 in FIG. 4; however, the data random access The memory 122, the weight random access memory 124 and the neural processing unit 126 are configured in a pipeline, so that after the first multiplication and accumulation operation starts (as shown in time-frequency cycle 1 in FIG. 5 ), the subsequent multiplication and accumulation operation (as shown in FIG. 5 5 time-frequency cycle 2-512) will start to execute in the subsequent time-frequency cycle. As far as a preferred embodiment is concerned, in response to the use of architectural instructions, such as MTNN or MFNN instructions (described in subsequent FIGS. Fetching actions, or microinstructions translated from architectural instructions, these NPUs 126 are briefly suspended.

In the time-frequency period 513 , the startup function unit 212 of each of the 512 neural processing units 126 executes the startup function at address 3 in FIG. 4 . Finally, at clock cycle 514, each NPU 126 of the 512 NPUs 126 will perform the execution of FIG. The write at address 4 in the middle starts the function unit output instruction, that is to say, the result 133 of NPU 0 will be written into word 0 of DRAM 122, and the result 133 of NPU 1 will be written into DRAM 122. Word 1 of the memory 122 is accessed, and so on, the result 133 of the NPU 511 is written into the word 511 of the DRAM 122 . A corresponding block diagram corresponding to the aforementioned operations of FIG. 5 is shown in FIG. 6A.

FIG. 6A is a block diagram showing that the neural network unit 121 of FIG. 1 executes the program of FIG. 4 . The neural network unit 121 includes 512 neural processing units 126 , a data RAM 122 receiving an address input 123 , and a weight RAM 124 receiving an address input 125 . At time-frequency period 0, the 512 neural processing units 126 execute initialization instructions. This operation is not shown in the figure. As shown in the figure, at clock cycle 1, 512 16-bit data words of column 17 are read from the DRAM 122 and provided to the 512 NPUs 126 . During the time-frequency period 1 to 512 , 512 16-bit weight texts from row 0 to row 511 are respectively read from the weight random access memory 122 and provided to the 512 neural processing units 126 . During the time-frequency period 1, the 512 neural processing units 126 perform corresponding multiplication and accumulation operations on the loaded data words and weight words. This operation is not shown in the figure. During clock cycles 2 to 512, the 512 multitasking registers 208 of the NPU 126 operate as a rotator with 512 16-bit words, and the columns previously allocated by the DRAM 122 The loaded data words 17 are rotated to adjacent NPUs 126 , and these NPUs 126 perform multiply-accumulate operations on the rotated corresponding data words and the corresponding weight words loaded from the weight RAM 124 . During the time-frequency period 513, the 512 startup function units 212 execute the startup instruction. This operation is not shown in the figure. During the clock cycle 514 , the 512 NPUs 126 will write their corresponding 512 16-bit results 133 back to the column 16 of the DRAM 122 .

As shown in the figure, the number of time-frequency cycles required to generate the resulting text (neuron output) and write back to the data RAM 122 or the weight RAM 124 is roughly the number of data inputs received by the current layer of the neural network (link ) square root of the quantity. For example, if the current layer has 512 neurons, and each neuron has 512 connections from the previous layer, the total number of these connections is 256K, and the number of time-frequency cycles required to generate the result of the current layer will be slightly greater than 512 . Therefore, the neural network unit 121 can provide extremely high performance in neural network calculation.

6B is a flow chart showing the execution of the architecture program by the processor 100 of FIG. 1 to utilize the neural network unit 121 to perform a typical multiply-accumulate activation function operation of the neurons associated with the hidden layer of the artificial neural network, as shown by FIG. 4 The operation of program execution. The example of FIG. 6B assumes that there are four hidden layers (indicated in the variable NUM_LAYERS of the initialization step 602), each hidden layer has 512 neurons, and each neuron is connected to all 512 neurons of the previous layer (through the program of FIG. 4 ). It should be understood, however, that the number of these layers and neurons was chosen to illustrate the present invention, and that the neural network unit 121 may apply similar calculations to embodiments with different numbers of hidden layers, each with a different number of neurons. , or embodiments where the neurons are not all connected. In one embodiment, the weights for non-existent neurons or non-existent neuron connections in this layer are set to zero. For a preferred embodiment, the architecture program writes the first set of weights into the weight RAM 124 and activates the NNU 121. When the NNU 121 is performing calculations associated with the first layer, the architecture The program will write the second set of weights into the weight random access memory 124, so that once the neural network unit 121 completes the calculation of the first hidden layer, the neural network unit 121 can start the calculation of the second hidden layer. In this way, the architecture program will go back and forth between the two areas of the weight RAM 124 to ensure that the NNU 121 can be fully utilized. The process starts at step 602 .

In step 602, as described in the relevant section of FIG. 6A , the processor 100 executing the architecture program writes the input value into the current neuron hidden layer of the DRAM 122, that is, into the column of the DRAM 122 17. These values may also already be located in the column 17 of the data RAM 122 as the result 133 of the operation of the neural network unit 121 on the previous layer (eg convolution, common source or input layer). Second, the framework program initializes the variable N to a value of 1. The variable N represents the current layer of hidden layers to be processed by the neural network unit 121 . Also, the architecture program initializes the variable NUM_LAYERS to a value of 4, since there are four hidden layers in this example. Then the flow goes to step 604 .

In step 604, the processor 100 writes the weight text of layer 1 into the weight RAM 124, such as columns 0 to 511 shown in FIG. 6A. Then the flow goes to step 606 .

In step 606 , the processor 100 uses the MTNN instruction 1400 specifying the function 1432 to be written into the program memory 129 , and writes the multiply-accumulate activation function program (as shown in FIG. 4 ) into the program memory 129 of the neural network unit 121 . The processor 100 then uses the MTNN instruction 1400 to start the NNU program, which specifies the function 1432 to start executing the program. Then the flow goes to step 608 .

In decision step 608, the framework program checks to see if the value of variable N is less than NUM_LAYERS. If yes, the process will go to step 612 ; otherwise, go to step 614 .

In step 612 , the processor 100 writes the weight text of layer N+1 into the weight RAM 124 , such as columns 512 to 1023 . Therefore, the architecture program can write the weight text of the next layer into the weight random access memory 124 when the neural network unit 121 performs the calculation of the hidden layer of the current layer, thereby completing the calculation of the current layer, that is, writing the data After random accessing the memory 122, the neural network unit 121 can immediately start to execute the hidden layer calculation of the next layer. Then proceed to step 614 .

In step 614, the processor 100 confirms whether the executing NNU program (for layer 1, executed at step 606, for layers 2-4, executed at step 618) has completed execution. For a preferred embodiment, the processor 100 reads the state register 127 of the neural network unit 121 to confirm whether the execution has been completed by executing the MFNN instruction 1500 . In another embodiment, the neural network unit 121 generates an interrupt, indicating that the multiply-accumulate startup function layer program has been completed. Flow then proceeds to decision step 616 .

In decision step 616, the builder program checks to see if the value of variable N is less than NUM_LAYERS. If yes, the process will go to step 618 ; otherwise, go to step 622 .

In step 618, the processor 100 updates the multiply-accumulate activation function program to enable the execution of the hidden layer calculation of layer N+1. Further, the processor 100 will update the DRAM 122 column value of the multiply-accumulate instruction at address 1 in FIG. column 16) and update the output column (eg update to column 15). The processor 100 then starts to update the NNU program. In another embodiment, the program of FIG. 4 specifies the same column of the output instruction at address 4 as the column specified by the multiply-accumulate instruction at address 1 (ie, the column read by the DRAM 122 ). In this embodiment, the current row of input data words will be overwritten (since the row of data words has been read into the multitasking register 208 and rotated between the NPUs 126 via the N word rotator, as long as the row The data literal need not be used for other purposes, such processing is permitted). In this case, there is no need to update the NNU program in step 618, but only to restart it. Then the process proceeds to step 622 .

In step 622 , the processor 100 reads the result of the NNU program of layer N from the DRAM 122 . However, if these results are only used in the next layer, the architecture program does not need to read these results from the DRAM 122, but can keep them in the DRAM 122 for the calculation of the next hidden layer use. The process then proceeds to step 624 .

In decision step 624, the framework program checks to see if the value of variable N is less than NUM_LAYERS. If yes, the process proceeds to step 626; otherwise, the process is terminated.

In step 626, the framework program increments the value of N by one. Then the flow returns to decision step 608 .

As shown in the example of FIG. 6B, roughly every 512 time-frequency cycles, these neural processing units 126 will perform a read and a write to the data random access memory 122 (through the neural network unit program of FIG. 4 operation effect). In addition, the neural processing units 126 read the weight random access memory 124 roughly every time-frequency cycle to read a list of weight texts. Therefore, the entire bandwidth of the WRAM 124 will be consumed by the NNU 121 performing hidden layer operations in a hybrid manner. In addition, assuming that there is a write and read buffer in one embodiment, such as buffer 1704 in FIG. In this way, the buffer 1704 performs writing to the weight random access memory 124 roughly every 16 clock cycles to write the weight text. Therefore, in an embodiment in which the weight random access memory 124 is a single port (as described in the corresponding section of FIG. Reads from the memory 124 enable the buffer 1704 to write to the weight random access memory 124 . However, in the DPWRAM 124 embodiment, these NPUs 126 need not be left alone.

FIG. 7 is a block diagram illustrating another embodiment of the neural processing unit 126 of FIG. 1 . The neural processing unit 126 of FIG. 7 is similar to the neural processing unit 126 of FIG. 2 . However, the NPU 126 of FIG. 7 additionally has a dual-input multitasking register 705 . The multitasking register 705 selects one of the inputs 206 or 711 to store in its register, and provides it to its output 203 in a subsequent clock cycle. Input 206 receives weight text from weight RAM 124 . Another input 711 is to receive the output 203 of the second multitasking register 705 of the adjacent NPU 126 . As far as a preferred embodiment is concerned, the input 711 of the NPU J receives the output 203 of the multitasking register 705 of the NPU 126 arranged at J-1, and the output 203 of the NPU J is provided to The input 711 of the multitasking register 705 of the NPU 126 arranged at J+1. In this way, the multitasking registers 705 of the N NPUs 126 can work together, like a rotator for N words, which operates similarly to the method shown in FIG. 3 above, but is used for weight words instead of data words . The multiplexing register 705 uses the control input 213 to control which of the two inputs will be selected by the multiplexing register 705 to store in its register and subsequently provided on the output 203 .

Utilizing the multitasking register 208 and/or the multitasking register 705 (as well as the multitasking registers in other embodiments shown in FIG. 18 and FIG. 23 ), a large-scale rotator is actually formed to transfer data from random access By rotating the data/weights of one column of memory 122 and/or weight RAM 124, neural network unit 121 does not need to use a very large multiplier between data RAM 122 and/or weight RAM 124. worker to provide the required data/weight text to the appropriate neural network unit.

Write back the accumulator value in addition to the result of the start function

For some applications, have the processor 100 receive back (eg, to the media register 118 via the MFNN instruction of FIG. Performing calculations does have its uses. For example, in one embodiment, the activation function unit 212 is not configured for the execution of the soft-max activation function to reduce the complexity of the activation function unit 212 . Therefore, the neural network unit 121 will output the unprocessed accumulator 202 value 217 or a subset thereof to the data random access memory 122 or the weight random access memory 124, and the architecture program can be transferred from the data random access memory in subsequent steps 122 or weight random access memory 124 reads and calculates this unprocessed value. However, the use of the raw accumulator 202 value 217 is not limited to performing soft max calculations, and other uses are also encompassed by the present invention.

FIG. 8 is a block diagram illustrating another embodiment of the neural processing unit 126 of FIG. 1 . The neural processing unit 126 of FIG. 8 is similar to the neural processing unit 126 of FIG. 2 . However, the NPU 126 of FIG. 8 includes a multiplexer 802 within the enable function unit 212 having a control input 803 . The width (in bits) of the accumulator 202 is greater than the width of the data words. The multiplexer 802 has multiple inputs to receive the data word-width portion of the output 217 of the accumulator 202 . In one embodiment, the width of the accumulator 202 is 41 bits, and the neural processing unit 216 can be used to output a 16-bit result text 133; thus, for example, the multiplexer 802 (or the multiplexer of FIG. 30 3032 and/or multiplexer 3037) has three inputs to receive bits [15:0], bits [31:16] and bits [47:32] of the accumulator 202 output 217, respectively. For a preferred embodiment, output bits not provided by the accumulator 202 (eg, bits [47:41]) are forced to be zero-valued bits.

The sequencer 128 will generate a value at the control input 803, and control the multiplexer 802 to select one of the words (such as 16 bits) in the accumulator 202 to respond to the instruction for writing the accumulator, for example, it is located at address 3 in the following figure 9 to 5 of the Write Accumulator instruction. As far as a preferred embodiment is concerned, the multiplexer 802 also has one or more inputs to receive the output of the activation function circuit (such as the components 3022, 3024, 3026, 3018, 3014 and 3016 in Figure 30), and these activation The output produced by the function circuit has a width equal to one data literal. The sequencer 128 will generate a value at the control input 803 to control the multiplexer 802 to select it among these enable function circuit outputs instead of selecting it among the words of the accumulator 202, in response to the enable function at address 4 in FIG. 4 The unit outputs instructions.

FIG. 9 is a table showing a program stored in the program memory 129 of the neural network unit 121 of FIG. 1 and executed by the neural network unit 121 . The example program of FIG. 9 is similar to the program of FIG. 4 . In particular, the instructions at addresses 0 to 2 are identical. However, the instructions at addresses 3 and 4 in FIG. 4 are replaced in FIG. 9 by a write to accumulator instruction, which instructs each of the 512 NPUs 126 to write their accumulator 202 output 217 as a result 133 back to the DRAM Three columns of memory 122 are fetched, columns 16-18 in this example. That is to say, the write accumulator instruction will instruct the sequencer 128 to output the data random access memory address 123 with a value of 16 and the write command in the first clock cycle, and output the data with a value of 17 in the second clock cycle The RAM address 123 and the write command output the data RAM address 123 and the write command with a value of 18 in the third clock cycle. As far as the preferred embodiment is concerned, the execution time of the write accumulator instruction can be overlapped with other instructions, so that the write accumulator instruction can actually be executed within these three clock cycles, where each Cycles are written to the ranks of the DRAM 122 . In an embodiment, the user specifies the value of the start function 2934 and the output command 2956 column of the control register 127 (FIG. 29A) to write the desired portion of the accumulator 202 into the DRAM 122 or WRAM. Access memory 124. Additionally, the write accumulator instruction may selectively write back a subset of accumulator 202 rather than the entire contents of accumulator 202 . In an embodiment, standard type accumulator 202 may be written back. This part will be described in more detail in subsequent chapters corresponding to FIG. 29 to FIG. 31 .

FIG. 10 is a sequence diagram showing that the neural network unit 121 executes the procedure of FIG. 9 . The timing diagram of FIG. 10 is similar to the timing diagram of FIG. 5 , wherein the time-frequency periods 0 to 512 are all the same. However, in the clock cycle 513-515, the activation function unit 212 of each neural processing unit 126 in the 512 neural processing units 126 executes one of the instructions for writing to the accumulator at addresses 3 to 5 in FIG. 9 . In particular, each of the 512 NPUs 126 will write bits [15:0] of the output 217 of the accumulator 202 as its result 133 back to the column of the DRAM 122 during the clock cycle 513 Corresponding text in 16; in the time-frequency cycle 514, each neural processing unit 126 in the 512 neural processing units 126 will write the bits [31:16] of the accumulator 202 output 217 as its result 133 and write back the data random access The corresponding text in column 17 of the memory 122; and in the time-frequency cycle 515, each neural processing unit 126 in the 512 neural processing units 126 will write the bits [40:32] of the accumulator 202 output 217 as its result 133 The corresponding text in column 18 of DRAM 122 is returned. For a preferred embodiment, bits [47:41] are forced to be zero.

shared boot function unit

FIG. 11 is a block diagram showing an embodiment of the neural network unit 121 in FIG. 1 . In the embodiment of Fig. 11, a neuron is divided into two parts, i.e. a start function unit part and an ALU part (this part also includes a shift register part), and each start function unit part is composed of a plurality of ALU parts shared. In FIG. 11 , the ALU portion refers to the NPU 126 , and the shared AFU portion refers to the AFU 1112 . Compared with the embodiment shown in FIG. 2 , each neuron includes its own activation function unit 212 . Accordingly, in an example of the embodiment in FIG. 11 , the neural processing unit 126 (ALU part) may include the accumulator 202 , the ALU 204 , the multitasking register 208 and the register 205 in FIG. 2 , but does not include The functional unit 212 is started. In the embodiment of FIG. 11 , the neural network unit 121 includes 512 neural processing units 126 , however, the present invention is not limited thereto. In the example of FIG. 11 , the 512 NPUs 126 are divided into 64 groups, labeled as Groups 0 to 63 in FIG. 11 , and each group has eight NPUs 126 .

The neural network unit 121 further includes a column buffer 1104 and a plurality of shared activation function units 1112 , and these activation function units 1112 are coupled between the neural processing unit 126 and the column buffer 1104 . The width (in bits) of the column buffer 1104 is the same as a column of the data RAM 122 or the weight RAM 124 , for example, 512 words. Each neural processing unit 126 group has an activation function unit 1112, that is, each activation function unit 1112 corresponds to the neural processing unit 126 group; like this, there are 64 activation function units 1112 in the embodiment of FIG. 11 Corresponding to 64 NPU 126 groups. The eight NPUs 126 of the same group share the activation function unit 1112 corresponding to this group. The invention is also applicable to embodiments having different numbers of activation function units and different numbers of NPUs in each group. For example, the present invention can also be applied to embodiments in which two, four or sixteen neural processing units 126 share the same activation function unit 1112 in each group.

Sharing the AFU 1112 helps to reduce the size of the NNU 121 . Size reduction comes at the expense of performance. That is to say, depending on the sharing ratio, additional time-frequency cycles may be required to generate the result 133 of the entire neural processing unit 126 array. For example, as shown in FIG. 12 below, in the case of a sharing ratio of 8:1 The next seven additional time-frequency cycles are required. In general, however, the aforementioned additional additional The number of time-frequency cycles (eg, 7) is relatively small. Therefore, sharing AFUs has very little impact on performance (eg, about 1 percent increase in computation time), and may be a cost-effective reduction in the size of the NNU 121 .

In one embodiment, each neural processing unit 126 includes an activation function unit 212 for performing relatively simple activation functions, and these simple activation function units 212 have a small size to be included in each neural processing unit 126 On the contrary, the shared complex activation function unit 1112 executes a relatively complex activation function, and its size will be significantly larger than the simple activation function unit 212 . In this embodiment, only when the specified complex activation function needs to be executed by the shared complex activation function unit 1112, additional time-frequency cycles are required. In the case that the specified activation function can be executed by the simple activation function unit 212, This additional time-frequency cycle is not needed.

12 and 13 are timing diagrams showing the neural network unit 121 of FIG. 11 executing the program of FIG. 4 . The timing diagram of FIG. 12 is similar to the timing diagram of FIG. 5 , and the time-frequency periods 0 to 512 of the two are the same. However, the operation in the time-frequency cycle 513 is not the same, because the neural processing unit 126 in FIG. 11 will share the activation function unit 1112; that is, the neural processing units 126 in the same group will share the activation function associated with this group Unit 1112, and Figure 11 shows this shared architecture.

Each column of the timing diagram of FIG. 13 corresponds to a consecutive time-frequency period indicated in the first row. The other lines are respectively corresponding to different activation function units 1112 among the 64 activation function units 1112 and indicate their operations. In the figure, only the operations of NPUs 0, 1, and 63 are shown to simplify the description. The time-frequency cycle in FIG. 13 corresponds to the time-frequency cycle in FIG. 12 , but shows that the neural processing unit 126 shares the operation of the activation function unit 1112 in a different manner. As shown in FIG. 13 , during the time-frequency period 0 to 512, the 64 activation function units 1112 are in the inactive state, and the neural processing unit 126 executes the initialization neural processing unit instruction, the multiply-accumulate instruction and the multiply-accumulate rotation instruction.

As shown in FIG. 12 and FIG. 13 , in the time-frequency period 513, the activation function unit 0 (associated with the activation function unit 1112 of group 0) starts to execute the specified activation function on the value 217 of the accumulator 202 of the neural processing unit 0, NPU 0 is the first NPU 216 in group 0, and the output of the activation function unit 1112 will be stored in word 0 of the row register 1104 . Also in the time-frequency period 513 , each activation function unit 1112 starts to execute the specified activation function on the accumulator 202 value 217 of the first NPU 126 in the corresponding NPU 216 group. Therefore, as shown in FIG. 13 , in the clock cycle 513, the activation function unit 0 starts to execute the specified activation function on the accumulator 202 of the neural processing unit 0 to generate a result of literal 0 to be stored in the column register 1104; The activation function unit 1 starts to execute the specified activation function on the accumulator 202 of the neural processing unit 8 to generate the result of the word 8 that will be stored in the column register 1104; The accumulator 202 executes the specified activation function to produce the result of the text 504 to be stored in the column register 1104 .

In the time-frequency cycle 514, the activation function unit 0 (the activation function unit 1112 associated with the group 0) starts to execute the specified activation function on the value 217 of the accumulator 202 of the neural processing unit 1, and the neural processing unit 1 is in the group 0. The second NPU 216 activates the output of the functional unit 1112 to be stored in word 1 of the column register 1104 . Also in the time-frequency period 514 , each activation function unit 1112 starts to execute the specified activation function on the accumulator 202 value 217 of the second NPU 126 in the corresponding NPU 216 group. Therefore, as shown in FIG. 13 , in the time-frequency period 514, the activation function unit 0 starts to execute the specified activation function on the accumulator 202 of the neural processing unit 1 to generate a result of literal 1 that will be stored in the column register 1104; The activation function unit 1 starts to execute the specified activation function on the accumulator 202 of the neural processing unit 9 to generate the result of the text 9 that will be stored in the column register 1104; The accumulator 202 executes the specified activation function to generate the result of the text 505 to be stored in the column register 1104 . Such processing will continue until the time-frequency cycle 520, and the activation function unit 0 (associated with the activation function unit 1112 of group 0) starts to perform the specified activation function on the accumulator 202 value 217 of the neural processing unit 7, and the neural processing unit 7 That is, the eighth (last) NPU 216 in group 0, and the output of the activation function unit 1112 will be stored in word 7 of the column register 1104 . Also in the time-frequency period 520 , each activation function unit 1112 starts to execute the specified activation function on the accumulator 202 value 217 of the eighth NPU 126 in the corresponding NPU 216 group. Therefore, as shown in FIG. 13 , in the time-frequency period 520, the activation function unit 0 starts to execute the specified activation function on the accumulator 202 of the neural processing unit 7 to generate the result of the literal 7 that will be stored in the column register 1104; The activation function unit 1 starts to execute the specified activation function on the accumulator 202 of the neural processing unit 15 to generate the result of the text 15 which will be stored in the column register 1104; The accumulator 202 executes the specified activation function to generate the result of the text 511 to be stored in the column register 1104 .

In clock cycle 521, once all 512 results of the 512 NPUs 126 have been generated and written into the column buffer 1104, the column buffer 1104 will begin to write its content into the DRAM 122 or Weight random access memory 124 . In this way, the activation function unit 1112 of each neural processing unit 126 group executes a part of the activation function instruction at address 3 in FIG. 4 .

The embodiment in which the AFU 1112 is shared among the ALU 204 group as shown in FIG. 11 is particularly helpful for use with the integer ALU 204 . This part will be described in subsequent chapters as corresponding to FIG. 29A to FIG. 33 .

MTNN and MFNN Architecture Instructions

FIG. 14 is a block diagram showing a move to neural network (MTNN) architecture instruction 1400 and its operation corresponding to the portion of the neural network unit 121 of FIG. 1 . The MTNN instruction 1400 includes an executable code field 1402 , a src1 field 1404 , a src2 field, a gpr field 1408 and an immediate field 1412 . The MTNN instruction is an architectural instruction, that is, the instruction is included in the instruction set architecture of the processor 100 . For a preferred embodiment, the ISA uses the default value of the execution code field 1402 to distinguish the MTNN instruction 1400 from other instructions in the ISA. The execution code 1402 of the MTNN instruction 1400 may or may not include a prefix commonly found in the x86 architecture.

The immediate field 1412 provides a value to specify the function 1432 to the control logic 1434 of the NNU 121 . For a preferred embodiment, this function 1432 is used as an immediate operand of the microinstruction 105 of FIG. 1 . These functions 1432 that can be executed by the neural network unit 121 include writing data RAM 122, writing weight RAM 124, writing program memory 129, writing control buffer 127, starting execution program memory 129 program, suspend execution of the program in the program memory 129 , notify request (such as interrupt) after completing the execution of the program in the program memory 129 , and reset the neural network unit 121 , but are not limited thereto. In a preferred embodiment, the NNU command set includes a command whose result indicates that the NNU program is complete. Additionally, the NNU instruction set includes an explicit generate interrupt instruction. As far as a preferred embodiment is concerned, the operation of resetting the neural network unit 121 includes keeping the data in the neural network unit 121, except the data random access memory 122, the weight random access memory 124, and the program memory 129 intact. effectively force reverting to a reset state (for example, emptying the internal state machine and setting it to an idle state). In addition, internal registers, such as the accumulator 202, are not affected by the reset function, but must be cleared explicitly, such as using the initialize NPU instruction at address 0 in FIG. 4 . In one embodiment, the function 1432 may include a direct execution function whose first source register contains a micro-operation (for example, refer to the micro-operation 3418 of FIG. 34 ). The direct execution function instructs the neural network unit 121 to directly execute the specified micro-operation. In this way, the architecture program can directly control the neural network unit 121 to perform operations, instead of writing instructions into the program memory 129 and subsequently instructing the neural network unit 121 to execute the instructions located in the program memory 129 or through the MTNN instruction 1400 (or Execution of the MFNN instruction 1500) of FIG. 15 . FIG. 14 shows an example of the function for writing to the DRAM 122 .

The gpr field specifies a general register within the general register file 116 . In one embodiment, each general purpose register is 64 bits. The general register file 116 provides the value of the selected general register to the NNU 121 as shown in the figure, and the NNU 121 uses the value as the address 1422 . This address 1422 selects a column of memory specified in function 1432 . For DRAM 122 or WRAM 124, the address 1422 additionally selects a data block whose size is twice the size of the media register location in the selected row (eg, 512 bits). For a preferred embodiment, this location is on a 512-bit boundary. In one embodiment, the multiplexer will select address 1422 (or address 1422 in the case of the MFNN instruction 1400 described below) or addresses 123/125/131 from the sequencer 128 to provide for data random access Memory 124/weight random access memory 124/program memory 129. In one embodiment, the data random access memory 122 has dual ports, so that the neural processing unit 126 can use the media buffer 118 to read/write to the data random access memory 122 while reading/writing the data random access memory 122 . In one embodiment, the weight RAM 124 also has dual ports for a similar purpose.

The src1 field 1404 and the src2 field 1406 in the figure both designate a media buffer of the media buffer file 118 . In one embodiment, each media buffer 118 is 256 bits. The media buffer file 118 will provide contiguous data (eg, 512 bits) from the selected media buffer to the data RAM 122 (either the weight RAM 124 or the program memory 129) for writing The selected column 1428 specified by the address 1422 and the location within the selected column 1428 specified by the address 1422 are shown in the figure. Through the execution of a series of MTNN instructions 1400 (and MFNN instructions 1500 described below), the architecture program executed on the processor 100 can fill up the data RAM 122 columns and the weight RAM 124 columns and transfer the program Writing into the program memory 129, such as the program described herein (as shown in FIG. 4 and FIG. 9 ), enables the neural network unit 121 to perform calculations on data and weights at a very fast speed to complete the artificial neural network. In one embodiment, the architecture program directly controls the NNU 121 instead of writing the program into the program memory 129 .

In one embodiment, instead of specifying two source registers (as specified by fields 1404 and 1406 ), the MTNN instruction 1400 specifies a starting source register and the number of source registers, ie, Q. This form of the MTNN instruction 1400 will instruct the processor 100 to write the media buffer 118 designated as the initial source buffer and the next Q-1 subsequent media buffers 118 into the neural network unit 121, that is, write the Designated data RAM 122 or weight RAM 124 . For a preferred embodiment, the instruction translator 104 translates the MTNN instruction 1400 into a required number of microinstructions for writing all Q specified media registers 118 . For example, in one embodiment, when the MTNN instruction 1400 specifies the register MR4 as the initial source register and Q is 8, the instruction translator 104 will translate the MTNN instruction 1400 into four microinstructions, wherein the first The first microinstruction is written into registers MR4 and MR5, the second microinstruction is written into registers MR6 and MR7, the third microinstruction is written into registers MR8 and MR9, and the fourth microinstruction is written into registers MR10 and MR11 . In another embodiment, the data path from the media buffer 118 to the NNU 121 is 1024 bits instead of 512 bits. In this case, the instruction translator 104 translates the MTNN instruction 1400 into two microinstructions, where The first microinstruction is written into registers MR4 to MR7, and the second microinstruction is written into registers MR8 through MR11. The present invention is also applicable to embodiments where the MFNN instruction 1500 specifies a starting destination register and the number of destination registers, so that each MFNN instruction 1500 can read from a column of the data RAM 122 or the weight RAM 124 Data blocks larger than a single media buffer 118 are fetched.

FIG. 15 is a block diagram showing a move to neural network (MTNN) architecture instruction 1500 and its operation corresponding to the portion of the neural network unit 121 of FIG. 1 . This MFNN instruction 1500 includes an execution code field 1502 , a dst field 1504 , a gpr field 1508 , and an immediate field 1512 . The MFNN instruction is an architectural instruction, that is, the instruction is included in the instruction set architecture of the processor 100 . For a preferred embodiment, the ISA uses the default value of the execution code field 1502 to distinguish the MFNN instruction 1500 from other instructions in the ISA. The execution code 1502 of the MFNN instruction 1500 may or may not include a prefix commonly found in the x86 architecture.

The immediate field 1512 provides a value to specify the function 1532 to the control logic 1434 of the NNU 121 . For a preferred embodiment, this function 1532 is used as an immediate operand of the microinstruction 105 of FIG. 1 . The functions 1532 that the neural network unit 121 can execute include reading the data RAM 122 , reading the weight RAM 124 , reading the program memory 129 , and reading the state register 127 , but not limited thereto. The example of FIG. 15 shows a function 1532 for reading data RAM 122 .

The gpr field 1508 specifies a general register within the general register file 116 . The general register file 116 provides the value of the selected general register to the NNU 121 as shown in the figure, and the NNU 121 takes this value as an address 1522 and in a manner similar to the address 1422 of FIG. 14 An operation is performed whereby a column of memory specified in function 1532 is selected. For DRAM 122 or WRAM 124, this address 1522 additionally selects a data block whose size is the location of the media register (eg, 256 bits) in the selected row. For a preferred embodiment, this location is on a 256-bit boundary.

The dst field 1504 specifies a media buffer within a media buffer file 118 . As shown, the media buffer file 118 receives data (eg, 256 bits) from the data RAM 122 (or weight RAM 124 or program memory 129) into the selected media buffer. Read from the selected column 1528 specified by address 1522 in data reception and the location specified by address 1522 in the selected column 1528 .

Port Configuration of the Internal Random Access Memory of the Neural Network Unit

FIG. 16 is a block diagram showing an embodiment of the data random access memory 122 of FIG. 1 . The DRAM 122 includes a memory array 1606 , a read port 1602 and a write port 1604 . Memory array 1606 is loaded with words of data which, for a preferred embodiment, are arranged in an array of N words in D columns as previously described. In one embodiment, the memory array 1606 includes an array of 64 horizontally arranged SRAM cells, each of which has a width of 128 bits and a height of 64 bits, thus providing a 64KB The DRAM 122 has a width of 8192 bits and 64 columns, and the die area used by the DRAM 122 is approximately 0.2 square millimeters. However, the present invention is not limited thereto.

For a preferred embodiment, the write port 1602 is coupled to the NPU 126 and the media buffer 118 in a multi-tasking manner. Further, the media buffers 118 may be coupled to the read ports through a result bus, which is also used to provide data to the reorder buffer and/or the result transfer bus to other execution units 112 . The NPUs 126 share the read port 1602 with the media buffer 118 to read the data RAM 122 . Furthermore, for a preferred embodiment, the write port 1604 is also coupled to the NPU 126 and the media buffer 118 in a multi-tasking manner. The NPUs 126 share the write port 1604 with the media buffer 118 to write into the DRAM 122 . In this way, the media buffer 118 can write the data random access memory 122 while the neural processing unit 126 is reading the data random access memory 122, and the neural processing unit 126 can also write the data to the random access memory 122 while the media buffer 118 is writing to it. While the data random access memory 122 is reading, the data random access memory 122 is being written. Doing so can improve performance. For example, the NPUs 126 can read the DRAM 122 (eg, continue to perform calculations), while the media buffer 118 can write more data words into the DRAM 122 at the same time. In another example, the neural processing units 126 can write calculation results into the DRAM 122 , and at the same time, the media buffer 118 can read the calculation results from the DRAM 122 . In one embodiment, the neural processing unit 126 can write a column of calculation results into the DRAM 122 and read a column of data words from the DRAM 122 at the same time. In one embodiment, the memory array 1606 is configured into memory banks. When NPU 126 accesses DRAM 122, all memory banks are enabled to access a complete column of memory array 1606; , only the specified memory block will be activated. In one embodiment, the width of each memory block is 128 bits, and the width of the media buffer 118 is 256 bits, so, for example, each access to the media buffer 118 requires the activation of two memories blocks. In one embodiment, one of these ports 1602/1604 is a read/write port. In one embodiment, these ports 1602/1604 are all read/write ports.

An advantage of having these NPUs 126 with the capability of a rotator as described herein is that it allows the architecture program (via the media buffer 118 ) to continuously provide data to the RAM 122 and the memory array needed to retrieve results from DRAM 122 while NPU 126 is performing calculations, this capability facilitates reducing the number of columns in memory array 1606 of DRAM 122 number, and thus can be reduced in size.

Internal Random Access Memory Buffer

FIG. 17 is a block diagram showing an embodiment of the weight RAM 124 and the buffer 1704 of FIG. 1 . The weight RAM 124 includes a memory array 1706 and a port 1702 . The memory array 1706 is loaded with weight words, which are arranged as an array of N words in W columns as described above for a preferred embodiment. In one embodiment, the memory array 1706 includes an array of 128 horizontally arranged SRAM cells, each of which has a width of 64 bits and a height of 2048 bits, thus providing a 2MB The weight random access memory 124 has a width of 8192 bits and 2048 columns, and the die area used by the weight random access memory 124 is approximately 2.4 square millimeters. However, the present invention is not limited thereto.

For a preferred embodiment, the port 1702 is coupled to the NPU 126 and the buffer 1704 in a multi-tasking manner. These NPUs 126 and buffers 1704 read from and write to the weight RAM 124 through the port 1702 . The buffer 1704 is also coupled to the media buffer 118 of FIG. 1 , so that the media buffer 118 can read from and write to the weight RAM 124 through the buffer 1704 . The advantage of this approach is that when the NPU 126 is reading from or writing to the weight RAM 124, the media buffer 118 can also write to or read from the buffer 118 (but if the NPU units 126 are executing, these NPUs 126 are preferably parked to avoid accessing WRAM 124 when buffer 1704 accesses WRAM 124 ). This approach can improve performance, especially because the media buffer 118 reads and writes to the WRAM 124 significantly less than the NPU 126 reads and writes to the WRAM 124 . For example, in one embodiment, NPU 126 reads/writes 8192 bits (one column) at a time, however, media register 118 is only 256 bits wide, and each MTNN instruction 1400 only writes two Media buffer 118, namely 512 bits. Thus, in the case where the architectural program executes sixteen MTNN instructions 1400 to fill the buffer 1704, the conflict between the NPU 126 and the architectural program accessing the WRAM 124 takes less than approximately all of the time. six percent. In another embodiment, the instruction translator 104 translates one MTNN instruction 1400 into two microinstructions 105, and each microinstruction will write a single data register 118 into the buffer 1704, so that the neural processing unit 126 and The frequency of conflicts generated by the architecture program when accessing the weighted random access memory 124 is further reduced.

In embodiments including buffer 1704 , multiple MTNN instructions 1400 are required to write to weight RAM 124 using the architectural program. One or more MTNN instructions 1400 specify a function 1432 to write the specified data block in the buffer 1704, and then an MTNN instruction 1400 specifies a function 1432 instructing the neural network unit 121 to write the contents of the buffer 1704 to the weight random access memory A selected column of 124. The size of a single data block is twice the number of bits in the media buffer 118 , and these data blocks are naturally aligned in the buffer 1704 . In one embodiment, each MTNN instruction 1400 specifying a function 1432 to write a specified block of data in the buffer 1704 includes a bitmask with bits corresponding to each block of the buffer 1704 . Data from the two specified source registers 118 is written into the data blocks of buffer 1704, the corresponding bit in the bitmask being set for each data block. This embodiment facilitates situations where there are duplicate data values within a column of the WRAM 124 . For example, to zero the buffer 1704 (and subsequently a column of the weight RAM 124), the programmer can load the source buffer with a value of zero and set all bits of the bitmask. In addition, bit masking also allows the programmer to write only selected data blocks in buffer 1704 while leaving other data blocks to maintain their previous data states.

In an embodiment that includes buffer 1704 , multiple MFNN instructions 1500 are required to read weight RAM 124 using an architectural program. The initial MFNN instruction 1500 specifies a function 1532 to load a specified column of the weight random access unit 124 into the buffer 1704, and then one or more MFNN instructions 1500 specify a function 1532 to read a specified block of data from the buffer 1704 to the destination cache. The size of a single data block is the number of bits in the media buffer 118 , and these data blocks are naturally aligned in the buffer 1704 . The technical features of the present invention are also applicable to other embodiments. For example, the weight random access memory 124 has a plurality of buffers 1704. By increasing the number of architecture programs that can be accessed by the neural processing unit 126 during execution, the neural processing unit can be further reduced. The conflict between 126 and the architecture program due to the access weighted random access memory 124 increases. In the time-frequency cycle when the neural processing unit 126 does not need to access the weighted random access memory 124, it is accessed by the buffer 1704 instead. possibility.

FIG. 16 depicts the DPRAM 122, however, the present invention is not limited thereto. The technical features of the present invention are also applicable to other embodiments in which the weight random access memory 124 is also a dual-port design. In addition, it is described in FIG. 17 that the buffer is used with the weight random access memory 124 , but the present invention is not limited thereto. The technical features of the present invention are also applicable to embodiments in which the DRAM 122 has a corresponding buffer similar to the buffer 1704 .

Dynamically configurable neural processing unit

FIG. 18 is a block diagram illustrating the dynamically configurable neural processing unit 126 of FIG. 1 . The neural processing unit 126 of FIG. 18 is similar to the neural processing unit 126 of FIG. 2 . However, the NPU 126 of FIG. 18 is dynamically configurable to operate in one of two different configurations. In a first configuration, the neural processing unit 126 of FIG. 18 operates similarly to the neural processing unit 126 of FIG. 2 . That is, in a first configuration, denoted herein as a "wide" configuration or a "single" configuration, the ALU 204 of the NPU 126 operates on a single wide data word and a single wide weight word (eg 16 bits) to perform operations to produce a single wide result. In contrast, in the second configuration, designated herein as the "narrow" configuration or the "dual" configuration, the NPU 126 will process two narrow data words and two narrow weight words (e.g., 8 ones) perform the operation to produce two narrow results respectively. In one embodiment, the configuration (wide or narrow) of the NPU 126 is achieved by an initialization NPU command (such as the command located at address 0 in FIG. 20 ). Alternatively, this configuration can also be achieved by an MTNN instruction with the function 1432 specifying the configuration (wide or narrow) of the NPU settings. For a preferred embodiment, the program memory 129 instructions or the MTNN instructions that determine the configuration (wide or narrow) fill the configuration register. For example, the output of the configuration register is provided to the ALU 204 , the enable function unit 212 , and the logic that generates the MTR control signal 213 . Basically, the components of the NPU 126 of FIG. 18 perform similar functions as the same numbered components in FIG. 2 , and reference can be taken therefrom to understand the embodiment of FIG. 18 . The following describes the embodiment of FIG. 18 including its differences from FIG. 2 .

The neural processing unit 126 of FIG. 18 includes two registers 205A and 205B, two three-input multitasking registers 208A and 208B, an arithmetic logic unit 204, two accumulators 202A and 202B, and two activation function units 212A and 202B. 212B. The registers 205A/ 205B respectively have half the width of the register 205 of FIG. 2 (eg, 8 bits). The registers 205A/205B respectively receive a corresponding narrow weight word 206A/B 206 (e.g., 8 bits) from the weight RAM 124 and provide their output 203A/203B to the ALU 204 in a subsequent clock cycle. Operand selection logic 1898. When NPU 126 is in a wide configuration, registers 205A/205B operate together to receive a wide weight word 206A/206B (e.g., 16 bits) from weight RAM 124, similar to the embodiment of FIG. 2 When the NPU 126 is in a narrow configuration, the registers 205A/205B will actually operate independently, each receiving a narrow weight text 206A/206B from the weight random access memory 124 (for example, 8 bit), so that the NPU 126 is actually equivalent to two narrow NPUs operating independently. However, regardless of the configuration of the NPU 126 , the same output bits of the WRAM 124 are coupled and provided to the registers 205A/ 205B. For example, NPU0 buffer 205A receives byte 0, NPU0 buffer 205B receives byte 1, NPU1 buffer 205A receives byte 2, NPU1 The buffer 205B of NPU 511 receives byte 3, and so on, the buffer 205B of NPU 511 receives byte 1023.

The multitasking registers 208A/ 208B respectively have half the width of the register 208 of FIG. 2 (eg, 8 bits). The multitasking register 208A will select one of the inputs 207A, 211A, and 1811A to store in its register and provide it from the output 209A in subsequent clock cycles. The multitasking register 208B will select one of the inputs 207B, 211B, and 1811B to store in the register. It is buffered and provided to operand select logic 1898 by output 209B in subsequent clock cycles. Input 207A receives a narrow data word (eg, 8 bits) from DRAM 122 and input 207B receives a narrow data word from DRAM 122 . When the NPU 126 is in a wide configuration, the multitasking registers 208A/208B will actually work together to receive a wide data word 207A/207B (e.g., 16 bits) from the data RAM 122, similar to In the multitasking register 208 in the embodiment of FIG. Narrow data words 207A/207B (for example, 8 bits), so that the NPU 126 is actually equivalent to two narrow NPUs operating independently. However, regardless of the configuration of the NPU 126 , the same output bits of the DRAM 122 are coupled and provided to the multitasking registers 208A/ 208B. For example, MMR 208A of NPU0 receives byte 0, MMR 208B of NPU0 receives byte 1, and MMR 208A of NPU1 receives byte 2. The multitasking register 208B of the neural processing unit 1 receives byte 3, and so on, the multitasking register 208B of the neural processing unit 511 receives byte 1023.

The input 211A receives the output 209A of the multitasking buffer 208A of the adjacent NPU 126 , and the input 211B receives the output 209B of the multitasking buffer 208B of the adjacent NPU 126 . Input 1811A receives output 209B of multitasking buffer 208B adjacent to NPU 126 , and input 1811B receives output 209A of multitasking buffer 208A adjacent to NPU 126 . The neural processing unit 126 shown in FIG. 18 belongs to one of the N neural processing units 126 shown in FIG. 1 and is labeled as neural processing unit J. That is to say, the neural processing unit J is a representative example of the N neural processing units. For a preferred embodiment, the multitasking register 208A input 211A of the NPU J receives the output 209A of the multitasking register 208A of the NPU 126 of Example J-1, and the multitasking buffer of the NPU J NPU 208A input 1811A will receive output 209B of multitasking register 208B of NPU 126 of example J-1, and output 209A of multitasking register 208A of NPU J will be simultaneously provided to NPU 126 of example J+1 Input 211A of the multitasking register 208A of the example J and the input 211B of the multitasking register 208B of the NPU 126 of the example J; the input 211B of the multitasking register 208B of the NPU J will receive the The multitasking buffer 208B outputs 209B, and the input 1811B of the multitasking buffer 208B of the NPU J receives the output 209A of the multitasking buffer 208A of the NPU 126 of the example J, and the multitasking buffer of the NPU J The output 209B of the register 208B is provided to both the multitasking register 208A input 1811A of the NPU 126 of the example J+1 and the multitasking register 208B input 211B of the NPU 126 of the example J+1.

The control input 213 controls each of the multiplexing registers 208A/208B, selects one of the three inputs to be stored in its corresponding register, and provided to the corresponding output 209A/209B in a subsequent step. When the neural processing unit 126 is instructed to load a column from the data random access memory 122 (such as the multiply-accumulate instruction at address 1 in FIG. , the control input 213 will control each of the multiplex registers in the multiplex registers 208A/208B to select a corresponding narrow data word 207A/207B from the corresponding narrow words of the selected column of the data random access memory 122 (eg 8 bits).

When the neural processing unit 126 receives an indication that it needs to rotate the value of the previously received data column (for example, the multiply-accumulate rotation instruction of address 2 in FIG. Each multitasking register in the multitasking register 208A/208B is controlled to select the corresponding input 1811A/1811B. In this case, the multi-tasking registers 208A/208B would actually operate independently so that the NPU 126 would actually act like two independent narrow NPUs. In this way, the multitasking registers 208A and 208B of the N NPUs 126 work together as a 2N narrow text rotator, and this part will be described in more detail corresponding to FIG. 19 .

When the NPU 126 receives an indication that it needs to rotate the value of the previously received data column, if the NPU 126 is in a wide configuration, the control input 213 will control the selection of each multitasking register in the multitasking register 208A/208B. Corresponding to input 211A/211B. In this case, the multi-tasking registers 208A/208B will operate together as if the NPU 126 is actually a single wide NPU 126 . In this way, the multitasking registers 208A and 208B of the N NPUs 126 work together as an N wide text rotator, similar to the manner described in FIG. 3 .

ALU 204 includes operand selection logic 1898, a wide multiplier 242A, a narrow multiplier 242B, a wide two-input multiplexer 1896A, a narrow two-input multiplexer 1896B, a wide adder 244A and a narrow adder device 244B. In fact, the ALU 204 can be understood as including operand selection logic, a wide ALU 204A (including the aforementioned wide multiplier 242A, the aforementioned wide multiplexer 1896A, and the aforementioned wide adder 244A) and a narrow ALU 204B (including the aforementioned narrow multiplier 242B, the aforementioned narrow multiplexer 1896B and the aforementioned narrow adder 244B). For a preferred embodiment, wide multiplier 242A can multiply two wide literals, similar to multiplier 242 of FIG. 2 , eg, a 16-bit by 16-bit multiplier. Narrow multiplier 242B can multiply two narrow literals, such as an 8-bit by 8-bit multiplier, to produce a 16-bit result. When NPU 126 is in the narrow configuration, wide multiplier 242A can be fully utilized with the assistance of operand selection logic 1898, which acts as a narrow multiplier for multiplying two narrow literals so that NPU 126 will Like two narrow neural processing units functioning efficiently. For a preferred embodiment, wide adder 244A adds the output of wide multiplexer 1896A to output 217A of wide accumulator 202A to produce a total 215A for wide accumulator 202A, which operates similarly to FIG. 2 The adder 244. Narrow adder 244B will add the output of narrow multiplexer 1896B and narrow accumulator 202B output 217B to produce a sum 215B for use by narrow accumulator 202B. In one embodiment, the narrow accumulator 202B has a width of 28 bits to avoid loss of accuracy when accumulating up to 1024 16-bit products. When the NPU 126 is in the wide configuration, the narrow multiplier 244B, the narrow accumulator 202B, and the narrow enable function unit 212B are preferably disabled to reduce power consumption.

The operand selection logic 1898 selects operands from 209A, 209B, 203A, and 203B to provide to other components of the ALU 204, which will be described in detail later. For a preferred embodiment, operand selection logic 1898 also has other functions, such as performing sign extension of signed numeric data literals and weight literals. For example, if NPU 126 is in a narrow configuration, operand select logic 1898 stretches the sign of the narrow data word and weight word to the width of the wide word before providing it to wide multiplier 242A. Similarly, if ALU 204 accepts an indication to pass a narrow data/weight literal (using wide multiplexer 1896A to skip wide multiplier 242A), operand select logic 1898 will extend the sign of the narrow data literal and weight literal to The width of the wide text is then provided to wide adder 244A. As far as a preferred embodiment is concerned, the logic for performing the symbol extension function also exists inside the ALU 204 of the NPU 126 of FIG. 2 .

Wide multiplexer 1896A receives the output of wide multiplier 242A and an operand from operand select logic 1898, and selects one of these inputs for wide adder 244A, and narrow multiplexer 1896B receives the output of narrow multiplier 242B. The output is ANDed with an operand from operand select logic 1898, and a selected one of these inputs is provided to narrow adder 244B.

The operand selection logic 1898 provides operands based on the configuration of the NPU 126 and the arithmetic and/or logic operations to be performed by the ALU 204. The arithmetic/logic operations are determined based on the functions specified by the instructions executed by the NPU 126 . For example, if the instruction instructs ALU 204 to perform a multiply-accumulate operation and NPU 126 is in a wide configuration, operand select logic 1898 provides a wide literal consisting of outputs 209A and 209B concatenated to wide multiplier 242A. 1 input, and a wide text formed by the concatenation of outputs 203A and 203B is provided to the other input, while the narrow multiplier 242B is not enabled, so that the operation of the neural processing unit 126 will be like a single Wide neural processing unit 126 of neural processing unit 126 . However, if the instruction instructs the ALU to perform a multiply-accumulate operation and NPU 126 is in the narrow configuration, operand select logic 1898 provides a stretched or expanded version of narrow data literal 209A to a wide multiplier 242A. input, and an extended version of narrow weight literal 203A is provided to another input; in addition, operand selection logic 1898 provides narrow data literal 209B to one input of narrow multiplier 242B and narrow weight literal 203B to another input one input. In order to achieve the operation of extending or expanding narrow literals as described above, if the narrow literal is signed, the operand selection logic 1898 will perform sign extension on the narrow literal; if the narrow literal is unsigned, the operand selection logic 1898 will add the upper zero value bit to the narrow text.

In another example, if the NPU 126 is in a wide configuration and the instruction instructs the ALU 204 to perform a weighted literal accumulation operation, the wide multiplier 242A is skipped and the operand select logic 1898 outputs 203A It is connected in series with 203B to provide wide multiplexer 1896A to provide wide adder 244A. However, if the NPU 126 is in a narrow configuration and the instruction instructs the ALU 204 to perform an accumulation of weighted literals, the wide multiplier 242A is skipped and the operand selection logic 1898 outputs a stretched version 203A is provided to wide multiplexer 1896A to provide to wide adder 244A; in addition, narrow multiplier 242B is skipped and operand select logic 1898 provides a stretched version of the output 203B to narrow multiplexer 1896B to provide Narrow adder 244B.

In another example, if NPU 126 is in a wide configuration and the instruction instructs ALU 204 to perform an accumulation operation of a data word, wide multiplier 242A is skipped and operand select logic 1898 outputs 209A It is connected in series with 209B to provide wide multiplexer 1896A to provide wide adder 244A. However, if the NPU 126 is in the narrow configuration and the instruction instructs the ALU 204 to perform an accumulation operation of a data word, the wide multiplier 242A is skipped and the operand select logic 1898 outputs a stretched version 209A is provided to wide multiplexer 1896A for wide adder 244A; in addition, narrow multiplier 242B is skipped and operand select logic 1898 provides a stretched version of the output 209B to narrow multiplexer 1896B for supply to Narrow adder 244B. Accumulation of weights/data literals facilitates averaging, which can be used as a pooling layer in some artificial neural network applications such as image processing.

As far as a preferred embodiment is concerned, the neural processing unit 126 further includes a second wide multiplexer (not shown), which is used to skip the wide adder 244A, so as to facilitate wide data/weight literals or The extended narrow data/weight text in the narrow configuration is loaded into the wide accumulator 202A, and a second narrow multiplexer (not shown) is used to skip the narrow adder 244B, so as to facilitate the narrow data/weight text in the narrow configuration The weight literal loads narrow accumulator 202B. For a preferred embodiment, the ALU 204 also includes wide and narrow comparator/multiplexer combinations (not shown), which receive corresponding accumulator values 217A/ 217B and the corresponding multiplexer 1896A/1896B output, so as to select the maximum value between the accumulator value 217A/217B and a data/weight text 209A/209B/203A/203B, common source (pooling) of some artificial neural network applications ) layer uses this operation, and this part will be described in more detail in subsequent chapters, for example corresponding to FIG. 27 and FIG. 28 . Additionally, operand select logic 1898 is configured to provide operands of value zero (for addition operations that add zero or to clear the accumulator) and operands of value one (for multiplication operations that multiply by one).

Narrow activation function unit 212B receives output 217B of narrow accumulator 202B and performs an activation function on it to produce narrow result 133B, and wide activation function unit 212A receives output 217A of wide accumulator 202A and performs an activation function on it to produce wide result 133A. When the neural processing unit 126 is in a narrow configuration, the wide activation function unit 212A will understand the output 217A of the accumulator 202A according to this configuration and execute the activation function on it to generate a narrow result, such as 8 bits. This part will be described in subsequent chapters as corresponding to FIG. 29A See Figure 30 for a more detailed description.

As previously mentioned, a single NPU 126 in the narrow configuration can actually act as two narrow NPUs, thus providing substantially more for smaller text than in the wide configuration. Up to twice the processing power. For example, suppose a neural network layer has 1024 neurons, and each neuron receives 1024 narrow inputs (with narrow weight text) from the previous layer, which would result in a million connections. For the neural network unit 121 with 512 neural processing units 126, under the narrow configuration (equivalent to 1024 narrow neural processing units), although narrow text is processed instead of wide text, the number of connections it can process Four times the wide configuration can be achieved (1 million links vs. 256K links), and the time required is roughly half that (approximately 1026 clock cycles vs. 514 clock cycles).

In one embodiment, the dynamically configured NPU 126 of FIG. 18 includes a three-input multitasking register similar to multitasking registers 208A and 208B instead of registers 205A and 205B to form a rotator, and the processing is randomized by weights. The operation of accessing the weight text string received by the memory 124 is similar to that described in the embodiment of FIG. 7 but applied to the dynamic configuration described in FIG. 18 .

FIG. 19 is a schematic block diagram showing that according to the embodiment of FIG. 18 , using 2N multitasking registers 208A/208B of N neural processing units 126 of the neural network unit 121 of FIG. A row of data words 207 retrieved from memory 122 performs like a rotator. In the embodiment of FIG. 19, N is 512, and the NPU 121 has 1024 multitasking registers 208A/208B, labeled 0 to 511, corresponding to 512 NPUs 126 and actually 1024 narrow NPUs, respectively. unit. The two narrow NPUs in the NPU 126 are labeled A and B, respectively, and the corresponding narrow NPUs in each multitasking register 208 are also labeled. Further, the multitasking register 208A of the neural processing unit 126 marked as 0 is marked as 0-A, the multitasking register 208B of the neural processing unit 126 marked as 0 is marked as 0-B, and the neural processing unit 126 marked as 1 The multitasking register 208A of unit 126 is designated 1-A, the multitasking register 208B of the NPU 126 designated 1 is designated 1-B, and the multitasking register 208A of the NPU 126 designated 511 is designated as 511-A, and the multitasking register 208B of the neural processing unit 126 marked as 511 is marked as 511-B, and its value also corresponds to the narrow neural processing unit described in FIG. 21 .

Each multiplexing register 208A receives its corresponding narrow data word 207A in one of the D columns of the DRAM 122, and each multiplexing register 208B receives its corresponding narrow data word 207A in one of the D columns of the DRAM 122. One of the columns receives its corresponding narrow data literal 207B. That is to say, MMR 0-A receives narrow data word 0 of DRAM 122 columns, MMR 0-B receives narrow data word 1 of DRAM 122 columns, MMR 0-B receives narrow data word 1 of DRAM 122 columns, 1-A receives the narrow data word 2 of 122 rows in the DRAM, the multitasking register 1-B receives the narrow data word 3 of the 122 rows in the DRAM, and so on, and the multitasking register 511-A receives The narrow data word 1022 of the DRAM 122 columns, and the multiplexing register 511-B receives the narrow data word 1023 of the 122 columns of the DRAM. In addition, multitasking register 1-A receives output 209A of multitasking register 0-A as its input 211A, multitasking register 1-B receives output 209B of multitasking register 0-B as its input 211B, and so on By analogy, the multitasking buffer 511-A receives the output 209A of the multitasking buffer 510-A as its input 211A, the multitasking buffer 511-B receives the output 209B of the multitasking buffer 510-B as its input 211B, and multiple Tasking buffer 0-A receives output 209A of multitasking buffer 511-A as its input 211A, and multitasking buffer 0-B receives output 209B of multitasking buffer 511-B as its input 211B. Each multiplexing register 208A/208B receives a control input 213 to control it to select the data word 207A/207B or the rotated input 211A/211B or the rotated input 1811A/1811B. Finally, multitasking register 1-A receives output 209B of multitasking register 0-B as its input 1811A, multitasking register 1-B receives output 209A of multitasking register 1-A as its input 1811B, and so on By analogy, the multitasking buffer 511-A receives the output 209B of the multitasking buffer 510-B as its input 1811A, the multitasking buffer 511-B receives the output 209A of the multitasking buffer 511-A as its input 1811B, and multiple Tasking buffer 0-A receives output 209B of multitasking buffer 511-B as its input 1811A, and multitasking buffer 0-B receives output 209A of multitasking buffer 0-A as its input 1811B. Each multiplexing register 208A/208B receives a control input 213 to control it to select the data word 207A/207B or the rotated input 211A/211B or the rotated input 1811A/1811B. In an operation mode, in the first time-frequency cycle, the control input 213 will control each multiplex register 208A/208B to select the data word 207A/207B to be stored in the register for subsequent provision to the arithmetic logic unit 204; Frequency cycle (such as the aforementioned M-1 time-frequency cycle), the control input 213 will control each multi-tasking register 208A/208B to select the rotation and then input 1811A/1811B to store in the register for subsequent provision to the arithmetic logic unit 204, this part It will be described in more detail in subsequent chapters.

FIG. 20 is a table showing a program stored in the program memory 129 of the neural network unit 121 of FIG. 1 and executed by the neural network unit 121 having the neural processing shown in the embodiment of FIG. 18 Unit 126. The example program of FIG. 20 is similar to the program of FIG. 4 . The differences are explained below. The initialize NPU instruction at address 0 specifies that NPU 126 will enter a narrow configuration. In addition, as shown in the figure, the multiply accumulate rotate instruction at address 2 specifies a count value of 1023 and requires 1023 clock cycles. This is because the example in Figure 20 assumes that there are actually 1024 narrow (e.g., 8-bit) neurons (i.e., neural processing units) in a layer, each with 1024 of the 1024 neurons from the previous layer of link inputs, so there are 1024K links in total. Each neuron receives an 8-bit data value from each connection input and multiplies this 8-bit data value by an appropriate 8-bit weight value.

FIG. 21 is a timing diagram showing that the neural network unit 121 executes the program of FIG. 20 . The neural network unit 121 has the neural processing unit 126 as shown in FIG. 18 executed in a narrow configuration. The timing diagram of FIG. 21 is similar to the timing diagram of FIG. 5 . The differences are explained below.

In the timing diagram of FIG. 21, the NPUs 126 would be in the narrow configuration because the initialize NPU instruction at address 0 initializes them to the narrow configuration. Therefore, the 512 NPUs 126 actually operate as 1024 narrow NPUs (or neurons) within the field as NPU0-A and NPU0- B (the two narrow NPUs of NPU 126 marked as 0), NPU 1-A and NPU 1-B (the two narrow NPUs of NPU 126 marked as 1), And so on until NPU 511-A and NPU 511-B (the two narrow NPUs of NPU 126 labeled 511 ) are indicated. To simplify the description, only the operations of the narrow NPUs 0-A, 0-B and 511-B are shown in the figure. Since the count value specified by the multiply accumulate rotate instruction at address 2 is 1023, 1023 clock cycles are required for operation. Therefore, the columns of the timing diagram in FIG. 21 include as many as 1026 clock cycles.

In clock cycle 0, each of the 1024 NPUs executes the initialization command shown in FIG. 4 , that is, the operation of assigning a value of zero to the accumulator 202 shown in FIG. 5 .

In clock cycle 1, each of the 1024 narrow NPUs executes the multiply-accumulate instruction at address 1 in Figure 20. As shown, narrow NPU 0-A adds the accumulator 202A value (i.e. zero) to column 17 narrow word 0 of data random access unit 122 and column 0 narrow word 0 of weight random access unit 124 Product; the narrow neural processing unit 0-B adds the accumulator 202B value (i.e. zero) to the product of the column 17 narrow text 1 of the data random access unit 122 and the column 0 narrow text 1 of the weight random access unit 124; accordingly By analogy, the narrow NPU 511 -B adds the value of the accumulator 202B (ie zero) to the product of the column 17 narrow text 1023 of the data random access unit 122 and the column 0 narrow text 1023 of the weight random access unit 124 .

In clock cycle 2, each of the 1024 narrow NPUs executes the first iteration of the multiply-accumulate-rotate instruction at address 2 in FIG. 20 . As shown, the narrow NPU 0-A adds the accumulator 202A value 217A to the rotated narrow data word 1811A received by the multitasking register 208B output 209B of the narrow NPU 511-B (i.e. Product of narrow data word 1023) received by data random access memory 122 and column 1 narrow word 0 of weight random access unit 124; narrow neural processing unit 0-B adds accumulator 202B value 217B The 0-A multiplexing register 208A outputs the rotated narrow data word 1811B received by 209A (that is, the narrow data word 0 received by the DRAM 122 ) and the column 1 narrow word of the weight RAM 124 1; and so on until the narrow NPU 511-B adds the accumulator 202B value 217B to the rotated narrow data literal 1811B ( That is, the product of the narrow data word 1022 received by the DRAM 122 and the column 1 narrow word 1023 of the weight RAM 124 .

At clock cycle 3, each of the 1024 narrow NPUs executes the second iteration of the multiply-accumulate-rotate instruction at address 2 in Figure 20. As shown, the narrow NPU 0-A adds the accumulator 202A value 217A to the rotated narrow data word 1811A received by the multitasking register 208B output 209B of the narrow NPU 511-B (i.e. The product of narrow data word 1022) received by data random access memory 122 and column 2 narrow word 0 of weight random access unit 124; The 0-A multiplexing register 208A outputs the rotated narrow data word 1811B received by 209A (that is, the narrow data word 1023 received by the DRAM 122 ) and the column 2 narrow word of the weight RAM 124 1; and so on until the narrow NPU 511-B adds the accumulator 202B value 217B to the rotated narrow data literal 1811B ( That is, the product of the narrow data word 1021) received by the data random access memory 122 and the column 2 narrow word 1023 of the weight random access unit 124. As shown in FIG. 21 , this operation will continue for the following 1021 time-frequency periods until the time-frequency period 1024 described below.

In clock cycle 1024, each of the 1024 narrow NPUs executes iteration 1023 of the multiply accumulate rotate instruction at address 2 in FIG. 20 . As shown, the narrow NPU 0-A adds the accumulator 202A value 217A to the rotated narrow data word 1811A received by the multitasking register 208B output 209B of the narrow NPU 511-B (i.e. The product of narrow data word 1) received by data random access memory 122 and column 1023 narrow word 0 of weight random access unit 124; narrow neural processing unit 0-B adds accumulator 202B value 217B by narrow neural processing unit The 0-A multiplexing register 208A outputs the rotated narrow data word 1811B received by the output 209A (that is, the narrow data word 2 received by the DRAM 122 ) and the row 1023 narrow word of the weight RAM 124 1; and so on until the narrow NPU 511-B adds the accumulator 202B value 217B to the rotated narrow data literal 1811B ( That is, the product of the narrow data word O) received by the DRAM 122 and the column 1023 narrow word 1023 of the weight RAM 124 .

In the time-frequency cycle 1025, the enable function unit 212A/212B of each of the 1024 narrow NPUs executes the enable function instruction located at address 3 in FIG. 20 . Finally, at clock cycle 1026, each of the 1024 narrow NPUs will write its narrow result 133A/133B back to the corresponding narrow text in column 16 of the data random access memory 122 to execute the The write at address 4 initiates the functional unit instruction. That is, narrow result 133A of NPU 0-A will be written into narrow literal 0 of DRAM 122, and narrow result 133B of NPU 0-B will be written into narrow literal 0 of DRAM 122. Literal 1, and so on, until narrow result 133B of NPU 511-B is written to narrow literal 1023 of DRAM 122 . FIG. 22 shows the aforementioned operations corresponding to FIG. 21 in a block diagram.

FIG. 22 is a schematic block diagram showing the neural network unit 121 of FIG. 1 , and the neural network unit 121 has the neural processing unit 126 shown in FIG. 18 to execute the program of FIG. 20 . This neural network unit 121 includes 512 neural processing units 126, i.e. 1024 narrow neural processing units, data random access memory 122, and weight random access memory 124, data random access memory 122 receives its address input 123, weight random access memory 122 Access memory 124 receives its address input 125 . Although not shown in the figure, in the time-frequency cycle 0, the 1024 narrow neural processing units will execute the initialization instruction shown in FIG. 20 . As shown in the figure, in clock cycle 1, 1024 8-bit data words of row 17 are read from DRAM 122 and provided to the 1024 narrow NPUs. In clock cycles 1 to 1024, 1024 8-bit weight texts of columns 0 to 1023 are respectively read from the weight RAM 124 and provided to the 1024 narrow NPUs. Although not shown in the figure, in time-frequency cycle 1, the 1024 narrow neural processing units perform their corresponding multiplication and accumulation operations on the loaded data literals and weight literals. During clock cycles 2 to 1024, the 1024 narrow NPU multitasking registers 208A/208B operate as a 1024 8-bit word rotator, relocating the previously loaded data to row 17 of RAM 122 The data words are rotated to adjacent narrow NPUs, and these narrow NPUs perform multiply-accumulate operations on the corresponding rotated data words and the corresponding narrow weight words loaded from the weight RAM 124 . Although not shown in the figure, in the clock cycle 1025, the 1024 narrow enable function units 212A/212B execute the enable command. The 1024 narrow NPUs will write their corresponding 1024 8-bit results 133A/ 133B back to the column 16 of the DRAM 122 in the clock cycle 1026 .

It can be found that, compared with the embodiment of FIG. 2 , the embodiment of FIG. 18 allows program designers to have the flexibility to choose to use wide data and weight literals (such as 16 bits) and narrow data and weight literals (such as 8 bits) for execution. Calculated to meet the accuracy requirements of specific applications. From one perspective, for narrow data applications, the embodiment of FIG. 18 can provide twice the performance compared to the embodiment of FIG. 205B, narrow ALU 204B, narrow accumulator 202B, and narrow AFU 212B), these additional narrow components increase the area of the NPU 126 by approximately 50%.

Trimodal Neural Processing Unit

FIG. 23 is a block diagram illustrating another embodiment of the dynamically configurable neural processing unit 126 of FIG. 1 . The NPU 126 of FIG. 23 can be used not only in wide and narrow configurations, but also in a third configuration, referred to herein as a "funnel" configuration. The neural processing unit 126 of FIG. 23 is similar to the neural processing unit 126 of FIG. 18 . However, the wide adder 244A in FIG. 18 is replaced in the neural processing unit 126 of FIG. An extended version of the output of device 1896B. The procedure executed by the neural network unit having the neural processing unit of FIG. 23 is similar to that of FIG. 20 . However, the initialize NPU instruction at address 0 therein initializes the NPUs 126 to a funnel configuration rather than a narrow configuration. Also, the multiply accumulate rotate instruction at address 2 has a count of 511 instead of 1023.

In the funnel configuration, the NPU 126 operates similarly to the narrow configuration. When executing a multiply-accumulate instruction at address 1 in FIG. 20 , the NPU 126 receives two narrow data words 207A/207B and two narrow weights Literal 206A/206B; wide multiplier 242A will multiply data literal 209A with weight literal 203A to produce wide multiplexer 1896A selected product 246A; narrow multiplier 242B will multiply data literal 209B with weight literal 203B to produce narrow Multiplexer 1896B selects product 246B. However, wide adder 2344A adds both product 246A (selected by wide multiplexer 1896A) and product 246B/2399 (selected by wide multiplexer 1896B) to wide accumulator 202A output 217A, while narrow adder 244B and Narrow accumulator 202B is then disabled. In addition, when executing the multiply-accumulate rotation instruction at address 2 in FIG. 20 in the funnel configuration, the control input 213 will cause the multitasking register 208A/208B to rotate two narrow words (such as 16 bits), that is, the multitasking buffer The switch 208A/208B will select its corresponding input 211A/211B as if in the wide configuration. However, wide multiplier 242A will multiply data word 209A by weight word 203A to produce product 246A selected by wide multiplexer 1896A; narrow multiplier 242B will multiply data word 209B by weight word 203B to produce narrow multiplexer product 246B selected by 1896B; and, wide adder 2344A will add both product 246A (selected by wide multiplexer 1896A) and product 246B/2399 (selected by wide multiplexer 1896B) to wide accumulator 202A output 217A, The narrow adder 244B and the narrow accumulator 202B are disabled as described above. Finally, when the activation function instruction at address 3 in FIG. 20 is executed in the funnel configuration, the wide activation function unit 212A executes the activation function on the total number of results 215A to generate a narrow result 133A, while the narrow activation function unit 212B does not activate. In this way, only the narrow NPU marked A will generate the narrow result 133A, and the narrow result 133B generated by the narrow NPU marked B is invalid. Therefore, the column that writes back the result (column 16 indicated by the instruction at address 4 in FIG. 20) will contain a hole because only narrow result 133A is valid and narrow result 133B is invalid. Thus, conceptually, each neuron (neural processing unit in Figure 23) executes two concatenated data inputs per time-frequency cycle, that is, multiplying two narrow data literals by their corresponding weights and combining the two In contrast, the embodiments of FIG. 2 and FIG. 18 only perform one link data input in each time-frequency cycle.

In the embodiment of FIG. 23 it can be seen that the number of resulting words (neuron outputs) generated and written back to the data RAM 122 or the weight RAM 124 is half the square root of the number of received data inputs (connections) , and the writeback column of the result has a hole, that is, every other narrow text result is invalid, more precisely, the narrow NPU result marked as B has no meaning. Thus, the embodiment of FIG. 23 is particularly efficient for neural networks having two consecutive layers, for example, the first layer has twice as many neurons as the second layer (e.g., the first layer has 1024 neurons fully connected to 512 neurons in the second layer). In addition, other execution units 122 (such as media units, such as the x86 advanced vector extension unit) can perform pack operations on a scattered result column (ie, with holes) to make it compact (ie, without holes) if necessary . Subsequently, when the neural processing unit 121 is performing other calculations associated with other columns of the data RAM 122 and/or the weight RAM 124 , the processed data columns can be used for calculations.

Hybrid Neural Network Unit Operations: Convolution and Common Source Computing Capabilities

The advantage of the neural network unit 121 described in the embodiment of the present invention is that the neural network unit 121 can simultaneously operate in a manner similar to a coprocessor executing its own internal program and in a processing unit similar to a processor to execute issued Architectural instructions (or microinstructions translated from architectural instructions). The architectural instructions are included in the architectural program executed by the processor with the neural network unit 121 . In this way, the neural network unit 121 can operate in a mixed manner, so as to maintain a high utilization rate of the neural processing unit 121 . For example, FIGS. 24 to 26 show the operation of the neural network unit 121 performing the convolution operation, wherein the neural network unit is fully utilized, and FIGS. 27 to 28 show the operation of the neural network unit 121 performing the common source operation. Convolutional layers, common source layers, and other digital data computing applications, such as image processing (eg, edge detection, sharpening, blurring, recognition/classification) require the use of these operations. However, the hybrid operation of the neural processing unit 121 is not limited to performing convolution or common source operations, and this hybrid feature can also be used to perform other operations, such as traditional neural network multiply-accumulate operations and activation function operations described in FIGS. 4 to 13 . That is to say, the processor 100 (more precisely, the reservation station 108) will issue the MTNN instruction 1400 and the MFNN instruction 1500 to the neural network unit 121, and the neural network unit 121 will write data into the memory 122/ 124/129 and read the result from the memory 122/124 written by the neural network unit 121. At the same time, in order to execute the program written into the program memory 129 by the processor 100 (through the MTNN1400 instruction), the neural network unit 121 Will read and write to memory 122/124/129.

FIG. 24 is a block diagram showing an example of a data structure used by the neural network unit 121 of FIG. 1 to perform convolution operations. The block diagram includes a convolution kernel 2402 , a data array 2404 , and the data RAM 122 and the weight RAM 124 in FIG. 1 . For a preferred embodiment, the data array 2404 (eg, corresponding to image pixels) is loaded in a system memory (not shown) connected to the processor 100 and is loaded by the processor 100 into the neural network unit 121 by executing the MTNN instruction 1400 The weight random access memory 124. The convolution operation convolves the first array with the second array, and the second array is the convolution kernel described herein. As described herein, a convolution kernel is a matrix of coefficients, and these coefficients may also be called weights, parameters, elements or values. For a preferred embodiment, the convolution kernel 2042 is static data of the architecture program executed by the processor 100 .

The data array 2404 is a two-dimensional array of data values, and the size of each data value (such as an image pixel value) is the size of a character in the data RAM 122 or weight RAM 124 (such as 16 bits or 8 bits). In this example, the data value is a 16-bit word, and the NNU 121 is a NPU 126 configured with a 512-wide configuration. In addition, in this embodiment, the neural processing unit 126 includes a multitasking register to receive the weight text 206 from the weight random access memory 124, such as the multitasking register 705 in FIG. The received column of data values performs a collective rotator operation, which will be described in more detail in subsequent chapters. In this example, the data array 2404 is a pixel array with 2560 rows×1600 columns. As shown in the figure, when the architecture program performs convolution calculation on the data array 2404 and the convolution kernel 2402, the data array 2402 will be divided into 20 data blocks, and each data block is a 512x400 data array 2406.

In this example, the convolution kernel 2402 is a 3x3 array of coefficients, weights, parameters, or elements. The first column of these coefficients is labeled C0,0; C0,1; and C0,2; the second column of these coefficients is labeled C1,0; C1,1; and C1,2; the third column of these coefficients is labeled C2,0; C2,1; and C2,2. For example, a convolution kernel with the following coefficients can be used to perform edge detection: 0,1,0,1,-4,1,0,1,0. In another embodiment, a convolution kernel with the following coefficients may be used to perform a Gaussian blur operation: 1,2,1,2,4,2,1,2,1. In this example, a division is usually performed on the final accumulated value, wherein the divisor is the sum of the absolute values of the elements of the convolution kernel 2042 , which is 16 in this example. In another example, the divisor may be the number of elements of the convolution kernel 2042 . In yet another example, the divisor may be a value used to compress the convolution operation to a target value range, and the divisor is determined by the element values of the convolution kernel 2042, the target range, and the range of the input value array performing the convolution operation .

Referring to FIG. 24 and FIG. 25 for details, the architecture program writes the coefficients of the convolution kernel 2042 into the DRAM 122 . As far as a preferred embodiment is concerned, all characters on each column of the nine consecutive columns (the number of elements in the convolution kernel 2402) of the DRAM 122 will use different elements of the convolution kernel 2402 to be listed as its Major order to be written. That is to say, as shown in the figure, each character in the same column is written with the first coefficient C0, 0; the next column is written with the second coefficient C0, 1; and the next column is written with the third coefficient C0 , 2 is written; the next column is written with the fourth coefficient C1, 0; and so on, until each character in the ninth column is written with the ninth coefficient C2, 2. In order to perform a convolution operation on the data matrix 2406 of the data block divided by the data array 2404, the neural processing unit 126 will repeatedly read the nine columns loaded with the coefficients of the convolution kernel 2042 in the data random access memory 122 in order, and this part will be described later Sections, especially the part corresponding to Fig. 26A, will be described in more detail.

Referring to FIG. 24 and FIG. 25 for details, the architecture program writes the values of the data matrix 2406 into the weight RAM 124 . When the NNU program performs the convolution operation, it will write the result array back to the weight RAM 124 . For a preferred embodiment, the architecture program will write the first data matrix 2406 into the weight random access memory 124 and start the neural network unit 121. When the neural network unit 121 is convolving the first data matrix 2406 with When the core 2402 performs the convolution operation, the architecture program will write the second data matrix 2406 into the weight random access memory 124, so that after the neural network unit 121 completes the convolution operation of the first data matrix 2406, it can start to execute the second data matrix 2406. The convolution operation of the data matrix 2406 will be described in more detail later corresponding to FIG. 25 . In this way, the architecture program will shuttle between the two areas of the weight RAM 124 to ensure that the NNU 121 is fully utilized. Thus, the example of FIG. 24 shows a first data matrix 2406A and a second data matrix 2406B, the first data matrix 2406A being the first data block corresponding to columns 0 to 399 occupying the weighted random access memory 124, and the second data matrix 2406B Matrix 2406B is the second data block corresponding to columns 500 to 899 in Occupancy Weight RAM 124 . In addition, as shown in the figure, the neural network unit 121 will write the results of the convolution operation back to the columns 900-1299 and columns 1300-1699 of the weight RAM 124, and then the architecture program will read from the weight RAM 124 Take these results. The data values loaded into the data matrix 2406 of the weight random access memory 124 are marked as "Dx, y", where "x" is the column number of the weight random access memory 124, and "y" is the text of the weight random access memory, or Called the number of rows. For example, the data word 511 in column 399 is denoted D399, 511 in FIG.

FIG. 25 is a flow chart showing that the processor 100 of FIG. 1 executes the architecture program to use the neural network unit 121 to perform the convolution operation of the convolution kernel 2042 on the data array 2404 of FIG. 24 . The process starts at step 2502.

In step 2502 , the processor 100 , that is, the processor 100 executing the structured program, writes the convolution kernel 2402 in FIG. 24 into the DRAM 122 in the manner shown and described in FIG. 24 . Additionally, the architecture program initializes the variable N to a value of 1. The variable N indicates the data block in the data array 2404 that the NNU 121 is processing. Additionally, the framework program initializes the variable NUM_CHUNKS to the value 20. Then the flow goes to step 2504 .

In step 2504 , as shown in FIG. 24 , the processor 100 writes the data matrix 2406 of the data block 1 into the weight random access memory 124 (such as the data matrix 2406A of the data block 1 ). Then the flow goes to step 2506 .

In step 2506 , the processor 100 writes the convolution program into the program memory 129 of the neural network unit 121 using an MTNN instruction 1400 specifying the function 1432 to write into the program memory 129 . The processor 100 then uses a designated function 1432 to start executing the program's MTNN instruction 1400 to start the neural network unit convolution program. An example of an NNU convolution procedure is described in more detail corresponding to FIG. 26A . Then the flow goes to step 2508 .

In decision step 2508, the framework program checks to see if the value of variable N is less than NUM_CHUNKS. If yes, the process will go to step 2512; otherwise, go to step 2514.

In step 2512, as shown in FIG. 24, the processor 100 writes the data matrix 2406 of data block N+1 into the weight random access memory 124 (eg, the data matrix 2406B of data block 2). Therefore, when the neural network unit 121 is performing a convolution operation on the current data block, the architecture program can write the data matrix 2406 of the next data block into the weight random access memory 124, so that after completing the convolution of the current data block After the operation, that is, after the weight random access memory 124 is written, the neural network unit 121 can immediately start to perform the convolution operation on the next data block.

In step 2514, the processor 100 confirms whether the executing NNU program (executed from step 2506 for block 1 and executed from step 2518 for blocks 2-20) has completed execution. For a preferred embodiment, the processor 100 reads the state register 127 of the neural network unit 121 to confirm whether the execution has been completed by executing the MFNN instruction 1500 . In another embodiment, the NNU 121 generates an interrupt indicating that the convolution procedure has been completed. Flow then proceeds to decision step 2516.

In decision step 2516, the framework program checks to see if the value of variable N is less than NUM_CHUNKS. If yes, the process proceeds to step 2518; otherwise, proceeds to step 2522.

In step 2518, the processor 100 updates the convolution program to execute on the data block N+1. More precisely, processor 100 will update the column value of the initialize NPU instruction corresponding to address 0 in weight RAM 124 to the first column of data matrix 2406 (e.g., to column 0 of data matrix 2406A or column 500 of data matrix 2406B), and the output column is updated (eg, to column 900 or 1300). Then the processor 100 starts to execute the updated neural network unit convolution procedure. Then the process proceeds to step 2522 .

In step 2522 , the processor 100 reads the execution result of the NNU convolution procedure of the data block N from the weight RAM 124 . Flow then proceeds to decision step 2524.

In decision step 2524, the framework program checks whether the value of variable N is less than NUM_CHUNKS. If yes, the process proceeds to step 2526; otherwise, it terminates.

In step 2526, the framework program increments the value of N by one. The process then returns to decision step 2508.

FIG. 26A is a program listing of the neural network unit program. The neural network unit program uses the convolution kernel 2402 of FIG. This program loops the instruction loop composed of the instructions of addresses 1 to 9 for a certain number of times. The initialize NPU instruction at address 0 specifies the number of times each NPU 126 executes this instruction loop, and in the example of FIG. 26A has a loop count value of 400, corresponding to the number of columns in the data matrix 2406 of FIG. 24 , The loop instruction (located at address 10) at the end of the loop will decrement the current loop count value, and if the result is non-zero, it will return to the top of the instruction loop (that is, return to the instruction at address 1). The initialize NPU instruction also clears accumulator 202 to zero. For a preferred embodiment, the loop instruction at address 10 also clears accumulator 202 to zero. In addition, the accumulator 202 can also be cleared to zero as the aforementioned multiply-accumulate instruction at address 1.

For each execution of the instruction loop in the program, the 512 neural processing units 126 will simultaneously perform convolution operations of 512 3x3 convolution kernels and 512 corresponding 3x3 sub-matrices of the data matrix 2406 . The convolution operation is the sum of nine products calculated from the elements of the convolution kernel 2042 and the corresponding elements in the corresponding sub-matrix. In the embodiment of FIG. 26A, the origin (central element) of each of these 512 corresponding 3x3 sub-matrices is the data literal Dx+1,y+1 in FIG. 24, where y (row number) is the neural processing unit 126 numbers, and x (column number) is the column number read by the multiplication and accumulation instruction of address 1 in the program of FIG. Instructions perform initialization processing, are also incremented when the multiply-accumulate instructions at addresses 3 and 5 are executed, and are also updated by the decrement instruction at address 9). Thus, in each cycle of the program, the 512 neural processing units 126 will calculate 512 convolution operations and write the results of the 512 convolution operations back to the instruction queue of the weight random access memory 124 . In this paper, the edge handling (edge handling) is omitted to simplify the description, but it should be noted that using the collective rotation characteristics of these neural processing units 126 will result in a large number of data matrix 2406 (for the image processor, the data matrix of the image) Two rows in the row data generate wrapping from one vertical edge to the other vertical edge (for example, from the left edge to the right edge, and vice versa). The instruction loop is now explained.

Address 1 is a multiply-accumulate instruction that specifies column 0 of data RAM 122 and implicitly utilizes the current column of weight RAM 124, which is preferably loaded in sequencer 128 (and determined by 0 to initialize it to zero to perform the operation passed in the first instruction cycle). That is to say, the instruction at address 1 causes each neural processing unit 126 to read its corresponding text from column 0 of the data random access memory 122, read its corresponding text from column 124 of the current weight random access memory, And perform a multiply-accumulate operation on the two literals. Thus, for example, the NPU 5 multiplies C0,0 by Dx,5 (where "x" is the current weight RAM 124 column), adds the result to the accumulator 202 value 217, and writes the total to back to the accumulator 202.

Address 2 is a multiply-accumulate instruction, which specifies that the row of the DRAM 122 is incremented (ie, incremented to 1), and then the row is read from the incremented address of the DRAM 122 . This instruction will also specify to rotate the value in the multitasking register 705 of each NPU 126 to the adjacent NPU 126, in this example, to read from the weight random access memory 124 in response to the instruction at address 1 Take the 2406 columns of the data matrix. In the embodiments of FIG. 24 to FIG. 26 , these NPUs 126 are used to rotate the value of the multitasking register 705 to the left, that is, to rotate from NPU J to NPU J-1, instead of as described above Fig. 3, Fig. 7 and Fig. 19 are rotated from NPU J to NPU J+1. It should be noted that in the embodiment where the neural processing unit 126 is rotated to the right, the architecture program will write the value of the convolution kernel 2042 into the data random access memory 122 in a different order (for example, rotate around its central row) to achieve similarity. The purpose of the convolution result. In addition, the architecture program can perform additional kernel preprocessing (such as transposition) when needed. Also, the instruction specifies a count value of 2. Thus, the instruction at address 2 causes each NPU 126 to read its corresponding word from column 1 of the DRAM 122, receive the rotated word into the multitasking register 705, and execute Multiply-accumulate operation. Since the count value is 2, this command also causes each NPU 126 to repeat the aforementioned operations. That is to say, the sequencer 128 will increment the column address 123 of the DRAM 122 (that is, increase to 2), and each NPU 126 will read its corresponding The text and the rotated text are received into the multitasking register 705, and a multiply-accumulate operation is performed on the two texts. Thus, for example, assuming that the current weight random access memory 124 is listed as 27, after executing the instruction at address 2, the neural processing unit 5 will combine the product of C0,1 and D27,6 with the product of C0,2 and D27,7 The multiplication is accumulated to its accumulator 202 . In this way, after the instruction of address 1 and address 2 is completed, the product of C0, 0 and D27, 5, the product of C0, 1 and D27, 6 and C0, 2 and D27, 7 will be accumulated in the accumulator 202, and all other Accumulated value from previously passed instruction cycle.

The operations performed by the instructions of addresses 3 and 4 are similar to the instructions of addresses 1 and 2, using the effect of incrementing the pointer in the column of the weight random access memory 124, these instructions will perform operations on the next column of the weight random access memory 124, and these The instruction will perform operations on the next three columns of the DRAM 122 , ie, columns 3 to 5 . That is to say, taking neural processing unit 5 as an example, after completing the instructions of addresses 1 to 4, the product of C0,0 and D27,5, the product of C0,1 and D27,6, the product of C0,2 and D27,7 , the product of C1, 0 and D28, 5, the product of C1, 1 and D28, 6, and the product of C1, 2 and D28, 7 will be accumulated in the accumulator 202, adding all other accumulated values from the previously delivered instruction cycle .

The instructions at addresses 5 and 6 perform operations similar to the instructions at addresses 3 and 4, and these instructions will affect the next column of weight RAM 124 and the next three columns of data RAM 122, columns 6 to 8 , to perform the operation. That is to say, taking neural processing unit 5 as an example, after completing the instructions of addresses 1 to 6, the product of C0, 0 and D27, 5, the product of C0, 1 and D27, 6, the product of C0, 2 and D27, 7 , the product of C1,0 and D28,5, the product of C1,1 and D28,6, the product of C1,2 and D28,7, the product of C2,0 and D29,5, the product of C2,1 and D29,6, and The product of C2,2 and D29,7 is accumulated in accumulator 202, adding all other accumulated values from previously passed instruction cycles. That is to say, after completing the instructions of addresses 1 to 6, assuming that when the instruction loop starts, the weight RAM 124 is listed as 27, taking the neural processing unit 5 as an example, the following 3x3 sub-matrix will be processed by the convolution kernel 2042 Convolution operation:

Generally speaking, after completing the instructions of addresses 1 to 6, the 512 neural processing units 126 have used the convolution kernel 2042 to perform convolution operations on the following 3x3 sub-matrices:

Where r is the column address value of the weight RAM 124 at the beginning of the instruction loop, and n is the number of the NPU 126 .

The instruction at address 7 will pass the value 217 of the accumulator 202 through the activation function unit 121 . The pass function passes a text whose size (in bits) is equivalent to the text read by the data RAM 122 and the weight RAM 124 (16 bits in this example). As far as a preferred embodiment is concerned, the user can specify the output format, for example, how many digits in the output digits are fractional digits, which will be described in more detail in subsequent chapters. Alternatively, instead of specifying a transfer enable function, the designation may specify a divide enable function that divides the accumulator 202 value 217 by a divisor, as described herein corresponding to FIGS. 29A and 30 , for example using FIG. One of the 30 "dividers" 3014/3016. For example, for a convolution kernel 2042 with a coefficient, such as the aforementioned Gaussian blur kernel with a coefficient of one-sixteenth, the instruction at address 7 would specify a division activation function (for example, divide by 16) instead of Specify a transfer function. In addition, before the architecture program writes the coefficients of the convolution kernels into the DRAM 122, the operation of dividing the coefficients of the convolution kernels 2042 by 16 is performed, and accordingly the position of the binary point of the value of the convolution kernels 2042 is adjusted, For example, use the data binary decimal point 2922 of FIG. 29 as described below.

The instruction at address 8 will write the output of the activation function unit 212 into the column of the weight RAM 124 specified by the current value of the output column register. This current value is initialized by the instruction at address 0, and is incremented each pass through the loop by the increment pointer within the instruction.

24-26 with a 3x3 convolution kernel 2402 example, the neural processing unit 126 will read the weight random access memory 124 to read a column of the data matrix 2406 approximately every three clock cycles, and approximately every Twelve time-frequency cycles will write the convolution kernel result matrix into the weight random access memory 124 . In addition, assuming that in an embodiment, there are write and read buffers such as the buffer 1704 in FIG. 124 for reading and writing, and the buffer 1704 performs a reading and writing operation on the weight random access memory about every sixteen time-frequency cycles, so as to read the data matrix and write the convolution kernel result matrix respectively. Therefore, approximately half of the bandwidth of the WRAM 124 is consumed by the convolution kernel operations performed by the NNU 121 in a hybrid manner. This example includes a 3x3 convolution kernel 2042, but the present invention is not limited thereto, other sizes of convolution kernels, such as 2x2, 4x4, 5x5, 6x6, 7x7, 8x8, etc., are also applicable to different neural network unit programs . In the case of using a larger convolution kernel, because the rotated version of the multiply-accumulate instruction (such as the instructions of addresses 2, 4, and 6 in Figure 26A, which is required for a larger convolution kernel) has a larger count value Therefore, the proportion of time for the neural processing unit 126 to read the weight random access memory 124 will be reduced, and therefore, the bandwidth usage ratio of the weight random access memory 124 will also be reduced.

Additionally, the architecture program allows the NNU program to overwrite columns of the input data matrix 2406 that are no longer needed, rather than writing the results of the convolution operations back to different columns of the weight RAM 124 (e.g., columns 900-1299 and 1300-1699). For example, for a 3x3 convolution kernel, the architecture program may write the data matrix 2406 into columns 2-401 of the weight RAM 124 instead of columns 0-399, while the NPU program writes The result of the convolution operation is written starting from column 0 of the weight random access memory 124 , and the column number is incremented each time the instruction cycle is passed through. This way, the NNU program only overwrites columns that are no longer needed. For example, after the first pass through the instruction loop (or more precisely, after the instruction at address 1 is executed which loads column 0 of the weight RAM 124), the data in column 0 may be overwritten, however, The data in columns 1-3 needs to be left for the operation of the second pass through the instruction cycle and cannot be overwritten; similarly, after the second pass through the instruction cycle, the data in column 1 can be overwritten, however, columns 2-4 The data needs to be left for the operation of the third pass through the instruction loop and cannot be overwritten; and so on. In this embodiment, the height of each data matrix 2406 (data block) can be increased (eg, 800 columns), so fewer data blocks can be used.

In addition, the architecture program can cause the NNU program to write the result of the convolution operation back to the DRAM 122 column above the convolution kernel 2402 (eg, above column 8), instead of writing the result of the convolution operation back to the weight random access memory 122 column. To access the memory 124, when the neural network unit 121 writes the result, the architecture program can read the result from the data random access memory 122 (for example, use the last written column 2606 address of the data random access memory 122 in FIG. 26). This configuration is suitable for embodiments with a single-port weight RAM 124 and a dual-port data RAM.

According to the operation of the neural network unit 121 in the embodiment of FIG. 24 to FIG. 26A, it can be found that each execution of the program in FIG. The calculation requires about 100,000 time-frequency cycles, which is significantly less than the number of time-frequency cycles required to perform the same task in a conventional manner.

FIG. 26B is a block diagram showing an embodiment of some fields of the control register 127 of the NNU 121 of FIG. 1 . This state buffer 127 includes a field 2602, indicating the address of the column in the weight random access memory 124 that was recently written by the neural processing unit 126; a field 2604 indicating the address of the column in the weight RAM 124 that was most recently read by the neural processing unit 126; and a field 2608 indicating the address of the column in the data random access memory 122 that was most recently The address of the column to read. In this way, the architecture program executed on the processor 100 can confirm the processing progress of the neural network unit 121 when reading and/or writing data to the data RAM 122 and/or the weight RAM 124 . Utilizing this capability, plus overwriting the input data matrix as described above (or writing the result to the DRAM 122 as described above), as described in the following example, the data array 2404 of FIG. 24 can be regarded as 5 chunks of 512x1600 to execute instead of 20 chunks of 512x400. Processor 100 writes the first 512x1600 block of data starting from column 2 of WRAM 124 and causes the NNU program to start (this program has a loop count of 1600 and loads WRAM 124 124 output columns initialized to 0). When the neural network unit 121 executes the neural network unit program, the processor 100 will monitor the output position/address of the weight random access memory 124, so as to (1) (use the MFNN instruction 1500) read the output data in the weight random access memory 124 The columns of the valid convolution operation results written by the NNU 121 (starting at column 0); and (2) overwriting the second 512x1600 data matrix 2406 (starting at column 2) over the valid volumes that have already been read In this way, when the NNU 121 completes the NNU program for the first 512x1600 data block, the processor 100 can immediately update the NNU program and restart the NNU program to execute on the second 512x1600 data block if necessary. data block. This procedure will be repeated three times to execute the remaining three 512x1600 data blocks, so that the neural network unit 121 can be fully utilized.

In one embodiment, the enabling function unit 212 has the ability to effectively perform an effective division operation on the value 217 of the accumulator 202. This part will be described in more detail in subsequent chapters, especially corresponding to FIG. 29A, FIG. 29B and FIG. 30. illustrate. For example, an enable function NNU instruction that divides the accumulator 202 value by 16 may be used for the Gaussian blur matrix described below.

The convolution kernel 2402 used in the example of FIG. 24 is a small static convolution kernel applied to the entire data matrix 2404, but the present invention is not limited thereto. This convolution kernel can also be a large matrix with specific The weights correspond to different data values of the data array 2404, such as convolution kernels commonly found in convolutional neural networks. When the neural network unit 121 is used in this way, the architecture program will exchange the positions of the data matrix and the convolution kernel, that is, the data matrix is placed in the data random access memory 122 and the convolution kernel is placed in the weight random access memory. In the memory 124, the number of rows required to execute the neural network unit program will be relatively small.

FIG. 27 is a block diagram showing an example of the weighted random access memory 124 in FIG. 1 filled with input data for pooling operation performed by the neural network unit 121 in FIG. 1 . The common-source operation is performed by the common-source layer of the artificial neural network. By obtaining the sub-region or sub-matrix of the input matrix and calculating the maximum or average value of the sub-matrix as the result matrix, that is, the common-source matrix, the input data matrix ( Such as image or image after convolution) size (dimension). In the examples of FIG. 27 and FIG. 28 , the common source operation calculates the maximum value of each sub-matrix. Common source operations are especially useful for artificial neural networks that perform object classification or detection, for example. Generally speaking, the common-source operation can actually reduce the input matrix by a factor of the detected sub-matrix elements, in particular, each dimension of the input matrix can be reduced by the number of elements in the corresponding dimension of the sub-matrix. In the example of FIG. 27 , the input data is a 512x1600 matrix of wide text (eg, 16 bits), stored in columns 0 to 1599 of the WRAM 124 . In Figure 27, these characters are marked by their column and row positions, for example, the characters in column 0 and row 0 are marked as D0, 0; the characters in column 0 and row 1 are marked as D0, 1; the characters in column 0 and row 2 are marked as D0, 1 It is marked as D0, 2; and so on, the text located in row 511 of column 0 is marked as D0, 511. Similarly, the text in column 1, row 0 is marked as D1, 0; the text in column 1, row 1 is marked as D1, 1; the text in column 1, row 2 is marked as D1, 2; and so on, the text in column 1, row 511 The text in row 1599 is marked as D1, 511; and so on, the text in column 1599, line 0 is marked as D1599, 0; the text in column 1599, line 1 is marked as D1599, 1, and the text in column 1599, line 2 is marked as D1599, 2 ; By analogy, the text located in row 1599 and line 511 is marked as D1599, 511.

FIG. 28 is a program listing of the NNU program that performs the common-source operation of the input data matrix of FIG. 27 and writes it back to the weight RAM 124. In the example of FIG. 28, the common source operation calculates the maximum value of each 4x4 sub-matrix in the input data matrix. This program executes the instruction loop consisting of instructions 1 to 10 multiple times. The initialize NPU instruction at address 0 specifies the number of times each NPU 126 executes an instruction loop, the loop count value being 400 in the example of FIG. The current loop count is decremented, and if the resulting result is non-zero, it returns to the top of the instruction loop (ie, to the instruction at address 1). The input data matrix in the weight random access memory 124 will be regarded by the neural network unit program as 400 mutually exclusive groups consisting of four adjacent columns, namely columns 0-3, columns 4-7, columns 8- 11. And so on until columns 1596-1599. Each group of four adjacent columns consists of 128 4x4 sub-matrices formed by the elements at the intersection of the four columns of this group and four adjacent rows, the adjacent rows That is, rows 0-3, rows 4-7, rows 8-11, and so on up to rows 508-511. Among these 512 neural processing units 126, every fourth neural processing unit 126 (a total of 128) that calculates as a group of four will perform a common source operation on a corresponding 4x4 sub-matrix, while the other three neural processing units 126 is not used. More precisely, NPUs 0, 4, 8, and so on up to NPU 508 perform common-source operations on their corresponding 4x4 sub-matrix whose leftmost row numbers correspond to Neural processing unit number, and the lower column corresponds to the column value of the current weight random access memory 124, this value will be initialized to zero by the initialization command at address 0 and will increase by 4 after repeating each command loop, this part will be described in subsequent chapters There will be more detailed instructions. These 400 repetitions of instruction cycles correspond to the number of 4x4 sub-matrix groups in the input data matrix of FIG. 27 (ie, the input data matrix has 1600 columns divided by 4). The initialize NPU instruction also clears the accumulator 202 to zero. For a preferred embodiment, the loop instruction at address 11 also clears accumulator 202 to zero. In addition, the maxwacc instruction at address 1 specifies that the accumulator 202 be cleared to zero.

The 128 used NPUs 126 simultaneously perform 128 common-source operations on 128 individual 4x4 sub-matrices in the current four-column group of the input data matrix each time an instruction loop of the program is executed. Further, the common source operation identifies the maximum value element among the 16 elements of the 4x4 sub-matrix. In the embodiment of FIG. 28, for each NPU y of the 128 used NPUs 126, the lower left element of the 4x4 sub-matrix is the element Dx,y in FIG. 27, where x is the column number of the current weight random access memory 124 when the instruction loop begins, and this column data is read by the maxwacc instruction of address 1 in the program of Figure 28 (this column number will also be initialized by the initialization neural processing unit instruction of address 0 , and is incremented each time the maxwacc instructions at addresses 3, 5, and 7 are executed). Therefore, for each cycle of this program, the 128 used NPUs 126 will write the maximum value elements of the corresponding 128 4x4 sub-matrices of the current column group back to the weight random access memory 124 Specify columns. The following describes this instruction cycle.

The maxwacc instruction at address 1 will implicitly use the current weight RAM 124 column, which is preferably loaded in the sequencer 128 (and initialized to zero by the instruction at address 0 to perform the first pass through the instruction loop operation). The instruction at address 1 causes each NPU 126 to read its corresponding text from the current column of the weight RAM 124, compare the text to the accumulator 202 value 217, and calculate the maximum of the two values stored in the accumulator 202. So, for example, the NPU 8 would identify the maximum value of the accumulator 202 value 217 and the data word Dx,8 (where "x" is the current weight RAM 124 column) and write it back to the accumulator 202 .

Address 2 is a maxwacc instruction, which specifies to rotate the value in the multitasking register 705 of each neural processing unit 126 to be adjacent to the neural processing unit 126. A column of input data array values read by the memory 124 is fetched. In the embodiment shown in FIGS. 27 to 28, the neural processing unit 126 is used to rotate the value of the multiplexer 705 to the left, that is, from the neural processing unit J to the neural processing unit J-1. Figure 26 section. In addition, this command specifies a count value of 3. In this way, the instruction at address 2 causes each neural processing unit 126 to receive the rotated text into the multitasking register 705 and confirm the maximum value between the rotated text and the value of the accumulator 202 , and then repeat this operation twice. That is to say, each neural processing unit 126 performs three operations of receiving the rotated text into the multitasking register 705 and confirming the maximum value of the rotated text and the value of the accumulator 202 . Thus, for example, assuming that the current weight random access memory 124 is listed as 36 when starting the instruction cycle, taking the neural processing unit 8 as an example, after executing the instructions at addresses 1 and 2, the neural processing unit 8 will be in its The accumulator 202 stores the maximum value in the accumulator 202 and the four weight RAM 124 words D36,8, D36,9, D36,10 and D36,11 at the beginning of the cycle.

The operations performed by the maxwacc instructions at addresses 3 and 4 are similar to the instructions at address 1. Using the effect of incrementing pointers in the columns of the weight random access memory 124, the instructions at addresses 3 and 4 will be executed on the next column of the weight random access memory 124. . That is to say, assuming that the current weight RAM 124 column is 36 at the beginning of the instruction cycle, and taking the neural processing unit 8 as an example, after completing the instructions of addresses 1 to 4, the neural processing unit 8 will store 36 in its accumulator 202 At the beginning of the storage cycle, accumulator 202 and eight weight RAM 124 words D36,8, D36,9, D36,10, D36,11, D37,8, D37,9, D37,10 and D37,11 maximum value.

The operations performed by the maxwacc instructions at addresses 5 to 8 are similar to the instructions at addresses 1 to 4, and the instructions at addresses 5 to 8 will be executed on the next two columns of the WRAM 124 . That is to say, assuming that the current weight RAM 124 column is 36 at the beginning of the instruction cycle, and taking the neural processing unit 8 as an example, after completing the instructions of addresses 1 to 8, the neural processing unit 8 will store 36 in its accumulator 202 Accumulator 202 and sixteen weight random access memory 124 words D36,8, D36,9, D36,10, D36,11, D37,8, D37,9, D37,10, D37,11, The maximum value among D38,8, D38,9, D38,10, D38,11, D39,8, D39,9, D39,10 and D39,11. That is to say, assuming that the current weight RAM 124 column is 36 at the beginning of the instruction cycle, taking the neural processing unit 8 as an example, after completing the instructions of addresses 1 to 8, the neural processing unit 8 will complete the confirmation of the following 4x4 sub-matrix The maximum value of:

Basically, after completing the instructions for addresses 1 to 8, each of the 128 used NPUs 126 will complete identifying the maximum value of the following 4x4 sub-matrix:

Where r is the column address value of the current weight random access memory 124 at the beginning of the instruction loop, and n is the number of the neural processing unit 126 .

The instruction at address 9 will pass the value 217 of the accumulator 202 through the activation function unit 212 . The transfer function transfers a text whose size (in bits) is equivalent to the text read from the weight RAM 124 (16 bits in this example). As far as a preferred embodiment is concerned, the user can specify the output format, for example, how many bits in the output bits are fractional bits, which will be described in more detail in subsequent chapters.

The instruction at address 10 will write the accumulator 202 value 217 into the column of weight RAM 124 specified by the current value of the output column buffer, which will be initialized by the instruction at address 0, using the The increment pointer increments this value after each pass through the loop. Further, the instruction at address 10 will write a wide word (eg, 16 bits) from the accumulator 202 into the weight RAM 124 . As far as a preferred embodiment is concerned, the command will write the 16 bits according to the output binary point 2916, which will be described in more detail below corresponding to FIG. 29A and FIG. 29B.

As mentioned above, one iteration of the instruction loop writes the columns of the weight RAM 124 to include holes with invalid values. That is, wide literals 1 to 3, 5 to 7, 9 to 11, and so on for result 133, up to wide literals 509 to 511 are invalid or unused. In one embodiment, enable function unit 212 includes a multiplexer to enable combining results into adjacent words of a column buffer, such as column buffer 1104 of FIG. 11 , to write back to the output weight RAM 124 column. For a preferred embodiment, the enable function instruction specifies the number of words in each hole, and the number of words in the hole controls the multiplexer merge result. In one embodiment, the number of holes can be specified as a value from 2 to 6 to combine outputs of 3x3, 4x4, 5x5, 6x6 or 7x7 sub-matrices from a common source. In addition, the architectural program executing on the processor 100 reads the resulting sparse (i.e., with holes) result columns from the weight RAM 124 and utilizes other execution units 112, such as media units using architectural coalescing instructions, such as x86 SIMD Extensions (SSE) instructions, which perform merge functions. In a manner similar to that described above concurrently and taking advantage of the hybrid nature of the NNU 121, an architectural program executing on the processor 100 can read the state register 127 to monitor the most recently written column of the WRAM 124 (e.g., FIG. 26B field 2602) to read a resulting sparse result column, combine it and write it back to the same column in the weight RAM 124, and thus complete the preparation as an input data matrix for the next step of the neural network Layers are used, such as convolutional layers or traditional neural network layers (aka multiply-accumulate layers). In addition, the embodiments described herein perform common-source operations with 4x4 sub-matrices, but the present invention is not limited thereto. The neural network unit program of FIG. Or 7x7, performing common-source operations.

As can be seen previously, the number of resultant columns written to the weight random access memory 124 is one quarter the number of columns of the input data matrix. Finally, the DRAM 122 is not used in this example. However, the data RAM 122 can also be used instead of the weight RAM 124 to perform the common source operation.

In the embodiments of FIG. 27 and FIG. 28 , the common source operation calculates the maximum value of the sub-region. However, the program of FIG. 28 can be adjusted to calculate subregion averages by replacing the maxwacc instruction with the sumwacc instruction (summing the weight literal and accumulator 202 value 217) and modifying the start function instruction at address 9 To divide the accumulated result by the number of elements in each sub-region (preferably by reciprocal multiplication as described below), sixteen in this example.

It can be found from the operation of the neural network unit 121 according to FIG. 27 and FIG. 28 that each execution of the program in FIG. 28 requires about 6000 time-frequency cycles to perform a common-source operation on the entire 512x1600 data matrix shown in FIG. 27 . The number of time-frequency cycles used by the calculation is significantly less than the number of time-frequency cycles required to perform similar tasks in traditional methods.

In addition, the architecture program can make the NNU program write the result of the common source operation back to the DRAM 122 column instead of writing the result back to the weight RAM 124, when the NNU 121 writes the result to the DRAM When accessing memory 122 (eg, using the address most recently written to column 2606 by DRAM 122 in FIG. 26B ), the fabric program reads the result from DRAM 122 . This configuration is suitable for an embodiment having a single-port weight RAM 124 and a dual-port data RAM 122 .

Fixed-point arithmetic with user-supplied binary point, full-precision fixed-point accumulation, user-specified reciprocal value, random rounding of accumulator value, and selectable start/output functions

Generally speaking, hardware units for performing arithmetic operations in a digital computing device can be generally divided into "integer" units and "floating point" units depending on whether the object of the arithmetic operation is an integer or a floating point number. Floating-point numbers have a magnitude (or mantissa), an exponent, and usually a sign. The exponent is a pointer to the position of the radix (radix) point (usually the binary point) relative to the value. Integers, by contrast, do not have an exponent, but only a value, and usually a sign. The floating-point unit allows the programmer to get the number he or she needs to work with from a very wide range of different values, and the hardware takes care of adjusting the exponent of that number when needed, without the programmer having to deal with it. As an example, suppose two floating point numbers 0.111x 10 ²⁹ are multiplied by 0.81x 10 ³¹ . (Although floating-point units usually work with base-2 floating-point numbers, this example uses decimal fractions, or base-10 floating-point numbers.) The floating-point unit automatically takes care of multiplying the mantissa, adding the exponents, and then The results were then normalized to a value of .8911 x 10 ⁵⁹ . In another example, assume that the same two floating point numbers are added. The floating point unit automatically takes care of aligning the binary point of the mantissa before adding to produce a total of .81111x 10 ³¹ values.

However, it is well known that such complex operations will increase the size of the floating point unit, increase power consumption, increase the number of time-frequency cycles required per instruction, and/or prolong the cycle time. For this reason, many devices, such as embedded processors, microcontrollers, and relatively low-cost and/or low-power microprocessors, do not have floating point units. As can be seen from the preceding examples, the complex structure of the floating-point unit contains the logic to perform exponent calculations associated with floating-point addition and multiplication/division Adder for subtracting operand exponents to confirm binary point alignment offset for floating-point addition), including offset for binary point alignment of mantissa in floating-point addition, including for floating-point addition The offset to normalize the result. In addition, the process often requires logic to perform rounding of floating-point results, perform conversions between integer formats and floating-point formats, and between different floating-point formats (such as extended precision, double precision, single precision, half precision) logic for , detectors for leading zeros and ones, and logic for handling special floating-point numbers, such as abnormal values, non-numeric values, and infinity values.

In addition, the verification of the correctness of the floating point unit will greatly increase its complexity due to the increase of the numerical space to be verified in the design, which will prolong the product development cycle and time to market. In addition, as mentioned above, floating-point arithmetic operations require separate storage and use of the mantissa and exponent fields of each floating-point number used for calculation, which increases the required storage space and/or the number of integers stored in a given storage space. In this case, the accuracy is reduced. Many of these disadvantages can be avoided by performing arithmetic operations on the integer unit.

Programmers often need to write programs that deal with decimals, which are values that are not complete numbers. Such a program may need to be executed on a processor that does not have a floating point unit, or the processor may have a floating point unit, but it may be faster for the processor's integer unit to execute integer instructions. To take advantage of the performance advantages of integer processors, programmers use known fixed-point arithmetic operations on fixed-point numbers. Such programs would include instructions that execute on integer units to process integers or integer data. The software knows that the data is fractional, and the software includes instructions to perform operations on integer data that deal with issues where the data is actually fractional, such as aligning offsetters. Basically, fixed-point software manually performs some or all of the functions that a floating-point unit can.

In this context, a "fixed-point" number (or value or operand or input or output) is a number whose storage bits are understood to contain bits to represent the fractional part of the fixed-point number, referred to herein as the "fractional bit". The storage bits for a fixed-point number are contained within a memory or register, such as an 8-bit or 16-bit word within a memory or register. In addition, all of the fixed-point storage bits are used to represent a value, and in some cases one of the bits is used to represent the sign, but none of the fixed-point storage bits are used to represent the exponent of the number. In addition, the number of decimal places or the position of the binary decimal point of this fixed-point number is specified in a storage space different from the storage position of the fixed-point number, and the number of decimal places or the position of the binary decimal point is indicated in a shared or common way, shared among a A set of fixed-point numbers that contains this fixed-point number, such as an input operand, an accumulated value, or a set of output results from an array of processing cells.

In the embodiments described here, the ALU is an integer unit, however, the enabling function unit includes floating point arithmetic hardware assistance or acceleration. This makes the ALU part smaller and faster, facilitating the use of more ALUs on a given chip space. This also means that more neurons can be arranged on a unit chip space, which is particularly beneficial to neural network units.

In addition, compared to each floating-point number requiring an exponent storage bit, the fixed-point number in the embodiment described herein expresses the number of decimal storage bits in the entire set of numbers with a pointer, however, this pointer resides in a single , Shared storage space and broadly point out all the numbers of the entire set, such as the input set of a series of operations, the set of cumulative numbers of a series of operations, the output set, and the number of decimal places. For a preferred embodiment, the user of the NNU can specify the number of decimal places for the set of numbers. Thus, it is understood that while in many contexts (such as mathematics in general) the term "integer" refers to a signed whole number, that is, a number without a fractional part, in the context of this paper, " The term "integer" may mean a number with a fractional part. In addition, in the context of this article, the term "integer" is used to distinguish it from floating-point numbers. For floating-point numbers, some bits in their respective storage spaces are used to express the exponent of the floating-point number. Similarly, integer arithmetic operations, such as integer multiplication or addition or comparison operations performed by the integer unit, assume that there are no exponents in the operands, so integer elements of the integer unit, such as integer multipliers, integer adders, and integer comparators, There is no need to include logic to handle exponents, such as shifting the mantissa to align the binary point for addition or comparison operations, or adding exponents for multiplication operations.

Additionally, embodiments described herein include a large hardware integer accumulator to accumulate a large series of integer operations (eg, 1000 multiply-accumulate operations) without loss of precision. This avoids the neural network unit from dealing with floating point numbers, while maintaining the full precision of the accumulator without saturating it or producing inaccurate results due to overflow. Once the series of integer operations have summed a result into the full-precision accumulator, the fixed-point hardware assist performs the necessary scaling and saturation operations to utilize the user-specified pointer to the number of decimal places in the accumulated value and the required number of decimal places in the output value. Quantity, convert this full-precision accumulative value into an output value, this part will be described in more detail in subsequent chapters.

When the accumulated value needs to be compressed from full precision form for use as an input to the activation function or for transfer, in a preferred embodiment, the activation function unit may optionally perform random rounding on the accumulated value, This part will be described in more detail in subsequent chapters. Finally, the neural processing unit may optionally be instructed to use different activation functions and/or output accumulated values in many different forms, depending on the different requirements of a given layer of the neural network.

FIG. 29A is a block diagram showing an embodiment of the control register 127 of FIG. 1 . The control register 127 may include a plurality of control registers 127 . As shown, the control register 127 includes the following fields: Configuration 2902, Signed Data 2912, Signed Weight 2914, Data Binary Point 2922, Weight Binary Point 2924, ALU Function 2926, Rounding Control 2932, Enable Function 2934 , Reciprocal 2942 , Offset 2944 , Output Random Access Memory 2952 , Output Binary Point 2954 , and Output Command 2956 . The value of the control register 127 can be written by using the MTNN command 1400 and the command of the NNU program, such as the start command.

The configuration 2902 value specifies whether the neural network unit 121 belongs to a narrow configuration, a wide configuration, or a funnel configuration, as previously described. The configuration 2902 also sets the size of the input text received by the data RAM 122 and the weight RAM 124 . In the narrow configuration and the funnel configuration, the size of the input text is narrow (eg, 8 or 9 digits), whereas in the wide configuration, the size of the input text is wide (eg, 12 or 16 digits). In addition, the configuration 2902 also sets the size of the output result 133 to be the same as the size of the input text.

Signed data value 2912 is true when it indicates that the data words received by DRAM 122 are signed values, and if false, indicates that these data words are unsigned values. When the signed weight value 2914 is true, it means that the weight text received by the weight random access memory 122 is a signed value, and if it is false, it means that the weight text is an unsigned value.

The data binary point 2922 value represents the binary point position of the data literal received by the data RAM 122 . For a preferred embodiment, for the position of the binary point, the data binary point 2922 value represents the number of bit positions of the binary point counted from the right. In other words, the data binary point 2922 represents the number of decimal places in the least significant bits of the data literal, ie, the number of bits to the right of the binary point. Similarly, the weight binary point 2924 value represents the binary point position of the weight literal received by the weight RAM 124 . For a preferred embodiment, when the ALU function 2926 is a multiplication and accumulation or output accumulation, the neural processing unit 126 determines the number of digits to the right of the binary decimal point of the value loaded in the accumulator 202 as the data binary decimal point 2922 Sum with weighted binary point 2924. So, for example, if the data binary point 2922 has a value of 5 and the weight binary point 2924 has a value of 3, the value in the accumulator 202 will have 8 bits to the right of the binary point. When the ALU function 2926 is a sum/maximum accumulator and data/weight literal or transfer data/weight literal, the NPU 126 will determine the number of digits to the right of the binary decimal point of the value loaded in the accumulator 202 respectively For data/weight binary decimal point 2922/2924. In another embodiment, a single accumulator binary point 2923 is specified instead of the individual data binary point 2922 and weight binary point 2924 . This part will be described in more detail later corresponding to FIG. 29B .

ALU function 2926 specifies a function to be performed by ALU 204 of NPU 126 . As previously mentioned, ALU function 2926 may include, but is not limited to, the following operations: multiply data literal 209 by weight literal 203 and add the product to accumulator 202; add accumulator 202 to weight literal 203; accumulate add accumulator 202 and data literal 209; maximum value of accumulator 202 and data literal 209; maximum value of accumulator 202 and weight literal 209; output accumulator 202; pass data literal 209; pass weight literal 209; output zero value. In one embodiment, the ALU function 2926 is specified by the NNU initialization instruction and used by the ALU 204 in response to an execution instruction (not shown). In one embodiment, the ALU function 2926 is specified by individual NNU instructions, such as the aforementioned multiply-accumulate and maxwacc instructions.

Rounding control 2932 specifies the form of rounding operations used by rounder 3004 (in FIG. 30). In one embodiment, the rounding modes that can be specified include, but are not limited to: no rounding, round-to-nearest, and random rounding. For a preferred embodiment, processor 100 includes random bit source 3003 (see FIG. 30 ) to generate random bits 3005 that are sampled to perform random rounding to reduce the possibility of rounding bias . In one embodiment, when the rounding bit 3005 is one and the sticky bit is zero, the NPU 126 rounds up if the sampled random bit 3005 is true, and if the sampled random bit 3005 is false, The NPU 126 would not round up. In one embodiment, random bit source 3003 generates random bits 3005 by sampling based on random electrical properties of processor 100, such as thermal noise of semiconductor diodes or resistors, although the invention is not limited thereto.

The activation function 2934 specifies a function for the value 217 of the accumulator 202 to produce the output 133 of the neural processing unit 126 . As described herein, activation functions 2934 include, but are not limited to: Sigmoid function; hyperbolic tangent function; soft add function; correction function; divide by a specified power of two; efficient division; pass the entire accumulator; and pass the accumulator in standard size, which are described in more detail in the following sections. In one embodiment, the activation function is specified by the NNU activation function instruction. In addition, the startup function can also be specified by the initialization command and used in response to the output command, for example, the startup function unit output command at address 4 in FIG. 4 . In this embodiment, the startup function command at address 3 in FIG. 4 will be included in within the output command.

The reciprocal 2942 value specifies a value that is multiplied by the accumulator 202 value 217 to achieve a divide operation on the accumulator 202 value 217 . That is, the reciprocal 2942 value specified by the user will be the reciprocal of the divisor actually intended to be performed. This is useful with convolution or common source operations as described in this paper. As far as a preferred embodiment is concerned, the user will designate the reciprocal 2942 value as two parts, which will be described in more detail later corresponding to FIG. 29C . In one embodiment, the control register 127 includes a field (not shown) that allows the user to specify one of multiple built-in divisor values for division. The size of these built-in divisor values is equivalent to that of a commonly used convolution kernel. Size, such as 9, 25, 36 or 49. In this embodiment, the activation function unit 212 stores the reciprocals of these built-in divisors to be multiplied by the value 217 of the accumulator 202 .

The offset 2944 specifies the number of bits by which the shifter of the enabled functional unit 212 will right-shift the value 217 of the accumulator 202 to divide it by a power of two. This is beneficial to operate with a convolution kernel whose size is a power of two.

The output RAM 2952 value specifies one of the data RAM 122 and the weight RAM 124 to receive the output result 133 .

The output binary point 2954 value represents the position of the binary point of the output result 133 . For a preferred embodiment, for the position of the binary point of the output result 133, the output binary point 2954 value represents the number of bit positions counted from the right. In other words, the output binary decimal point 2954 represents the number of decimal places in the least significant digits of the output result 133 , that is, the number of digits to the right of the binary decimal point. The activation function unit 212 will perform rounding, compression, Operations for saturation and size conversion.

Output command 2956 will output results 133 from many facets of control. In one embodiment, the boot function unit 121 utilizes the concept of a standard size, which is twice the width size (in bits) specified by the configuration 2902 . Thus, for example, if configuration 2902 sets the size of input text received by data RAM 122 and weight RAM 124 to be 8 bits, the standard size would be 16 bits; in another example, if Configuration 2902 sets the size of the input text received by DRAM 122 and WRAM 124 to be 16 bits, the standard size would be 32 bits. As described herein, accumulators 202 are larger in size (eg, 28 bits for narrow accumulator 202B and 41 bits for wide accumulator 202A) to accommodate intermediate computations, such as 1024 versus 512 NNUs The full precision of the multiply-accumulate instruction. Thus, the accumulator 202 value 217 is larger (in bits) than the standard size, and for most values of the activation function 2934 (except passing the entire accumulator), the activation function unit 212 (such as described below in the paragraph corresponding to FIG. 30 standard size compressor 3008) will compress the accumulator 202 value 217 to the size of the standard size. The first default value of the output command 2956 will instruct the start function unit 212 to execute the specified start function 2934 to generate an internal result and output this internal result as the output result 133, the size of the internal result is equal to the size of the original input text, i.e. the standard size half of. The second default value of the output command 2956 will instruct the start function unit 212 to execute the specified start function 2934 to generate an internal result and output the lower half of the internal result as the output result 133, the size of which is equal to the size of the original input text Twice of the standard size; and the third default value of the output command 2956 will instruct the startup function unit 212 to output the upper half of the internal result of the standard size as the output result 133 . A fourth default value of output command 2956 would instruct enable functional unit 212 to output the unprocessed least significant word of accumulator 202 as output result 133; and a fifth default value of output command 2956 would instruct enable functional unit 212 to output accumulator The unprocessed middle significant text of 202 is output as the output result 133; the sixth default value of the output command 2956 will instruct the startup function unit 212 to convert the unprocessed most significant text of the accumulator 202 (its width is specified by the configuration 2902) It is output as an output result 133 , which is described in more detail in the previous section corresponding to FIGS. 8 to 10 . As mentioned above, outputting the full accumulator 202 size or the standard size of the internal result helps other execution units 112 of the processor 100 to execute activation functions, such as the soft-max activation function.

The fields described in FIG. 29A (and FIG. 29B and FIG. 29C ) are located inside the control register 127 , however, the present invention is not limited thereto, and one or more fields may also be located in other parts of the NNU 121 . For a preferred embodiment, many of these fields may be included within the NNU instructions and decoded by the sequencer 128 to generate microinstructions 3416 (see FIG. 34 ) to control the ALU 204 and/or enable Function unit 212 . In addition, these fields can also be included in the micro-operation 3414 (please refer to FIG. 34 ) stored in the media register 118 to control the ALU 204 and/or enable the function unit 212 . This embodiment can reduce the usage of the initialization NNU instruction, while in other embodiments the initialization NNU instruction can be eliminated.

As before, the NNU instruction may specify a reference to a memory operand (e.g., a literal from the data RAM 122 and/or weight RAM 123) or a rotated operand (e.g., from the multitasking register 208/705 ) to perform arithmetic logic instruction operations. In one embodiment, the NNU instruction may also specify an operand as the register output of the activation function (such as the output of the register 3038 in FIG. 30 ). Additionally, as previously mentioned, the NNU instruction may specify to increment the current column address of the data RAM 122 or the weight RAM 124 . In one embodiment, the NNU instruction may specify an immediate signed integer difference to be added to the current column for the purpose of incrementing or decrementing a value other than one.

FIG. 29B is a block diagram showing another embodiment of the control register 127 of FIG. 1 . Control register 127 of FIG. 29B is similar to control register 127 of FIG. 29A , however, control register 127 of FIG. 29B includes an accumulator binary point 2923 . Accumulator binary point 2923 represents the binary point position of accumulator 202 . For a preferred embodiment, the accumulator binary point 2923 value represents the number of bit positions from the right of this binary point position. In other words, the accumulator binary point 2923 represents the number of decimal places in the least significant bits of the accumulator 202 , ie, the bits to the right of the binary point. In this embodiment, the accumulator binary point 2923 is explicitly indicated, rather than being implicitly identified as in the embodiment of FIG. 29A.

FIG. 29C is a block diagram illustrating an embodiment of storing the reciprocal 2942 of FIG. 29A in two parts. The first part 2962 is an offset value representing the number 2962 of leading zeros to be suppressed in the true reciprocal value of the accumulator 202 value 217 multiplied by the user. The number of leading zeros is the number of consecutive zeros immediately to the right of the binary point. The second part 2694 is the reciprocal value suppressed by leading zeros, that is, the real reciprocal value after removing all leading zeros. In one embodiment, the number of suppressed leading zeros 2962 is stored as 4 bits, and the reciprocal value of leading zeros suppressed 2964 is stored as an 8-bit unsigned value.

For example, assume that the user wants to multiply the value 217 of the accumulator 202 by the reciprocal value of the value 49. The reciprocal of the value 49 in two dimensions and set to 13 decimal places would be 0.0000010100111 with five leading zeros. In this way, the user will fill the value 5 with the number of suppressed leading zeros 2962, and fill the value 10100111 with the reciprocal value of leading zeros suppressed 2964. After the accumulator 202 value 217 is multiplied by the leading zero suppression reciprocal value 2964 by the reciprocal multiplier "Divider A" 3014 (see FIG. 30 ), the resulting product is shifted right by the number of suppressed leading zeros 2962. Such an embodiment facilitates the use of relatively few bits to express the reciprocal 2942 value to achieve high precision requirements.

FIG. 30 is a block diagram showing an embodiment of the activation function unit 212 of FIG. 2 . The enable function unit 212 includes the control logic 127 of FIG. 1, a positive type converter (PFC) and an output binary point aligner (OBPA) 3002 to receive the accumulator 202 value 217, a rounder 3004 to receive the accumulator 202 Value 217 with a pointer to the number of bits shifted out by the output binary point aligner 3002, a random bit source 3003 as previously described to generate random bits 3005, a first multiplexer 3006 to receive the positive type converter and output binary point alignment The output of the device 3002 and the output of the rounder 3004, a standard size compressor (CCS) and saturator 3008 to receive the output of the first multiplexer 3006, a bit selector and saturator 3012 to receive the standard size compressor and saturator 3012 The output of the saturator 3008, a corrector 3018 to receive the output of the standard size compressor and saturator 3008, an inverse multiplier 3014 to receive the output of the standard size compressor and saturator 3008, a right shifter 3016 to receive The output of the standard size compressor and saturator 3008, a hyperbolic tangent (tanh) block 3022 to receive the output of the bit selector and saturator 3012, an S-shaped block 3024 to receive the output of the bit selector and saturator 3012, a Soft addition module 3026 to receive the output of bit selector and saturator 3012, a second multiplexer 3032 to receive hyperbolic tangent module 3022, S-type module 3024, soft addition module 3026, corrector 3018, reciprocal multiplier 3014 and The output of the right shifter 3016 and the standard size output 3028 delivered by the standard size compressor and saturator 3008, a sign restorer 3034 to receive the output of the second multiplexer 3032, a size converter and saturator 3036 to Receive the output of the symbol restorer 3034, a third multiplexer 3037 to receive the output of the size converter and saturator 3036 and the accumulator output 217, and an output buffer 3038 to receive the output of the multiplexer 3037, and its output That is the result 133 in FIG. 1 .

The positive type converter and output binary point aligner 3002 receives the accumulator 202 value 217 . For a preferred embodiment, accumulator 202 value 217 is a full precision value as previously described. That is, the accumulator 202 has enough bits of storage to load the accumulator, which is the sum produced by the integer adder 244 by adding a series of products produced by the integer multiplier 242, without discarding. The individual products of the multipliers 242 or any one bit of the respective totals of the adders to maintain accuracy. For a preferred embodiment, the accumulator 202 has at least enough bits to hold the maximum number of product accumulations that the NNU 121 can be programmed to generate. For example, please refer to the program in FIG. 4 , under the wide configuration, the neural network unit 121 can be programmed to generate a maximum number of multiply-accumulates of 512, and the width of the accumulated number 202 is 41. In another example, please refer to the program in FIG. 20 , under the narrow configuration, the neural network unit 121 can be programmed to generate a maximum number of product accumulations of 1024, and the width of the accumulation number 202 is 28 bits. Basically, full precision accumulator 202 has at least Q bits, where Q is the sum of M and log ₂ P, where M is the bit width of the integer product of multiplier 242 (for example, for narrow multiplier 242 is 16 bits, or 32 bits for wide multiplier 242), and P is the maximum allowable number of products that accumulator 202 can accumulate. As far as a preferred embodiment is concerned, the maximum number of multiply-accumulate is specified according to the program specification of the programmer of the neural network unit 121 . In one embodiment, the sequence is based on the assumption that a previous multiply-accumulate instruction is used to load the data/weight literal 206/207 column from the data/weight RAM 122/124 (such as the instruction at address 1 in FIG. 4 ). The maximum value of the count of the multiplication and accumulation neural network unit instruction (such as the instruction of address 2 in FIG. 4 ) executed by the device 128 is, for example, 511.

The design of the ALU 204 of the NPU 126 is simplified by using an accumulator 202 with sufficient bit width to perform accumulation operations on the maximum number of full precision values allowed to be accumulated. In particular, this alleviates the need to use logic to saturate the sum produced by the integer adder 244, which would overflow a small accumulator, while keeping track of the accumulator's binary point position To confirm whether an overflow occurs to confirm whether a saturation operation needs to be performed. As an example, for a design with a non-full precision accumulator but with saturation logic to handle overflow of the non-full precision accumulator, assume the following.

(1) The range of the data literal value is between 0 and 1 and all storage bits are used to store decimal places. The range of weight literal values is between -8 and +8 and all but three storage bits are used to store decimal places. The range of accumulated values used as input to a hyperbolic tangent activation function is between -8 and 8, and all but three storage bits are used to store decimal places.

(2) The bit width of the accumulator is non-full precision (such as only the bit width of the product).

(3) Assuming that the accumulator is full precision, the final accumulated value will also be between -8 and 8 (such as +4.2); however, the product before "point A" in this sequence will produce positive values more frequently , and the product after point A will produce negative values more frequently.

In this case, incorrect results (such as results other than +4.2) may be obtained. This is because at some point ahead of point A, when it is necessary to bring the accumulator to a value above its saturation maximum of +8, such as +8.2, the extra 0.2 is lost. The accumulator will even keep the remaining multiply-accumulate result at saturation, losing more positive values. Therefore, the final value of the accumulator may be smaller than the value calculated using the accumulator with full precision bit width (ie, less than +4.2).

The positive type converter 3004 converts the value 217 of the accumulator 202 to a positive type when it is negative, and generates an extra bit to indicate the original value's sign. Converting negative numbers into positive types can simplify the operation of the subsequent startup function unit 121 . For example, after this processing, only positive values will be input into the hyperbolic tangent module 3022 and the sigmoid module 3024, thus simplifying the design of these modules. In addition, the rounder 3004 and the saturator 3008 can also be simplified.

The output binary point aligner 3002 shifts or scales the positive type value to the right to align it with the output binary point 2954 specified in the control register 127 . For a preferred embodiment, the output binary point aligner 3002 calculates the number of decimal places of the accumulator 202 value 217 (such as specified by the accumulator binary point 2923 or the addition of the data binary point 2922 and the weight binary point 2924 total) minus the number of decimal places for the output (as specified, for example, by the output binary decimal point 2954) as the offset. Thus, for example, if the accumulator 202 binary decimal point 2923 is 8 (i.e. the above embodiment) and the output binary decimal point 2954 is 3, the output binary decimal point aligner 3002 will right-shift the positive type value by 5 bits to generate The result is provided to the multiplexer 3006 and the rounder 3004.

The rounder 3004 performs rounding operations on the accumulator 202 value 217 . For a preferred embodiment, the rounder 3004 generates a rounded version of the positive type value generated by the positive type converter and the output binary point aligner 3002, and provides the rounded version to the multiplexer 3006. The rounder 3004 performs rounding operations according to the aforementioned rounding control 2932 , which includes random rounding using random bits 3005 as described herein. Multiplexer 3006 selects one of its multiple inputs, namely the positive input from positive type converter and output binary point aligner 3002, in accordance with rounding control 2932 (which may include random rounding as described herein). The type value is either a rounded version from the rounder 3004 and the selected value is provided to the normal size compressor and saturator 3008. For a preferred embodiment, multiplexer 3006 selects the output of positive type converter and output binary point aligner 3002 if the rounding control specifies no rounding, otherwise selects the output of rounder 3004 output. In other embodiments, additional rounding operations may also be performed by the enable function unit 212 . For example, in one embodiment, when the bit selector 3012 compresses the output bits of the standard-size compressor and saturator 3008 (as described later), the bit selector 3012 performs truncation based on missing low order bits input operation. In another example, the product of the reciprocal multiplier 3014 (described later) is rounded. In yet another example, the size converter 3036 needs to convert to an appropriate output size (described later), and this conversion may involve dropping some low-order bits used to determine rounding, and then rounding is performed.

The standard size compressor 3008 compresses the output of the multiplexer 3006 to a standard size. Thus, for example, standard size compressor 3008 can compress the 28-bit output of multiplexer 3006 to 16 bits if NPU 126 is in narrow configuration or funnel configuration 2902; With configuration 2902, the standard size compressor 3008 can compress the 41-bit output value of the multiplexer 3006 to 32 bits. However, before compressing to the standard size, if the uncompressed value is greater than the maximum value that can be expressed in the standard form, the saturator 3008 will fill the uncompressed value to the maximum value that can be expressed in the standard form. For example, if any bit of the uncompressed value to the left of the most significant uncompressed value bit is a value 1, the saturator 3008 is filled to a maximum value (eg, all 1s).

For a preferred embodiment, the hyperbolic tangent module 3022, the S-type module 3024, and the soft-add module 3026 all include look-up tables, such as programmable logic array (PLA), read-only memory (ROM), combinational logic gate wait. In one embodiment, in order to simplify and reduce the size of these modules 3022/3024/3026, the input values provided to these modules have the form of 3.4, that is, three integer characters and four decimal places, that is, the input values have four The ones digit is to the right of the binary point and has three digits to the left of the binary point. These values were chosen because at the extremes of the input value range (-8, +8) of the 3.4 version, the output value will asymptotically approach its min/max values. However, the present invention is not limited thereto, and the present invention can also be applied to other embodiments where the binary point is placed in a different position, such as in a 4.3 pattern or a 2.5 pattern. The bit selector 3012 will select the bits that meet the 3.4 form specification among the bits output by the standard size compressor and saturator 3008, which involves compression processing, that is, some bits will be lost, because the standard form has more bits . However, before selecting/compressing the standard size compressor and saturator 3008 output value, if the uncompressed value is greater than the maximum value expressible in the 3.4 form, the saturator 3012 will fill the uncompressed value to the expressible 3.4 form maximum value. For example, if any of the bits to the left of the most significant 3.4 pattern bits in the pre-compressed value have the value 1, the saturator 3012 will be filled to the maximum value (eg, filled to all 1s).

The hyperbolic tangent module 3022 , the sigmoid module 3024 and the soft adder module 3026 execute the corresponding activation function (as described above) on the 3.4 type values output by the standard size compressor and saturator 3008 to generate a result. As far as a preferred embodiment is concerned, what the hyperbolic tangent module 3022 and the sigmoid module 3024 produce is a 0.7-type 7-bit result, that is, zero integer characters and seven decimal places, that is, the input value has seven digits to the right of the binary point. For a preferred embodiment, the soft-add module 3026 produces a 7-bit result of the 3.4 format, ie, the same format as the module 3026 input. For a preferred embodiment, the outputs of the hyperbolic tangent module 3022, the sigmoid module 3024, and the soft-add module 3026 are extended to normal form (such as adding leading zeros where necessary) and aligned so that the binary point is represented by Outputs the value specified with 2954 binary decimal points.

Corrector 3018 produces a corrected version of the output values of standard size compressor and saturator 3008 . That is, if the output value of the standard-size compressor and saturator 3008 (which has its sign shifted down the pipeline as described above) is negative, the corrector 3018 will output a value of zero; otherwise, the corrector 3018 will output its input value. For a preferred embodiment, the output of the corrector 3018 is of standard form and has a binary point specified by the output binary point 2954 value.

Reciprocal multiplier 3014 multiplies the output of standard-size compressor and saturator 3008 by the user-specified reciprocal value specified in reciprocal value 2942 to produce a standard-sized product, which is effectively the standard-size compressor and saturator The output value of device 3008 is the quotient calculated with the reciprocal of the reciprocal value 2942 as the divisor. For a preferred embodiment, the output of the reciprocal multiplier 3014 is of standard form and has a binary point specified by the output binary point 2954 value.

Right shifter 3016 shifts the output of standard-size compressor and saturator 3008 by the user-specified number of bits specified in offset value 2944 to produce the normal-size quotient. For a preferred embodiment, the output of the right shifter 3016 is of standard form and has a binary point specified by the output binary point 2954 value.

The multiplexer 3032 selects the appropriate input specified by the value of the activation function 2934, and provides its selection to the symbol restorer 3034. If the original accumulator 202 value 217 is negative, the symbol restorer 3034 will output the multiplexer 3032 Converts a value of positive type to a negative type, for example to a two's complement type.

The size converter 3036 converts the output of the sign restorer 3034 to an appropriate size according to the value of the output command 2956 as described in FIG. 29A . For a preferred embodiment, the output of the sign restorer 3034 has a binary point specified by the output binary point 2954 value. For a preferred embodiment, the size converter 3036 discards the upper half of the sign restorer 3034 output for the first default value of the output command. In addition, if the output of sign restorer 3034 is positive and exceeds the maximum value that can be expressed by the character size specified by configuration 2902, or if the output is negative and is smaller than the minimum value that can be expressed by the character size, saturator 3036 will output it Expressable maximum/minimum values to fill up to this text size respectively. For the second and third default values, the size converter 3036 passes the output of the sign restorer 3034 .

The multiplexer 3037 selects one of the output of the data converter and saturator 3036 and the output 217 of the accumulator 202 according to the output command 2956 to provide to the output register 3038 . Further, for the first and second default values of output command 2956, multiplexer 3037 selects the lower text of the output of size converter and saturator 3036 (size specified by configuration 2902). For the third default, the multiplexer 3037 selects the upper text of the output of the size converter and saturator 3036 . For the fourth default value, the multiplexer 3037 will select the lower text of the unprocessed accumulator 202 value 217; for the fifth default value, the multiplexer 3037 will select the middle text of the unprocessed accumulator 202 value 217; For the sixth default value, the multiplexer 3037 will select the upper text of the value 217 of the unprocessed accumulator 202 . As mentioned above, for a preferred embodiment, the enabling function unit 212 adds a zero value upper digit to the upper text of the value 217 of the unprocessed accumulator 202 .

FIG. 31 shows an example of the operation of the activation function unit 212 of FIG. 30 . As shown in the figure, the configuration 2902 of the neural processing unit 126 is set to a narrow configuration. Additionally, the signed data 2912 and signed weight 2914 values are true. In addition, the data binary decimal point 2922 value indicates that there are 7 bits to the right of the binary decimal point position for the data random access memory 122 word, and an example value of the first data word received by the neural processing unit 126 is 0.1001110. In addition, the value of the weight binary point 2924 indicates that for the weight random access memory 124 text, there are 3 bits to the right of the binary point position, and the example value of the first weight text received by the neural processing unit 126 is 00001.010.

The 16-bit product of the first data and the weight literal (this product is added to the initial zero value of the accumulator 202 ) renders 000000.1100001100. Since the data binary point 2912 is 7 and the weight binary point 2914 is 3, there will be 10 bits to the right of the implied accumulator 202 binary point. In the case of a narrow configuration, as shown in this embodiment, the accumulator 202 has a width of 28 bits. For example, after completing all the arithmetic and logic operations (for example, all 1024 multiplication and accumulation operations in FIG. 20 ), the value 217 of the accumulator 202 will be 0000000000000000001.1101010100.

The output binary point 2954 value means that there are 7 bits to the right of the output binary point. Therefore, after passing the output binary point aligner 3002 and standard size compressor 3008, the accumulator 202 value 217 is scaled, rounded and compressed to a standard form value, ie 000000001.1101011. In this example, the output binary point address represents 7 decimal places, while the accumulator 202 binary point position represents 10 decimal places. Therefore, the output binary point aligner 3002 calculates a difference of 3 and scales it by right-shifting the accumulator 202 value 217 by 3 bits. In FIG. 31 it is shown that the value 217 of the accumulator 202 will lose the 3 least significant bits (binary number 100). Also, in this example, the rounding control 2932 value indicates that random rounding is used, and the sample random bit 3005 is assumed to be true in this example. Thus, as before, the least significant bit is rounded up because of the rounding bit of the accumulator 202 value 217 (the most significant bit ) is one, and the sticky bit (the result of the Boolean OR operation of the 2 least significant bits of the 3 bits shifted out due to the scaling operation of the accumulator 202 value 217 ) is zero.

In this example, the start function 2934 indicates that a sigmoid function is used. Thus, the bit selector 3012 selects the bits of the standard type value such that the input to the S-type block 3024 has three integer characters and four decimal places, as before, ie the value 001.1101 shown. The output value of the S-type module 3024 will be put into the standard form, ie the value 000000000.1101110 shown.

The output command 2956 of this example specifies a first default, the size of the text represented by the output configuration 2902, in this case narrow text (8 bits). In this way, the size converter 3036 will convert the standard S-type output value into an 8-bit quantity, which has an implicit binary decimal point, that is, there are 7 bits to the right of the binary decimal point, and an output value 01101110 is generated, as shown in the figure shown in .

FIG. 32 shows a second example of the operation of the activation function unit 212 of FIG. 30 . The example of FIG. 32 depicts the operation of the enable function unit 212 when the enable function 2934 indicates passing the value 217 of the accumulator 202 in a standard size. As shown, this configuration 2902 is set to a narrow configuration of the neural processing unit 216 .

In this example, the width of the accumulator 202 is 28 bits, and there are 10 bits to the right of the position of the accumulator 202 binary point (this is because in one embodiment the sum of the data binary point 2912 and the weight binary point 2914 is 10, or in another embodiment the accumulator binary point 2923 is explicitly assigned to have a value of 10). For example, after performing all arithmetic logic operations, the value 217 of the accumulator 202 shown in FIG. 32 is 000001100000011011.1101111010.

In this example, an output binary point value of 2954 means that there are 4 bits to the right of the binary point for the output. Thus, after passing the output binary point aligner 3002 and the standard size compressor 3008, the accumulator 202 value 217 is saturated and compressed to the shown standard value of 111111111111.1111, which is received by the multiplexer 3032 as the standard size Pass the value 3028.

In this example two output commands 2956 are shown. The first output command 2956 specifies a second default value, ie, to output the text below the standard form size. Since the size indicated by configuration 2902 is narrow literal (8 bits), the standard size would be 16 bits, and the size converter 3036 would select the lower 8 bits of the standard size pass value 3028 to produce 8 bits as shown The value is 11111111. The second output command 2956 specifies a third default, which is to output the upper text of the standard form size. Thus, the size converter 3036 would select the upper 8 bits of the standard size transfer value 3028 to produce the 8-bit value 11111111 as shown.

FIG. 33 shows a third example of the operation of the activation function unit 212 of FIG. 30 . The example of FIG. 33 illustrates the operation of the enable function unit 212 when the enable function 2934 indicates that the entire unprocessed accumulator 202 value 217 is to be delivered. As shown, this configuration 2902 is set to a wide configuration of the NPU 126 (eg, 16-bit input text).

In this example, the width of the accumulator 202 is 41 bits, and there are 8 bits to the right of the accumulator 202 binary point position (this is because in one embodiment the sum of the data binary point 2912 and the weight binary point 2914 is 8, or in another embodiment the accumulator binary point 2923 is explicitly assigned to have a value of 8). For example, after performing all arithmetic logic operations, the value 217 of the accumulator 202 shown in FIG. 33 is 001000000000000000001100000011011.11011110.

Three output commands 2956 are shown in this example. The first output command specifies the fourth default value, that is, the text below the unprocessed accumulator 202 value; the second output command specifies the fifth default value, that is, the middle text that outputs the unprocessed accumulator 202 value; And the third output command specifies the sixth default value, that is, the upper text that outputs the unprocessed accumulator 202 value. Because the size indicated by the configuration 2902 is wide text (16 bits), as shown in Figure 33, in response to the first output command 2956, the multiplexer 3037 will select the 16-bit value 0001101111011110; in response to the second output command 2956, the multiplexer 3037 The 16-bit value 0000000000011000 will be selected; and in response to the third output command 2956 , the multiplexer 3037 will select the 16-bit value 0000000001000000.

As mentioned above, the neural network unit 121 can execute on integer data instead of floating point data. In this way, each neural processing unit 126 , or at least the ALU 204 part thereof, can be simplified. For example, the ALU 204 does not need to incorporate an adder for the multiplier 242, which is required in floating-point operations to add the exponents of the multipliers. Similarly, the ALU 204 does not need to include a shifter for the adder 234 to align the binary point of the addend in floating point operations. Those of ordinary skill in the art will understand that floating point units are often very complex; therefore, the examples described herein are only simplified for the ALU 204, which has hardware fixed-point assistance and allows the user to specify the relevant binary point The integer embodiment of can also be used to simplify other parts. Using an integer unit for the ALU 204 results in a smaller (and faster) NPU 126 compared to the floating-point embodiment, which facilitates the integration of a large NPU 126 array into the NNU Within 121. The portion of the activation function unit 212 may handle scaling and saturation of the accumulator 202 value 217 based on user specification, the number of decimal places required for the accumulator, and the number of decimal places required for the output value, preferably based on user specification. Any additional complexity and concomitant increase in size, energy and/or time consumption in the fixed-point hardware assistance of AFU 212 can be amortized by sharing AFU 212 among ALU 204, which This is because, as shown in the embodiment of FIG. 11 , the number of startup function units 1112 can be reduced in the embodiment of sharing.

Embodiments described herein can enjoy many of the advantages of using integer arithmetic units to reduce hardware complexity (compared to using floating point arithmetic units), while still being able to perform arithmetic operations on fractional numbers, ie, numbers with a binary decimal point. The advantage of floating-point arithmetic is that it can provide data arithmetic operations on individual values of data falling within a very wide range of values (actually only limited by the size of the exponent range, so a very large range). That is, each floating point number has its own potentially unique exponent value. However, the embodiments described herein understand and take advantage of the property that certain applications have input data that is highly parallel and falls within a relatively narrow range such that all parallel data have the same "exponent". Thus, these embodiments allow the user to assign the binary point position to all input values and/or accumulated values at once. Similarly, by understanding and taking advantage of the fact that parallel outputs have similar ranges, these embodiments allow the user to assign the binary point position to all output values at once. An artificial neural network is an example of such an application, although embodiments of the invention may also be applied to perform computations for other applications. By assigning the binary point position to multiple inputs at once rather than to individual input numbers, embodiments of the present invention can use memory space more efficiently (eg, require less memory) than using floating point arithmetic And/or improve precision while using a similar amount of memory, since the bits of the exponent used for floating point operations can be used to increase numerical precision.

Furthermore, embodiments of the present invention understand that there may be a loss of precision when performing accumulations on a large series of integer operations (such as overflow or loss of less significant decimal places), and thus provide a solution, primarily by using a sufficiently large accumulator to avoid loss of precision.

Direct execution of micro-operations in neural network units

FIG. 34 is a block diagram showing partial details of the processor 100 and the neural network unit 121 of FIG. 1 . The neural network unit 121 includes the pipeline stage 3401 of the neural processing unit 126 . Each pipeline stage 3401 is distinguished by a stage register, and includes combinational logic to realize the operations of the neural processing unit 126 herein, such as Boolean logic gates, multiplexers, adders, multipliers, comparators, and so on. Pipeline stage 3401 receives uops 3418 from multiplexer 3402 . Micro-ops 3418 flow down to pipeline stage 3401 and control its combinatorial logic. Micro-op 3418 is a collection of bits. For a preferred embodiment, the micro-operation 3418 includes data RAM 122 memory address 123 bits, weight RAM 124 memory address 125 bits, program memory 129 memory address 131 bits, multitasking register 208/705 control signals 213/713, and many fields of control registers 217 (eg, the control registers of FIGS. 29A-29C ). In one embodiment, micro-operations 3418 include approximately 120 bits. Multiplexer 3402 receives uops from three different sources and selects one of them as uop 3418 to provide to pipeline stage 3401 .

A source of micro-operations for the multiplexer 3402 is the sequencer 128 of FIG. 1 . The sequencer 128 decodes the NNU instruction received from the program memory 129 and generates a micro-operation 3416 accordingly to the first input of the multiplexer 3402 .

The second source of micro-operations for the multiplexer 3402 is the decoder 3404 that receives microinstructions 105 from the reservation station 108 of FIG. For a preferred embodiment, as mentioned above, the microinstruction 105 is generated by the instruction translator 104 in response to the translation of the MTNN instruction 1400 and the MFNN instruction 1500 . Microinstruction 105 may include an immediate field to specify a specific function (specified by an MTNN instruction 1400 or an MFNN instruction 1500), such as starting and stopping execution of a program in program memory 129, executing a microoperation directly from media register 118 , or reading/writing the memory of the neural network unit as described above. The decoder 3404 decodes the microinstruction 105 and generates a microoperation 3412 accordingly, which is provided to the second input of the multiplexer. For a preferred embodiment, for some functions 1432/1532 of the MTNN instruction 1400/MFNN instruction 1500, the decoder 3404 does not need to generate a micro-operation 3412 to be sent down the pipeline 3401, such as writing to the control buffer register 127, start executing the program in the program memory 129, suspend the execution of the program in the program memory 129, wait for the program in the program memory 129 to finish executing, read from the state register 127, and reset the neural network unit 121.

The third source of micro-operations for the multiplexer 3402 is the media buffer 118 itself. In terms of a preferred embodiment, as described above corresponding to FIG. 14 , the MTNN instruction 1400 can specify a function to instruct the neural network unit 121 to directly execute a microprocessor that is provided by the media buffer 118 to the third input of the multiplexer 3402. Operation 3414. Direct execution of the micro-operations 3414 provided by the architectural media buffer 118 is beneficial for testing the NNU 121, such as built-in self-test (BIST), or debugging actions.

For a preferred embodiment, the decoder 3404 generates a mode pointer 3422 to control the selection of the multiplexer 3402 . When the MTNN instruction 1400 specifies a function to start executing a program from the program memory 129, the decoder 3404 will generate a mode pointer 3422 value so that the multiplexer 3402 selects the micro-operation 3416 from the sequencer 128 until an error occurs or until the decoding Encoder 3404 encounters an MTNN instruction 1400 specifying a function to stop executing the program from program memory 129. When the MTNN instruction 1400 specifies a function to instruct the neural network unit 121 to directly execute a micro-operation 3414 provided by the media buffer 118, the decoder 3404 will generate a mode pointer 3422 value to make the multiplexer 3402 select from the specified media buffer 118 micro-operations 3414. Otherwise, the decoder 3404 generates a mode pointer 3422 value to make the multiplexer 3402 select the micro-operation 3412 from the decoder 3404 .

Variable Rate Neural Network Unit

In many cases, after executing a program, the neural network unit 121 enters an idle state and waits for the processor 100 to process some things that need to be processed before executing the next program. For example, assuming a situation similar to that described in FIG. 3 to FIG. 6A, the neural network unit 121 will activate a function program (also called a feed forward neural network layer program) for a multiply-accumulate )) is executed two or more times in a row. Compared with the time taken by the neural network unit 121 to execute the program, the processor 100 obviously takes a longer time to write the 512 KB weight value into the weight random access memory 124 for the next neural network unit program. In other words, the neural network unit 121 executes the program for a short time, and then enters the standby state until the processor 100 writes the next weight value into the weight random access memory 124 for the next program execution. This situation can be referred to FIG. 36A , which will be described in detail later. In this case, the neural network unit 121 can run at a lower clock frequency to prolong the execution time of the program, so that the energy consumption required for executing the program can be distributed to a longer time range, so that the neural network unit 121, and even the entire The processor 100 is maintained at a lower temperature. This situation is called relaxation mode, which can be referred to FIG. 36B , which will be described in detail later.

FIG. 35 is a block diagram showing a processor 100 with a variable rate neural network unit 121 . The processor 100 is similar to the processor 100 of FIG. 1 , and components with the same reference numbers in the figure are also similar. The processor 100 of FIG. 35 also has a time-frequency generation logic 3502 coupled to the functional units of the processor 100, these functional units are the instruction fetch unit 101, the instruction cache 102, the instruction translator 104, the renaming unit 106, and the reservation station 108 , the neural network unit 121 , other execution units 112 , the memory subsystem 114 , the general buffer 116 and the media buffer 118 . The time-frequency generation logic 3502 includes a time-frequency generator, such as a phase-locked loop (PLL), to generate a time-frequency signal having a dominant time frequency or a time-frequency frequency. For example, the primary clock frequency can be 1GHz, 1.5GHz, 2GHz and so on. The time frequency means the number of cycles per second, such as the number of oscillations of the time-frequency signal between high and low states. Preferably, the time-frequency signal has a duty cycle, that is, half of the cycle is in a high state and the other half is in a low state; in addition, the time-frequency signal may also have an unbalanced cycle, that is, the duty cycle of the time-frequency signal is It spends more time in the high state than it does in the low state, and vice versa. Preferably, the phase-locked loop is used to generate main time-frequency signals of multiple time-frequency. Preferably, the processor 100 includes a power management module, which automatically adjusts the main clock frequency according to various factors, these factors include the dynamic detection of the operating temperature of the processor 100, utilization (utilization), and information from system software (such as the operating system, Basic Input Output System (BIOS)) commands that indicate desired performance and/or power saving metrics. In one embodiment, the power management module includes microcode of the processor 100 .

Time-frequency generation logic 3502 also includes a time-frequency distribution network, or clock tree. The time-frequency tree will distribute the main time-frequency signals to the functional units of the processor 100, as shown in FIG. Send to the instruction cache 102, send the time-frequency signal 3506-10 to the instruction translator 104, send the time-frequency signal 3506-9 to the renaming unit 106, send the time-frequency signal 3506-8 to the reservation station 108, and send the time-frequency signal 3506-8 to the reservation station 108. The frequency signal 3506-7 is transmitted to the neural network unit 121, the time-frequency signal 3506-4 is transmitted to other execution units 112, the time-frequency signal 3506-3 is transmitted to the memory subsystem 114, and the time-frequency signal 3506-5 is transmitted to the general buffer 116 , and transmits time-frequency signal 3506 - 6 to media buffer 118 , these signals are collectively referred to as time-frequency signal 3506 . The time-frequency tree has nodes or lines to transmit primary time-frequency signals 3506 to their corresponding functional units. In addition, preferably, the time-frequency generation logic 3502 may include a time-frequency buffer. When it is necessary to provide a cleaner time-frequency signal and/or to increase the voltage level of the main time-frequency signal, especially for a farther node, The time-frequency buffer regenerates the main time-frequency signal. In addition, each functional unit has its own sub-time-frequency tree to regenerate and/or boost the received voltage level of the corresponding main time-frequency signal 3506 when necessary.

The NNU 121 includes clock-frequency reduction logic 3504 that receives the mitigation pointer 3512 and the main clock-frequency signal 3506-7 to generate a second clock-frequency signal. The second time-frequency signal has a time frequency. At this time, if the frequency is not the same as the main time frequency, or it is in the relaxation mode, the value is reduced from the main time frequency to reduce heat generation. This value is programmed to the relaxation pointer 3512. The time-frequency reduction logic 3504 is similar to the time-frequency generation logic 3502. It has a time-frequency distribution network, or a time-frequency tree, to distribute the second time-frequency signal to various functional blocks of the neural network unit 121. Signal 3508-1 is passed to NPU array 126, time-frequency signal 3508-2 is passed to sequencer 128 to pass time-frequency signal 3508-3 to interface logic 3514, collectively referred to as second time-frequency signal 3508 . Preferably, these neural processing units 126 include a plurality of pipeline stages 3401 , as shown in FIG. 34 , the pipeline stages 3401 include pipeline staging buffers for receiving the second clock frequency signal 3508 - 1 from the clock frequency reduction logic 3504 .

The NNU 121 also has an interface logic 3514 to receive the primary time-frequency signal 3506-7 and the second time-frequency signal 3508-3. The interface logic 3514 is coupled between the lower portion of the front end of the processor 100 (such as the reservation station 108, the media buffer 118 and the general purpose buffer 116) and various functional blocks of the NNU 121. These functional blocks are real-time frequency reduction logic 3504, Data RAM 122 , weight RAM 124 , program memory 129 and sequencer 128 . Interface logic 3514 includes data RAM buffer 3522 , weight RAM buffer 3524 , decoder 3404 of FIG. 34 , and buffer pointer 3512 . Easing pointer 3512 is loaded with a value specifying how slowly NPU array 126 will execute NNU program instructions. Preferably, the relaxation pointer 3512 specifies a divisor value N, and the time-frequency reduction logic 3504 divides the main time-frequency signal 3506-7 by this divisor value to generate the second time-frequency signal 3508, so that the time frequency of the second time-frequency signal It will be 1/N. Preferably, the value of N can be programmed as any one of a plurality of different default values, and these default values can make the time-frequency reduction logic 3504 correspondingly generate a plurality of second time-frequency signals 3508 with different time frequencies, and these time-frequency less than the main time frequency.

In one embodiment, the time-frequency reduction logic 3504 includes a time-frequency divider circuit for dividing the main time-frequency signal 3506-7 by the relaxation pointer 3512 value. In one embodiment, the time-frequency reduction logic 3504 includes a time-frequency gate (such as an AND gate), and the time-frequency gate can gate the main time-frequency signal 3506-7 through an enable signal, and the enable signal is activated every time the main time-frequency signal The true value will only be generated once in N cycles. As an example, consider a circuit that includes a counter that counts up to N to generate an enable signal. When the accompanying logic circuit detects that the output of the counter matches N, the logic circuit generates a true value pulse on the second time-frequency signal 3508 and resets the counter. Preferably, the value of the buffer pointer 3512 can be programmed by an architectural instruction, such as the MTNN instruction 1400 of FIG. 14 . Preferably, before the architecture program instructs the neural network unit 121 to start executing the neural network unit program, the architecture program operating on the processor 100 will program the relaxation value to the relaxation pointer 3512, and this part will be described later corresponding to FIG. 37 More detailed instructions.

The weight RAM buffer 3524 is coupled between the weight RAM 124 and the media buffer 118 as a buffer for data transmission therebetween. Preferably, weight RAM buffer 3524 is similar to one or more embodiments of buffer 1704 of FIG. 17 . Preferably, the portion of the WRAM buffer 3524 receiving data from the media buffer 118 takes the main time-frequency signal 3506-7 having the main time frequency as the time frequency, and the WRAM buffer 3524 receives the data from the weight random access The part of the memory 124 receiving data uses the second clock frequency signal 3508-3 with the second clock frequency as the clock frequency, and the second clock frequency can be lowered or not from the main clock frequency according to the value programmed in the relaxation pointer 3512, that is, The downscaling or no is performed depending on whether the NNU 121 is operating in a moderate or normal mode. In one embodiment, the weight random access memory 124 is a single port. As described above in FIG. 126 or the column buffer 1104 in FIG. 11 is accessed in an arbitrated fashion. In another embodiment, the weight random access memory 124 is a dual-port, as described above in FIG. 1104 are accessed in parallel.

Similar to the weight RAM buffer 3524 , the data RAM buffer 3522 is coupled between the data RAM 122 and the media buffer 118 as a buffer for data transmission therebetween. Preferably, data RAM buffer 3522 is similar to one or more embodiments of buffer 1704 of FIG. 17 . Preferably, the portion of the data RAM buffer 3522 receiving data from the media buffer 118 uses the primary clock signal 3506-7 having the primary clock rate as the clock frequency, and the data RAM buffer 3522 receives data from the data random access The part of the memory 122 receiving data uses the second clock frequency signal 3508-3 with the second clock frequency as the clock frequency, and the second clock frequency can be lowered or not from the main clock frequency according to the value programmed in the relaxation pointer 3512, that is, The downscaling or no is performed depending on whether the NNU 121 is operating in a moderate or normal mode. In one embodiment, the data random access memory 122 is a single port. As described above in FIG. 126 or the column buffer 1104 in FIG. 11 is accessed by arbitration. In another embodiment, the DRAM 122 is dual-ported. As described above in FIG. 1104 are accessed in parallel.

Preferably, regardless of whether data RAM 122 and/or weight RAM 124 are single-ported or dual-ported, interface logic 3514 includes data RAM buffer 3522 and weight RAM buffer 3524 to synchronize main The time-frequency domain and the second time-frequency domain. Preferably, the data random access memory 122, the weight random access memory 124 and the program memory 129 all have a static random access memory (SRAM), which includes individual read enable signals, write enable signals and memory select enable signal.

As mentioned above, the neural network unit 121 is an execution unit of the processor 100 . The execution unit is a functional unit in the processor that executes the translated microinstructions of the architectural instructions or executes the architectural instructions itself, for example, executes the microinstructions 105 translated from the architectural instructions 103 in FIG. 1 or the architectural instructions 103 itself. The execution units receive operands from general-purpose registers of the processor, such as general-purpose register 116 and media register 118 . The execution unit will generate a result after executing the micro instruction or the architectural instruction, and the result will be written into the general register. The MTNN instruction 1400 and the MFNN instruction 1500 shown in FIGS. 14 and 15 are examples of the architectural instructions 103 . Microinstructions are used to implement architectural instructions. More precisely, the collective execution of one or more microinstructions translated by the execution unit from the architectural instruction is to execute the operation specified by the architectural instruction on the input specified by the architectural instruction to generate a result defined by the architectural instruction.

FIG. 36A is a timing diagram showing an example of the operation of the processor 100 with the NNU 121 operating in the normal mode, ie operating at the main clock frequency. In a sequence diagram, time progresses from left to right. The processor 100 executes the architecture program at the main clock frequency. More precisely, the front end of processor 100 (e.g., instruction fetch unit 101, instruction cache 102, instruction translator 104, renaming unit 106, and reservation station 108) fetches, decodes, and issues architectural instructions to The neural network unit 121 and other execution units 112 .

Initially, the architecture program executes an architecture instruction (such as the MTNN instruction 1400 ), and the processor front end 100 issues the architecture instruction to the NNU 121 to instruct the NNU 121 to start executing the NNU program in its program memory 129 . Previously, the architecture program would execute the architecture instruction to write the value specifying the main clock frequency into the relaxation pointer 3512 , even if the neural network unit is in the normal mode. More precisely, programming to the value of the ease pointer 3512 causes the clock frequency reduction logic 3504 to generate the second clock frequency signal 3508 at the primary clock frequency of the primary clock signal 3506 . Preferably, in this example, the clock buffer of the clock reduction logic 3504 simply boosts the voltage level of the main clock signal 3506 . In addition, before, the architecture program will execute the architecture instructions to write the data RAM 122 , the weight RAM 124 and write the NNU program into the program memory 129 . In response to the NNU program MTNN instruction 1400, the NNU 121 will start executing the NNU program at the dominant clock frequency because the relaxation pointer 3512 is programmed with the dominant clock frequency value. After the neural network unit 121 starts to execute, the architecture program will continue to execute the architecture instructions at the main time frequency, including mainly writing and/or reading the data random access memory 122 and the weight random access memory 124 with the MTNN instruction 1400, to complete For the next instance (instance) of the neural network unit program, it is also called invocation or preparation for execution (run).

In the example of FIG. 36A , compared to the time it takes for the architecture program to write/read data RAM 122 and weight RAM 124 , NNU 121 can execute in significantly less time (e.g. a quarter of the time) to complete the execution of the neural network unit program. For example, the NNU 121 takes about 1000 clock cycles to execute the NNU program while operating at the main clock rate, whereas the architectural program takes about 4000 clock cycles. In this way, the NNU 121 will be in the standby state for the rest of the time, which is a relatively long time in this example, such as about 3000 main clock cycles. As shown in the example of FIG. 36A , depending on the size and configuration of the neural network, the preceding pattern is repeated, possibly many times in a row. Since the NNU 121 is a relatively large and transistor-intensive functional unit in the processor 100, the operation of the NNU 121 will generate a large amount of heat, especially when operating at the main clock frequency.

FIG. 36B is a timing diagram showing an example of the operation of the processor 100 with the NNU 121 operating in the relaxed mode. The frequency of operation in the relaxed mode is lower than the main frequency. The timing diagram of FIG. 36B is similar to that of FIG. 36A. In FIG. 36A, the processor 100 executes the architectural program at the main clock frequency. This example assumes that the architecture program and NNU program in FIG. 36B are the same as those of FIG. 36A. However, before starting the NNU program, the architecture program executes the MTNN instruction 1400 to program the relaxation pointer 3512 with a value that causes the clock rate reduction logic 3504 to generate the second clock rate at a second clock rate less than the primary clock rate Signal 3508. That is, the architecture program will put the NNU 121 in the relaxed mode of FIG. 36B instead of the normal mode of FIG. 36A . In this way, the neural processing unit 126 executes the neural network unit program at the second frequency, and in the moderation mode, the second frequency is lower than the main frequency. This example assumes that the moderation pointer 3512 is programmed with a value that designates the secondary clock rate as one quarter of the primary clock rate. In this way, the time spent by the neural network unit 121 on executing the neural network unit program in the relaxed mode will be four times the time spent in the normal mode, as shown in FIG. 36A and FIG. The length of time that the network unit 121 is in the standby state is significantly shortened. In this way, the NNU 121 in FIG. 36B executes the NNU program and consumes energy for about four times as long as the NNU 121 in FIG. 36A executes the program in the normal mode. Therefore, the heat energy generated by the neural network unit 121 in FIG. 36B executing the neural network unit program per unit time is about a quarter of that in FIG. 36A , and has the advantages described herein.

FIG. 37 is a flowchart showing the operation of the processor 100 of FIG. 35 . The operation described in this flow chart is similar to the operation described above in relation to Fig. 35, Fig. 36A and Fig. 36B. The process starts at step 3702.

In step 3702 , the processor 100 executes the MTNN instruction 1400 to write the weights into the weight RAM 124 and write the data into the data RAM 122 . Then the flow goes to step 3704.

In step 3704, the processor 100 executes the MTNN instruction 1400 to program the relaxation pointer 3512 with a value specifying a clock frequency lower than the main clock frequency, ie, even if the NNU 121 is in relaxation mode. The process then proceeds to step 3706.

In step 3706, the processor 100 executes the MTNN instruction 1400 to instruct the NNU 121 to start executing the NNU program, that is, a manner similar to that shown in FIG. 36B. The process then proceeds to step 3708.

In step 3708, the NNU 121 starts executing the NNU program. At the same time, the processor 100 will execute the MTNN instruction 1400 to write the new weights into the weight random access memory 124 (may also write new data into the data random access memory 122), and/or execute the MFNN instruction 1500 to obtain The data RAM 122 reads the results (and possibly the weight RAM 124 as well). The process then proceeds to step 3712.

In step 3712, the processor 100 executes the MFNN instruction 1500 (for example, reads the status register 127) to detect that the neural network unit 121 has finished program execution. Assuming that the architectural program chooses a good value for the relaxation pointer 3512, the time it takes for the NNU 121 to execute the NNU program is the same as the processor 100 executes part of the architectural program to access the WRAM 124 and/or the DRAM. The time taken to access the memory 122 is shown in FIG. 36B. The process then proceeds to step 3714.

In step 3714, the processor 100 executes the MTNN instruction 1400 to program the relaxation pointer 3512 with a value specifying the main clock frequency, ie the NNU 121 is in normal mode. Proceed to step 3716 next.

In step 3716, the processor 100 executes the MTNN instruction 1400 to instruct the NNU 121 to start executing the NNU program, that is, a manner similar to that shown in FIG. 36A . The process then proceeds to step 3718.

In step 3718, the NNU 121 starts executing the NNU program in normal mode. The flow ends at step 3718.

As mentioned above, compared to executing the NNU program in the normal mode (that is, executing at the main clock frequency of the processor), the execution in the relaxed mode can spread the execution time and avoid generating high temperature. Furthermore, when the neural network unit executes the program in the relaxed mode, the neural network unit generates heat energy at a lower frequency, and the heat energy can pass smoothly through the neural network unit (such as semiconductor devices, metal layers and underlying substrates) It is exhausted from the surrounding package and cooling mechanism (such as heat sink, fan), and therefore, the devices (such as transistors, capacitors, wires) in the neural network unit are more likely to operate at a lower temperature. Overall, operating in moderation mode also helps reduce device temperatures in other parts of the processor die. The lower operating temperature, especially for the junction temperature of these devices, can reduce the generation of leakage current. In addition, since the amount of current flowing per unit time is reduced, the inductance noise and IR drop noise are also reduced. In addition, the temperature reduction also has a positive impact on the negative bias temperature instability (NBTI) and positive bias instability (PBSI) of the metal oxide semiconductor field effect transistor (MOSFET) in the processor, which can improve reliability and/or Or device and processor part life. The temperature is reduced and can mitigate Joule heating and electromigration effects within the metal layers of the processor.

Communication Mechanisms Between Architectural Programs and Non-Architectural Programs Regarding Shared Resources of Neural Network Units

As mentioned above, in the examples of FIGS. 24 to 28 and FIGS. 35 to 37 , the resources of the data RAM 122 and the weight RAM 124 are shared. The NPU 126 shares the data RAM 122 and the weight RAM 124 with the front end of the processor 100 . More precisely, the NPU 126 and the front end of the processor 100 , such as the media buffer 118 , both read and write to the data RAM 122 and the weight RAM 124 . In other words, the architecture program executed on the processor 100 and the NNU program executed on the NNU 121 will share the data RAM 122 and the weight RAM 124, and in some cases, as described above As mentioned above, it is necessary to control the flow between the architecture program and the neural network unit program. The resources of program memory 129 are also shared to some extent, since the architectural program writes to it and the sequencer 128 reads it. The embodiments described herein provide a high-performance solution to control the flow of accessing shared resources between the architecture program and the NNU program.

In the embodiments described herein, the NNU program is also referred to as a non-architectural program, the NNU instruction is also referred to as a non-architectural instruction, and the NNU instruction set (also referred to as an NPU instruction set as previously described) ) is also known as the non-architectural instruction set. A non-architectural instruction set is different from an architectural instruction set. In embodiments in which the processor 100 includes an instruction translator 104 to translate architectural instructions into microinstructions, the non-architectural instruction set is also different from the microinstruction set.

FIG. 38 is a block diagram showing the sequencer 128 of the neural network unit 121 in detail. Sequencer 128 provides memory addresses to program memory 129 to select non-architectural instructions to provide to sequencer 128, as previously described. As shown in FIG. 38 , the memory address is loaded into the program counter 3802 of the sequencer 128 . Sequencer 128 will normally increment in program memory 129 address order unless sequencer 128 encounters a non-architectural instruction, such as a loop or branch instruction, in which case sequencer 128 will update program counter 3802 to The target address of the control instruction, ie, the address of the non-architectural instruction that is updated to be the target of the control instruction. Thus, address 131 loaded in program counter 3802 specifies the address in program memory 129 of the non-architectural instructions of the non-architectural program currently being fetched for execution by NPU 126 . The value of the program counter 3802 can be obtained by the architecture program through the NNU program counter field 3912 of the state register 127, as described in FIG. 39 later. In this way, the architectural program can determine the location to read/write data to the data RAM 122 and/or the weight RAM 124 according to the progress of the non-architectural program.

The sequencer 128 also includes a loop counter 3804 that operates with non-architectural loop instructions, such as the loop to 1 instruction at address 10 in FIG. 26A and the loop to 1 instruction at address 11 in FIG. 28 . In the example of FIG. 26A and FIG. 28 , the loop counter 3804 is loaded with the value specified by the non-architectural initialization instruction at address 0, for example, the value 400 is loaded. Each time the sequencer 128 encounters a loop instruction and jumps to the target instruction (such as the multiply-accumulate instruction at address 1 in Figure 26A or the maxwacc instruction at address 1 in Figure 28), the sequencer 128 will cause the loop counter 3804 decrease. Once the loop counter 3804 is decremented to zero, the sequencer 128 moves to the next in-sequence non-architectural instruction. In another embodiment, the first time a loop instruction is encountered, the loop counter is loaded with the loop count value specified in a loop instruction, thereby eliminating the need to initialize the loop counter 3804 with a non-architectural initialization instruction. Thus, the value of the loop counter 3804 will indicate the number of times the non-architectural program's loop group has yet to be executed. The value of the loop counter 3804 can be obtained by the framework program through the loop count field 3914 of the state register 127, as shown in FIG. 39 . In this way, the architectural program can determine the location to read/write data to the data RAM 122 and/or the weight RAM 124 according to the progress of the non-architectural program. In one embodiment, the sequencer includes three additional loop counters for nested loops in non-architectural programs, and the values of these three loop counters can also be read through the status register 127 . There is a bit in the loop instruction to indicate which of the four loop counters is provided for the current loop instruction.

The sequencer 128 also includes an iteration count counter 3806 . The iteration count counter 3806 is paired with non-architectural instructions, such as the multiply-accumulate instruction at address 2 in FIG. 4, FIG. 9, FIG. 20 and FIG. 26A, and the maxwacc instruction at address 2 in FIG. "instruction. In the foregoing example, each execute instruction specifies execution counts 511, 511, 1023, 2, and 3, respectively. When the sequencer 128 encounters an execution instruction specifying a non-zero iteration count, the sequencer 128 loads the iteration count counter 3806 with the specified value. In addition, the sequencer 128 will generate appropriate micro-operations 3418 to control logic execution within the pipeline stage 3401 of the NPU 126 in FIG. 34 and to decrement the iteration counter 3806 . If the iteration counter 3806 is greater than zero, the sequencer 128 again generates appropriate micro-operations 3418 to control the logic within the NPU 126 and decrement the iteration counter 3806 . The sequencer 128 will continue to operate in this manner until the value of the iteration counter 3806 reaches zero. Therefore, the value of the iteration count counter 3806 is the number of operations specified in the non-architectural execution instruction to be executed (these operations are multiplication and accumulation of accumulated values and data/weight literals, taking the maximum value, summing operations, etc.). The value of the iteration count counter 3806 can be obtained through the iteration count field 3916 of the status register 127 by using the framework program, as described in FIG. 39 . In this way, the architectural program can determine the location to read/write data to the data RAM 122 and/or the weight RAM 124 according to the progress of the non-architectural program.

FIG. 39 is a block diagram showing several fields of the control and state register 127 of the NNU 121. These fields include the address 2602 of the WRAM column most recently written to by the NPU 126 executing the non-architectural program, the address 2604 of the WRAM column most recently read by the NPU 126 executing the non-architectural program, Unit 126 executes the address 2606 of the DRAM column most recently written by the non-architectural program, and NPU 126 executes the address 2608 of the DRAM column most recently read by the non-architectural program, as previously described in FIG. 26B . In addition, these fields also include a NNU program counter 3912 field, a loop counter 3914 field, and an iteration counter 3916 field. As mentioned above, the architecture program can read the data in the state register 127 to the media register 118 and/or the general register 116, for example, through the MFNN instruction 1500 to read the NNU program counter 3912, the loop counter 3914 and the iteration The value of the times counter 3916 field. The value of program counter field 3912 reflects the value of program counter 3802 in FIG. 38 . The value of loop counter field 3914 reflects the value of loop counter 3804 . The value of the iteration counter field 3916 reflects the value of the iteration counter 3806 . In one embodiment, the sequencer 128 updates the values of the program counter field 3912, the loop counter field 3914 and the iteration counter field 3916 each time the program counter 3802, the loop counter 3804, or the iteration count counter 3806 needs to be adjusted. In this way, the value of these fields will be the current value when the framework program reads it. In another embodiment, when the NNU 121 executes the architectural instruction to read the state register 127, the NNU 121 only fetches the values of the program counter 3802, the loop counter 3804 and the number of iterations counter 3806 and provides them back to the architecture instructions (eg, to media cache 118 or general cache 116).

From this, it can be found that the values of the fields of the state register 127 in FIG. 39 can be understood as the information of the execution progress of the non-architectural instruction in the process of being executed by the neural network unit. Some specific aspects about the progress of non-architectural program execution, such as program counter 3802 value, loop counter 3804 value, iteration counter 3806 value, last read/write weight RAM 124 address 125 field 2602/2604, And the fields 2606/2608 of the last read/written data RAM 122 address 123 have been described in previous chapters. Architectural programs executing on the processor 100 can read the non-architectural program progress values of FIG. 39 from the state register 127 and use this information to make decisions, eg, through architectural instructions such as compare and branch instructions. For example, the architecture program will determine the columns for data/weight read/write to data RAM 122 and/or weight RAM 124 to control data RAM 122 or weight RAM 124 The inflow and outflow of data to memory 124 is especially for large data sets and/or overlapping execution of different non-architectural instructions. Examples of these decisions using architectural procedures can be found in the preceding and following chapters of this paper.

For example, as described above in FIG. 26A , the architectural program sets the non-architectural program to write the result of the convolution operation back to the column above the convolution kernel 2402 in the DRAM 122 (such as above column 8), and when the neural When the network unit 121 writes a result using the address most recently written to the column 2606 of the DRAM 122 , the fabric program reads the result from the DRAM 122 .

In another example, the architected program utilizes information from the state register 127 field of FIG. 38 to confirm that the non-architected program divides the data array 2404 of FIG. 24 into five 512 x 1600 data blocks to perform the convolution, as previously described in FIG. 26B The progress of the operation. The architectural program writes the first 512x 1600 data block of the 2560x 1600 data array into WRAM 124 and starts the non-architectural program with a loop count of 1600 and WRAM 124 initialized output columns to 0. When the NNU 121 executes the non-architectural program, the architectural program reads the state register 127 to confirm the last written column 2602 of the weight RAM 124, so that the architectural program can read valid volumes written by the non-architectural program The result of the convolution operation, and after reading, use the next 512x 1600 data block to overwrite the effective convolution operation result, so that after the neural network unit 121 completes the execution of the first 512x 1600 data block by the non-architecture program, the processor 100 can immediately update the unarchitected program if necessary and start the unarchitected program again to execute the next 512x 1600 data block.

In another example, assume that the architecture program causes NNU 121 to perform a series of typical NNN multiply-accumulate activation functions, where weights are stored in weight RAM 124 and results are written back to data RAM 122 . In this case, once a column of the weight RAM 124 is read by the architecture program, it will not be read again. In this way, after the current weight has been read/used by the non-architecture program, the architecture program can be used to start to rewrite the weight on the weight random access memory 124 to provide the next example of the non-architecture program (such as the following A neural network layer) is used. In this case, the architecture program reads the state register 127 to obtain the address of the last read column 2604 of the WRAM to determine where to write the new weight group in the WRAM 124 .

In another example, assume that the architected program knows that the non-architected program includes an executed instruction with a large iteration count, such as the non-architected multiply-accumulate instruction at address 2 in FIG. 20 . In this case, the architectural program needs to know the iteration count 3916 to know approximately how many time-frequency cycles are still needed to complete the non-architectural instruction to determine which of the two or more actions the architectural program will take next . For example, an architectural program may relinquish control to another architectural program, such as an operating system, if it takes a long time to complete its execution. Similarly, assume that the architected program knows that the non-architected program includes a loop group with a relatively large loop count, such as the non-architected program of FIG. 28 . In this case, the architectural program needs to know the loop count 3914 in order to know approximately how many time-frequency cycles are still needed to complete the non-architectural instruction to determine which of two or more actions to take next.

In another example, assume that the architecture program causes the NNU 121 to perform a common-source operation similar to that described in FIGS. 27 and 28 , where the data to be shared is stored in the weight RAM 124 and the result is written back Weight random access memory 124 . However, unlike the examples of FIGS. 27 and 28 , it is assumed that the results of this example will be written back to the top 400 columns of the weight RAM 124 , such as columns 1600 to 1999 . In this case, after the non-architectural program completes reading the four columns of weight RAM 124 data that it wants to share, the non-architectural program will not read again. Therefore, once the current four columns of data have been read/used by the non-architecture program, the architecture program can be used to start weighting the new data (such as the weight of the next instance of the non-architecture program, for example, performing typical The non-architectural program of the multiply-accumulate start function operation) overwrites the data in the weight RAM 124 . In this case, the architecture program reads the state register 127 to obtain the address of the last read column 2604 of the weight RAM to determine the location where the new weight group is written into the weight RAM 124 .

Temporal recurrent neural network acceleration

Traditional feed-forward neural networks do not have memory to store previous inputs to the network. Feedforward neural networks are often used to perform tasks in which multiple inputs to the network over time are independent, and so are multiple outputs. In contrast, temporal recurrent neural networks are often helpful for tasks where the order of inputs to the neural network over time within the task is important. (The sequence here is often referred to as a time step.) Thus, a temporal recurrent neural network includes a conceptual memory, or internal state, to load information resulting from computations performed by the network in response to previous inputs in the sequence, temporal recurrent The output of the neural network is related to this internal state and the input of the next time step. Tasks such as speech recognition, language modeling, text generation, language translation, image description generation, and some forms of handwriting recognition are examples where temporal recurrent neural networks can perform well.

Three well-known examples of temporal recurrent neural networks are Elman temporal recurrent neural networks, Jordan temporal recurrent neural networks and long short-term memory (LSTM) neural networks. The Elman RTNN contains content nodes to remember the hidden layer state of the RTNN at the current time step, which will be used as input to the hidden layer at the next time step. Jordan RTNNs are similar to Elman RTNNs, except that the content nodes remember the output layer state of the RTNN instead of the hidden layer state. The LSTM neural network includes a LSTM layer composed of LSTM cells. Each LSTM cell has a current state and current output at the current time step, and a new state and new output at a new or subsequent time step. LSTM cells include input gates, output gates, and forget gates, which allow neurons to lose their memorized states. These three temporal recurrent neural networks will be described in more detail in subsequent chapters.

As described in this paper, for temporal recurrent neural networks, such as Elman or Jordan temporal recurrent neural networks, each execution of the neural network unit takes a set of input layer node values and performs the necessary calculations to pass through time. The recurrent neural network propagates to produce output layer node values as well as hidden and content layer node values. Thus, the input layer node values will be associated with the time steps at which hidden, output, and content layer node values are computed; and the hidden, output, and content layer node values will be associated with the time steps at which those node values were generated. The input layer node value is the sampling value of the system simulated by the time recurrent neural network, such as video, voice sampling, and snapshot of commercial market data. For LSTM neural networks, each execution of the neural network unit takes a time step, taking a set of cell input values and performing the necessary calculations to produce the cell output values (and cell state and input gates, forgetting gates and the output gate value), which can also be understood as propagating the input value of the memory cell through the memory cell of the long-term short-term memory layer. Therefore, the memory cell input value will be related to the time step of calculating the memory cell state and the input gate, forgetting gate and output gate value; and the memory cell state and the input gate, forgetting gate and output gate value will be related to the time when these node values are generated step.

Content layer node values, also known as state nodes, are the state values of the neural network based on the input layer node values associated with previous time steps, not just the current time step. The calculations performed by the NNU for a time step (eg, hidden layer node value calculations for Elman or Jordan time recurrent neural networks) are a function of the content layer node values produced by previous time steps. Therefore, the network state value (content node value) at the beginning of a time step affects the output layer node values produced during this time step. Additionally, the value of the network state at the end of a time step is affected by the values of the input nodes at this time step and the value of the network state at the beginning of the time step. Similarly, for LSTM cells, the state value of the memory cell is related to the input value of the memory cell at the previous time step, not only to the input value of the memory cell at the current time step. Because the calculation performed by the neural network unit for the time step (such as the calculation of the next memory cell state) is a function of the memory cell state value produced by the previous time step, the network state value (memory cell state value) at the beginning of the time step will affect this. The output value of the memory cell generated in the time step, and the network state value at the end of this time step will be affected by the input value of the memory cell at this time step and the previous network state value.

Fig. 40 is a block diagram showing an example of an Elman temporal recurrent neural network. The Elman temporal recurrent neural network of FIG. 40 includes input layer nodes, or neurons, labeled D0, D1 through Dn, collectively referred to as a plurality of input layer nodes D and individually collectively referred to as input layer nodes D; hidden layer nodes/neurons, Marked as Z0, Z1 to Zn, collectively referred to as multiple hidden layer nodes Z and individually collectively referred to as hidden layer nodes Z; output layer nodes/neurons, marked as Y0, Y1 to Yn, collectively referred to as multiple output layer nodes Y Individually referred to as an output layer node Y; and content layer nodes/neurons, denoted as C0, C1 to Cn, collectively referred to as a plurality of content layer nodes C and individually referred to as a content layer node C. In the example of the Elman time recurrent neural network of Figure 40, each hidden layer node Z has an input connected to the output of each input layer node D, and has an input connected to the output of each content layer node C; each output layer node Y There is an input connected to the output of each hidden layer node Z; and each content layer node C has an input connected to the output of the corresponding hidden layer node Z.

In many respects, Elman temporal recurrent neural networks operate similar to traditional feed-forward artificial neural networks. That is, for a given node, each input link of this node will have an associated weight; the value received by the node at an input link will be multiplied by the associated weight to produce a product; the node will The products associated with all input links are summed to produce a total (possibly including an offset term in this total); generally speaking, the activation function is also executed on this total to produce the node's output value, which is sometimes called The startup value for this node. For traditional feedforward networks, data always flows in the direction from the input layer to the output layer. That is, the input layer provides a value to the hidden layer (often there are multiple hidden layers), the hidden layer produces its output value to the output layer, and the output layer produces an output that can be accessed.

However, unlike the traditional feedforward network, the Elman temporal recurrent neural network also includes some feedback connections, that is, the connection from hidden layer node Z to content layer node C in Fig. 40 . The operation of the Elman time recurrent neural network is as follows. When the input layer node D provides an input value to the hidden layer node Z at a new time step, the content node C will provide a value to the hidden layer Z. This value is the hidden layer node Z corresponding to The output value of the previous input, that is, the current time step. In this sense, the content node C of the Elman temporal recurrent neural network is a memory based on the input values at previous time steps. 41 and 42 will illustrate an example of the operation of the neural network unit 121 performing the computation associated with the Elman temporal recurrent neural network of FIG. 40 .

To illustrate the present invention, an Elman temporal recurrent neural network is a temporal recurrent neural network comprising at least one input node layer, one hidden node layer, one output node layer and one content node layer. For a given time step, the content node layer stores the results generated by the hidden node layer at the previous time step and fed back to the content node layer. The result fed back to the content layer may be the execution result of the activation function or the result of the accumulation operation performed by the hidden node layer without executing the activation function.

41 is a block diagram showing the data configuration in the data random access memory 122 and the weight random access memory 124 of the neural network unit 121 when the neural network unit 121 performs calculations associated with the Elman time recurrent neural network of FIG. 40 An example of . In the example of FIG. 41 , it is assumed that the Elman temporal recurrent neural network in FIG. 40 has 512 input nodes D, 512 hidden nodes Z, 512 content nodes C, and 512 output nodes Y. In addition, it is also assumed that this Elman time recurrent neural network is fully connected, that is, all 512 input nodes D are connected to each hidden node Z as input, all 512 content nodes C are connected to each hidden node Z as input, and all 512 hidden nodes Node Z is connected to each output node Y as an input. In addition, the neural network unit 121 is configured as 512 neural processing units 126 or neurons, such as a wide configuration. Finally, this example assumes that the links associated with content node C to hidden node Z all have a weight of 1, so there is no need to store these weight values of 1.

As shown in the figure, the lower 512 columns (columns 0 to 511 ) of the weight RAM 124 are loaded with weight values associated with the links between the input node D and the hidden node Z. More precisely, as shown in the figure, column 0 holds the weights associated with the input link from input node D0 to hidden node Z, i.e., text 0 would hold the weights associated with the link between input node D0 and hidden node Z0 Weight, text 1 will load the weight associated with the link between input node D0 and hidden node Z1, text 2 will load the weight associated with the link between input node D0 and hidden node Z2, and so on, text 511 will load the weight associated with The weight of the link between input node D0 and hidden node Z511; column 1 loads the weight associated with the input link from input node D1 to hidden node Z, that is, text 0 will load the weight associated with input node D1 and hidden node Z0 Link weights, text 1 will load the weight associated with the link between input node D1 and hidden node Z1, text 2 will load the weight associated with the link between input node D1 and hidden node Z2, and so on, text 511 will load Weights associated with the link between input node D1 and hidden node Z511; up to column 511, column 511 loads the weights associated with the input link from input node D511 to hidden node Z, i.e. a literal 0 would load the weight associated with input node D511 The weight of the connection with the hidden node Z0, text 1 will load the weight associated with the connection between the input node D511 and the hidden node Z1, text 2 will load the weight associated with the connection between the input node D511 and the hidden node Z2, and so on By analogy, the text 511 will carry the weight associated with the link between the input node D511 and the hidden node Z511. This configuration and usage are similar to the embodiments described above with respect to FIGS. 4 to 6A .

As shown in the figure, the subsequent 512 columns (columns 512 to 1023) of the weight RAM 124 are similarly loaded with weights associated with the link between the hidden node Z and the output node Y.

Data RAM 122 is loaded with Elman time recurrent neural network node values for a series of time steps. Further, the DRAM 122 is loaded in groups of three columns to provide node values for a given time step. As shown in the figure, taking a DRAM 122 with 64 columns as an example, the DRAM 122 can be loaded with node values for 20 different time steps. In the example in Figure 41, columns 0 to 2 hold node values for time step 0, columns 3 to 5 for time step 1, and so on, columns 57 to 59 for time step 19 node value. The first column in each group holds the value of the input node D for this time step. The second column in each group holds the value of hidden node Z for this time step. The third column in each group holds the value of the output node Y for this time step. As shown, each row of DRAM 122 is loaded with the node value of its corresponding neuron or NPU 126 . That is, row 0 loads the node values associated with nodes D0, Z0, and Y0, whose calculations are performed by NPU0; row 1 loads the node values associated with nodes D1, Z1, and Y1, whose calculations are performed by NPU Executed by processing unit 1; and so on, line 511 loads the node values associated with nodes D511, Z511 and Y511, and its calculation is performed by neural processing unit 511. This part will be described in more detail later corresponding to FIG. 42 illustrate.

As indicated in FIG. 41, for a given time step, the value of the hidden node Z in the second column of each set of three-column memories will be the value of the content node C at the next time step. That is to say, the value of node Z calculated and written by the neural processing unit 126 in one time step will become the value of node C used by the neural processing unit 126 to calculate the value of node Z in the next time step ( together with the value of input node D for this next time step). The initial value of content node C (the value of node C used at time step 0 to calculate the value of node Z in column 1) is assumed to be zero. This will be described in more detail in subsequent relevant chapters corresponding to the non-architecture program in FIG. 42 .

Preferably, the value of the input node D (column 0, 3 in the example of FIG. 41 , and so on to the value of column 57) is written/filled by the architectural program executing on the processor 100 through the MTNN instruction 1400 The RAM 122 is read/used by non-architectural programs executed on the NNU 121, such as the non-architectural program in FIG. 42 . Conversely, the values of hidden/output nodes Z/Y (columns 1 and 2, 4 and 5 in the example of FIG. The architectural program is written/filled into the DRAM 122 and is read/used by the architectural program executing on the processor 100 through the MFNN instruction 1500 . The example of FIG. 41 assumes that the architecture program will perform the following steps: (1) For 20 different time steps, fill the data random access memory 122 with the value of the input node D (column 0, 3, and so on to column 57 ); (2) start the non-architecture program of FIG. and (5) repeat steps (1) to (4) several times until the task is completed, such as the calculation required to recognize the speech of the mobile phone user.

In another execution mode, the architecture program will perform the following steps: (1) for a single time step, fill the data random access memory 122 (such as column 0) with the value of the input node D; (2) start the non-architecture Program (the modified version of the non-architecture program in Fig. 42 does not need a loop, and only accesses a single group of three columns of the data random access memory 122); (3) detect whether the non-architecture program has been executed; (4) from the data The random access memory 122 reads out the value of the output node Y (eg column 2); and (5) repeat steps (1) to (4) several times until the task is completed. Which of the two methods is better depends on the sampling method of the input value of the temporal recurrent neural network. For example, if the task allows the input to be sampled and the computation to be sampled at multiple time steps (say around 20 time steps), the first approach is ideal as it may lead to more computational resource efficiencies and/or or better performance, but if the task only allows sampling to be performed at a single time step, then the second approach is required.

The third embodiment is similar to the aforementioned second method, but unlike the second method using a single set of three-column data random access memory 122, the non-architectural program of this method uses multiple sets of three-column memory, that is, at each time The steps use different sets of three-column memory, this part is similar to the first way. In this third embodiment, preferably, the architecture program includes a step before step (2). In this step, the architecture program will update the non-architecture program before it starts, for example, the address 1 in the instruction The DRAM 122 ranks are updated to point to the next set of three ranks.

FIG. 42 is a table showing a program stored in the program memory 129 of the neural network unit 121. The program is executed by the neural network unit 121 and uses data and weights according to the configuration of FIG. 41 to achieve an Elman time recurrent neural network. Some instructions in the non-architectural program of Figure 42 (and Figure 45, Figure 48, Figure 51, Figure 54 and Figure 57) are as described above (such as multiplication and accumulation (MULT-ACCUM), loop (LOOP), initialization (INITIALIZE) instructions ), the following paragraphs assume that these instructions are consistent with the preceding instructions unless stated otherwise.

The example program in Figure 42 contains 13 non-architectural instructions located at addresses 0 to 12, respectively. The instruction at address 0 (INITIALIZE NPU, LOOPCNT=20) clears accumulator 202 and initializes loop counter 3804 to a value of 20 to execute the loop group (instructions at addresses 4 to 11) 20 times. Preferably, the initialization instruction also makes the NNU 121 in a wide configuration, so that the NNU 121 is configured as 512 NPUs 126 . As described in subsequent chapters, during the execution of instructions at addresses 1 to 3 and addresses 7 to 11, the 512 neural processing units 126 operate as 512 corresponding hidden layer nodes Z, while instructions at addresses 4 to 6 During execution, the 512 neural processing units 126 operate as 512 corresponding output layer nodes Y.

Instructions at addresses 1 to 3 do not belong to the loop group of the program and are executed only once. These instructions calculate the initial value of the hidden layer node Z and write it to column 1 of the data random access memory 122 for the first execution of the instructions at addresses 4 to 6 to calculate the first time step (time step 0) The output layer node Y of . In addition, the value of the hidden layer node Z calculated by the instructions of addresses 1 to 3 and written to column 1 of the DRAM 122 will become the value of the content layer node C for the first time of the instructions of addresses 7 and 8 Execute use to calculate the value of hidden layer node Z for the second time step (time step 1).

During the execution of the instructions at addresses 1 and 2, each neural processing unit 126 in the 512 neural processing units 126 will perform 512 multiplication operations, and the 512 input node D values located at column 0 of the data random access memory 122 The weights corresponding to the rows of the NPU 126 in columns 0 to 511 of the WRAM 124 are multiplied to generate 512 multiplications that are accumulated in the accumulator 202 of the corresponding NPU 126 . During the execution of the instruction at address 3, the values of the 512 accumulators 202 of the 512 NPUs will be transferred and written into column 1 of the DRAM 122 . That is to say, the output command at address 3 will write the value of the accumulator 202 of each neural processing unit 512 in the 512 neural processing units into column 1 of the data random access memory 122, and this value is the initial hidden layer Z value , then this instruction clears accumulator 202.

The operations performed by the instructions at addresses 1 to 2 of the non-architectural program of FIG. 42 are similar to the operations performed by the instructions at addresses 1 to 2 of the non-architectural instructions of FIG. 4 . Further, the instruction at address 1 (MULT_ACCUM DR ROW 0) will instruct each neural processing unit 126 in the 512 neural processing units 126 to read the corresponding text of row 0 of the data random access memory 122 into its multitasking cache The device 208 reads the corresponding word of column 0 of the weight random access memory 124 into its multitasking register 705 , multiplies the data word and the weight word to generate a product and adds the product to the accumulator 202 . The instruction of address 2 (MULT-ACCUM ROTATE, WR ROW+1, COUNT=511) instructs each NPU 126 in these 512 NPUs to transfer the text from adjacent NPUs 126 into its multitasking buffer 208 (utilizing the 512 word rotators collectively operated by the 512 multitasking registers 208 of the neural network unit 121, these registers being the cache into which the instruction at address 1 instructs to read the columns of the data random access memory 122 device), read the corresponding text of the next column of the weight random access memory 124 into its multitasking register 705, multiply the data text and the weight text to generate a product and add this product to the accumulator 202, and perform the aforementioned operation 511 Second-rate.

In addition, a single non-architectural output instruction (OUTPUT PASSTHRU, DR OUT ROW 1, CLRACC) at address 3 in FIG. The program passes the value of the accumulator 202, while the program of FIG. 4 executes the start function on the value of the accumulator 202). That is to say, in the program of Figure 42, the startup function executed on the value of the accumulator 202, if any, is specified in the output instruction (also specified in the output instruction of addresses 6 and 11), rather than as shown in Figure 4 The procedure shown is specified in a different non-architectural start function directive. Another embodiment of the non-architectural program of FIG. 4 (and FIG. 20, FIG. 26A and FIG. 28), that is, the operation of the start function instruction and the write output instruction (such as address 3 and 4 of FIG. 4) are combined into one shown in FIG. 42 The single non-architectural output instruction shown is also within the scope of the invention. The example of FIG. 42 assumes that the nodes of the hidden layer (Z) do not perform activation functions on accumulator values. However, embodiments in which the hidden layer (Z) performs activation functions on accumulator values are also within the scope of the present invention. These embodiments can use the instructions at addresses 3 and 11 to perform operations such as sigmoid, hyperbolic tangent, correction functions, etc.

Compared with the instructions at addresses 1 to 3 which are only executed once, the instructions at addresses 4 to 11 are located in the program loop and will be executed a number of times specified by the loop count (for example, 20). The first nineteen executions of the instructions at addresses 7 to 11 calculate the value of hidden layer node Z and write it into the DRAM 122 for the second to twenty executions of the instructions at addresses 4 to 6 to calculate the remaining time Output layer node Y for step (time steps 1 to 19). (The last/twentieth execution of the instructions at addresses 7 to 11 computes and writes the values for hidden layer node Z to column 61 of DRAM 122, however, these values are not used.)

In the first execution (corresponding to time step 0) of the instructions at addresses 4 and 5 (MULT-ACCUM DR ROW+1, WR ROW 512 and MULT-ACCUMROTATE, WR ROW+1, COUNT=511), these 512 Each neural processing unit 126 in the neural processing unit 126 will perform 512 multiplication operations, and the value of the 512 hidden nodes Z of column 1 of the data random access memory 122 (these values are determined by a single execution of the instructions of addresses 1 to 3 and generate and write) are multiplied by the weights corresponding to the rows of the NPU 126 in the columns 512 to 1023 of the weighted random access memory 124 to generate 512 multiplications that are accumulated in the accumulator 202 of the corresponding NPU 126 . In the first execution of the instruction at address 6 (OUTPUT ACTIVATIONFUNCTION, DR OUT ROW+1, CLR ACC), the activation function (such as sigmoid, hyperbolic tangent, correction function) will be executed for these 512 accumulated values to calculate the output The value of the layer node Y, the execution result will be written into column 2 of the data random access memory 122 .

In the second execution of the instructions at addresses 4 and 5 (corresponding to time step 1), each of the 512 neural processing units 126 will perform 512 multiplication operations, and the data random access memory 122 The values of 512 hidden nodes Z in column 4 (these values are generated and written by the first execution of the instructions at addresses 7 to 11) are multiplied by weights corresponding to this NPU in columns 512 to 1023 of random access memory 124 126 row weights to generate 512 multiplications and accumulate them in the accumulator 202 of the corresponding neural processing unit 126, and in the second execution of the instruction at address 6, the activation function will be executed for these 512 accumulated values to calculate the output The value of layer node Y, this result is written into column 5 of data random access memory 122; Each neural processing unit 126 will perform 512 multiplication operations, and write the values of 512 hidden nodes Z in column 7 of the DRAM 122 (these values are generated and written by the second execution of the instructions at addresses 7 to 11) ) is multiplied by the weight corresponding to the row of the neural processing unit 126 in the columns 512 to 1023 of the weight random access memory 124, to generate 512 multiplications accumulated in the accumulator 202 of the corresponding neural processing unit 126, and the instruction at address 6 In the third execution of , the startup function will be executed for these 512 accumulated values to calculate the value of the node Y of the output layer, and this result will be written into column 8 of the data random access memory 122; and so on, at addresses 4 and 5 In the twentieth execution of the instruction (corresponding to time step 19), each neural processing unit 126 in the 512 neural processing units 126 will perform 512 multiplication operations, and 512 columns 58 of the data random access memory 122 The value of hidden node Z (these values are generated and written by the nineteenth execution of the instructions at addresses 7 to 11) is multiplied by the weight of weights, so as to generate 512 multiplications and accumulate them in the accumulator 202 of the corresponding neural processing unit 126, and in the twentieth execution of the instruction at address 6, the activation function will be executed for these 512 accumulated values to calculate the output layer node Y The value of , the execution result is written into the column 59 of the data random access memory 122 .

In the first execution of the instructions at addresses 7 and 8, each of the 512 NPUs 126 accumulates the values of the 512 content nodes C of column 1 of the DRAM 122 to its accumulator register 202, these values are generated by a single execution of instructions at addresses 1-3. Further, the instruction at address 7 (ADD_D_ACC DRROW+0) will instruct each neural processing unit 126 in the 512 neural processing units 126 to put the data random access memory 122 into the current column (in the process of the first execution, it is The corresponding text of column 0) is read into its multitasking register 208, and this text is added to the accumulator 202. The instruction of address 8 (ADD_D_ACC ROTATE, COUNT=511) instructs each neural processing unit 126 in these 512 neural processing units 126 to transfer the text from the adjacent neural processing unit 126 into its multitasking register 208 (utilized by the neural network The 512 multitasking registers 208 of the unit 121 are collectively operated to form a 512-word rotator, and these multitasking registers are the registers that the instruction at address 7 indicates to read into the column of the data random access memory 122), and this The text is added to the accumulator 202, and the aforementioned operation is performed 511 times.

In the second execution of the instructions at addresses 7 and 8, each of the 512 NPUs 126 will accumulate the values of the 512 content nodes C in column 4 of the DRAM 122 to Its accumulator 202, these values are generated and written by the first execution of the instructions of addresses 9 to 11; in the third execution of the instructions of addresses 7 and 8, each of the 512 neural processing units 126 The processing unit 126 will accumulate the values of the 512 content nodes C in column 7 of the data random access memory 122 to its accumulator 202, and these values are generated and written in by the second execution of the instructions of addresses 9 to 11; By analogy, in the twentieth execution of the instructions at addresses 7 and 8, each of the 512 NPUs 126 will transfer the 512 content nodes of column 58 of the DRAM 122 to The value of C is accumulated to its accumulator 202, and these values are generated and written by the nineteenth execution of the instructions at addresses 9-11.

As before, the example of FIG. 42 assumes that the weight associated with the link from content node C to hidden layer node Z has a value of one. However, in another embodiment, the connections in the Elman time recurrent neural network have non-zero weight values, and these weights are placed in the weight random access memory 124 (such as columns 1024 to 1535) before the program of FIG. 42 is executed. ), the program instruction at address 7 is MULT-ACCUM DR ROW+0, WR ROW 1024, and the program instruction at address 8 is MULT-ACCUM ROTATE, WR ROW+1, COUNT=511. Preferably, the instruction at address 8 does not access the weight RAM 124 , but the instruction at address 7 rotates the value read from the weight RAM 124 into the multitasking register 705 . More bandwidth can be reserved for the architecture program to access the weight RAM 124 by not accessing the weight RAM 124 within 511 time-frequency cycles of executing the address 8 instruction.

In the first execution (corresponding to time step 1) of the instructions at addresses 9 and 10 (MULT-ACCUM DR ROW+2, WR ROW 0 and MULT-ACCUM ROTATE, WR ROW+1, COUNT=511), this 512 Each of the NPUs 126 performs 512 multiplication operations to multiply the values of the 512 input nodes D in column 3 of the data RAM 122 by columns 0 to 511 of the weight RAM 124 The weights corresponding to the row of the neural processing unit 126 to generate 512 products, together with the accumulation operation performed by the instructions of addresses 7 and 8 on the 512 values of the content node C, are accumulated in the accumulator 202 corresponding to the neural processing unit 126 to To calculate the value of hidden layer node Z, in the first execution of the instruction at address 11 (OUTPUT PASSTHRU, DR OUT ROW+2, CLR ACC), the 512 accumulator 202 values of the 512 neural processing units 126 are passed and Write to column 4 of data random access memory 122, and accumulator 202 will be cleared; In the second execution (corresponding to time step 2) of the instruction of address 9 and 10, the Each neural processing unit 126 will perform 512 multiplication operations, and multiply the values of the 512 input nodes D in column 6 of the data random access memory 122 by the weight corresponding to the neural processing unit 126 in columns 0 to 511 of the random access memory 124 to generate 512 products, together with the accumulative operation performed by the instructions at addresses 7 and 8 on the 512 values of content node C, they are accumulated in the accumulator 202 of the corresponding neural processing unit 126 to calculate the value of hidden layer node Z value, in the second execution of the instruction at address 11, the 512 accumulator 202 values of the 512 neural processing units 126 are transferred and written into the column 7 of the data random access memory 122, and the accumulator 202 will be clear; and so on, in the nineteenth execution (corresponding to time step 19) of the instruction of address 9 and 10, each neural processing unit 126 in these 512 neural processing units 126 can execute 512 multiplication operations, will The values of the 512 input nodes D in column 57 of the data RAM 122 are multiplied by the weights of the row corresponding to this neural processing unit 126 in columns 0 to 511 of the weight RAM 124 to produce 512 products, along with the address The accumulative operation performed by instructions 7 and 8 on the value of 512 content nodes C is accumulated in the accumulator 202 of the corresponding neural processing unit 126 to calculate the value of hidden layer node Z, and the nineteenth time of the instruction at address 11 During execution, the 512 accumulators 202 values of the 512 NPUs 126 are passed and written into the column 58 of the DRAM 122, and the accumulators 202 are cleared. As mentioned above, the value of hidden layer node Z generated and written in the twentieth execution of the instructions at addresses 9 and 10 is not used.

The instruction at address 12 (LOOP 4) will decrement the loop counter 3804 and fall back to the instruction at address 4 if the new loop counter 3804 value is greater than zero.

FIG. 43 is a block diagram showing an example of a Jordan temporal recurrent neural network. The Jordan temporal recurrent neural network of Figure 43 is similar to the Elman temporal recurrent neural network of Figure 40, with input layer node/neuron D, hidden layer node/neuron Z, output layer node/neuron Y, and content layer node/neuron Yuan C. However, in the Jordan time recurrent neural network in Figure 43, the content layer node C uses the output feedback from its corresponding output layer node Y as its input connection, instead of the hidden layer node in the Elman time recurrent neural network as shown in Figure 40 The output of Z is connected as its input.

To illustrate the present invention, a Jordan temporal recurrent neural network is a temporal recurrent neural network comprising at least one input node layer, one hidden node layer, one output node layer and one content node layer. At the beginning of a given time step, the content node layer stores the results generated by the output node layer at the previous time step and fed back to the content node layer. The result fed back to the content layer can be the result of the activation function or the result of the accumulation operation performed by the output node layer without executing the activation function.

44 is a block diagram showing the data configuration in the data random access memory 122 and weight random access memory 124 of the neural network unit 121 when the neural network unit 121 performs calculations associated with the Jordan Time Recurrent Neural Network of FIG. 43 An example of . In the example of FIG. 44 , it is assumed that the Jordan time recurrent neural network in FIG. 43 has 512 input nodes D, 512 hidden nodes Z, 512 content nodes C, and 512 output nodes Y. In addition, it is also assumed that this Jordan time recurrent neural network is fully connected, that is, all 512 input nodes D are connected to each hidden node Z as input, all 512 content nodes C are connected to each hidden node Z as input, and all 512 hidden nodes Node Z is connected to each output node Y as an input. Although the example of the Jordan time recurrent neural network in Figure 44 will apply the activation function to the value of the accumulator 202 to generate the value of the output layer node Y, however, this example assumes that the value of the accumulator 202 before the activation function is applied to the content Layer node C, not the real output layer node Y value. Furthermore, the neural network unit 121 is provided with 512 neural processing units 126 , or neurons, for example in a wide configuration. Finally, this example assumes that the weights associated with the links from content node C to hidden node Z all have a value of 1; thus there is no need to store these weight values as one.

Like the example in FIG. 41 , as shown in the figure, the lower 512 columns (columns 0 to 511) of the WRAM 124 will be loaded with the weight values associated with the connection between the input node D and the hidden node Z, and the WRAM 124 The next 512 columns (columns 512 to 1023) of fetch memory 124 are loaded with weight values associated with the link between hidden node Z and output node Y.

The data random access memory 122 is loaded with Jordan time recurrent neural network node values for a series of time steps similar to the example in FIG. 41; however, the example of FIG. node value. As shown in the figure, in an embodiment of DRAM 122 having 64 columns, DRAM 122 can be loaded with node values required for 15 different time steps. In the example in Figure 44, columns 0 to 3 hold node values for time step 0, columns 4 to 7 for time step 1, and so on, columns 60 to 63 for time step 15 node value. The first column of this quad holds the value of the input node D for this time step. The second column of the quad holds the value of hidden node Z for this time step. The third column of the quad holds the value of the content node C for this time step. The fourth column of the quad is loaded with the value of the output node Y for this time step. As shown, each row of DRAM 122 is loaded with the node value of its corresponding neuron or NPU 126 . That is to say, row 0 loads the node values associated with nodes D0, Z0, C0, and Y0, and its computation is performed by NPU 0; row 1 loads the node values associated with nodes D1, Z1, C1, and Y1, and its computation is executed by NPU 1; and so on, line 511 loads the node values associated with nodes D511, Z511, C511 and Y511, and its computation is executed by NPU 511. This part will be described in more detail later corresponding to FIG. 44 .

The value of content node C at a given time step in Figure 44 is generated during this time step and serves as the input for the next time step. That is to say, the value of node C calculated and written by the neural processing unit 126 in this time step will become the value of node C used by the neural processing unit 126 to calculate the value of node Z in the next time step ( together with the value of input node D for this next time step). The initial value of content node C (ie, the value of node C used to calculate the value of column 1 node Z at time step 0) is assumed to be zero. This part will be described in more detail in the subsequent chapter corresponding to the non-architectural program of Figure 45.

As previously described in FIG. 41, preferably, the value of the input node D (column 0, 4 in the example of FIG. 44, and so on to the value of column 60) is executed by the architecture program on the processor 100 through the MTNN instruction 1400 is written/filled into the DRAM 122 and read/used by non-architectural programs executing on the NNU 121, such as the non-architectural program of FIG. 45 . On the contrary, the value of hidden node Z/content node C/output node Y (in the example of Figure 44, columns 1/2/3, 5/6/7, and so on to the values of columns 61/62/63) The DRAM 122 is written/filled by the non-architectural program executing on the NNU 121 and read/used by the architectural program executing on the processor 100 via the MFNN instruction 1500 . The example of FIG. 44 assumes that the architecture program will perform the following steps: (1) For 15 different time steps, fill the data random access memory 122 with the value of the input node D (column 0, 4, and so on to column 60 ); (2) start the non-architecture program of FIG. and (5) repeat steps (1) to (4) several times until the task is completed, such as the calculation required to recognize the utterance of the mobile phone user.

In another execution mode, the architecture program will perform the following steps: (1) for a single time step, fill the data random access memory 122 (such as column 0) with the value of the input node D; (2) start the non-architecture Program (the modified version of the non-architecture program of Fig. 45, without looping, and only accessing a single group of four columns of the DRAM 122); (3) detecting whether the non-architecture program has been executed; (4) from the data random access Access memory 122 to read out the value of output node Y (eg column 3); and (5) repeat steps (1) to (4) several times until the task is completed. Which of the two methods is better depends on the sampling method of the input value of the temporal recurrent neural network. For example, if the task allows the input to be sampled and the calculation to be performed over multiple time steps (say, around 15 time steps), the first approach is ideal because it leads to more computational resource efficiency and / or better performance, but if the task only allows sampling to be performed in a single time step, then the second method is required.

The third embodiment is similar to the aforementioned second method, but unlike the second method using a single group of four data random access memory 122 columns, the non-architectural program of this method uses multiple groups of four-column memories, that is, in each Time steps use different sets of quad-column memory, this part is similar to the first way. In this third embodiment, preferably, the architecture program includes a step before step (2). In this step, the architecture program will update the non-architecture program before it starts, for example, the address 1 in the instruction The data RAM 122 ranks are updated to point to the next set of four ranks.

FIG. 45 is a table showing the program stored in the program memory 129 of the neural network unit 121. The program is executed by the neural network unit 121 and uses data and weights according to the configuration of FIG. 44 to achieve the Jordan time recurrent neural network. The non-architectural program in Figure 45 is similar to the non-architectural program in Figure 42, and the differences between the two can refer to the descriptions in relevant sections of this document.

The example program in FIG. 45 includes 14 non-architectural instructions located at addresses 0 to 13, respectively. The instruction at address 0 is an initialization instruction to clear the accumulator 202 and initialize the loop counter 3804 to a value of 15 to execute the group of 15 loops (instructions at addresses 4 to 12). Preferably, the initialization instruction also causes the NNU 121 to be configured in a wide configuration as 512 NPUs 126 . As described herein, during the execution of instructions at addresses 1 to 3 and addresses 8 to 12, the 512 neural processing units 126 correspond to and operate as 512 hidden layer nodes Z, while instructions at addresses 4, 5 and 7 During execution, the 512 neural processing units 126 correspond to and operate as 512 output layer nodes Y.

The commands of addresses 1 to 5 and address 7 are the same as the commands of addresses 1 to 6 in FIG. 42 and have the same functions. The instructions at addresses 1 to 3 calculate the initial value of node Z of the hidden layer and write it into column 1 of the DRAM 122 for the first execution of the instructions at addresses 4, 5 and 7 to calculate the first time Output layer node Y at step (time step 0).

During the first execution of the output instruction at address 6, the 512 accumulator 202 values generated by the instructions at addresses 4 and 5 are accumulated (these values are then used by the output instruction at address 7 to calculate and write The value of the output layer node Y) will be transferred and written into column 2 of the data random access memory 122, these values are the value of the content layer node C generated in the first time step (time step 0) and will be generated in the second time step (time step 1); during the second execution of the output instruction at address 6, these 512 accumulator 202 values generated by the accumulation of instructions at addresses 4 and 5 (next, these values will be stored at address 7 The output instruction used to calculate and write the value of the output layer node Y) will be passed and written into the column 6 of the data random access memory 122, these values are the content layer generated in the second time step (time step 1) The value of node C is used in the third time step (time step 2); and so on, during the fifteenth execution of the output instruction at address 6, these 512 instructions generated by the accumulation of addresses 4 and 5 Accumulator 202 values (which are then used by the output instruction at address 7 to calculate and write the value of output layer node Y) are passed and written to column 58 of DRAM 122, and these values are the first The content layer node C value generated in fifteen time steps (time step 14) (and read by the instruction at address 8, but not used).

The commands at addresses 8 to 12 are substantially the same as the commands at addresses 7 to 11 in FIG. 42 and have the same functions, with only one difference. The difference is that the instruction (ADD_D_ACC DR ROW+1) at address 8 in FIG. The column number of the data random access memory 122 is incremented by zero. This difference is due to the difference in data configuration in the data random access memory 122. In particular, the configuration of a group of four columns in FIG. ), while the configuration of three columns in Figure 41 does not have this independent column, but allows the value of the content layer node C and the value of the hidden layer node Z to share the same column (such as columns 1, 4, 7, etc.). Fifteen executions of the instructions at addresses 8 to 12 compute the value of node Z of the hidden layer and write it to DRAM 122 (write to columns 5, 9, 13, and so on up to column 57) for address 4 , the second to sixteenth executions of the instructions of 5 and 7 are used to compute the output layer node Y for the second to fifteenth time steps (time steps 1 to 14). (The last/fifteenth execution of the instructions at addresses 8-12 computes and writes the values for hidden layer node Z to column 61 of DRAM 122, but these values are not used.)

The loop instruction at address 13 will decrement the loop counter 3804 and fall back to the instruction at address 4 if the new loop counter 3804 value is greater than zero.

In another embodiment, the design of the Jordan time recurrent neural network uses the content node C to load the activation function value of the output node Y, and the activation function value is the accumulated value after the activation function is executed. In this embodiment, because the output node Y has the same value as the content node C, the non-architectural instruction at address 6 is not included in the non-architectural program. Therefore, the number of columns used in the data random access memory 122 can be reduced. To be more precise, the columns (for example, columns 2, 6, and 59) loaded with the value of the content node C in FIG. 44 do not exist in this embodiment. In addition, each time step of this embodiment only needs three columns of the DRAM 122, and 20 time steps will be used instead of 15, and the address of the instruction of the non-architectural program in FIG. 45 will also be properly adjusted. .

long short-term memory cells

The use of LSTM cells in temporal recurrent neural networks is a well-known concept in the art. For example, Long Short-Term Memory, Sepp Hochreiter and Jürgen Schmidhuber, Neural Computation, November 15, 1997, Vol.9, No.8, Pages 1735-1780; Learning to Forget: Continual Prediction with LSTM, Felix A. Gers, Jürgen Schmidhuber, and Fred Cummins, Neural Computation, October 2000, Vol.12, No.10, Pages 2451-2471; all available from MIT Press Journals. LSTM cells can be constructed in many different types. The LSTM cells 4600 of FIG. 46 described below are LSTMs as described in the tutorial at http://deeplearning.net/tutorial/lstm.html titled LSTM Networks for Sentiment Analysis. Memory cells are models, and a copy of this tutorial was downloaded on October 19, 2015 (hereinafter referred to as the "Long Short-Term Memory Tutorial") and provided in the U.S. Application Data Disclosure Report of this case. This LSTM cell 4600 may be used to generally describe the ability of the NNU 121 embodiments described herein to efficiently perform computations associated with LSTM. It is worth noting that these embodiments of the neural network unit 121 , including the embodiment described in FIG. 49 , can efficiently perform calculations associated with other LSTM cells other than the LSTM cells described in FIG. 46 .

Preferably, the neural network unit 121 can be used to perform calculations for a temporal recurrent neural network with LSTM layers connected to other layers. For example, in this LSTM tutorial, the network includes a mean common source layer to receive the output (H) of the LSTM cells of the LSTM layer, and a logistic regression layer to receive the output of the mean common source layer.

FIG. 46 is a block diagram showing an embodiment of an LSTM cell 4600.

As shown in the figure, the LSTM 4600 includes a memory cell input (X), a memory cell output (H), an input gate (I), an output gate (O), a forgetting gate (F), and a memory cell state (C ) and candidate memory cell states (C'). The input gate (I) can gate the signal transmission from the memory cell input (X) to the memory cell state (C), while the output gate (O) can gate the signal transmission from the memory cell state (C) to the memory cell output (H) . The memory cell state (C) is fed back as a candidate memory cell state (C') for a time step. The forgetting gate (F) can gate the candidate memory cell state (C'), and the candidate memory cell state will be fed back and become the memory cell state (C) of the next time step.

The embodiment of Figure 46 uses the following equations to calculate the various aforementioned values:

(1) I=SIGMOID(Wi*X+Ui*H+Bi)

(2) F＝SIGMOID(Wf*X+Uf*H+Bf)

(3) C'＝TANH(Wc*X+Uc*H+Bc)

(4) C=I*C'+F*C

(5) O＝SIGMOID(Wo*X+Uo*H+Bo)

(6)H＝O*TANH(C)

Wi and Ui are weight values associated with the input gate (I), and Bi is an offset value associated with the input gate (I). Wf and Uf are weight values associated with the forget gate (F), and Bf is an offset value associated with the forget gate (F). Wo and Uo are weight values associated with the output gate (O), and Bo is an offset value associated with the output gate (O). As mentioned above, equations (1), (2) and (5) respectively calculate the input gate (I), the forgetting gate (F) and the output gate (O). Equation (3) calculates the candidate memory cell state (C'), and equation (4) calculates the candidate memory cell state (C') with the current memory cell state (C) as input, the current memory cell state (C) is The state of the memory cell at the current time step (C). Equation (6) calculates the memory cell output (H). However, the present invention is not limited thereto. Embodiments of LSTMs that use other methods to calculate input gates, forgetting gates, output gates, candidate memory cell states, memory cell states, and memory cell outputs are also covered by the present invention.

To illustrate the present invention, the LSTM includes memory cell input, memory cell output, memory cell state, candidate memory cell state, input gate, output gate and forgetting gate. For each time step, the input gate, output gate, forgetting gate and candidate memory cell states are functions of memory cell input at the current time step, memory cell output at the previous time step and related weights. The state of the memory cell at this time step is a function of the state of the memory cell at the previous time step, the state of the candidate memory cell, the input gate and the output gate. In this sense, the memory cell state feeds back the memory cell state used to compute the next time step. The output of the memory cell at this time step is a function of the calculated state of the memory cell and the output gate at this time step. A LSTM neural network is a neural network with a layer of LSTM cells.

47 is a block diagram showing the data random access memory 122 and weight random access memory of the neural network unit 121 when the neural network unit 121 performs calculations associated with the LSTM cell 4600 layer of the LSTM neural network of FIG. 46 . An example of the configuration of data within the memory 124 is taken. In the example of FIG. 47, the neural network unit 121 is configured as 512 neural processing units 126 or neurons, such as a wide configuration, but only 128 neural processing units 126 (such as neural processing units 0 to 127) produce values will be used because the LSTM layer in this example has only 128 LSTM cells 4600 .

As shown in the figure, the weight random access memory 124 is loaded with the weight values, offset values and intermediate values of the corresponding NPUs 0 to 127 of the NNU 121 . Rows 0 to 127 of the weight RAM 124 are loaded with weight values, offset values and intermediate values of the corresponding NPUs 0 to 127 of the NNU 121 . Each of columns 0 to 14 is loaded with 128 of the following values corresponding to the aforementioned equations (1) to (6) to provide to the neural processing unit 0 to 127, these values are: Wi, Ui, Bi, Wf, Uf, Bf, Wc, Uc, Bc, C', TANH(C), C, Wo, Uo, Bo. Preferably, the weight values and offset values - Wi, Ui, Bi, Wf, Uf, Bf, Wc, Uc, Bc, Wo, Uo, Bo (located in columns 0 to 8 and columns 12 to 14) - are performed by The architectural program of the processor 100 is written/filled into the weight random access memory 124 through the MTNN instruction 1400 and read/used by the non-architectural program executed in the neural network unit 121 , such as the non-architectural program of FIG. 48 . Preferably, the intermediate values - C', TANH(C), C (in columns 9 to 11) - are written/filled into the weight RAM 124 and read by the non-architectural program executing on the NNU 121 / use, as described later.

As shown in the figure, DRAM 122 is loaded with input (X), output (H), input gate (I), forget gate (F) and output gate (O) values for a series of time steps. Further, the memory is loaded with values of X, H, I, F and O in groups of five for a given time step. Taking a DRAM 122 with 64 columns as an example, as shown in the figure, the DRAM 122 can be loaded with memory cell values for 12 different time steps. In the example of Figure 47, columns 0 to 4 hold the memory cell values for time step 0, columns 5 to 9 hold the memory cell values for time step 1, and so on, columns 55 to 59 hold the memory cell values for time step 11 The memory cell value to use. The first column in this five-column memory holds the X value for this time step. The second column in the five-column memory holds the H value for this time step. The third column in the five-column memory holds the I value for this time step. The fourth column in the five-column memory holds the F value for this time step. The fifth column in the five-column memory holds the O value for this time step. As shown, each row within DRAM 122 is loaded with values for use by a corresponding neuron or NPU 126 . That is, row 0 loads the value associated with LSTM cell 0, and its calculation is performed by NPU0; row 1 loads the value associated with LSTM cell 1, and its calculation is performed by NPU 1; and so on, row 127 loads the value associated with the long-term short-term memory cell 127, and its calculation is performed by the neural processing unit 127, as described in detail in subsequent FIG. 48 .

Preferably, the X values (located in columns 0, 5, 9, and so on to column 55) are written/filled into the DRAM 122 by the architectural program executing on the processor 100 through the MTNN instruction 1400, and by The non-architectural program executed in the neural network unit 121 is read/used, such as the non-architectural program shown in FIG. 48 . Preferably, I values, F values and O values (located in columns 2/3/4, 7/8/9, 12/13/14, and so on to columns 57/58/59) are implemented by the neural processing unit The non-architectural program of 121 is written/filled into the DRAM 122, which will be described in detail later. Preferably, the H values (located in columns 1, 6, 10, and so on to column 56) are written/filled into DRAM 122 and read/used by non-architectural programs executing in NPU 121 , and is read by the architecture program executing on the processor 100 through the MFNN instruction 1500 .

The example of FIG. 47 assumes that the architectural program will perform the following steps: (1) For 12 different time steps, fill the data random access memory 122 with the value of input X (column 0, 5, and so on to column 55) ; (2) start the non-architecture program of Figure 48; (3) detect whether the non-architecture program has been executed; (4) read the value of the output H from the data random access memory 122 (column 1, 6, and so on to column 59); and (5) repeating steps (1) to (4) several times until the task is completed, such as the calculation required to recognize the utterance of the mobile phone user.

In another execution mode, the architectural program will perform the following steps: (1) for a single time step, fill the data random access memory 122 (such as column 0) with the value of input X; (2) start the non-architectural program (the modified version of Fig. 48 non-architecture program does not need to loop, and only accesses a single group of five rows of data random access memory 122); (3) detect whether the non-architecture program has been executed; (4) from the data random access memory Access the memory 122 to read out the value of the output H (eg column 1); and (5) repeat steps (1) to (4) several times until the task is completed. Which of the two methods is superior can be determined according to the sampling method of the input X value of the long-term short-term memory layer. For example, if the task allows the input to be sampled and the computation performed at multiple time steps (e.g. around 12 time steps), the first approach is ideal as it may result in more computational resource efficiency and/or or better performance, but if the task only allows sampling to be performed at a single time step, then the second approach is required.

The third embodiment is similar to the aforementioned second method, but unlike the second method using a single group of five-column data random access memory 122, the non-architectural program of this method uses multiple groups of five-column memory, that is, at each time Steps use different five-column memory, this part is similar to the first way. In this third embodiment, preferably, the architecture program includes a step before step (2). In this step, the architecture program will update the non-architecture program before it starts, for example, the address 0 in the instruction The DRAM 122 ranks are updated to point to the next set of five ranks.

FIG. 48 is a table showing the program stored in the program memory 129 of the neural network unit 121. The program is executed by the neural network unit 121 and uses data and weights according to the configuration of FIG. 47 to achieve calculations associated with the LSTM cell layer. . The example program in FIG. 48 includes 24 non-architectural instructions located at addresses 0 to 23, respectively. The instruction at address 0 (INITIALIZE NPU, CLRACC, LOOPCNT=12, DR IN ROW=-1, DR OUT ROW=2) clears accumulator 202 and initializes loop counter 3804 to a value of 12 to execute 12 loop groups (address 1 to 22 instructions). The initialization command also initializes the column to be read in the DRAM 122 to a value of -1, and after the first execution of the command at address 1, the value is incremented to zero. The initialization command also initializes the column to be written in the data random access memory 122 (such as the register 2606 in FIG. 26 and FIG. 39 ) as column 2 . Preferably, the initialization instruction will cause the NNU 121 to be in a wide configuration, so that the NNU 121 will be configured with 512 NPUs 126 . As described in subsequent chapters, during the execution of the instructions at addresses 0 to 23, 128 of the 512 neural processing units 126 correspond to and operate as 128 long-term short-term memory cells 4600 .

In the first execution of the instructions of addresses 1 to 4, each of the 128 neural processing units 126 (that is, neural processing units 0 to 127) will perform the corresponding long-short-term memory cell 4600 for the first time Step (time step 0) calculates the input gate (I) value and writes the I value to the corresponding text of column 2 of the data random access memory 122; Each neural processing unit 126 in the neural processing unit 126 will calculate the I value for the second time step (time step 1) of the corresponding LSTM cell 4600 and write the I value into the corresponding row 7 of the data random access memory 122. Corresponding text; and so on, in the twelfth execution of the instructions of addresses 1 to 4, each neural processing unit 126 in the 128 neural processing units 126 will correspond to the twelfth time of the corresponding long-short-term memory cell 4600 The step (time step 11) calculates the I value and writes the I value into the corresponding text of column 57 of the DRAM 122, as shown in FIG. 47 .

Further, the multiply-accumulate instruction at address 1 will read the next column behind the current column of the data random access memory 122 (column 0 in the first execution, column 5 in the second execution, and so on, and column 5 in the second execution. Twelve executions are column 55), which contains the memory cell input (X) value associated with the current time step, and this instruction will read column 0 containing the Wi value in the weight random access memory 124, and read the aforementioned The values are multiplied to produce the first multiplication accumulation to accumulator 202 which has just been cleared by the initialize instruction at address 0 or the instruction at address 22. Subsequently, the multiply-accumulate instruction at address 2 will read the next data random access memory 122 column (column 1 in the first execution, column 6 in the second execution, and so on, and column 6 in the twelfth execution. Column 56), this column contains the memory cell output (H) value associated with the current time step, this command will also read the column 1 containing the Ui value in the weight random access memory 124, and multiply the aforementioned values to generate the first The two times are accumulated and added to the accumulator 202 . The H value associated with the current time step is read by the instruction at address 2 (and instructions at addresses 6, 10, and 18) from the data random access memory 122, generated at the previous time step, and written into the data by the output instruction at address 22 RAM 122; however, in the first execution, the instruction at address 2 will write an initial value into column 1 of the DRAM as the H value. Preferably, the architectural program will write the initial H value into column 1 of the data random access memory 122 (for example, using the MTNN instruction 1400) before starting the non-architectural program of FIG. Other embodiments in which the program includes an initialization command to write the initial H value into column 1 of the DRAM 122 also belong to the scope of the present invention. In one embodiment, the initial H value is zero. Next, the add weight text to accumulator instruction (ADD_W_ACC WR ROW 2 ) at address 3 reads row 2 containing the Bi value in the weight RAM 124 and adds it to the accumulator 202 . Finally, the output instruction at address 4 (OUTPUT SIGMOID, DR OUT ROW+0, CLR ACC) will execute an S-type start function for the value of the accumulator 202 and write the execution result into the current output column of the DRAM 122 (in The first execution is column 2, the second execution is column 7, and so on, the twelfth execution is column 57) and accumulator 202 is cleared.

In the first execution of the instructions of addresses 5 to 8, each neural processing unit 126 in the 128 neural processing units 126 will calculate its forgetting for the first time step (time step 0) of the corresponding long-short-term memory cell 4600 gate (F) value and write the F value into the corresponding text of column 3 of the data random access memory 122; The processing unit 126 will calculate its forgetting gate (F) value for the second time step (time step 1) of the corresponding LSTM cell 4600 and write the F value into the corresponding text in column 8 of the data random access memory 122; By analogy, during the twelfth execution of the instructions at addresses 5 to 8, each neural processing unit 126 in the 128 neural processing units 126 will perform the twelfth time step (time Step 11) Calculate the value of the forgetting gate (F) and write the F value into the corresponding text in column 58 of the data random access memory 122, as shown in FIG. 47 . The instructions at addresses 5 to 8 calculate the value of F in a manner similar to the instructions at addresses 1 to 4 described above, however, the instructions at addresses 5 to 7 read Wf from columns 3, 4, and 5 of the weight RAM 124, respectively. , Uf and Bf values to perform multiplication and/or addition operations.

During the twelve executions of the instructions at addresses 9 to 12, each of the 128 NPUs 126 will calculate its candidate memory cell state (C) for the corresponding time step of the corresponding LSTM 4600 ') value and write the C' value into the corresponding text in column 9 of the weight RAM 124 . The instructions at addresses 9 to 12 calculate the value of C' similarly to the instructions at addresses 1 to 4 described above, however, the instructions at addresses 9 to 11 read from columns 6, 7, and 8 of the weight RAM 124, respectively. Wc, Uc and Bc values to perform multiplication and/or addition operations. In addition, the output instruction at address 12 will execute the hyperbolic tangent activation function instead of (as the output instruction at address 4 executes) the sigmoid activation function.

Further, the multiply-accumulate instruction at address 9 will read the current column of the data random access memory 122 (column 0 in the first execution, column 5 in the second execution, and so on, in the tenth The second execution is column 55), this current column contains the memory cell input (X) value associated with the current time step, this instruction will also read the column 6 containing the We value in the weight random access memory 124, and convert the aforementioned The values are multiplied to produce the first multiplication accumulation to accumulator 202 which was just cleared by the instruction at address 8. Next, the multiply-accumulate instruction at address 10 will read the next column of the data random access memory 122 (column 1 in the first execution, column 6 in the second execution, and so on, in the twelfth The first execution is column 56), this column contains the memory cell output (H) value associated with the current time step, this instruction will also read the column 7 containing the value of Uc in the weight random access memory 124, and compare the previous value The multiplication results in a second multiplication accumulation which is added to the accumulator 202 . Next, the add weight literal to accumulator instruction at address 11 reads column 8 containing the value of Bc from weight RAM 124 and adds it to accumulator 202 . Finally, the output instruction at address 12 (OUTPUT TANH, WR OUT ROW 9, CLR ACC) will execute the hyperbolic tangent activation function on the value of the accumulator 202 and write the execution result to column 9 of the weight random access memory 124, and clear the accumulated device 202.

During the twelve executions of the instructions at addresses 13 to 16, each of the 128 NPUs 126 will calculate a new memory cell state (C ) value and write this new C value into the corresponding text of the column 11 of the weight random access memory 122, and each neural processing unit 126 also calculates tanh(C) and writes it into the column of the weight random access memory 124 The corresponding text of 10. Further, the multiply-accumulate instruction at address 13 will read the next column behind the current column of the data random access memory 122 (column 2 in the first execution, column 7 in the second execution, and so on, In the twelfth execution is column 57), this column contains the value of the input gate (I) associated with the current time step, and this instruction reads the value of the candidate memory cell state (C') contained in the weight random access memory 124 column 9 (just written to by the instruction at address 12), and the preceding values are multiplied to produce the first multiplied accumulation to accumulator 202 which has just been cleared by the instruction at address 12. Next, the multiply-accumulate instruction at address 14 will read the next column of the data random access memory 122 (column 3 in the first execution, column 8 in the second execution, and so on, in the twelfth The second execution is column 58), which contains the forgetting gate (F) value associated with the current time step, and reads the current memory cell state (C ) value (written by the most recent execution of the instruction at address 15) to column 11, and the preceding value is multiplied to produce a second product that is added to accumulator 202. Next, the output instruction at address 15 (OUTPUT PASSTHRU, WR OUT ROW 11 ) passes the accumulator 202 value and writes it to column 11 of the weight RAM 124 . It should be understood that the C value read by the instruction at address 14 from column 11 of the DRAM 122 is the C value generated and written in the latest execution of the instructions at addresses 13 to 15 . The output instruction at address 15 does not clear accumulator 202 so that its value can be used by the instruction at address 16. Finally, the output instruction at address 16 (OUTPUT TANH, WR OUT ROW 10, CLR ACC) will execute the hyperbolic tangent activation function on the value of the accumulator 202 and write its execution result to column 10 of the weight random access memory 124 for address 21 The instruction of is used to calculate the memory cell output (H) value. The instruction at address 16 clears accumulator 202 .

In the first execution of the instructions at addresses 17 to 20, each of the 128 NPUs 126 will calculate its output for the first time step (time step 0) of the corresponding LSTM 4600 gate (O) value and write the O value into the corresponding text of column 4 of the data random access memory 122; The processing unit 126 calculates the value of the output gate (O) for the second time step (time step 1) of the corresponding LSTM cell 4600 and writes the value of O into the corresponding text in column 9 of the DRAM 122; By analogy, during the twelfth execution of the instructions at addresses 17 to 20, each neural processing unit 126 in the 128 neural processing units 126 will perform the twelfth time step (time Step 11) Calculate the value of the output gate (O) and write the value of O into the corresponding text of column 58 of the data random access memory 122, as shown in FIG. 47 . The instructions at addresses 17 through 20 compute the value of O in a manner similar to the instructions at addresses 1 through 4 described above, however, the instructions at addresses 17 through 19 read Wo from column 12, column 13, and column 14 of weight RAM 124, respectively. , Uo and Bo values to perform multiplication and/or addition operations.

In the first execution of the instructions at addresses 21 to 22, each of the 128 neural processing units 126 will calculate its memory for the first time step (time step 0) of the corresponding long-short-term memory cell 4600 cell output (H) value and write the H value into the corresponding text of column 6 of the data random access memory 122; The neural processing unit 126 will calculate the memory cell output (H) value for the second time step (time step 1) corresponding to the long-term short-term memory cell 4600 and write the H value into the corresponding column 11 of the data random access memory 122 text; and so on, in the twelfth execution of the instructions of addresses 21 to 22, each neural processing unit 126 in the 128 neural processing units 126 will address the twelfth time step of the corresponding long-short-term memory cell 4600 (Time step 11) Calculate the memory cell output (H) value and write the H value into the corresponding text in column 60 of the data random access memory 122, as shown in FIG. 47 .

Further, the multiply-accumulate instruction at address 21 will read the third column behind the current column of the data random access memory 122 (column 4 in the first execution, column 9 in the second execution, and so on. , in the twelfth execution is column 59), this column contains the output gate (O) value associated with the current time step, and this instruction reads the column 10 containing the tanh (C) value in the weight random access memory 124 (written by the instruction at address 16), and the aforementioned values are multiplied to produce a multiplied accumulation to accumulator 202 which was just cleared by the instruction at address 20. Subsequently, the output instruction at address 22 transfers the accumulator 202 value and writes it to the next second output column 11 of the DRAM 122 (column 6 in the first execution and column 6 in the second execution). is column 11, and so on, the twelfth execution is column 61), and the accumulator 202 is cleared. It should be understood that the H value written into the data random access memory 122 column by the instruction of address 22 (in the first execution is column 6, in the second execution it is column 11, and so on, in the tenth The second execution (column 61) is the H value consumed/read in the subsequent execution of the instructions at addresses 2, 6, 10 and 18. However, the H value written to column 61 in the twelfth execution will not be consumed/read by the execution of instructions at addresses 2, 6, 10 and 18; for a preferred embodiment, this value will be Consumed/read by infrastructure programs.

The instruction at address 23 (LOOP 1) will decrement the loop counter 3804 and fall back to the instruction at address 1 if the new loop counter 3804 value is greater than zero.

FIG. 49 is a block diagram showing an embodiment of a neural network unit 121 with output buffer masking and feedback capabilities within a group of neural processing units. FIG. 49 shows a single NPU group 4901 consisting of four NPUs 126 . Although FIG. 49 only shows a single neural processing unit group 4901, it should be understood that each neural processing unit 126 in the neural network unit 121 will be included in one neural processing unit group 4901. Therefore, there will be a total of N/ J NPU groups 4901, where N is the number of NPUs 126 (e.g., 512 for a wide configuration, 1024 for a narrow configuration) and J is the number of neurons within a single group 4901 The number of processing units 126 (for example, four for the embodiment of FIG. 49 ). In FIG. 49 , the four NPUs 126 in the NPU group 4901 are referred to as NPU0, NPU1, NPU2 and NPU3.

Each neural processing unit in the embodiment of FIG. 49 is similar to the aforementioned neural processing unit 126 in FIG. 7 , and components with the same number in the figure are also similar. However, the multitasking register 208 is adapted to include four additional inputs 4905, the multitasking register 705 is adapted to include four additional inputs 4907, and the select input 213 is adapted to select from the original inputs 211 and 207 plus the additional A selection from input 4905 is provided to output 209 , and selection input 713 is adjusted to select from original inputs 711 and 206 and additional input 4907 is provided to output 203 .

As shown in the figure, the column buffer 1104 in FIG. 11 is the output buffer 1104 in FIG. 49 . Further, the words 0, 1, 2 and 3 of the output buffer 1104 shown in the figure receive the corresponding outputs of the four AFUs 212 associated with the NPUs 0, 1, 2 and 3. The output buffer 1104 of this part contains N words corresponding to the NPU group 4901, and these words are called an output buffer word group. In the embodiment of FIG. 49, N is four. The four words from output buffer 1104 are fed back to multiplex registers 208 and 705 and received as four additional inputs 4905 by multiplex register 208 and as four additional inputs 4907 by multiplex register 705 . The feedback action of the output buffer literal group to its corresponding NPU group 4901 enables the arithmetic instructions of the non-architectural program to be read from the literal of the output buffer 1104 associated with the NPU group 4901 (i.e., the output buffer literal group Group) to select one or two characters as its input. For an example, please refer to the non-architectural program in FIG. That is, the output buffer 1104 literal specified in the non-architectural instruction will confirm the value generated by the select input 213/713. This capability effectively enables output buffer 1104 to act as a sort of scratch pad memory, enabling non-architectural programs to reduce the number of writes to and subsequent reads from data RAM 122 and/or weight RAM 124 , such as values that are intermediately generated and used during reduction. Preferably, the output buffer 1104, or column buffer 1104, includes a one-dimensional buffer array for storing 1024 narrow words or 512 wide words. Preferably, reading from the output buffer 1104 can be performed within a single clock cycle, and writing to the output buffer 1104 can also be performed within a single clock cycle. Unlike the data RAM 122 and the weight RAM 124, which can be accessed by architectural programs and non-architectural programs, the output buffer 1104 cannot be accessed by architectural programs, but can only be accessed by non-architectural programs.

Output buffer 1104 will be adjusted to receive mask input 4903 . Preferably, mask input 4903 includes four bits corresponding to four words of output buffer 1104 associated with four NPUs 126 of NPU group 4901 . Preferably, if the mask input 4903 bit corresponding to the text of the output buffer 1104 is true, the text of the output buffer 1104 will maintain its current value; otherwise, the text of the output buffer 1104 will be activated by the function The output of unit 212 is updated. That is, if the mask input 4903 bit corresponding to the text of the output buffer 1104 is false, the output of the enable function unit 212 will be written into the text of the output buffer 1104 . In this way, the output instruction of the non-architecture program can selectively write the output of the activation function unit 212 into some words of the output buffer 1104 and keep the current values of other words of the output buffer 1104 unchanged. For an example, please refer to Subsequent instructions of the non-architecture program in FIG. 51 are instructions at addresses 6, 10, 13 and 14 in the figure. That is to say, the text specified in the output buffer 1104 in the non-architecture program depends on the value generated from the mask input 4903 .

To simplify the description, the input 1811 of the multitasking register 208/705 (as shown in FIG. 18 , FIG. 19 and FIG. 23 ) is not shown in FIG. 49 . However, embodiments that support both dynamically configurable NPU 126 and output buffer 1104 feedback/masking are also within the scope of the present invention. Preferably, in these embodiments, the output buffer text group can be correspondingly and dynamically configured.

It should be understood that although the number of neural processing units 126 in the neural processing unit group 4901 in this embodiment is four, the present invention is not limited thereto, and the number of neural processing units 126 in the group is larger or smaller Examples all belong to the category of the present invention. In addition, as far as an embodiment with a shared activation function unit 1112 is concerned, as shown in FIG. Quantities have synergistic effects. The masking and feedback capabilities of the output buffers 1104 in the NPU group are particularly helpful for improving the computational efficiency associated with the LSTM cells 4600, as described in detail in FIG. 50 and FIG. 51 .

FIG. 50 is a block diagram showing that when the neural network unit 121 performs calculations associated with a layer of 128 long-short-term memory cells 4600 in FIG. 46, the data random access memory 122 of the neural network unit 121 of FIG. 49, An example of data configuration in the weight random access memory 124 and the output buffer 1104 . In the example of FIG. 50 , the neural network unit 121 is configured as 512 neural processing units 126 or neurons, for example, in a wide configuration. Like the example in FIG. 47 and FIG. 48 , there are only 128 LSTM cells 4600 in the LSTM layer in the example in FIG. 50 and FIG. 51 . However, in the example of FIG. 50 , the values generated by all 512 NPUs 126 (eg, NPUs 0 to 127 ) are used. When executing the non-architectural program in FIG. 51 , each neural processing unit group 4901 collectively operates as a long-term short-term memory cell 4600 .

As shown, DRAM 122 is loaded with cell input (X) and output (H) values for a series of time steps. Further, for a given time step, there will be a pair of two-column memories loaded with X value and H value respectively. Taking a DRAM 122 with 64 columns as an example, as shown in the figure, the memory cell values loaded in the DRAM 122 can be used in 31 different time steps. In the example of Figure 50, columns 2 and 3 hold values for time step 0, columns 4 and 5 for time step 1, and so on, columns 62 and 63 for time step 30. The first column of the pair of two-column memories holds the X value for the time step, and the second column holds the H value for the time step. As shown in the figure, each group of four rows in the DRAM 122 corresponds to the memory of the NPU group 4901 loaded with values for its corresponding LSTM 4600 . That is to say, rows 0 to 3 are loaded with values associated with LSTM cell 0, whose calculations are performed by NPUs 0-3, that is, NPU group 0; rows 4 to 7 are loaded with values associated with LSTM The value of cell 1, whose calculation is performed by NPU 4-7, that is, NPU group 1; and so on, rows 508 to 511 are loaded with the value associated with long short-term memory cell 127, whose calculation is performed by NPU The processing units 508-511 execute, that is, the neural processing unit group 127 executes, as shown in the subsequent FIG. 51 in detail. As shown in the figure, column 1 is not used, and column 0 is loaded with the initial memory cell output (H) value. For a preferred embodiment, the zero value can be filled by the framework program, but the present invention is not limited to Therefore, using non-architectural program instructions to fill the initial memory cell output (H) value of column 0 also falls within the scope of the present invention.

Preferably, the X value (located in columns 2, 4, 6 and so on to column 62) is written/filled into the DRAM 122 by the architectural program executing on the processor 100 through the MTNN instruction 1400, and is executed by It is read/used by the non-architectural program of the neural network unit 121, such as the non-architectural program shown in FIG. 50 . Preferably, the H values (located in columns 3, 5, 7 and so on to column 63) are written/filled into the DRAM 122 and read/used by the non-architectural program executed in the NNU 121, Details are described later. Preferably, the H value is read by the architecture program executed on the processor 100 through the MFNN instruction 1500 . It should be noted that the non-architectural program of FIG. 51 assumes that each set of four lines of memory corresponding to the NPU group 4901 (such as lines 0-3, lines 4-7, lines 5-8, and so on to lines 508- 511), the four X values in a given column are filled with the same value (eg, by a framework program). Similarly, the non-architectural program of FIG. 51 calculates and writes the same value for the four H values for a given column in each set of four rows of memory corresponding to NPU group 4901 .

As shown in the figure, the WRAM 124 is loaded with the weights, offsets and cell state (C) values required by the NPU of the NNU 121 . In each set of four rows of memory corresponding to NPU group 121 (e.g. rows 0-3, rows 4-7, rows 5-8 and so on to rows 508-511): (1) row number divided by 4 The row whose remainder is equal to 3 will load the values of Wc, Uc, Bc, and C in its columns 0, 1, 2, and 6 respectively; (2) the row number divided by 4 and the remainder equal to 2 will be loaded in its column 3, 4, and 5 respectively load the values of Wo, Uo, and Bo; (3) the row number whose remainder equals 1 after dividing the row number by 4 will load the values of Wf, Uf, and Bf in columns 3, 4, and 5 respectively; and (4) For the row whose remainder is equal to 0 when the row number is divided by 4, the values of Wi, Ui and Bi will be loaded in columns 3, 4 and 5 respectively. Preferably, these weights and offsets—Wi, Ui, Bi, Wf, Uf, Bf, Wc, Uc, Bc, Wo, Uo, Bo (in columns 0 to 5)—are implemented by the architecture of processor 100 The program is written/filled into the weight RAM 124 through the MTNN instruction 1400, and read/used by the non-architectural program executed in the neural network unit 121, such as the non-architectural program in FIG. 51 . Preferably, the intermediate C value is written/filled into the weight random access memory 124 and read/used by the non-architectural program executed in the neural network unit 121 , as described in detail later.

The example of FIG. 50 assumes that the architecture program will perform the following steps: (1) for 31 different time steps, fill the value of the input X into the data random access memory 122 (column 2, 4, and so on to column 62); (2) start the non-architecture program in Figure 51; (3) detect whether the non-architecture program has been executed; (4) read the value of the output H from the data random access memory 122 (column 3, 5, and so on to the column 63); and (5) repeating steps (1) to (4) several times until the task is completed, such as the calculation required for recognizing the speech of the mobile phone user.

In another execution mode, the architectural program will perform the following steps: (1) for a single time step, fill the data random access memory 122 (such as column 2) with the value of input X; (2) start the non-architectural program (the modified version of the non-architecture program in Figure 51 does not need to loop, and only accesses the single pair of two columns of the data random access memory 122); (3) detect whether the non-architecture program has been executed; (4) from the data random access memory Access memory 122 to read out the value of output H (eg column 3); and (5) repeat steps (1) to (4) several times until the task is completed. Which of the two methods is superior can be determined according to the sampling method of the input X value of the long-term short-term memory layer. For example, if the task allows the input to be sampled and the computation to be sampled at multiple time steps (e.g. around 31 time steps), the first approach is ideal as it may result in more computational resource efficiencies and/or or better performance, but if the task only allows sampling to be performed at a single time step, then the second approach is required.

The third embodiment is similar to the aforementioned second method, but unlike the second method using a single pair of two-column data random access memory 122, the non-architectural program of this method uses multiple pairs of memory columns, that is, at each time step This part is similar to the first way, using a different pair of memory columns. Preferably, the architecture program of this third embodiment includes a step before step (2). In this step, the architecture program will update the non-architecture program before it starts, for example, the data in the instruction of address 1 The RAM 122 ranks are updated to point to the next pair of two-rank memories.

As shown in the figure, for the neural processing units 0 to 511 of the neural network unit 121, after the execution of instructions at different addresses in the non-architectural program of FIG. (C'), the input gate (I), the forget gate (F), the output gate (O), the intermediate value of the memory cell state (C) and tanh (C), in each output buffer text group (such as output buffer The device 1104 corresponds to the group of four characters of the neural processing unit group 4901, such as characters 0-3, 4-7, 5-8 and so on to 508-511), and the remainder of dividing the character number by 4 is 3 The text is represented as OUTBUF[3], the text whose number is divided by 4 and the remainder is 2 is represented as OUTBUF[2], the text whose number is divided by 4 and the remainder is 1 is represented as OUTBUF[1], and the text number divided by 4 A literal with a remainder of 0 is represented as OUTBUF[0].

As shown in the figure, after the instruction at address 2 in the non-architectural program of FIG. The initial memory cell output (H) value of 4600. After the instruction at address 6 is executed, for each neural processing unit group 4901, the OUTBUF[3] text of the output buffer 1104 will be written into the candidate memory cell state (C') value of the corresponding long-term short-term memory cell 4600, The other three words of the output buffer 1104 will maintain their previous values. After the instruction at address 10 is executed, for each neural processing unit group 4901, the OUTBUF[0] text of the output buffer 1104 will be written into the input gate (I) value of the corresponding long-term short-term memory cell 4600, OUTBUF[1 ] text will be written into the forgetting gate (F) value corresponding to the long-term short-term memory cell 4600, and the OUTBUF[2] text will be written into the output gate (O) value corresponding to the long-term short-term memory cell 4600, while the OUTBUF[3] text will be is to maintain its previous value. After the instruction at address 13 is executed, for each neural processing unit group 4901, the OUTBUF[3] text of the output buffer 1104 will be written into the new memory cell state (C) value of the corresponding long-term short-term memory cell 4600 ( For the output buffer 1104, the value of C containing the slot (slot) 3 is written into column 6 of the weight random access memory 124, as described in detail in subsequent FIG. 51 ), while the other three words of the output buffer 1104 are is to maintain its previous value. After the instruction at address 14 is executed, for each neural processing unit group 4901, the OUTBUF[3] text of the output buffer 1104 will be written into the tanh(C) value of the corresponding long-term and short-term memory cell 4600, and the output buffer The other three characters of 1104 maintain their previous values. After the instruction at address 16 is executed, for each NPU group 4901 , all four words of the output buffer 1104 will be written into the new memory cell output (H) value of the corresponding LSTM 4600 . The aforementioned execution flow of addresses 6 to 16 (that is, excluding the execution of address 2 because address 2 is not part of the program loop) will be repeated thirty times, returning to the program loop of address 3 as address 17.

FIG. 51 is a table showing the program stored in the program memory 129 of the neural network unit 121. This program is executed by the neural network unit 121 of FIG. 49 and uses data and weights according to the configuration of FIG. layer calculations. The example program in FIG. 51 contains 18 non-architectural instructions located at addresses 0 to 17, respectively. The instruction at address 0 is an initialization instruction to clear the accumulator 202 and initialize the loop counter 3804 to a value of 31 to execute the loop group (instructions at addresses 1 to 17) 31 times. This initialization command will also initialize the column to be written in the data random access memory 122 (such as the register 2606 in FIG. 26/FIG. 39) to a value of 1, and after the first execution of the instruction at address 16, this value will be increased to 3. Preferably, the initialization instruction will cause the NNU 121 to be in a wide configuration, so that the NNU 121 will be configured with 512 NPUs 126 . As described in subsequent chapters, during the execution of the instructions at addresses 0 to 17, the 128 NPU groups 4901 formed by the 512 NPUs 126 operate as 128 corresponding LSTMs 4600 .

The instructions at addresses 1 and 2 do not belong to the loop group of the program and will only be executed once. These instructions generate an initial cell output (H) value (eg, 0) and write it to all words in the output buffer 1104 . The instruction at address 1 will read the initial H value from column 0 of DRAM 122 and place it in accumulator 202 which is cleared by the instruction at address 0. The instruction at address 2 (OUTPUT PASSTHRU, NOP, CLR ACC) will pass the accumulator 202 value to the output buffer 1104, as shown in FIG. 50 . The "NOP" mark in the output command of address 2 (and other output commands in Figure 51) indicates that the output value will only be written into the output buffer 1104, and will not be written into the memory, that is, it will not be written into the data random access memory 122 or weight random access memory 124 . The instruction at address 2 will also clear the accumulator 202.

The instructions at addresses 3 to 17 are located in the loop group, and the number of executions thereof is the value of the loop count (eg, 31).

Each execution of the instructions at address 3 to 6 will calculate the value of tanh(C’) at the current time step and write it into the text OUTBUF[3], which will be used by the instruction at address 11. More precisely, the multiply-accumulate instruction at address 3 will read the memory cell input associated with this time step from the current read column of the data random access memory 122 (such as columns 2, 4, 6 and so on to column 62) (X) value, reads the Wc value from column 0 of the weight RAM 124, and multiplies the aforementioned values to generate a product that is added to the accumulator 202 that is cleared by the instruction at address 2.

The multiply-accumulate instruction of address 4 (MULT-ACCUM OUTBUF[0], WR ROW 1) will read the H value (that is, all four neural processing units 126 of the neural processing unit group 4901) from the text OUTBUF[0], from the weight Column 1 of the random access memory 124 reads the value of Uc and multiplies the aforementioned value to generate a second product which is added to the accumulator 202 .

The add weight text to accumulator instruction (ADD_W_ACC WR ROW 2 ) at address 5 reads the Bc value from row 2 of the weight RAM 124 and adds it to the accumulator 202 .

The output instruction at address 6 (OUTPUT TANH, NOP, MASK[0:2], CLR ACC) will execute the hyperbolic tangent activation function on the value of the accumulator 202, and only write the execution result into the text OUTBUF[3] (that is, Only the NPU 126 of the NPU group 4901 whose number is 3 after dividing by 4 will write this result), and the accumulator 202 will be cleared. That is to say, the output instruction at address 6 will mask the text OUTBUF[0], OUTBUF[1] and OUTBUF[2] (as indicated by the instruction term MASK[0:2]) and maintain its current value, as shown in Figure 50 . Furthermore, the output instruction at address 6 is not written to memory (as denoted by the instruction term NOP).

Each execution of the instructions of addresses 7 to 10 will calculate the input gate (I) value, forget gate (F) value and output gate (O) value of the current time step and write them into the text OUTBUF[0], OUTBUF[ 1], and OUTBUF[2], these values will be used by the instructions at addresses 11, 12 and 15. More precisely, the multiply-accumulate instruction at address 7 will read the memory cell input associated with this time step from the current read column of the data random access memory 122 (such as columns 2, 4, 6 and so on to column 62) (X) values, read the Wi, Wf and Wo values from column 3 of the weight RAM 124, and multiply the aforementioned values to generate a product which is added to the accumulator 202 which is cleared by the instruction at address 6. More precisely, in the neural processing unit group 4901, the neural processing unit 126 whose number is 0 when dividing by 4 will calculate the product of X and Wi, and the neural processing unit 126 whose number is 1 when dividing by 4 will calculate X and The product of Wf, and the neural processing unit 126 whose number has a remainder of 2 when divided by 4 calculates the product of X and Wo.

The multiply-accumulate instruction at address 8 reads the H value (that is, all four NPUs 126 of the NPU group 4901 ) from the text OUTBUF[0], and reads Ui, Uf from column 4 of the weight random access memory 124 and the value of Uo, and multiply the aforementioned value to generate a second product which is added to the accumulator 202 . More precisely, in the neural processing unit group 4901, the neural processing unit 126 whose number is 0 when dividing by 4 will calculate the product of H and Ui, and the neural processing unit 126 whose number is 1 when dividing by 4 will calculate H and The product of Uf, and the NPU 126 whose number has a remainder of 2 when divided by 4 calculates the product of H and Uo.

The add weight text to accumulator instruction (ADD_W_ACC WR ROW 2 ) at address 9 reads the Bi, Bf and Bo values from row 5 of the weight RAM 124 and adds them to the accumulator 202 . More precisely, in the neural processing unit group 4901, the neural processing units 126 whose numbers divide by 4 with a remainder of 0 will perform the addition calculation of Bi values, and the neural processing units 126 whose numbers divide by 4 with a remainder of 1 will perform Bf The addition calculation of the value, and the neural processing unit 126 whose number is divided by 4 with a remainder of 2 will perform the addition calculation of the Bo value.

The output command at address 10 (OUTPUT SIGMOID, NOP, MASK[3], CLR ACC) will execute the S-type startup function for the value of the accumulator 202 and write the calculated I, F and O values into the text OUTBUF[0] respectively, OUTBUF[1] and OUTBUF[2], this instruction will also clear the accumulator 202 without writing to the memory. That is to say, the output instruction at address 10 will mask the word OUTBUF[3] (as represented by the instruction term MASK[3]) and maintain the current value of the word (that is, C'), as shown in FIG. 50 .

Each execution of the instructions of addresses 11 to 13 will calculate the new memory cell state (C) value generated by the current time step and write it into column 6 of the weight random access memory 124 for use in the next time step (that is, for the next time step The instruction at address 12 is used when the next cycle is executed), more precisely, the value written in column 6 corresponds to the text whose label divides by 4 and the remainder is 3 in the four lines of text of the neural processing unit group 4901 . In addition, each execution of the instruction at address 14 will write the value of tanh (C) into OUTBUF[3] for the instruction at address 15.

More precisely, the multiply-accumulate instruction at address 11 (MULT-ACCUM OUTBUF[0], OUTBUF[3]) reads the input gate (I) value from the literal OUTBUF[0] and the candidate value from the literal OUTBUF[3] The value of the cell state (C') is stored and multiplied to generate a first product which is added to the accumulator 202 which is cleared by the instruction at address 10. More precisely, each neural processing unit 126 in the four neural processing units 126 of the neural processing unit group 4901 calculates the first product of the I value and the C' value.

The multiply-accumulate instruction at address 12 (MULT-ACCUM OUTBUF[1], WR ROW 6) will instruct the NPU 126 to read the forget gate (F) value from the text OUTBUF[1], from column 6 of the WRAM 124 The corresponding word is read and multiplied to produce a second product which is added to the first product generated in accumulator 202 by the instruction at address 11 . More precisely, for the NPUs 126 in the NPU group 4901 whose label divides by 4 with a remainder of 3, the text read from column 6 is the current memory cell state (C) value calculated at the previous time step , the sum of the first product and the second product is the new memory cell state (C). However, for the other three NPUs 126 of the NPU group 4901, the text read from column 6 is a don't care value, because the accumulated value generated by these values will not be used and will not be used. That is, it will not be put into the output buffer 1104 by the instructions of address 13 and 14, but will be cleared by the instruction of address 14. That is to say, only the new memory cell state (C) value generated by the neural processing unit 126 of the neural processing unit group 4901 whose label divides by 4 with a remainder of 3 will be used, that is, used by the instructions of addresses 13 and 14 . For the second through thirty-first executions of the instruction at address 12, the C value read from column 6 of weight RAM 124 is the value written by the instruction at address 13 in the previous execution of the loop group. However, for the first execution of the instruction at address 12, the C value in column 6 is the initial value written by the architected program before starting the non-architected program of FIG. 51 or by an adjusted version of the non-architected program .

The output instruction at address 13 (OUTPUT PASSTHRU, WR ROW 6, MASK[0:2]) will only pass the accumulator 202 value, which is the calculated C value, to the text OUTBUF[3] (that is, only the NPU The neural processing unit 126 in the group 4901 whose label divides by 4 with a remainder of 3 will write its calculated C value into the output buffer 1104), and column 6 of the weight random access memory 124 will use the updated output buffer Writer 1104, as shown in Figure 50. That is to say, the output command at address 13 will mask the characters OUTBUF[0], OUTBUF[1] and OUTBUF[2] and maintain their current values (ie, I, F and O values). As mentioned above, only the C values in the four-line text corresponding to the neural processing unit group 4901 in column 6 will be used, that is, the C value in the text with a remainder of 3 when dividing by 4, that is, used by the instruction at address 12; therefore, the non-architectural program The values in row 0-2, row 4-6, and so on to row 508-510 in column 6 of the weight RAM 124 are ignored, as shown in FIG. 50 (ie, I, F and O values).

The output instruction at address 14 (OUTPUT TANH, NOP, MASK[0:2], CLR ACC) will execute the hyperbolic tangent activation function on the value of the accumulator 202, and write the calculated tanh (C) value into the text OUTBUF[3 ], which clears the accumulator 202 without writing to memory. The output command at address 14, like the output command at address 13, will mask the characters OUTBUF[0], OUTBUF[1] and OUTBUF[2] and maintain their original values, as shown in FIG. 50 .

Each execution of the instructions at addresses 15 to 16 will calculate the memory cell output (H) value generated at the current time step and write it into the second column behind the current output column of the data random access memory 122, and its value will be determined by the architecture The program is read and used for the next time step (ie used by the instructions at addresses 3 and 7 in the next loop execution). More precisely, the multiply-accumulate instruction at address 15 reads the output gate (O) value from literal OUTBUF[2], reads the tanh(C) value from literal OUTBUF[3], and multiplies them to produce a product Add accumulator 202 cleared by the instruction at address 14. More precisely, each of the four NPUs 126 in the NPU group 4901 calculates the product of the value O and tanh(C).

The output instruction at address 16 passes the accumulator 202 value and writes the calculated H value to column 3 in the first execution, writes the calculated H value to column 5 in the second execution, and so on in The calculated H value is written into column 63 in the thirty-first execution, as shown in FIG. In addition, as shown in FIG. 50 , these calculated H values will be put into the output buffer 1104 for the subsequent use of the instructions at addresses 4 and 8 . The output instruction at address 16 also clears accumulator 202. In one embodiment, the LSTM cell 4600 is designed so that the output command at address 16 (and/or the output command at address 22 in FIG. 48 ) has an activation function, such as a sigmoid or hyperbolic tangent function, rather than a transfer-accumulate device 202 value.

The loop instruction at address 17 will decrement the loop counter 3804 and fall back to the instruction at address 3 if the new loop counter 3804 value is greater than zero.

It can be found that, because of the feedback and shielding capabilities of the output buffer 1104 in the embodiment of the neural network unit 121 of FIG. 49 , the number of instructions in the loop group of the non-architectural program of FIG. 51 is approximately 34% reduction. In addition, because of the feedback and masking capabilities of the output buffer 1104 in the embodiment of the neural network unit 121 in FIG. three times. The aforementioned improvements are helpful for some architectural program applications using the neural network unit 121 to perform LSTM cell layer calculations, especially for applications where the number of LSTM cells 4600 in the LSTM cell layer is less than or equal to 128.

The embodiments of FIGS. 47-51 assume that the weights and offsets remain unchanged at each time step. However, the present invention is not limited thereto, and other embodiments in which weights and offsets change with time steps also fall within the scope of the present invention, wherein the weight random access memory 124 is not filled into a single group as shown in FIGS. 47 to 50 Instead, fill in different sets of weights and offset values at each time step, and the address of the weight random access memory 124 of the non-architectural program in FIGS. 48 to 51 will be adjusted accordingly.

Basically, in the aforementioned embodiments of FIG. 47 to FIG. 51, weights, offsets and intermediate values (such as C, C' values) are stored in the weight random access memory 124, and input and output values (such as X, H values ) is stored in the DRAM 122 . This feature facilitates embodiments where the data RAM 122 is dual-ported and the weight RAM 124 is single-ported because there is more traffic to the data RAM 122 from non-architectural programs and architectural programs . However, because the weight random access memory 124 is relatively large, in another embodiment of the present invention, the memory for storing non-architecture and architecture program write values is exchanged (that is, the data random access memory 122 and the weight random access memory 122 are interchanged). access memory 124). That is, W, U, B, C', tanh(C) and C values are stored in DRAM 122 and X, H, I, F and O values are stored in WRAM 124 ( Adjusted embodiment of FIG. 47); and W, U, B, and C values are stored in DRAM 122 while X and H values are stored in Weight RAM 124 (adjusted embodiment of FIG. 50 ). Because the weight RAM 124 is larger, these embodiments can process more time steps in a batch. For applications utilizing the architecture of the neural network unit 121 to perform computations, this feature is beneficial for certain applications that can benefit from more time steps and may be a memory of a single-port design (such as weight random access memory 124) Provide sufficient bandwidth.

FIG. 52 is a block diagram showing an embodiment of the neural network unit 121 . In this embodiment, the group of neural processing units has output buffer masking and feedback capabilities, and the activation function unit 1112 is shared. The neural network unit 121 of FIG. 52 is similar to the neural network unit 121 of FIG. 47, and the components with the same reference numbers in the figure are also similar. However, the four enable function units 212 of FIG. 49 are replaced in this embodiment by a single shared enable function unit 1112, which receives four outputs 217 from the four accumulators 202 and generates Four outputs to text OUTBUF[0], OUTBUF[1], OUTBUF[2] and OUTBUF[3]. The operation of the neural network unit 212 in FIG. 52 is similar to the embodiment described above in FIGS. 49 to 51 , and the operation mode of the shared activation function unit 1112 is similar to the embodiment described in FIGS. 11 to 13 above.

Figure 53 is a block diagram showing that the data random access memory 122 of the neural network unit 121 of Figure 49, weight Another embodiment of the data configuration in the random access memory 124 and the output buffer 1104 . The example of FIG. 53 is similar to the example of FIG. 50 . However, in Figure 53, Wi, Wf and Wo values are in column 0 (instead of column 3 as in Figure 50); Ui, Uf and Uo are in column 1 (rather than in column 4 as in Figure 50); Bi, Bf and Bo values are in column 2 (instead of column 5 as in Figure 50); C values are in column 3 (instead of column 6 as in Figure 50). In addition, the content of the output buffer 1104 of FIG. 53 is similar to that of FIG. 50, but because of the difference between the non-architectural program of FIG. 54 and FIG. The instruction at address 7 appears in the output buffer 1104 after execution (rather than the instruction at address 10 as shown in Figure 50); the contents of the fourth column (ie I, F, O and C values) appear after the instruction at address 10 is executed In the output buffer 1104 (instead of the address 13 instruction as shown in Figure 50); the contents of the fifth column (ie I, F, O and tanh (C) values) appear in the output buffer after the execution of the instruction at address 11 1104 (rather than the instruction of address 14 as shown in Figure 50); and the content of the sixth column (i.e. the H value) appears in the output buffer 1104 after the execution of the instruction of address 13 (rather than the instruction of address 16 as shown in Figure 50 ), as described later.

FIG. 54 is a table showing the program stored in the program memory 129 of the neural network unit 121. This program is executed by the neural network unit 121 of FIG. 49 and uses data and weights according to the configuration of FIG. layer calculations. The example program of FIG. 54 is similar to the program of FIG. 51 . More precisely, in Figure 54 and Figure 51, the instructions of addresses 0 to 5 are the same; the instructions of addresses 7 and 8 in Figure 54 are identical to the instructions of addresses 10 and 11 in Figure 51; and the instructions of addresses 10 to 14 in Figure 54 The commands are the same as those of addresses 13 to 17 in Fig.51.

However, the instruction at address 6 in FIG. 54 does not clear the accumulator 202 (in contrast, the instruction at address 6 in FIG. 51 clears the accumulator 202). In addition, the instructions at addresses 7 to 9 in FIG. 51 do not appear in the non-architectural program of FIG. 54 . Finally, regarding the instruction at address 9 in FIG. 54 and the instruction at address 12 in FIG. 51, except that the instruction at address 9 in FIG. Except column 6 of the read weight random access memory, other parts are the same.

Because of the difference between the non-architectural program of FIG. 54 and the non-architectural program of FIG. 51 , the configuration of FIG. 53 will reduce the number of columns of the weight RAM 124 by three, and the number of instructions in the program loop will also be reduced by three. The loop group size in the non-architected program of FIG. 54 is substantially only half the size of the loop group in the non-architected program of FIG. 48 , and roughly 80% of the size of the loop group in the non-architected program of FIG. 51 .

FIG. 55 is a block diagram showing portions of the neural processing unit 126 according to another embodiment of the present invention. More precisely, for a single neural processing unit 126 of the plurality of neural processing units 126 of FIG. Inputs 206, 711 and 4907 are associated therewith. In addition to the input of FIG. 49 , the multitasking register 208 and the multitasking register 705 of the NPU 126 each receive an index_within_group input 5599 . The number within group input 5599 indicates the number of a particular NPU 126 within its NPU group 4901 . Thus, for example, taking an embodiment in which each NPU group 4901 has four NPUs 126, within each NPU group 4901, one of the NPUs 126 numbers the inputs within its group. 5599 receives the value zero, one of the neural processing units 126 receives the value one in its intragroup number input 5599, one of the neural processing units 126 receives the value two in its intragroup number input 5599, and one of the neural processing units 126 receives the value three in its intragroup number input 5599. In other words, the number input 5599 in the group received by the neural processing unit 126 is the remainder of dividing the number of the neural processing unit 126 in the neural network unit 121 by J, where J is the neuron in the neural processing unit group 4901 The number of processing units 126 . Thus, for example, NPU 73 receives a value of one on its group numbered input 5599, NPU 353 receives a value of three on its group numbered input 5599, and NPU 6 receives a value on its group numbered input 5599. 5599 receives the value two.

In addition, when the control input 213 specifies a default value, denoted as "SELF" herein, the multitasking register 208 will select the output buffer 1104 output 4905 corresponding to the value of the number input 5599 within the group. Therefore, when a non-architectural instruction specifies to receive data from output buffer 1104 with a value of SELF (labeled OUTBUF[SELF] in instructions at addresses 2 and 7 in FIG. Its corresponding text is received from the output buffer 1104 . Therefore, for example, when the neural network unit 121 executes the non-architectural instructions of addresses 2 and 7 in FIG. To receive the text 73 from the output buffer 1104, the multitasking register 208 of the neural processing unit 353 will select the fourth (number 3) input from the four inputs 4905 to receive the text 353 from the output buffer 1104, and the neural processing unit 353 The multiplexing register 208 of the processing unit 6 selects the third input (number 2 ) among the four inputs 4905 to receive the word 6 from the output buffer 1104 . Although not used in the non-architectural program of FIG. 57, non-architectural instructions can also use SELF values (OUTBUF[SELF]) to specify the receipt of data from the output buffer 1104 so that the control input 713 specifies a default value for each neural processing unit The multitasking buffer 705 of 126 receives its corresponding text from the output buffer 1104 .

Figure 56 is a block diagram showing the data random access memory 122 and weight random access memory 122 of the neural network unit 121 when the neural network unit performs calculations associated with the Jordan Time Recurrent Neural Network of Figure 43 and utilizes the embodiment of Figure 55 An example of data configuration in memory 124 . The weight configuration in the weight random access memory 124 in the figure is the same as the example in FIG. 44 . The configuration of values in the data random access memory 122 in the figure is similar to the example of FIG. 44, except that in this example, each time step has a corresponding pair of two-column memories to load the input layer node D value and the output layer node Y values instead of using a set of four columns of memory as in the example of Figure 44. That is to say, in this example, the hidden layer Z value and the content layer C value are not written into the DRAM 122 . Instead, the output buffer 1104 is used as a class scratch storage for the hidden layer Z value and the content layer C value, as described in detail in the non-architectural procedure of FIG. 57 . The feedback feature of the aforementioned OUTBUF[SELF] output buffer 1104 can make the operation of the non-architectural program faster (this is two writes and two reads to the data random access memory 122, and two operations for the output Buffer 1104 performs two write and two read actions) and reduces the space of the DRAM 122 used in each time step, so that the data loaded in the DRAM 122 of this embodiment Approximately twice as many time steps as the embodiment of Figures 44 and 45 can be used, as shown in the Figures, ie 32 time steps.

FIG. 57 is a table showing the program stored in the program memory 129 of the neural network unit 121. The program is executed by the neural network unit 121 and uses data and weights according to the configuration of FIG. 56 to achieve a Jordan time recurrent neural network. The non-architected procedure of FIG. 57 is similar to the non-architected procedure of FIG. 45 with the differences described below.

The example program in FIG. 57 has 12 non-architectural instructions located at addresses 0 to 11, respectively. The initialize instruction at address 0 clears the accumulator 202 and initializes the value of the loop counter 3804 to 32, so that the loop group (instructions at addresses 2 to 11) is executed 32 times. The output instruction at address 1 places the zero value of accumulator 202 (cleared by the instruction at address 0 ) into output buffer 1104 . It can be observed that, during the execution of the instructions at addresses 2 to 6, the 512 neural processing units 126 correspond to and operate as 512 hidden layer nodes Z, and during the execution of the instructions at addresses 7 to 10, Corresponding to and operating as 512 output layer nodes Y. That is, the 32 executions of the instructions at addresses 2 to 6 will calculate the hidden layer node Z value for 32 corresponding time steps and put it into the output buffer 1104 for the corresponding 32 executions of the instructions at addresses 7 to 9 Execution is used to calculate the output layer node Y of these 32 corresponding time steps and write it into the data random access memory 122, and the corresponding 32 executions of the instruction at address 10 are provided to convert these 32 corresponding The content layer node C of the time step is put into the output buffer 1104 . (The content layer node C placed in the output buffer 1104 at the 32nd time step will not be used.)

In the first execution of the instructions of addresses 2 and 3 (ADD_D_ACC OUTBUF[SELF] and ADD_D_ACC ROTATE, COUNT=511), each of the 512 NPUs 126 will transfer the 512 NPUs of the output buffer 1104 The content node C values are accumulated to its accumulator 202 , and these content node C values are generated and written by the execution of instructions at addresses 0 to 1 . In the second execution of the instructions at addresses 2 and 3, each of the 512 NPUs 126 will accumulate the 512 content node C values of the output buffer 1104 into its accumulator 202, which Node C values are generated and written by the execution of instructions at addresses 7-8 and 10. More precisely, the instruction at address 2 will instruct the multitasking register 208 of each NPU 126 to select its corresponding output buffer 1104 word, as previously described, and add it to the accumulator 202; the instruction at address 3 will Instructs the NPU 126 to rotate the content node C value within the 512-word rotator formed by the collective operation of the multitasking registers 208 connected among the 512 NPUs such that Each neural processing unit 126 may accumulate the 512 content node C values to its accumulator 202 . The instruction at address 3 will not clear the accumulator 202, so the instructions at addresses 4 and 5 can add the value of the input layer node D (multiplied by its corresponding weight) to the content layer node accumulated by the instructions at addresses 2 and 3 C value.

In each execution of the instructions of addresses 4 and 5 (MULT-ACCUM DR ROW+2, WR ROW 0 and MULT-ACCUM ROTATE, WRROW+1, COUNT=511), each of the 512 neural processing units 126 The processing unit 126 will perform 512 times of multiplication, and associate the column in the data random access memory 122 with the current time step (for example: column 0 for time step 0, column 2 for time step 1, And so on, for the 512 input node D values that are column 62) for time step 31, multiplied by the weight of the row corresponding to this NPU 126 in columns 0 to 511 of the weight random access memory 124, 512 products are generated, and the accumulation results of the 512 content node C values executed by the instructions of addresses 2 and 3 are accumulated to the corresponding accumulator 202 of the neural processing unit 126 to calculate the value of the layer Z of the hidden node.

In each execution of the instruction at address 6 (OUTPUT PASSTHRU, NOP, CLRACC), the 512 accumulator 202 values of the 512 neural processing units 126 are transferred and written to the corresponding text of the output buffer 1104, and the accumulator 202 will be cleared.

During the execution of the instructions at addresses 7 and 8 (MULT-ACCUM OUTBUF[SELF], WR ROW 512 and MULT-ACCUMROTATE, WR ROW+1, COUNT=511), each neural processing unit in the 512 neural processing units 126 Unit 126 will perform 512 multiplication operations to multiply the 512 hidden node Z values in the output buffer 1104 (generated and written by the corresponding executions of the instructions at addresses 2 to 6) by the weight RAM 124 The weights corresponding to the rows of the NPU 126 in the columns 512 to 1023 of , to generate 512 multiplications accumulated to the accumulator 202 of the corresponding NPU 126 .

In each execution of the instruction at address 9 (OUTPUT ACTIVATION FUNCTION, DR OUT ROW+2), the activation function (such as hyperbolic tangent function, sigmoid function, correction function) will be executed for the 512 accumulated values to calculate the output node Y value, this output node Y value will be written into the column corresponding to the current time step in the data random access memory 122 (for example: column 1 for time step 0, column 3 for time step 1 , and so on, column 63 for time step 31). The instruction at address 9 does not clear accumulator 202.

During each execution of the instruction at address 10 (OUTPUT PASSTHRU, NOP, CLRACC), the 512 values accumulated by the instructions at addresses 7 and 8 will be put into the output buffer 1104 for the next execution of the instructions at addresses 2 and 3 used, and the accumulator 202 will be cleared.

The loop instruction at address 11 will decrement the value of the loop counter 3804, and if the new value of the loop counter 3804 is still greater than zero, a return to the instruction at address 2 is indicated.

As described in the section corresponding to FIG. 44, in the example of implementing the Jordan time recurrent neural network using the non-architectural program of FIG. The example assumes that the accumulator 202 value is passed to the content layer node C before the start function is applied, rather than the real output layer node Y value. However, for the Jordan Time RNN that applies the activation function to the accumulator 202 value to generate the content layer node C, the instruction at address 10 will be removed from the non-architectural program of FIG. 57 . In the embodiments described herein, the Elman or Jordan temporal recurrent neural network has a single hidden node layer (as shown in FIGS. Computations associated with temporal recurrent neural networks with multiple hidden layers can be efficiently performed in a manner similar to that described herein.

As described above in the section corresponding to FIG. 2 , each neural processing unit 126 operates as a neuron in an artificial neural network, and all neural processing units 126 in the neural network unit 121 are effectively processed in a massively parallel manner. Computes the neuron output values for one level of this network. The parallel processing of the neural network units, especially the rotator using the multi-tasking registers of the neural processing units, is not intuitively expected by the traditional way of computing the output of the neuron layer. Further, traditional approaches typically involve computations associated with a single neuron or a very small subset of neurons (e.g., using parallel arithmetic units to perform multiplications and additions), and then proceed to execute computations associated with the same level below The calculation of a neuron, and so on, continues in a sequential manner until the calculation of all neurons in this level is completed. In contrast, in the present invention, in each time-frequency cycle, all the neural processing units 126 (neurons) of the neural network unit 121 will execute in parallel a small set of calculations associated with generating all neuron outputs (such as a single multiplication and accumulation calculations). After about M time-frequency periods are over—M is the number of connected nodes in the current level—the neural network unit 121 will calculate the output of all neurons. In many artificial neural network configurations, because there are a large number of neural processing units 126, the neural network unit 121 can calculate its neuron output values for all neurons in the entire layer at the end of M time-frequency periods. As described herein, this computation is efficient for all types of artificial neural network computations, including but not limited to feed-forward and temporally recurrent neural networks such as Elman, Jordan and long short-term memory networks. Finally, although in the embodiment herein, the neural network unit 121 is configured as 512 neural processing units 126 (for example, adopting a wide text configuration) to perform the calculation of the time recurrent neural network, the present invention is not limited thereto, and the neural network unit 121 is configured as an embodiment of 1024 NPUs 126 (e.g., in a narrow-text configuration) to perform RTNN calculations, and NPUs 121 with other numbers of NPUs 126 than 512 and 1024 as previously described, Also belong to the category of the present invention.

But the above-mentioned person is only preferred embodiment of the present invention, when can not limit the scope of the present invention implementation with this, promptly all the simple equivalent changes and modifications that are done according to the patent scope of the present invention and the content of the description of the invention, All still belong to the scope that the patent of the present invention covers. For example, software can perform functions, manufacture, shape, simulate, describe and/or test, etc. of the devices and methods described in the present invention. This can be achieved by common programming languages (such as C, C++), hardware description languages (HDL) including Verilog HDL, VHDL, etc., or other existing programs. This software can be installed on any known computer-usable media, such as tape, semiconductor, magnetic disk, optical disk (such as CD-ROM, DVD-ROM, etc.), network connection, wireless or other communication media. Embodiments of the apparatus and methods described herein may be embodied in a semiconductor intellectual property core, such as a microprocessor core (eg, implemented in a hardware description language) and translated to hardware through the fabrication of an integrated circuit. Furthermore, the apparatus and methods described herein may also comprise a combination of hardware and software. Therefore, any embodiments described herein are not intended to limit the scope of the present invention. In addition, the present invention can be applied to microprocessor devices of general general-purpose computers. Finally, those skilled in the art can use the concepts and embodiments disclosed in the present invention as a basis to design and adjust different structures to achieve the same purpose, which is within the scope of the present invention.

Claims (21)

1. A neural network unit, characterized in that, comprising: a first memory, loaded with elements of a data matrix; a second memory loaded with elements of a convolution kernel; and A neural processing unit (NPU) array, coupled to the first memory and the second memory, each of the neural processing units includes: a multitasking register having an output, and receiving a corresponding element from a column of the first memory and receiving the multitasking register output of an adjacent neural processing unit; a buffer having an output and receiving a corresponding element from a column of the second memory; an accumulator with an output; and An arithmetic unit receives the outputs of the register, the multitasking register and the accumulator, and performs a multiply-accumulate operation on them; Wherein, for each of the multiple sub-matrices of the data matrix, each of the arithmetic units selectively receives the element from the first memory or the output from the multi-tasking buffer of the adjacent neural processing unit performing a series of multiplication and accumulation operations on the element, and accumulating a convolution operation result of the sub-matrix and the convolution kernel into the accumulator. 2. The neural network unit according to claim 1, wherein the neural network unit writes the convolution operation results of the sub-matrices into the first and the second memories. 3. The neural network unit according to claim 1, wherein the neural network unit is included in a processor, the processor includes architectural registers, and the processor executes an instruction set of the processor architectural instructions to write the data matrix from the architectural buffer to the first memory, write the convolution kernel from the architectural buffer to the second memory, and write the volume from the first and second memories The result of the product operation is read into the architectural register. 4. The neural network unit according to claim 1, wherein the neural processing unit array comprises N neural processing units, the convolution kernel is a matrix of K x K, and each of the sub-matrixes is a K x A matrix of K, the data matrix is a J x N matrix, and each of the J columns of the first memory is loaded with N elements of different columns of the J columns of the data matrix. 5. The neural network unit according to claim 4, wherein the plurality of sub-matrices is approximately (J x N)/(K x K) sub-matrices. 6. The neural network unit according to claim 4, characterized in that, each column in the K x K columns of the second memory is loaded with columns as the main order in which the K x K elements of the convolution kernel are different N instances of elements. 7. The neural network unit according to claim 6, wherein the neural network unit performs the following steps: (a) the neural network unit initializes a first column address to point to the first column of the data matrix in the first memory; (b) the neural processing unit clears the accumulator to zero; (c) the neural network unit initializes a second column address to point to the first column of the K x K columns in the second memory; (d) For each of the K iterations: For an example: the multitasking register receives the column data matrix element pointed to by the first column address from the first memory to provide to the arithmetic unit, and increments the first column address; the buffer receives the column convolution kernel element pointed to by the second column address from the second memory to provide to the arithmetic unit, and increments the second column address; and the arithmetic unit performs the multiply-accumulate operation; and For K-1 examples: the multitasking register receives the multitasking register output of the adjacent neural processing unit to provide to the arithmetic unit; The buffer receives the column convolution kernel element pointed to by the second column address from the second memory to provide to the arithmetic unit, and the neural network unit increments the second column address; and The arithmetic unit performs the multiply-accumulate operation. 8. The neural processing unit according to claim 7, wherein the neural processing unit performs the following steps: (e) the neural network unit writes the convolution operation result into a column of the first or the second memory; and (f) Decrement the first column address by K-2. 9. The neural network unit according to claim 8, wherein the neural network unit iterates steps (b) to (f) approximately J times. 10. The neural network unit according to claim 8, characterized in that, in step (e), the neural processing unit will perform a division operation on the result of the convolution operation, and the results of the division operation are written to into the column in the first or the second memory instead of the result of the convolution operation. 11. A method for operating a neural network unit, characterized in that the neural network unit has a neural processing unit (NPU) array, and each of the neural processing units includes a multitasking register, a register, an accumulator and an arithmetic unit, wherein the multitasking register has an output and the multitasking register receives a corresponding element from a column of a first memory and receives the multitasking register output of an adjacent neural processing unit, The register has an output, and the register receives a corresponding element from a column of a second memory, the accumulator has an output, and the arithmetic unit receives the register, the multitasking register and the accumulator output, and perform a multiply-accumulate operation on it, the method comprising: using the first memory to load elements of a data matrix; using the second memory to load elements of a convolution kernel; For each of the submatrices of the data matrix: selectively receiving, with each of the arithmetic units, the element from the first memory or the element from the multitasking register output of the neighboring NPU; and A series of multiplication and accumulation operations are performed, and a convolution operation result of the sub-matrix and the convolution kernel is accumulated into the accumulator. 12. The method of claim 11, further comprising: The convolution operation results of the sub-matrices are written into the first memory and the second memory. 13. The method according to claim 11, wherein the neural network unit is included in a processor, and the processor includes an architectural register, the method further comprising: Using the processor, execute an architectural instruction of an instruction set of the processor to write the data matrix from the architectural register to the first memory, and write the convolution kernel from the architectural register to the second memory, and read the convolution operation result from the first and second memories to the architectural register. 14. The method according to claim 11, wherein the neural processing unit array comprises N neural processing units, the convolution kernel is a matrix of K x K, and each sub-matrix is a matrix of K x K matrix, the data matrix is a J x N matrix, and each of the J columns of the first memory is loaded with N elements of different columns of the J columns of the data matrix. 15. The method according to claim 14, wherein the plurality of sub-matrices is approximately (J x N)/(K x K) sub-matrices. 16. The method according to claim 14, characterized in that, each column in the K x K columns of the second memory takes columns as the main order to load the different elements of the K x K elements of the convolution kernel N examples. 17. The method of claim 16, further comprising: (a) initializing a first column address to point to the first column of the data matrix in the first memory; (b) clearing the accumulator to zero; (c) initializing a second column address to point to the first column of the K x K columns in the second memory; (d) For each of the K iterations: For an example: receiving the column data matrix element pointed to by the first column address from the first memory for providing to the arithmetic unit, and incrementing the first column address; receiving the column convolution kernel element pointed to by the second column address from the second memory to provide to the arithmetic unit, and incrementing the second column address; and using the arithmetic unit, performing the multiply-accumulate operation; and For K-1 examples: using the multitasking register, receiving the multitasking register output of the neighboring neural processing unit for providing to the arithmetic unit; and receiving the column convolution kernel element pointed to by the second column address from the second memory to provide to the arithmetic unit, and incrementing the second column address; and With the arithmetic unit, the multiply-accumulate operation is performed. 18. The method of claim 17, further comprising: (e) writing the result of the convolution operation into a column of the first or the second memory; and (f) Decrement the first column address by K-2. 19. The method of claim 18, further comprising: Steps (b) to (f) are iterated J times. 20. The method according to claim 18, wherein in step (e), a division operation is performed on the result of the convolution operation, and the results of the division operation are written into the first or The column in the second memory is not the result of the convolution operation. 21. A computer program product encoded on at least one non-transitory computer usable medium for use by a computer device, comprising: Computer-usable program code embodied in the medium for describing a neural network unit, the computer-usable program code comprising: a first program code for describing a first memory loaded with elements of a data matrix; The second program code is used to describe a second memory, and the second memory is loaded with elements of a convolution kernel; The third program code is used to describe a neural processing unit (NPU) array, the neural processing unit array is coupled to the first memory and the second memory, each of the neural processing units includes: a multitasking register having an output, and receiving a corresponding element from a column of the first memory and receiving the multitasking register output of an adjacent neural processing unit; a buffer having an output and receiving a corresponding element from a column of the second memory; an accumulator with an output; and an arithmetic unit receiving the outputs of the register, the multitasking register and the accumulator, and performing a multiply-accumulate operation thereon; and Wherein, for each of the multiple sub-matrices of the data matrix, each of the arithmetic units selectively receives the element from the first memory or the output from the multi-tasking buffer of the adjacent neural processing unit performing a series of multiplication and accumulation operations on the element, and accumulating a convolution operation result of the sub-matrix and the convolution kernel into the accumulator.