CN115658146B - A kind of AI chip, tensor processing method and electronic equipment - Google Patents

Abstract

The application relates to an AI chip, a tensor processing method and electronic equipment, and belongs to the technical field of computers. The AI chip includes: the device comprises a vector register, an engine unit and a vector operation unit; the engine unit is used for acquiring source operands required by a vector operation instruction from original tensor data according to a tensor addressing mode; the vector register is used for storing the source operand acquired by the engine unit; the vector operation unit is used for acquiring a source operand of the vector operation instruction from the vector register to perform operation to obtain an operation result. In the application, the vector register is used for storing the source operand acquired by the engine unit, so that the vector operation unit can directly acquire the source operand required by the vector operation instruction from the vector register to perform operation.

Description

A kind of AI chip, tensor processing method and electronic equipment

technical field

The application belongs to the field of computer technology, and specifically relates to an AI chip, a tensor processing method and electronic equipment.

Background technique

With the development of artificial intelligence (AI), neural networks have become one of the most popular artificial intelligence technologies today. Calculations between tensors are often involved in neural networks. The architecture of the current AI chip used for tensor calculations is shown in Figure 1. When performing tensor calculations, the general steps are as follows: Steps 1. The processing unit (processing core) calculates the address of the tensor element (tensor element) corresponding to one or more source operands; Step 2. Read one or more source operands from the memory into the vector operation unit according to the above address ; Step 3, the vector operation unit calculates the result.

The AI chip only supports real-time addressing when addressing. After calculating the address of the tensor element, it is necessary to read one or more source operands from the memory into the vector operation unit in time according to the above address. If an error occurs, re-addressing is required, resulting in low operational efficiency and high overhead.

Contents of the invention

In view of this, the purpose of this application is to provide an AI chip, a tensor processing method and electronic equipment to improve the existing AI chip that only supports real-time addressing, and it is easy to occur that if an error occurs in the middle of the calculation, it needs to be re-addressed, resulting in calculation efficiency. low cost and high cost.

The embodiment of the application is realized like this:

In the first aspect, the embodiment of the present application provides an AI chip, including: a vector register, an engine unit, and a vector operation unit; Source operand, the number of source operands obtained by one addressing of the tensor addressing mode is greater than or equal to 1; the vector register is connected to the engine unit, and the vector register is used to store the source operation obtained by the engine unit number; the vector operation unit is connected to the vector register, and the vector operation unit is used to obtain the source operand of the vector operation instruction from the vector register to perform an operation to obtain an operation result. There can be one or more tensor elements determined by the engine unit in one addressing mode using the tensor addressing mode.

In the embodiment of the present application, the engine unit adopts the tensor addressing mode to obtain the source operands required by the vector operation instruction from the tensor data, so that at least one source operand can be obtained in one addressing, which greatly improves the speed of obtaining the source operands. The efficiency makes it possible to find the required source operand with fewer instructions; at the same time, the vector register is used to store the source operand obtained by the engine unit, so that the vector operation unit can directly obtain the vector operation from the vector register The source operand of the instruction is used for calculation. This design method can realize the separation of addressing and operation, and can support obtaining the source operand required by the vector operation instruction in advance, and then directly read it when performing the operation. Even if an error occurs in the middle of the operation, there is no need to re-address, and the corresponding source operand can be directly obtained from the vector register, which can improve the efficiency of the operation.

With reference to a possible implementation manner of the embodiment of the first aspect, the parameters of the tensor addressing mode include: a starting address representing the starting point of addressing, an end address representing the end point of addressing, and an address representing the offset magnitude of the addressing pointer. Step size, the size that characterizes the shape of the data obtained by addressing.

In the embodiment of this application, by improving and extending the existing slice addressing mode, the parameters contained in the tensor addressing mode (which can also be regarded as a new slice addressing mode) are changed from the original slice addressing mode The three parameters included in the mode are expanded to include more parameters (including size information used to characterize the shape of the data obtained by addressing), so that addressing efficiency can be improved and redundant instructions can be reduced.

With reference to a possible implementation manner of the embodiment of the first aspect, the parameters of the tensor addressing mode further include a characteristic parameter characterizing the preservation of an incomplete shape of the data obtained by addressing.

In the embodiment of this application, by introducing characteristic parameters (such as represented by partial) that characterize the retention of the data shape obtained by addressing as an incomplete shape, it is possible to configure the value of the characteristic parameter, so that it is possible to flexibly decide whether to retain the incomplete shape The data contained by the shape.

With reference to a possible implementation manner of the embodiment of the first aspect, the tensor addressing mode includes a nested double tensor addressing mode, and the double tensor addressing mode includes: an outer iteration addressing mode and an inner iteration addressing mode An addressing mode, wherein the inner iterative addressing mode addresses the tensor data obtained by addressing in the outer iterative addressing mode.

In the embodiment of this application, the nested double tensor addressing mode is used for addressing to realize the independent addressing of the outer iterative addressing mode and the inner iterative addressing mode. Compared with the single addressing mode, the addressing The addressing efficiency is higher, so that only one instruction is needed to realize the addressing of two single-layer instructions, and the inner iteration addressing mode is based on the tensor data obtained by the outer iteration addressing mode, and the outer iteration addressing mode The data addressed by the address mode is only an intermediate result, and does not need to be read and written. Compared with the addressing of the single-layer tensor addressing mode, the data reading of the first layer (outer iterative addressing mode) is reduced. Write.

With reference to a possible implementation manner of the embodiment of the first aspect, the engine unit is further connected to an external memory, and the external memory is used to store the original tensor data.

In the embodiment of this application, when the vector register is connected to the external memory through the engine unit, at this time, the external memory can be directly used to store the original tensor data instead of storing the source operand obtained by the engine unit, so that the storage of the external memory The space does not need to be designed to be very large (because tensor operations involve a large amount of repeated data, the data volume of the data addressed by the tensor addressing mode may be larger than the original tensor data).

With reference to a possible implementation manner of the embodiment of the first aspect, there are multiple vector registers, and different vector registers store different source operands.

In the embodiment of the present application, by setting multiple vector registers, different vector registers store different source operands, so that when performing vector operations, required operands can be read from each vector register at the same time, which can improve operation efficiency.

With reference to a possible implementation manner of the embodiment of the first aspect, the engine unit includes: a plurality of addressing engines, one of the addressing engines corresponds to one of the vector registers, and each of the addressing engines is configured to The respective tensor addressing modes obtain the source operands required by the vector operation instructions from the corresponding raw tensor data.

In the embodiment of the present application, by setting multiple addressing engines, and one addressing engine corresponds to one vector register, each addressing engine can start from the corresponding original tensor according to its own tensor addressing mode during addressing. Obtaining the source operands required by the vector operation instructions in the data can not only improve the addressing efficiency, but also do not interfere with each other.

With reference to a possible implementation manner of the embodiment of the first aspect, the engine unit further includes: a main engine, configured to send a control command to each of the addressing engines, and control each of the addressing engines to The vector addressing mode is used for addressing, and the source operand required for the vector operation instruction is obtained from the corresponding original tensor data.

In the embodiment of the present application, the independent addressing of each addressing engine is controlled by setting the main engine to send control commands to each addressing engine, and the centralized control is performed by the main engine to ensure the accuracy of the source operands obtained by each addressing engine The shape (shape) is consistent, thus ensuring the accuracy of the operation.

With reference to a possible implementation of the embodiment of the first aspect, the control commands include: Advance advance command, Reset reset command, and NOP no-operation command. When the addressing engine sends a control command, send a combined control command composed of at least one control command in the three control commands, so as to control each of the addressing engines according to their respective tensor addressing modes in the original Different dimensions of tensor data are addressed.

In the embodiment of the present application, by sending a combined control command composed of at least one of the Advance command, the Reset command, and the NOP empty operation command to the addressing engine, for example, the combined control command is: "W cmd : NOP; H cmd: NOP", "W cmd: Advance; H cmd: NOP", etc., so as to control each addressing engine in different dimensions of the original tensor data (such as width W, Height H and other dimensions) for addressing, only need to combine simple commands, so that multiple functions can be realized.

In the second aspect, the embodiment of the present application also provides an electronic device, including: a memory and the AI chip provided in the first aspect above; a memory for storing tensor data required for calculation; the AI chip and the memory Connecting, the AI chip is used to obtain the source operands required by the vector operation instruction from the original tensor data stored in the memory according to the tensor addressing mode, and to obtain the source operand according to the vector operation instruction The operands are operated and the result of the operation is obtained.

In the third aspect, the embodiment of the present application also provides a tensor processing method, including:

According to the tensor addressing mode, the source operands required by the vector operation instruction are obtained from the original tensor data, wherein, the number of source operands obtained by one addressing of the tensor addressing mode is greater than or equal to 1; the acquired The source operand is stored in a vector register; the source operand of the vector operation instruction is acquired from the vector register to perform an operation to obtain an operation result.

The method can be applied to the aforementioned AI chip of the first aspect and/or the aforementioned electronic device of the second aspect. In this method, there can be one or more tensor elements obtained by one addressing using the tensor addressing mode. Regarding the principles and beneficial effects of the method, reference may be made to the relevant descriptions in other aspects of the embodiments or implementation manners.

With reference to a possible implementation of the embodiment of the third aspect, obtaining the source operands required by the vector operation instruction from the original tensor data according to the tensor addressing mode includes: using the control command sent by the addressing engine to the main engine Next, the source operands required by the vector operation instruction are obtained from the original tensor data according to the tensor addressing mode.

With reference to a possible implementation of the embodiment of the third aspect, the parameters of the tensor addressing mode include: a starting address representing the starting point of addressing, an end address representing the end point of addressing, and an address representing the offset magnitude of the addressing pointer. Step size, the size that characterizes the shape of the data obtained by addressing.

With reference to a possible implementation manner of the embodiment of the third aspect, the tensor addressing mode includes a nested double tensor addressing mode, and the double tensor addressing mode includes: an outer iteration addressing mode and an inner iteration addressing mode Addressing mode: according to the tensor addressing mode, the source operand required for the vector operation instruction is obtained from the original tensor data, including: using the outer iteration addressing mode to select at least one candidate from the original tensor data An area, the candidate area includes a plurality of tensor elements; the source operand required by the vector operation instruction is obtained from the at least one candidate area by using the inner iteration addressing mode.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some of the present application. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort. The above and other objects, features and advantages of the present application will be more clearly shown by the accompanying drawings. Like reference numerals designate like parts throughout the drawings. The drawings are not intentionally scaled and drawn according to the actual size, and the emphasis is on illustrating the gist of the application.

FIG. 1 is a schematic structural diagram of an AI chip and a memory in the prior art.

FIG. 2 is a schematic diagram of the principle of addressing in the slice addressing mode in the prior art.

FIG. 3 shows a schematic structural diagram of the connection between an AI chip and a memory provided by an embodiment of the present application.

FIG. 4 shows a schematic diagram of a principle of broadcasting provided by an embodiment of the present application.

FIG. 5 shows another schematic diagram of the principle of broadcasting provided by the embodiment of the present application.

FIG. 6 shows a schematic diagram of the principle of the first tensor addressing mode provided by the embodiment of the present application.

FIG. 7 shows a schematic diagram of the principle of the second tensor addressing mode provided by the embodiment of the present application.

FIG. 8 shows a schematic structural diagram of another AI chip provided by an embodiment of the present application.

FIG. 9 shows a schematic diagram of the principle of addressing according to the addressing mode of 0:8:2:5 provided by the embodiment of the present application.

FIG. 10 shows a schematic diagram of the principle of addressing according to the addressing mode of 0:5:2:1 provided by the embodiment of the present application.

FIG. 11 shows a schematic structural diagram of an electronic device provided by an embodiment of the present application.

FIG. 12 shows a schematic diagram of the principle of addressing according to the addressing mode of 0:4:1:3 provided by the embodiment of the present application.

FIG. 13 shows a schematic diagram of a tensor processing method provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

It should be noted that like numerals and letters denote similar items in the following figures, therefore, once an item is defined in one figure, it does not require further definition and explanation in subsequent figures. The term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements but also other elements not expressly listed elements, or also elements inherent in such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

After research, the inventor found that, for the calculation process between tensors in the prior art, it takes a long time to calculate the address of the tensor element corresponding to the source operand, that is, a large part of the time is wasted in calculating the source operand The address of the corresponding tensor element is mainly because the existing method is based on the slice addressing mode of numpy (Numerical Python, an open source numerical computing extension of Python). For example, for a tensor array a, use the idea of a[start:stop:step] to extract a new array, and can support any dimension in the multidimensional array for processing. Among them, start represents the start address of addressing, stop represents the end address of addressing, and step is equivalent to stride (step size), which represents the offset range of the addressing pointer. Taking the slice addressing mode start:stop:step=0:5:2 as an example, its addressing principle is shown in Figure 2. This slice addressing mode can only get one tensor element at a time, for example, (0, 0), (2, 0), (4, 0), (0, 2), (2, 2 ), (4,2), (0,4), (2,4), (4,4). This slice addressing mode can only get one tensor element at a time. Since tensor calculation involves a large number of source operands, if you want to obtain multiple tensor elements corresponding to many source operands, you need to use a large number of Instructions are addressed multiple times, which causes a large number of instructions to be wasted on addressing, that is, wasted on ineffective calculations.

Based on this, this application provides a brand new AI chip, which can determine the address of at least one tensor element corresponding to the source operand in one addressing, which can greatly improve the efficiency of calculating the address of the tensor element corresponding to the source operand , so that the required source operand can be found with only a few instructions, which greatly reduces redundant instructions, improves effective instruction density, improves performance, and simplifies programming. At the same time, the vector register is used to store the source operand obtained by the engine unit, so that the vector operation unit can directly obtain the source operand of the vector operation instruction from the vector register for operation. The required source operands can be read directly when performing subsequent operations, so as to realize the separation of addressing and operation. Even if there is an error in the middle of the operation, there is no need to re-address, and the corresponding source operation can be obtained directly from the vector register. The number is enough, which can improve the efficiency of operation.

The AI chip can accelerate the addressing and operation of tensors at the hardware level. The hardware can support multiple tensor addressing modes, and has good compatibility, which is conducive to the rapid completion of calculations between tensors.

In addition, the embodiment of the present application also proposes a new addressing mode for tensor addressing in terms of flexibility, ease of use, and processing efficiency. The design idea of accessing the elements in the array is further expanded and designed a new slice-based tensor addressing mode; on the other hand, an addressing mode that can be nested and capable of double tensor addressing is designed . These new addressing modes can all be applied on the AI chip provided by the embodiment of this application, which is beneficial to reduce the total amount of instructions required, improve hardware processing performance, and improve the processing performance of tensors when faced with calculations between tensors. processing efficiency.

The principle of the AI chip provided by the embodiment of the present application will be described below with reference to FIG. 3 . The AI chip includes a vector register, an engine unit, and a vector operation unit. Wherein, the vector register is directly connected to the vector operation unit, and the engine unit may be directly connected to the vector operation unit.

The engine unit is used to obtain the source operands required for vector operation instructions from the original tensor data according to the tensor addressing mode, and the source operands (number) obtained by one addressing in the tensor addressing mode are greater than or equal to 1. Tensor data is multi-dimensional data, which is usually expanded in a certain way (such as linear layout, tile layout, etc.) and stored in devices with storage functions such as memory and on-chip memory.

The vector register is used to store the source operand obtained by the engine unit, so that even if an error occurs in the middle of the operation, there is no need to re-address, and the corresponding source operand can be obtained directly from the vector register.

The vector operation unit is connected with the vector register, and the vector operation unit is used to obtain the source operand of the vector operation instruction from the vector register for operation to obtain an operation result. Wherein, the vector operation instruction refers to an instruction that can calculate more than two operands at the same time. The types of vector operation instructions can be various operation types such as addition, subtraction, multiplication, and multiply-accumulate.

The vector operation unit is directly connected to the vector register. Since the data stored in the vector register is the source operand required by the vector operation instruction addressed from the original tensor data according to the tensor addressing mode, the vector operation unit directly accesses the vector The register can get the content of the tensor address.

Wherein, when the vector register is directly connected to the engine unit, the engine unit is also connected to the external memory. At this time, it is equivalent to that the vector register is connected to the external memory through the engine unit. External memory is used to store raw tensor data. Take a vector operation instruction with three operands, such as A*B+C as an example, where A, B, and C are all source operands, where A, B, and C can be single elements or multiple array of elements. The memory is used to store the original tensor data where operand A is located, store the original tensor data where operand B is located, and store the original tensor data where operand C is located. The engine unit obtains the operand A of the vector operation instruction from the original tensor data where the operand A is located according to the tensor addressing mode, stores it in the vector register, and obtains the vector operation from the original tensor data where the operand B is located The operand B of the instruction is stored in the vector register, and the operand C of the vector operation instruction is obtained from the original tensor data where the operand C is located, and stored in the vector register.

Wherein, the aforementioned AI chip may be an integrated circuit chip, which has data processing capability and can be used to process operations between tensors. For example, it can be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; it can also be a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), Field Programmable Gate Array (Field Programmable Gate Array, FPGA) or other programmable logic devices. The general-purpose processor can also be a microprocessor or the AI chip can also be any conventional processor, such as a graphics processor (Graphics Processing Unit, GPU), a general-purpose graphics processor (General Purposecomputing on Graphics Processing Unit, GPGPU )wait.

The inventors found that the operation process of tensor elements is relatively regular, and there are two general operation processes as follows:

The first is to perform a one-to-one operation on tensor elements in two or more tensor data with the same dimension size (called element-by-element operation or element-wise operation for short). For example, assuming that the width (W) and height (H) of the two tensor data are both 5, that is, W*H is 5*5, then the tensor elements in the two tensor data are performing one-to-one operations When , it is a one-to-one operation between tensor elements at corresponding positions.

The second is to operate on a certain tensor element in one tensor data and a set of tensor elements in another tensor data (called broadcasting). For example, for an image tensor (the first operand), it is necessary to divide each pixel value by 255 (the second operand), assuming that the dimensions of the two operands are: the first operation The number of channels, width and height of the number are: 3, 224, 224 respectively; the number of channels, width and height of the second operand are: 1, 1, 1 respectively.

At this time, the second operand will be broadcast in three dimensions (namely, the number of channels, width, and height mentioned above), that is, before performing the division operation, it is necessary to convert the second operand with the dimension (1, 1, 1) Two operands (value 255), expanded into a tensor of dimension (3, 224, 224), and the value of each dimension of this tensor is 255. At this time, the dimensions of the two operands are the same, and the division operation can be performed, and then the tensor elements in the two tensor data can be operated one-to-one.

For another example, suppose the number of channels, width and height of the first operand are respectively: 3,224,224, and the number of channels, width and height of the second operand are respectively: 3,1,224, then for the second operand , you only need to broadcast in the width direction, that is, expand the second operand with dimension (3, 1, 224) into a tensor with dimension (3, 224, 224), at this time, the dimensions of the two operands Same size, division operation is possible.

Among them, for a broadcast operation to occur, the dimensions of the two tensor data need to be the same, and at least one dimension of one of the tensor data is 1. When performing broadcast processing, all other dimensions of the broadcast tensor data will be broadcasted Elements are replicated along the dimension of .

In order to better understand the above-mentioned broadcasting operation, the following will be described in conjunction with the schematic diagrams shown in FIG. 4 and FIG. 5 . In Figure 4, since the operand 5 is different from the array "np.arange(3)" (that is, the first array) in the width dimension, it is necessary to broadcast 5 in the width direction before adding , so that the dimensions in the width direction are consistent with the dimensions of the array "np.arange(3)", and then added. Similarly, in Figure 5, due to the array "np.arange(3).reshape((3,1))" (that is, the second array) and the array "np.arange(3)", the width and height The dimensions are different, so it is necessary to broadcast the array "np.arange(3).reshape((3, 1))" in the width direction, and the array "np.arange(3)" in the height direction Broadcast so that the dimensions of the two are the same, and then add them together.

The principles and details of element-by-element operations and broadcast operations on tensors are well known in the art and will not be introduced here.

In addition, the address calculation modes of common tensor elements (such as the slice addressing mode) are also relatively regular. Among them, slice refers to accessing tensor elements in a certain order or span.

In addition to improving the architecture of the AI chip provided by this application, in order to improve the addressing efficiency of tensor elements, the inventors of this application proposed a method based on making full use of the regularity of existing tensor operations. A brand new tensor addressing mode. The tensor addressing mode improves and expands the existing slice addressing mode, so that the parameters contained in the tensor addressing mode (which can also be regarded as a new slice addressing mode) are changed from the original slice addressing mode. The three parameters included in the address mode are expanded to include more parameters, which can improve addressing efficiency and reduce redundant instructions.

The parameters of the tensor addressing mode in this application include: the starting address representing the starting point of addressing (such as represented by start), the end address representing the end point of addressing (such as represented by stop), and the value representing the offset magnitude of the addressing pointer The step size (such as represented by step), the size representing the shape of the data obtained by addressing (such as represented by size). Then the expression of the tensor addressing mode can be recorded as [start:stop:step:size]. Among them, size is used to describe the size of the data shape (shape) obtained by addressing, that is, at each step, an element point contained in a shape will be extracted instead of only one point. The values of start, stop, step, and size in the above expressions are all configurable to apply to calculations between various tensors.

Optionally, the parameters of the tensor addressing mode also include: characteristic parameters that characterize the retention of the data shape obtained by addressing as an incomplete shape (such as represented by partial, used to reflect the integrity of the local tensor data shape), here , the expression for the tensor addressing mode is [start:stop:step:size:partial]. As an implementation, when partial=false, that is, when the partial parameter is set to false, the points contained in the incomplete shape at the edge will be discarded, that is, not retained; when partial=true, that is, the partial parameter is set to When set to true, the points contained in the incomplete shape at the edge need to be kept. Compared with the parameters of the existing slice addressing mode, the parameters of the tensor addressing mode in this application add two parameters, size and partial.

For a better understanding, the following takes the tensor addressing mode [start:stop:step:size] =0:6:2:3 as an example to explain the cases of partial=false and partial=true respectively. Among them, in this expression, start=0, stop=6, step=2, size=3.

When partial=false, the points contained in the incomplete shape on the edge will be discarded. The schematic diagram is shown in Figure 6. In this case, every time you slide a step, it will judge whether the currently addressed element point can form a complete shape according to the size. If it cannot form a complete shape, then the point addressed by this step will be discarded. When partial=true, the points contained in the incomplete shape on the edge need to be kept, and its schematic diagram is shown in Figure 7. In this case, each step will still judge whether the currently addressed point can form a complete shape according to the size. Even if it cannot form a complete shape, the point addressed by this step needs to be retained.

By comparing Figure 6 and Figure 7, it can be seen that when partial=false, the tensor element points contained in the incomplete shape at the edge in Figure 6 will be discarded. In the examples in Figure 6 and Figure 7, a complete shape contains 9 points. Since the points of the shape on the edge are less than 9, it is an incomplete shape. When partial=false, the points contained in the incomplete shape will be Abandoned; when partial=true, the points contained in the incomplete shape need to be kept.

By comparing Figure 6 (or Figure 7 ) with Figure 2, it can be seen that: using the tensor addressing mode shown in this application, one step can determine multiple tensor elements. One addressing can determine the addresses of tensor elements corresponding to multiple source operands. For example, in the above example, one (ie, one step) addressing will address the addresses of 9 tensor elements, while using the existing In this way, 9 addressings are required to address the addresses of the 9 tensor elements, and correspondingly, more instructions are required to address the addresses of the 9 tensor elements. Therefore, compared with the existing addressing mode, the addressing mode provided by the embodiment of the present application can greatly improve the efficiency of calculating the address of the tensor element corresponding to the source operand, so that only fewer instructions are needed to find the required The number of source operands can reduce redundant instructions, increase effective instruction density, improve performance, and simplify programming.

It is understandable that when each step only needs to determine one tensor element, it can be realized by configuring size=1. At this time, each step only addresses one tensor element. The addressing mode at this time is the same as the current Some addressing modes are similar (the existing addressing logic can be compatible with the hardware by changing the value of the parameter), and its schematic diagram is shown in Figure 1. Or, directly use the existing addressing mode for addressing.

It should be noted that when each step only needs to address 1 tensor element, its size parameter can also be configured to be greater than 1. In this case, the required tensor is further selected from the shape obtained according to size and step element. Therefore, when each step only needs to address 1 tensor element, the size may not be 1. In this way, the tensor addressing mode provided by the embodiment of the present application can be compatible to realize the existing slice addressing effect, taking into account processing efficiency and addressing flexibility.

Among them, the number of vector registers can be multiple, and the source operands (which are part of the tensor data) stored in different vector registers can be different. Take a vector operation instruction with three operands, such as A*B+C, as an example. At this time, the number of vector registers may include three, for example, vector register 1 , vector register 2 , and vector register 3 . Vector register 1 can be used to store source operand A, vector register 2 can be used to store source operand B, and vector register 3 can be used to store source operand C. This application does not limit the specific instruction expression form.

In a vector operation instruction, each operand corresponds to an addressing mode, and the addressing modes of multiple operands match each other. Broadcast operations may occur during the matching process, and the shape of multiple operands after matching same.

It can be understood that different operands (such as A and B above) can also be stored in the same vector register, as long as the space of the vector register is large enough, therefore, the operands stored in different vector registers in the above example can be Different situations are to be understood as limitations on the present application.

When there are multiple vector registers, one or more addressing engines can be used to address tensor data stored in multiple vector registers.

In order to improve addressing efficiency, under an optional implementation manner, as shown in Figure 8, the engine unit includes: a plurality of addressing engines, one addressing engine corresponds to a vector register, and each addressing engine is used to The tensor addressing mode obtains the source operands required by the vector operation instruction from the corresponding raw tensor data. For example, taking 3 vector registers as an example to store the 3 required source operands respectively, it can be performed by 3 addressing engines (for example, respectively marked as addressing engine 1, addressing engine 2, and addressing engine 3). addressing. Wherein, the addressing engine 1 corresponds to the vector register 1, and is used to obtain the source operand A required for the vector operation instruction from the original tensor data where the source operand A is located according to the tensor addressing mode of the addressing engine 1; The addressing engine 2 corresponds to the vector register 2, and is used to obtain the source operand B required for the vector operation instruction from the original tensor data where the source operand B is located according to the tensor addressing mode of the addressing engine 2; The addressing engine 3 corresponds to the vector register 3, and is used to obtain the source operand C required by the vector operation instruction from the original tensor data where the source operand C is located according to the tensor addressing mode of the addressing engine 3.

It can be understood that, in an optional implementation mode, the same addressing engine can also obtain multiple source operands required for vector operation instructions from multiple original tensor data, for example, by addressing engine 1 to obtain the source operand A, source operand B, and source operand C of the preceding example.

Wherein, each addressing engine, such as addressing engine 1, addressing engine 2, and addressing engine 3 mentioned above, is independent and does not interfere with each other. When each addressing engine obtains the source operand required by the vector operation instruction, it uses an independent tensor addressing mode for independent addressing, that is, each source operand of the vector operation instruction corresponds to an independent tensor volume addressing mode. For example, the above source operand A corresponds to tensor addressing mode 1, source operand B corresponds to tensor addressing mode 2, and source operand C corresponds to tensor addressing mode 3. The types of parameters contained in tensor addressing mode 1, tensor addressing mode 2, and tensor addressing mode 3 can be the same. The exact values may vary.

It can be understood that when multiple addressing engines are independently addressing, all addressing engines may use the tensor addressing mode provided by this application for addressing, or some of them may use the tensor addressing mode provided by this application The addressing mode is used for addressing, and the remaining part is addressed using the existing addressing mode (index addressing mode, numpy's basic slice addressing mode). For example, tensor addressing mode 1, tensor addressing mode 2, and tensor addressing mode 3 can not only be the same type of addressing mode (such as the new slice addressing mode), but also correspond to different addressing modes Mode, for example, tensor addressing mode 1 corresponds to the tensor addressing mode in this application (new slice addressing mode), tensor addressing mode 2 adopts the original 3-parameter slice addressing mode, tensor addressing Mode 3 uses the original indexed addressing mode. That is to say, compared with adopting all existing addressing modes, the addressing efficiency can also be improved to a certain extent, and the number of instructions required for addressing can be reduced.

As an implementation manner, in addition to its own independent addressing mode, each addressing engine can also be configured to support the ability to process broadcast operations.

In order to facilitate the addressing of each addressing engine, in an optional implementation manner, the engine unit also includes: a main engine, which is connected to each addressing engine respectively, and the main engine is used to send each addressing engine The control command controls each addressing engine to address according to the tensor addressing mode, and obtains the source operand required by the vector operation instruction from the corresponding original tensor data. Addressing efficiency can be improved by introducing a master engine to centrally control these independent addressing engines. The main engine sends control commands to the addressing engine of the operand at each step, so that the corresponding addressing engine traverses each dimension of the tensor data in order from low-dimensional to high-dimensional, and addresses in the corresponding dimension.

Wherein, the control commands include an Advance command, a Reset command, and a NOP empty operation command. That is, the main engine sends at least one control command among the above three control commands to the addressing engine of the operand at each step. That is, each time the main engine sends a control command to each addressing engine, it sends a combined control command composed of at least one of the above three control commands to control each addressing engine according to its respective tensor Addressing modes address different dimensions of raw tensor data.

The Advance forward command means to advance (advance or increase) a step in this dimension.

The Reset command indicates that when this dimension comes to an end, it needs to start addressing in this dimension from the beginning.

The NOP command indicates that no action has taken place on this dimension.

In order to better understand the logic of how the main engine controls the addressing engine to perform addressing according to the above-mentioned Advance command, Reset command, and NOP command, the schematic diagram shown in Figure 7 is explained below:

Step1: W cmd: NOP; H cmd: NOP, where cmd represents a command, and the state at this time is shown in (1) in Figure 7;

Step2: W cmd: Advance; H cmd: NOP, the state at this time is shown in (2) in Figure 7;

Step3: W cmd: Advance; H cmd: NOP, the state at this time is shown in (3) in Figure 7;

Step4: W cmd: Reset; H cmd: Advance, the state at this time is shown in (4) in Figure 7;

Step5: W cmd: Advance; H cmd: NOP, the state at this time is shown in (5) in Figure 7;

Step6: W cmd: Advance; H cmd: NOP, the state at this time is shown in (6) in Figure 7;

Step7: W cmd: Reset; H cmd: Advance, the state at this time is shown in (7) in Figure 7;

Step8: W cmd: Advance; H cmd: NOP, the state at this time is shown in (8) in Figure 7;

Step9: W cmd: Advance; H cmd: NOP, the state at this time is shown in (9) in Figure 7.

If the first dimension of the tensor data (it can be any dimension of the tensor data) needs to be broadcasted, when the addressing engine corresponding to the tensor data is addressing on the first dimension of the tensor data, upon receiving Advance When forwarding the command, keep the addressing pointer unchanged, so that when the addressing engine reads the source operand from the vector register, it will always read the source operand pointed by the current addressing pointer, so as to achieve the purpose of broadcasting. Correspondingly, if the first dimension of the tensor data (it can be any dimension of the tensor data) needs to be broadcasted, when the main engine sends a control command to control the addressing engine to address the first dimension of the tensor data, When the addressing engine comes to an end on the first dimension of the tensor data, the main engine will not send the Reset reset command, but can always send the Advance command to read the source operand pointed by the current addressing pointer , until the broadcast is completed, and then send other control commands.

For a better understanding, the following description will be made in conjunction with the schematic diagram shown in FIG. 4 . Assuming that the vector operation unit needs to complete the operations shown in Figure 4, the addressing engine 1 is responsible for obtaining the operand in the array "np.arange(3)", and the addressing engine 2 is responsible for obtaining the operand 5. At the initial moment, the main engine Send the Advance command to addressing engine 1 and addressing engine 2, addressing engine 1 obtains operand 0 in the array of "np.arange(3)", addressing engine 2 obtains operand 5; at the next moment, the master The engine sends an Advance command to addressing engine 1 and addressing engine 2, and addressing engine 1 gets the operand 1 in the array of "np.arange(3)" one step ahead, and addressing engine 2 receives the Advance command , keep the addressing pointer unchanged, and still output operand 5 at this time; at the next moment, the main engine sends the Advance command to addressing engine 1 and addressing engine 2, and addressing engine 1 further obtains "np.arange(3 )” in the array of 2 operands, when the addressing engine 2 receives the Advance command, it keeps the addressing pointer unchanged, and still outputs operand 5 at this time.

In an optional implementation, the tensor addressing mode can also include nested double tensor addressing modes, and the double tensor addressing mode includes: outer iterator addressing mode (outer iterator) and inner iterator addressing mode ( inneriterator), where the inner iterative addressing mode addresses based on the tensor data obtained by the outer iterative addressing mode.

As an implementation, the outer iterator addressing mode (outer iterator) can be the aforementioned new slice addressing mode using the parameters start: stop: step: size: partial. In this case, the local area can be quickly selected The tensor data (such as the shape defined based on these parameters) provides the addressable data basis for the inner iteration addressing mode. In some application scenarios, the local area selected by the outer iteration addressing mode is the complete shape obtained in the case of partial=true, so that there is no need to use additional instructions to indicate mode switching, or use additional instructions To handle the data shape change (from complete to incomplete) that may be faced in the inner iterative addressing mode.

The nested double tensor addressing mode is used for addressing. Compared with other addressing modes, the addressing efficiency is higher, so that only one instruction is needed to realize the addressing of two single-layer instructions, and, The inner iterative addressing mode is based on the tensor data obtained by the outer iterative addressing mode. Compared with the addressing of the single-layer tensor addressing mode, the first layer (outer iterative addressing mode) data read and write.

Wherein, the expressions of the outer iteration addressing mode and the inner iteration addressing mode may be: [start:stop:step:size], or [start:stop:step:size:partial]. Parameter values in expressions for outer and inner iteration addressing modes can be different.

For a better understanding, the expression of the external iterative addressing mode is [start:stop:step:size]=0:8:2:5 as an example, and its schematic diagram is shown in Figure 9, which is traversed in Figure 9 Part can be regarded as a feature map of size 8*8. The value of the inner iterative addressing mode expression is 0:5:2:1 as an example, and its addressing schematic diagram is shown in Figure 10. Addressing according to the addressing mode of 0:8:2:5, you can get 4 parts of parameters, that is, the parameters contained in the shapes of (1), (2), (3), and (4) in Figure 9. The inner iterative addressing mode is addressed on the basis of the tensor data obtained by the outer iterative addressing mode, that is, the addressing mode of 0:5:2:1 is used to address (1), (2), ( 3), (4) to address multiple tensor elements obtained.

It can be seen from the examples in Figure 9 and Figure 10 that the outer iterative addressing mode is used to select at least one candidate area from tensor data, and the candidate area contains multiple tensor elements, such as selecting (1), (2), (3), (4) and other areas; the inner iteration addressing mode is used to obtain the source operation required for the vector operation instruction from at least one candidate area. The inner iteration addressing mode is based on (1), (2), (3), and (4) in Figure 9 to address to obtain the source operand required by the vector operation instruction.

It should be noted that the inner iterative addressing mode does not necessarily wait until the outer iterative addressing mode is all addressed before addressing. For example, when the outer iterative addressing mode is addressed to (1) in Figure 9 When it contains 5*5 elements, the internal iterative addressing mode can address on this basis. After the internal iterative addressing mode completes the addressing of this part of the parameters contained in (1) in Figure 9, and Addressing can continue in the outer iterative addressing mode. When the outer iterative addressing mode addresses the 5*5 elements contained in (2) in Figure 9, the inner iterative addressing mode will address on this basis. After the inner iterative addressing mode completes the addressing of this part of the data included in (2) in Figure 9, it continues the outer iterative addressing mode, and so on, until all the addressing is completed. In this way, the data addressed by the outer iterative addressing mode is only an intermediate result, and does not need to be read and written. Compared with the addressing of the single-layer tensor addressing mode, the first layer (outer iterative addressing) is reduced. mode) data read and write.

It can be understood that, in addition to the addressing mode (new slice addressing mode) shown in this application, the outer iterative addressing mode and the inner iterative addressing mode can also be extended, such as using the existing Some addressing modes (slice addressing mode, index addressing mode) are used for addressing, so there can be multiple combinations, some exemplary combinations are shown in Table 1 below.

Table 1

Outer iteration addressing mode New slice addressing mode New slice addressing mode slice addressing mode slice addressing mode indexed addressing mode slice addressing mode indexed addressing mode indexed addressing mode New slice addressing mode Intra-iteration addressing mode New slice addressing mode slice addressing mode New slice addressing mode slice addressing mode indexed addressing mode indexed addressing mode slice addressing mode New slice addressing mode indexed addressing mode

It should be noted that when the outer iterative addressing mode adopts the new slice addressing mode for addressing, each time the pointer moves by one step, (1), (2), (3), and (4) in Figure 9 can be obtained The 5*5 element parameters contained in the shape shown. When the outer iterative addressing mode uses the existing slice addressing mode or index addressing mode for addressing, since only one element point can be obtained for each sliding step, to obtain (1) and (2) in Figure 9 , (3), and (4), these element parameters contained in the shape shown in (4) need to be addressed multiple times to ensure that no matter what addressing mode is used in the outer iteration addressing mode, the new slice addressing mode can be obtained. The data obtained by addressing in the addressing mode provides a basis for the inner iterative addressing mode. That is, no matter what addressing mode the outer iterative addressing mode uses, the data obtained is consistent, for example, the shape shown in (1), (2), (3), and (4) in Figure 9 is obtained The multiple tensor element parameters contained in it are only the parameters contained in a shape can be obtained by sliding a step when using the new slice addressing mode. If other addressing modes are used, multiple addressing (sliding) is required. Multiple steps) to get the parameters contained in the shape obtained by sliding a step in the new slice addressing mode.

Based on the same inventive concept, an embodiment of the present application further provides an electronic device, as shown in FIG. 11 , the electronic device includes: a memory and the above-mentioned AI chip. The AI chip is connected to the memory, and the AI chip is used to obtain the source operand required by the vector operation instruction from the original tensor data stored in the memory according to the tensor addressing mode, and to perform the obtained source operand according to the vector operation instruction Perform calculations to obtain calculation results.

Among them, the memory is used to store the original tensor data required for operation, which can be various common memories, such as Random Access Memory (Random Access Memory, RAM), Read Only Memory (Read Only Memory, ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electric Erasable Programmable Read-Only Memory (EEPROM) . Wherein, the random access memory may be a static random access memory (Static Random Access Memory, SRAM) or a dynamic random access memory (Dynamic Random Access Memory, DRAM). In addition, the memory may be a single data rate (Single Data Rate, SDR) memory, or a double data rate (Double Data Rate, DDR) memory.

In order to better understand the interaction process between the AI chip and the memory, the following vector operation instruction is a vector operation instruction with three operands, such as D=A*B+C, as an example. First, the original tensor data where the operand A is located is (such as H*W=4*4) into memory, write the original tensor data of operand B (such as H*W=4*4) into memory, and write the original tensor data of operand C (such as H*W=4*4) Write to the memory, the engine unit obtains the operand A of the vector operation instruction from the original tensor data where the operand A is located according to the tensor addressing mode, and writes it into the vector register 1, the engine unit The unit obtains the operand B of the vector operation instruction from the original tensor data where the operand B is located according to the tensor addressing mode, and writes it into the vector register 2, and the engine unit obtains the operand B from the original tensor data where the operand C is located according to the tensor addressing mode Raw tensor data takes the operand C of the vector operation instruction and writes it into vector register 3. When a tensor operation is required, the vector operation unit directly obtains the source operand A corresponding to the vector operation instruction from the vector register 1, obtains the source operand B corresponding to the vector operation instruction from the vector register 2, and obtains the source operand B corresponding to the vector operation instruction from the vector register 3. Obtain the source operand C corresponding to the vector operation instruction for operation, and obtain the operation result.

Among them, assuming the tensor addressing mode [start:stop:step:size] =0:4:1:3, for each 4*4 original tensor data, the engine unit will generate a 3*3*4 address , and then read the data at these addresses, get 3*3*4 data (that is, the operand), and send it to the vector register for storage. Wherein, the principle of addressing by the engine unit according to the addressing mode of 0:4:1:3 is shown in FIG. 12 .

It can be understood that, in the case that the storage space of the memory is sufficient, in the above process, all the original tensor data where the above three operands are located can be written into the memory, and then sequentially search according to the tensor addressing mode address, write the addressed data into the vector register; or first write the original tensor data of one of the operands into the memory, and then write the original tensor data of the operand according to the tensor addressing mode The data is addressed, and the addressed operand is written into the vector register. After that, the original tensor data of one of the operands is written into the memory, and then the operand is based on the tensor addressing mode. The original tensor data is addressed, and the addressed operand is written into the vector register. In this application, the order in which the original tensor data of each operand is written into the memory and the order in which the vector register is written from the memory are not limited.

In addition, in the above process, the original tensor data of different operands can be addressed by the same addressing engine, or the original tensor data of different operands can be addressed by different addressing engines site.

Wherein, the above-mentioned electronic devices may be, but not limited to, electronic devices such as mobile phones, tablets, computers, servers, vehicle-mounted devices, wearable devices, and edge boxes.

Based on the same inventive concept, the embodiment of the present application also provides a tensor processing method, as shown in FIG. 13 . The principle of the tensor processing method will be described below with reference to FIG. 13 . This method can be applied to the aforementioned AI chip, and can be applied to the aforementioned electronic equipment.

S1: Obtain the source operands required by the vector operation instruction from the tensor data according to the tensor addressing mode, where the number of source operands acquired by one addressing in the tensor addressing mode is greater than or equal to 1.

The tensor addressing mode in the embodiment of the present application improves and expands the existing slice addressing mode to include more parameters. For example, the parameters of the tensor addressing mode include: The start address (such as represented by start), the end address representing the addressing end point (such as represented by stop), the step size representing the offset range of the addressing pointer (such as represented by step), and the size of the data shape represented by addressing ( Such as expressed in size). Then the expression for the tensor addressing mode is [start:stop:step:size]. Among them, size is used to describe the size of the data shape (shape) obtained by addressing, that is, at each step, all element points contained in a shape will be extracted, not just one point. Therefore, the address of the tensor element corresponding to at least one source operand can be determined in one addressing, which can greatly improve the efficiency of calculating the address of the tensor element corresponding to the source operand, so that only fewer instructions are needed to find The required source operands greatly reduce redundant instructions, increase effective instruction density, improve performance, and simplify programming.

In one embodiment, the engine unit in the above-mentioned AI chip can obtain the source operands required by the vector operation instruction from the original tensor data stored in the external memory according to the tensor addressing mode.

S2: Store the obtained source operand into the vector register.

After the engine unit obtains the source operand according to the tensor addressing mode, it can store the obtained source operand in the vector register, where different source operands can be stored in different vector registers for subsequent reading When the operands are taken for operation, they can be read in parallel to improve efficiency.

S3: Acquiring source operands required by the vector operation instruction from the vector register to perform an operation to obtain an operation result.

After the source operand required by the vector operation instruction is obtained, the obtained source operand can be operated according to the vector operation instruction to obtain an operation result. In an optional implementation manner, the vector operation unit in the above-mentioned AI chip can obtain the source operands required by the vector operation instruction from the vector register to perform the operation, and obtain the operation result.

It can be understood that, according to the content of the tensor addressing mode disclosed in the embodiment of the present application, efficient addressing can be performed in each link of tensor processing. For example, for the calculation process between tensors, in addition to calculating the final In addition to the effective calculation steps of this step of the result, whether it is to read the tensor to be calculated or to write the calculated tensor result, it will involve the calculation of the address of the tensor element. This is because when reading the tensor to be calculated When tensor needs to read data according to the address of the tensor element, and after reading the tensor to be calculated from the corresponding address and performing operations between tensors, if it comes to the link of outputting results, in some scenarios It may be necessary to write the tensor operation result to the address used to store the tensor result, and the process of writing the operation result may also require the calculation of the address of the tensor element. Based on the principle of the tensor addressing mode disclosed in this application, the tensor can be quickly read and written under the condition of efficient addressing.

The tensor processing method provided by the embodiment of the present application has the same realization principle and technical effect as the aforementioned AI chip embodiment. For a brief description, the parts not mentioned in the method embodiment can refer to the aforementioned AI chip embodiment. Corresponding content.

It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. For the same and similar parts in each embodiment, refer to each other, that is, Can.

The above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto. Anyone familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the application. Should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (9)

1. An AI chip, comprising:

the engine unit is used for acquiring source operands required by a vector operation instruction from original tensor data according to a tensor addressing mode, wherein the number of the source operands acquired by one-time addressing of the tensor addressing mode is greater than or equal to 1, and parameters of the tensor addressing mode comprise: the method comprises the steps of representing a starting address of an addressing starting point, representing an end address of an addressing end point, representing a step length of the offset amplitude of an addressing pointer and representing the size of a data shape obtained by addressing, wherein tensor elements extracted by each step length are element points contained in one data shape;

the vector register is connected with the engine unit and used for storing the source operand acquired by the engine unit;

and the vector operation unit is connected with the vector register and is used for acquiring a source operand of the vector operation instruction from the vector register to perform operation so as to obtain an operation result.

2. The AI chip of claim 1, wherein the parameters of the tensor addressing mode further include characteristic parameters characterizing a preservation of the shape of the data resulting from addressing as an incomplete shape.

3. The AI chip of claim 1, wherein the tensor addressing mode comprises a nested dual tensor addressing mode, the dual tensor addressing mode comprising: the device comprises an outer iteration addressing mode and an inner iteration addressing mode, wherein the inner iteration addressing mode carries out addressing on tensor data obtained by addressing in the outer iteration addressing mode.

4. The AI chip of any of claims 1-3, wherein the engine unit is further coupled to an external memory for storing the raw tensor data; the engine unit includes:

and each addressing engine is used for acquiring a source operand required by a vector operation instruction from corresponding original tensor data according to a respective tensor addressing mode.

5. The AI chip of claim 4, wherein the engine unit further includes:

and the main engine is used for sending a control command to each addressing engine, controlling each addressing engine to address according to the respective tensor addressing mode, and acquiring a source operand required by the vector operation instruction from the corresponding original tensor data.

6. An electronic device, comprising:

the memory is used for storing tensor data required by operation; and

the AI chip of any of claims 1-5, the AI chip coupled to the memory, the AI chip to obtain source operands required by a vector operation instruction from the original tensor data stored by the memory according to a tensor addressing mode, and to perform an operation on the obtained source operands according to the vector operation instruction to obtain an operation result.

7. A tensor processing method, comprising:

obtaining source operands required by a vector operation instruction from original tensor data according to a tensor addressing mode, wherein the number of the source operands obtained by one-time addressing of the tensor addressing mode is more than or equal to 1;

storing the obtained source operands to a vector register, wherein the parameters of the tensor addressing mode include: the method comprises the steps of representing a starting address of an addressing starting point, representing an end address of an addressing end point, representing a step length of the offset amplitude of an addressing pointer, and representing the size of a data shape obtained by addressing, wherein tensor elements extracted each time the step length is advanced are element points contained in one data shape;

and acquiring a source operand of the vector operation instruction from the vector register to operate to obtain an operation result.

8. The method of claim 7, wherein obtaining source operands required by a vector operation instruction from raw tensor data according to a tensor addressing mode comprises:

and acquiring source operands required by the vector operation instruction from the original tensor data according to the tensor addressing mode by utilizing the addressing engine under the control command sent by the main engine.

9. The method of claim 8, wherein the control command comprises: the main engine sends a combined control command formed by combining at least one control command in the three control commands every time the main engine sends the control command to each addressing engine so as to control each addressing engine to address in different dimensions of the original tensor data according to respective tensor addressing modes.

Images