CN116227558A - Neural network dynamic exit lightweight method and system for multiple continuous reasoning - Google Patents

Abstract

The invention provides a neural network dynamic exit light-weight method and a system aiming at multiple continuous reasoning, comprising the following steps: step 1: constructing an inference model based on a neural network, predicting the position of network exit for each inference within a preset small time range, and correspondingly predicting a calculation configuration, wherein the calculation configuration comprises frequency and voltage; for a plurality of inferences in a preset large time range, calibrating the frequency and the voltage of the processor through residual inference workload and time constraint; step 2: the configuration is calculated according to the prediction and calibration to execute the neural network to realize dynamic voltage and frequency adjustment. Compared with a classical deep learning network, the method can realize energy saving of 63.8%, ensure that multiple times of neural network reasoning is completed within a specified time, terminate the reasoning in advance and obtain an accurate result by early exit, and reduce calculation and energy cost.

Description

Neural network dynamic exit lightweight method and system for multiple consecutive inferences

Technical Field

The present invention relates to the field of neural network technology, and in particular to a neural network dynamic exit lightweight method and system for multiple consecutive reasonings.

Background Art

Deep learning methods, such as convolutional neural networks, have achieved great success in multi-purpose applications. However, one of the challenges in deploying deep learning models on resource-constrained systems is their huge energy cost. As a dynamic inference method, early exit adds a dropout layer to the network, which can terminate the inference early and obtain accurate results to save energy. The current passive decision-making for energy regulation of early exit cannot adapt to the ongoing inference state, different inference workloads and time constraints, let alone guide the reasonable configuration of the computing platform during the inference process to save potential energy.

Patent document US20210056357A1 discloses a system and method for implementing a flexible, input-adaptive deep learning neural network; patent document US20210012178A1 discloses a system, method, and device for early exit convolution; patent document EP3997621A1 discloses a system, method, and device for early exit convolution. Although the above documents propose a method for early exit, they cannot predict the early exit point.

Patent document CN114997370A discloses a low-power neural network system based on predictive exit and its implementation method. Although the patent document proposes a prediction of early exit points, it can only predict a single neural network inference and reduce the amount of calculation and system energy consumption. The prediction accuracy is low, and it is impossible to predict with high accuracy from the first few layers of the network or even the first time, and it is impossible to predict the early exit point of a large network model with high accuracy.

Summary of the invention

In view of the defects in the prior art, the purpose of the present invention is to provide a method and system for dynamically exiting lightweight neural networks for multiple consecutive reasonings.

The method for dynamically exiting a neural network from lightweighting for multiple consecutive reasonings provided by the present invention includes:

Step 1: Build a neural network-based inference model, predict the location where the network exits for each inference within a preset small time range, and predict the computing configuration accordingly, the computing configuration including frequency and voltage;

Step 2: For multiple inferences within a preset large time range, calibrate the processor frequency and voltage based on the remaining inference workload and time constraints;

Step 3: Calculate the configuration based on the prediction and calibration to execute the neural network to achieve dynamic voltage and frequency adjustment.

Preferably, a coordination period of reasoning, i.e., a deadline of the reasoning task, is set during the reasoning process, and the expression is:

Among them, λ represents the tightness of the task deadline, 0<λ≤1, the smaller λ is, the tighter the deadline is; _ti represents the actual completion time of each reasoning; _Tc is the coordination cycle; M is the number of reasoning tasks; _Ti represents the coordination deadline of the i-th reasoning task.

Preferably, when performing an inference task, the power consumed in the calculation process is active power P _active , which is the sum of dynamic power P _D , static power P _S , and constant power P _C ;

The power consumed by the computing platform when it is idle is the idle power P _idle , which is the sum of the static power P _S and the constant power P _C ;

_PD ＝ ^CV2f

_{PS ＝ VNtrIS}

Where C is the capacitance of the switching logic gate; V is the power supply voltage of the processor; f is the processor clock frequency; N _tr is the number of logic gates; I _S is the normalized static current of each logic gate;

The energy consumption of each inference in _Ti is:

The network layer before early exit runs at (V _H , f _H ). After the network exits at point t _i , the computing platform reduces the voltage and frequency to the lowest level (V _L , f _L ) through the dynamic voltage and frequency scaling technology DVFS. Therefore, the energy consumption of inference task j _i is expressed as:

E _i represents the energy consumption of reasoning task j _i ; V _H represents the high power supply voltage of the processor; f _H represents the high clock frequency of the processor; V _L represents the low power supply voltage of the processor.

Preferably, in an inference task j _i , the network exit position is first predicted, and the remaining number of layers is established to complete the inference; then (V, f) is established for prediction based on the remaining number of layers and the worst-case execution time _Ti until the end of the current inference; based on the remaining amount of inference on T _c , the cross-inference calibrator is enabled through the feedback control strategy, and (V, f) is calibrated according to the multi-inference progress, the total inference workload and the time constraint; based on the prediction and calibration, the processor is updated to the appropriate computing configuration at runtime through DVFS.

从而估计出现有层的最终输出，分类器的估计结果为：Preferably, in the inference task j _i , the existing layer includes a BoF pooling layer and a FC layer. Let _yi be the intermediate result of the i-th layer of the existing layer, N _c be the number of object classes in the data set, and the BoF pooling layer is used as a feature aggregation extracted from _yi . In BoF pooling, a set of feature vectors called codebooks is used to describe _yi . The weight of each codebook is generated by measuring the similarity between the codebook and _yi . The size of the codebook weight is greater than N _c , so the FC layer is further used as a classifier to adjust the result of BoF pooling to

Thus, the final output of the existing layer is estimated, and the estimated result of the classifier is:

是分类器的估计结果；Where _fw (x,i) is the intermediate result of the i-th layer; x is the initial network input; _Wi is the parameter of the i-th exit layer, and the exit layer is the BoF pooling layer and the FC layer;

is the estimated result of the classifier;

Train the parameters in the existing layers offline, and choose the same cross entropy loss function when training _Wi ;

为交叉熵；Where p(i) and q(i) are the actual and predicted class distributions of each object, respectively;

is the cross entropy;

为正确结果；首先，在没有任何现有层的情况下，用以下公式训练模型：When training the existing layer, the training set data {x1, x2, …, xN} is input into the model, N is the number of training data, and the existing layer parameters are optimized by batch gradient descent. Let L _total be the number of layers in the network, and the target vector set is

To get the correct result; first, without any existing layers, train the model with the following formula:

表示参数梯度；x_j表示训练数据向量元素；r_j表示目标数据向量元素；Where W' is the updated overall network parameter; η is the learning rate. If the accuracy cannot meet the requirements, adjust η; j represents the vector position;

represents parameter gradient; x _j represents training data vector element; r _j represents target data vector element;

The original model W is fixed, and based on the existing layers, the model is trained again to optimize the parameters of the BoF pooling layer and the FC layer. The expression is:

After training the exit layer, the average feature weight μ _i is calculated as the parameter for the exit decision, and the expression is:

Among them, k represents the kth overall classification target;

During the inference process of the initial network input _xj , at each early exit layer, the weight ratio αW is the maximum feature weight divided by _μi multiplied by the user-specified hyperparameter β. The larger the maximum feature weight, the more confident the classification is. If αW is greater than 1, the inference is terminated, and the result of this early exit layer is applied as the final result, expressed as:

Preferably, the predictor starts predicting from the _L0th layer of the neural network, and the predictor input is the execution result vector _y0 of the _L0th layer of the neural network and its corresponding exit layer execution result vector

两端添加0的方式，将向量

由

维扩展到

维，并记为

其中N_c为向量

长度同时也是推理结果分类的数量，K为特征池化组中特征向量的个数，具体计算过程如下述公式所示：The above results are expanded in dimension by using the vector

Add 0 at both ends to convert the vector

Depend on

Dimensions expanded to

dimension, and recorded as

Where N _c is a vector

The length is also the number of inference result categories, K is the number of feature vectors in the feature pooling group, and the specific calculation process is shown in the following formula:

l represents the lth element;

将进行一维卷积操作并获得新的特征权重向量

其中卷积权重为长度为K的一维向量h，具体计算过程如下述公式所示：The output vector is zero-padded

A one-dimensional convolution operation will be performed and a new feature weight vector will be obtained

The convolution weight is a one-dimensional vector h of length K. The specific calculation process is shown in the following formula:

替换为

并重复计算式，得到L₀+2位置的出料层的预测结果，记为

By recursively converting the intermediate estimation results

Replace with

Repeat the calculation formula to obtain the predicted result of the discharge layer at position L ₀ +2, which is recorded as

Place the prediction result of the exit layer in any layer after L ₀ to obtain the PREDICT function, which is described as follows:

其中，

表示L₀后的预测出点的最小值；

in,

Indicates the minimum value of the predicted output point after L ₀ ;

为L₀+ζ层的预测出口置信度，因此，L₀+ζ是预测函数预测的退出点，如果找不到ζ，则引入一个超参数τ，即ζ∈[1，τ]，τ≤L_totaal-L₀，τ的上界表示禁止最后一层以外的预测结果；如果[1，τ]中没有整数满足条件，则设ζ＝τ；In the formula,

is the prediction exit confidence of the L ₀ +ζ layer. Therefore, L ₀ +ζ is the exit point predicted by the prediction function. If ζ cannot be found, a hyperparameter τ is introduced, that is, ζ∈[1,τ], τ≤L _totaal -L ₀ . The upper bound of τ indicates that the prediction results outside the last layer are prohibited. If no integer in [1,τ] satisfies the condition, ζ＝τ is set.

及时完成模型的L_total而不退出，表达式为：Convert the existing prediction results ζ of PREDICT to the mid-level frequency f _M,i . For the inference task j _i , a relatively high-level conservative approach is applied in the interval between the start of inference and the L ₀ prediction.

The L _total of the model is completed in time without exiting, and the expression is:

由Δf_i校准；Where _fH is the default maximum frequency of the computing platform. The computing platform can complete reasoning with L _total layers and will not exit early due to time _Ti ; frequency

Calibrated by Δf _i ;

When predicting _L0 , adjust f to the intermediate frequency _fM,i according to the predicted exit point, and run the network until exiting at _Ti . Given a network with L _total layers and early prediction ζ, the intermediate frequency is calculated as:

where f _M,i is the lowest frequency predicted to complete inference at time T _i .

Preferably, for the reasoning task j _i+1 , the cross-reasoning DVFS strategy provides a calibration suggestion Δf _i+1 after completing all previous tasks including j _i . The calibration is implemented by a discrete incremental proportional integral PI regulator, and the expression is:

Δf _i+1 =Δf _i +(K _P +K _I (t _i -t _i-1 ))e(t _i )-K _P e(t _i-1 )

Where K _P and K _I are the proportional coefficient and integral coefficient respectively; index i represents the i-th reasoning task in the current coordination cycle; _ti represents the relative completion time of reasoning j _i from the beginning of the coordination cycle; _{ti -t i-1} is the time interval between the completion of reasoning tasks j _i and j _i-1 ;

其中第一项是推理工作负载平衡的参考速度，第二项是到目前为止的处理速度。The input deviation e(t _i ) of the PI regulator is an evaluation of the inference progress of T _c based on the total inference workload and the inference execution progress since the beginning of the coordination period, and is expressed as:

The first term is the reference speed for inference workload balancing, and the second term is the processing speed so far.

保守地完成不提前退出的推理，在推理层L₀进行早期退出预测时，根据内推断预测器导出的f_M，i，以及根据交叉推断校准导出的Δf_i，DVFS调控器建立适当的频率配置

表达式为：Preferably, the DVFS governor configures the frequency and the corresponding voltage, executes the network according to the prediction and calibration of the small time scale and the large time scale respectively, and at the beginning of each inference, Δf _i is derived based on the cross-inference calibration, and the DVFS governor sets the calculation configuration to

Inference without early exit is done conservatively. When early exit prediction is performed at the inference layer _L0 , the DVFS governor establishes an appropriate frequency configuration based on fM _,i derived from the internal inference predictor and Δf _i derived from the cross-inference calibration.

The expression is:

Therefore, the energy consumption is:

为预测层L₀及其之前的网络层j_i完成的时间；

为给定

所对应的电压；由于f的选择是基于推理中预测的剩余工作量，因此推理内预测可以使j_i由T_i完成；交叉推断校准目标通过T_c完成M个顺序推断。in,

is the time it takes to complete the prediction layer _L0 and its previous network layer j _i ;

For a given

The corresponding voltage; since the selection of f is based on the remaining workload predicted in inference, the intra-inference prediction can enable j _i to be completed by _Ti ; the cross-inference calibration target completes M sequential inferences through T _c .

The neural network dynamic exit lightweight system for multiple consecutive reasonings provided by the present invention includes:

Module M1: Build a neural network-based inference model, predict the location of network exit for each inference within a preset small time range, and predict the computing configuration accordingly, the computing configuration includes frequency and voltage; for multiple inferences within a preset large time range, perform processor frequency and voltage calibration based on the remaining inference workload and time constraints;

Module M2: Configured to execute a neural network based on prediction and calibration calculations to achieve dynamic voltage and frequency regulation.

Preferably, a coordination period of reasoning, i.e., a deadline of the reasoning task, is set during the reasoning process, and the expression is:

Where λ represents the tightness of the task deadline, 0<λ≤1, the smaller λ is, the tighter the deadline is; _ti represents the actual completion time of each reasoning; _Tc is the coordination cycle; M is the number of reasoning tasks; _Ti represents the coordination deadline of the i-th reasoning task;

When performing inference tasks, the power consumed in the calculation process is active power P _active , which is the sum of dynamic power P _D , static power P _S , and constant power P _C ;

The power consumed by the computing platform when it is idle is the idle power P _idle , which is the sum of the static power P _S and the constant power P _C ;

_PD ＝ ^CV2f

_{PS ＝ VNtrIS}

Where C is the capacitance of the switching logic gate; V is the power supply voltage of the processor; f is the processor clock frequency; N _tr is the number of logic gates; I _S is the normalized static current of each logic gate;

The energy consumption of each inference in _Ti is:

The network layer before early exit runs at (V _H , f _H ). After the network exits at point t _i , the computing platform reduces the voltage and frequency to the lowest level (V _L , f _L ) through the dynamic voltage and frequency scaling technology DVFS. Therefore, the energy consumption of the inference task j _i is expressed as:

E _i represents the energy consumption of reasoning task j _i ; V _H represents the high supply voltage of the processor; f _H represents the high clock frequency of the processor; V _L represents the low supply voltage of the processor;

In an inference task j _i , first predict the network exit position and establish the remaining number of layers to complete the inference; then, based on the remaining number of layers and the worst-case execution time _Ti until the end of the current inference, establish (V,,f) for prediction; based on the remaining amount of inference on T _c , enable the cross-inference calibrator through the feedback control strategy, and calibrate (V,f) according to the multi-inference progress, total inference workload and time constraints; based on the prediction and calibration, update the processor to the appropriate computing configuration at runtime through DVFS.

Compared with the prior art, the present invention has the following beneficial effects:

The present invention predicts where the network will exit in each inference, and adjusts the computing configuration (i.e., frequency and voltage) accordingly within a small time range. At the same time, based on multiple inferences within a large time range, combined with the remaining inference workload and time constraints, it provides computing configuration (i.e., frequency and voltage) recommendations, and executes the neural network based on the predicted and calibrated configuration, thereby achieving dynamic voltage and frequency regulation. Compared with classic deep learning networks, the present invention can achieve energy savings of up to 63.8%, while ensuring that multiple neural network inferences are completed within the specified time. Early exit can terminate the inference in advance and obtain accurate results, reducing computing and energy costs.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

FIG1 is a diagram showing the structure of a neural network based on early exit according to the present invention.

DETAILED DESCRIPTION

The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

Embodiment 1:

As shown in FIG1 , for a neural network application that completes multiple neural network inferences within a period of time, the present invention provides a neural network dynamic exit lightweight method for multiple continuous inferences, which reduces the computing power and power consumption of neural network calculations by combining the inference state with the dynamic voltage and frequency adjustment of the processor at runtime, and specifically includes the following steps:

Step 1: In each inference of a small time period, predict where the network will exit and accordingly predict the computing configuration in the small time range, the computing configuration including frequency and voltage; for multiple inferences in a large time range, provide processor frequency and voltage calibration by the remaining inference workload and deadline constraints;

Step 2: Calculate the configuration based on the prediction and calibration to execute the neural network to achieve dynamic voltage and frequency scaling.

A more detailed description follows.

The coordination cycle of reasoning, that is, the deadline of the reasoning task, is expressed as:

λ represents the tightness of the task deadline, 0<λ≤1, the smaller λ is, the tighter the deadline is; _ti represents the actual completion time of each reasoning; _Tc is the coordination cycle; M is the number of reasoning tasks; _Ti represents the coordination deadline of the i-th reasoning task.

When performing inference tasks, the power consumed in the calculation process is active power P _active , which is the sum of dynamic power P _D , static power P _S , and constant power P _C ;

The power consumed by the computing platform when it is idle is the idle power P _idle , which is the sum of the static power P _S and the constant power P _C ;

_PD ＝ ^CV2f

_{PS ＝ VNtrIS}

Where C is the capacitance of the switching logic gate; V is the power supply voltage of the processor; f is the processor clock frequency; N _tr is the number of logic gates; I _s is the normalized static current of each logic gate;

The energy consumption of each inference in _Ti is:

Due to the early deployment of the neural network, the network layer before the early exit runs at (V _H , f _H ). After the network exits at point _ti , the computing platform reduces the voltage and frequency to the lowest level (V _L , f _L ) through DVFS until _ti . Therefore, the energy consumption of the inference task j _i is expressed as:

E _i represents energy consumption; V _H represents the high supply voltage of the processor; f _H represents the high clock frequency of the processor; V _L represents the low supply voltage of the processor;

An appropriate (V, f) is established to adjust the remaining workload and time constraints, and (V, f) is adjusted at runtime based on the prediction of the network exit position and the remaining inference to be completed before the deadline T _c .

In an inference task j _i , the exit position of the inference network is first predicted and the number of remaining layers is established to complete the inference; then, based on the remaining layers and the worst-case execution time _Ti until the end of the current inference, an appropriate (V, f) prediction is established to reduce energy costs on a small time scale; considering the remaining amount of inference on T _c , the cross-inference calibrator enabled by the feedback control strategy calibrates (V, f) according to the multi-inference progress, total inference workload and time constraints to balance the workload, energy and time costs on a large time scale; based on the prediction and calibration, the DVFS governor will update the processor to the appropriate computing configuration (i.e., frequency and voltage) at runtime to save energy while meeting the deadline.

从而估计出现有层的最终输出。In the inference task j _i , the design of the existing layers is based on existing works, and all the existing layers are further modified to share the same topology. It contains BoF (Bag-of-Features) pooling layers and FC (full-connected) fully connected layers. Let _yi be the intermediate result of the i-th layer, and N _c be the number of object classes in the dataset. The BoF pooling layer acts as a feature aggregation extracted from _yi . In BoF pooling, a set of feature vectors called codebooks is used to describe _yi . The weight of each codebook is generated by measuring the similarity between the codebook and _yi . Since the size of the codebook weight is usually larger than N _c , the FC layer is further used as a classifier to adjust the result of BoF pooling to

Thereby estimating the final output of the existing layer.

The functions of the existing layers are recorded as:

是分类器的估计结果。Where _fw (x,i) is the intermediate result of the i-th layer, x is the initial network input; _Wi is the parameter of the i-th exit layer (BoF and FC);

is the estimated result of the classifier.

The parameters in the existing layers need to be trained offline. Since the existing layers work by estimating the final network output, the same cross entropy loss function is selected when training _Wi .

为交叉熵；Where p(i) and q(i) are the actual and predicted class distributions of each object, respectively;

is the cross entropy;

为正确结果；首先，在没有任何现有层的情况下，用以下公式训练模型：When training the existing layer, the training set data {x1, x2, …, xN} is input into the model, N is the number of training data, and the existing layer parameters are optimized by batch gradient descent. Let L _total be the number of layers in the network, and the target vector set is

To get the correct result; first, without any existing layers, train the model with the following formula:

表示参数梯度；x_j表示训练数据向量元素；r_j表示目标数据向量元素；Where W' is the updated overall network parameter, η is the learning rate, if the accuracy cannot meet the requirements, η can be adjusted; j represents the vector position;

represents parameter gradient; x _j represents training data vector element; r _j represents target data vector element;

After that, the original model W is fixed, and the model is trained again based on the existing layers to optimize the parameters of the BoF pooling layer and the FC layer. The expression is:

After training the exit layer, the average feature weight μ _i is calculated as the parameter for the exit decision, and the expression is:

k represents the kth overall classification target.

During inference of the initial network input _xj , at each early exit layer, the weight ratio αW is the maximum feature weight divided by _μi multiplied by the user-specified hyperparameter β. The larger the maximum feature weight, the more confident the classification is. Once αW is greater than 1, inference terminates and the result of this early exit layer is applied as the final result.

Assume that the predictor starts predicting from the _L0th layer of the neural network, and the predictor input is the execution result vector _y0 of the _L0th layer of the neural network and its corresponding exit layer execution result vector

两端添加0的方式，将向量

由

维扩展到

维，并记为

其中N_c为向量

长度同时也是推理结果分类的数量，K为特征池化组中特征向量的个数。具体计算过程如下述公式所示：The zero-filling module expands the dimension of the above results by

Add 0 at both ends to convert the vector

Depend on

Dimensions expanded to

dimension, and recorded as

Where N _c is a vector

The length is also the number of inference result categories, and K is the number of feature vectors in the feature pooling group. The specific calculation process is shown in the following formula:

l represents the lth element.

将进行一维卷积操作并获得新的特征权重向量

其中卷积权重为长度为K的一维向量h。具体计算过程如下述公式所示：The output vector after the zero padding module

A one-dimensional convolution operation will be performed and a new feature weight vector will be obtained

The convolution weight is a one-dimensional vector h of length K. The specific calculation process is shown in the following formula:

替换为

并重复计算式，得到L₀+2位置的出料层的预测结果，记为

按照上述步骤，可以计算L₀之后任何现有层的预测结果。By recursively converting the intermediate estimation results

Replace with

Repeat the calculation formula to obtain the predicted result of the discharge layer at position L ₀ +2, which is recorded as

Following the above steps, the prediction results of any existing layers after _L0 can be calculated.

Placing the prediction result of the exit layer in any layer after L ₀ , we will get the PREDICT function, which is described as follows:

表示L₀后的预测出点的最小值；in,

Indicates the minimum value of the predicted output point after L ₀ ;

为L₀+ζ层的预测出口置信度。因此，L₀+ζ是预测函数预测的退出点。如果找不到ζ，进一步引入一个超参数τ，即ζ∈[1，τ]，τ≤L_total-L₀。τ的上界表示禁止最后一层以外的预测结果。如果[1，τ]中没有整数满足条件，则设ζ＝τ。预测的时间复杂度为O(N_c×K×τ)，同时在预测中引入超参数。L₀、β和τ可由高级用户调优，可以平衡不同应用场景的预测精度和计算成本。In the formula,

is the prediction exit confidence of the L ₀ +ζ layer. Therefore, L ₀ +ζ is the exit point predicted by the prediction function. If ζ cannot be found, a hyperparameter τ is further introduced, that is, ζ∈[1,τ], τ≤L _total -L ₀ . The upper bound of τ indicates that prediction results other than the last layer are prohibited. If no integer in [1,τ] satisfies the condition, then ζ＝τ is set. The time complexity of the prediction is O(N _c ×K×τ), and hyperparameters are introduced in the prediction. L ₀ , β and τ can be tuned by advanced users to balance the prediction accuracy and computational cost of different application scenarios.

及时完成模型的L_total而不退出，表达式为：Frequency prediction: Convert the existing prediction results ζ of PREDICT to appropriate "mid-level" frequencies f _M,i . For inference task j _i , a relatively "high-level" prediction is conservatively applied in the interval between the start of inference and the L ₀ prediction.

The L _total of the model is completed in time without exiting, and the expression is:

由Δf_i校准。Where _fH is the default maximum frequency of the computing platform. The computing platform can complete reasoning with L _total layers and will not exit early due to time _Ti .

Calibrated by _Δfi .

When predicting L ₀ , we adjust f to the appropriate “middle layer” f _M,i according to the predicted exit point, and run the network until exiting at _Ti . Given a network with L _total layers and early exit prediction ζ, the computation frequency drops to:

where f _M,i is the lowest frequency prediction that can be inferred by time _Ti .

Since our goal is to execute reasoning tasks in a bursty manner, for reasoning task j _i+1 , the cross-reasoning DVFS strategy provides a calibration suggestion Δf _i+1 after completing all previous tasks including j _i , taking into account the workload and time constraints on a larger time scale T _c . The calibration is implemented by a discrete incremental proportional integral (PI) regulator, expressed as:

Δf _i+1 =Δf _i +(K _P +K _I (t _i -t _i-1 ))e(t _i )-K _P e(t _i-1 )

Where K _P and K _I are the proportional coefficient and integral coefficient respectively. Index i represents the i-th reasoning task in the current coordination cycle. _ti represents the relative completion time of reasoning j _i from the beginning of the coordination cycle. _ti - _ti-1 is the time interval between the completion of reasoning tasks j _i and j _i-1 .

其中第一项是推理工作负载平衡的参考速度，第二项是到目前为止的处理速度。The input deviation e(t _i ) of the PI regulator is an evaluation of the inference progress of T _c based on the total inference workload and the inference execution progress since the beginning of the coordination period, and is expressed as:

The first term is the reference speed for inference workload balancing, and the second term is the processing speed so far.

In practical applications, a smaller time period αT _c is used, 0＜α≤1. The interval between αT _c and T _c is a margin reserved by feedback control to eliminate the effect of execution time overshoot. The appropriate choice of α can strike a balance between saving energy and meeting deadlines. The output of the cross-inference corrector is the frequency calibration of the next inference j _i+1 of the DVFS governor configuration calculation platform Δf _i+1 .

保守地完成不提前退出的推理。在推理层L₀进行早期退出预测时，DVFS调控器建立适当的频率配置

根据内推断预测器导出的f_M，i，以及根据交叉推断校准导出的Δfi，表达式为：Finally, the DVFS governor configures the frequency and corresponding voltage to execute the network based on the prediction and calibration of small and large time scales, respectively, to further save energy and meet the network inference deadline. The DVFS governor operates at a smaller granularity (V,f) than inference. At the beginning of each inference, Δf _i is derived based on the cross-inference calibration of . The DVFS governor sets the computation configuration to

Inference without early exit is done conservatively. When early exit prediction is made at inference level _L0 , the DVFS governor establishes an appropriate frequency configuration

f _M,i , derived from the intra-inference predictor, and Δfi , derived from the cross-inference calibration, are expressed as:

The resulting energy consumption would therefore be:

为预测层L₀及其之前的网络层j_i完成的时间。

为给定

所对应的电压。由于f的选择是基于推理中预测的剩余工作量，因此推理内预测可以使j_i由T_i完成。交叉推断校准目标通过T_c完成M个顺序推断。由于V和f是计算性能的线性尺度(与推断成反比)，而对P_D和P_S的三次尺度和线性尺度，双时间尺度电源管理有效地降低了E。in,

is the time it takes to complete the prediction layer _L0 and its previous network layer j _i .

For a given

The voltage corresponding to . Since the choice of f is based on the remaining workload predicted in inference, intra-inference prediction allows j _i to be completed by _Ti . The cross-inference calibration target completes M sequential inferences through T _c . Since V and f are linear scales of computational performance (inversely proportional to inference), and cubic and linear scales for P _D and _PS , dual-time-scale power management effectively reduces E.

The processor voltage and frequency can be adjusted at the application level through system calls (sycall). According to our observation, the transient time of each (V, f) change is about 1ms to 3ms. The transient process of changing the processor (V, f) is incorporated into our actual system evaluation.

For each inference at runtime, the computational complexity of in-inference exit prediction is O(N _c ×K×τ), and the cross-inference calibration and DVFS governor are both O(1).

We use VGG-19 and ResNets-18 as backbone models on the commonly used CIFAR-10, CIFAR-100, SVHN and CINIC datasets to evaluate the inference accuracy and timing performance of the present invention. The evaluation is performed on NVIDIA Jetson TX2, GPU f _H = 1.30050GHz, f _L = 0.11475GHz. Serial and I/O operations are performed on the CPU, and a large number of parallel and computationally intensive segments are offloaded to the GPU, which is automatically managed by the PyTorch library. The energy cost is evaluated by a Tektronix MDO32 oscilloscope and a TCP2020 current probe with a sampling frequency of 50kHz. The prediction and calibration parameters of different benchmarks are listed in Table 1.

Table 1

During inference, the frequency is calculated based on the prediction of the early exit point. The accuracy of the prediction directly determines the time and energy consumption of the task. Therefore, we tested the prediction accuracy of datasets and models with different L _0. According to the prediction and inference accuracy results, the rest of the Vgg-19 and Resnet-18 are set to L ₀ = 6 and 7 respectively.

The inference accuracy is then evaluated, and the results are shown in Table 2. The present invention achieves the same accuracy as other early exit methods on both models, which is 1%-3% lower than the classic CNN. Because in the early exit prediction, even if the prediction is wrong, the network will continue to make the next prediction instead of forcing the inference exit, and no additional loss of inference accuracy will be introduced. The remaining power management settings have no effect on the inference accuracy, because the existing layer predictions of ζ are determined first to achieve high inference accuracy, and power management is built on this basis.

Table 2

We evaluate the timing performance and show that our proposed method achieves more focused and shorter execution time than other methods due to the effective dual-timescale DVFS management. α can strike a balance between saving energy and meeting deadlines. α = 0.75 results in slightly longer total execution time than α = 0.5, while it saves more energy.

当λ＝1时，所有的方法都能满足最后期限。Finally, we evaluate the deadline T _c satisfaction rate under different values of λ. As a stress test, we compare the timing performance under tighter deadlines, i.e.

When λ=1, all methods can meet the deadline.

To measure the power consumption during runtime, the Jetson TX2 board was directly connected to a Keysight E36231A power supply at 19V. A current probe and an oscilloscope monitored the current fed to the board. Table 3 summarizes the average energy consumption for each task. Energy measurements on a real platform show that the proposed method saves 63.8% energy compared to a classic deep learning network and 21.5% energy compared to early exit under the state-of-the-art exit strategy. In most cases, α = 0.75 consumes less energy than α = 0.5.

Table 3

The main source of overhead is early exit PREDICT. Therefore, we evaluate the time cost of 1750 early exit PREDICT for each benchmark. The maximum time cost for all benchmarks is below 300μs. The box of ResNets-18 is wider than that of VGG-19 because ResNets-18 is expected to exit with a larger ζ. The applicability and scalability of the proposed online power management are demonstrated by the time cost.

Embodiment 2:

The present invention also provides a lightweight system for dynamic exit of a neural network for multiple continuous reasonings. The lightweight system for dynamic exit of a neural network for multiple continuous reasonings can be implemented by executing the process steps of the lightweight method for dynamic exit of a neural network for multiple continuous reasonings. That is, those skilled in the art can understand the lightweight method for dynamic exit of a neural network for multiple continuous reasonings as a preferred implementation of the lightweight system for dynamic exit of a neural network for multiple continuous reasonings.

According to the present invention, a lightweight system for dynamic exit of a neural network for multiple continuous reasoning includes: module M1: constructing a reasoning model based on a neural network, predicting the location of network exit for each reasoning within a preset small time range, and correspondingly predicting the computing configuration, wherein the computing configuration includes frequency and voltage; for multiple reasonings within a preset large time range, calibrating the processor frequency and voltage through the remaining reasoning workload and time constraints; module M2: executing the neural network according to the predicted and calibrated computing configuration, thereby realizing dynamic voltage and frequency adjustment.

In the reasoning process, the reasoning coordination cycle, i.e., the reasoning task deadline, is set, and the expression is:

Where λ represents the tightness of the task deadline, 0<λ≤1, the smaller λ is, the tighter the deadline is; _ti represents the actual completion time of each reasoning; _Tc is the coordination cycle; M is the number of reasoning tasks; _Ti represents the coordination deadline of the i-th reasoning task;

When performing inference tasks, the power consumed in the calculation process is active power P _active , which is the sum of dynamic power P _D , static power P _S , and constant power P _C ;

The power consumed by the computing platform when it is idle is the idle power P _idle , which is the sum of the static power P _S and the constant power P _C ;

_PD ＝ ^CV2f

_{PS ＝ VNtrIS}

Where C is the capacitance of the switching logic gate; V is the power supply voltage of the processor; f is the processor clock frequency; N _tr is the number of logic gates; I _S is the normalized static current of each logic gate;

The energy consumption of each inference in _Ti is:

The network layer before early exit runs at (V _H , f _H ). After the network exits at point t _i , the computing platform reduces the voltage and frequency to the lowest level (V _L , f _L ) through the dynamic voltage and frequency scaling technology DVFS. Therefore, the energy consumption of inference task j _i is expressed as:

E _i represents the energy consumption of reasoning task j _i ; V _H represents the high supply voltage of the processor; f _H represents the high clock frequency of the processor; V _L represents the low supply voltage of the processor;

In an inference task j _i , first predict the network exit position and establish the remaining number of layers to complete the inference; then, based on the remaining number of layers and the worst-case execution time _Ti until the end of the current inference, establish (V,,f) for prediction; based on the remaining amount of inference on T _c , enable the cross-inference calibrator through the feedback control strategy, and calibrate (V,f) according to the multi-inference progress, total inference workload and time constraints; based on the prediction and calibration, update the processor to the appropriate computing configuration at runtime through DVFS.

Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other at will.

Claims (10)

1. A lightweight method for dynamic exit of a neural network for multiple consecutive reasonings, characterized by comprising: Step 1: Build a neural network-based inference model, predict the location where the network exits for each inference within a preset small time range, and predict the computing configuration accordingly, the computing configuration includes frequency and voltage; for multiple inferences within a preset large time range, calibrate the processor frequency and voltage based on the remaining inference workload and time constraints; Step 2: Calculate the configuration based on the prediction and calibration to execute the neural network to achieve dynamic voltage and frequency scaling. 2. According to the lightweight method of dynamic exit of neural network for multiple continuous reasoning according to claim 1, it is characterized in that the reasoning coordination cycle, i.e., the reasoning task deadline, is set during the reasoning process, and the expression is:

Among them, λ represents the tightness of the task deadline, 0＜λ≤1, the smaller λ is, the tighter the deadline is; _ti represents the actual completion time of each reasoning; _Tc is the coordination cycle; M is the number of reasoning tasks; _Ti represents the coordination deadline of the i-th reasoning task. 3. The method for dynamic exit of a neural network for multiple continuous reasoning according to claim 2 is characterized in that, when performing a reasoning task, the power consumed in the calculation process is active power P _active , which is the sum of dynamic power P _D , static power P _S , and constant power P _C ; The power consumed by the computing platform when it is idle is the idle power P _idle , which is the sum of the static power P _S and the constant power P _C ; _PD ＝ ^CV2f _{PS ＝ VNtrIS} Where C is the capacitance of the switching logic gate; V is the power supply voltage of the processor; f is the processor clock frequency; N _tr is the number of logic gates; I _S is the normalized static current of each logic gate; The energy consumption of each inference in _Ti is:

The network layer before early exit runs at (V _H , f _H ). After the network exits at point t _i , the computing platform reduces the voltage and frequency to the lowest level (V _L , f _L ) through the dynamic voltage and frequency scaling technology DVFS. Therefore, the energy consumption of inference task j _i is expressed as:

E _i represents the energy consumption of reasoning task j _i ; V _H represents the high power supply voltage of the processor; f _H represents the high clock frequency of the processor; V _L represents the low power supply voltage of the processor. 4. According to claim 3, the lightweight method for dynamic exit of a neural network for multiple continuous reasoning is characterized in that, in an reasoning task j _i , the network exit position is first predicted, and the remaining number of layers is established to complete the reasoning; then, based on the remaining number of layers and the worst-case execution time _Ti until the end of the current reasoning, (V, f) is established for prediction; based on the remaining amount of reasoning on T _c , a cross-reasoning calibrator is enabled through a feedback control strategy, and (V, f) is calibrated according to the multi-reasoning progress, the total reasoning workload and the time constraint; based on the prediction and calibration, the processor is updated to the appropriate computing configuration at runtime through DVFS. 5. The method for dynamic exit of a neural network for multiple continuous reasoning according to claim 3 is characterized in that, in the reasoning task j _i , the existing layer includes a BoF pooling layer and an FC layer, assuming that _yi is the intermediate result of the i-th layer of the existing layer, N _c is the number of object classes in the data set, and the BoF pooling layer is used as a feature aggregation extracted from _yi . In BoF pooling, a set of feature vectors called codebooks is used to describe _yi . The weight of each codebook is generated by measuring the similarity between the codebook and _yi . The size of the codebook weight is greater than N _c , so the FC layer is further used as a classifier to adjust the result of BoF pooling to

Thus, the final output of the existing layer is estimated, and the estimated result of the classifier is:

Where fw(x, i) is the intermediate result of the i-th layer; x is the initial network input; _Wi is the parameter of the i-th exit layer, and the exit layer is the BoF pooling layer and the FC layer;

is the estimated result of the classifier; Train the parameters in the existing layers offline, and choose the same cross entropy loss function when training _Wi ;

Where p(i) and q(i) are the actual and predicted class distributions of each object, respectively;

is the cross entropy; When training the existing layer, the training set data {x1, x2, …, xN} is input into the model, N is the number of training data, and the existing layer parameters are optimized by batch gradient descent. Let L _total be the number of layers in the network, and the target vector set is

To get the correct result; first, without any existing layers, train the model with the following formula:

Where W′ is the updated overall network parameter; η is the learning rate. If the accuracy cannot meet the requirements, adjust η; j represents the vector position;

represents parameter gradient; x _j represents training data vector element; r _j represents target data vector element; The original model W is fixed, and based on the existing layers, the model is trained again to optimize the parameters of the BoF pooling layer and the FC layer. The expression is:

After training the exit layer, the average feature weight μ _i is calculated as the parameter for the exit decision, and the expression is:

Among them, k represents the kth overall classification target; During the inference process of the initial network input _xj , at each early exit layer, the weight ratio αW is the maximum feature weight divided by _μi multiplied by the user-specified hyperparameter β. The larger the maximum feature weight, the more confident the classification is. If αW is greater than 1, the inference is terminated, and the result of this early exit layer is applied as the final result, expressed as:

6. The method for dynamic exit of a neural network for multiple continuous reasoning according to claim 5 is characterized in that the predictor starts prediction from the _L0th layer of the neural network, and the predictor input is the execution result vector _y0 of the _L0th layer of the neural network and its corresponding exit layer execution result vector

The above results are expanded in dimension by using the vector

Add 0 at both ends to convert the vector

Depend on

Dimensions expanded to

dimension, and recorded as

Where N _c is a vector

The length is also the number of inference result categories, K is the number of feature vectors in the feature pooling group, and the specific calculation process is shown in the following formula:

l represents the lth element; The output vector is zero-padded

A one-dimensional convolution operation will be performed and a new feature weight vector will be obtained

The convolution weight is a one-dimensional vector h of length K. The specific calculation process is shown in the following formula:

By recursively converting the intermediate estimation results

Replace with

Repeat the calculation formula to obtain the predicted result of the discharge layer at position L ₀ +2, which is recorded as

Place the prediction result of the exit layer in any layer after L ₀ to obtain the PREDICT function, which is described as follows:

in,

Indicates the minimum value of the predicted output point after L ₀ ;

In the formula,

is the prediction exit confidence of the L ₀ +ζ layer. Therefore, L ₀ +ζ is the exit point predicted by the prediction function. If ζ cannot be found, a hyperparameter τ is introduced, that is, ζ∈[1,τ], τ≤L _total -L ₀ . The upper bound of τ indicates that the prediction results outside the last layer are prohibited. If no integer in [1,τ] satisfies the condition, ζ＝τ is set. Convert the existing prediction results ζ of PREDICT to the mid-level frequency f _M,i . For the inference task j _i , a relatively high-level conservative approach is applied in the interval between the start of inference and the L ₀ prediction.

The L _total of the model is completed in time without exiting, and the expression is:

Where _fH is the default maximum frequency of the computing platform. The computing platform can complete reasoning with L _total layers and will not exit early due to time _Ti ; frequency

Calibrated by Δf _i ; When predicting _L0 , adjust f to the intermediate frequency _fM,i according to the predicted exit point, and run the network until exiting at _Ti . Given a network with L _total layers and early prediction ζ, the intermediate frequency is calculated as:

where f _M,i is the lowest frequency predicted to complete inference at time _Ti . 7. The method for dynamic exit lightweighting of a neural network for multiple consecutive reasoning according to claim 6, characterized in that for the reasoning task j _i+1 , the cross-reasoning DVFS strategy provides a calibration suggestion Δf _i+1 after completing all previous tasks including j _i , and the calibration is implemented by a discrete incremental proportional integral PI regulator, and the expression is: Δf _i+1 =Δf _i +(K _P +K _I (t _i -t _i-1 ))e(t _i )-K _P e(t _i-1 ) Where K _P and K _I are the proportional coefficient and integral coefficient respectively; index i represents the i-th reasoning task in the current coordination cycle; _ti represents the relative completion time of reasoning j _i from the beginning of the coordination cycle; _{ti -t i-1} is the time interval between the completion of reasoning tasks j _i and j _i-1 ; The input deviation e(t _i ) of the PI regulator is an evaluation of the inference progress of T _c based on the total inference workload and the inference execution progress since the beginning of the coordination period, and is expressed as:

The first term is the reference speed for inference workload balancing, and the second term is the processing speed so far. 8. The method for dynamic exit of a neural network for multiple consecutive inferences according to claim 7 is characterized in that the frequency and the corresponding voltage are configured by the DVFS governor, and the network is executed according to the prediction and calibration of the small time scale and the large time scale respectively. At the beginning of each inference, Δf _i is derived based on the cross-inference calibration, and the DVFS governor sets the calculation configuration to

Inference without early exit is done conservatively. When early exit prediction is performed at the inference layer _L0 , the DVFS governor establishes an appropriate frequency configuration based on fM _,i derived from the internal inference predictor and Δf _i derived from the cross-inference calibration.

The expression is:

Therefore, the energy consumption is:

in,

is the time it takes to complete the prediction layer _L0 and its previous network layer j _i ;

For a given

The corresponding voltage; since the selection of f is based on the remaining workload predicted in inference, the intra-inference prediction can enable j _i to be completed by _Ti ; the cross-inference calibration target completes M sequential inferences through T _c . 9. A lightweight system for dynamic exit of a neural network for multiple consecutive reasonings, characterized by comprising: Module M1: Build a neural network-based inference model, predict the location where the network exits for each inference within a preset small time range, and predict the computing configuration accordingly, the computing configuration including frequency and voltage; Module M2: For multiple inferences within a preset large time range, processor frequency and voltage calibration is performed based on the remaining inference workload and time constraints; Module M3: Configured to execute a neural network based on prediction and calibration calculations to achieve dynamic voltage and frequency regulation. 10. The neural network dynamic exit lightweight system for multiple continuous reasoning according to claim 9 is characterized in that a reasoning coordination cycle, i.e., a reasoning task deadline, is set during the reasoning process, and the expression is:

Where λ represents the tightness of the task deadline, 0＜λ≤1, the smaller λ is, the tighter the deadline is; _ti represents the actual completion time of each reasoning; _Tc is the coordination cycle; M is the number of reasoning tasks; _Ti represents the coordination deadline of the i-th reasoning task; When performing inference tasks, the power consumed in the calculation process is active power P _active , which is the sum of dynamic power P _D , static power P _S , and constant power P _C ; The power consumed by the computing platform when it is idle is the idle power P _idle , which is the sum of the static power P _S and the constant power P _C ; _PD ＝ ^CV2f _{PS ＝ VNtrIS} Where C is the capacitance of the switching logic gate; V is the power supply voltage of the processor; f is the processor clock frequency; N _tr is the number of logic gates; I _S is the normalized static current of each logic gate; The energy consumption of each inference in _Ti is:

The network layer before early exit runs at (V _H , f _H ). After the network exits at point t _i , the computing platform reduces the voltage and frequency to the lowest level (V _L , f _L ) through the dynamic voltage and frequency scaling technology DVFS. Therefore, the energy consumption of inference task j _i is expressed as:

E _i represents the energy consumption of reasoning task j _i ; V _H represents the high supply voltage of the processor; f _H represents the high clock frequency of the processor; V _L represents the low supply voltage of the processor; In an inference task j _i , the network exit position is first predicted and the remaining number of layers is established to complete the inference; then (V, f) is established for prediction based on the remaining number of layers and the worst-case execution time Ti until the end of the current inference; based on the remaining amount of inference on T _c , the cross-inference calibrator is enabled through the feedback control strategy to calibrate (V, f) according to the multi-inference progress, total inference workload and time constraints; based on the prediction and calibration, the processor is updated to the appropriate computing configuration at runtime through DVFS.

Images