CN116720566A - A model performance optimization method and related equipment - Google Patents

Abstract

This application discloses a model performance optimization method, which is applied to a terminal. In the method, the terminal sends a first request to the server; within the first time, receives feedback information for the first request sent by the server, and the feedback information It can be used to determine the first operation mode of each operator among multiple operators of the neural network model. Through this method, the optimal operation method of each operator among the multiple operators of the terminal's neural network model can be obtained based on the terminal-cloud interaction, so that the terminal performs operations on the neural network model using the optimal operation method. During operation, better operating performance can be obtained without the need for developers to perform complex performance optimization operations for various different types and models of equipment that the neural network model may be deployed on, which improves the optimization efficiency of the neural network model in the terminal.

Description

A model performance optimization method and related equipment

Technical field

This application relates to the field of artificial intelligence technology, specifically to a model performance optimization method and related equipment.

Background technique

Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the nature of intelligence and produce a new class of intelligent machines that can respond in a manner similar to human intelligence. Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

With the rapid development of AI, neural network models (for example, deep neural network models) are deployed in various devices to play a role in the processing and analysis of various information such as images, videos, and voices. However, the composition of computing units in different devices is different, and different computing units are good at different computing scenarios, resulting in large performance differences when the same neural network model is run on different devices or computing units. In addition, the hardware and software versions in the device are often constantly updated and iterated, which also affects the running performance of the neural network model in the device.

Currently, in order to obtain better performance of neural network models on multiple devices, developers need to deploy the trained neural network models on multiple devices and conduct targeted testing and optimization respectively. The optimization efficiency is low and the workload is high. larger.

Contents of the invention

This application provides a model performance optimization method to solve the problem that in order to obtain better performance of neural network models on multiple devices, developers currently need to deploy the neural network models obtained after training on multiple devices and perform targeted operations respectively. Test optimizations comprehensively and optimize problems with low efficiency. This application also provides corresponding devices, equipment, computer-readable storage media, computer program products, etc.

The first aspect of this application provides a method for optimizing model performance, which is applied to a terminal. The method includes: sending a first request to a server, where the first request carries operator information of the neural network model and operating environment information of the neural network model; Within the first time, feedback information sent by the server in response to the first request is received, and the feedback information can be used to determine the first operation mode of each operator among multiple operators of the neural network model.

In the first aspect, the first operation mode is used to indicate the setting mode of the operating parameters when the operator performs operations on the terminal. For example, the operating parameters may include the operating frequency of the computing unit corresponding to the operator.

The feedback information can be used to determine the first operation mode of each operator in the neural network model. Specifically, the feedback information can be used to determine the first calculation unit corresponding to each operator in the neural network model. , and the first operation method of each operator in the neural network model in the corresponding first calculation unit. At this time, the feedback information may include either of the following two situations: (1) The feedback information may include information about the model operation strategy corresponding to the neural network model. The model operation strategy includes multiple calculations of the neural network model. The first operation method corresponding to each operator in the operator. (2) The feedback information includes operator performance information corresponding to the neural network model, and the operator performance information included in the feedback information meets the specified optimization conditions. In this case, if the operator performance information included in the feedback information satisfies the specified optimization conditions, it can be understood that the feedback information includes sufficient operator performance information. In one example, determining that the operator performance information included in the feedback information satisfies the specified optimization condition may be that the operator performance information included in the feedback information is related to each computing unit of the terminal, thereby being able to support the terminal from the computing unit of the terminal. , it is determined that the computing unit with better operating performance for the corresponding operator is the first computing unit of the corresponding operator, and the operation method that is determined to have better operation performance with the corresponding operator is the first operation method of the corresponding operator.

As can be seen from the above, in the first aspect, for the neural network model of the terminal, the optimal operation method (i.e., the first operation method) of each of the multiple operators of the neural network model can be obtained based on the relevant feedback information of the server. (one operation mode), so that when the terminal performs operations on the neural network model in the corresponding first operation mode, it can obtain better operating performance without the need for developers to deploy various different types and models of the neural network model. equipment, perform complex performance optimization operations, and improve the optimization efficiency of the neural network model in the terminal.

In a possible implementation of the first aspect, the method further includes: obtaining a first operation result of the data to be processed based on the feedback information.

In this possible implementation, the data to be processed can come from an application program corresponding to the neural network model. For example, the application can implement the face recognition function. At this time, the data to be processed in the application is the image to be processed. Alternatively, the application can implement a text translation function, in which case the data to be processed in the application is text information.

In a possible implementation of the first aspect, the first operation mode of each operator among the plurality of operators includes the operation mode of the corresponding operator in the corresponding first calculation unit, and the first calculation unit is the terminal. One or more of the computational units.

In this possible implementation, the type of computing unit may include a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processing unit (NPU). ), one or more of tensor processing unit (TPU). The first computing unit is one or more of the computing units of the terminal. When the first computing unit is a plurality of the computing units of the terminal, the type of each first computing unit may be the same or different, and each first computing unit may be of the same or different type. The hardware configuration of the computing units can be the same or different.

In a possible implementation of the first aspect, the feedback information also carries first reference information; the method further includes: after obtaining the first operation result, when the difference between the first performance information and the first reference information If the preset conditions are met, first performance information is sent to the server, where the first performance information is used to indicate the operating performance of at least one operator with respect to the corresponding first operation mode.

In this possible implementation manner, if the difference between the first performance information and the first reference information meets the preset condition, it may indicate that the difference between the first performance information and the first reference information is relatively large. When the corresponding operator performance information when the terminal is actually running is significantly different from the corresponding operator performance information in the relationship list, the server can update the corresponding operator performance information stored in the server based on the operator performance information when the terminal is actually running. .

In a possible implementation of the first aspect, the method further includes: no feedback information is received within the first time, or the feedback information cannot be used to determine each of the multiple operators of the neural network model. The second operation mode of each operator in the preset neural network model is used to obtain the second operation result of the data to be processed.

In this possible implementation, the feedback information sent by the server cannot be used to determine the first operation method of each operator in the neural network model. The feedback information cannot support the terminal to determine the operator in the neural network model. The first operation mode corresponding to all operators of .

For a certain operator in the neural network model, the feedback information cannot be used to determine the first operation method corresponding to the operator. The feedback information may not contain information about the operator, or the feedback information may contain The operator performance information of the operator is incomplete, for example, the operator's performance information about certain computing units and/or certain computing methods in the terminal is missing, etc.

In this possible implementation, when the terminal fails to obtain the optimal operation strategy for the neural network model from the server, it can determine each of the multiple operators of the neural network model based on the existing model operation strategy in the terminal. sub-corresponding second operation mode, thereby optimizing the operating performance of the neural network model. That is to say, in this possible implementation, the second operation mode corresponding to each operator in the neural network model may be based on the operator information and running environment information of the neural network model by the terminal. And determined by the model running strategy stored locally in the terminal.

In this possible implementation, when the neural network model is loaded next time, the operation of sending the first request to the server and subsequent operations can be re-executed to try to optimize the performance of the neural network model again.

In a possible implementation of the first aspect, the method further includes: after obtaining the second operation result, sending second performance information to the server, where the second performance information is used to indicate at least one operator about the corresponding second operation. mode operating performance.

In this possible implementation, in some examples, the terminal fails to obtain feedback information from the server that can be used to determine the first operation mode corresponding to each of the multiple operators of the neural network model. It may be that the server The operator performance information corresponding to the neural network model is missing in the server, or the operator performance of the neural network model in the server is incomplete. At this time, the terminal sends the second performance information to the server, and when the server lacks relevant operator performance information, it can supplement it based on the second performance information. Of course, in other examples, it may be that the communication connection between the terminal and the server fails, causing the terminal to be unable to receive feedback information. At this time, the terminal may not send the second performance information to the server, or the terminal may send the second performance information to the server, but the server does not update the stored operator performance information based on the second performance information.

In a possible implementation of the first aspect, an application program and a target software development kit SDK are installed in the terminal, and the application program corresponds to the neural network model; the above step: sending a first request to the server, including: through the application program Send a second request to the target SDK, and the second request contains the information of the neural network model; when the target SDK contains the target configuration information, the first request is generated based on the target configuration information and the second request, and the target SDK sends the third request to the server. In a request, the target configuration information is used to indicate the information content about the neural network model that the terminal can send to the server.

In this possible implementation, the target configuration information is used to indicate that the function of optimizing model performance through device-cloud interaction has been enabled. In this way, each application in the terminal can enable the function of optimizing model performance through device-cloud interaction according to their own needs through the target SDK, thereby providing interaction with the server for each different application in the terminal through the target SDK. In order to obtain the functions of the optimized neural network model, there is no need to set up related functions to interact with the server in different applications during the development stage, which improves the development efficiency of the application and facilitates the terminal execution of related functions to interact with the server. operate.

In a possible implementation of the first aspect, the feedback information includes operator performance information and/or model operation strategy information for the neural network model, and the operator performance information includes the operating performance of the corresponding operator with respect to the specified parameters. The information, the model operation strategy includes the first operation method of each operator in the plurality of operators of the neural network model.

In this possible implementation, the specified parameters can be considered as factors affecting the operating performance of the operator. Among them, the specified parameters corresponding to different operators can be different. For example, the specified parameters may be one or more of the following parameters: operator parameters (for example: dimensions of input data, dimensions of output data, size of convolution kernel), version of the AI inference framework, computing unit Type, operating frequency of the computing unit.

In a possible implementation of the first aspect, the neural network model is not associated with the hardware of the terminal.

In this possible implementation, the neural network model is not associated with the hardware of the terminal, which means that the neural network model is not adapted to the hardware of the terminal such as the chip. In this way, the compatibility and universality of the neural network model can be ensured when deployed on various devices. In addition, developers do not need to specifically optimize the operating performance of the neural network model on terminals for different types and models of terminals during the development stage, which reduces the developer's workload during the development stage and improves development efficiency.

A second aspect of this application provides a model performance optimization method, applied to a server. The method includes: receiving a first request sent by a terminal, where the first request carries operator information of the neural network model and operating environment information of the neural network model; Feedback information is sent to the terminal, and the feedback information is obtained based on the operator information and the running environment information, and the feedback information can be used to determine the first operation mode of each operator among the multiple operators of the neural network model.

In the first aspect, after receiving the first request sent by the terminal, the server can query the operator performance information corresponding to the neural network model based on the operator information and operating environment information carried in the first request. If the neural network model is query Based on the operator performance information corresponding to the model, feedback information can be obtained based on the operator performance information corresponding to the neural network model.

In a possible implementation of the second aspect, the method further includes: receiving first performance information sent by the terminal, the difference between the first performance information and the first reference information meets a preset condition, and the feedback information carries First reference information; based on the first performance information, update the corresponding operator performance information in the server.

In this possible implementation, when the operator performance information when the terminal is actually running is significantly different from the corresponding operator performance information in the server, the server can calculate the corresponding operator performance information stored in the server based on the operator performance information when the terminal is actually running. Operator performance information is updated.

In a possible implementation of the second aspect, the method further includes: if the server does not obtain feedback information, storing operator information and operating environment information of the neural network model.

In this possible implementation, when the server cannot obtain feedback information, it stores the operator information and operating environment information of the neural network model, thereby indicating that the server is missing relevant operator performance information. The server stores the operator information and operating environment information of the neural network model, which can facilitate the server to maintain and update the missing relevant operator performance information.

In a possible implementation of the second aspect, the method further includes: receiving second performance information sent by the terminal, the second performance information being used to indicate the operating performance of at least one operator with respect to the corresponding second operation mode; according to the first 2. Performance information, operator information and operating environment information of the neural network model, and update the server.

In this possible implementation, the server can receive the second performance information from the terminal to supplement the operator performance information missing in the server, thereby making the information stored in the server more complete.

The third aspect of the present application provides a model performance optimization device, which is applied to a terminal. The device has the function of implementing the method of the above-mentioned first aspect or any of the possible implementation methods of the first aspect. This function can be implemented by hardware, or it can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as: sending module, receiving module and processing module.

A fourth aspect of the present application provides a terminal, which includes at least one processor, a memory, and computer-executable instructions stored in the memory and executable on the processor. When the computer-executable instructions are executed by the processor, the processor executes as follows: Methods of the above first aspect or any possible implementation of the first aspect.

A fifth aspect of the present application provides a computer-readable storage medium that stores one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes any of the above-mentioned first aspects or possible methods of the first aspect. Ways to implement it.

A sixth aspect of the present application provides a computer program product that stores one or more computer execution instructions. When the computer execution instructions are executed by a processor, the processor executes the above first aspect or any possible implementation of the first aspect. Methods.

A seventh aspect of the present application provides a chip system. The chip system includes a processor and is used to support a terminal to implement the functions involved in the above-mentioned first aspect or any possible implementation manner of the first aspect. In a possible design, the chip system may also include a memory, which is used to store necessary program instructions and data for the computer device. The chip system may be composed of chips, or may include chips and other discrete devices.

The eighth aspect of this application provides a model performance optimization device, which can be applied to a server, and has the function of implementing the above second aspect or any of the possible implementation methods of the second aspect. This function can be implemented by hardware, or it can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, such as a receiving module and a sending module.

A ninth aspect of the present application provides a server, which includes at least one processor, a memory, and computer-executable instructions stored in the memory and executable on the processor. When the computer-executable instructions are executed by the processor, the processor executes: The above second aspect or any possible implementation method of the second aspect.

A tenth aspect of the present application provides a computer-readable storage medium that stores one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes any of the above-mentioned second aspects or possible second aspects. Ways to implement it.

An eleventh aspect of the present application provides a computer program product that stores one or more computer execution instructions. When the computer execution instructions are executed by a processor, the processor executes the above second aspect or any possible implementation of the second aspect. way method.

A twelfth aspect of the present application provides a chip system. The chip system includes a processor and is used to support the server in implementing the functions involved in the above-mentioned second aspect or any possible implementation manner of the second aspect. In a possible design, the chip system may also include a memory, which is used to store necessary program instructions and data for the computer device. The chip system may be composed of chips, or may include chips and other discrete devices.

Among them, the technical effects brought by the third to seventh aspects or any one of the possible implementation methods can be found in the technical effects brought by the first aspect or the related possible implementation methods of the first aspect, and the eighth to tenth aspects For the technical effects brought by the two aspects or any one of the possible implementation methods, please refer to the technical effects brought by the second aspect or the related possible implementation methods of the second aspect, and will not be described again here.

Description of the drawings

Figure 1 is a schematic diagram of an artificial intelligence main body framework provided by an embodiment of the present invention;

Figure 2 is an exemplary schematic diagram of a server and a terminal provided by an embodiment of the present invention;

Figure 3 is an exemplary schematic diagram of the information interaction process between the server and the terminal provided by the embodiment of the present invention;

Figure 4 is an exemplary schematic diagram of information interaction between a server and multiple terminals provided by an embodiment of the present invention;

Figure 5 is another exemplary schematic diagram of the information interaction process between the server and the terminal provided by the embodiment of the present invention;

Figure 6 is another exemplary schematic diagram of the information interaction process between the server and the terminal provided by the embodiment of the present invention;

Figure 7 is an exemplary schematic diagram of the information interaction process between the application program, the target SDK and the server provided by the embodiment of the present invention;

Figure 8 is a schematic diagram of an embodiment of the model performance optimization device provided by the embodiment of the present application;

Figure 9 is a schematic diagram of another embodiment of the model performance optimization device provided by the embodiment of the present application;

Figure 10 is a schematic structural diagram of a terminal provided by an embodiment of the present application;

Figure 11 is a schematic structural diagram of a server provided by an embodiment of the present application.

Detailed ways

The embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention. The terms used in the embodiments of the present invention are only used to explain specific embodiments of the present invention and are not intended to limit the present invention.

The embodiments of the present application are described below with reference to the accompanying drawings. Persons of ordinary skill in the art know that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

In this application, "at least one" refers to one or more, and "plurality" refers to two or more. "And/or" describes the relationship between associated objects, indicating that there can be three relationships, for example, A and/or B, which can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A, B can be singular or plural. The character "/" generally indicates that the related objects are in an "or" relationship. "At least one of the following" or similar expressions thereof refers to any combination of these items, including any combination of single or plural items. The terms "first", "second", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms so used are interchangeable under appropriate circumstances, and are merely a way of distinguishing objects with the same attributes in describing the embodiments of the present application. Furthermore, the terms "include" and "having" and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, product or apparatus comprising a series of elements need not be limited to those elements, but may include not explicitly other elements specifically listed or inherent to such processes, methods, products or equipment.

Since the embodiments of the present application relate to the field of artificial intelligence, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

Please refer to Figure 1. Figure 1 shows a structural schematic diagram of the main framework of artificial intelligence. The following is an analysis of the above artificial intelligence theme from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). framework is explained. Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has gone through the condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (providing and processing technology implementation) to the systematic industrial ecological process.

(1)Infrastructure

Infrastructure provides computing power support for artificial intelligence systems, enables communication with the external world, and supports it through basic platforms. Communicate with the outside through sensors; computing power is provided by smart chips (central processing unit (CPU), graphics processing unit (GPU), neural network processing unit (NPU), tensor processing It provides tensor processing unit (TPU), application specific integrated circuit (ASIC), field programmable gate array (FPGA) and other hardware acceleration chips); the basic platform includes distributed computing framework and Network and other related platform guarantees and support can include cloud storage and computing, interconnection networks, etc. For example, sensors communicate with the outside world to obtain data, which are provided to smart chips in the distributed computing system provided by the basic platform for calculation.

(2)Data

Data from the upper layer of the infrastructure is used to represent data sources in the field of artificial intelligence. The data involves graphics, images, voice, and text, as well as IoT data of traditional devices, including business data of existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.

(3)Data processing

Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

Among them, machine learning and deep learning can perform symbolic and formal intelligent information modeling, extraction, preprocessing, training, etc. on data.

Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formal information to perform machine thinking and problem solving based on reasoning control strategies. Typical functions are search and matching.

Decision-making refers to the process of decision-making after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

(4) General ability

After the data is processed as mentioned above, some general capabilities can be formed based on the results of further data processing, such as algorithms or a general system, such as translation, text analysis, computer vision processing, speech recognition, and image processing. identification, etc.

(5) Intelligent products and industry applications

Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Its application fields mainly include: intelligent terminals, intelligent transportation, Smart healthcare, autonomous driving, smart cities, etc.

(6)Neural network

The neural network can be composed of neural units. The neural unit can refer to an operation unit that takes xs (ie, input data) and intercept 1 as input. The output of the operation unit can be:

Among them, s=1, 2,...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

(7)Operator

An operator refers to a function that implements a specific function. For example, take the reshape operator, which is used to reinterpret the shape of tensor data. As another example, take the transpose operator, which is used to adjust the dimension order of tensor data. In this application, commonly used functions used to build deep learning model algorithms are collectively called operators, and any operation performed on any function can be considered an operator. For example, convolution is a mathematical method of integral transformation. For example, if the function f3 is generated by two functions f1 and f2, then f1, f2 and f3 can be used as an operator respectively.

With the rapid development of AI, neural network models have been widely used in various devices such as mobile phones, TVs, desktop computers, and vehicle-mounted equipment to achieve functions such as speech recognition and image processing, greatly improving device performance and improving user experience. usage experience.

However, in the actual deployment process, when evaluating the running performance of the neural network model in the terminal, the following problems often exist:

(1) At present, the structures of neural network models used in various application scenarios are quite different. For example, in speech processing scenarios, the transformer model is often used, while in image processing scenarios, deep neural network (DNN) is often used. , recurrent neural network (RNN) model or convolutional neural networks (CNN) model, making optimization difficult.

(2) Different devices contain different computing units, and different computing units (for example, CPU, GPU, NPU, TPU) are good at different computing scenarios. For example, NPU has better computing performance for matrix calculations, while CPU is better at logical operations, scalar calculations, etc. For example, a neural network model with a large proportion of matrix calculations can achieve several times to several times better performance when running on NPU than running on CPU. The ten-fold performance improvement results in large performance differences when the same neural network model is run on different devices or computing units.

(3) The hardware and software versions in the device are often constantly updated and iterated, and the running performance of the neural network model in the device will be affected by the hardware and software versions. Therefore, in order to ensure the running performance of the neural network model in the device, Developers need to continuously optimize and adapt the neural network model as hardware and software are updated and iterated. Otherwise, the running performance of the neural network model in the device is likely to be greatly affected, thus affecting the user experience.

Based on the above problems, in the actual development process, in order to meet the deployment needs of applications in various devices and obtain better performance of neural network models on multiple devices, developers need to deploy neural network models on multiple devices. equipment and perform targeted performance optimization respectively.

The following is a brief introduction to a currently commonly used model performance optimization method.

Currently, for a certain device, after loading the neural network model, the device can analyze the neural network model. In addition, the structure of the neural network model may be simply optimized, such as optimizing the operators in the neural network model. Fusion or pruning, and obtain operator information in the neural network model after simple optimization. After obtaining the operator information in the neural network model, the operator information of the neural network model can be sent to the computing library corresponding to different computing units in the device, thereby obtaining the computing unit support verification results returned by the computing library. The computing unit support verification results may include operating performance information such as the computing unit's operating speed and power consumption for operators in the neural network model. After the device obtains the computing unit support verification results, it can make decisions based on the computing unit support verification results to determine how to allocate operators in the neural network model to the computing units of the device for calculations. In this way, the device can optimize model performance based on the results of this decision.

However, this commonly used model performance optimization method for a certain device will cause the process from loading the neural network model to applying the neural network model to the device to perform calculations, which takes a long time, and the model performance optimization strategy in the device is usually relatively simple, and the calculation The support verification results of units are usually less, resulting in less than ideal model performance optimization results.

In addition, the model parameters of a neural network model optimized for a specific device will be deeply coupled with the device, resulting in poor compatibility between the optimized neural network model and other devices. Therefore, the optimization results corresponding to different devices are usually not universal. , making the performance optimization operation of neural network models for each device usually cumbersome and cannot be simplified.

It can be seen that at present, the optimization efficiency of the operating performance of the neural network model in the terminal is low and the workload is large.

Based on this, embodiments of the present application provide a model performance optimization method to solve the problem that in order to obtain better performance of neural network models on multiple devices, developers currently need to deploy the trained neural network models on multiple devices. The equipment is tested and optimized separately to optimize problems with low efficiency.

The following is a detailed introduction to the model performance optimization method in the embodiment of this application.

The model performance optimization method of the embodiment of the present application can be applied to a device-cloud collaboration system, which includes a terminal and a server.

For example, the terminal can be a mobile phone (mobile phone), a tablet computer (pad), a computer with wireless transceiver functions, a virtual reality (VR) terminal, an augmented reality (AR) terminal, an industrial control ( Wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical, wireless terminals in smart grid, and transportation safety Wireless terminals, wireless terminals in smart cities (smart cities), wireless terminals in smart homes (smart homes), wireless terminals in the Internet of Things (IoT), etc.

The server in the embodiment of this application may be a server or a server cluster.

Data transmission can be carried out between the terminal and the server based on the specified wireless communication method or wired communication method. There may be multiple communication methods between the terminal and the server, which are not limited in the embodiments of this application.

As shown in Figure 2, it is an exemplary schematic diagram of a server and a terminal.

In Figure 2, the server 201 can interact with multiple terminals (such as the terminal 211, the terminal 212 and the terminal 213 shown in Figure 2). Among them, the equipment types and equipment models of terminal 211, terminal 212 and terminal 213 may be the same or different. Server 201 may also be connected to a data storage system.

An application is installed in the terminal, and the application can contain a neural network model. The application can be downloaded and installed by the terminal from the application market or other channels, or it can be included in the terminal when it leaves the factory.

In some embodiments, the neural network model is not associated with the terminal's hardware.

In other words, the neural network model is not adapted to the terminal's hardware such as chips. In this way, the compatibility and universality of the neural network model can be ensured when deployed on various devices. In addition, developers do not need to specifically optimize the operating performance of the neural network model on terminals for different types and models of terminals during the development stage, which reduces the developer's workload during the development stage and improves development efficiency.

The embodiments of this application do not limit the type of the neural network model. For example, the neural network model may be a regression model, DNN, CNN or RNN.

The neural network model corresponding to the application program can be included in the installation package of the application program, or after the terminal installs the application program, the neural network model can be downloaded to the terminal based on the specified configuration information in the application program. This neural network model can be used to process data to be processed in applications.

This model performance optimization method can be applied to various information processing fields such as natural language processing, image processing, audio and video processing in the field of artificial intelligence. The specific functions of the application and the functions of the neural network model are not limited here.

For example, the application can implement the face recognition function. At this time, the application can include a neural network model for face recognition, and the image to be processed obtained by the application can be processed through the neural network model. Process to obtain the corresponding face recognition results. At this time, the data to be processed in the application is the image to be processed. Alternatively, the application can implement a text translation function, and the text information obtained by the application can be translated through the neural network model to obtain the translation result. At this time, the data to be processed in the application is text information.

Referring to Figure 3, specifically, the model performance optimization method in the embodiment of the present application may include steps 301-305.

Step 301: The terminal sends a first request to the server.

The first request carries operator information of the neural network model and operating environment information of the neural network model.

In this embodiment of the present application, the server can provide a designated application programming interface (API), and the terminal calls the API and sends the first request to the server based on the communication connection with the server.

Generally speaking, operator information can include information about each operator in the neural network model.

Among them, the number and type of operators in the neural network model can be determined based on actual application scenarios, and are not limited here. For example, the operators in the neural network model may include one or more of convolution operators, pooling operators, activation operators, and fully connected operators, and each operator may include one or more , the operator parameters between operators of the same type can be the same or different. For example, the neural network model can include two convolution operators, two pooling operators and a fully connected operator. Operators in the neural network model can correspond to layers in the neural network.

The operator information may include one or more of the following information of the corresponding operator: operator type, operator weight information, and one or more operator parameter information. The operator parameter information is different from the weight of the operator. For example, the operator parameter information may include the input data and/or the dimensions of the output data of the operator, and the relevant parameters of the elements in the operator (such as the convolution operator the size of the convolution kernel).

In addition, in some examples, the first request may also include association relationship information between operators of the neural network model.

The association information between operators may include information such as the connection relationship between operators in the neural network model and the data flow direction between operators. The correlation information between operators is represented by the graph structure of the neural network model.

The specific forms of the operator information and the association information between operators in the first request are not limited here.

For example, the first request may record the operator information of each operator in the neural network model in the form of a list. Alternatively, the first request may include a complete neural network model. In this case, the complete neural network model not only includes operator information of each operator, but also includes association relationship information between operators.

The running environment information corresponding to the neural network model may include hardware information and/or software information associated with the neural network model when it is run on the terminal.

For example, the software information may include information about the application program corresponding to the neural network model and/or information about the AI inference framework. Among them, the AI inference framework is used to control the running process of the neural network model on the terminal. For example, the AI inference framework can load and parse the neural network model, obtain the information of the operators in the neural network model, and then obtain the information of the neural network model. The binary executable file corresponding to the operator is then allocated to the computing unit of the terminal for calculation. The information of the application program may include one or more of the name, identifier, version number and other information of the application program. The information of the AI inference framework may include one or more of the name, identifier, version number and other information of the AI inference framework. kind.

The hardware information may include information about the chip corresponding to the neural network model in the terminal. The chip information may include one or more of the chip supplier, chip model, and information about the computing unit included in the chip.

The type of computing unit in the chip may include a CPU, or one or more of a GPU, NPU, and TPU. Information about the computing unit included in the chip may include one or more of the type, model, performance parameters (such as the operating frequency of the computing unit), the computing unit's support for operators, and other information.

In addition, the first request may also carry identification information such as the name of the terminal and the International Mobile Equipment Identity (IMEI), so that the server can identify the sender of the first request.

In this embodiment of the present application, the triggering conditions that trigger the terminal to send the first request to the server may be in various situations, which are not limited here.

For example, the neural network model corresponds to the application program. The terminal may send a first request to the server every time it starts the application program; or it may send the first request to the server when the terminal starts the application program for the first time; or it may When the terminal starts the application program and applies the neural network model for the first time, it sends the first request to the server; or, the terminal may start the application program and detects that the information of the neural network model that optimizes the running performance is not stored in the terminal. When the application is running, the first request is sent to the server; or, the terminal can periodically trigger the operation of sending the first request to the server while the application is running; or, after detecting the updated information of the neural network model, the terminal can send the first request to the server. The server sends the first request.

Step 302: The server receives the first request sent by the terminal.

After receiving the first request sent by the terminal, the server can query the operator performance information corresponding to the neural network model based on the operator information and operating environment information carried in the first request. If the operator performance information corresponding to the neural network model is queried, information, you can obtain feedback information based on the operator performance information corresponding to the neural network model.

For example, the server may include operator performance information and index information corresponding to each operator performance information. In this way, the operator information and running environment information carried in the first request can be matched with each index information. If the operator information and running environment information carried in the first request are successfully matched with a certain index information, then the operator information and running environment information carried in the first request can be matched with a certain index information. The operator performance information corresponding to the successfully matched index information is used as the operator performance information corresponding to the neural network model, thereby obtaining feedback information.

Among them, the operator performance information includes information about the operating performance of the corresponding operator with respect to specified parameters.

The specified parameters can be considered as factors affecting the operating performance of the operator. Since the operation operations related to the operator are performed by the computing unit in the terminal, the specified parameters usually include parameters related to the computing unit. Among them, the specified parameters corresponding to different operators can be different. For example, the specified parameter may be one or more of the following parameters: the data dimension of the operator (for example: the dimension of the input data, the dimension of the output data), operator parameters, version of the AI inference framework, computing unit Type, operating frequency of the computing unit.

Operator parameters are related to the type of operator, and the operator parameters in different types of operators can be different. Taking the convolution operator as an example, the operator parameters of the convolution operator may include the size of the convolution kernel in the convolution operator, the moving step size of the convolution kernel in the convolution operation, etc. Generally speaking, although the types and structures of current neural network models are diverse, the types of operators involved are relatively limited, and the selection ranges of operator parameters and data dimensions of the same type of operators are relatively limited, and Probably the same in many cases. Therefore, operator performance information for any type of operator can generally be used in a variety of neural network models containing that operator.

Step 303: If the server obtains feedback information for the first request based on the operator information and running environment information, it sends the feedback information to the terminal.

The feedback information can be used to determine the first operation mode of each of the multiple operators of the neural network model.

The first operation mode is used to indicate the setting mode of the operating parameters when the operator performs operations on the terminal. The operating parameters may include parameters that affect the operation performance corresponding to the operator. Exemplarily, the operation parameters may include parameters related to the computing unit.

In some embodiments, the first operation mode of each operator in the plurality of operators includes the operation mode of the corresponding operator in the corresponding first computing unit, and the first computing unit is one or more of the computing units of the terminal. indivual.

Since in the terminal, the operations corresponding to the operators are usually performed by the computing unit in the terminal, it can be considered that the first operation method of each operator among the multiple operators of the neural network model specifically refers to:

The first operation mode of the first calculation unit corresponding to each operator in the neural network model.

At this time, the first operation mode is used to indicate the setting mode of the operating parameters when the operator performs operations on the corresponding first computing unit. By way of example, the operating parameters may include the operating frequency of the computing unit.

The first computing unit is one or more of the computing units of the terminal. When the first computing unit is a plurality of the computing units of the terminal, the type of each first computing unit may be the same or different, and each first computing unit may be of the same or different type. The hardware configuration of the computing units can be the same or different.

The first operation method in which the feedback information can be used to determine each of the multiple operators of the neural network model may specifically refer to: the feedback information can be used to determine that the feedback information can be used to determine the multiple operators of the neural network model. a first computing unit corresponding to each operator, and a first operation mode of each operator in the corresponding first computing unit of the neural network model.

At this time, the feedback information can include either of the following two situations:

(1) The feedback information may include information about the model operation strategy corresponding to the neural network model. The model operation strategy includes the first computing unit corresponding to each of the multiple operators of the neural network model, and the neural network model. Each operator in the plurality of operators is in the first operation mode of the corresponding first calculation unit.

(2) The feedback information includes operator performance information corresponding to the neural network model, and the operator performance information included in the feedback information meets the specified optimization conditions.

In this case, if the operator performance information included in the feedback information satisfies the specified optimization conditions, it can be understood that the feedback information includes sufficient operator performance information. In one example, determining that the operator performance information included in the feedback information satisfies the specified optimization condition may be that the operator performance information included in the feedback information is related to each computing unit of the terminal, thereby being able to support the terminal from the computing unit of the terminal. , it is determined that the computing unit with better operating performance for the corresponding operator is the first computing unit of the corresponding operator, and the operation method that is determined to have better operation performance with the corresponding operator is the first operation method of the corresponding operator.

In some embodiments, the feedback information includes operator performance information and/or model operation strategy information for the neural network model. The operator performance information includes information about the operating performance of the corresponding operator with respect to specified parameters. The model operation strategy includes neural network model operation strategy information. The first operation mode of each operator in the network model.

In one example, the feedback information may include operator performance information of the neural network model. The operator performance information corresponding to the neural network model can indicate the operating performance of the operators in the neural network model when operating in at least one operation mode.

In another example, the feedback information may include information about the first operation mode corresponding to each of the multiple operators of the neural network model.

Step 304: The terminal receives the feedback information sent by the server within the first time.

The time interval between the first time and the time of sending the first request may be determined according to actual application scenarios. For example, the time interval between the first time and the time of sending the first request can be pre-configured by the developer on the terminal; or it can also be pre-determined and sent by the server based on information such as the time required for the server to obtain feedback information. to the terminal, and then configure it from the terminal.

If the terminal does not receive the feedback information of the first request sent by the server within the first time, or the feedback information received by the terminal cannot be used to determine the first operation of each operator among the multiple operators of the neural network model In this way, please refer to the description of relevant embodiments of step 503 in Figure 5 and subsequent steps, which will not be described again here.

Feedback information can be used to determine the first operation mode of each operator in the neural network model, which can include either of the following two situations:

(1) The feedback information may include information on the model operation strategy corresponding to the neural network model.

The model operation strategy includes a first operation method corresponding to each operator in the neural network model.

At this time, based on the feedback information, the terminal can directly obtain information about the first operation mode corresponding to each of the multiple operators of the neural network model.

(2) The feedback information includes operator performance information corresponding to the neural network model, and the operator performance information included in the feedback information meets the specified optimization conditions.

In this case, if the operator performance information included in the feedback information satisfies the specified optimization conditions, it can be understood that the feedback information includes sufficient operator performance information.

For the terminal, there are many ways to determine that the operator performance information included in the feedback information satisfies the specified optimization conditions.

In one example, the feedback information may include operator performance information of each operator of the neural network model, and the operator performance information included in the feedback information is related to each computing unit of the terminal, thereby being able to support the terminal. From the computing units of the terminal, it is determined that the computing unit with better operating performance for the corresponding operator is the first computing unit of the corresponding operator, and the computing method that has better operating performance for the corresponding operator is determined to be the third computing unit of the corresponding operator. A method of operation.

For example, the neural network model contains convolution operators, and the terminal chips include CPU, NPU and GPU.

If the feedback information includes the operator performance information of the convolution operator on the CPU and the operator performance information of the convolution operator on the NPU, but does not include the operator performance information of the convolution operator on the GPU, then the feedback can be considered In the information, the operator performance information about the convolution operator is incomplete and does not include enough operator performance information. Therefore, it is considered that the operator performance information in the feedback information does not meet the specified optimization conditions.

And if the feedback information includes the operator performance information of the convolution operator on the CPU, the operator performance information of the convolution operator on the NPU, and the operator performance information of the convolution operator on the GPU, then it can be considered that the feedback information, The operator performance information about the convolution operator is relatively complete and meets the specified optimization conditions. Therefore, the feedback information contains sufficient operator performance information for the convolution operator.

At this time, the terminal can compare the operator performance information of the convolution operator on the CPU, the operator performance information of the convolution operator on the NPU, and the operator performance information of the convolution operator on the GPU, and select the one with the best operating performance. The calculation unit serves as the first calculation unit of the convolution operator.

In one example, the determination may be to determine, for each operator among the multiple operators of the neural network model, the first calculation unit corresponding to each operator, and the corresponding first calculation unit for each operator. The first operation mode of the unit.

In another example, the operators of the neural network model can also be combined to obtain N first operator sets, and then the first computing unit corresponding to each first operator set and each first The first operation mode of the operator set in each corresponding first calculation unit, where N is a positive integer, and each first operator set contains at least one operator. There can be many strategies for combining each operator, which are not limited here. For example, based on the type and number of operators and the relationship between operators, each operator can be combined to obtain N first operator sets. For example, if the neural network model includes multiple adjacent convolution operators, and the operator parameters such as operator dimensions of the multiple adjacent convolution operators are the same or close, then the multiple adjacent convolution operators can be The convolution operators are combined to obtain a first operator set, and the convolution operators in the first operator set can be run by the same first computing unit.

Generally speaking, the operating performance of the neural network model on the terminal is largely determined by the operating performance of each of the multiple operators of the neural network model, and the operators in the neural network model are determined by different computing units. Execution, the impact caused by causing the neural network model to be executed across computing units is usually small and is not usually a determining factor in the operational performance of the neural network model. That is to say, if each operator in the neural network model is assigned to the corresponding computing unit with the best operating performance to perform calculations, then the overall operating performance of the neural network model on the terminal is usually the optimal operating performance. Or better operating performance.

Therefore, in the embodiment of the present application, for the neural network model of the terminal, the optimal operation method (i.e., the first operation method) of each of the multiple operators of the neural network model can be obtained based on the relevant feedback information of the server. mode), so that when the terminal performs calculations on the neural network in the first calculation mode, it can obtain better operating performance, without the need for developers to perform complex operations on various types and models of equipment that may be deployed in the neural network model. Performance optimization operations improve the optimization efficiency of neural network models on the terminal.

In some embodiments, the method further includes step 305.

Step 305: The terminal obtains the first operation result of the data to be processed based on the feedback information.

Among them, the data to be processed can come from the application program corresponding to the neural network model. The type of data to be processed is not limited here. For example, the application can implement the face recognition function. At this time, the data to be processed in the application is the image to be processed. Alternatively, the application can implement a text translation function, in which case the data to be processed in the application is text information.

In the embodiment of the present application, the corresponding operator can be compiled based on the first computing unit and the first computing method corresponding to each operator in the neural network model to obtain an algorithm that can be efficiently loaded and run by the corresponding first computing unit. binary file. In this way, when obtaining the data to be processed, the running timing of each first computing unit can be determined according to the structure of the neural network model, and based on the running timing, the binary file of the corresponding operator is loaded and run through each first computing unit to Obtain the first running result for the data to be processed.

In some examples, after obtaining a binary file of each operator that can be efficiently loaded and run by the corresponding first computing unit, the terminal can store the binary file, thereby receiving multiple subsequent requests for each set of data to be processed. When processing instructions, it is convenient for the first computing unit to load and run efficiently. Moreover, after the binary file is stored in the terminal, it can not only be loaded and run repeatedly as needed during the current running of the application corresponding to the neural network model, but can also be run in the next or subsequent runs of the application. During the process, it is loaded and run repeatedly.

In other examples, after obtaining the binary file of each operator that can be efficiently loaded and run by the corresponding first computing unit, due to the large size of the binary file or the small storage resources in the terminal, the terminal The binary file is only temporarily stored in the memory during the current running of the application corresponding to the neural network model, and the binary file is deleted from the terminal after the current running of the application. In this example, the terminal can trigger the operation of sending the first request to the server every time the application is started, and subsequent operations to re-determine the operators in each neural network model every time the application is started. The corresponding first computing unit and the corresponding first operating mode are obtained, and the corresponding binary file is obtained for loading and running by the first computing unit.

In this embodiment of the present application, the server can interact with multiple terminals to perform performance optimization for the neural network models of multiple terminals.

For example, as shown in Figure 4, it is an exemplary schematic diagram of information interaction between a server and multiple terminals.

The server 40 can interact with the terminal 41, the terminal 42 and the terminal 43 respectively to provide model performance optimization functions for the terminal 41, the terminal 42 and the terminal 43 respectively. Among them, the operators in the neural network model 10 of the application program 1 in the terminal 41 can be calculated by the CPU 411, GPU 412 and NPU 413 respectively, and the operators in the neural network model 20 of the application program 2 in the terminal 42 can be calculated by the CPU 421 and NPU 413 respectively. The GPU 422 performs calculations, and the operators in the neural network model 30 of the application program 3 in the terminal 43 can be calculated by the CPU 431 and the NPU 432 respectively.

It can be seen that through the embodiments of this application, the server can efficiently and quickly provide optimal operating modes for the corresponding neural network models of multiple terminals, and the optimization efficiency is high.

In some embodiments, as shown in Figure 5, after the above step 302, the model performance optimization method further includes:

Step 503: If the terminal does not receive feedback information within the first time, or the feedback information cannot be used to determine the first operation mode of each operator among the multiple operators of the neural network model, then the terminal performs the operation based on the preset neural network model. The second operation mode of each operator in the network model obtains the second operation result of the data to be processed.

Wherein, the second operation method of each operator among the multiple operators of the neural network model may specifically refer to: the second operation method of each operator among the multiple operators of the neural network model in the corresponding second calculation unit. Way.

The second computing unit is one or more of the computing units of the terminal. When the second computing unit is a plurality of computing units of the terminal, the type of each second computing unit may be the same or different, and the hardware configuration of each second computing unit may be the same or different.

It can be understood that the second computing unit and the first computing unit in the embodiment of the present application both belong to the computing unit of the terminal. The determination strategy of the first computing unit and the determination strategy of the second computing unit are usually independent of each other. Therefore, the first computing unit The calculation unit and the second calculation unit may be the same or different. The first calculation method corresponding to the first calculation unit and the second calculation method corresponding to the second calculation unit may be the same or different, and are not limited here.

There may be many reasons why the terminal does not receive the feedback information. For example, the communication connection between the terminal and the server may fail, or the server may not send the feedback information.

The feedback information received by the terminal within the first time cannot be used to determine the first operation mode corresponding to each of the multiple operators of the neural network model. It may be that the transmission speed of the communication connection between the terminal and the server is relatively slow. Slow, causing the terminal to fail to fully receive the feedback information sent by the server, or it may be that the feedback information sent by the server cannot be used to determine the first operation mode corresponding to each of the multiple operators of the neural network model.

The feedback information sent by the server cannot be used to determine the first operation method of each operator in the neural network model. The feedback information cannot support the terminal to determine the first operation method corresponding to all operators in the neural network model. Operation method.

For a certain operator in the neural network model, the feedback information cannot be used to determine the first operation method corresponding to the operator. The feedback information may not contain information about the operator, or the feedback information may contain The operator performance information of the operator is incomplete, for example, the operator's performance information about certain computing units and/or certain computing methods in the terminal is missing, etc. At this time, it may be that the server has not queried the complete operator performance information corresponding to multiple operators of the neural network model, or it may be that the terminal has not received the complete information sent by the server within the first time.

In the embodiment of the present application, the information about the second operation mode corresponding to each of the multiple operators of the neural network model may be pre-stored in the terminal, or the terminal may calculate the operation method according to the neural network based on a preset operation strategy. It is determined by the operator information of the model and the operating environment information of the neural network model. The preset calculation strategy is not limited here.

Through the embodiments of the present application, when the terminal fails to obtain the operation strategy for the neural network model from the server, it can determine the corresponding operator of each of the multiple operators of the neural network model based on the existing model operation strategy in the terminal. The second operation method, thereby optimizing the operating performance of the neural network model.

The next time the neural network model is loaded, the operation of sending the first request to the server and subsequent operations can be re-executed to try to optimize the performance of the neural network model again.

In some embodiments, the server stores information on the correspondence between index information and operator performance information. The index information is used to match the operator information and the running environment information. The operator performance information is used in the corresponding index information. When the operator information and running environment information are successfully matched, feedback information is obtained.

In one example, the correspondence between the index information and the operator performance information can be stored in the form of a relationship list. The corresponding relationship can also have other storage forms, which are not limited here.

The following uses the relationship list as an example to introduce the information stored in the server.

In this example, the relationship list is used to indicate the correspondence between index information and operator performance information. Generally speaking, each index information in the relationship list corresponds to each operator performance information one-to-one. An index information and corresponding operator performance information can form a set of correspondence relationships. The relationship list can include one or more sets of corresponding relationships.

For example, the index information may include reference operator information and reference running environment information of the corresponding operator. The reference operator information and reference running environment information may be used to compare the operator information and the running environment information in the first request. match.

Reference operator information includes one or more of the following information:

Operator type, operator weight information, one or more operator parameter information (for example: the dimensions of the operator’s input data, the dimensions of the output data, the size of the convolution kernel in the convolution operator, etc.), the relationship information between them.

The reference operating environment information includes reference hardware operating environment information and reference software operating environment information.

Reference hardware operating environment information includes one or more of the following information:

The supplier of the chip, the chip model, the type of computing unit contained in the chip, the model of the computing unit, performance parameters (such as the operating frequency of the computing unit), and the computing unit's support for operators.

Reference software running environment information includes one or more of the following information:

Information about the application program corresponding to the neural network model (such as the name, identification and/or version number of the application program), and information about the AI inference framework (such as the name and version number of the AI inference framework).

When a certain index information successfully matches the operator information and running environment information in the first request, the operator performance information corresponding to the successfully matched index information can be used to obtain feedback information. The success of matching a certain index information with the operator information and running environment information in the first request may be that matching information is found in the index information between the operator information and the running environment information in the first request.

When all index information in the relationship list fails to match the operator information and running environment information in the first request, it can be considered that the server cannot obtain feedback information.

In the embodiment of this application, through the relationship list, the server can quickly and efficiently query the operator performance information of the operator in the neural network model associated with the first request, and after querying the operator performance information of the operator in the neural network model When receiving information, get feedback information.

Moreover, the server can maintain and update the operator performance information stored in the server and the index information corresponding to the operator performance information.

The following is a detailed introduction to the specific maintenance and update methods of the relationship list for several scenarios.

1. In one scenario, when the server fails to query the operator performance information corresponding to the neural network model of the first request in the server for the first request, it performs the corresponding information in the server based on the second performance information subsequently uploaded by the terminal. renew.

Specifically, as shown in Figure 5, in some embodiments, the method also includes steps 504-507:

Step 504: If the server does not obtain feedback information, it stores the operator information and operating environment information of the neural network model.

The server stores the operator information and running environment information of the neural network model, thereby indicating that the server is missing relevant operator performance information.

The server stores the operator information and operating environment information of the neural network model, which can facilitate the server to maintain and update the missing relevant operator performance information. For example, it can be subsequently performed based on the second performance information and related index information sent by the terminal. In addition, you can also manually supplement this part of operator performance information and corresponding index information.

Step 505: After obtaining the second operation result, the terminal sends the second performance information to the server.

The second performance information is used to indicate the operating performance of at least one operator with respect to the corresponding second operation mode.

Step 506: The server receives the second performance information sent by the terminal.

Step 507: The server updates the server based on the second performance information, the operator information of the neural network model, and the operating environment information.

In the embodiment of the present application, the terminal fails to obtain feedback information from the server that can be used to determine the first operation mode corresponding to each of the multiple operators of the neural network model. At this time, it may be that the server lacks the neural network model. The operator performance information corresponding to the network model, or the operator performance of the neural network model in the server is incomplete. Therefore, after obtaining the second operation result, the terminal can send second performance information to the server. The second performance information is used to indicate the operating performance of all operators in the neural network model with respect to the corresponding second operation mode, or it can only indicate The operating performance of some operators in the neural network model with respect to the corresponding second operation mode. The second performance information may be operator performance information not involved in the feedback information, that is, operator performance information that the server may lack.

In this way, the server can receive the second performance information from the terminal to supplement the operator performance information missing in the server, thereby making the corresponding information in the server more complete. Specifically, the index information can be obtained according to the operator information and operating environment information of the neural network model, the operator performance information corresponding to the index information can be obtained according to the second performance information, and then the index information and the corresponding operator performance information can be used as one The group correspondence is added to the relationship list.

Of course, in some cases, the communication connection between the terminal and the server may fail, causing the terminal to be unable to receive feedback information. At this time, the terminal may not send the second performance information to the server, or the terminal may send the second performance information to the server, but the server does not update the stored operator performance information based on the second performance information.

2. In another scenario, when the operator performance information when the terminal is actually running is significantly different from the corresponding operator performance information in the relationship list, the server can calculate the operator performance information in the server based on the operator performance information when the terminal is actually running. The corresponding operator performance information is updated.

Specifically, in some embodiments, the feedback information also carries first reference information;

As shown in Figure 6, after the above step 305, the model performance optimization method also includes steps 601-603:

Step 601: After obtaining the first operation result, the terminal sends the first performance information to the server when the difference between the first performance information and the first reference information meets the preset conditions.

The first performance information is used to indicate the operating performance of at least one operator with respect to the corresponding first operation mode.

Step 602: The server receives the first performance information sent by the terminal.

Step 603: The server updates the corresponding operator performance information in the server based on the first performance information.

In this embodiment of the present application, the first reference information may indicate the expected operating performance corresponding to the corresponding operator. The expected operating performance is the expected operating performance when performing the operation operation of the corresponding operator in the first operation mode.

In this embodiment of the present application, if the difference between the first performance information and the first reference information meets the preset condition, it may indicate that the difference between the first performance information and the first reference information is relatively large.

Preset conditions can be determined based on how the performance is described.

For example, in the first performance information, the running performance corresponding to the operator is described by the corresponding running time, and in the first reference information, the expected running performance corresponding to the operator is also described by the corresponding running time, then the first performance information and The difference between the first reference information can be represented by the difference or error rate between the running time in the first performance information and the running time in the first reference information. If the difference or error rate is greater than the preset deviation threshold, Then it can be considered that the difference between the first performance information and the first reference information meets the preset conditions.

In actual scenarios, when the corresponding operator performance information of the terminal during actual operation is significantly different from the corresponding operator performance information in the relationship list, it may be that there is a large error in the relevant operator performance information stored in the server. Therefore, the terminal Sending the first performance information to the server can provide data reference for the server and promptly correct operator performance information with large errors in the server.

For example, the first reference information includes a reference running time of 3 milliseconds when the convolution operator is operated in the GPU at the first frequency, and in the terminal, when the first operation result is obtained, the GPU of the terminal operates at the first frequency. The actual running time of frequency when operating this convolution operator is 2 milliseconds. At this time, the error rate of the actual running time relative to the reference running time is 50%, which is greater than the preset deviation threshold of 20%. It can be considered that the running performance of the terminal's GPU when operating the convolution operator at the first frequency is consistent with The difference in the first reference information meets the preset conditions. Therefore, the terminal can use the GPU's running performance of the convolution operator as the first performance information and send it to the server. The server then uses the corresponding operator performance information stored in the server. Change to 2 milliseconds.

In some examples, after receiving the first performance information, the server may not immediately update the corresponding information in the server based on the first performance information. Instead, when the server receives actual performance information for the same operator from multiple terminals, Then update the corresponding operator performance information in the server. In this way, the relevant operator performance information can be authenticated multiple times through multiple terminals, and only when it is determined that the relevant operator performance information in the server has indeed a large error, the operator performance information in the server will be verified. The corresponding operator performance information is updated.

3. You can manually update the operator performance information and corresponding index information in the server.

When the chip version and/or software version in the terminal is updated, the neural network model in the application is updated, the operator's calculation method has new optimization methods, etc., the operator performance data in the server is no longer available. To meet the needs of the terminal, at this time, the operator performance data and corresponding index information stored in the server can be updated by maintenance personnel. Among them, according to the situation to be updated, test cases can be constructed and operator performance tests can be performed to collect new operator performance information, thereby updating relevant information in the server based on the collected operator performance information.

Based on any embodiment of the above model performance optimization method, the following is a detailed introduction to the internal processing flow of the terminal involved in the model performance optimization method.

In some embodiments, an application program and a target software development kit (SDK) are installed in the terminal, and the application program sends a first request to the server corresponding to the neural network model, including:

Send a second request to the target SDK through the application, and the second request contains information about the neural network model;

When the target SDK contains target configuration information, the first request is generated based on the target configuration information and the second request, and the target SDK sends the first request to the server. The target configuration information is used to indicate the neural network information that the terminal can send to the server. The information content of the model.

As shown in Figure 7, it is an exemplary schematic diagram of the information interaction process between the application program in the terminal, the target software development tool kit SDK, and the server.

In the embodiment of this application, the target configuration information is used to indicate that the function of optimizing model performance through device-cloud interaction has been enabled.

Among them, the application can enable the model performance optimization function based on device-cloud interaction in any of the above embodiments through the target SDK. At this time, the target SDK can obtain the target configuration information.

After turning on the model performance optimization function based on device-cloud interaction, the application can send a second request to the target SDK through the API provided by the target SDK. The second request includes the information of the neural network model.

The target SDK generates the first request based on the target configuration information and the neural network model information, and can send the first request to the server through the target SDK. Then, the feedback information sent by the server can be received through the target SDK, so that the terminal can perform subsequent operations based on the feedback information.

In the embodiment of this application, the target SDK can be used to provide different applications in the terminal with the function of interacting with the server to obtain an optimized neural network model, without the need to set settings in different applications during the development stage. The related functions of interacting with the server improve the efficiency of application development and facilitate the terminal to perform related operations of interacting with the server.

As above, the embodiments of the present application have introduced the model performance optimization method from many aspects. The following describes the model performance optimization device of the present application applied to the terminal and the model performance optimization device of the present application with reference to the accompanying drawings.

As shown in Figure 8, this embodiment of the present application provides a model performance optimization device 80, which can be applied to the terminal in the above embodiment.

An embodiment of the device 80 includes:

The sending module 801 is used to send a first request to the server, where the first request carries operator information of the neural network model and operating environment information of the neural network model;

The receiving module 802 is configured to receive feedback information for the first request sent by the server within the first time. The feedback information can be used to determine the first operation mode of each operator among the multiple operators of the neural network model.

Optionally, the feedback information also carries first reference information;

The sending module 801 is also used for:

After obtaining the first calculation result, when the difference between the first performance information and the first reference information meets the preset condition, the first performance information is sent to the server, where the first performance information is used to indicate at least one calculation result. sub-relationship to the operating performance of the corresponding first operation mode.

Optionally, the device 80 also includes a processing module 803;

Processing module 803 is used for:

Within the first time, no feedback information is received, or the feedback information cannot be used to determine the first operation mode of each of the multiple operators of the neural network model, then the multiple operators based on the preset neural network model The second operation mode of each operator in the operators obtains the second operation result of the data to be processed.

Optionally, the sending module 801 is also used to:

After obtaining the second operation result, second performance information is sent to the server, and the second performance information is used to indicate the operating performance of at least one operator with respect to the corresponding second operation mode.

Optionally, an application program and a target software development kit SDK are installed in the terminal, and the application program corresponds to the neural network model;

Send module 801 is used for:

Send a second request to the target SDK through the application, and the second request contains information about the neural network model;

When the target SDK contains target configuration information, the first request is generated based on the target configuration information and the second request, and the target SDK sends the first request to the server. The target configuration information is used to indicate the neural network information that the terminal can send to the server. The information content of the model.

As shown in Figure 9, this embodiment of the present application provides a model performance optimization device 90, which can be applied to the server in the above embodiment.

An embodiment of the device 90 includes:

The receiving module 901 is configured to receive the first request sent by the terminal, where the first request carries operator information of the neural network model and operating environment information of the neural network model;

The sending module 902 is used to send feedback information to the terminal. The feedback information is obtained based on the operator information and the running environment information, and the feedback information can be used to determine the first value of each operator among the multiple operators of the neural network model. Operation method.

Optionally, the device 90 also includes an update module 903;

The receiving module 901 is also used to receive the first performance information sent by the terminal, the difference between the first performance information and the first reference information meets the preset conditions, and the feedback information carries the first reference information;

The update module 903 is used to update the corresponding operator performance information in the server based on the first performance information.

Optionally, the device 90 further includes a storage module 904;

The storage module 904 is used to store the operator information and running environment information of the neural network model if the server does not obtain feedback information.

Optionally, the receiving module 901 is also configured to receive second performance information sent by the terminal, where the second performance information is used to indicate the operating performance of at least one operator with respect to the corresponding second operation mode;

The update module 903 is used to update the server according to the second performance information, the operator information of the neural network model and the running environment information.

Figure 10 is a schematic diagram of a possible logical structure of the terminal 100 provided by the embodiment of the present application. The terminal 100 is used to implement the functions of the terminal involved in any of the above embodiments. The terminal 100 includes: a memory 1001, a processor 1002, a communication interface 1003 and a bus 1004. Among them, the memory 1001, the processor 1002, and the communication interface 1003 implement communication connections between each other through the bus 1004.

The memory 1001 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1001 can store programs. When the program stored in the memory 1001 is executed by the processor 1002, the processor 1002 and the communication interface 1003 are used to perform steps 301, 304-305, 503, 505, 601 etc.

The processor 1002 may be a central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or a digital signal processor (Digital Signal Processor). signal processing (DSP), off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components or any combination thereof, for executing relevant programs to achieve the above In the embodiment, the sending module, the receiving module and the processing module in the model performance optimization device applied to the terminal need to perform functions, or perform steps 301, 304-305, 304-305, etc. of the model performance optimization method embodiment of the method embodiment of the present application. 503, 505, 601, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly implemented by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in this field. The storage medium is located in the memory 1001. The processor 1002 reads the information in the memory 1001, and performs steps 301, 304-305, 503, 505, 601, etc. of the above-mentioned model performance optimization method embodiment in conjunction with its hardware.

The communication interface 1003 uses a transceiver device such as but not limited to a transceiver to implement communication between the terminal 100 and other devices or communication networks. For example, information can be exchanged with the server through the communication interface 1003.

The bus 1004 can implement a path for transmitting information between various components of the terminal 100 (eg, the memory 1001, the processor 1002, and the communication interface 1003). The bus 1004 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one thick line is used in Figure 10, but it does not mean that there is only one bus or one type of bus.

In another embodiment of the present application, a computer-readable storage medium is also provided. Computer-executable instructions are stored in the computer-readable storage medium. When the processor of the device executes the computer-executed instructions, the device executes the above-mentioned steps in Figure 10 The steps performed by the processor.

In another embodiment of the present application, a computer program product is also provided. The computer program product includes computer-executable instructions, and the computer-executable instructions are stored in a computer-readable storage medium; when the processor of the device executes the computer-executed instructions When , the device performs the steps performed by the processor in Figure 10 above.

In another embodiment of the present application, a chip system is also provided. The chip system includes a processor, and the processor is configured to implement the steps performed by the processor in FIG. 10 . In a possible design, the chip system may also include a memory, a memory, a device for storing data writing, necessary program instructions and data. The chip system may be composed of chips, or may include chips and other discrete devices.

Figure 11 is a schematic diagram of a possible logical structure of the server 110 provided by the embodiment of the present application. The server 110 is used to implement the functions of the server involved in any of the above embodiments. The server 110 includes: a memory 1101, a processor 1102, a communication interface 1103 and a bus 1104. Among them, the memory 1101, the processor 1102, and the communication interface 1103 implement communication connections between each other through the bus 1104.

The memory 1101 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1101 can store programs. When the program stored in the memory 1101 is executed by the processor 1102, the processor 1102 and the communication interface 1103 are used to perform steps 302-303, 504, 506-507, 602-603 etc.

The processor 1102 may adopt a central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), a digital signal processor ( digital signal processor (DSP), off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any combination thereof, for executing relevant programs to Realize the functions required to be performed by the receiving module, sending module, storage module, update module, etc. in the model performance optimization device of the server in the above embodiment, or perform step 302- of the model performance optimization method embodiment of the method embodiment of the present application. 303, 504, 506-507, 602-603, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly implemented by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other mature storage media in this field. The storage medium is located in the memory 1101. The processor 1102 reads the information in the memory 1101 and performs steps 302-303, 504, 506-507, 602-603, etc. of the above-mentioned model performance optimization method embodiment in conjunction with its hardware.

The communication interface 1103 uses a transceiver device such as but not limited to a transceiver to implement communication between the server 110 and other devices or communication networks. For example, information interaction can be performed with the terminal involved in any of the above embodiments.

Bus 1104 may implement a path for transmitting information between various components of server 110 (eg, memory 1101, processor 1102, and communication interface 1103). The bus 1104 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one thick line is used in Figure 11, but it does not mean that there is only one bus or one type of bus.

In another embodiment of the present application, a computer-readable storage medium is also provided. Computer-executable instructions are stored in the computer-readable storage medium. When the processor of the device executes the computer-executed instructions, the device executes the above-mentioned steps in Figure 11 The steps performed by the processor.

In another embodiment of the present application, a computer program product is also provided. The computer program product includes computer-executable instructions, and the computer-executable instructions are stored in a computer-readable storage medium; when the processor of the device executes the computer-executed instructions When , the device performs the steps performed by the processor in Figure 11 above.

In another embodiment of the present application, a chip system is also provided. The chip system includes a processor, and the processor is configured to implement the steps performed by the processor in FIG. 11 . In a possible design, the chip system may also include a memory, a memory, a device for storing data writing, necessary program instructions and data. The chip system may be composed of chips, or may include chips and other discrete devices.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of the present application.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be described again here.

In the several embodiments provided in the embodiments of this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated. to another system, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.

A unit described as a separate component may or may not be physically separate. A component shown as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

Functions may be stored in a computer-readable storage medium when implemented in the form of software functional units and sold or used as independent products. Based on this understanding, the technical solutions of the embodiments of the present application are essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the embodiments of this application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program code. .

The above are only specific implementation modes of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto.

Claims (26)

1. A model performance optimization method, characterized in that, applied to a terminal, the method includes: Send a first request to the server, where the first request carries operator information of the neural network model and operating environment information of the neural network model; Within the first time, receive feedback information for the first request sent by the server, and the feedback information can be used to determine the first operation of each operator among multiple operators of the neural network model. Way. 2. The method of claim 1, further comprising: obtaining a first operation result of the data to be processed based on the feedback information. 3. The method according to claim 1 or 2, characterized in that the first operation mode of each operator in the plurality of operators includes the operation mode of the corresponding operator in the corresponding first calculation unit, so The first computing unit is one or more of the computing units of the terminal. 4. The method according to any one of claims 1-3, characterized in that the feedback information also carries first reference information; The method also includes: After obtaining the first operation result, when the difference between the first performance information and the first reference information meets a preset condition, the first performance information is sent to the server, wherein: The first performance information is used to indicate the operating performance of at least one of the operators with respect to the corresponding first operation mode. 5. The method according to any one of claims 1-4, characterized in that the method further includes: Within the first time, if the feedback information is not received, or the feedback information cannot be used to determine the first operation mode of each operator among the multiple operators of the neural network model, then based on the preset The second operation mode of each operator in the neural network model is used to obtain the second operation result of the data to be processed. 6. The method of claim 5, further comprising: After obtaining the second operation result, second performance information is sent to the server, where the second performance information is used to indicate the operating performance of at least one of the operators with respect to the corresponding second operation mode. 7. The method according to any one of claims 1 to 6, characterized in that an application program and a target software development kit SDK are installed in the terminal, and the application program corresponds to the neural network model; The sending of the first request to the server includes: Send a second request to the target SDK through the application program, where the second request contains information about the neural network model; When the target SDK contains target configuration information, the first request is generated based on the target configuration information and the second request, and the target SDK sends the first request to the server, so The target configuration information is used to indicate the information content about the neural network model that the terminal can send to the server. 8. The method according to any one of claims 1 to 7, characterized in that the feedback information includes operator performance information and/or model operation strategy information for the neural network model, and the operator The performance information includes information on the operating performance of the corresponding operator with respect to specified parameters, and the model operation strategy includes a first operation method for each operator in the plurality of operators of the neural network model. 9. The method according to any one of claims 1 to 8, characterized in that the neural network model is not associated with the hardware of the terminal. 10. A model performance optimization method, characterized in that it is applied to a server, and the method includes: Receive a first request sent by the terminal, where the first request carries operator information of the neural network model and operating environment information of the neural network model; Send feedback information to the terminal, where the feedback information is obtained based on the operator information and the running environment information, and the feedback information can be used to determine each of the multiple operators of the neural network model. The first operation mode of an operator. 11. The method of claim 10, further comprising: Receive the first performance information sent by the terminal, the difference between the first performance information and the first reference information meets a preset condition, and the feedback information carries the first reference information; Based on the first performance information, corresponding operator performance information in the server is updated. 12. The method of claim 10, further comprising: If the server does not obtain the feedback information, it stores the operator information and running environment information of the neural network model. 13. The method of claim 12, further comprising: Receive second performance information sent by the terminal, where the second performance information is used to indicate the operating performance of at least one of the operators with respect to the corresponding second operation mode; The server is updated according to the second performance information, the operator information and the running environment information of the neural network model. 14. A model performance optimization device, characterized in that it is applied to a terminal, and the device includes: A sending module, configured to send a first request to the server, where the first request carries operator information of the neural network model and operating environment information of the neural network model; A receiving module, configured to receive feedback information for the first request sent by the server within the first time, where the feedback information can be used to determine each of the multiple operators of the neural network model. The first mode of operation of sub. 15. The device according to claim 14, wherein the feedback information also carries first reference information; The sending module is also used to: After obtaining the first operation result, when the difference between the first performance information and the first reference information meets a preset condition, the first performance information is sent to the server, wherein: The first performance information is used to indicate the operating performance of at least one of the operators with respect to the corresponding first operation mode. 16. The device according to claim 14 or 15, characterized in that the device further comprises a processing module; The processing module is used for: Within the first time, if the feedback information is not received, or the feedback information cannot be used to determine the first operation mode of each operator among the multiple operators of the neural network model, then based on the preset The second operation mode of each operator in the neural network model is used to obtain the second operation result of the data to be processed. 17. The device according to claim 16, characterized in that the sending module is also used to: After obtaining the second operation result, second performance information is sent to the server, where the second performance information is used to indicate the operating performance of at least one of the operators with respect to the corresponding second operation mode. 18. The device according to any one of claims 14 to 17, wherein an application program and a target software development kit SDK are installed in the terminal, and the application program corresponds to the neural network model; The sending module is used for: Send a second request to the target SDK through the application program, where the second request contains information about the neural network model; When the target SDK contains target configuration information, the first request is generated based on the target configuration information and the second request, and the target SDK sends the first request to the server, so The target configuration information is used to indicate the information content about the neural network model that the terminal can send to the server. 19. A model performance optimization device, characterized in that it is applied to a server, and the device includes: A receiving module, configured to receive a first request sent by the terminal, where the first request carries operator information of the neural network model and operating environment information of the neural network model; A sending module, configured to send feedback information to the terminal, where the feedback information is obtained based on the operator information and the running environment information, and the feedback information can be used to determine multiple parameters of the neural network model. The first operation mode of each operator in the operator. 20. The device according to claim 19, characterized in that the device further comprises an update module; The receiving module is also configured to receive the first performance information sent by the terminal, the difference between the first performance information and the first reference information meets a preset condition, and the feedback information carries the first reference information; The update module is configured to update corresponding operator performance information in the server based on the first performance information. 21. The device according to claim 19, wherein the device further comprises a storage module; The storage module is used to store operator information and running environment information of the neural network model if the server does not obtain the feedback information. 22. The device according to claim 21, wherein the device further comprises an update module; The receiving module is also configured to receive second performance information sent by the terminal, where the second performance information is used to indicate the operating performance of at least one of the operators with respect to the corresponding second operation mode; The update module is configured to update the server according to the second performance information, the operator information of the neural network model and the operating environment information. 23. A terminal, characterized in that the terminal includes at least one processor, a memory and instructions stored on the memory and executable by the at least one processor, and the at least one processor executes the instructions , to implement the steps of the method described in any one of claims 1-9. 24. A computer-readable storage medium with a computer program stored thereon, characterized in that when the computer program is executed by a processor, the method of any one of claims 1-9 is implemented. 25. A server, characterized in that the server includes at least one processor, a memory and instructions stored on the memory and executable by the at least one processor, and the at least one processor executes the instructions , to implement the steps of the method described in any one of claims 10-13. 26. A computer-readable storage medium with a computer program stored thereon, characterized in that when the computer program is executed by a processor, the method of any one of claims 10-13 is implemented.