CN116450519A - DL model precision testing method, device, equipment, server and storage medium - Google Patents

Abstract

The invention discloses a DL model precision testing method, device, equipment, server and storage medium, which belong to the field of deep learning and are used for testing the precision of the deep learning model. Considering that the existing standard hardware can be used as a reference for the accurate execution results of the DL model, and the reliability of the development code for IR conversion can be reflected by the execution results of the DL model, this application can first obtain the target DL model in the specified type The execution result in the standard hardware is used as the expected result, and then the original IR of the DL compiler is converted into an IR recognizable by the artificial intelligence AI chip by using the development code to be tested to obtain an executable file, and then the executable file is placed in the AI chip The execution result of the test is used as the observation result. Through the expected result and the observation result, the reliability of the developed code to be tested can be evaluated to guide the design of the developed code, which is conducive to improving the accuracy of the DL model, that is, the DL algorithm.

Description

DL model accuracy testing method, device, equipment, server and storage medium

technical field

The present invention relates to the field of deep learning, in particular to a DL model accuracy testing method, and the present invention also relates to a DL model accuracy testing device, equipment, server and computer-readable storage medium.

Background technique

When the DL (Deep Learning, deep learning) algorithm is applied, the DL model is usually input to the DL compiler for compilation, and then the compiled content is sent to the back-end hardware for the execution of the DL algorithm. The back-end hardware usually selects CPU (Central Processing Unit, central processing unit) or GPU (Graphics Processing Unit, graphics processing unit), etc. At present, there is a new practice of using AI (Artificial Intelligence, artificial intelligence) chips as back-end hardware in order to utilize AI chips Medium and efficient AI algorithm, but it needs to develop code to convert the original IR (Intermediate Representation, intermediate representation) of the DL compiler into the IR recognizable by the AI chip, so that the AI chip can execute the DL algorithm, but different staff have designed it for conversion The development codes of IR are different, and the reliability of different development codes is also different. If the reliability of the development codes cannot be ensured, the accuracy of the DL algorithm will be seriously affected.

Therefore, how to provide a solution to the above technical problems is a problem that those skilled in the art need to solve at present.

Contents of the invention

The purpose of the present invention is to provide a DL model accuracy testing method, the reliability of the code to be tested can be evaluated through the expected result of CPU execution and the observation result of AI chip execution, so as to guide the design of reliable development code, which is beneficial to Improve the accuracy of the DL model, that is, the DL algorithm; another object of the present invention is to provide a DL model accuracy test device, equipment, server, and computer-readable storage medium, through which the expected result executed by the CPU and the observed result executed by the AI chip can be easily The reliability of the code to be tested can be evaluated, so as to guide the design of a reliable development code, which is conducive to improving the accuracy of the DL model, that is, the DL algorithm.

In order to solve the above technical problems, the present invention provides a DL model accuracy testing method, including:

The execution result of the target DL model compiled by the deep learning DL compiler in the specified type of standard hardware is taken as the expected result;

Using the code to be tested to convert the original intermediate representation IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip, and obtain an executable file;

Taking the execution result of the executable file in the AI chip as the observation result;

Evaluate the reliability of the developed code to be tested according to the expected result and the observed result.

Preferably, the DL model accuracy testing method also includes:

converting the target DL model into the original IR in the target format through the DL compiler in advance;

The execution result of the target DL model compiled by the deep learning DL compiler in the specified type of standard hardware as the expected result is specifically:

controlling the DL compiler to compile the target DL model according to the original IR of the target format;

Taking the execution result of the compiled target DL model on the specified type of standard hardware as the expected result;

The described use of the development code to be tested converts the original intermediate representation IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip, and the executable file obtained is specifically:

The original IR of the target format of the DL compiler is converted into an IR recognizable by the artificial intelligence AI chip by using the development code to be tested, and an executable file is obtained.

Preferably, the original IR that converts the target DL model into the target format through the DL compiler in advance is specifically:

Pre-select the DL model of the specified type of frame from the DL model library as the target DL model;

converting the target DL model into the original IR in target format by the DL compiler.

Preferably, the DL model library includes the DL model and the original IR of the pre-built DL model;

The pre-selected DL model of a specified type of frame from the DL model library as the target DL model is specifically:

Pre-select the DL model of the specified type of frame from the DL model library;

Judging whether the DL model of the specified type of framework already exists in the DL model library in the original IR of the pre-built DL model;

If it exists, the original IR of the target format of the DL compiler is converted into an IR recognizable by the artificial intelligence AI chip by using the development code to be tested, and the step of obtaining an executable file is performed;

If not, converting the target DL model into the original IR in target format by the DL compiler.

Preferably, the use of the development code to be tested converts the original intermediate representation IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip, and the executable file is specifically:

Select one as the development code to be tested from a plurality of candidate development codes corresponding to the framework type of the target DL model in the development code library;

Using the code to be tested to convert the original intermediate representation IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip, and obtain an executable file;

After evaluating the reliability of the developed code to be tested according to the expected result and the observed result, the DL model accuracy testing method further includes:

Judging whether there is untested development code corresponding to the framework type of the target DL model in the development code library;

If it exists, the step of selecting one as the development code to be tested from a plurality of candidate development codes corresponding to the framework type of the target DL model in the development code base is executed.

Preferably, after evaluating the reliability of the developed code to be tested according to the expected result and the observed result, the DL model accuracy testing method further includes:

The control prompter prompts the reliability evaluation result of the development code to be tested.

Preferably, after determining whether there is untested development code corresponding to the framework type of the target DL model in the development code library, the DL model accuracy testing method further includes:

If it does not exist, generate horizontal comparison information of the reliability evaluation results of all the developed codes that correspond to the framework type of the target DL model and have been tested;

The prompter is controlled to prompt the horizontal comparison information of each of the development codes corresponding to the DL compiler.

Preferably, after determining whether there is untested development code corresponding to the framework type of the target DL model in the development code library, the DL model accuracy testing method further includes:

If it does not exist, it is judged whether all the DL models in the DL model library have been tested;

If not completed, the step of selecting a DL model of a specified type of frame from the DL model library in advance is performed.

Preferably, the DL model accuracy testing method also includes:

modifying the development code specified in the development code library in response to a development code modification instruction;

Adding the specified development code to the development code library in response to the development code adding instruction.

Preferably, the DL compiler is an end-to-end deep learning compiler TVM, and the target format is JS object numbered musical notation JSON.

Preferably, the evaluating the reliability of the development code under test according to the expected result and the observed result is specifically:

Based on the chi-square test method, the reliability of the development code to be tested is evaluated according to the expected result and the observed result.

In order to solve the above technical problems, the present invention also provides a DL model accuracy testing device, including:

The first acquisition module is used to take the execution result of the target DL model compiled by the deep learning DL compiler in the specified type of standard hardware as the expected result;

The control module is used to convert the original intermediate representation IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip by using the development code to be tested, and obtain an executable file;

The second acquisition module is used to take the execution result of the executable file in the AI chip as the observation result;

An evaluation module, configured to evaluate the reliability of the developed code to be tested according to the expected result and the observed result.

In order to solve the above technical problems, the present invention also provides a DL model accuracy testing equipment, including:

memory for storing computer programs;

A processor, configured to implement the steps of the above-mentioned DL model accuracy testing method when executing the computer program.

In order to solve the above technical problems, the present invention also provides a server, including a server body and the above-mentioned DL model accuracy testing device connected to the server body.

In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned DL model accuracy testing method is implemented. A step of.

The present invention provides a DL model accuracy testing method, considering that the existing standard hardware can be used as a reference for the accurate execution result of the DL model, and the reliability of the development code for IR conversion can be reflected by the execution result of the DL model , so this application can first obtain the execution result of the target DL model in the specified type of standard hardware and take it as the expected result, and then use the code to be tested to convert the original IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip , and get the executable file, and then take the execution result of the executable file in the AI chip as the observation result, and then evaluate the reliability of the code to be tested and develop through the expected result and observation result, so as to guide the design of a reliable development code, It is conducive to improving the accuracy of the DL model, that is, the DL algorithm.

The present invention also provides a DL model accuracy testing device, equipment, server and computer-readable storage medium, which have the same beneficial effect as the above DL model accuracy testing method.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the prior art and the accompanying drawings that need to be used in the embodiments. Obviously, the accompanying drawings in the following description are only some of the present invention. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

Fig. 1 is a schematic flow chart of a DL model accuracy testing method provided by the present invention;

Fig. 2 is a schematic flow chart of another DL model accuracy testing method provided by the present invention;

Fig. 3 is a schematic structural view of a DL model accuracy testing device provided by the present invention;

Fig. 4 is a schematic structural diagram of a DL model accuracy testing device provided by the present invention;

FIG. 5 is a schematic structural diagram of a computer-readable storage medium provided by the present invention.

Detailed ways

The core of the present invention is to provide a DL model accuracy testing method, the reliability of the code to be tested can be evaluated through the expected result of CPU execution and the observation result of AI chip execution, so as to guide the design of reliable development code, which is beneficial to Improve the accuracy of the DL model, that is, the DL algorithm; another core of the present invention is to provide a DL model accuracy test device, equipment, server and computer-readable storage medium, through the expected results executed by the CPU and the observed results executed by the AI chip. The reliability of the code to be tested can be evaluated, so as to guide the design of a reliable development code, which is conducive to improving the accuracy of the DL model, that is, the DL algorithm.

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Please refer to FIG. 1. FIG. 1 is a schematic flowchart of a DL model accuracy testing method provided by the present invention. The DL model accuracy testing method includes:

S101: Taking the execution result of the target DL model compiled by the deep learning DL compiler on the specified type of standard hardware as the expected result;

Specifically, considering the technical problems in the above background technology, combined with the consideration that the accurate execution results of the DL model on the existing standard hardware (such as CPU or GPU) can be used as a standard reference, and the development code for IR conversion is reliable. The performance can be reflected by the execution results of the DL model, so this application intends to compare the "execution results obtained in the AI chip after IR conversion of the developed code to be tested" with "the execution results of the DL model by standard hardware" for evaluation The reliability of the development code to be tested, and because the execution of the DL model usually needs to be compiled by the DL compiler first, and then executed on the back-end hardware, so in this step, the target DL model compiled by the deep learning DL compiler is first , the execution result in the specified type of standard hardware is taken as the expected result, so as to serve as the data basis for the subsequent steps.

In order to better illustrate the embodiment of the present invention, please refer to FIG. 2, which is a schematic flow chart of another DL model accuracy testing method provided by the present invention. In FIG. 2, when the DL model is to be executed by the CPU, you can Set the target of the DL compiler to the target corresponding to the target ID 2, and import the JSON file corresponding to the target DL model, and the compiler can deploy the DL model to the CPU to run and get result 1 (that is, the expected result). When you want to execute the DL model through the AI chip, you can set the target of the DL compiler to the target corresponding to the target ID 1, and import the JSON file corresponding to the target DL model, and then call the development code to be tested corresponding to the target DL model that is connected to the AI chip , convert the original IR of the DL compiler into an IR recognizable by the AI chip, and obtain an executable file. The result 2 of the executable file executed on the AI chip is the observation result.

Wherein, both the target ID 1 and the target ID 2 can be independently set, for example, the target ID 1 can be set to AIPU, and the target ID 2 can be set to LLVM, etc., which are not limited in this embodiment of the present invention.

Specifically, the file in JSON format may include two parts, namely a model part and a parameter part of the target DL model, the model part may be saved as Mod.json, and the parameter part may be saved as Param.json.

Wherein, the execution type of standard hardware can be of various types, besides CPU or GPU, it can also be of various types, for example, it can be ARM processing and tensor processor, etc., which is not limited in this embodiment of the present invention.

S102: Use the development code to be tested to convert the original intermediate representation IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip, and obtain an executable file;

Specifically, in order to make the DL model compiled by the DL compiler run smoothly on the AI chip, it is necessary to develop codes to realize the operator conversion from the original IR (that is, Relay IR) of the DL compiler to the IR recognizable by the AI chip. The purpose of the embodiment of the present invention is to detect the reliability of the development code, so in this step, the development code to be tested can be used to convert the original intermediate representation IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip, and obtain an executable file and use it as the data basis for subsequent steps.

S103: Taking the execution result of the executable file in the AI chip as the observation result;

Specifically, after the executable file is obtained, the executable file can be run in the AI chip, that is, the calculation of the target DL model is executed in the AI chip, and the obtained execution result is used as the observation result, so as to be used as the next step. data base.

S104: Evaluate the reliability of the developed code to be tested according to the expected results and the observed results.

Specifically, after the expected results and observation results are obtained, since the observation results should be consistent with the expected results under ideal conditions, on the basis of this relationship, the reliability of the developed code to be tested can be evaluated according to the expected results and observation results, so that Guiding the design of the development code is conducive to improving the accuracy of the DL model executed by the AI chip.

The present invention provides a DL model accuracy testing method, considering that the existing standard hardware can be used as a reference for the accurate execution result of the DL model, and the reliability of the development code for IR conversion can be reflected by the execution result of the DL model , so this application can first obtain the execution result of the target DL model in the specified type of standard hardware and take it as the expected result, and then use the code to be tested to convert the original IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip , and get the executable file, and then take the execution result of the executable file in the AI chip as the observation result, and then evaluate the reliability of the code to be tested and develop through the expected result and observation result, so as to guide the design of a reliable development code, It is conducive to improving the accuracy of the DL model, that is, the DL algorithm.

On the basis of above-mentioned embodiment:

As a preferred embodiment, the DL model accuracy testing method also includes:

Convert the target DL model to the original IR in the target format through the DL compiler in advance;

The execution result of the target DL model compiled by the deep learning DL compiler in the specified type of standard hardware is the expected result:

Control the DL compiler to compile the target DL model according to the original IR of the target format;

The execution result of the compiled target DL model on the specified type of standard hardware is taken as the expected result;

Use the development code to be tested to convert the original intermediate representation IR of the DL compiler into the IR recognizable by the artificial intelligence AI chip, and obtain the executable file as follows:

Use the development code to be tested to convert the original IR in the target format of the DL compiler into an IR recognizable by the artificial intelligence AI chip, and obtain an executable file.

Specifically, considering that different DL models adopt different file formats when being imported into the DL compiler, this leads to a very long time-consuming when some DL models are imported into the DL compiler, which reduces the work efficiency, and for a specific DL compiler, the import speed of the DL model of a specific file format is relatively fast, so in order to improve the import efficiency, the DL compiler can be used in advance in the embodiment of the present invention Convert the target DL model to the original IR in the target format, so that when the DL compiler compiles the DL model, it can import the original IR in the target format and compile based on the original IR in the target format, which improves work efficiency .

The DL compilers can be of various types, and the corresponding target formats are also different for different DL compilers, which is not limited in this embodiment of the present invention.

As a preferred embodiment, the original IR that converts the target DL model into the target format through the DL compiler in advance is specifically:

Pre-select the DL model of the specified type of frame from the DL model library as the target DL model;

The target DL model is converted to the original IR in the target format by a DL compiler.

Specifically, in order to efficiently test the development code of DL models of various framework types, in the embodiment of the present invention, a DL model library can be pre-designed, and then a DL model of a specified type of framework can be selected from the DL model library as the target DL model, Then, the target DL model is converted into the original IR of the target format through the DL compiler, which can realize the verification of the development code of the DL model of various framework types contained in the DL model library, and improve work efficiency.

As a preferred embodiment, the DL model library includes the DL model and the original IR of the pre-built DL model;

Pre-select the DL model of the specified type of frame from the DL model library as the target DL model, specifically:

Pre-select the DL model of the specified type of frame from the DL model library;

Determine whether the DL model of the specified type of framework already has the original IR of the pre-built DL model in the DL model library;

If it exists, then perform the steps of converting the original IR of the target format of the DL compiler into an IR recognizable by the artificial intelligence AI chip by using the development code to be tested, and obtaining an executable file;

If it does not exist, the target DL model is converted to the original IR in the target format by the DL compiler.

Specifically, considering that in addition to some existing complete DL models that need to be detected, for some uncommon DL models, it is also necessary to design and develop codes and detect the developed codes, and the function of the developed codes is to convert the DL model The original IR is converted into the IR recognizable by the AI chip. Therefore, for the uncommon DL model, the embodiment of the present invention does not construct a complete DL model, but pre-builds a part of the complete DL model: "DL model original IR ", which not only realizes the test of the development code for "uncommon DL models", but also directly reduces the workload.

Specifically, when determining the target DL model, both the complete DL model in the DL model library and the DL model corresponding to the original IR of the pre-built DL model can be selected as the target DL model. After selecting the DL model of the specified type of frame in the DL model library, judge whether the DL model of the specified type of frame already exists in the DL model library. The original IR of the pre-built DL model, if not, can convert the target DL model through the DL compiler The original IR in the target format, if it exists, can directly use the development code to be tested to convert the original IR in the target format of the DL compiler into an IR recognizable by the artificial intelligence AI chip, and obtain an executable file.

Among them, there can be various DL models, such as Tensorflow, Pytorch, onnx, PaddlePaddle, Caffe, shufflenet, and faster rcnn, etc., which are not limited in this embodiment of the present invention.

As a preferred embodiment, use the code to be tested to convert the original intermediate representation IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip, and obtain the executable file as follows:

Select one of the multiple candidate development codes corresponding to the framework type of the target DL model in the development code library as the development code to be tested;

Use the development code to be tested to convert the original intermediate representation IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip, and obtain an executable file;

After evaluating the reliability of the developed code to be tested according to the expected results and observation results, the DL model accuracy testing method also includes:

Determine whether there is untested development code corresponding to the framework type of the target DL model in the development code base;

If it exists, the step of selecting one as the development code to be tested from among multiple candidate development codes corresponding to the frame type of the target DL model in the development code library is performed.

Specifically, considering that there may be multiple types of development codes for the DL model of the same framework type, it is necessary to test the various development codes corresponding to the same framework type. Therefore, in order to improve work efficiency, the embodiment of the present invention In the development code library, one can be selected from multiple candidate development codes corresponding to the framework type of the target DL model as the development code to be tested, and then the original intermediate representation IR of the DL compiler can be converted into artificial The smart AI chip can recognize the IR and get the executable file. Finally, after getting the reliability of the development code to be tested, it can also judge whether there is untested development code corresponding to the framework type of the target DL model in the development code base , if it exists, you can continue to select one of the multiple candidate development codes corresponding to the framework type of the target DL model in the development code base as the development code to be tested, so as to automatically implement multiple different development codes corresponding to the same framework type. The testing process of developing code improves the degree of automation and work efficiency.

As a preferred embodiment, after evaluating the reliability of the developed code to be tested according to the expected results and observation results, the DL model accuracy testing method further includes:

The control prompter prompts the reliability evaluation result of the developed code to be tested.

Specifically, in order to facilitate the staff to know the reliability evaluation result of the development code to be tested in time, the embodiment of the present invention may control the prompter to prompt the reliability evaluation result of the development code to be tested.

Wherein, the prompter can be of various types, for example, it can be a display, etc., which is not limited in this embodiment of the present invention.

As a preferred embodiment, after determining whether there is untested development code corresponding to the framework type of the target DL model in the development code library, the DL model accuracy testing method further includes:

If it does not exist, generate horizontal comparison information of the reliability evaluation results of all developed codes corresponding to the framework type of the target DL model and tested;

The control prompter prompts the horizontal comparison information of each development code corresponding to the DL compiler.

Specifically, considering that the staff may need to make a horizontal comparison of the reliability evaluation results of multiple development codes corresponding to the DL model of the same framework type, in order to improve work efficiency and user experience, in the embodiment of the present invention, the development code library can After there is no untested development code corresponding to the framework type of the target DL model, generate horizontal comparison information of the reliability evaluation results of all tested development codes corresponding to the framework type of the target DL model, and control the prompter Prompt the horizontal comparison information of each development code corresponding to the DL compiler.

Wherein, the horizontal comparison information may include various contents, for example, it may include reliability ranking of each development code to be tested, etc., which is not limited in this embodiment of the present invention.

As a preferred embodiment, after determining whether there is untested development code corresponding to the framework type of the target DL model in the development code library, the DL model accuracy testing method further includes:

If it does not exist, it is judged whether all the DL models in the DL model library have been tested;

If not completed, perform the step of selecting a DL model of a specified type of frame from the DL model library in advance.

Specifically, in order to realize automated reliability testing of development codes corresponding to multiple DL models contained in the DL model library, in the embodiment of the present invention, it may be determined that there is no code corresponding to the framework type of the target DL model in the development code library. When developing code without testing, it is judged whether all DL models in the DL model library have been tested. If not, the step of selecting a DL model of a specified type of frame from the DL model library in advance can be performed to further improve the degree of automation and work efficiency.

As a preferred embodiment, the DL model accuracy testing method also includes:

modifying the development code specified in the development code base in response to the development code modification instruction;

In response to the development code adding instruction, the specified development code is added to the development code base.

Specifically, considering that there is a need for the staff to modify the development code existing in the development code base or add new development code to the development code base, these two interfaces are provided in the embodiment of the present invention, which can respond to the development The code modification instruction modifies the development code specified in the development code library, and may also respond to the development code adding instruction to add the specified development code to the development code library.

As a preferred embodiment, the DL compiler is an end-to-end deep learning compiler TVM, and the target format is JS object numbered musical notation JSON.

Specifically, TVM is a mature and efficient DL compiler, and its import speed for JSON format files is relatively fast, so JSON is selected as the target format.

Of course, except for this specific case, the DL compiler and its corresponding target format may also be of other types, which are not limited in this embodiment of the present invention.

As a preferred embodiment, evaluating the reliability of the development code to be tested according to the expected results and observation results is specifically as follows:

Based on the chi-square test method, the reliability of the developed code to be tested is evaluated according to the expected results and the observed results.

Specifically, considering that the expected results and observation results conform to the chi-square test standard, the development detection method can be used in the embodiment of the present invention to evaluate the reliability of the development code to be tested according to the expected results and observation results, which can be specifically:

Among them, x ² is the detection value of the chi-square test, f0 is the observed result, and f _e is the expected result.

Among them, when the detection value of chi-square detection is not greater than the preset threshold, the error comes from the effective number, and at this time it can be confirmed that the operator of the development code to be tested is implemented correctly. If the detection value is greater than the preset threshold, the problem needs to be confirmed at the model stage The operator then corrects.

Specifically, the preset threshold can be set independently, for example, it can be e ^-5 , etc., which is not limited in this embodiment of the present invention.

In order to better describe the embodiment of the present invention, please refer to Table 1 below, which is a table of Shufflenet model accuracy test results.

Table 1

Please refer to FIG. 3. FIG. 3 is a schematic structural diagram of a DL model accuracy testing device provided by the present invention. The DL model accuracy testing device includes:

The first acquisition module 31 is used to take the execution result of the target DL model compiled by the deep learning DL compiler in a specified type of standard hardware as the expected result;

The control module 32 is used to convert the original intermediate representation IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip by using the development code to be tested, and obtain an executable file;

The second acquisition module 33 is used to take the execution result of the executable file in the AI chip as the observation result;

An evaluation module 34, configured to evaluate the reliability of the developed code to be tested according to expected results and observation results.

The present invention provides a DL model accuracy test device, considering that the existing standard hardware can be used as a reference for the accurate execution result of the DL model, and the reliability of the development code for IR conversion can be reflected by the execution result of the DL model , so this application can first obtain the execution result of the target DL model in the specified type of standard hardware and take it as the expected result, and then use the code to be tested to convert the original IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip , and get the executable file, and then take the execution result of the executable file in the AI chip as the observation result, and then evaluate the reliability of the code to be tested and develop through the expected result and observation result, so as to guide the design of a reliable development code, It is conducive to improving the accuracy of the DL model, that is, the DL algorithm.

For the introduction of the DL model accuracy testing device provided by the embodiment of the present invention, please refer to the aforementioned embodiment of the DL model accuracy testing method, and the embodiment of the present invention will not be repeated here.

Please refer to FIG. 4. FIG. 4 is a schematic structural diagram of a DL model accuracy testing device provided by the present invention. The DL model accuracy testing device includes:

Memory 41, used to store computer programs;

The processor 42 is configured to implement the steps of the method for testing the accuracy of the DL model in the foregoing embodiments when executing the computer program.

Specifically, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer-readable instructions, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. When the processor executes the computer program stored in the memory, the following steps can be implemented: the execution result of the target DL model compiled by the deep learning DL compiler in the specified type of standard hardware is used as the expected result; The original intermediate representation IR of the DL compiler is converted into the IR recognizable by the artificial intelligence AI chip, and the executable file is obtained; the execution result of the executable file in the AI chip is taken as the observation result; the development under test is evaluated according to the expected result and the observation result code reliability.

The present invention provides a DL model accuracy testing device, considering that the existing standard hardware can be used as a reference for the accurate execution result of the DL model, and the reliability of the development code for IR conversion can be reflected by the execution result of the DL model , so this application can first obtain the execution result of the target DL model in the specified type of standard hardware and take it as the expected result, and then use the code to be tested to convert the original IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip , and get the executable file, and then take the execution result of the executable file in the AI chip as the observation result, and then evaluate the reliability of the code to be tested and develop through the expected result and observation result, so as to guide the design of a reliable development code, It is conducive to improving the accuracy of the DL model, that is, the DL algorithm.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps may be implemented: converting the target DL model into the original IR in the target format through a DL compiler in advance;

The execution result of the target DL model compiled by the deep learning DL compiler in the specified type of standard hardware is the expected result:

Control the DL compiler to compile the target DL model according to the original IR of the target format;

The execution result of the compiled target DL model on the specified type of standard hardware is taken as the expected result;

Use the development code to be tested to convert the original intermediate representation IR of the DL compiler into the IR recognizable by the artificial intelligence AI chip, and obtain the executable file as follows:

Use the development code to be tested to convert the original IR in the target format of the DL compiler into an IR recognizable by the artificial intelligence AI chip, and obtain an executable file.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps may be implemented: pre-selecting a DL model of a specified type of frame from the DL model library as the target DL model;

The target DL model is converted to the original IR in the target format by a DL compiler.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps can be implemented: the DL model library includes the DL model and the original IR of the pre-built DL model;

Pre-select the DL model of the specified type of frame from the DL model library as the target DL model, specifically:

Pre-select the DL model of the specified type of frame from the DL model library;

Determine whether the DL model of the specified type of framework already has the original IR of the pre-built DL model in the DL model library;

If it exists, then perform the steps of converting the original IR of the target format of the DL compiler into an IR recognizable by the artificial intelligence AI chip by using the development code to be tested, and obtaining an executable file;

If it does not exist, the target DL model is converted to the original IR in the target format by the DL compiler.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps can be implemented: select one of the multiple candidate development codes corresponding to the framework type of the target DL model in the development code base As the development code under test;

Use the development code to be tested to convert the original intermediate representation IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip, and obtain an executable file;

After evaluating the reliability of the developed code to be tested according to the expected results and observation results, the DL model accuracy testing method also includes:

Determine whether there is untested development code corresponding to the framework type of the target DL model in the development code base;

If it exists, the step of selecting one as the development code to be tested from among multiple candidate development codes corresponding to the frame type of the target DL model in the development code library is performed.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps may be implemented: the control prompter prompts the reliability evaluation result of the developed code to be tested.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps can be implemented: If it does not exist, generate reliability evaluations of all tested development codes corresponding to the framework type of the target DL model Horizontal comparison information of the results;

The control prompter prompts the horizontal comparison information of each development code corresponding to the DL compiler.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps can be implemented: if it does not exist, then judge whether all the DL models in the DL model library have been tested;

If not completed, perform the step of selecting a DL model of a specified type of frame from the DL model library in advance.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps may be implemented: modify the development code specified in the development code library in response to the development code modification instruction;

In response to the development code adding instruction, the specified development code is added to the development code base.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps can be implemented: the DL compiler is an end-to-end deep learning compiler TVM, and the target format is JS object numbered musical notation JSON.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps may be implemented: based on the chi-square test method, evaluating the reliability of the developed code to be tested according to the expected result and the observed result.

For the introduction of the DL model accuracy testing equipment provided by the embodiment of the present invention, please refer to the aforementioned embodiment of the DL model accuracy testing method, and the embodiments of the present invention will not be repeated here.

The present invention also provides a server, including a server body and the above DL model accuracy testing device connected to the server body.

The present invention provides a server, considering that the existing standard hardware can be used as a reference for the accurate execution results of the DL model, and the reliability of the developed code for IR conversion can be reflected by the execution results of the DL model, so the present application The execution result of the target DL model in the specified type of standard hardware can be obtained first as the expected result, and then the original IR of the DL compiler can be converted into an IR recognizable by the artificial intelligence AI chip by using the development code to be tested, and the identifiable IR can be obtained. Execute the file, and then take the execution result of the executable file in the AI chip as the observation result. The reliability of the code to be tested can be evaluated through the expected result and the observation result, so as to guide the design of a reliable development code, which is conducive to improving DL The model is also the accuracy of the DL algorithm.

For the introduction of the server provided by the embodiment of the present invention, please refer to the aforementioned embodiment of the DL model accuracy testing method, and the embodiment of the present invention will not be repeated here.

Please refer to FIG. 5. FIG. 5 is a schematic structural diagram of a computer-readable storage medium provided by the present invention. A computer program 51 is stored on the computer-readable storage medium 50. When the computer program 51 is executed by the processor 41, the aforementioned implementation The steps of the DL model accuracy test method in the example.

Specifically, the readable storage medium may include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, etc., which can store various programs. The medium of the code. A computer program is stored on the storage medium, and when the computer program is executed by the processor, the following steps are implemented: the execution result of the target DL model compiled by the deep learning DL compiler in a specified type of standard hardware is taken as the expected result; The test and development code converts the original intermediate representation IR of the DL compiler into the IR recognizable by the artificial intelligence AI chip, and obtains the executable file; the execution result of the executable file in the AI chip is taken as the observation result; according to the expected result and the observation result Assess the reliability of the developed code under test.

The present invention provides a computer-readable storage medium, considering that the existing standard hardware can be used as a reference for the accurate execution results of the DL model, and the reliability of the developed code for IR conversion can be reflected by the execution results of the DL model , so this application can first obtain the execution result of the target DL model in the specified type of standard hardware and take it as the expected result, and then use the code to be tested to convert the original IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip , and get the executable file, and then take the execution result of the executable file in the AI chip as the observation result, and then evaluate the reliability of the code to be tested and develop through the expected result and observation result, so as to guide the design of a reliable development code, It is conducive to improving the accuracy of the DL model, that is, the DL algorithm.

For the introduction of the computer-readable storage medium provided by the embodiment of the present invention, please refer to the foregoing embodiment of the DL model accuracy testing method, and the embodiments of the present invention will not be repeated here.

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for relevant details, please refer to the description of the method part. It should also be noted that in this specification, relative terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations There is no such actual relationship or order between the operations. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article or apparatus comprising that element.

The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (15)

1. A DL model accuracy testing method, is characterized in that, comprises: The execution result of the target DL model compiled by the deep learning DL compiler in the specified type of standard hardware is taken as the expected result; Using the code to be tested to convert the original intermediate representation IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip, and obtain an executable file; Taking the execution result of the executable file in the AI chip as the observation result; Evaluate the reliability of the developed code to be tested according to the expected result and the observed result. 2. The DL model accuracy testing method according to claim 1, wherein the DL model accuracy testing method further comprises: converting the target DL model into the original IR in the target format through the DL compiler in advance; The execution result of the target DL model compiled by the deep learning DL compiler in the specified type of standard hardware as the expected result is specifically: controlling the DL compiler to compile the target DL model according to the original IR of the target format; Taking the execution result of the compiled target DL model on the specified type of standard hardware as the expected result; The described use of the development code to be tested converts the original intermediate representation IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip, and the executable file obtained is specifically: The original IR of the target format of the DL compiler is converted into an IR recognizable by the artificial intelligence AI chip by using the development code to be tested, and an executable file is obtained. 3. The DL model accuracy testing method according to claim 2, wherein the original IR that converts the target DL model into the target format by the DL compiler in advance is specifically: Pre-select the DL model of the specified type of frame from the DL model library as the target DL model; converting the target DL model into the original IR in target format by the DL compiler. 4. The DL model accuracy testing method according to claim 3, wherein the DL model library includes DL models and pre-built DL model original IRs; The pre-selected DL model of a specified type of frame from the DL model library as the target DL model is specifically: Pre-select the DL model of the specified type of frame from the DL model library; Judging whether the DL model of the specified type of framework already exists in the DL model library in the original IR of the pre-built DL model; If it exists, the original IR of the target format of the DL compiler is converted into an IR recognizable by the artificial intelligence AI chip by using the development code to be tested, and the step of obtaining an executable file is performed; If not, converting the target DL model into the original IR in target format by the DL compiler. 5. The DL model accuracy testing method according to claim 4, characterized in that, the original intermediate representation IR of the DL compiler is converted into the recognizable IR of the artificial intelligence AI chip by using the development code to be tested, and obtained Executable files are: Select one as the development code to be tested from a plurality of candidate development codes corresponding to the framework type of the target DL model in the development code library; Using the code to be tested to convert the original intermediate representation IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip, and obtain an executable file; After evaluating the reliability of the developed code to be tested according to the expected result and the observed result, the DL model accuracy testing method further includes: Judging whether there is untested development code corresponding to the framework type of the target DL model in the development code library; If it exists, the step of selecting one as the development code to be tested from a plurality of candidate development codes corresponding to the framework type of the target DL model in the development code base is executed. 6. The DL model accuracy testing method according to claim 5, characterized in that, after evaluating the reliability of the development code to be tested according to the expected results and the observation results, the DL model accuracy testing method further include: The control prompter prompts the reliability evaluation result of the development code to be tested. 7. The DL model accuracy testing method according to claim 6, characterized in that, after determining whether there is untested development code corresponding to the frame type of the target DL model in the described development code base, the DL model accuracy testing methods also include: If it does not exist, generate horizontal comparison information of the reliability evaluation results of all the developed codes that correspond to the framework type of the target DL model and have been tested; The prompter is controlled to prompt the horizontal comparison information of each of the development codes corresponding to the DL compiler. 8. The DL model accuracy testing method according to claim 7, characterized in that, after determining whether there is untested development code corresponding to the framework type of the target DL model in the described development code base, the DL model accuracy testing methods also include: If it does not exist, it is judged whether all the DL models in the DL model library have been tested; If not completed, the step of selecting a DL model of a specified type of frame from the DL model library in advance is performed. 9. The DL model accuracy testing method according to claim 5, wherein the DL model accuracy testing method further comprises: modifying the development code specified in the development code library in response to a development code modification instruction; Adding the specified development code to the development code library in response to a development code adding instruction. 10. The DL model accuracy testing method according to claim 2, wherein the DL compiler is an end-to-end deep learning compiler TVM, and the target format is JS object numbered musical notation JSON. 11. The DL model accuracy testing method according to any one of claims 1 to 10, wherein the evaluation of the reliability of the development code to be tested according to the expected result and the observed result is specifically: Based on the chi-square test method, the reliability of the development code to be tested is evaluated according to the expected result and the observed result. 12. A DL model accuracy testing device, characterized in that, comprising: The first acquisition module is used to take the execution result of the target DL model compiled by the deep learning DL compiler in the specified type of standard hardware as the expected result; The control module is used to convert the original intermediate representation IR of the DL compiler into an IR recognizable by the artificial intelligence AI chip by using the development code to be tested, and obtain an executable file; The second acquisition module is used to take the execution result of the executable file in the AI chip as the observation result; An evaluation module, configured to evaluate the reliability of the developed code to be tested according to the expected result and the observed result. 13. A DL model accuracy testing device, characterized in that, comprising: memory for storing computer programs; A processor, configured to implement the steps of the DL model accuracy testing method according to any one of claims 1 to 11 when executing the computer program. 14. A server, characterized by comprising a server body and the DL model accuracy testing device according to claim 13 connected to the server body. 15. A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the DL model according to any one of claims 1 to 11 is realized The steps of the accuracy test method.