CN107590037A - A kind of method that EDPP tests are carried out to server GPU - Google Patents

Abstract

The invention discloses a method for performing an EDPP test on a server GPU. The method comprises the steps of: building a test environment; performing multiple EDPP tests on multiple GPUs at the same time, and restarting the test program for each test; recording the test process and results . Compared with the prior art, the test process of the present invention does not require manual participation, which enhances the reliability of test results; the test realizes EDPP test on multiple GPUs at the same time, which greatly shortens the time used for testing and improves work efficiency.

Description

A method for EDPP testing of server GPU

technical field

The invention relates to the technical field of server GPU testing, in particular to a method for performing EDPP testing on server GPUs.

Background technique

Computing server - PCIE-SWITH (PCIE, full name peripheral component interconnect express, is a high-speed serial computer expansion bus standard; SWITH, is a server) is a server with a new architecture, the server in China The field of cabinet servers can meet customers' needs for high-performance computing servers.

However, in the production and development stage of the server, the testing of its graphics processing unit (English: Graphics Processing Unit, abbreviation: GPU) is an important link. Due to the large number of test items and the length of each test item during the test, a single loop EDPP (EDPP, English full name is electrical date peak processing, power supply data peak processing) test takes about 3 hours, and it is multi-node At the same time, the required test items are continued, so the workload is huge.

Existing testing methods are all manual testing and each GPU is tested one by one. It takes a long time to use and requires manual selection, which increases the workload of the staff. The work efficiency is low, and manual selection will inevitably lead to errors and omissions.

Contents of the invention

In order to overcome the deficiencies in the above-mentioned prior art, the purpose of the present invention is to provide a method for performing EDPP testing on server GPUs with a high degree of automation and high testing efficiency.

The technical scheme that the present invention adopts to solve the technical problem is: a kind of method that server GPU is carried out EDPP test, comprises the following steps:

Set up a test environment;

Perform multiple EDPP tests on multiple GPUs at the same time, and restart the test program for each test;

Record the test process and results.

Further, the specific process of setting up the test environment is: make the GPUBOX fully configured, connect the GPUBOX of PCIE-SWITCH to the test server through Mini SAS HD, and copy the graphics card driver and test tools to the operating system on the server.

Further, the described multiple EDPP tests are carried out simultaneously to multiple GPUs, and the specific process of restarting the test program for each test is as follows:

Enter the test node IP and test times variables;

Judging whether the test times variable is within the preset test times range;

If the number of tests variable is not within the preset number of times, then end the test;

If the number of tests variable is within the preset number of times, it is judged whether the number of tests variable is 1;

If the variable of the number of tests is not 1, then close the running test program and proceed to the next step; if the variable of the number of tests is 1, then call the graphics card driver matched with the test tool to proceed to the next step;

Log in to the test node, make each test node start the test program at the same time, select the EDPP test item to test, and output the test result, add 1 to the test times variable, and repeat the test.

Further, the specific steps of calling the graphics card driver matched with the test tool are:

Execute the driver version check command to judge whether the existing graphics card driver is the driver required for the test according to the driver version number;

If it is the driver required for testing, directly call the graphics card driver;

If it is not the driver required for the test, uninstall the graphics card driver and call the copied graphics card driver.

Further, log in the test node through the ssh service, and record the output information, the output information includes one or more information in the load, number, GPU usage, power consumption, frequency, temperature and state of each GPU, and Recorded through the log file.

Further, the test result and the output information are recorded in the same log file.

The beneficial effects of the present invention are:

1. The present invention fully configures the entire PCIE-SWITH cabinet, and performs EDPP testing on GPUs at the same time with multiple nodes. Simultaneous testing of 32 GPUs is realized in a single cycle, which greatly reduces the testing time and improves work efficiency;

2. The present invention records the test results and GPU information into the same log file, which facilitates subsequent viewing of GPU performance and accurate positioning of problems, and avoids manual testing and mistakes in GPU selection.

3. In the method of the present invention, the number of tests is set to multiple times, and the test program is restarted for each test, so as to avoid the impact of the last test on this test and ensure the accuracy and reliability of each test result.

Description of drawings

Fig. 1 is a schematic flow chart of the method of the present invention.

FIG. 2 is a schematic flow diagram of performing multiple EDPP tests simultaneously on a fully configured GPU in the present invention.

detailed description

In order to clearly illustrate the technical features of this solution, the present invention will be described in detail below through specific implementation modes and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. To simplify the disclosure of the present invention, components and arrangements of specific examples are described below. Furthermore, the present invention may repeat reference numerals and/or letters in different instances. This repetition is for the purpose of simplicity and clarity and does not in itself indicate a relationship between the various embodiments and/or arrangements discussed. It should be noted that components illustrated in the figures are not necessarily drawn to scale. Descriptions of well-known components and processing techniques and processes are omitted herein to avoid unnecessarily limiting the present invention.

In the present invention, the PCIE-SWITH whole cabinet is fully equipped and connected to the test server, and the GPU is called by the test tool on the test server side, so that the GPU of the whole cabinet can perform EDPP test at the same time.

Wherein test tool is the test tool that GPU is carried out EDPP test, and the Nvqual test tool that the present invention selects is NVQual_P40_v4_18, uses the test tool in this test tool to need collocation driver to use, and the driver program that the present invention selects is P40gpu:driver_367. 57.

As shown in Figure 1, the concrete operation steps of this method are,

S1, build a test environment;

S2, performing multiple EDPP tests on multiple GPUs at the same time, and restarting the test program for each test;

S3, recording the testing process and results.

In step S1, the setup of the test environment includes the setup of the hardware test environment and the setup of the software test environment. The setup of the hardware test environment refers to fully configuring the GPUBOX of the two PCIE-SWITCH complete cabinets with the test server (server) Connect with Mini SAS HD cable. The full configuration refers to the installation of 16 GPUs on a single GPUBOX. One GPUBOX corresponds to two servers. Among them, GPUBOX is the box where the GPU is installed. The construction of the software environment is realized by copying the graphics card driver and test tools to the operating system on the server side. At the beginning of the test, first turn on the GPUBOX, and turn on the corresponding server after 1 minute to ensure the stability of the test system.

As shown in Figure 2, the specific steps for performing multiple EDPP tests on a fully configured GPU at the same time in step S2 are:

S21, input test node IP and test times variable;

S22, judging whether the test times variable is within the preset test times range;

S23, if the test number variable is not within the preset number range, then end the test;

S24, if the test times variable is within the preset times range, then judge whether the test times variable is 1;

S25, if the number of tests variable is not 1, then close the running test program;

S26, log in the test node, make each test node start the test program at the same time, select the EDPP test item to test, and output the test result, add 1 to the test times variable, and go to step S22;

S27, if the number of tests variable is 1, then determine whether the existing graphics card driver is the driver required for the test;

S28, if it is the driver required for the test, then directly call the graphics card driver, and go to step S25;

S29. If the driver is not required for the test, uninstall the graphics card driver, call the copied graphics card driver, and go to step S25.

In step S21, input the test node IP and test times variables in the command input line of the test program. The present invention inputs 4 test node IPs, wherein each test node IP corresponds to 8 GPUs, so the present invention performs EDPP test on 32 GPUs at the same time. In step S22, the preset number of times is multiple times, and the present invention selects 10 times, and the preset number of times is the number of repeated tests for every 32 GPUs. The test results obtained by multiple tests are more accurate and more reliable. The time taken for one test is about 2.5 hours, compared with the traditional three-hour test for one GPU, which greatly improves the work efficiency.

In step S24, the judgment of whether the number of tests is 1 is to further check and judge whether there is a built-in graphics card driver under the server operating system. If there is a built-in graphics card driver, uninstall the built-in graphics card driver and use the built-in test environment The newly copied video card driver. This is because the graphics card driver that comes with the operating system cannot drive the existing test tools, so the EDPP test of the GPU cannot be performed.

In step S25, in order to ensure the stability of the test process, first close the running test tool, so that the test node of each test restarts the test program, so as to avoid the impact that the test program of the last test did not close on this test. Guarantee the accuracy of each test result.

In step S26, log in to the test node through the ssh service, record and output information, and the output information includes the load, number, GPU usage, power consumption, frequency, temperature, and status of each GPU. And record through the log file, the following is an example of the output content of 8 GPUs:

Current Application Clocks: [1531MHz]

sgemm Workload-M:3584N:1024K:4096L:100000I:10000P:X:T:60-PASS

index, timestamp, utilization.gpu[%], power.draw[W], clocks.current.graphics[MHz], temperature.gpu, pstate, clocks_throttle_reasons.hw_slowdown

0,2017/04/01 16:27:42.062,0%,14.00W,544MHz,31,P8,Not Active

1,2017/04/01 16:27:42.077,0%,13.19W,544MHz,30,P8,Not Active

2,2017/04/01 16:27:42.091,0%,12.39W,544MHz,30,P8,Not Active

3,2017/04/01 16:27:42.106,0%,13.38W,544MHz,30,P8,Not Active

4,2017/04/01 16:27:42.121,0%,12.89W,544MHz,31,P8,Not Active

5,2017/04/01 16:27:42.136,0%,12.99W,544MHz,30,P8,Not Active

6,2017/04/01 16:27:42.152,0%,13.40W,544MHz,29,P8,Not Active

7, 2017/04/01 16:27:42.167, 0%, 12.79W, 544MHz, 33, P8, Not Active.

In step S26, the EDPP test item of the Nvqual test tool is selected to test the capability of the GPU to supply power to a peak value under fluctuating load pressure. If the test is passed, the test is output as Pass, otherwise the test is output as Fail. Record the test results and the output information of the GPU in the same log file, which is convenient for subsequent viewing of GPU performance and accurate positioning of problems.

In step S27, the judgment of whether the existing graphics card driver is the driver required for the test is realized according to the version number of the driver program by executing the driver version checking command. Wherein the driver version inspection command is check_driver="nvidia-smi|grep-i version", wherein nvidia-smi is an instruction to grab current GPU information, and grep-iversion is an instruction to grab the driver version in the area information, among the present invention , the version of the graphics card driver matching the GPU test is Tesla P40, and the model is 367.57.

During the EDPP test, the pressure is turned on on the test nodes at the same time, so that the test nodes reach the maximum value at the same time, where the pressure is the ability of the GPU to transmit and process various data blocks of different sizes per unit time.

In step S3, the recording of the test process and results is to record the test results and the output information of the GPU through a log file.

The above is only a preferred embodiment of the present invention. For those of ordinary skill in the art, without departing from the principle of the present invention, some improvements and modifications can also be made, and these improvements and modifications are also considered as the present invention. protection scope of the invention.

Claims (6)

1. a kind of method that EDPP tests are carried out to server GPU, it is characterized in that, comprise the following steps：

Build test environment；

Multiple EDPP tests are carried out simultaneously to multiple GPU, test program is restarted in test every time；

Test process and result are recorded.

2. a kind of method that EDPP tests are carried out to server GPU according to claim 1, it is characterized in that, it is described to build The detailed process of test environment is：Make GPUBOX full configurations, PCIE-SWITCH GPUBOX is surveyed by Mini SAS HD connections Server end is tried, and copies video driver and testing tool to the operating system of server end.

3. a kind of method that EDPP tests are carried out to server GPU according to claim 1, it is characterized in that, it is described to more Individual GPU carries out multiple EDPP tests simultaneously, and the detailed process that test program is restarted in test every time is：

Input test node IP and testing time variable；

Judge testing time variable whether in the range of default testing time；

If testing time variable not in the range of preset times, terminates to test；

If testing time variable in the range of preset times, judges whether testing time variable is 1；

If testing time variable is not 1, the test program being currently running is closed, carries out the operation of next step；If test Degree variables are 1, then call the video driver matched with testing tool, carry out the operation of next step；

Test node is logined, makes each test node while starts test program, selects EDPP test items to be tested, and export survey Test result, testing time variable is set to add 1, retest.

4. a kind of method that EDPP tests are carried out to server GPU according to claim 3, it is characterized in that, the calling The video driver matched with testing tool concretely comprises the following steps：

Driver version inspection order is performed, judges whether existing video driver is test institute according to the version number of driver Need to drive；

If being driven needed for test, then the video driver is directly invoked；

If not driving needed for test, then the video driver is unloaded, calls the video driver of copy.

5. a kind of method that EDPP tests are carried out to server GPU according to claim 4, it is characterized in that, taken by ssh Test node is logined in business, and records output information, load of the output information including each GPU, numbering, GPU utilization rates, work( One or more information in consumption, frequency, temperature and state, and pass through log file records.

6. a kind of method that EDPP tests are carried out to server GPU according to claim 5, it is characterized in that, the test As a result it is recorded in output information in same log files.