CN108763043A - Acquisition methods, the apparatus and system of pressure testing data in a kind of storage device - Google Patents

Abstract

The present application discloses a method for acquiring stress test data in a storage device, which includes regularly collecting target stress test data from the storage device through a data collection script; saving the target stress test data into a data file; The target stress test data is analyzed in the data file to obtain the stress test data. The acquisition method can more quickly acquire relevant data generated by the storage device during the test process, and accurately analyze it, effectively improving work efficiency and data accuracy, and also greatly reducing the test cost of the storage system. The present application also discloses a device, system and computer-readable storage medium for acquiring stress test data in a storage device, all of which have the above beneficial effects.

Description

A method, device and system for acquiring stress test data in a storage device

technical field

The present application relates to the field of distributed storage systems, in particular to a method for acquiring stress test data in a storage device, and also relates to an acquisition device, system and computer-readable storage medium for stress test data in a storage device.

Background technique

With the continuous development of information technology, data is gradually being valued as a valuable resource. How to quickly process data resources and obtain expected results has become one of the key issues in the transformation from resources to assets. Various activities in people's work and life will generate data. Collecting these data and then analyzing and processing them can obtain useful information and realize the transformation of resources into assets, thus catalyzing the rapid development of big data and high-performance computing. As one of the core elements of data resources, data storage has also ushered in a period of rapid development.

Traditional network storage systems use centralized storage servers to store all data. With the development of big data, storage servers have become the bottleneck of system performance and the focus of reliability and security, which are increasingly unable to meet the needs of large-scale storage applications. . The distributed network storage system adopts an expandable system structure, which not only improves the reliability, availability and access efficiency of the system, but also is easy to expand, thus being accepted and recognized by more and more enterprise units.

A distributed storage system generally consists of 1 to N nodes to provide high performance and massive data storage. In the actual development and testing environment, developers often verify the function, performance and stability of some distributed storage systems after adding new features. Furthermore, when performing performance and stability verification work, staff need to conduct long-term stress tests on distributed storage system clusters, and perform statistics and analysis on relevant data generated during the testing process. However, due to artificial participation in statistics and analysis of relevant data, it will consume a lot of time and manpower, reduce work efficiency, and greatly increase its testing cost; resulting in inaccurate data acquisition.

Therefore, it is an urgent problem to be solved by those skilled in the art how to quickly obtain relevant data generated by the storage device during the testing process and accurately analyze it to improve work efficiency and data accuracy while reducing testing costs.

Contents of the invention

The purpose of this application is to provide a method for obtaining stress test data in a storage device, which can more quickly obtain relevant data generated by the storage device during the test process, and accurately analyze it, effectively improving the work efficiency. efficiency and data accuracy, and also greatly reduce the test cost of the storage system; another purpose of the present application is to provide an acquisition device, system and computer-readable storage medium for stress test data in a storage device, all of which have the above-mentioned benefits Effect.

In order to solve the above technical problems, the present application provides a method for acquiring stress test data in a storage device, the acquisition method comprising:

Collect the target stress test data from the storage device regularly through the data collection script;

Save the target stress test data into a data file;

Analyzing the target stress test data in the data file through a data parsing script to obtain stress test data.

Preferably, the regularly collecting target stress test data from the storage device through the data collection script includes:

Running the data collection script through a task timing running mechanism on the storage node of the storage device;

The target stress test data is collected by using the data collection script.

Preferably, the data collection script includes: top command, free command, dstst command, iostat command and ceph-s command.

Preferably, analyzing the target stress test data in the data file through the data parsing script, and obtaining the stress test data includes:

The target pressure data is analyzed by a text analysis tool of Linux to obtain the stress test data.

Preferably, the acquisition method also includes:

Save the stress test data to a txt file;

Import the stress test data in the txt file into an EXCEL file.

In order to solve the above technical problems, the present application also provides an acquisition device for stress test data in a storage device, the acquisition device includes:

The collection module is used to regularly collect target stress test data from the storage device through the data collection script;

A saving module, configured to save the target stress test data into a data file;

The parsing module is configured to parse the target stress test data in the data file through a data parsing script to obtain stress test data.

Preferably, the collection module includes:

A script running submodule, configured to run the data collection script on the storage node of the storage device through a task timing running mechanism;

The data collection sub-module is configured to use the data collection script to collect the target stress test data.

Preferably, the acquisition device further includes:

A data saving module, configured to save the stress test data to a txt file;

A data import module, configured to import the stress test data in the txt file into an EXCEL file.

In order to solve the above technical problems, the present application also provides an acquisition system for stress test data in storage devices, the acquisition system includes:

memory for storing computer programs;

A processor configured to implement the steps of any one of the methods for acquiring stress test data in the storage device described above when executing the computer program.

In order to solve the above-mentioned technical problems, the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, any one of the above-mentioned storage Steps of a method for obtaining stress test data in a device.

A method for acquiring stress test data in a storage device provided by the present application includes regularly collecting target stress test data from the storage device through a data collection script; saving the target stress test data into a data file; The target stress test data is analyzed in the data file to obtain stress test data.

It can be seen that the technical solution provided by this application can automatically collect various types of data generated by the storage system during the test process from the corresponding storage nodes through the corresponding data collection tools at regular intervals, that is, the above-mentioned target pressure test data, and through specific The text analysis tool automatically analyzes the target stress test data, so that the staff can get rid of the need for human participation when collecting CEPH cluster data, analyzing performance and stability, and effectively avoiding statistical errors caused by human participation. Data related to the performance and stability of the storage system can be obtained more accurately; at the same time, due to the need for human participation, it can effectively reduce the workload and work pressure of development and testing personnel, and further reduce the consumption of labor costs, thereby Improve overall work efficiency.

A device, system, and computer-readable storage medium for acquiring stress test data in a storage device provided in the present application all have the above-mentioned beneficial effects, and will not be repeated here.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present application, and those skilled in the art can also obtain other drawings according to the provided drawings without creative work.

FIG. 1 is a schematic flow diagram of a method for obtaining stress test data in a storage device provided by the present application;

FIG. 2 is a schematic flowchart of another method for obtaining stress test data in a storage device provided by the present application;

FIG. 3 is a schematic diagram of an acquisition device for stress test data in a storage device provided by the present application;

FIG. 4 is a schematic diagram of a system for acquiring stress test data in a storage device provided by the present application.

Detailed ways

The core of this application is to provide a method for obtaining stress test data in a storage device, which can more quickly obtain relevant data generated by the storage device during the test process, and accurately analyze it, effectively improving the work efficiency. Efficiency and data accuracy, and also greatly reduce the test cost of the storage system; another core of the application is to provide a pressure test data acquisition device, system and computer-readable storage medium in the storage device, all of which have the above-mentioned benefits Effect.

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

Please refer to FIG. 1. FIG. 1 is a schematic flowchart of a method for obtaining stress test data in a storage device provided by the present application. The obtaining method may include:

S101: regularly collect target stress test data from the storage device through the data collection script;

Specifically, in the process of verifying the performance and stability of the storage system, relevant data generated during the verification process, that is, the above-mentioned target stress test data, can be regularly collected by setting a corresponding data collection script. Among them, the above-mentioned target stress test data can be collected according to actual needs, and is not unique, such as the storable space, usage status and network transmission speed of each node in the storage system.

Preferably, the above-mentioned regular collection of target stress test data from the storage device through the data collection script may include: running the data collection script through a task timing operation mechanism on the storage node of the storage device; using the data collection script to collect the target stress test data.

Specifically, in the storage system, a corresponding task timing mechanism can be set on each storage node, and the above data collection script can be run through the task timing mechanism, so as to realize automatic collection of target stress test data. Wherein, the application does not specifically limit the timing of the automatic collection, and it can be set according to the actual needs of the performance verification.

Preferably, the above data collection script may include: top command, free command, dstst command, iostat command and ceph-s command.

Specifically, various collection tools can be set in the data collection script to realize the collection of target stress test data. In this application, the data collection script may include top command, free command, dstst command, iostat command and ceph-s command. Among them, the top command can be used to collect the memory information of each storage node, and determine the number and capacity of the memory; the free command can be used to collect the memory usage information of the storage system; the dstat command can be used to collect the network transmission speed between storage nodes, and Corresponding to the read and write speed of the disk; the iostat command can be used to collect the read and write speed of a single disk of the storage node and its performance utilization rate; the ceph-s command can be used to collect the overall performance data of the CEPH cluster, including the read and write speed and the read and write per second Write op number (operation, operand) and so on. Of course, the types and quantities of the collection tools set in the above data collection script are not limited thereto, and can be increased or decreased accordingly according to the actual situation, without affecting the implementation of the technical solution.

S102: saving the target stress test data into a data file;

S103: Analyze the target stress test data in the data file through the data analysis script to obtain the stress test data.

Specifically, after the target stress test data is collected, it can be saved in the corresponding data file, and further, the corresponding analysis tool can be used, that is, the above data analysis script to analyze the target stress test data in the data file , to obtain the stress test data related to the performance and stability of the storage system required by the staff. Wherein, the stress test data corresponds to the above-mentioned target stress test data, and may include various related data such as CPU idle rate, memory usage, network transmission speed, hard disk read and write speed, single hard disk resource usage, and cluster overall performance data.

Preferably, analyzing the target stress test data in the data file through the data analysis script and obtaining the stress test data may include: analyzing the target stress data through a Linux text analysis tool to obtain the stress test data.

Specifically, when analyzing the target stress test data, the target stress test data can be customized and analyzed through the text analysis tool that comes with Linux, and various data related to the performance and stability of the storage system can be obtained by parsing in a fixed format, namely Stress test data. The text analysis tool of Linux has multi-functional and high-efficiency performance characteristics, and can be used to read, calculate, sort and generate reports for data, making it easier to analyze stress test data and generate corresponding reports.

The method for obtaining stress test data in the storage device provided by this application enables the staff to get rid of the need for human participation when collecting CEPH cluster data, analyzing performance and stability, and effectively avoids statistics caused by human participation. error, can obtain data related to the performance and stability of the storage system more accurately; at the same time, because it is separated from the need for human participation, it can effectively reduce the workload and work pressure of development and testing personnel, and further reduce the consumption of labor costs , so as to improve the overall work efficiency.

On the basis of the above embodiments, please refer to FIG. 2 , which is a schematic flowchart of another method for acquiring stress test data in a storage device provided by the present application.

As a preferred implementation example, as shown in steps S104 to S105 in Figure 2, the method for obtaining stress test data in the storage device may also include:

S104: saving the stress test data to a txt file;

S105: Import the stress test data in the txt file into an EXCEL file.

Specifically, after automatically analyzing and obtaining the required stress test data, it can also be saved to a txt file, and further imported from the txt file into an EXCEL file to obtain the data required for a test report. According to the test report, various related performances and stability of the storage system can be determined more simply and intuitively.

In order to solve the above problem, please refer to FIG. 3, which is a schematic diagram of an acquisition device for stress test data in a storage device provided by the present application. The acquisition device may include:

The collection module 10 is used to regularly collect target stress test data from the storage device through the data collection script;

Preservation module 20, is used for preserving target pressure test data in the data file;

The parsing module 30 is configured to parse the target stress test data in the data file through the data parsing script to obtain the stress test data.

As a preferred embodiment, the above acquisition module 10 may include:

The script running sub-module is used to run the data collection script through the task timing running mechanism on the storage node of the storage device;

The data collection sub-module is used to collect the target stress test data by using the data collection script.

As a preferred embodiment, the above parsing module 30 may include:

The data analysis sub-module is used to analyze the target pressure data through the text analysis tool of Linux to obtain the pressure test data.

As a preferred embodiment, the acquisition device may also include:

Data saving module, used to save stress test data to txt file;

The data import module is used to import the stress test data in the txt file into the EXCEL file.

For the introduction of the device provided in this application, please refer to the above method embodiment, and this application will not repeat it here.

In order to solve the above problems, please refer to FIG. 4, which is a schematic diagram of a system for acquiring stress test data in a storage device provided by the present application. The acquisition system may include:

memory 10 for storing computer programs;

When the processor 20 is used to execute the computer program stored in the memory 10, the following steps can be implemented:

The target stress test data is regularly collected from the storage device through the data collection script; the target stress test data is saved in the data file; the target stress test data is analyzed in the data file through the data analysis script to obtain the stress test data.

For the introduction of the system provided by this application, please refer to the above method embodiment, and this application will not repeat it here.

In order to solve the above problems, the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps can be realized:

The target stress test data is regularly collected from the storage device through the data collection script; the target stress test data is saved in the data file; the target stress test data is analyzed in the data file through the data analysis script to obtain the stress test data.

The computer-readable storage medium may include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, etc., which can store program codes. medium.

For the introduction of the computer-readable storage medium provided by the present application, please refer to the foregoing method embodiments, and the present application does not repeat it here.

Each embodiment in the description is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for the related information, please refer to the description of the method part.

Professionals can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software or a combination of the two. In order to clearly illustrate the possible For interchangeability, in the above description, the composition and steps of each example have been generally described according to their functions. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be directly implemented by hardware, software modules executed by a processor, or a combination of both. Software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other Any other known storage medium.

The method, device, system, and computer-readable storage medium for acquiring stress test data in a storage device provided by the present application have been described in detail above. In this paper, specific examples are used to illustrate the principles and implementation methods of the present application, and the descriptions of the above embodiments are only used to help understand the methods and core ideas of the present application. It should be pointed out that for those skilled in the art, without departing from the principles of the application, some improvements and modifications can be made to the application, and these improvements and modifications also fall within the protection scope of the claims of the application.

Claims (10)

1. A method for obtaining stress test data in a storage device, comprising: Collect the target stress test data from the storage device regularly through the data collection script; Save the target stress test data into a data file; Analyzing the target stress test data in the data file through a data parsing script to obtain stress test data. 2. The acquisition method according to claim 1, wherein said collecting target stress test data from a storage device at regular intervals through a data collection script comprises: Running the data collection script through a task timing running mechanism on the storage node of the storage device; The target stress test data is collected by using the data collection script. 3. The acquisition method according to claim 2, wherein the data collection script comprises: top command, free command, dstst command, iostat command and ceph-s command. 4. The method according to claim 1, wherein the analyzing the target stress test data in the data file through the data parsing script, and obtaining the stress test data comprises: The target pressure data is analyzed by a text analysis tool of Linux to obtain the stress test data. 5. The acquisition method according to claim 4, further comprising: Save the stress test data to a txt file; Import the stress test data in the txt file into an EXCEL file. 6. An acquisition device for stress test data in a storage device, characterized in that it comprises: The collection module is used to regularly collect target stress test data from the storage device through the data collection script; A saving module, configured to save the target stress test data into a data file; The parsing module is configured to parse the target stress test data in the data file through a data parsing script to obtain stress test data. 7. The acquisition device according to claim 6, wherein the acquisition module comprises: A script running submodule, configured to run the data collection script on the storage node of the storage device through a task timing running mechanism; The data collection sub-module is configured to use the data collection script to collect the target stress test data. 8. The acquisition device according to claim 6, further comprising: A data saving module, configured to save the stress test data to a txt file; A data import module, configured to import the stress test data in the txt file into an EXCEL file. 9. A system for acquiring stress test data in a storage device, comprising: memory for storing computer programs; The processor is configured to implement the steps of the method for acquiring stress test data in the storage device according to any one of claims 1 to 5 when executing the computer program. 10. A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the computer program according to any one of claims 1 to 5 is implemented. The steps of the method for obtaining the stress test data in the storage device.