CN114297071A - Quality evaluation method and device for automatic test case - Google Patents

Abstract

The invention discloses a quality evaluation method and a quality evaluation device for an automatic test case, wherein the method comprises the following steps: executing all the test cases in the test case set; obtaining code coverage data of all node services in a distributed system; determining a test quality for the set of test cases based on the code coverage data. According to the scheme, index data capable of reflecting the automatic test cases and the software quality is obtained by collecting distributed micro-service operation period data, the effect of the automatic test cases is evaluated, the test quality and the test efficiency in the software research and development process are improved, and the effect of completing the software version quality evaluation and the test process through the same number of cases or fewer automatic test cases is achieved.

Description

Quality assessment method and device for automated test cases

technical field

The invention relates to the technical field of software development, and more particularly, to a quality evaluation method and device for automated test cases.

Background technique

The software system architecture is gradually changing from a centralized architecture to a distributed architecture, and new systems are mostly developed using distributed frameworks and microservices architectures. After the system goes online, with business development and continuous R&D investment, the number of microservice modules gradually increases, and the complexity of the system increases exponentially. In order to ensure the efficiency of the software R&D process under the premise of controllable quality, according to the R&D iteration characteristics of distributed systems and continuous operation systems, software R&D teams and test teams will mostly introduce automated testing systems for quality assurance in the system iteration process and Regression Testing. In the process of actual practice, with the increase of business code and related testing requirements, the number of test cases has also increased significantly; but with the increase of cases, the test process and quality have not improved with the increase of the number of cases. Therefore, it is necessary to provide a technical evaluation method to evaluate the effect of automated test cases.

SUMMARY OF THE INVENTION

In view of this, the present invention provides the following technical solutions:

A quality assessment method for automated test cases, including:

Execute all test cases in the test case collection;

Obtain the code coverage data of all node services in the distributed system;

A test quality of the set of test cases is determined based on the code coverage data.

Optionally, the obtaining code coverage data of all node services in the distributed system includes:

Collect the code coverage data of all node services in the distributed system after the execution of the test case is completed by the data collection component mounted on each node in the distributed system;

Wherein, the data acquisition component is implemented by loading a startup script in a development environment based on a configuration file previously formulated in the version library.

Optionally, also include:

After the download of the code coverage data collected by the data collection component is completed, the collected data of the data collection component is controlled to return to zero.

Optionally, determining the test quality of the test case set based on the code coverage data includes:

The CI/CD tool open application programming interface API is invoked through HTTP, and the construction task preconfigured in the CI/CD tool is invoked, so as to realize the determination of the test quality of the test case set.

Optionally, determining the test quality of the test case set based on the code coverage data includes:

Pull source code from the repository, and execute compilation and construction based on the source code, where the source code is the source code corresponding to the test service of the test case set;

Pull the code coverage data;

Summarize the source code and the code coverage data to obtain the test quality of the test case set.

Optionally, after obtaining the code coverage data of all node services in the distributed system, the method further includes:

Relevant data indicator associations are determined based on the case execution data of the test case and the code coverage data, and the relevant data indicator associations are used to guide the design of subsequent test cases.

Optionally, the code coverage data includes code coverage and version incremental code coverage.

A quality assessment device for automated test cases, comprising:

The case execution module is used to execute all test cases in the test case collection;

The data acquisition module is used to obtain the code coverage data of all node services in the distributed system;

A quality determination module, configured to determine the test quality of the test case set based on the code coverage data.

Optionally, the data acquisition module is specifically configured to: collect the code coverage data of all node services in the distributed system after the execution of the test case is completed by using the data acquisition component mounted on each node in the distributed system. ;

Wherein, the data acquisition component is implemented by loading a startup script in a development environment based on a configuration file previously formulated in the version library.

Optionally, the quality determination module includes:

a code pulling module, used for pulling source code from the repository, and performing compilation and construction based on the source code, where the source code is the source code corresponding to the test service of the test case set;

a data pulling module for pulling the code coverage data;

A data processing module, configured to summarize the source code and the code coverage data to obtain the test quality of the test case set.

It can be known from the above technical solutions that, compared with the prior art, the embodiment of the present invention discloses a quality assessment method and device for an automated test case. The method includes: executing all the test cases in the test case set; Code coverage data for all node services; determining the test quality of the set of test cases based on the code coverage data. The above solution obtains an indicator data that can reflect the automated test cases and software quality by collecting the data during the running period of distributed microservices, and evaluates the effect of the automated test cases, which helps to improve the test quality and the test efficiency in the software R&D process. , to achieve the effect of completing the software version quality assessment and testing process through the same number of automated test cases or less.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without creative work.

1 is a flowchart of a quality assessment method for an automated test case disclosed in an embodiment of the application;

FIG. 2 is a flowchart for determining the test quality of a test case set disclosed by an embodiment of the present application;

3 is a flowchart of another method for evaluating the quality of an automated test case disclosed in an embodiment of the application;

FIG. 4 is a schematic diagram of the implementation of the overall architecture deployment of the quality assessment of the test case disclosed in the embodiment of the present application;

5 is a schematic structural diagram of a quality assessment device for an automated test case disclosed in an embodiment of the application;

FIG. 6 is a schematic structural diagram of a quality determination module disclosed in an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

1 is a flowchart of a method for evaluating the quality of an automated test case disclosed in an embodiment of the present application. Referring to FIG. 1 , the method for evaluating the quality of an automated test case may include:

Step 101: Execute all test cases in the test case set.

The test case collection includes multiple test cases, and the test objects of these test cases are the same service. When testing the service under test, all test cases in the corresponding test case collection need to be executed.

Step 102: Obtain code coverage data of all node services in the distributed system.

Regarding code coverage, it is an evaluation method for case coverage. In the development process, developers often run tests locally and export them through relevant tools to form reports. For example, by executing 10 forward and reverse test cases, All business code lines can be executed, which means that the test code coverage of these 10 cases is 100%. It is generally believed that the higher the code coverage after the test case is executed, the better the test case coverage and the higher the test quality.

The code coverage data can be some data during the operation of the microservice, such as, but not limited to, the transaction volume of the microservice node, JVM operating parameters, indicators, and some manually added data collection point data.

In this application, the code coverage data of all node services in the distributed system will be obtained only after all the test cases in the test case set are executed. The traditional coverage collection stage is in the latter stage of the code construction process. Through CI (Continuous Integration, continuous integration) / CD (Continuous Delivery, continuous delivery) tools, code collection is achieved by locally executing unit test cases after the construction is completed. However, this method does not provide real feedback on the quality of the test environment after code deployment, and is not integrated with automated testing. However, in the embodiment of the present application, after the automated test is completed, that is, after the execution of all the test cases in the test case set, the relevant data is collected and analyzed, so that the validity of the automated test case in the test environment can be effectively evaluated. This ensures test quality.

Step 103: Determine the test quality of the test case set based on the code coverage data.

After the code coverage data is obtained, corresponding statistical analysis can be performed based on the code coverage data to obtain indicator data that can reflect the automated test cases and software quality, which can guide the design and implementation of automated test cases, improve the test quality, and finally feedback to the During the process management of software R&D iteration and ultimately improve the software quality, that is, to determine the test quality of the test case set, help the corresponding personnel to better arrange and handle the subsequent test work, and improve the test work efficiency.

The quality evaluation method of the automated test case described in this embodiment obtains index data that can reflect the quality of the automated test case and software by collecting distributed microservice runtime data, and evaluates the effect of the automated test case, which is helpful for Improve the test quality and test efficiency in the software development process, and achieve the effect of completing the software version quality assessment and test process through the same number of automated test cases or less.

In the above embodiment, the obtaining the code coverage data of all node services in the distributed system may include: collecting the distribution data after the execution of the test case is completed by using the data collection component mounted on each node in the distributed system. The code coverage data of all node services in the system; wherein, the data acquisition component is implemented by loading a startup script in the development environment based on the configuration file pre-established in the version library.

Based on the above content, the quality assessment method for an automated test case may further include: after the code coverage data collected by the data collection component is downloaded, controlling the collected data of the data collection component to return to zero for the next cycle use .

In one implementation, the determining the test quality of the test case set based on the code coverage data may include: calling the CI/CD tool open application programming interface API through HTTP, and calling up the pre-configured in the CI/CD. A build task in the tool that implements the determination of the test quality of the set of test cases.

FIG. 2 is a flowchart of determining the test quality of a test case set disclosed in an embodiment of the present application. With reference to FIG. 2 , the determining the test quality of the test case set based on the code coverage data may include:

Step 201: Pull source code from the repository, and perform compilation and construction based on the source code.

The source code is the source code corresponding to the test service of the test case set.

Step 202: Pull the code coverage data.

Step 203: Summarize the source code and the code coverage data to obtain the test quality of the test case set.

The specific implementation of determining the test quality of the test case set will be described in detail in a later example, and will not be described here.

FIG. 3 is a flowchart of another method for evaluating the quality of an automated test case disclosed in an embodiment of the present application. Referring to Figure 3, in one implementation, the method may include:

Step 301: Execute all test cases in the test case set.

Step 302: Obtain code coverage data of all node services in the distributed system.

Step 303: Determine the test quality of the test case set based on the code coverage data.

Step 304: Determine the correlation of relevant data indicators based on the case execution data of the test case and the code coverage data.

The correlation of the relevant data indicators is used to guide the design of subsequent test cases. The code coverage data includes code coverage and version incremental code coverage.

In this implementation, the effect of the correlation of the indicator can be verified by observing the defect data of the version. Among them, the defect data refers to the number of defects generated by manual testing in each iterative version, and the quality of the version can be reflected by the defect level and quantity. Under the distributed microservice architecture, automated testing can improve test efficiency. By increasing the number of automated test cases and evaluating the quality of automated cases by automatically collecting coverage data, and comparing with manual test results, the effectiveness of the aforementioned indicator collection and evaluation can be obtained.

For the application system of the distributed microservice architecture, after the introduction of the automated test case platform, related functions such as automatic case sending and case management are realized through the interface API call. Since the distributed system will be deployed on multiple computer systems, through the The node mounts the test code coverage collection plug-in for each service. After the automated test case test is completed, the data collection of each node service is collected, summarized and displayed by scheduling the unified code coverage collection program. For the design logic of the application, Continuously improve the coverage data of automated test results by adding test cases in a targeted manner, and finally establish the associated observation data of automated test cases, coverage, and version test results defects, which are used to evaluate the actual effectiveness of the method. Fewer test cases achieve better test quality and ensure the quality of software development and upgrades.

In practical applications, the overall implementation scheme of the quality assessment method for automated test cases is mainly reflected in the distributed system architecture, centralized data collection and analysis, and the integration of functions of automated test platforms.

FIG. 4 is a schematic diagram of the deployment and implementation of the overall architecture of the quality assessment of the test case disclosed in the embodiment of the present application. With reference to FIG. 4 , in a specific implementation:

The overall process of the solution is triggered by software version management on the software version library, and the automatic construction and deployment of distributed service applications is triggered by submitting code merge requests. This solution is oriented to distributed microservices, which is characterized by many nodes and complex deployment. From the improvement of its own development efficiency, the automatic process of code submission, construction, deployment, automated testing, and test coverage collection is an effective pipeline method to improve performance. By formulating the corresponding configuration file in the version library, the jacoco code coverage statistics plug-in can be mounted to realize the collection of code coverage data by loading the jacoco (third-party component) agent in the development environment such as FAT/UAT by starting the script. Among them, Jacoco is a tool that supports dynamic mounting from the running state service node through the proxy business class, and records the code running data when the business class code is executed.

After the automatic deployment is completed, the execution of the overall test case is triggered by the automatic deployment platform or manually. After the automated test platform executes all the required cases, it calls the Jenkins (an open source CI/CD tool) open application programming interface API through HTTP, and invokes the build tasks configured on Jenkins in advance.

After the build task is started, first pull the source code of the same version of the target service from the GitLab (version management software) repository to execute the compilation and build, and then connect to the target service jacoco service port through TCP to download the coverage data and let the target service jacoco through the instruction Collect data to zero.

After the build process and collection are completed, Jenkins executes the sonar service address, target project ID, security The certification ID and other implementations implement the upload of the aggregated application service code coverage data for this automated test.

Through sonarqube (an open source platform for managing the quality of source code), the test code coverage data of software engineering application service modules can be viewed on the page side, and the relevant data of time cycle and version iteration summary are provided for analysis. Combined with relevant case execution data, code coverage, version incremental code coverage and other related data, the correlation of relevant data indicators can be established, and the quality level of the software iteration process can be evaluated by observing the changes in the data indicators. The correlation is reflected in the combination of the number of automated test cases and the code coverage data, so that the converted test cases can be designed for the lines of code that are not covered, so as to increase the number of cases and improve the code coverage.

Through the detailed coverage of the code files provided by sonarqube, relevant test cases can be designed in a targeted manner and added to the automated case set, and the coverage data indicators can be improved in the next iteration execution. Improve the quality assurance capability of automated testing by continuously improving coverage, from the original purpose of software quality improvement.

Through the above solutions, the R&D and testing links can be effectively integrated, the resource input of invalid test cases can be solved, and the overall R&D testing efficiency can be improved. In the traditional solution, the test is limited to the black-box test of the interface function, and more than 800 automated test cases are realized. The actual code coverage collection data is 20-25% on average and fails to cover most of the distributed interactions. In this application scheme, the coverage data and code file coverage implementation can be combined, and more than 120 targeted design cases can achieve an overall code coverage rate of more than 50%. % Defects are reduced. Overall, by improving the quality of automated cases, the test work for version optimization and upgrade has been greatly reduced, and the efficiency of software R&D iteration has been improved.

The technical solution of the present application is designed for a distributed system, and solves the data collection and aggregation functions of distributed services through configuration files, environment management and other aspects. In addition, the overall technical solution is realized by integrating open source software and frameworks. The solution is simple to implement and has certain universality and promotion value.

For the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, Certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

The method is described in detail in the above disclosed embodiments of the present invention, and the method of the present invention can be implemented by various forms of devices. Therefore, the present invention also discloses a device, and specific embodiments are given below for detailed description.

FIG. 5 is a schematic structural diagram of an apparatus for evaluating the quality of an automated test case disclosed in an embodiment of the present application. Referring to FIG. 5 , the apparatus for evaluating the quality of an automated test case 50 may include:

The case execution module 501 is used for executing all test cases in the test case set.

The data obtaining module 502 is configured to obtain the code coverage data of all node services in the distributed system.

A quality determination module 503, configured to determine the test quality of the test case set based on the code coverage data.

The quality evaluation device of the automated test case described in this embodiment obtains index data that can reflect the quality of the automated test case and software by collecting distributed microservice runtime data, and evaluates the effect of the automated test case, which is helpful for Improve the test quality and test efficiency in the software development process, and achieve the effect of completing the software version quality assessment and test process through the same number of automated test cases or less.

In one implementation, the data acquisition module is specifically configured to: collect the code coverage data of all node services in the distributed system after the execution of the test case is completed through the data acquisition component mounted on each node in the distributed system. . Wherein, the data acquisition component is implemented by loading a startup script in a development environment based on a configuration file previously formulated in the version library.

In one implementation, the data acquisition module is further configured to: after the download of the code coverage data acquired by the data acquisition component is completed, control the acquisition data of the data acquisition component to return to zero.

In one implementation, the quality determination module is specifically used for: calling the CI/CD tool open application programming interface API through HTTP, calling up the construction task preconfigured in the CI/CD tool, and realizing the test quality of the test case collection. ok.

FIG. 6 is a schematic structural diagram of a quality determination module disclosed in an embodiment of the present application. 6, in one implementation, the quality determination module 503 includes:

The code pulling module 601 is configured to pull source code from the repository, and execute compilation and construction based on the source code, where the source code is the source code corresponding to the test service of the test case set.

A data pulling module 602 is configured to pull the code coverage data.

The data processing module 603 is configured to summarize the source code and the code coverage data to obtain the test quality of the test case set.

In one implementation, the quality assessment device for an automated test case further includes an indicator association module, configured to determine the correlation of relevant data indicators based on the case execution data of the test case and the code coverage data, and the correlation of the relevant data indicators. Used to guide the design of subsequent test cases.

The quality assessment device for any automated test case described in the foregoing embodiment includes a processor and a memory. In the foregoing embodiment, a case execution module, a data acquisition module, a quality determination module, a code pulling module, and a data pulling module are included. , data processing modules, etc. are all stored in the memory as program modules, and the processor executes the above-mentioned program modules stored in the memory to realize corresponding functions.

The processor includes a kernel, and the kernel calls the corresponding program module from the memory. The kernel can set one or more, and realize the processing of return data by adjusting the kernel parameters.

Memory may include non-persistent memory in computer readable media, random access memory (RAM) and/or non-volatile memory, such as read only memory (ROM) or flash memory (flash RAM), the memory including at least one memory chip.

Embodiments of the present invention provide a storage medium on which a program is stored, and when the program is executed by a processor, implements the method for evaluating the quality of an automated test case described in the foregoing embodiments.

An embodiment of the present invention provides a processor for running a program, wherein when the program runs, the method for evaluating the quality of an automated test case described in the foregoing embodiment is executed.

Further, this embodiment provides an electronic device including a processor and a memory. The memory is used for storing executable instructions of the processor, and the processor is configured to execute the quality assessment method of the automated test case described in the above embodiments by executing the executable instructions.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method.

It should also be noted that in this document, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply those entities or operations There is no such actual relationship or order between them. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The steps of a method or algorithm described in conjunction with the embodiments disclosed herein may be directly implemented in hardware, a software module executed by a processor, or a combination of the two. A software module can be placed in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other in the technical field. in any other known form of storage medium.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A quality evaluation method for an automated test case is characterized by comprising the following steps:

executing all the test cases in the test case set;

obtaining code coverage data of all node services in a distributed system;

determining a test quality for the set of test cases based on the code coverage data.

2. The method for automatically evaluating the quality of a test case according to claim 1, wherein the obtaining code coverage data of all node services in the distributed system comprises:

acquiring code coverage data of all node services in the distributed system after the execution of the test case is finished through a data acquisition component mounted on each node in the distributed system;

the data acquisition component is realized by loading a start script in a development environment based on a configuration file which is made in a version library in advance.

3. The method for automatically evaluating the quality of test cases of claim 2, further comprising:

and after the code coverage data acquired by the data acquisition assembly is downloaded, controlling the acquired data of the data acquisition assembly to return to zero.

4. The method of claim 1, wherein determining the test quality of the set of test cases based on the code coverage data comprises:

and calling an open Application Programming Interface (API) of the CI/CD tool through HTTP, calling a construction task which is pre-configured in the CI/CD tool, and determining the test quality of the test case set.

5. The method of claim 1, wherein determining the test quality of the set of test cases based on the code coverage data comprises:

pulling a source code from a version library, and executing compiling construction based on the source code, wherein the source code is a source code corresponding to the test service of the test case set;

pulling the code coverage data;

and summarizing the source codes and the code coverage data to obtain the test quality of the test case set.

6. The method for automatically evaluating the quality of a test case according to claim 1, further comprising, after obtaining the code coverage data of all node services in the distributed system:

determining a correlation data index correlation based on the case execution data of the test case and the code coverage data, the correlation data index correlation being used to guide the design of subsequent test cases.

7. The method for quality assessment of automated test cases according to any of claims 1 to 6, wherein said code coverage data comprises code coverage and version delta code coverage.

8. An automated test case quality assessment apparatus, comprising:

the case execution module is used for executing all the test cases in the test case set;

the data acquisition module is used for acquiring code coverage data of all node services in the distributed system;

a quality determination module to determine a test quality of the set of test cases based on the code coverage data.

9. The apparatus for quality assessment of automated test cases of claim 8, wherein the data acquisition module is specifically configured to: acquiring code coverage data of all node services in the distributed system after the execution of the test case is finished through a data acquisition component mounted on each node in the distributed system;

the data acquisition component is realized by loading a start script in a development environment based on a configuration file which is made in a version library in advance.

10. The automated test case quality assessment apparatus of claim 8, wherein said quality determination module comprises:

the code pulling module is used for pulling a source code from the version library and executing compiling construction based on the source code, wherein the source code is the source code corresponding to the test service of the test case set;

the data pulling module is used for pulling the code coverage data;

and the data processing module is used for summarizing the source code and the code coverage data to obtain the test quality of the test case set.

Images