CN101604288A - A Software Quality Evaluation Method Based on Test Data - Google Patents

Abstract

A method for evaluating software quality based on test data, firstly determine the evaluation formula of functional test case set and non-functional test case set evaluation formula, determine the software quality sub-characteristic evaluation formula according to the test case set evaluation formula, according to the software quality The sub-feature evaluation formula determines the software quality characteristic evaluation formula, and the software quality evaluation formula is determined according to the software quality characteristic evaluation formula. The invention determines the software quality evaluation confidence degree, and determines the software quality comprehensive evaluation formula according to the software quality evaluation result and the software quality evaluation confidence degree. The invention evaluates the software quality objectively and has high accuracy.

Description

A kind of method for evaluating software quality based on test data

Technical field

The present invention relates to a kind of method for evaluating software quality in the soft project based on test data.

Technical background

Along with the development of infotech, computer software has been penetrated in the every field of national economy, and closely bound up with people's productive life.Therefore, the quality of software quality is subjected to people's attention day by day." quality of software product is the life of software enterprise ", this viewpoint has become the common recognition of industry.To the software implementation software quality evaluation, be the important means that another promotion software quality after software quality management and software test improves.Software quality evaluation is the quality level of evaluation software quantitatively, accurately controls, manages and improve the quality of software, for software typing, examination, evaluation and industrialization provide objective, just scientific basis.

The formal software quality models of releasing of ISO is the ISO/IEC9126 series standard, and this standard provides the software quality metric model of the three-decker of " characteristic-sub-feature-measurement metric ", as Fig. 1.This model is a tree structure, and high level has defined 6 mass propertys, and the middle layer has defined corresponding quality sub-feature, and the 3rd layer is measurement metric.The corresponding relation of mass property and quality sub-feature as shown in Table 1.

The corresponding relation of mass property and quality sub-feature in form 1 ISO/IEC 9126 models

Mass property	The quality sub-feature
		Functional	Adaptability, accuracy, interoperability, secret and safe, functional compliance
Reliability	Maturity, fault-tolerance, the easy compliance of restorative, reliability
		Easy usability	The easy compliance of the property understood, learnability, ease for operation, attractability, ease for use
Efficient	Time response, the utilization of resources, efficient compliance
		Maintainable	The compliance of easily analytical, malleable, stability, Easy Test, maintainability
Portable	Adaptability, easily installation property, compossibility, easily replaceability, portable compliance

The ISO/IEC9126 standard does not have quantisation metric unit, but points out and can define voluntarily according to the different situations of enterprise.

The Jia Xinghua of BJ University of Aeronautics ﹠ Astronautics; Yan Haihua; in " computer technology and application progress " (Page545-551 in 2007) " based on software external quality assessment schemes research of test ", a kind of method for evaluating software quality based on test data has been proposed.

This method provides customizable software quality models based on international standard ISO/IEC 9126 standards, provides the measurement metric of quantification based on test data.It is defined as test use cases with the measurement metric in the model, provides the evaluation of software quality sub-feature, software quality characteristics and software quality by the evaluation to test use cases and execution result thereof.

This method provides customizable software quality models.Particular content for according to the quality requirement of different user and dissimilar softwares to mass property in the quality model and correlator characteristic make amendment, define, interpolation or cutting, promptly define mass property, the quality sub-feature of this project according to the feature of project itself, thereby make model can satisfy the specific quality requirement of user, easier metric operations.This method pass test data (test use cases and execution result) is carried out software test and quality model related, the cambium layer aggregated(particle) structure.The corresponding quality evaluation system of being set up as shown in Figure 2.

The test use cases definition sees Appendix 1.

This method is the test use cases (proper testing set of uses case TSD, illegal test use cases TSF, limit testing set of uses case TST) that each quality sub-feature has been set up three acquiescences, and is related with the respective quality sub-feature by them.And stipulate that they confront contribution rate (the being also referred to as weight) Q respectively of Quantum Properties assessment ₁, Q ₂, Q ₃, Q is arranged at this ₁+ Q ₂+ Q ₃=1.The evaluation personnel also can expand the classification of test use cases as required accordingly, but will satisfy ∑ Q _i=1,0≤Q _i≤ 1.

This method is divided into functional test set of uses case and non-functional test use cases two classes with test use cases, and each test use cases belongs to functional test set of uses case or non-functional test use cases.If the affiliated mass property of quality sub-feature is functional under this test use cases, then this test use cases is the functional test set of uses case, otherwise is the non-functional test use cases.

Below provide the judgement schematics of functional test set of uses case, non-functional test use cases, quality sub-feature, mass property and software oeverall quality.

Functional test set of uses case judgement schematics is: $S_T=\frac{N_1*D-\underset{j=1}{\overset{M}{\mathrm{\Σ}}}{p}_j*x_j}{N_1*D},$ Wherein, N ₁Represent test case to concentrate the sum of the test case of design, M represents the quantity of unsanctioned test case, and on behalf of test case, D concentrate the diversity factor of test case, x _jRepresent the significance level of j unsanctioned test case, p _jRepresent the order of severity of problem report of the discovery of j unsanctioned test case.Test case importance is divided into high, normal, basic [1,0.5,0.25], and the seriousness of problem report also is divided into high, normal, basic [1.0,0.5,0.25], and the seriousness classification value of the importance of test case and problem report can rule of thumb be adjusted.

Non-functional test use cases judgement schematics is: $S_T=\frac{N_2*D-M}{N_2*D},$ Wherein, N ₂Represent test case to concentrate the sum of the test case of design, M represents the quantity of unsanctioned test case, and on behalf of test case, D concentrate the diversity factor of test case.

The judgement schematics of quality sub-feature is defined as: $M_s=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{Q}_i*S_{T_i},$ Wherein, supposing has n test use cases in quality sub-feature bottom, and the weight of each test use cases correspondence is Q _i, wherein $\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{Q}_i=1,$ Be the evaluation of each test use cases, if mass property is functional under this quality sub-feature, then

Calculate according to functional test set of uses case judgement schematics, if mass property is a non-functional under this quality sub-feature, then Calculate according to non-functional test use cases judgement schematics.The quality evaluation result of quality sub-feature is the weight sum of products of each test use cases evaluation score and test use cases place quality sub-feature.

The judgement schematics of mass property is defined as: $M_c=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{R}_i*M_{\mathrm{si}}$ , wherein, supposing has n quality sub-feature in mass property bottom, and the mark of i quality sub-feature is M _Si, its corresponding weight is R _i, wherein $\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{R}_i=1$ 。Therefore the quality evaluation result of mass property is the mark of the quality sub-feature under it and the weight sum of products of quality sub-feature place mass property.

The software quality evaluation formula definition is: $E_{\mathrm{quality}}=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{R}_i*M_{\mathrm{ci}}$ , wherein, supposing has n mass property in mass property bottom, and the mark of i mass property is M _Ci, its corresponding weight is R _i, wherein $\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{R}_i=1$ 。Therefore the software quality evaluation result is the mark and the mass property weight sum of products of affiliated mass property.

The weak point of above-mentioned method for evaluating software quality mainly contains following three aspects.

(1) do not consider the situation of the corresponding a plurality of software issues of test case in the functional test set of uses case judgement schematics.

(2) tolerance of factor of evaluation diversity factor is to be provided by manual review, influenced by evaluation expert's subjectivity.

(3) do not provide in this model the software quality evaluation tolerance of degree of confidence as a result.

The evaluation result that the existing quality evaluating method of the feasible utilization of these deficiencies is estimated software is objective inadequately, and accuracy is relatively poor.

Summary of the invention

The technology of the present invention is dealt with problems: overcome the deficiencies in the prior art, a kind of method for evaluating software quality based on test data is provided, this method has realized that the software quality evaluation result is objective, the accuracy height.

Technical solution of the present invention: a kind of method for evaluating software quality based on test data, prior art is improved, show that the present invention considers the situation of the corresponding a plurality of software issues of a test case when Function of Evaluation property testing set of uses case, improve functional test set of uses case judgement schematics; The present invention defines the measure of factor of evaluation diversity factor in the test use cases judgement schematics; The present invention defines the software quality evaluation degree of confidence; The present invention provides the method for coming the comprehensive evaluation software quality in conjunction with software quality evaluation and software quality evaluation degree of confidence.

Concrete steps of the present invention are as follows:

Step 1: determine the test use cases evaluation

Test use cases is divided into functional test set of uses case and non-functional test use cases two classes, and each test use cases belongs to functional test set of uses case or non-functional test use cases; If the affiliated mass property of quality sub-feature is functional under this test use cases, then this test use cases is the functional test set of uses case, otherwise is the non-functional test use cases;

Described functional test set of uses case is evaluated as: $S_T=\frac{N_1*D-\underset{j=1}{\overset{L}{\mathrm{\Σ}}}{p}_j*x_j}{N_1*D}$ , wherein, N ₁Represent actual test case number, when existing a test case to cause the situation of a plurality of software issues, regard this test case as with software issue number respective amount a plurality of test cases, it calculates by the correspondence problem number when calculating the use-case sum; L represents the sum of software issue, p _jThe seriousness degree of representing j software issue, x _jRepresent the importance degree of the corresponding test case of j software issue, D represents the test use cases diversity factor, 0≤D≤1;

Described non-functional test use cases is evaluated as: $S_T=\frac{N_2*D-M}{N_2*D},$ Wherein, N ₂Represent test case to concentrate the sum of the test case of design, M represents the quantity of unsanctioned test case, and on behalf of test case, D concentrate the diversity factor of test case;

Step 2: determine the evaluation of software quality sub-feature

The software quality sub-feature is evaluated as: $M_s=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{Q}_i*S_{T_i}$ , wherein, supposing has n test use cases, Q in quality sub-feature bottom _iBe the weight of each test use cases correspondence, wherein $\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{Q}_i=1,$

For each test use cases is estimated, if mass property is functional under this quality sub-feature, then

Calculate according to functional test set of uses case judgement schematics in the step 1, if mass property is a non-functional under this quality sub-feature, then

Calculate according to non-functional test use cases judgement schematics in the step 1;

Step 3: determine the software quality characteristics evaluation

Software quality characteristics is evaluated as: $M_c=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{R}_i*M_{\mathrm{si}}$ , wherein, supposing has n quality sub-feature in mass property bottom, and the evaluation score of i quality sub-feature is M _Si, to calculate according to step 2 software quality sub-feature judgement schematics, the weight of i quality sub-feature correspondence is R _i, wherein $\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{R}_i=1;$

Step 4: software quality evaluation

Software quality evaluation is: $E_{\mathrm{quality}}=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{R}_i*M_{\mathrm{ci}}$ , wherein, suppose below software quality, to have n mass property, the evaluation score of i mass property is M _Ci, to calculate according to step 3 software quality sub-feature judgement schematics, the weight of i mass property correspondence is R _i, wherein $\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{R}_i=1;$

Step 5: determine the software quality evaluation confidence scores

S＝S _tcset＝S _white*S _black

Wherein, S is the software quality evaluation confidence scores, S _TcsetBe the adequacy of software test case, S _WhiteBe software white-box testing adequacy, S _BlackBe software Black-box Testing adequacy;

White-box testing adequacy S _White=Coverage _XThe requirement of supposing software dialogue box test-types is that X covers, and then the coverage rate of X covering is

Black-box Testing adequacy deterministic process is as follows:

(1) at first, provide leaf node, i.e. the adequacy of test use cases;

(2) certain non-leaf node x has n child, and the weight of each child is p _i, corresponding adequacy grade is s _i, then this non-leaf node adequacy is evaluated as $S_x={\mathrm{\Σ}}_{i=1}^n{p}_i{s}_i;$

(3) the rest may be inferred, can get the adequacy S of software Black-box Testing _Black

Step 6: determine software quality comprehensive evaluation

The degree of confidence factor of software quality evaluation is joined in the software quality evaluation system, provide software quality comprehensive evaluation formula: E in conjunction with the software quality evaluation mark _Total=E _Quality* S

Wherein, E _TotalRepresent software quality comprehensive evaluation mark, E _QualityBe the software quality evaluation mark, S is the degree of confidence of software quality evaluation mark.

The present invention extrapolates software quality sub-feature evaluation result according to the test use cases evaluation result, extrapolate the mass property evaluation result according to software quality sub-feature evaluation result, extrapolate quality evaluation result by the mass property evaluation result, extrapolate the software quality comprehensive evaluation result by the degree of confidence of quality evaluation result and quality evaluation result.

The present invention's advantage compared with prior art is:

(1) the present invention has improved the test use cases evaluation, and provides the metric function of test use cases factor of evaluation diversity factor, has solved prior art to test use cases evaluation comprehensively and not enough objective problem inadequately.

(2) the present invention provides software quality evaluation result's degree of confidence, takes all factors into consideration the Black-box Testing adequacy and white-box testing adequacy situation defines the software test case adequacy, and with the degree of confidence of this adequacy as the software quality evaluation result.

(3) the present invention provides software quality comprehensive evaluation, degree of confidence factor in conjunction with software quality evaluation mark and software quality evaluation provides software quality comprehensive evaluation, the confidence level and the accuracy of software quality evaluation have been improved, solved and utilized prior art objective inadequately, the problem that accuracy is relatively poor the evaluation result of software.

Description of drawings

Fig. 1 is the formal software quality models synoptic diagram of releasing of existing ISO;

Fig. 2 publishes the synoptic diagram of document for background technology;

Fig. 3 is realization flow figure of the present invention.

Embodiment

As shown in Figure 3, the present invention is based on the method for evaluating software quality of test data, at first determine functional test set of uses case judgement schematics and non-functional test use cases judgement schematics, determine software quality sub-feature judgement schematics according to the test use cases judgement schematics, determine the software quality characteristics judgement schematics according to software quality sub-feature judgement schematics, determine the software quality evaluation formula according to the software quality characteristics judgement schematics.The present invention has determined the software quality evaluation degree of confidence, and determines software quality comprehensive evaluation formula according to software quality evaluation result and software quality evaluation degree of confidence, and its concrete steps are as follows:

Step 1: determine the test use cases evaluation

Quality evaluation index system of the present invention is same as the prior art, as Fig. 2.

The present invention is based on international standard ISO/IEC 9126 standards and provide customizable software quality models, particular content is same as the prior art, and corresponding quality evaluation system as shown in Figure 2.The present invention is based on test data the measurement metric in the model is defined as test use cases, provide the evaluation of software quality sub-feature, software quality characteristics and software quality by evaluation test use cases and execution result thereof.

The present invention is the test use cases (proper testing set of uses case TSD, illegal test use cases TSF, limit testing set of uses case TST) that each quality sub-feature has been set up three acquiescences, and is related with the respective quality sub-feature by them.And stipulate that they confront contribution rate (the being also referred to as weight) Q respectively of Quantum Properties assessment ₁, Q ₂, Q ₃, Q is arranged at this ₁+ Q ₂+ Q ₃=1.The evaluation personnel also can expand the classification of test use cases as required accordingly, but will satisfy ∑ Q _i=1,0≤Q _i≤ 1.

The present invention is divided into functional test set of uses case and non-functional test use cases two classes with test use cases, and each test use cases belongs to functional test set of uses case or non-functional test use cases.If the affiliated mass property of quality sub-feature is functional under this test use cases, then this test use cases is the functional test set of uses case, otherwise is the non-functional test use cases.

Non-functional test use cases judgement schematics is identical with original technological assessment formula: $S_T=\frac{N_2*D-M}{N_2*D},$ Wherein, N ₂Represent test case to concentrate the sum of the test case of design, M represents the quantity of unsanctioned test case, and on behalf of test case, D concentrate the diversity factor of test case.

The present invention improves functional test set of uses case judgement schematics.The present invention considers the situation of the corresponding a plurality of software issues of a test case, improves functional test set of uses case judgement schematics to be: $S_T=\frac{N_1*D-\underset{j=1}{\overset{L}{\mathrm{\Σ}}}{p}_j*x_j}{N_1*D},$ Wherein, N ₁Represent actual test case number, when existing a test case to cause the situation of a plurality of software issues, regard this test case as with software issue number respective amount a plurality of test cases, it calculates by the correspondence problem number when calculating the use-case sum.L represents the sum of software issue, p _jThe seriousness degree of representing j software issue, x _jThe importance degree of representing the corresponding test case of j software issue.D represents the test use cases diversity factor, and 0≤D≤1 draws after this value is evaluated test case by evaluation personnel or tester in the document " based on the software external quality assessment schemes research of test ".Provide the metric function of factor of evaluation diversity factor below the present invention.

The present invention at first introduces the diversity factor of test case, provides the diversity factor of test use cases then.

The diversity factor value of two test cases can be 0 (not having difference) or 1 (variant).To provide test case diversity factor computing method from Black-box Testing and two angles of white-box testing below.

For Black-box Testing, test case diversity factor computing method are as follows.

1, test case tc1 and test case tc2 promptly are positioned at same test use cases among Fig. 2 in software quality models.

If all input points of 2 tc1 and tc2 are all in same equivalence class, and the input sequence and the operating process of all input points is all identical among tc1 and the tc2, and tc1 and tc2 all represent the situation on border condition or non-border, and the diversity factor of tc1 and tc2 is 0 so, otherwise are 1.

For white-box testing, according to different test coverage criterions, test case otherness judgment criterion difference, test case diversity factor also can be different.Provide the judgment criterion of judging two test case diversity factoies at different white-box testing covering methods below.Suppose:

1, test case tc1 and test case tc2 promptly are positioned at same test use cases among Fig. 2 in software quality models.

2, for set A={ a ₁, a ₂..., a _nAnd set B={ b ₁, b ₂..., b _m, the necessary and sufficient condition of A=B is: $A=B\⇔m=n\∩\∀i\∈[1,n],a_i=b_i.$

Criterion 1: cover at statement, suppose tc1, the statement sequence that tc2 carries out is respectively:

Sentence _tc1＝{sent _a1，sent _a2，…，sent _an}Sentence _tc2＝{sent _b1，sent _b2，…，sent _bm}

If Sentence _Tc1=Sentence _Tc2, then the diversity factor of tc1 and tc2 is 0, otherwise is 1;

Criterion 2: cover (or branch covers) at judging, suppose tc1, the branched sequence that tc2 carries out is respectively:

Decision _tc1＝{dec _a1，dec _a2，…，dec _an}Decision _tc2＝{dec _b1，dec _b2，…，dec _bm}

Tc1, value (Boolean value, the as follows) sequence of respective branches point is respectively in the branched sequence that tc2 carries out:

Value_dec _tc1＝{v_dec _a1，v_dec _a2，…，v_dec _an}Value_dec _tc2＝{v_dec _b1，v_dec _b2，…，v_dec _bm}

If Decision _Tc1=Decision _Tc2And Value_dec _Tc1=Value_dec _Tc2, then the diversity factor of tc1 and tc2 is 0, otherwise is 1;

Criterion 3: cover at condition, suppose tc1, the value sequence of each condition in each judgement that tc2 carries out is respectively:

Dec_con _tc1＝{dec_con _a1，…，dec_con _an}Dec_con _tc2＝{dec_con _b1，…，dec_con _bm}

If Dec_con _Tc1=Dec_con _Tc2, then the diversity factor of tc1 and tc2 is 0, otherwise is 1;

Criterion 4: cover at judgement/condition, suppose tc1, the value sequence of each judgement that tc2 carries out is respectively:

Decision _tc1＝{dec _a1，dec _a2，…，dec _an}Decision _tc2＝{dec _b1，dec _b2，…，dec _bm}

The value sequence of each condition in each judgement that tc1, tc2 carry out is respectively:

Condition _tc1＝{con _a1，con _a2，…，con _ap}Condition _tc2＝{con _b1，con _b2，…，con _bq}

If Decision _Tc1=Decision _Tc2And Condition _Tc1=Condition _Tc2, then the diversity factor of tc1 and tc2 is 0, otherwise is 1.

If two test cases all do not have difference from the angle of white-box testing and Black-box Testing, the diversity factor that then shows these two test cases is 0, otherwise is 1.

On the basis of definition test case diversity factor, the test use cases diversity factor is defined as follows.

1. certain test use cases A is tcset _A={ tc ₁, tc ₂..., tc _n;

2. test case tc _AiWith test case tc _AjDiversity factor be e (tc _Ai, tc _Aj);

3.tcset _AThe Validity Test set of uses case be

Tcset _EA={ tc _A1, tc _A2..., tc _Am| tc _Ai∈ tcset _A, e (tc _Ai, tc _Aj)=1, i ≠ j and i, j ∈ [1, n] };

4.tcset _AMaximum Validity Test set of uses case be

tcset _{EA_max}＝{tcset _B|tcset _B＝tcset _EA，|tcset _B|＝max{|tcset _EA|}}；

5.tcset _ADiversity factor be $E_{{\mathrm{tcset}}_A}=\left|{\mathrm{tcset}}_{\mathrm{EA}\_\max }\right|/\left|{\mathrm{tcset}}_A\right|$

Step 2: determine the evaluation of software quality sub-feature

The software quality sub-feature is evaluated as: $M_s=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{Q}_i*S_{T_i}$ , wherein, supposing has n test use cases, Q in quality sub-feature bottom _iBe the weight of each test use cases correspondence, wherein $\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{Q}_i=1,$

Be each test use cases evaluation score, if mass property is functional under this quality sub-feature, so

Calculate according to functional test set of uses case judgement schematics in the step 1, if mass property is not functional under this quality sub-feature, so

Calculate according to non-functional test use cases judgement schematics in the step 1;

Step 3: determine the software quality characteristics evaluation

Software quality characteristics is evaluated as: $M_c=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{R}_i*M_{\mathrm{si}}$ , wherein, supposing has n quality sub-feature in mass property bottom, and the evaluation score of i quality sub-feature is M _Si, to calculate according to step 2 software quality sub-feature judgement schematics, the weight of i quality sub-feature correspondence is R _i, wherein $\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{R}_i=1;$

Step 4: software quality evaluation

Software quality evaluation is: $E_{\mathrm{quality}}=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{R}_i*M_{\mathrm{ci}}$ , wherein, suppose below software quality, to have n mass property, the evaluation score of i mass property is M _Ci, comment 6 valency formula to calculate according to step 3 software quality sub-feature, the weight of i mass property correspondence is R _i, wherein $\underset{i=1}{\overset{n}{\mathrm{\Σ}}}{R}_i=1;$

Step 5: determine the software quality evaluation degree of confidence

Do not consider the degree of confidence of software quality evaluation in the method for evaluating software quality of prior art introduction, this makes when test is insufficient, the software quality evaluation that uses this method still can obtain, and this obviously is irrational.Owing to undertaken by test case and execution result thereof when coming the evaluation software quality based on test data, so test case designs fully more, the degree of confidence of the evaluation result that draws by this evaluation method is also high more.Therefore the present invention represents the degree of confidence of software quality evaluation with the adequacy of software test case.

Because white-box testing is based on the software logic structure, and Black-box Testing is based on software requirement, in order to examine or check the adequacy of software test case more all sidedly, the present invention takes all factors into consideration the white-box testing adequacy and the Black-box Testing adequacy comes the software test case adequacy is measured.

The adequacy of software white-box testing depends on the requirement of dialogue box test-types, as statement covering, branch's covering, condition covering and the covering of judgement/condition etc., be typically implemented in (the criticality difference of software, the test request of software will be different) in the test contract of software testing plan document or software.Provide the step of white-box testing adequacy tolerance below.

1. the requirement of software dialogue box test-types is that X covers.

2.X the coverage rate that covers is

, the type of code definition sees Table lattice 2.

3. white-box testing adequacy S _White=Coverage _X

Type of code definition in form 2 white-box testings

Test-types	Total code	Run time version	Unreachable code
				Statement covers	The sum of executable statement	The executable statement number that is performed	Unreachable statement number
Branch covers	Total branches order	The number of branches of carrying out	Unreachable number of branches
				Condition covers	Total logical condition number	The number of the logical condition of carrying out	Unreachable logical condition number
The path covers	The total path number	The path number of carrying out	Unreachable path number

The layout strategy that the adequacy of Black-box Testing is taked based on test case is considered.The present invention comes software for calculation Black-box Testing adequacy according to Fig. 2, and computation process is as follows.

1. at first, provide the adequacy of leaf node (test use cases).

A, provide the Test Strategy { method that certain test use cases is selected for use ₁, method ₂..., method _n, suppose that the weight of each selected Test Strategy is identical.

B, judge whether every kind of given Test Strategy has omission, thereby provide adequacy S the Test Strategy of this test use cases design _{Tcset_method}

Form 3 adequacy table of gradings

The adequacy grade	Insufficient	More abundant	Fully	Very abundant
					Adequacy numerical value	??[0， ??0.7]	??(0.7， ??0.8]	??(0.8， ??0.9]	??(0.9， ??1.0]

√ is not if omit the Test Strategy of this test use cases design, then thinks S _{Tcset_method}=1;

√ according to the importance of the Test Strategy of omitting, provides the adequacy grade S of Test Strategy if omission is arranged according to form 3 _{Tcset_method}

√ based on the consideration of risk, advises S if Test Strategy is only selected " other " _{Tcset_method}Select 0.5.

C, provide adequacy criterion, and provide every kind of tactful method according to this criterion to every kind of Test Strategy of this test use cases design _iThe adequacy of (i ∈ [1, n]) $\{S_{{\mathrm{method}}_1},S_{{\mathrm{method}}_2},...,S_{{\mathrm{method}}_n}\}.$ Form 4 provides Black-box Testing strategic equivalence class commonly used and divides boundary value analysis, the description of cause-and-effect diagram and coverage rate computing method.

D, in conjunction with above analysis result, calculate this test use cases Black-box Testing adequacy according to following tolerance formula: $S_{\mathrm{black}\_\mathrm{tcset}}=S_{\mathrm{tcset}\_\mathrm{method}}*(1/n)*{\mathrm{\Σ}}_{i=1}^n{S}_{{\mathrm{method}}_i}$

Form 4 Black-box Testing strategies commonly used cover counting rate meter

2. certain non-leaf node x has n child, and the weight of each child is p _i, corresponding adequacy grade is s _i, then this non-leaf node adequacy is evaluated as $S_x={\mathrm{\Σ}}_{i=1}^n{p}_i{s}_i.$

3. the rest may be inferred, can get the adequacy S of software Black-box Testing _Black

In conjunction with more than the white-box testing adequacy that obtains and the metric function of Black-box Testing adequacy, the present invention provides software test case adequacy metric function and is: S=S _Tcset=S _White* S _Black

Wherein, S is the degree of confidence of software quality evaluation mark, S _TcsetBe the adequacy of software test case, S _WhiteBe software white-box testing adequacy, S _BlackBe software Black-box Testing adequacy.The present invention defines the degree of confidence that the software test case adequacy is the software quality evaluation mark.In engineering is used,, can suppose that this adequacy is defaulted as 1 in real process if white-box testing adequacy or Black-box Testing adequacy are difficult to obtain.

Step 6: determine software quality comprehensive evaluation at last

The present invention joins the degree of confidence factor of software quality evaluation in the software quality evaluation system, provides software quality comprehensive evaluation formula: E in conjunction with the software quality evaluation mark _Total=E _Quality* S

Wherein, E _TotalRepresent software quality comprehensive evaluation mark, E _QualityBe the software quality evaluation mark, S is the degree of confidence of software quality evaluation mark.From this formula as can be known, the comprehensive evaluation of software quality depends on this software quality is estimated credibility with the gained evaluation score.

The weight of the test use cases of each quality sub-feature is provided according to the significance level of each test case set pair software quality by the tester, and the AHP method is adopted in the establishment of the weight of quality sub-feature, mass property, and its content introduction sees Appendix 2.

Annex 1: test use cases definition

Test use cases typically refers in order to test or the set of the test case of a certain function of verifying software or test item.And test case is to be used for describing certain specific test case, is made up of test data and associated test procedure usually.

Test use cases has five attribute factors among the present invention, promptly test case pass through quantity X, by quantity Y, the software problem reporting of discovery is not counted P, the otherness D of test case, the cardinal B of test case, with TS (X, Y, P, D B) represents.Wherein B is the radix of test use cases, refers to the concentrated test case number that will comprise at least of test case.

The application of annex 2:AHP method in software quality evaluation

1). set up and pass judgment on criterion

Adopt Paired Comparisons, the judge criterion such as the following table of foundation:

Form 5 is passed judgment on the criterion table

Scale	Implication
		??1	Represent that two factors compare, have equal importance
??3	Represent that two factors compare, the former is more important slightly than the latter
		??5	Represent that two factors compare, the former is obvious more important than the latter
??7	Represent that two factors compare, the former is quite more important than the latter
		??9	Represent that two factors compare, the former is extremely more important than the latter
??2，4，6，8	The intermediate value of adjacent judgement

2). the input judgment matrix

After having determined comparison criterion, need to compare of the influence of several factors, thereby determine the proportion that they account in target same target.An elements A of hypothetical target level is to the elements A of next level ₁, A ₂..., A _iDominance relation is arranged, can set up the comparator matrix in twos between each element so.Like this for mass property A _iJust can obtain the comparator matrix in twos on n rank: S=(s _Ij) _n

3). weight analysis extracts

After step is on top finished, obtained judgment matrix S, the extraction of judgment matrix being carried out weight obtains absolute weight vectors W`={W <sub>1</sub>`, W ₂` ..., W <sub>n</sub>`} ^T, extracting method generally adopts the characteristic root method, and solution formula is as follows: SW`=λ <sub>Max</sub>W`.λ wherein _MaxBe the maximum characteristic root of matrix S, W` is exactly corresponding proper vector, and W` is carried out normalized, promptly obtains weight vectors W.Same method, the Quantum Properties of also can confronting is handled, the structure judgment matrix.

4). consistency check

If comparative result is on all four before and after being, then judgment matrix should have following character: (1) s _Ij＞0, (2) $s_{\mathrm{ij}}=\frac{1}{s_{\mathrm{ji}}},$ (3) s _Ii=1; Obtain such n rank matrix, must do n* (n-1)/2 judgement, because the judgement of mass property is had subjectivity and complicacy, so might not make judgment matrix satisfy s _Ij* s _Jk=s _Ik, just judgment matrix S not necessarily has consistance, so after obtaining maximum characteristic root, do consistency check to judgment matrix.Need whether seriously non-unanimity of judgment matrix S that check constructs, so that determine whether to accept S.

Definition 1: satisfy relational expression s _Ij* s _Jk=s _Ik, $\&ForAll;i,j,k\&Element;\mathrm{1,2,3},...,n$ Just reciprocal matrix be called consistent matrix.

Theorem 1: the maximum characteristic root λ of just reciprocal matrix S _MaxMust be arithmetic number, the important arithmetic number that is of its character pair vector.The mould of all the other eigenwerts of S is all strict with λ _Max

Theorem 2: if S is consistent matrix, then:

(1) S must be just reciprocal matrix.

(2) the transposed matrix AT of S also is consistent matrix.

(3) any two row of S are proportional, and scale factor is greater than zero, thus rank (S)=1 (same, any two row of S are also proportional).

(4) the eigenvalue of maximum n=λ of S _Max, wherein n is the rank of matrix S.All the other characteristic roots of S are zero.

(5) if the eigenvalue of maximum λ of S _MaxThe characteristic of correspondence vector is W=(w ₁, w ₂, w ₃..., w _n) ^T, s then _Ij=w _i/ w _j, $\&ForAll;i,j\&Element;\mathrm{1,2,3},...,n,$ That is:

$s=\left(\begin{array}{cccc}\frac{w_1}{w_1}& \frac{w_1}{w_2}& \&CenterDot;\&CenterDot;\&CenterDot;& \frac{w_1}{w_n}\\ \frac{w_1}{w_1}& \frac{w_1}{w_2}& \&CenterDot;\&CenterDot;\&CenterDot;& \frac{w_1}{w_n}\\ \&CenterDot;\&CenterDot;\&CenterDot;& \&CenterDot;\&CenterDot;\&CenterDot;& \&CenterDot;\&CenterDot;\&CenterDot;& \&CenterDot;\&CenterDot;\&CenterDot;\\ \frac{w_n}{w_1}& \frac{w_n}{w_2}& \&CenterDot;\&CenterDot;\&CenterDot;& \frac{w_n}{w_n}\end{array}\right)$

The just reciprocal matrix S in theorem 3:n rank is consistent matrix and if only if its maximum characteristic root n=λ _Max, and when the non-unanimity of just reciprocal matrix A, λ must be arranged _Max＞n.

According to theorem 3, can be by λ _MaxWhether equal n and check whether judgment matrix S is consistent matrix.Because characteristic root depends on s continuously _IjSo, λ _MaxThan n is big must be many more, the nonuniformity degree of S is also just serious more, λ _MaxCorresponding standardized feature vector also just can not reflect X={x more truly ₁..., x _nShared proportion in influence to factor.Therefore, the judgment matrix that the decision maker is provided is necessary to do a consistency check, whether can accept it with decision.

Step to the consistency check of judgment matrix is as follows:

(i) calculate coincident indicator CI: $C.I=\frac{{\mathrm{\&lambda;}}_{\max }-n}{n-1}$

(ii) search corresponding mean random coincident indicator RI.To n=1 ..., 9, Saaty has provided the value of RI, and is as shown in the table:

Form 6 average homogeneity index RI

The value of RI obtains like this, with 500 sample matrix of random device structure: randomly from 1～9 and inverse extract the just reciprocal matrix of numeral structure, try to achieve the mean value λ of maximum characteristic root _Max`, and definition $R.I=\frac{{\mathrm{\&lambda;}}_{\max }^{`}-n}{n-1}$

(iii) calculate consistance ratio CR: $\mathrm{CR}=\frac{\mathrm{CI}}{\mathrm{RI}}$ 。When CR＜0.10, think that the consistance of judgment matrix is an acceptable, otherwise the reply judgment matrix is done suitably to revise.

5). level always sorts and consistency check

What obtain above is the weight vectors of a group element to certain element in its last layer.Finally to obtain in each element, particularly lowermost layer each scheme for the ordering weight of target, thereby carry out Scheme Choice.Total ordering weight wants from top to down that the weight under the single criterion is synthesized.

If last layer time (A layer) comprises A ₁..., A _mM factor altogether, their level weight that always sorts is respectively a ₁..., a _mNext level (B layer) of establishing again thereafter comprises n factor B ₁..., B _m, they are about A _jThe single preface weight of level be respectively b _1f..., b _Nf(work as B _iWith A _jDuring onrelevant, b _Ij=0).Ask in the B layer each factor about the weight of general objective, i.e. the level of each factor of the B layer weight b that always sorts ₁..., b _n, the mode shown in the according to the form below of calculating is carried out, promptly $b_i=\underset{j=1}{\overset{m}{\mathrm{\&Sigma;}}}{b}_{\mathrm{ij}}*a_j,$ i＝1，…，n。

The total sequencing table of form 7 levels

Total ordering also need be done consistency check to level, and check is still successively carried out to low layer by high level as the total ordering of level.

If in the B layer with A _jThe paired relatively judgment matrix of relevant factor has been finished consistency check in single preface, trying to achieve single preface coincident indicator is CI (j), (j=1, m), corresponding mean random coincident indicator is RI (j) (CI (j), RI (j) try to achieve when the single preface of level), then the B layer consistance ratio at random that always sorts: $\mathrm{CR}=\frac{\underset{j=1}{\overset{m}{\mathrm{\&Sigma;}}}\mathrm{CI}\left(j\right)a_j}{\underset{j=1}{\overset{m}{\mathrm{\&Sigma;}}}\mathrm{RI}\left(j\right)a_j}$

When CR＜0.10, think that the total ranking results of level has satisfied consistance and accepts this analysis result.

For certain true project A, according to software feature and quality requirements, set up corresponding quality model, and carried out quality assessment by this method based on test data.The following evaluation of detailed process:

(1) .A project quality model customization

Mass property	The quality sub-feature
		Functional	Adaptability, security, easily installation property, interoperability, border characteristic
Performance	Performance Characteristics, strength characteristics, capacity characteristic

(2). the mass property weight

According to the application (annex 2) of AHP method in software quality evaluation, by filling in the relative importance between the mass property, the absolute weight information that obtains mass property is as follows:

Mass property	Functional	Performance
			Functional	??1.0	??3.0
Performance	??0.333333	??1.0
			Absolute weight information	??0.75	??0.25

(3). quality sub-feature weight

According to the application (annex 2) of AHP method in software quality evaluation,, obtain the absolute weight information of quality sub-feature by filling in the relative importance between the quality sub-feature.

Functional quality sub-feature weight analysis:

The quality sub-feature	Adaptability	Security	Easily installation property	Interoperability	The border characteristic
						Adaptability	??1.0	??1.0	??1.0	??1.0	??1.0
Security	??1.0	??1.0	??1.0	??1.0	??1.0
						Easily installation property	??1.0	??1.0	??1.0	??1.0	??1.0
Interoperability	??1.0	??1.0	??1.0	??1.0	??1.0
						The border characteristic	??1.0	??1.0	??1.0	??1.0	??1.0
Absolute weight information	??0.2	??0.2	??0.2	??0.2	??0.2

With the quality sub-feature weight that can determine performance with quadrat method.

(4). the test use cases weight design

The test use cases of design acquiescence:

Name	Weight	Diversity factor	Basis use-case number
				Test use cases	??1.0	??1.0	??1

Also can be separately weight be set for the test use cases of each quality sub-feature respectively.

(5). implementation evaluation

Below by certain true project A is carried out software test, obtain test result data, comprise the failure number of test case, by number, software issue number, information such as the software issue order of severity, and calculate the mark of each test use cases as stated above.

By the mark of test use cases, the mark that can draw the quality sub-feature in conjunction with test use cases proportion in affiliated quality sub-feature is as follows:

The quality sub-feature	Adaptability	Security	Easily installation property	Interoperability	The border characteristic	Performance Characteristics	Strength characteristics	Capacity characteristic
									Evaluation score	??61.7	??41.18	??67.52	??81.82	??62.5	??100	??0.0	??20.1

By the mark of quality sub-feature, it is as follows that bond quality sub-feature proportion in affiliated mass property can draw the mark of mass property:

Mass property	Functional	Performance
			Evaluation score	??62.94	??39.6

By the mark of mass property, it is as follows that bond quality characteristic proportion in the evaluation of software oeverall quality can draw the mark of software quality:

The software quality mark

??57.11

Because this project is not carried out white-box testing, fails to obtain the white-box testing adequacy, supposes that the white-box testing adequacy is 1.The Black-box Testing aspect, the Black-box Testing adequacy computation process with test use cases under the quality sub-feature " adaptability " is the Black-box Testing adequacy that example provides this test use cases below.

1. the Test Strategy selected for use of this test use cases={ equivalence class is divided, boundary value analysis };

2. the adequacy S of the Test Strategy of this test use cases design _{Tcset_method}=1;

3. according to form 3, the adequacy that provides every kind of Test Strategy for 0.95,0.93};

4. this test use cases Black-box Testing adequacy is

$S_{\mathrm{black}\_\mathrm{tcset}}=S_{\mathrm{tcset}\_\mathrm{method}}*1/2*{\mathrm{\&Sigma;}}_{i=1}^2{S}_{{\mathrm{method}}_i=1*1/2*(0.95+0.93)=0.94.}$

The Black-box Testing adequacy computation process of all the other test use cases is similar.At last obtaining project A software Black-box Testing adequacy according to the computation process of the step 5 Black-box Testing adequacy of instructions is 0.98.

Thereby, project A software quality evaluation degree of confidence S=S _White* S _Black=1*0.98=0.98.Project A software quality comprehensive evaluation mark is 57.11*0.98=55.9678.

Above-mentioned example has carried out complete software quality evaluation to true project A.Example A provides the quality model that satisfies this project demands based on the three-decker of software quality models in the international standard ISO/IEC9126 standard at detailed programs A, thereby has realized the customizable of software quality models.

Example is being considered under the situation of test case by number, failure number, problem report number, test case importance, software problem reporting seriousness factor, calculate the mark of test use cases, the mark of quality sub-feature, the mark of mass property and the mark of software quality, and in conjunction with the degree of confidence of software test case, comprehensively provide the software quality evaluation mark, estimated the quality level of software quantitatively.The software quality mark that the A project draws at last is 55.9678, and this evaluation result has reflected the quality condition of A project software.According to the quality requirements of A project, can be improved software, repair associated disadvantages, thereby improve the quality of software.The evaluation of above-mentioned example by software quality is quantized, thus for improving the quality of software, finally provide objective, just scientific basis for software typing, examination, evaluation and industrialization.

Obviously, content of the present invention described here can have many variations, and this variation can not be thought and departs from the spirit and scope of the present invention.Therefore, the change that all it will be apparent to those skilled in the art all is included within the covering scope of these claims.

Claims (1)

1, a kind of method for evaluating software quality based on test data is characterized in that step is as follows:

Step 1: determine the test use cases evaluation

Test use cases is divided into functional test set of uses case and non-functional test use cases two classes, and each test use cases belongs to functional test set of uses case or non-functional test use cases; If the affiliated mass property of quality sub-feature is functional under this test use cases, then this test use cases is the functional test set of uses case, otherwise is the non-functional test use cases;

Described functional test set of uses case is evaluated as: $S_T=\frac{N_1*D-\underset{j=1}{\overset{L}{\mathrm{\&Sigma;}}}{p}_j*x_j}{N_1*D},$ Wherein, N ₁Represent actual test case number, when existing a test case to cause the situation of a plurality of software issues, regard this test case as with software issue number respective amount a plurality of test cases, it calculates by the correspondence problem number when calculating the use-case sum; L represents the sum of software issue, p _jThe seriousness degree of representing j software issue, x _jRepresent the importance degree of the corresponding test case of j software issue, D represents the test use cases diversity factor, 0≤D≤1;

Described non-functional test use cases is evaluated as: $S_T=\frac{N_2*D-M}{N_2*D},$ Wherein, N ₂Represent test case to concentrate the sum of the test case of design, M represents the quantity of unsanctioned test case, and on behalf of test case, D concentrate the diversity factor of test case, 0≤D≤1;

Step 2: determine the evaluation of software quality sub-feature

The software quality sub-feature is evaluated as: $M_s=\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{Q}_i*S_{T_i},$ Wherein, suppose n test use cases, Q are arranged in quality sub-feature bottom _iBe the weight of each test use cases correspondence, wherein $\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{Q}_i=1,$

For each test use cases is estimated, if mass property is functional under this quality sub-feature, then

Calculate according to functional test set of uses case judgement schematics in the step 1, if mass property is a non-functional under this quality sub-feature, then Calculate according to non-functional test use cases judgement schematics in the step 1;

Step 3: determine the software quality characteristics evaluation

Software quality characteristics is evaluated as: $M_c=\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{R}_i*M_{\mathrm{si}},$ Wherein, supposing has n quality sub-feature in mass property bottom, and the evaluation score of i quality sub-feature is M _Si, to calculate according to step 2 software quality sub-feature judgement schematics, the weight of i quality sub-feature correspondence is R _i, wherein $\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{R}_i=1;$

Step 4: software quality evaluation

Software quality evaluation is: $E_{\mathrm{quality}}=\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{R}_i*M_{\mathrm{ci}},$ Wherein, suppose to have n mass property below software quality, the evaluation score of i mass property is M _Ci, to calculate according to step 3 software quality sub-feature judgement schematics, the weight of i mass property correspondence is R _i, wherein $\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{R}_i=1;$

Step 5: determine the software quality evaluation confidence scores

S＝S _tcset＝S _white*S _black

Wherein, S is the software quality evaluation confidence scores, S _TcsetBe the adequacy of software test case, S _WhiteBe software white-box testing adequacy, S _BlackBe software Black-box Testing adequacy;

White-box testing adequacy S _White=Coverage _XThe requirement of supposing software dialogue box test-types is that X covers, and then the coverage rate of X covering is

Black-box Testing adequacy deterministic process is as follows:

(1) at first, provide leaf node, i.e. the adequacy of test use cases;

(2) certain non-leaf node x has n child, and the weight of each child is p _i, corresponding adequacy grade is s _i, then this non-leaf node adequacy is evaluated as $S_x={\mathrm{\&Sigma;}}_{i=1}^n{p}_i{s}_i;$

(3) the rest may be inferred, can get the adequacy S of software Black-box Testing _Black

Step 6: determine software quality comprehensive evaluation

The degree of confidence factor of software quality evaluation is joined in the software quality evaluation system, provide software quality comprehensive evaluation formula: E in conjunction with the software quality evaluation mark _Total=E _Quality* S

Wherein, E _TotalRepresent software quality comprehensive evaluation mark, E _QualityBe the software quality evaluation mark, S is the degree of confidence of software quality evaluation mark.

Images