1

Software Quality Assurance

Đảm bảo chất lượng phần mềm
Lecture 1: Introduction to SQA

2
Outline
• Basic concepts and terms
• Definition of Software Quality
• Quality Models
• Software Quality Assurance Process

3
1.1. Basic concepts

4
Basic concepts and terms
Các khái niệm và thuật ngữ cơ bản
• Verification vs Validation:
• Verification (to verify – xác minh): Determining whether the
product of a given development phase satisfies the requirements
established before the start of this phase.
• Verify if the system is built right
• Validation (to validate – kiểm chứng): Confirming that a product
meets its customer’s expectation
• Validate if we built a right system

5
Failure– Fault – Error – Defect
(Lỗi)
• Failure: is an event that the behavior of a system does not
conform to that prescribed in the system specification
• Failure – failed function caused by an error
• Error: is an erroneous state of the software
• Faulty state
• Fault: is the adjudged cause of an error
• Human error: mistake by engineer
• Fault/Defect/Bug: anomaly in the software
• Debugging – Fault localizing (Sửa lỗi hay xác định lỗi)

6
1.2. Software Quality

7
Five views of Quality
5 quan điểm về chất lượng sản phẩm nói chung
• Transcendental view (Quan điểm siêu việt về chất
lượng): quality is something we can recognize but not
define
• User view: quality is fitness for purpose
• Manufacturing view: quality is conformance to
specification
• Product view: quality is tied to inherent product
characteristics
• Value-based view (Economic): quality depends on the
amount the customer is willing to pay for it

8
Software Quality
Chất lượng phần mềm là gì?
• In general, quality means that a product should meet its
specifications

9
Problems in Software Quality Assurance

• A tension between:
• Customer quality requirements (efficiency, reliability etc.)
• Developer quality requirements (reusability, maintainability, etc.)
• Software specifications are usually incomplete and often
inconsistent
• Some quality requirements are difficult to specify in an
unambiguous way

10
1.3. Software quality model

11
Software Quality Model
Các mô hình chất lượng phần mềm
• Such general definitions of software quality are not
sufficient in practice
• Thus, software quality is described by specific quality
models
• Two main approaches:
• Standard Models: McCall, ISO/IEC 9126, ISO/IEC 25000, Boehm,
Dromey
• Application or company specific quality models: FURPS, GQM
Approach

12
Factors – Criteria – Metrics
Nhân tố – Tiêu chí – Độ đo
• Each quality model contains:
• Factors (to specify): describe the external view of the software, as
viewed by users
• Criteria (to build): describe the internal view of the software, as
seen by developers. Considered as characteristics which define the
quality factors
• Metrics (to control): defined and used to provide a scale and
method for measurement

13
tor-Criteria-Metrics-Model
Factors – Criteria – Metrics in Software
Quality Model
ation into :
rs (to specify): They
be the external view Software Quality
software, as viewed
users.
Quality Quality Quality
ia (to build): They factor 1 factor 2 factor 3
be the internal view
software, as seen by
veloper.
Quality Quality Quality
cs (to control): They criterion 1 criterion 2 criterion 3
fined and used to
e a scale and
d for measurement. Quality indicators (metrics)

14
e WS04/05 – Software Quality Assurance – I Introduction I-31
McCall’s Quality Factors
• A quality factor represents a behavioral characteristic of
the system
• 3 groups
• Product Operation
• Product Revision
• Product Transition
• 11 quality factors

15
McCall’s Factor Model Tree
Cây nhân tố chất lượng của mô hình McCall

Source: from Internet

16
11 quality factors of McCall’s Model
11 nhân tố chất lượng của mô hình McCall

Source: Software
Testing and Quality Assurance
Theory and Practice

17
McCall’s Quality Criteria
Các tiêu chí chất lượng của mô hình McCall
• A quality criterion is an attribute of a quality factor that is
related to software development
• Ex: Modularity is a quality criterion related to the quality factor
maintainability
• Traceability is a quality criterion related to the correctness of a
system
• There are 23 quality criteria of McCall’s Model

18
 23 quality criteria of McCall
23 tiêu chí chất lượng đưa ra bởi McCall
Quality Criteria Definition
Access audit Ease with which software and data can be checked for
(kiểm tra truy nhập) compliance with standards or other requirements
Access control Provisions for control and protection of the software and data
(kiểm soát truy nhập)
Accuracy Precision of computations and output
(Tính chính xác)
Communication Degree to which standard protocols and interfaces are used
commonality
(Độ tương đồng giao
tiếp)
Completeness Degree to which a full implementation of the required
(Tính đầy đủ) functionalities has been achieved

19
 23 quality criteria of McCall
23 tiêu chí chất lượng đưa ra bởi McCall
Quality Criteria Definition
Communicativeness Ease with which inputs and outputs can be assimilated
(Sự sẵn sàng giao tiếp)
Conciseness Compactness of the source code, in terms of lines of code
(Tính xúc tích)
Consistency Use of uniform design and implementation techniques and
(Tính nhất quán) notation throughout a project
Data commonality Use of standard data representations
(Độ tương đồng dữ
liệu)
Error tolerance Degree to which continuity of operation is ensured under
(Độ dung thứ lỗi) adverse conditions
Execution efficiency Run time efficiency of the software

20
 23 quality criteria of McCall
23 tiêu chí chất lượng đưa ra bởi McCall
Quality Criteria Definition
Expandability Degree to which storage requirements or software functions
(Khả năng mở rộng) can be expanded
Generality Breadth of the potential application of software components
(Tính khái quát hoá)
Hardware Degree to which the software is dependent on the
independence underlying hardware
(Độc lập phần cứng)
Instrumentation Degree to which the software provides for measurement of
(Thiết bị đo đạc) its use or identification of errors
Modularity Provision of highly independent modules
(Tính đầy đủ)
Operability Ease of operation of the software
(Khả năng vận hành)

21
 23 quality criteria of McCall
23 tiêu chí chất lượng đưa ra bởi McCall
Quality Criteria Definition
Self-documentation Provision of in-line documentation that explains implementation
(Tự đặc tả tài liệu) of components
Simplicity Ease with which the software can be understood
(Tính đơn giản)
Software system Degree to which the software is independent of its software
independence environment—nonstandard language constructs, operating
(Độc lập phần mềm) system, libraries, database management system, etc.
Software efficiency Run time storage requirements of the software
(Hiệu năng phần mềm)
Traceability Ability to link software components to requirements
(Khả năng theo vết)
Training Ease with which new users can use the system
(Đào tạo)

22
 Testing and Quality Assurance
Theory and Practice
Relationship between Quality factors and criteria

23

528 CHAPTER 17 SOFTWARE QUALITY

Source: Software
Mối quan hệ giữa nhân tố và tiêu chí chất lượng

Quality criteria Quality factors

Traceability
Completeness Correctness
Consistency
Accuracy Reliability
Error tolerance
Execution efficiency
Efficiency
Storage efficiency
Access control
Integrity
Access audit
Operability
Training Usability
Communicativeness
Simplicity
Conciseness Maintainability
Instrumentation
Self-descriptiveness Testability
Expandability
Flexibility
Generality
Portability
Modularity
Software–system independence Reusability
Machine independence
Interoperability
Communications commonality
Data commonality
Figure 17.1 Relation between quality factors and quality criteria [6].
• If an effort is made to improve one quality factor, another quality factor
may be degraded. For example, if an effort is made to make a software
product testable, the efficiency of the software is likely to go down. To
make code testable, programmers may not be able to write compact code.
Relationship between Quality factors and criteria
Mối quan hệ giữa nhân tố và tiêu chí chất lượng
• 2 characteristics of relationship:
• If an effort is made to improve one quality factor, another quality
factor may be degraded
• Make software testable è the efficiency of the software is likely to
go down
• Make the product portable èreduce the efficiency
• Some quality factors positively impacts others
• an effort to enhance the correctness of a system will increase its
reliability
• an effort to enhance the testability of a system will improve its
maintainability

Source: Software
Testing and Quality Assurance
Theory and Practice

IT4501 - Software Engineering Department -

6/19/22 24
SoICT/HUST
Quality Metrics
Độ đo chất lượng
• A quality metric is a measure that captures some aspects
of a quality criterion
• The metrics can be derived as:
• Formulate a set of relevant questions concerning the quality
criteria and seek a yes or no answer for each question
• Divide the number of yes answers by the number of questions
to obtain a value between 0 and 1
• Example: self-descriptiveness
• Is all documentation written clearly and simply such that
procedures, functions and algorithms can be easily understood?
• Is the design rationale behind a module clearly understood?

6/19/22 25
Boehm's Quality Model
Mô hình chất lượng của Boehm
• Attempts to automatically and qualitatively evaluate the
quality of software
• High-level characteristics address three classifications:
• Utility
• Maintainability
• Portability
• Intermediate-level characteristics address 7 quality factors

6/19/22 26
7 quality factors of Boehm’s Model
Factors Criteria

Portability Self-contentedness, device independence

Self-contentedness, accuracy, completeness,

Reliability
robustness, integrity, consistency

Efficiency Accountability, device efficiency, accessibility

Usability Completeness

Testability (Human Accountability, communicativeness, self

Engineering) descriptiveness, structured

Understandability Consistency, structured, conciseness

Modifiability
Structured, expandability
(Flexibility)

6/19/22 27
Dromey’s Quality Model
• Dromey quality model proposed a framework to evaluate
requirement, design and implementation phases
• High-level product properties for the implementation
quality model include:
• Correctness
• Internal
• Contextual
• Descriptive

6/19/22 28
Dromey’s quality factors and criteria
Các nhân tố và tiêu chí chất lượng của mô hình Dromey

Factors Criteria

Correctness Functionality, Reliability

Internal Maintainability, Efficiency, Reliability

Maintainability, Reusability, Portability,

Contextual
Reliability

Maintainability, Efficiency, Reliability,

Descriptive
Usability

6/19/22 29
 The Six Quality Characteris
ISO/IEC 9126 Quality Model
a chất
Mô hình Software (ISO/IEC
lượng ISO/IEC 9126 9126)
Software qua
features and
software prod
ability to satis
needs. (ISO 9
Software qua
A set of attrib
product by wh
described and
software qual
be refined into
sub-character
1991, 3.13)

Each charact
set of sub-cha
Each sub-cha
Source: from Internet
evaluated by
Some metrics
several sub-c
[ISO/IEC 9126]
6/19/22 30

Dr. Holger Giese WS04/05 – Software Quality Assurance – I Introduction

 532 CHAPTER 17 SOFTWARE QUALITY
31
20 quality sub characteristics of ISO/IEC

Quality characteristic Quality subcharacteristics

Suitability
Accuracy
Functionality
Interoperability
Security
Maturity
Reliability Fault tolerance
Recoverability
Understandability
Usability Learnability
Operability
Time behavior
Efficiency
Resource behavior
Analyzability
Changeability
Maintainability
Stability
Testability
Adaptability
9126
Installability
Portability
Conformance

6/19/22
Replaceability
Figure 17.2 ISO 9126 sample quality model refines standard’s features into
subcharacteristics. (From ref. 4.  1996 IEEE.)
Comparison of McCall and ISO/IEC 9126
So sánh hai mô hình (sự tương đồng)
• Quality factor of McCall model is in fact quality
characteristic in ISO/IEC model
• Quality criterion of McCall is sub-quality-characteristic of
ISO/IEC model
• Some high-level quality factors/characteristics are found in
both models
• reliability
• usability
• efficiency
• maintainability
• portability

6/19/22 32
Comparison of McCall and ISO/IEC 9126
So sánh hai mô hình (sự khác nhau)
• ISO 9126 model emphasizes characteristics visible to
users whereas McCall model considers internal qualities
• Ex: Developpers produce reusable components whereas its impact
is not perceived by customers
• In McCall’s model, one quality criterion can impact several
quality factors whereas in the ISO 9126 model, one
subcharacteristic impacts exactly one quality characteristic
• A high-level quality factor, testability, in the McCall model
is a low-level subcharacteristic of maintainability in the
ISO 9126 model

6/19/22 33
Hewlett Packard: F.U.R.P.S
Mô hình FURPS của HP
• Result of a statistical project survey at Hewlett Packard
1987 to improve its products:
• Factors:
• Functionality: functions it performs, their generality and security
• Usability: consistency, documentation
• Reliability: frequency and severity of failure, accuracy of output
• Performance: response time, resource consumption
• Supportability: can it be extended, adapted, corrected?
• FURPS is originally a company specific quality model

6/19/22 34
Factors and Criteria of FURPS

Factors Criteria

Functionality Capability, security

Consistency, user documentation, training

Usability
materials

Frequency and security of failure, recoverability,

Reliability
predictability, accuracy, mean time between failure

Speed efficiency, availability, accuracy,

Performance throughput, response time, recovery time,
resource usage
Testability, extensibility, adaptability,
Supportability maintainability, compatibility, configurability,
serviceability, install ability, localizability

6/19/22 35
GQM: Goal-Question-Metric
Mô hình GQM
• A measurement program can be more successful if
designed with the goals in mind
• GQM approach provides a framework with 3 steps:
• List the major goals of the development/maintenance project
• Derive from each goal the questions that must be answered to
determine if the goals are being met
• Decide what must be measured to answer the questions
adequately

6/19/22 36
 University of Paderborn
Software Engineering Group

Example
Exampleof GQM
GOAL: Evaluate effectiveness of coding standard

QUESTIONS: Who is using What is coder What is code

Standard? productivity? quality?

METRICS:
Proportion of Experience of Errors
Code
Coders Coders Effort
size
-using standard -with standard
-using language -with language
-with environment
etc

Dr. Holger Giese WS04/05 – Software Quality Assurance – I Introduction I-39

6/19/22 37
GQM Discussion
Ưu điểm và nhược điểm của mô hình GQM
• Benefits:
• generates only those measures relevant to the goal
• several measurements may be needed to answer a single question
• a single measurement may apply to more than one question
• the goal provides the purpose for collecting the data
• Not evident from the GQM
• The model needed to combine the measurement in a sensible way so
that the question can be answered
• It must be supplemented by one or more models that express the
relationship among metrics
• Disadvantages:
• Additional efforts to derive the goals and metrics
• Error prone compared to standard models

6/19/22 38
1.4. Quality Assurance Process
Quy trình đảm bảo chất lượng phần mềm
“Quality Assurance is any systematic process of checking to see
whether a product or service being developed is meeting its
specified requirements”

6/19/22 39
 QA Process
• Need to be integrated with any other
processes in the development life
cycle of the software system
• Project management process
• Code Development process
• Requirements Management process
• Configuration Management process

6/19/22 40

40
QA Dependencies – Project Management
Vai trò của QA trong quá trình quản lý dự án
• QA should be considered in planning stage
• Define quality goals & criteria
• Identify applicable QA/testing procedures
• Pre-assign necessary resources (including personnel,
hardware and infrastructure environment)

6/19/22 41
QA Dependencies - Requirements Management
Vai trò của QA trong quản lý yêu cầu phần mềm
• Keep product requirements in a well organized way
• To identify all important aspects of functionality and receive
customer's confirmation
• To define restrictions and exceptions
• To establish common understanding across the entire project team
• To provide basis for project estimates, code development and
validation processes
• To be a starting point for effective communication and change
management during product evolution process

6/19/22 42
QA Dependencies - Configuration Management
QA đối với quy trình quản lý cấu hình
• Configuration Management presents on any software project
• To allow distributed teams simultaneously to work on the same
code
• To keep the code baseline free from breakages introduced by bug
fixing
• To know code version where new error appeared for the first time
and to identify what change had introduced it
• To realize all advantages of automated testing on each and every
code version published

6/19/22 43
QA Dependencies - Code Development
Vai trò của QA đối với quy trình phát triển mã nguồn
• Benefits of applying a Code Development Process
• Unified coding style suitable for further product maintenance and
updates without single team member lock-in
• Earliest breakage detection and the most effective cost on fixing bug
• Test-oriented code design
• Localize errors and minimize impacts on dependent product
components
• Share knowledge between team members and Improve personal
skills

6/19/22 44
 QA Core Components - Testing Process
Quy trình kiểm thử là nền tảng của quy trình QA

• Testing is implemented in
separate and/or overlapped test
phase/types
• Testing on different product
levels: unit, feature, features
interaction, system
• Testing during product
maintenance: smoke, sanity,
regression
• Testing of special aspects:
bring-up, stress, stability,
performance,
interoperability, compliance
 Test Management
Quản lý kiểm thử trong quy trình QA
• Perform from the very beginning through the end of a project to
achieve goals of improved product quality
• Major Objects of Test Management
• Testing scope / strategy
• Approaches and Constraints
• Features / Components / Functional Areas to be or not to be tested
• Test Item Pass / Fail and Test Cycle Suspension / Resumption Criteria
• Test Tasks and Deliverables, Environmental Needs
• Responsibilities, Staffing and Training Needs / Collaboration with other
project groups
• Schedule, Risk and Contingencies
• Reporting and Quality Metrics

6/19/22 46
 Test Preparation
Chuẩn bị kiểm thử trong quy trình QA
• Define tests and test environment required for each test
phase/type according to test plans
• Basic test preparation activities
• Analysis of functional requirements, specifications and functional
areas's
• Development of Requirements Traceability Matrix (RTM)
• Design and Development of the test cases for all planned phases/types
• Using approaches and techniques to ensure effective and optimal test
suites
• Various test approaches (Black Box, White Box)
• Effective test techniques: Functionality, Equivalent classes, Boundary,
Negative/Positive Combinatorial..
• Development of additional tools/utilities and Test automation
• Development and setup of specific test environment

6/19/22 47
 Test Execution
Thực thi kiểm thử trong quy trình QA
• Straightforward apply test definitions elaborated during the
Test Preparation Phase to the software product
• Usual Test Execution activities:
• Preparation of developed test environment
• Execution of test cases
• Deep analysis of execution results
• Collecting/Recording of test execution results and failures found
• Support development team to reproduce and to analyze failures
• Validation of fixes
• Analysis of escaped defects
• Update test cases/test environment according to the results of Test
Execution/Escaped Defect Analysis

6/19/22 48
Summary
Tóm lược lại nội dung bài học
• Principal terms of Quality Assurance
• IEEE’s definition of Software Quality
• Quality models: McCall, Boehm, Dromey, ISO/IEC 9126,
FURPS, GQM
• QA process and main activities of:
• Test Preparation
• Test Management
• Test Execution

6/19/22 49
 Thank you
for your
attention!!!

50
1
 Software Quality Assurance
Đảm bảo chất lượng phần mềm
Lecture 2: Software Metrics

2
Contents
• Motivation of Software Metrics
• Lines of code
• Halstead’s metrics
• McCabe Cyclomatic Number
• Maintainability Index (MI)
• OOP Software metrics

3
2.1. Motivation of
Software Metrics

4
How to achieve a successful software project?

• In order to conduct a successful software project we must

understand
• the scope of work to be done
• the risks incurred
• the resources required
• the tasks to be accomplished
• the milestones to be tracked
• the cost
• the schedule to be followed

5
Project Management
• Before a project can be planned
• objectives and scope should be established
• alternative solutions should be considered
• technical and management constraints should be identified
• This information is required to estimate costs, tasks, and a
schedule

6
Software Metrics
• Metrics help us understand the technical process that is
used to develop a product
• The process is measured to improve it and the product is measured
to increase its quality
• Measuring software projects is still controversial
• Not yet clear which are the appropriate metrics for a software
project
• whether people, processes or products can be compared using
such metrics

7
Why use metrics?
• Without measurements there is no real way to determine if
the process is improving
• They allow the establishment of meaningful goals for
improvement
• Metrics allow an organization to identify the causes of
defects that have the greatest effect on software
development

8
Applying metrics
• When metrics are applied to a product they help identify
• which user requirements are likely to change
• which modules are most error prone
• how much testing should be planned for each module

9
Direct measurements
• Measurements can be either direct or indirect
• Direct measures are taken from a feature of an item (e.g.,
length)
• Lines of code, execution speed, memory size, defects reported

10
Indirect Measurements
• Indirect measurements associate a measure to a feature of
the object being measured (e.g., quality is based upon
counting rejects)
• functionality, quality, complexity, efficiency, reliability and
maintainability

11
Code Complexity Measurements
Các độ đo độ phức tạp mã nguồn
• Code complexity correlates with the defect rate and
robustness of the application program
• Code complexity measurement tools
• Testwell CMT++ for C, C++ and C#
• Testwell CMTJava for Java
• Code with good complexity
• contains less errors
• is easier to understand
• is easier to maintain

12
How to use code complexity measurements
• Code complexity metrics are used to locate complex code
• To obtain a high quality software with low cost of testing
and maintenance, the code complexity should be measured
as early as possible in coding --> developers can adapt their
code when recommended values are exceeded

13
Code complexity metrics
• Metrics shown by Testwell CMT++/CMTJava
• Lines of code metrics
• McCabe Cyclomatic number
• Halstead Metrics
• Maintainability Index

14
2.2. Lines of code

15
Lines of code metrics
• Most traditional measures used to quantify software complexity
• Simple, easy to count and very easy to understand
• Do not take into account the intelligence content and the layout
of the code
• Testwell CMT++/CMTJava calculates the following lines of code
metrics
• LOCphy: number of physical lines
• LOCbl: number of blank lines (a blank line inside a comment block is
considered to be a comment line)
• LOCpro: number of program lines (declarations, definitions, directives and
code)
• LOCcom: number of comment lines

16
Lines of code metrics - Recommendations
Những giá trị khuyên dùng cho chiều dài mã nguồn
• Function length
• Function length should be 4 to 40 program lines
• A function definition contains at least a prototype, one line of code,
a pair of braces, which makes 4 lines
• A function longer than 40 program lines probably implements
many functions
• Exception: functions containing one selection statement with
many branches) --> decomposing them into smaller functions
often decreases readability

17
Lines of code metrics - Recommendations
Những giá trị khuyên dùng cho chiều dài mã nguồn

• File length
• File length should be 4 to 400 program lines
• The smallest entity that may reasonably occupy a whole source file
is a function and the minimum length of a function is 4 lines
• Files longer than 400 program lines (10..40 functions) are usually
too long to be understood as a whole

18
Lines of code metrics - Recommendations
Những giá trị khuyên dùng cho chiều dài mã nguồn
• Comments
• At least 30% and at most 75% of a file should be comments
• If less than one third of a file is comments the file is either very
trivial or poorly explained
• If more than 75% of a file are comments, the file is not a program
but a document
• Exception: in a well-documented header file percentage of
comments may sometimes exceed 75%

19
2.3. Halstead’s Metrics

20
Halstead's Metrics
• Developed by Maurice Halstead
• Introduced in 1977
• Used and experimented extensively since that time
• One of the oldest measures of program complexity
• Strong indicators of code complexity
• Often used as a maintenance metric

21
Halstead's Metrics (cont.)
• Based on interpreting the source code as a sequence of
tokens and classifying each token to be an operator or an
operand
• Operator: toán tử – Operand: toán hạng
• Then is counted
• number of unique (distinct) operators (n1)
• number of unique (distinct) operands (n2)
• total number of operators (N1)
• total number of operands (N2)
• All other Halstead measures are derived from these four
quantities with certain fixed formulas as described later

22
Operators and Operands
• Operators
• traditional: +, -, *, /, ++, --, etc.
• keywords: return, if, continue, break, try, catch etc.
• Operands
• identifiers
• constants

23
 Halstead’s Metrics
Một số độ đo của Halstead
• Program length (N): the program length is the sum of the
total number of operators and operands in the program:
• N = N1 + N2
• Vocabulary size (n): the vocabulary size is the sum of the
number of unique operators and operands
• n = n1 + n2
• Program volume (V): information content of the program
• V = N * log2(n)
• Halstead's volume describes the size of the implementation of
an algorithm

24
Halstead's metrics - Recommendations
Một số khuyến cáo về giá trị độ đo Halstead
• The volume of a function should be at least 20 and at most 1000
• The volume of a parameter less one-line function that is not
empty is about 20
• A volume greater than 1000 tells that the function probably
does too many things
• The volume of a file should be at least 100 and at most 8000
• These limits are based on volumes measured for files whose
LOCpro and v(G) are near their recommended limits

25
 Other Halstead’s metrics
• Difficulty level (D)
• The difficulty level or error proneness (D) of the program is
proportional to the number of unique operator in the program
• D is also proportional to the ration between the total number of
operands and the number of unique operands
• i.e., if the same operands are used many times in the program, it
is more prone to errors
• D = (n1/2) * (N2 / n2)
• Program level (L)
• The program level (L) is the inverse of the error proneness of the
program
• i.e., A low level program is more prone to errors than a high level
program
• L= 1/D

26
 Other Halstead’s metrics
• Effort to implement (E)
• The effort to implement or understand a program is
proportional to the volume and to the difficulty level of the
program
• E= V*D
• Time to implement (T)
• The time to implement or understand a program (T) is
proportional to the effort
• Halstead has found that dividing the effort by 18 give an
approximation for the time in seconds
• T = E / 18

27
Other Halstead’s metrics
• Number of delivered bugs (B)
• The number of delivered bugs (B) correlates with the overall
complexity of the software
• B = (E^2/3)/3000
• Estimates for the number of errors in the implementation
• Delivered bugs in a file should be less than 2
• Experiences have shown that when programming with C or C++, a
source file almost always contains more errors than B suggests
• B is an important metric for dynamic testing
• The number of delivered bugs approximates the number of errors in
a module
• As a goal at least that many errors should be found from the module
in its testing

28
 Halstead’s metric Example
Ví dụ về độ đo Halstead Halstead metrics: Example
void sort ( int *a, int n ) {
• Given a source code in C/C++ • Ignore
int i, j, t; • Count
• Calculate some basic metrics of
Halstead if ( n < 2 ) return;
• n1 for ( i=0 ; i < n-1; i++ ) {
for ( j=i+1 ; j < n ; j++ ) {
• n2
if ( a[i] > a[j] ) {
• N1 t = a[i];
• N2 a[i] = a[j];
a[j] = t;
• What are the operators/operands in
}
this source code? Oper
}
} Oper
IT4501 - Software Engineering Department - SoICT/HUST } 6/19/22 V = 80 log2(24) 392 25
/ SET / W&I Inside the boundaries [20;1000]
27-4-2011 PAGE 6
Halstead Example (cont.)

• Step 1:
• List all the operands
and operators in the
source file
• Count the number of
distinct operands and
distinct operators
• Step 2:
• Calculate V, D, E...

6/19/22 30
2.4. McCabe
Cyclomatic Number

31
McCabe Cyclomatic Number
Chỉ số phức tạp của McCabe
• The cyclomatic complexity v(G) has been introduced by
Thomas McCabe in 1976
• Measures the number of linearly-independent paths
through a program module (Control Flow)
• The McCabe complexity is one of the more widely-
accepted software metrics, it is intended to be independent
of language and language format
• Considered as a broad measure of soundness and
confidence for a program

32
McCabe Cyclomatic Number – v(G)
• v(G) is the number of conditional branches
• v(G) = 1 for a program consisting of only sequential
statements
• For a single function, v(G) is one less than the number of
conditional branching points in the function
• The greater the cyclomatic number is, the more execution
paths there are through the function, and the harder it is to
understand

33
 McCabe Cyclomatic Number
Công thức tính chỉ số McCabe
• Describes the structural
complexity
McCabe complexity (1976)
• Control flow
• Data flow
In general
• Graph based metrics
• Number-of#vertices
• v(G) = #edges vertices +2
• Number of edges
• Maximum length (depth of graph)
For control flow graphs
• In general: v(G) = #edges -
• v(G) =#vertices
#binaryDecisions
+2 + 1, or
• v(G)•=For
#IFs + #LOOPs
control +1
flow graphs
• v(G) = #binaryDecision + 1 6/19/22 34

• v(G)
Number of paths in+ the
= #IFs control
#LOOPs + 1 flow graph.

A.k.a. “cyclomatic complexity” Boundaries

• v(function) 15
McCabe Cyclomatic Number
Ý nghĩa của chỉ số McCabe
• For dynamic testing, the cyclomatic number v(G) is one of
the most important complexity measures
• Because the cyclomatic complexity describes the control
flow complexity
• it is obvious that modules and functions having high cyclomatic
number need more test cases than modules having a lower
cyclomatic number
• Rule: each function should have at least as many test cases as
indicated by its cyclomatic number

35
McCabe Cyclomatic Number - Example
Halstead metrics: Example
void sort ( int *a, int n ) {
• Count IFs and • Ignore the func
int i, j, t; • Count operato
LOOPs
• IF: 2, LOOP: 2 if ( n < 2 ) return;
for ( i=0 ; i < n-1; i++ ) {
• v(G) = 5
for ( j=i+1 ; j < n ; j++ ) {
if ( a[i] > a[j] ) {
t = a[i];
a[i] = a[j];
a[j] = t;
} T
} Operators N
} Operands N
} V = 80 log2(24) 392
/ SET / W&I Inside the boundaries [20;1000]
27-4-2011 PAGE 6
6/19/22 36
Other examples based on CFG
28

Calculate cyclomatic complexity

37
Other examples based on CFG
29

Calculate cyclomatic complexity

E=1 , N=2 , P=1 E=4 , N=4 , P=1 E=2 , N=4 , P=2

V =1 – 2 + 2*1= 1 V = 4 – 4 + 2 * 1= 2 V = 2 – 4 + 2*2 = 2

E=12 , N=7 , P=1

V = 12 – 7 + 2*1= 7

E=13, N=11 , P=3

V = 13 – 11 + 2*3 = 8

38
Other examples based on source code

39
Other examples based on source code

6/19/22 40
Other examples based on source code

41
Other examples based on source code

6/19/22 42
McCabe Cyclomatic Number – Recommendations
Khuyến cáo đối với chỉ số McCabe
• The cyclomatic number of a function should be less than 15.
• i.e., if a function has a cyclomatic number of 15, there are at least 15
(but probably more) execution paths through it
• More than 15 paths are hard to identify and test
• Functions containing one selection statement with many branches
make up a exception
• A reasonable upper limit cyclomatic number of a file is 100

43
2.5. Maintainability
Index

44
 Maintainability Index (MI)
Chỉ số bảo trì được của chương trình
• Calculated with certain formulas from lines of code measures,
McCabe measure and Halstead measures
• Indicates when it becomes cheaper and/or less risky to rewrite
the code instead to change it
• Two variants of Maintainability Index:
• One that contains comments (MI) and one that does not contain
comments (MIwoc)
• Actually there are three measures:
• MIwoc: Maintainability Index without comments
• MIcw: Maintainability Index comment weight
• MI: Maintainability Index = MIwoc + MIcw

45
MI’s formula
Công thức tính chỉ số MI
• MIwoc = 171 - 5.2 * ln(aveV) - 0.23 * aveG - 16.2 *
ln(aveLOC)
• aveV = average Halstead Volume V per module
• aveG = average extended cyclomatic complexity v(G) per module
• aveLOC = average count of lines LOCphy per module
• perCM = average percent of lines of comments per module
• MIcw = 50 * sin(√2,4*perCM)

46
MI Recommendations
Các khuyến cáo đối với chỉ số MI
• Maintainability Index (MI, with comments) values:
• 85 and more: good maintainability
• 65-85: Moderate maintainability
• < 65: Difficult to maintain with really bad pieces of code
• big, uncommented, unstructured module can lead to a MI value
can be event negative

47
Maintainability Index Halstead
Example metrics: Example

• Halstead's V = 392 void sort ( int *a, int n ) { • Ignore the fu

int i, j, t; • Count opera
• McCabe's v(G) = 5
• LOC = 14 if ( n < 2 ) return;
for ( i=0 ; i < n-1; i++ ) {
• MI = 96 --> easy to for ( j=i+1 ; j < n ; j++ ) {
maintain! if ( a[i] > a[j] ) {
t = a[i];
a[i] = a[j];
a[j] = t;
}
} Operators
} Operands
} V =6/19/22
80 log2(24) 392 48
/ SET / W&I Inside the boundaries [20;1000]
27-4-2011 PAGE 6
2.6. OOP Software
Metrics

IT4501 - Software Engineering Department -

6/19/22 49
SoICT/HUST
What about modularity?
Thế nào là tính mô đun hoá của chương trình
What about modularity?

• Generally, modularity is Design A Design B • Cohesion: calls

the degree to which a inside the
module
system may be separated • Coupling: calls
and recombined between the
modules
• High cohesion and Low
coupling is a good design A B
Cohesion Lo Hi
Coupling Hi Lo
• Squares are modules, lines are calls,
ends of the lines are functions.
• Which design is better?

/ SET / W&I 27-4-2011 PAGE 20

6/19/22 50
Modularity metrics: Fan-in and Fan-out
Chỉ số về tính mô đun hoá
• Fan-in of M: number of modules calling functions in M
• Fan-out of M: number of modules called by M
• Modules with fan-in = 0
• deadcode
• outside of the system boundaries
• Henry and Kafura's information flow complexity
• Fan-in and fan-out can be defined for procedures and functions
• HK: take global data structures into account:
• fan-in: flow of information into procedures
• fan-out: flow of information out of procedures
• HK = SLOC x (fan-in x fan-out)^2
• SLOC: the length (in lines of code) of the procedure without
comments and blank lines
51
Object-oriented metrics
• Object-oriented metrics measure the structure or
behaviour of an object-oriented system
• The system is viewed as a collection of objects rather than
as functions/procedures with messages passed from object
to object
• Some traditional metrics can be adapted to work within the
OO-domain:
• LOC, percentage of comments, Cyclomatic complexity
• These metrics are applied within methods

52
Chidamber and Kemerer's metrics
Độ đo của Chidamber và Kemerer
• Weighted methods per class (WMC)
• Response for a class (RFC)
• Lack of cohesion of methods (LCOM)
• Coupling between objects (CBO)
• Depth of inheritance tree (DIT)
• Number of children (NOC)

53
 Weighted Methods per Class (WMC)
Phương thức trên một lớp
• WMC is a count of the methods implemented within a class or
the sum of the complexities of the methods
• method complexity is measured by cyclomatic complexity
• The number of methods and their complexity is a predictor of
how much time and effort is required to develop and maintain
the class
• The larger the number of methods in class, the greater the
potential impact on children
• children inherit all of the methods defined in the parent class
• Classes with large number of methods are likely to be more
application specific, limiting the possibility of reuse

54
Response for a Class (RFC)
Phản hồi của một lớp
• The RFC is the count of the set of all methods that can be invoked in
response to a message to an object of the class or by some method in the
class
• This includes all methods accessible within the class hierarchy
• This metric looks at the combination of the complexity of a class through
the number of methods and the amount of communication with other
classes
• The larger the number of methods that can be invoked from a class
through messages, the greater the complexity of the class
• If a large number of methods can be invokes in response to a message, the
testing and debugging of the class becomes complicated since it requires
a greater level of understanding on the part of the tester

55
Lack of cohesion (LCOM)
(Sự thiếu độ kết dính)
• Lack of cohesion measures the dissimilarity of methods in a class by
instance variables or attributes
• A highly cohesive module should stand alone
• High cohesion indicates
• a good class subdivision
• simplicity and high reusability
• LCOM(C)=P-Q if P > Q and = 0 if otherwise
• P: #pairs of distinct methods in C that do not share variables
• Q: #pairs of distinct methods in C that share variables
• Lack of cohesion or low cohesion increases complexity, thereby
increasing the likelihood of errors during the development process
• Classes with low cohesion could probably be subdivided into two or
more subclasses with increased cohesion

56
Coupling between object classes (CBO)
Sự kết nối giữa các đối tượng của các lớp
• CBO is a count of the number of other classes to which a class is
coupled
• measured by counting the number of distinct non-inheritance related class
hierarchies on which a class depends
• Excessive coupling is detrimental to modular design and prevents
reuse
• the more independent class is, the easier it is reused in another application
• The larger the number of couples, the higher the sensitivity to
changes in other parts of the design
• therefore maintenance is more difficult
• Strong coupling complicates a system since a class is harder to
understand, change or correct
• Complexity can be reduced by designing systems with the weakest
possible coupling between classes
• This improves modularity and promotes encapsulation

57
Depth of Inheritance Tree (DIT)
Độ sâu cây kế thừa
• The depth of a class within the inheritance hierarchy is the
maximum number of steps from the class node to the root
of the tree and is measured by the number of ancestor
classes
• The deeper a class is within the hierarchy, the greater the
number of methods it is likely to inherit making it more
complex to predict its behavior
• Deeper trees constitute greater design complexity
• More methods and classes are involved
• But the greater the potential for reuse of inherited methods

58
Number of Children (NOC)
Số lượng lớp con
• The number of children is the number of immediate
subclasses subordinate to a class in the hierarchy
• If a class has a large number of children, it may require
more testing of the methods of that class, thus increasing
the testing time
• NOC is an indicator of the potential influence a class can
have on the system
• The greater the NOC, the greater the likelihood of improper
abstraction of the parent
• But the greater the NOC, the greater the reuse since inheritance is
a form of reuse

59
Package-level Metrics
Các độ đo ở mức gói
• Proposed by Robert Cecil Martin (Uncle Bob)
• Focus on the relationship between packages in the project
Do you still remember?
• Martin's metrics include
• Efferent Coupling (Ce)
• Afferent Coupling (Ca)
• Instability (I)
• Many intra-package dependencies: high cohesion
• Abstractness (A) i i
Ai or Ai
• 2
Normalized Distance from Main SequenceN i(D) Ni Ni 1
• Few inter-package dependencies: low coupling
i, j
Ei , j
2Ni N j
• Joint measure
k k 1 k
1 6/19/22 2 60
MQ Ai Ei , j k - Number of
k i1 k ( k 1) i 1 j i 1 packages
/ SET / W&I 27-4-2011 PAGE 21
Efferent Coupling (CE)
• This metric is used to
measure interrelationship
between classes (outgoing
dependencies)
• It is a count for the number of
classes in a given package
which depend on the classes
of other packages
• An indicator of the package's
responsibility • The high value of Ce indicates instability of a
package
• change in any of the numerous external
classes can cause the need for changes to the
package
• Preferred values for Ce are in the range of 0 to 20
6/19/22 55
Afferent Coupling (CA)
• Another type of
dependencies between
packages, i.e., incoming
dependencies
• It measures the number
of classes and interfaces
• High values of Ca usually suggest high
from other packages that component stability.
depend on classes in the • Many other classes depend on the class
analysed package • This class can't be modified significantly because
the probability of spreading such changes
increases
• Preferred values for Ca are in the range of 0 to
500 62
 Instability (I)
• This metric is used to measure the relative susceptibility of
class to changes
• I = Ce / (Ce + Ca)
• On the basis of value of I, we can distinguish two types of
components
• The ones having many outgoing dependencies and not many of
incoming ones (value of I is close to 1), which are rather unstable due to
the possibility of easy changes to these packages
• The ones having many incoming dependencies and not many of
outgoing ones (value of I is close to 0), therefore they are rather more
difficult in modifying due to their greater responsibility

63
 Abstractness (A)
• This metric is used to measure the degree of abstraction of the
package and is somewhat similar to the instability
• A = Ta / (Ta + Tc)
• Ta: number of abstract classes in a package
• Tc: number of concrete classes in this package
• Packages that are stable (metric I close to 0) should also be
abstract (metric A close to 1)
• Packages that are unstable (metric I close to 1) should consists
of concrete classes (metric A close to 0)

64
 Main Sequence
Đường trình tự chính
• Combining A and I
formulates the existence of
main sequence
• Classes that were well
designed should group
themselves around the
graph end points along the
main sequence

65
Normalized distance from main sequence (D)
• This metric is used to
measure the balance
between stability and
abstractness and is
calculated using the
following formula
• D = | A + I - 1|
• If we put a given class
on a graph of the main
sequence, its distance
from the main
sequence will be
proportional to the
value of D

6/19/22 66
Benefits of Normalized distance from main sequence (D)
Ý nghĩa của chỉ số D

• The value of D should be as low as possible so that the

components were located close to the main sequence
• A = 0 and I = 0, a package is extremely stable and concrete,
the situation is undesirable because the package is very stiff
and cannot be extended
• A = 1 and I = 1, rather impossible situation because a
completely abstract package must have some connection
to the outside, so that the instance that implements the
functionality defined in abstract classes contained in this
package could be created

67
 Summaries
Tổng kết
• Students must understand the motivation of using software
metrics to improve the quality of source code and also to
improve the quality of software systems
• Give a general perspective about software metrics that are
commonly used in software engineering
• General metrics of lines of code
• McCabe Cyclomatic Number
• Halstead’s metrics
• Maintainability Index
• OOP metrics

68
 Thank you
for your
attention!!!

69
1
 Software Quality Assurance
Đảm bảo chất lượng phần mềm
Lecture 3: Introduction to
Software Testing

2
Outline
• Definition and Motivation of Software Testing
• Basic terms of Software Testing
• Testing Activities
• Testing Levels

IT4501 - Software Engineering Department -

6/19/22 3
SoICT/HUST
3.1. Definitions &
Motivation

4
 What is software testing?
• IEEE defines software testing as:
• "A process of analyzing a software item to detect the
difference between existing and required conditions
(defects/errors/bugs) and to evaluate the features of the
software item"
• Note that...
• ...one needs to know the required conditions
• ...one needs to be able to observe the existing conditions

5
Software Testing is a process
• Evaluating a program or a system
• Verify that the product is right
• Validate that the right product is developed
• Evaluate the product quality
• Find defects

6
Testing takes time

7
 It does It does
work not work
Objectives of
Testing Reduce Reduce
the risk the cost
of failure of testing

6/19/22 8
3.2. Basic terms of
Software Testing

9
Test case
Trường hợp kiểm thử/Ca kiểm thử
• Simply, a pair of <input, expected outcome>
• Example: a square root function of nonnegative numbers
• Two kinds of testing function
• stateless function/system
• example: compiler
• state-oriented system
• example: ATM machine to withdraw function

10
 Expected Outcome
• An outcome of program execution may include
• Single values produced by the program
• State changes
• Sequence or set of values which must be interpreted together for the
outcome to be validated
• The concept of oracle in test designing: any entity (program,
process, body of data...) producing the expected outcome of
a particular test
• Expected outcome should be computed when designing test
case
• Without executing program but using program’s
requirements/specifications

11
Complete Testing
Hoàn tất kiểm thử
• Complete Testing means:
• There are no undiscovered faults at the end of the test phase
• It is impossible to complete testing for most of the
software systems
• Domain of possible inputs of a program is too large to be
completely used in testing
• valid and invalid values of data input
• Design issues may be too complex to completely test
• Impossible to create all possible execution environments of the
system

12
 Centre Issue in Testing
Vấn đề trung tâm của kiểm thử
• Select a subset of the input
domain to test a program
• Let D be the input of the
program P
• Select subset D1 ⊂ D to test P:
D1 exercise only in part P1 ⊂ P
(the execution behavior of P)
• Selection of a subset of input domain should be done
• We said that we were in a systematic and careful manner so that the
attempting to deduce properties deduction is as accurate and complete as possible
of P on selected inputs D1 • The idea of coverage is considered while selecting
test cases

13
3.3. Testing Activities

14
Testing Activities
Các hoạt động kiểm thử
• Identifying an objective to be
tested
• Select inputs
• Compute the expected
outcome
• Setup the execution
environment of the program
• Execute the program
• Analyze the test result
• Writing test report if test case
is failed 6/19/22 11

15
Sources of Information for Test Case
Nguồn dữ liệu sử dụng để thiết kế ca kiểm thử
• Objective of designing test cases: generate effective tests
at a lower cost
• Sources of Information:
• Requirements and functional specifications
• Source code
• Input and Output domains
• Operational Profile
• Fault Model

16
Requirements and Functional Specifications
• The process of software development begins by capturing
user needs
• The number of user needs identified at the beginning
depends on the specific life-cycle model
• ex: waterfall vs agile
• Test case designing is performed based on software
requirements
• Requirements can be specified in different manners
• Plaintext, equations, figures, flowcharts...
• Use cases, entity-relationship diagrams, class diagrams
• Using formal language and notation as finite-state machine...

17
Source Code
• Requirement specifications describe the intended behavior
of a system
• Source Code describes the actual behavior of the system
• Test cases must be designed based on the program

18
Input and Output domains
• Input domain
• Valid data and Invalid data
• Some values in the input domain may have special
meanings and should be treated separately
• Special values in the input and output domains of a
program should be considered while testing the program

19
Operational Profile
• Operation/Usage profile is a quantitative characterization
of how a system will be used
• It is important to test a system by considering the ways it
will actually be used
• To guide test engineers in selecting test cases using
samples of system usage
• Test inputs are assigned a probability distribution or profile
according to their occurrences in actual operation

20
Fault Model
• Encountered faults are a excellent source of information in
designing new test cases
• Different classes of faults: initialization faults, logic faults,
interface faults
• 3 types of fault-based testing:
• Error guessing
• Fault seeding
• Mutation testing

21
Test case designing techniques
• White Box Testing
• Structural testing techniques
• Examining source code with a focus on control flow and data flow
• Applied to individual units of a program
• Black Box Testing
• Functional testing techniques
• Examining functional specifications of testing modules
• Applied to both an entire system and the individual program units
• These strategies are used by different groups of people at
different times during a software life cycle
• Ex: programmers use both to test their own code while quality
assurance engineers apply the idea of functional testing

22
Test Planning and Design
• The purpose of system test planning is to get ready and
organized for test execution
• Test plan provides a framework, scope, details of resource
needed, effort required, schedule of activities and a budget
• Test design is a critical phase of software testing
• System requirements are critically studied
• System featured to be tested are identified
• Objectives and detailed behaviors of test cases are defined
• Test-Driven Development (TDD): test cases are designed
and implemented before the production code is written
• key practice of modern agile software development processes

23
Monitoring and Measuring test execution
• Motivation: to explore the progress of testing and reveal the
quality level of the system
• Test execution metrics are categorized into 2 classes
• Metrics for monitoring test execution
• Metrics for monitoring defects
• Quantitative measurements
• Number of defects detected by test cases
• Number of test cases designed/executed per day
• Number of defects found by the customers that were not found by
the test engineers

24
3.4. Testing Levels

25
Testing Levels
Mức độ kiểm thử
• Testing is performed
at different levels
involving the whole
system or parts of it
throughout the life
cycle of a software
product
• Four stages of testing
• Unit test
• Integration test
• System test
6/19/22
• Acceptance test 126

26
Unit Testing
Kiểm thử đơn vị
• Unit = smallest testable software component
• Objects
• Procedures/Functions
• Reusable components
• Tested in isolation
• Usually done by programmer during development
• A pair tester can help
• Developer guide or coding guideline can require
• Also known as component or module testing

27
Dynamic Unit Test Environment
• An environment for
dynamic unit testing is
created by emulating
the context of the unit
under test
• The context consists of:
• A caller of the unit
• All the other units called
by the unit

28
Test Driver
• The caller unit is called test driver
• A program that invokes the unit under test
• The unit under test executes with input values received from the
driver and return a value to the driver
• The driver compares the actual outcome with the expected
outcome and reports the ensuing test result

29
Stubs
• The units called by the unit under test are called stubs
• A stub is a dummy subprogram that replaces a unit that is called
by the unit under test
• A stub performs two tasks:
• It shows an evidence that the stub was called
• It returns a precomputed value to the caller so that the unit
under test can continue its execution

30
Integration Testing
Kiểm thử tích hợp
• At the unit testing level, the system exists in pieces under
the control of the programmers
• Put the modules together to construct the complete
system
• The path from tested components to constructing a
deliverable system contains two major testing phases:
• Integration testing
• System testing (introduced in Lecture 6)

31
Motivation of Integration Testing
• Different modules are generally created by groups of different
developers
• Interface errors between modules created by different
programmers and even by the same programmers are
rampant
• Unit testing of individual modules is carried out in a controlled
environ- ment by using test drivers and stubs. It is therefore
difficult to predict the behavior of a module in its actual
environment after the unit testing is performed
• Some modules are more error prone than other modules,
because of their inherent complexity. It is essential to identify
the ones causing most failures.

32
Objectives of Integration Testing
• To build a WORKING version of the system by putting the
modules together
• To ensure that additional modules work as expected
without disturbing the functionalities of the modules
already put together
• Integration test is completed when the system is fully
integrated together:
• All the test cases have been executed
• All the severe and moderate defects found have been fixed and the
system is retested

33
 Interfaces between modules

• Modules are interfaced with other

modules to realize the system’s
functional requirements
C1 C2
• An interface between two modules
allows one module to access the
service provided by the other
• An interface between two modules
implements a mechanism for passing
control and data between modules

34
Different types of Interfaces between modules

Three common paradigms for interfacing modules

• Procedure Call Interface: a procedure in one module call a
procedure in another module. The caller passes on control and
data to the called module. The called one can pass data to the
caller while returning control back to the caller
• Shared Memory Interface: A block of memory is shared
between two modules. Data are written into the memory
block by one module and are read from the block by the other
• Message Passing Interface: One module prepares a message
by initializing the fields of a data structure and sending the
message to another module. This form is common in client-
server or web-based system

35
Interface Errors

If all the unit tested modules work individually, why can

these modules not work when put together?

36
Interface Errors
• Construction: interface specification is separated from the
implementation code (i.e., in C/C++ through #include
header.h). Inappropriate use of #include statements cause
construction errors
• Inadequate Functionality: Implicit assumptions in one part of
a system that another part of the system would perform a
function
• Location of Functionality: Disagreement on or
misunderstanding about the location of a functional capability
within the software leads to this sort of error
• Changes in Functionality: Changing one module without
correctly adjusting for that change in other related modules
affects the functionality of the program

37
 Interface Errors (cont.)
• Added Functionality: A completely new functional module
was added as a system modification
• Misuse of Interface: one module makes an error in using the
interface of a called module. Interface misuse can take the
form of wrong parameter type, wrong parameter order or
wrong number of parameters passed
• Misunderstanding of Interface: A calling module may
misunderstand the interface specification of a called module.
The called module may assume that some parameters passed
to it satisfy a certain condition whereas the caller does not
ensure that the condition holds.
• ...

38
Advantages of Integration Testing
• Defects are detected early
• It is easier to fix defects detected early
• Get earlier feedback on the health and acceptability of the
individual modules and on the overall system
• Scheduling of defect fixes is flexible and it can overlap with
development

39
Who do integration testing?
• System integration testing is performed by the system
integration group, aka a build engineering group
• Integration test engineers who should be familiar with the
interface mechanisms
• Team of engineers who built the modules
• The system architects because of the fact that they have a bigger
picture of the system

40
Granularity of System Integration Testing

• System integration testing is performed at different

levels of granularity
• Use both of two techniques
• Black box testing
• White box testing
• Different levels:
• Intrasystem Testing
• Intersystem Testing
• Pairwise Testing

41
Intrasystem Testing
• Low-level integration testing with the objective of
combining the modules together to build a cohesive
system
• Combining modules progresses in an incremental manner
• Example: in a client-server system, individual modules of
client and server are combined in an incremental fashion, to
be tested
• Source of testing: low-level design document which details
the specification of modules

42
Intersystem Testing
• A high-level testing phase which requires interfacing
independently tested systems
• All systems are connected together
• Testing is conducted from end to end: ensuring that the
interaction between systems work together but not to conduct
a comprehensive test
• Only one feature is tested at a time and on a limited basis
• Test cases are derived from the high-level design document,
which details the overall system architecture
• Ex: integrating client and server to test after perform
individually tested system (client and server)

43
Pairwise Testing
• There can be many intermediate levels of system
integration testing between intrasystem testing and
intersystem testing
• Pairwise testing is a kind of intermediate level of
integration testing
• Only two interconnected systems in an overall system are
tested at a time
• The purpose is to ensure that two systems under
consideration can function together
• All other systems are assumed to behave as expected

44
System Integration Testing Techniques
Các kĩ thuật kiểm thử tích hợp hệ thống

• Incremental
• Top-down
• Bottom-up
• Sandwich
• Big bang

45
Incremental technique
• Integration testing is conducted in an incremental
manner as a series of test cycles
• In each test cycle: a few more modules are integrated
with an existing and tested build to generate a larger
build
• Objective:
• Complete one cycle of testing
• Fix all the errors found
• Continue the next cycle of testing
• Finish building a complete and ready system for system
testing

46
How many test cycles for system
integration testing?

The number of system integration test cycles and the

total integration time depend on:
• Number of modules in the system
• Relative complexity of the modules (McCabe Cyclomatic
Complexity)
• Relative complexity of the interfaces between modules
• Number of modules needed to be clustered together in each
test cycle
• Whether the modules to be integrated have been
adequately tested before
• Turnaround time for each test-debug-fix cycle

47
Constructing a build in each test cycle
• Constructing a build is a process by which individual modules
are integrated to form an interim software image
• A software image is a compiled software binary
• The final build will be a candidate for system testing
• Creating a daily build is very popular to facilitate a faster
delivery of the system
• Focusing on small incremental testing, steadily increasing the
number of test cases and regression testing from build to build

48
Advanced Testing Level
• System Testing
• Acceptance Testing
• Regression Testing

49
Summary
• Students should understand principle terms of software
testing
• Have a general understand of V-model
• Understand different phase/activities of testing process

52
 Thank you
for your
attention!!!

53
1
 Software Quality Assurance
Đảm bảo chất lượng phần mềm
Lecture 4: Black Box Approach
for Test Case Designing

2
Contents
Những nội dung chính

• Motivation of Blackbox Testing

• Equivalence class partitioning

• Boundary value analysis

• Decision Table

• State Transition Testing

3
4.1. Blackbox Testing

4
What is Black box testing
• One of testing techniques allowing to design good test
cases
• Black box testing is a strategy in which testing is based on
the requirements and specifications
• Verify the output without inspecting the internal workings
• We do not know:
• How the box handles errors
• Whether the inputs are executing all paths of code

5
 Black Box vs. White Box
Black-Box vs. White-Box
• Black Box: functional testing
• Unit test • External/user view: • Internal/developer view:
– Check conformance with – Allows tester to be confident about
• Integration test specification test coverage

• System test • Abstraction from details: • Based on control or data flow:

– Source code not needed – Easier debugging
• Programmers & Test Engineers • Scales up: • Does not scale up:
& Quality Assurance Engineers – Different techniques at – Mostly applicable at unit and
different levels of granularity integration testing levels
• White Box: structural testing
• Unit test USE BOTH!
• Integration test
• Programmers & Test Engineers Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

6
Black box testing techniques
• Equivalence class partitioning
• Boundary value analysis
• Decision Table
• State Transition Testing

7
4.2. Equivalence class
partitioning (ECP)

8
Equivalence Class Partitioning
Phân chia lớp tương đương
• A method testing that divides the input domain into
classes of data
• An Equivalence Class is
• Consist of a set of data that acts the same way
• Any data value within class is equivalent
• 2 classes
• Valid class
• Invalid class

9
Equivalent Class Testing
Kiểm thử lớp tương đương
• Step 1: Identify all equivalent classes from the requirement
and specification of the function
• Step 2: Pick up at least 1 value in each equivalence class to
create test case

Negative 0 Positive Negative 0 Positive

integer integer integer integer

Select -10
Select 100

Step 1 Step 2

10
Example: Insurance System
• Specification
• System shall reject over-age insurance applicants
• Reject male insurance applicants over the age of 80 years on day of
application
• Reject female insurance applicants over the age of 85 years on day
of application
• Identify equivalent classes?

11
Answer
• Input: Gender & Age
• Output: accept/reject
Classes Test Cases
C1: Input: Males over 80 T1: male, 83, reject
C2: Input: Males 80 or under T2: male, 56, accept
C3: Input: Females 85 or under T3: female, 83, accept
C4: Input: Females over 85 T4: female, 87, reject

• What’s about the invalid data?

12
Equivalence Class Testing Guidelines
Một số chú ý khi sử dụng phân lớp tương đương
If input condition
• is a range, e.g., x = [0..9]
• is an ordered list of values, e.g., owner = {1, 2, 3, 4}
• ==> one valid and two invalid classes are defined
• is a set, e.g., vehicle = {car, motorcycle, truck}
• is a "must be" condition (boolean), e.g., "first character of the identifier
must be a letter"
• ==> one valid and one invalid class are defined
• is anything else
• ==> partition further

13
Exercise 1: Program to determine employability

Age Employment Status

0-15 Don't hire

16-17 Can hire part-time

18-55 Can hire full-time

56-99 Don't hire

How many equivalent classes? How many test case should be designed?

14
Exercise 2: Examination Judgment Program
• Specification
• Two subjects: Maths and Physics
• Student passed if:
• Scores of both mathematics and physics are greater than or
equal to 70 out of 100
• average of mathematics and physics is greater than or equal to
80 out of 100
• Failed if otherwise

15
Exercise 2
Score of Average
Physics. Score is 80.

(2)
100
(5)
(1)
80 Score Math. Physics Result
(4)
70
(1) 55 85 Failed
(3)
(2) 67 97 Passed
60
Average
Score
(3) 96 68 Passed
is 80. (4) 77 80 Passed
40
(7) (6) (5) 85 92 Passed
(6) 79 58 Failed
20
(7) 52 58 Failed
Score of
Math.

0 20 40 60 70 80 100

16
Advantage of Equivalence Partitioning

Reduce the number of test case Ensure each class is tested

17
Disadvantages of Equivalence Partitioning
• The identification of equivalence classes relies on the
experiences of tester
• Not consider boundary values

18
4.3. Boundary Value
Analysis (BVA)

19
Boundary Value Analysis
Phân tích giá trị biên
• A supplementary method for the equivalence partitioning
method
• Allow to select test cases to represent each side of the class
boundaries
• Steps:
• Identify the equivalence classes
• Identify the boundary of each class
• Create test case for each boundary value by choosing one point on
the boundary, one point just below the boundary, and one point
just above the boundary

20
Example 14

Example – boundary value testing

• Employability Module Age Employment status
• 6 equivalence 0-15 Don’t hire

classes (4 valid 16-17 Can hire part-time

18-54 Can hire full-time
classes)
55-99 Don’t hire
• For each
boundary value,
pick up 3 values -1 0 1 15 16 17 17 18 19 54 55 56 98 99 100
on/below/upper
• 15 Test cases
Other test cases?

6/19/22 21

21
Example: Human’s age
• Another way to choose values
• One point on the boundary
• One point beside the boundary in invalid class

70 years old

18 60
Children Adult Elderly

17 61
30 years old

22
Advantage of Boundary Value Analysis
• Easy to understand and control
• Allow to complete the equivalence class method
• Achieve quality test cases

23
Limitations
• Does not work well with Boolean variables
• Very complex if variables are not independent
• Most suited to systems in which much of the input data
takes on values within ranges or within sets

24
 Example: The nextDate function
• Problem statement:
• nextDate is a function of three variables: month(M), day(D),
year(Y). It returns the date of the day after the input date. The
month, day and year variables have integer values subject to
these conditions:
• C1: 1 <= M <= 12
• C2: 1 <= D <= 31
• C3: 1850 <= Y <= 2050
• How many equivalent classes there are?

25
variables have integer values subject to these conditions:
• C1:1 <= month <=12;
•
Equivalent Classes
C2: 1<= day <= 31;
• C3: 1850 <= year <= 2050
Valid ECs
M1 = {month: 1 <= month <= 12} M2 = {month: month < 1}

D1={day:1<=day<=31} M2 = {month: month > 12}

Y1 = {year: 1850 <= year <= 2050} D2={day:day<1}

Can we define better equivalence classes?

What about combinations of M,D & Y?

26
 15

NextDate – multiple
Equivalent Classes variables
Remember?
• C1:1 <= month <=12;
• Which months have 30 days? 31 days?
• C2: 1<= day <= 31;
• February has 28
• C3: 1850 <=or 29<=
year days depend on whether year is leap
2050
yearCan
or we
notdefine better equivalence classes?
Valid ECs
M1 = {month has 30 days}
M2 = {month has 31 days}
M3 ={February}

D1={1<=day<=27}, D2={day=28}
D3= {day = 29}, D4= {day = 30}, D5 {day = 31}
Y1 = {year: year is a non-leap year}
Y2 = {year: year is a leap year}

What about combinations of M,D & Y?

27
Design Test cases
16

Case ID Month Day Year Expected

Output
C1 -1 15 1902 Value of
month not in
range
C2
C3
C4
…

There are too many combinations of M, D and Y è a lot of test case

28
4.4. Decision Table
Technique

29
Decision Table
Bảng quyết định
• Is an exellent tool to capture certain kinds of system
requirements and to document internal system design.
• Often used to record complex business rules that a system
must implement
• In addition, they can serve as a guide to creating test cases

30
 20

Generalform:
General Form
Rule 1 Rule 2 … Rule p
Condition 1
Condition 2
Conditions
…
Condition n
Action 1
Action 2
Actions
…
Action m

31
Example: Car Insurance Discount
• Specification
• If married and number of employed years >=3 then discount 70%
• If married and number of employed years < 3 then
• If good student, discount 50%
• Otherwise, 20%
• If not married and number of employed years >=3, discount 60%
• In case of not married and number of employed years < 3
• If good student, discount 40%
• Otherwise 0%

32
 21

Answer
Example: car insurance discount
Rule 1 Rule 2 Rule 3 Rule 4 Rule 5 Rule 6 Rule 7 Rule 8

Married? Yes Yes Yes Yes No No No No

# years >= 3 >= 3 <3 <3 >= 3 >= 3 <3 <3

C employed

Good student? Yes No Yes No Yes No yes No

A Discount 70 70 50 20 60 60 40 0

8 test cases

33
 Answer 22

Example - simplified

Rule Rule 3 Rule 4 Rule Rule 7 Rule 8

1-2 5-6

Married? Yes Yes Yes No No No

# years >= 3 <3 <3 >= 3 <3 <3

employed 6 test cases
C
Good student? - Yes No - Yes No

A Discount 70 50 20 60 40 0

34
Some terms
Một số thuật ngữ 23

Definitions
Rule Rule 3 Rule 4 Rule Rule 7 Rule 8
1-2 5-6 Limited entry
table –
conditions are
Married? Yes Yes Yes No No No binary only
# years >= 3 <3 <3 >= 3 <3 <3
C employed Extended entry
table – conditions
Good student? - Yes No - Yes No
may take on
Discount 70 50 20 60 40 0 multiple values
A

Don’t-care-entry – condition is irrelevant

or not applicable; any value is OK

Decision tables are declarative – order between entries does not matter

35
Using Table Decision for Designing Test Cases
Sử dụng bảng Using
quyết địnhtables
decision để thiết kế ca kiểm
for testing 24

thử • Rules become test cases

• Conditions become inputs
• Actions become expected results:
Test Case 1 Test Case 2 … Test Case p
• Rules becomes Condition 1

Test Cases Inputs

Condition 2
…
• Conditions Condition n
become inputs Action 1

• Actions become Expected

Results
Action 2
…
expected results Action m

36
Exercise 1: Triangle Program
• The program accepts three integers a,b,c as inputs
• These integers are interpreted as representing the lengths
of sides of a triangle
• These variables a,b,c must satisfy the following conditions
• a < b + c, b < a + c, c < a + b
• The program must determine whether the inputs are not a
triangle, an isosceles or an equilateral triangle

37
 26

Decision table
Decision table for the triangle program
Rule Rule Rule Rule Rule Rule Rule Rule Rule
1 2 3 4 5 6 7 8 9
a,b,c forms a
triangle
a=b
C
a=c
b=c
NaT
Isosceles
A
Scalene
Equilateral

38
Decision Table 27

Decision table for the triangle program

Rule Rule Rule Rule Rule Rule Rule Rule Rule
1 2 3 4 5 6 7 8 9
a,b,c forms a N Y Y Y Y Y Y Y Y
triangle
a=b _ Y Y Y Y N N N N
C
a=c _ Y Y N N Y Y N N
b=c _ Y N Y N Y N Y N
NaT
Isosceles
A
Scalene
Equilateral

39
 28

Decision table
Decision Tablefor the triangle program
Rule Rule Rule Rule Rule Rule Rule Rule Rule
1 2 3 4 5 6 7 8 9
a,b,c forms a N Y Y Y Y Y Y Y Y
triangle
a=b _ Y Y Y Y N N N N
C
a=c _ Y Y N N Y Y N N
b=c _ Y N Y N Y N Y N
NaT X

Impossible

Impossible

Impossible
Isosceles X X X
A
Scalene X
Equilateral X

40
Design Test Case
Testing the triangle program 29

1 2 3 4 5 6 7 8 9 10 11 a b c Exp
a<b+c

b<c+a

c<b+a
C
a=b

a=c

b=c

NaT

Isosceles
A
Scalene

Equilateral

41
Design Test Case
Testing the triangle program 30

1 2 3 4 5 6 7 8 9 10 11 a b c Exp
a<b+c N
b<c+a -
a<b+a -
C
a=b -
a=c -
b=c -
NaT X
Isosceles
A
Scalene

Equilateral

42
 Design Test Case
Testing the triangle program 31

1 2 3 4 5 6 7 8 9 10 11 a b c Exp
a<b+c N 4 2 1 NaT
b<c+a -
c<b+a -
C
a=b -
a=c -
b=c -
NaT X
Isosceles
A
Scalene

Equilateral

43
Design Test Case
Testing the triangle program 32

1 2 3 4 5 6 7 8 9 10 11 a b c Exp
a<b+c N Y Y T 4 2 1 NaT
b<c+a - N Y T
c<b+a - - N T
C
a=b - - - T
a=c - - - T
b=c - - - T
NaT X
Isosceles
A
Scalene

Equilateral

44
Design Test Case
Testing the triangle program 33

1 2 3 4 5 6 7 8 9 10 11 a b c Exp
a<b+c N Y Y T 4 2 1 NaT
b<c+a - N Y T 2 4 1 NaT
c<b+a - - N T 1 4 2 NaT
C
a=b - - - T 3 3 3 Eq
a=c - - - T
b=c - - - T
NaT X X X
Isosceles
A
Scalene

Equilateral X

45
Design Test Case
Testing the triangle program 34

1 2 3 4 5 6 7 8 9 10 11 a b c Exp
a<b+c N Y Y T T T 4 2 1 NaT
b<c+a - N Y T T T 2 4 1 NaT
c<b+a - - N T T T 1 4 2 NaT
C
a=b - - - T T T 3 3 3 Eq
a=c - - - T F T
b=c - - - T T F
NaT X X X
Isosceles
A
Scalene

Equilateral X

46
Design Test Case
Testing the triangle program 35

1 2 3 4 5 6 7 8 9 10 11 a b c Exp
a<b+c N Y Y T T T 4 2 1 NaT
b<c+a - N Y T T T 2 4 1 NaT
c<b+a - - N T T T 1 4 2 NaT
C
a=b - - - T T T 3 3 3 Eq
a=c - - - T F T
b=c - - - T T F
NaT X X X
Isosceles
A
Scalene

Equilateral X

47
Design Test Case
Testing the triangle program 36

1 2 3 4 5 6 7 8 9 10 11 a b c Exp
a<b+c N Y Y Y Y Y Y Y Y Y Y 4 2 1 NaT
b<c+a - N Y Y Y Y Y Y Y Y Y 2 4 1 NaT
c<b+a - - N Y Y Y Y Y Y Y Y 1 4 2 NaT
C
a=b - - - Y Y Y Y N N N N 3 3 3 Eq
a=c - - - Y N Y N Y Y N N
b=c - - - Y Y N N Y N Y N
NaT X X X 5 5 3 Iso
Isosceles X X X
A
Scalene X 5 3 5 Iso
Equilateral X 3 5 5 Iso
3 4 5 Sca

48
Remember?
• C1:1 <= month <=12;
• C2: 1<= day <= 31;
Exercise 2: the nextDate function
• C3: 1850 <= year <= 2050
Can we define better equivalence classes?
Valid ECs
M1 = {month has 30 days}
M2 = {month has 31 days}
M3 ={February}

D1={1<=day<=27}, D2={day=28}
D3= {day = 29}, D4= {day = 30}, D5 {day = 31}
Y1 = {year: year is a non-leap year}
Y2 = {year: year is a leap year}

What about combinations of M,D & Y?

49
Decision Table for the
The NextDate Function
nextDate function
40

1 2 3 4
month in M30
month in M31
February
December
1<=day<28
C
Day=28
Day=29
Day=30
Day=31
Leap year
Increment Year
Increment Month
Increment Day
A
Reset year
Reset month
Reset day

50
 Decision Table for the nextDate function
The NextDate Function 41

1 2 3 4
month in M30 T - - -
month in M31 - T - -
February - - T -
December - - - T
1<=day<28 T T - T
C
Day=28 - - T -
Day=29 - - - -
Day=30 - - - - Can we simplify the conditions?
Day=31 - - - -
Leap year F F F F
Increment Year
Increment Month X
Increment Day X X X
A
Reset year
Reset month
Reset day X

51
Decision Table Function
The NextDate for the nextDate function 43

1 2 3 4
month

C day

year

Increment Year
Increment Month
Increment Day
A Reset year
Reset month
Reset day
Error

M31 = { 1,3,5,7,8,10 }; M30 = { 4,6,9,11 }; M2 = { 2 }; M12 = { 12 };D27={1,2,...,27};

D28={28},D29={29 },D30={30},D31={31} YN = not a leap year; YL = leap year

52
Decision Table for the nextDate function
44
1 2 3 4 5 6 7 8 9
month M30 M30 M30 M31 M31 M12 M12 M2 M2 M2 M2 M2 M2

day D27 D30 D31 D27 D31 D27 D31 D30 D27 D28 D28 D29 D29
D28 D28 D28 D31
C
D29 D29 D29
D30 D30
year - - - - - - - - - YL YN YL YN

Increment X
Year
Increment X X X X
Month
Increment X X X X X
Day
A
Reset year
Reset X X X
month
Reset day X X X
Error X X X

53
Design TestforCases
Test cases The NextDate Function
45

Case ID Month Day Year Expected output

1-3 April 15 2001 April 16, 2001

4 April 30 2001 May 1, 2001

5 April 31 2001 Error

6-9 January 15 2001 January 16, 2001

10 January 31 2001 February 1, 2001

11-14 December 15 2001 December 16, 2001

15 December 31 2001 January 1, 2002

16 February 15 2001 February 16, 2001

17 February 28 2004 February 29, 2004

18 February 28 2001 March 1, 2001

19 February 29 2004 March 1, 2004

20 February 29 2001 Error

21, 22 February 30 2001 Error

54
Advantages
• This technique is useful for applications characterized by
any of the following:
• Prominent if-then-else logic
• Logical relationships among input variables
• Calculations involving subsets of the input variables
• Cause-and-effect relationships between inputs and outputs

55
 Disadvantages

• Decision table
may become very
large if we have Condition Rule
1
Rule 2 Rule 3 Rule 4 Rule 5 Rule 6

many conditions
• We need to factor
large tables into
smaller ones to
remove
redundancy Action

56
Exercise 3: Road charge
• A program that calculates road user charges for foreign
heavy vehicles
• Inputs:
• distance that the driver are intending to drive on a road
• the emission category of the vehicle (0 or 1 or 2)
• the number of axles of the truck
• Driver can pay for day or week
• Chargeable period for day is 00.00-24.00. For example, if
the journey begins at 21h30 on one day and finishes at
02.20 next day, payment for two days is required

57
Note that the chargeable period for day is 00.00-24.00. So, if a journey on a toll road begins, for
example, at 21.30 on one day and finishes at 02.30 next day, payment for two days is required.

Exercise 3: Tarif Tables

Tolls are paid in SEK.”

Toll Tariff 2014

Max
3 3 3 4 4 4
axles

Emission
0 1 2 or cleaner 0 1 2 or cleaner
Class

1 day 67 67 67 67 67 67

1 week 220 194 169 347 313 279

58
Exercise 4: Bus tarif
• A program allows to calculate automatically the monthly
bus tarif
• Inputs: age and distance
Distance <=10 km Distance > 10 km
• Age Infant charge 0 0
• 0-3 years old: infant charge Child charge 100 130
Adult charge 200 250
• 4-14 years old: child charge
Silver charge 160 200
• 15-59 years old: adult charge
• 60-99 years old: silver charge
• Do not treat at the age of 100 or more
• Distance: <=10 or >10 km

59
Exercise 4: Bus tarif (in EUR)

Distance <=10 km Distance > 10 km

Infant charge 0 0
Child charge 100 130
Adult charge 200 250
Silver charge 160 200

IT4501 - Software Engineering Department -

6/19/22 60
SoICT/HUST
Exercise 4:
• With input distance, list test cases which using Equivalent
and Boundary (if possible)
• With input age, list test cases which using Equivalent and
Boundary if possible
• List test cases combination of distance and age by
Equivalent method
• List test cases combination of distance and age by
Boundary method
• Create decision table base on input distance and age

61
Exercise 5: Register Form
• A register form allows to create a username and a password
and add the account to the database
• Specification:
• Username: from 3 to 15 characters, no space and special characters
including #,$,@
• Password: should be 8 digits, first character is not zero
• Using Equivalent Class and Boundary Value Analysis to
design test cases

62
4.5. State Transition
Testing

63
Basic ideas
• Applied when an application gives a different output for the
same input, depending on what has happened in the earlier
state
• The system can be in a (finite) number of different states
and the transitions from one state to another are
determined by rules
• Have to determine states, events, actions and transitions
that should be tested

64
Basic Terms
Thuật ngữ cơ bản
• The states that the system may occupy
• ex: a document is closed or opened
• The transitions from one state to another
• The events that cause a transition
• ex: the action “closing” a document cause the transition form the
state “opened” to “closed”
• The actions that result from a transition
• ex: an error message, a warning ...

65
Design Test Cases
• Create a set of test cases such that all states are visited at
least once. May miss important transitions.
• Create a set of test cases such that all states are triggered
at least once. May miss both states and transitions
• Create a set of test cases such that all transitions are
exercised at least once. Subsumes (includes) all-states and
all-events
• Stronger: cover all possible pairs of transitions (1-switch coverage)
• Even stronger: cover all possible triplets of transitions (2-switch
coverage)
• Create a set of test cases such that all paths are executed
at least once. Subsumes all others. Can be infeasible -
consider loops

66
Example 1: ATM machine
• Go to the ATM
• Insert card – Verify PIN
• Withdraw money
• Successful if sufficient balance
• Refused if insufficient balance
• User can try to type PIN 3 times – If not valid, card is
rejected
• How many states the system has?
• Draw the state-transition diagram

67
State Transition Diagram

• S1: Start State

• S2: Wait for PIN c
• S3: 1st try invalid
• S4: 2nd try invalid
• S5: 3rd try invalid
• S6: Card rejected
• S7: Access account

6/19/22 68

68
All states coverage
• TC1: Start State –
Valid PIN – Access
Account
• TC2: Start State – 1st
try invalid – 2nd try
invalid – 3rd try invalid
– Card Rejected

69

69
All transitions coverage

• Which test cases

will coverage all
transitions of the
system?

6/19/22 70

70
Example 2: Flight Booking System
52

Flight booking
• The customer system
provides some information and makes a
booking.
The customer provides some information and makes a booking. He then
• He then
has ahas a certain
certain amountamount
of time toof time
make theto make the
reservation.
reservation.
Event

giveInfo/ Booking
startPayTimer Made payMoney

Action Transition Paid

Entry point
State

71
 54

Flight Booking
Flight bookingSystem Complete
system complete

payMoney
giveInfo/ Booking
startPayTimer Made checkIn
Paid
cancel
cancel/refund
time Ticketed
Expires Cancelled/
Customer
Cancelled
/Unpaid fly
Used

72
Design Test Case for Flight Booking System
• Draw a table of states and events
• Identify test cases to satisfy all states coverage
• Identify test cases to satisfy all transitions coverage

73
 Thank you
for your
attention!!!

74
1
 Software Quality Assurance
Đảm bảo chất lượng phần mềm
Lecture 5: White Box Approach

2
Contents
• Control Flow Testing
• Predicate Testing
• Data Flow Testing
• Unit-level Testing
• Mutation Testing

3
5.1. White Box Testing

4
 Effective Test Case Design
• Exhaustive testing (use of all possible inputs and conditions) is
impractical
• Must use a subset of all possible test cases
• Must have high probability of detecting faults
• Need processes that help us selecting test cases
• Effective testing - detect more faults
• Focus attention on specific types of faults
• Know you are testing the right thing
• Efficient testing - detect faults with less effort
• Avoid duplication
• Systematic techniques are measurable and repeatable

5
Basic Testing Strategies
Basic Testing Strategies
• Motivation:
• Effective Test
Case Design
• Compare 2
approaches
• Black Box
• White Box
Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

6
Black Box vs. White Box
• Black Box: functional
Black-Box vs. White-Box
testing
• Unit test System
• Integration test
Specification
• System test Implementation

• Programmers & Test

Engineers & Quality
Assurance Engineers Missing functionality:
Cannot be revealed by white-
Unexpected functionality:
Cannot be revealed by black-box
box techniques techniques
• White Box: structural
testing Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

• Unit test
• Integration test
• Programmers & Test
Engineers

7
5.2. Control Flow Testing
Kiểm thử luồng điều khiển

IT4501 - Software Engineering Department -

6/19/22 8
SoICT/HUST
Basic terms
• Program unit: entry point to exit point
• Procedures, Functions, Modules, Components
• 2 kinds of statement
• Assignment statement
• Conditional statement
• Program path
• A sequence of statements from entry point to exit point of a
program unit
• A program unit may contains many program paths
• An execution instance of the unit
• A given set of input data, the unit executes a different
path

9
Control Flow Graph
Đồ thị luồng điều khiển
CFG symbols
• Represent the 4.3 CONTROL FLOW

graphical structure of
a program unit
True False
• A sequence of Computation Decision

statements from
entry point to exit
point of the unit
Sequential computation Decision point Merge point
depicted using graph
Figure 4.2 Symbols in a CFG.
notions
computation. A maximal sequential computation can be represented
Lund Universitysingle rectangle
/ Faculty of or by many
Engineering/ Department rectangles,
of Computer eachEngineering
Science / Software corresponding
Research Groupto one state
source code.
We label each computation and decision box with a unique intege
branches of a decision box are labeled with T and F to represent the tru
evaluations, respectively, of the condition within the box.10We will not la
node, because one can easily identify the paths in a CFG even withou
considering the merge nodes. Moreover, not mentioning the merge nod
 Control Flow Testing
White-Box Testing (Ch 4, 5)
• Main idea: select a few paths Exhaustive Testing
in a program unit and observe
whether or not the selected If-then-else There are many possible paths!
paths produce the expected 520 (~1014 ) different paths
outcome
• Executing a few paths while
loop < 20x
trying to assess the behavior
of the entire program unit
Selective Testing
Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

10

11
 Select as few paths as possible
Selective White-box Testing
• Selecting relevant paths
based on some criteria Selective:
a selected path
• All paths
• Statement Coverage
üControl flow testing
• Branch Coverage
üData flow testing
• Predicate Coverage üLoop testing
• Most applicable to new üFault-based testing
software for unit test
Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

12
 Outline of Control Flow Testing
90 CHAPTER 4 CONTROL FLOW TESTING

• Inputs Path
selection
• Source code of unit Work criteria

• Path selection criteria

process
• Generate CFG: draw CFG Program
unit
Draw a
control flow Control
flow graph
Select
paths
Selected
paths
graph
from source code of the unit
Inputs
• Selection of paths: selected Generate
paths to satisfy path test input
data
selection criteria
• Generation of test input data No
Are
the selected
paths
feasible?

Yes

Process of generating test input data

Fig 4.1 Output Test input

data
Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

Figure 4.1 Process of generating test input data for control flow testing.

Selection of Paths: Paths are selected from the CFG to satisfy the path selec-
13
tion criteria, and it is done by considering the structure of the CFG.
Generation of Test Input Data: A path can be executed if and only if a
 Path selection criteria

• Example: Life Insurance Example

Age, Gender 1

• Given the source 2

bool AccClient(agetype T if(female) F
code of the function age; gndrtype gender) T 3 F
AccClient bool accept
Age <85
T 4
Age <80
F

if(gender=female)
• Draw the CFG accept := age < 85;
Accept = true 5 Accept = false 6
else
accept := age < 80;
Return accept 7
return accept

Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

14
 Path selection criteria
• Example:
• Given the source Life Insurance Example
Age, Gender 1

code of the 2
bool AccClient(agetype T if(female) F

function AccClient age; gndrtype gender) T 3 F

Age <85

• Draw the CFG bool accept

if(gender=female)
T 4
Age <80
F

accept := age < 85;

Accept = true 5 Accept = false 6
else
accept := age < 80;
Return accept 7
return accept

Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

15
All path coverage criterion
Tiêu chí lựa chọn tất cả các đường dẫn
• Objective: Design all All paths
Female Age < 85 Age < 80
possible test cases so Yes Yes Yes Age, Gender 1

Yes Yes No
that all paths of the Yes No Yes T
2
if(female) F

program are executed Yes

No
No
Yes
No
Yes
T 3
Age <85
F
T 4 F
No Yes No Age <80

• 4 test cases satisfy the No

No
No
No
Yes
No
all path coverage <Yes,Yes,*> 1-2(T)-3(T)-5-7
Accept = true 5 Accept = false 6

criterion <Yes,No,No> 1-2(T)-3(F)-6-7 Return accept 7

<No,Yes,Yes> 1-2(F)-4(T)-5-7
<No,*,No> 1-2(F)-4(F)-6-7
Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

16
Statement coverage criterion
Tiêu chí bao phủ hết tập câu lệnh
Statement Coverage
• Main idea: Execute bool AccClient(agetype
each statement at age; gndrtype gender) Age, Gender 1

bool accept
least once if(gender=female)
T if(female)
2
F

• A possible concern accept := age < 85;

else
T
Age <85
F
T
Age <80
4
F

may be: accept := age < 80;

6
return accept 5

• dead code Accept = true Accept = false

AccClient(83, female)->accept Return accept

7

AccClient(83, male) ->reject

Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

17
Disadvantages of statement coverage
• Statement coverage is the weakest, indicating the fewest
number of test cases
• Bugs can easily occur in the cases that statement coverage
cannot see
• The most significantly shortcoming of statement coverage
is that it fails to measure whether you test simple If
statements with a false decision outcome.

18
 Branch coverage criterion
Tiêu chí bao phủ nhánh
• Also called Decision Coverage
• A branch is an outgoing edge from a node
• A rectangle node has at most one out going branch
• All diamond nodes have 2 outgoint branches
• A decision element in a program may be one of
• If – then
• Switch – case
• Loop
• Main idea: selecting paths such that every branch is included
in at least one path

19
Example

AccClient(83,
Branch Coverage /1 female)->accept

• We test the path bool AccClient(agetype Age, Gender 1

age; gndrtype gender)

• 1-2(T)-3(T)-5-7 bool accept
T
2
if(female) F

if(gender=female) T 3
Age <85
F
4 F
accept := age < 85; T
Age <80
true
else
accept := age < 80; Accept = true 5 Accept = false6
return accept
Return accept 7

Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

19

20
Example
AccClient(83, male)
Branch Coverage /2 ->reject

• We test the path bool AccClient(agetype Age, Gender 1

age; gndrtype gender)

• 1-2(F)-4(F)-6-7 bool accept
T if(female) 2 F

if(gender=female) T 3
Age <85
F
T 4 F
accept := age < 85; Age <80

else
accept := age < 80; Accept = true 5 Accept = false 6
false
return accept
Return accept 7

Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

21
Example

AccClient(78, male)-
Branch Coverage /3 >accept

• We test the path bool AccClient(agetype Age, Gender 1

• 1-2(F)-4(T)-5-7 age; gndrtype gender)

bool accept
T if(female) 2 F

if(gender=female) T 3
Age <85
F
4 F
accept := age < 85; T
Age <80
true
else
accept := age < 80;
Accept = true 5 Accept = false 6
true
return accept
Return accept 7

Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

22
Example
AccClient(88,
Branch Coverage /4 female) ->reject

• We test the path bool AccClient(agetype Age, Gender 1

age; gndrtype gender)

• 1-2(T)-3(F)-6-7 bool accept T
2
if(female) F

if(gender=female) T 3 F
Age <85
accept := age < 85; T 4
Age <80
F
false
else
accept := age < 80;
false Accept = true 5 Accept = false 6
return accept
Return accept 7

Lund University / Faculty of Engineering/ Department of Computer Science / Software Engineering Research Group

23
Comparing 3 criteria
• (1) All path coverage: assure 100% paths executed
• (2) Statement coverage: pick enough paths to assure that
every source statement is executed at least once
• (3) Branch coverage: assure that every branch has been
exercised at least once under some test
• (1) implies (3), (3) implies (2)
• These 3 criteria are also called as Path Testing Techniques

24
Example of statement and branch coverage

25
Example of statement and branch coverage
• Question 1:
• Does every decisions have a T and F values?
• Yes è Implies branch coverage
• Question 2:
• Is every link covered at least once?
• Yes è Implies statement coverage

Decisions Process Link

Path
4 6 7 9 a b c d e f g h i j k l m
abcde T T x x x x x
abhkgde F T F x x x x x x x
abhlibcde F T T x x x x x x x x
abcdfjgde T F T x x x x x x x x
abcdfmibcde T F F x x x x x x x

26
Example: Exponential Function

27
Example: Exponential Function

28
Example: Bubble sort

29
Example: Bubble sort

30
Limitations of path testing techniques
• Path Testing is applicable to new unit
• Limitations
• Interface mismatches and mistakes are not taken
• Not all initialization mistakes are caught by path testing
• Specification mistakes are not caught

31
5.3. Predicate Coverage
Tiêu chí bao phủ điều kiện

IT4501 - Software Engineering Department -

6/19/22 32
SoICT/HUST
 Basic concepts
• Predicate: are expresions that can be evaluated to a boolean
value (true or false)
• Predicate may contain:
• Boolean variables
• Non-boolean variables that are compared with the relational operators {≥, ≤, =, ≠
, <, >}
• Boolean function calls
• The internal structure is created by logical operators:
• ¬, →, ↔,∧,∨, ⨁
• A clause: is a predicate that does not contain any of the logic operator.
Example: (a=b) ∨ C ∧ p(x) contains 3 clauses

33
How to create a Path Predicate Expression?

• Write down the predicates

for the decisions you meet
along a path
• The result is a set of path
predicate expressions
• All of these expressions must
be satisfied to achieve a
selected path

34
Predicate Faults
Các lỗi điều kiện có thể xảy ra
• An incorrect Boolean operator is used
• An incorrect Boolean variable is used
• Missing or extra Boolean variables
• An incorrect relational operator is used
• Parentheses are used incorrectly

35
 Predicate Coverage

• To ensure all predicates in the

source code are implemented
correctly
• Main idea:
• For each predicate p, test
cases must assure that p
evaluates to true at least
once and p evaluates to
false at least once

36
Example

37
 Clause Coverage
• Main idea: For each
individual condition
(clause) c, c should
evaluate to true and c
should evaluate to false
at least once.

38
Example

39
 Predicate vs. Clause Coverage
• Does Predicate coverage
subsume clause coverage?
• Does clause coverage
subsume predicate coverage?
• Naturally, we want to test
both the predicate and
individual clauses

40
Predicate and Clause Coverage
• PC does not fully exercise all the clauses
• For example: p = a v b. In most programming languages, at run
time, when a evaluates to True, p evaluates to True and b does not
exercise.
• CC does not always ensure PC
• This is, we can satisfy CC without causing the predicate to
be both true and false
• This is definitively not what we want
• We need to come up with another approach
• Inversely, with PC, it is not always that all individual clauses
evaluate to both true and false

41
 Combinatorial Coverage
Bao phủ kết hợp
• Main idea: For each
predicate p, test cases
should ensure that the
clauses in p should
evaluate to each possible
combination of truth
values

42
Example 1
• Write all the clauses and the Combination of Clauses (CoC)
of the given predicate P = ((a >b) v C) ∧ p(x)

43
Example 1 - Answer

44
What is the problem with combinatorial
coverage?
• Combinatorial coverage is very expensive if we have
multiple clauses in the predicate
• For each decision point with N conditions, we need 2^N
test cases

45
Motivation
• Predicate should evaluate to both true and false in test
cases
• Each individual condition (clause) is tested independently
of each other
• Each individual clause should affect the predicate
• i.e., changing the value of an individual clause, the value of
predicate is also changed

46
Active clause
47
• Major clause: the
clause which is being
focused upon
• Minor clause: all other
clauses in the predicate
• Determination: clause
c in p determines p

47
Example: P = a ⋀ (b ⋁ c)
• Identify, when considering a as active clause, which values
should be assigned to b and c?
• Answer:
• a is active clause, the values of b and c are those such that
changing the value of a changes the value of P.
• So, (b v c) = 1, the value of P is determined by the value of a. We
can assign b = 1, c = 1
• Test case T = {(a=T, b=T), (a=F,b=F)} satisfy GACC, but not
PC, all both cases, P = T

48
Active Clause Coverage (ACC)
• For each predicate p and each major clause c of p, choose
minor clauses so that c determines p. 2 test requirements
for assuring ACC: for each c: c evaluates to true and c
evaluates to false
• With a decision point consists of N conditions, only N+1
test cases are required. Why?

49
Example

50
Example - Answer

51
Resolving the Ambiguity problem
Giải quyết vấn đề nhập nhằng
• With the example p =
a v (b ∧ c), a is active
clause
• How we can choose
the value of b and c? c
should be different
from b or not?

52
General Active Clause Coverage (GACC)
• For each clause c, choose minor
clauses’ values such that c
determines p
• Clause c has to evaluate to both
true and false
• Minor clauses don’t need to be
the same when clause c is true as
well as when c is false.
• cj(ci=true)=cj (ci = false) for all cj
OR cj(ci=true)!=cj (ci = false) for all
cj

Does GACC subsume predicate coverage? NOT always, ex: P = A <-> 53

6/19/22 B

53
General Active Clause Coverage (GACC)

a and (b or c) 1 0

a is active 1. . 0 . .

b is active . 1 . . 0 .

c is active . . 1 . . 0

All active values are in diagonal

54
General Active Clause Coverage (GACC)

a and (b or c) 1 0
a is active 11 0 0 0 1
Values of minor clauses are different
when active clause evaluates to true b is active 11 0 1 0 0
and false
c is active 1 0 1 1 0 0

Test case selected: 1 1 0, 1 0 1, 0 0 1, 1 0 0

55
 General Active Clause Coverage (GACC)

a and (b or c) 1 0
a is active 11 0 0 1 0
Values of minor clauses are the same
when active clause evaluates to true b is active 11 0 1 0 0
and false
c is active 1 0 1 1 0 0

Test case selected: 1 1 0, 1 0 1, 0 1 0, 1 0 0

56
Example

57
GACC and subsumption of CC/PC
• By definition: GACC subsumes CC, that is, satisfying GACC
leads to satisfy CC, but not PC
• Example: P = a <->b.
• Test case T = {(a=T, b=T), (a=F,b=F)} satisfy GACC, but not PC, all both
cases, P = T
• Test case T = {(a=T, b=F), (a=F,b=T)} satisfy GACC too, but not PC either, all
both cases, P = F

58
Correlated Active Clause Coverage (CACC)
• For each clause ci in P, choose minor clauses cj such that ci
determines the predicate P
• Clause ci has to evaluate to true or false in test cases
• The values chosen for the minor clauses cj must cause P to
be true for one value of the major clause ci and false for the
other value of ci
• P(ci = true) != P(ci = false)
• Minor clauses don’t need to be the same. That is:
• cj(ci=true)=cj (ci = false) for all cj OR cj(ci=true)!=cj (ci = false) for all cj

59
 Example
• When a is active
clause, which rows
can be chosen to
satisfy CACC?
• When a is active, values
of b and c are chosen
such that P(a=true) !=
P(a=false)
• Any of rows 1, 2, 3 and
any of row 5,6,7 - one
of total nine pairs
satisfies CACC

60
Test cases for CACC

a and (b or c) 1 0
a is active 11 0 0 1 0
b is active 11 0 1 0 0
c is active 1 0 1 1 0 0

Test case selected: 1 1 0, 1 0 1, 0 1 0, 1 0 0

61
CACC and subsumption of GACC/CC/PC
• By definition, CACC subsumes GACC thus CC and also PC
• Example: P = a<->b
• Which test cases satisfy PC?
• Which test cases satisfy CACC?

62
Example: P = a ⋀ (b ↔ c)
• Which test cases
satisfy GACC?
• Which test cases
satisfy CACC?

63
Restricted Active Clause Coverage
(RACC)
• For each predicate P and each active clause ci in P, choose
minor clause cj so that ci determines P. ci evaluates to both
true and false in test cases
• Predicate P evaluates to true in one case of major clause
and false in the other (that is CACC)
• The values choosen for minor clauses cj must be the same
when ci evaluates to true as when ci evaluates to false.
• That is, cj(ci=true)=cj (ci = false) for all cj
• This is the stricter version of CACC

64
 RACC Example
• When a is active clause,
which rows can be
chosen to satisfy RACC?
• Rows 1 and 8 are not
chosen because a is
not active in these
two cases
• b and c should have
the same values as
when a is true as well
as when a is false
• One of three pairs:
{1,5}, {2, 6} and {3, 7}
satisfy RACC

65
Test cases for RACC

a and (b or c) 1 0
a is active 11 0 0 1 0
b is active 11 0 1 0 0
c is active 1 0 1 1 0 0

Test case selected: 1 1 0, 1 0 1, 0 1 0, 1 0 0

66
RACC and subsumptions of
CACC/GACC/PC/CC
• By definition, RACC subsumes CACC, thus subsumes GACC, CC
and PC
• RACC often leads to infeasible test requirements
• Example: Consider the predicate P = ((a > b) && (b < c)) || (c>a) =
(X && Y)|| Z
• When Z is active clause, we choose X = 1, Y = 0. Infeasible test case: Z
= 0, X = 1, Y = 0

67
Logical Operators in Source Code
• & | ~ ^ correspond also to
logical operators.
• What is the difference
between (a&&b) and
(a&b)? (a||b) and (a|b)?

68
Code transformation

if if (a){
(a&&b) if (b)
S1; S1;
else else
S2; S2;
} else
S2;

69
 Exercise 1

• Identify test cases for

• PC
• CC
• CoC
• GACC
• CACC
• RACC

70
Exercise 2
Example: predicates/clauses in source code

• Determine predicates public static int daysInMonth(int m, int y) {

if (m <= 0 || m > 12)
and clauses in the source throw new IllegalArgumentException("Invalid month: "
if (m == 2) {
+ m);

code if (y % 400 == 0 || (y % 4 == 0 && y % 100 != 0))

return 29;

• Identify reachable else

return 28;
predicates of the source }
if (m <= 7) {
code if (m % 2 == 1)
return 31;
Predicates and clauses
p1: c1 || c2 ; c1: m <= 0, c2: m > 12
• Identify test cases that }
return 30;
p2: c3; c3: m == 2;

satisfy PC, CC and CoC if (m % 2 == 0)

return 31; p3: c4 || (c5 && c6) ; c4: y
c5: y % 4 == 0, c6: y % 100 !=
% 400 == 0,
0;
return 30;
• Are there infeasible } p4: c7; c7: m <= 7;

requirements ? p5: c8; c8: m % 2 == 1;

p6: c9; c9: m % 2 == 0

71
 Exercise 2
• Determine predicates Exercise 3
and clauses in the
source code public static int daysInMonth(int m, int y) {
if (m <= 0 || m > 12) // p1

• Identify reachable
throw new IllegalArgumentException("Invalid month: " + m);
if (m == 2) { // p2
predicates of the if (y % 400 == 0 || (y % 4 == 0 && y % 100 != 0)) // p3
return 29;
source code else
return 28;
Reachability predicates - r(p)

• Identify test cases }

if (m <= 7) { // p4
r(p1) = true (always reached)
r(p2) = ¬ p1 = m >= 0 ⋀ m < 12
that satisfy PC, CC and if (m % 2 == 1) // p5
return 31; r(p3) = r(p2) ⋀ p2 = (m > 0 ⋀ m < 12 ⋀ m == 2)
CoC }
return 30; = (m == 2)

• Are there infeasible if (m % 2 == 0) // p6 r(p4) = r(p2) ⋀ ¬ p2

return 31; r(p5) = r(p4) ⋀ p4 = (m > 0 ⋀ m < 12 ⋀ m != 2)
requirements ? }
return 30; ⋀ m <= 7

r(p6) = r(p4) ⋀ ¬ p4 = (m > 0 ⋀ m < 12 ⋀ m !=

2) ⋀ m > 7

Identify test cases that satisfy PC, CC, CoC (in this order).
13 Are there infeasible requirements?

72
PC Test Cases

Test Input
# PC m y
1 p1 0 -
2 ¬p1 1 -
3 p2 2 -
4 ¬p2 1 -
5 p3 2 2000
6 ¬p3 2 2017
7 p4 3 -
8 ¬p4 8 -
9 p5 3 -
10 ¬p5 4 -
11 p6 8 -
12 ¬p6 9 -

73
 CC Test Cases

# 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
CC c1 ¬c1 c2 ¬c2 c3 ¬c3 c4 ¬c4 c5 ¬c5 c6 ¬c6 c7 ¬c7 c8 ¬c8 c9 ¬c9

m 0 1 13 1 2 1 2 2 2 2 2 2 3 8 3 4 8 9
y - - - - - - 2000 2017 2000 2017 2017 2000 - - - - - -

74
CoC Test Case

# c1 c2 c3 c4 c5 c6 c7 c8 c9

1 T T infeasible

2 T F m=0

3 F T m=13

4 F F m=1

5 T m=2

6 F m=1

7 m=3 T

8 m=8 F

9 m=3 T

10 m=4 F

11 m=8 T

12 m=9 F

75
CoC Test Case

# c1 c2 c3 c4 c5 c6 c7 c8 c9

13 infeasible T T T

14 m=2, y=2000 T T F

15 infeasible T F T

16 infeasible T F F

17 m=2, y=2016 F T T

18 m=2, y=2100 F T F

19 m=2, y=2017 F F T

20 infeasible F F F

76
Exercise 3 Exercise 4
public static TClass triangleType(int a, int b, int c) {
if (a <= 0 || b <= 0 || c <= 0) /* p1 */
return INVALID;
if (a >= b + c || b >= a+c || c >= a+b) /* p2 */
return INVALID;
int count = 0;
if (a == b) /* p3 */ Identify:
count++;
if (a == c) /* p4 */ 1) the reachability predicates;
count++;
if (b == c) /* p5 */ 2) TR(CC) and TR(PC)
count++;
if (count == 0) /* p6 */ 3) test cases that satisfy a) PC b)
return SCALENE; CC
if (count == 1) /* p7 */
return ISOSCELES; 4) determination predicates for
return EQUILATERAL; the clauses of p1 and p2
}
5) TR(CACC)
6) test cases that satisfy a)
CACC b) RACC [are there
infeasible requirements?]
28

77
5.4. Data Flow Testing
Kiểm thử luồng dữ liệu

IT4501 - Software Engineering Department -

6/19/22 78
SoICT/HUST
Motivation
• S2: define x, assign to
x a value
• S6: use x to assign to
other variable
• If Branch Coverage
• 1-2-4-5-7
• 1-3-4-6-7
• We cannot explore
simultaneously the
relationship between
definition of x in
statement 2 and the
use of x in statement 6

79
Motivation
• GOAL: Try to ensure that values are computed and used
correctly
• Consider: how data gets accessed and modified in the system
and how it can get corrupted
• Common access-related bugs:
• Using an undefined or uninitialized variable
• Deallocating or reinitializing a variable before it is constructed,
initialized or used
• Deleting a collection object leaving its members
unaccessible

80
Variable Definition
Định nghĩa biến
• A program variable is DEFINED whenever its value is modified
• on the left hand side of an assigment statement, e.g., y = 20
• in an input statement, e.g., read(y)
• as an call-by-reference parameter in a subroutine call, e.g.,
update(x, &y)

81
Variable Use
Sử dụng biến
• A program variable is USED whenever its value is read:
• On the right hand side of an assignment statement, e.g., y = x +
17, x is used, y is defined
• as an call-by-value parameter in a subroutine or function call,
e.g., y = sqrt(x)
• in the predicate of a branch statement, e.g., if (x > 0){...}

82
 Variable use: p-use and c-use

• Use in the predicate of a branch statement is a predicate-use or p-

use
• Any other use is a computation-use or c-use
• For example, in the code below, there is a p-use of x and a c-use of y

if (x > 0) {
print(y);
}

83
Variable Use
• A variable can also be used and then re-defined in a single
statement when it appears
• on both sides of an assignment statement, e.g., y = y + x
• as an call-by-reference parameter in a subroutine call, e.g.,
increment(&y)

84
 Terminology
Các thuật ngữ khác

• Definition-Clear-Path
• A path is definition-clear (“def-
clear”) with respect to a
variable v if it has no variable
re-definition of v on this path
• p1 is def-clear path of v while
p2 is not

85
 Terminology
Các thuật ngữ khác

• Complete-Path
• A path is complete if the initial
node of this path is a entry
node and the final node of this
path is an exit node
• p1, p2 are not complete while
p3 is

86
 Definition-Use Pair (DU-pair)
Cặp định nghĩa – sử dụng

• A definition-use pair (du-pair) with respect

to a variable v is a pair (d, u) such that
• d is a node defining v
• u is a node or edge using v
• When it is a p-use of v, u is an out-
going edge of the predicate
statement
• There is a def-clear path with respect
to v from d to u

87
Example

1. input(A,B)
if (B>1){
2. A = A + 7
}
3. if (A > 10){
4. B = A + B
}
5. output(A,B)

88
Example: Identifying DU-Pairs of
Variable A

89
Example: Identifying DU-Pairs of
Variable A

90
Example: Identifying DU-Pairs of
Variable A

91
Example: Identifying DU-Pairs of
Variable B

92
Data Flow Test Coverage Criteria
Các tiêu chí kiểm thử bao phủ luồng dữ liệu

• All-defs coverage – Bao phủ tất cả các điểm định nghĩa dữ liệu trên đồ thị
• All-uses coverage – Bao phủ tất cả các điểm sử dụng dữ liệu trên đồ thị
• All-DU-Paths coverage – Bao phủ tất cả các đường dẫn DU trên đồ thị
• All-p-uses/Some-c-uses coverage – Bao phủ tất cả các điểm sử dụng dữ
liệu trong các câu lệnh rẽ nhánh và một vài điểm sử dụng dữ liệu
• All-c-uses/Some-p-uses coverage – Bao phủ tất cả các điểm sử dụng dữ
liệu và một vài điểm sử dụng dữ liệu trong các câu lệnh rẽ nhánh
• All-p-uses coverage – Bao phủ tất cả các loại sử dụng dữ liệu trong điều
kiện
• All-c-uses coverage – Bao phủ tất cả các loại sử dụng thông thường của
dữ liệu

93
All-Defs Coverage
Bao phủ tất cả các điểm định nghĩa
• For every program variable v, at least one def-clear path from
every definition of v to at least one c-use or one p-use of v must
be covered

94
 Example 1

• All Defs requires

• d0(x) to a use
• Satisfactory Path:
• <0-1-2-4-6>

95
 Example 2

• Identify all test cases to satisfy All-

defs
• Two variables: A and B
• Consider a test case executing
path
• t1: <1,2,3,4,5>
• t1 satisfy All-defs or not?

96
Example – Considering Variable A

97
Example – Considering Variable B

98
Answer
• Since <1,2,3,4,5> covers at least one def-clear path
from every definition of A or B to at least one c-use
or p-use of A or B è At least one du-pair from every
definition of A (n1 and n2) and of B (n1, n4) are
covered!
• All-Defs coverage is achieved

99
All-Uses Coverage
Bao phủ tất cả các điểm sử dụng
• For every program variable v, at least one def-clear path
from every definition of v to every c-use and every p-use
(including all outgoing edges of the predicate statement) of
v must be covered
• Requires that all du-pairs covered

100
Example 1
• All-Uses requires
• a: d0(x) to a u2(x)
• b: d0(x) to u3(x)
• c: d0(x) to u5(x)
• Satisfactory Path:
• <0-1-2-4-5-6>
satisfies a, c
• <0-1-3-4-6>
satisfies b
Example 2

• Consider two executing paths:

• t2: <1,3,4,5>
• t3: <1,2,3,5>
• Do all three test cases t1, t2,
t3 provide All-Uses coverage?

102
Example: Def-clear paths by t2 for
Variable A

103
Example: Def-clear paths by t2 for
Variable B

104
Example: Def-clear paths by t3 for
Variable A

105
Example: Def-clear paths by t3 for
Variable B

106
Answer
• Since all the du-pairs for variables A and B are
covered by 3 test cases è All-Uses coverage is
achieved
• Note: all du-pairs means: at least one def-clear path
from every definition to every c-use/p-use of the
variable

107
Example 3

108
Du-Pairs with respect to Variable X

109
Du-Pairs with respect to Variable X

110
Du-Pairs with respect to Variable X

111
Du-Pairs with respect to Variable X

112
Du-Pairs with respect to Variable X

113
Du-Pairs with respect to Variable X

114
More Dataflow Terms and Definitions
• A path (either partial or complete) is simple if all edges
within the path are distinct, i.e., different
• A path is loop-free if all nodes within the path are
distinct, i.e., different

115
Example: Simple and Loop-Free Paths

116
DU-Path
• A path <n1, n2, ... , nj, nk> is a du-path with respect to a
variable v, if v is defined at node n1 and either:
• there is a c-use of v at node nk and <n1, n2, ... , nj, nk> is a def-
clear simple path, or
• there is a p-use of v at edge <nj, nk> and <n1, n2, ... , nj> is a def-
clear loop-free path

117
Example 3: Identify du-path

118
 All DU-Paths Coverage

For every program

variable v, every du-
path from every
definition of v to
every c-use and
every p-use of v
must be covered

119
Example 2: All DU-
paths coverage

• Identify all DU-Paths

for variable A and B

120
Example 3: All DU-
paths coverage

• Identify all DU-

Paths for variable X
and Y

121
Data Flow Testing
Summary

• Relationship between
Data Flow Testing and
Path Testing
• Path testing with
Branch Coverage and
Data-flow testing with
All-uses is a very good
combination in
practice
Exercise 1

• Draw a data flow

graph for the
binsearch function
• Find a set of complete
paths satisfying all-
defs criterion with
respect to variable mid
• Do the same for
variable high
Exercise 2

• Using All-DU paths to

test the pow function
• Using All-Uses
combining with
Branch Coverage to
test this function
5.5. Mutation Testing
Kiểm thử đột biến

125
Mutation Testing
• Mutation testing (Kiểm thử đột biến) is a structural
testing method
• i.e., We have to use the structure of code to guide the test
process
• We cover the following aspects of Mutation Testing
• What is a mutation?
• What is mutation testing?
• When should we use mutation testing?
• Mutations
• Mutation Testing tools

126
What is a mutation?
• A mutation is a small change in the program
• Such small changes are intended to model low level
defects that arise in the process of coding systems
• Ideally, mutations should model low-level defect
creation

127
When should we use mutation
testing?
• Structural test suites are dỉrected at identifying defects
in the code
• One goal of mutation testing: To assess or improve
the efficacy of test suites in discovering defects
• Defects remaining in the code is measured through
Residual Defect Density (RDD)
• RDD is measured in defects per thousand lines of code
• Given the program under test P
• P satisfies all the tests in a test suite T
• Mutation testing on P provide an estimate of RDD on P

128
Using Mutation Testing to estimate RDD

• Given program P, test suite T

• Suppose we have 𝑟 , an estimate of RDD on P – could
be based on the number of faults our tests have
already detected
• Generate n mutants of P
• Test each mutant with the test suite T
• Find the number, k, of mutants that are killed by T
• RDD = r x (n-k)/k
• k/n is a measure of the adequacy of T in finding defects
in P

129
An approach to Mutation
• Consider mutation operators in the form of rules that
match a context and create some systematic mutation
of the context to create a mutant
• Simple approach: to consider all possible mutants but
this may create a very large number of mutants

130
Kinds of mutation
• Value Mutation
• Changing the values of constants or parameters (by
adding or subtracting a value)
• e.g., changing loop bounds
• Decision Mutation
• Modifying conditions to reflect potential slips and errors
in the coding of conditions in program
• e.g., replacing < by >
• Statement Mutation
• Deleting certain lines to reflect omissions in coding
• Swapping the order of lines of code
• Changing operations in arithmetic expressions
• ...

131
Mutations for Inter-class Testing

132
Value Mutation (đột biến giá trị)
• Typical examples:
• Changing values to one larger or smaller
• Swapping values in initializations
• The commonest approach is to change constants by
one in an attempt to generate a one-off error
• Coverage criterion: Perturb all constants in the
program or unit at least once or twice

133
Value Mutation - Example

134
Decision Mutation
(Đột biến điều kiện)
• Typical examples:
• Modelling “one-off” error by changing < to <= or vice versa
(common in checking loop bounds)
• Modelling confusion about larger and smaller by changing < to
> or vice versa
• Getting parenthesisation wrong in logical expressions:
mistaking precedence between && and ||
• Coverage Criterion: consider one mutation for each
condition in the program

135
Decision Mutation
(Đột biến điều kiện)

136
Statement Mutation
(Đột biến câu lệnh)
• Typical examples:
• Deleting a line of code
• Duplicating a line of code
• Permuting the order of statements
• Coverage Criterion:
• Applying this procedure to each statement in the program

137
Statement Mutation
(Đột biến câu lệnh)

138
Mutation Testing Tools
• Stryker JS: mutation testing for javascript
• Mutant: automated code reviews via mutation testing
• Infection: PHP mutation testing library
• Pitest: State of the art mutation testing for the JVM
• Cosmic Ray: mutation testing for Python
• Muter: automated mutation testing for Swift
• ...

139
 Thank you
for your
attention!!!

140
1
 Software Quality Assurance
Đảm bảo chất lượng phần mềm
Lecture 6: Software Testing Process

2
Contents
Những nội dung chính

• Software Testing Lifecycle

• Planning and Organization

• System Integration Testing

• Acceptance Test

3
6.1. Software Testing
Lifecycle

4
Process models from waterfall to agile

5
Phases of testing in the development process

6
V-model Rationale
• A modified version of waterfall model
• Tests are created at the point the activity they validate
is being carried out
• e.g., The acceptance test is created when the system analysis
is carried out
• Failure to meet the test requires a further iteration
beginning with the activity that has failed the validation
• V-model focuses on creating tests in a structured
manner

7
Boehm’s Spiral Model

8
Spiral Model Rationale
• The spiral model focuses on controlling project risk and
attempting formally to address project risk throughout
the lifecyle
• V&V activity is spread through the lifecycle with more
explicit validation of the preliminary specification and
early stages of the design
• At the early states, there may be no code available è
working with models of the system and environment
and verifying that the model exhibits the required
behaviors

9
XP principles
• eXtreme Programming advocates working directly with
code almost all the time
• 12 principles of XP summarise the approach
• Development is Test Driven
• Tests play a central role in refactoring activity
• Agile development mantra: Embrace Change

10
12 principles of XP (recap)
1. Test driven development
2. Planning game
3. On-site customer
4. Pair programming
5. Continuous integration
6. Refactoring
7. Small releases
8. Simple design
9. System metaphor
10. Collective code ownership
11. Coding standard
12. 40-hour work week

11
XP Rationale

12
6.2. Planning and
Organisation

13
Testing as a planned activity
• Testing should be planned for during Requirement
Definitions and Specification
• Allocate human and computer resources for testing
• Develop testing tools, test drivers, databased for test
data etc.
• Testability is one of the requirement of every software
system
• A Test Plan is usually developed to describe the testing
process in detail
• Testing should be traceable back to requirements and
specification

14
Testing Documentation
• Testing requirements and Test Specification are
developed during the requirements and specification
phases of the system design
• Test Plan describes the sequence of testing activities
and describes the testing software that will be
constructed
• Test Procedure specifies the order of system
integration and modules be tested
• Also describes unit test and test data
• Logging of tests conducted and archiving of test results
are often important tasks of Test Procedure

15
Test Plan components
• Establish test objectives

• Designing test cases

• Writing test cases

• Testing test cases

• Executing tests

• Evaluate test results

16
Test Organisation
• Support decision making
• Enhance teamwork
• Independency
• Balance testing quality
• Assist test management
• Ownership of test technology
• Resources utilization
• Career path

17
7 approaches for test organisation
1. Each person’s responsibility
2. Each unit’s responsibility
3. Dedicated resource
4. Test organisation in QA
5. Test organisation in development
6. Centralized test organisation
7. Test technology centre

18
1. Each person’s responsibility

19
2. Each unit’s responsibility

20
3a. Dedicated Resources

21
3b. Dedicated resources at large
scale

22
4. Test organisation in QA

23
5. Test organisation in development

24
6. Centralized test organisation

25
7. Test technology centre

26
Which organisation to be chosen?
• Depending on
• size
• maturity
• focus
• localization

The solution is often a mixture of different approaches

27
Discussion
• Comparing two alternative organisations

28
6.3. System Testing

29
System test
• System testing is very heterogeneous and what we
include in the system test will depend on the particular
application
• The following is a list of kinds of tests we might
consider applying and some assessment of their
strengths and weaknesses.
• System testing can be very expensive and time
consuming and can involve the construction of
physical components and software to provide the test
environment that may exceed the cost of the primary
software development.

30
Capacity Testing
• When: systems that are intended to cope with high volumes
of data should have their limits tested and we should
consider how they fail when capacity is exceeded
• What/How: usually we will construct a harness that is
capable of generating a very large volume of simulated data
that will test the capacity of the system or use existing
records
• Why: we are concerned to ensure that the system is fit for
purpose say ensuring that a medical records system can
cope with records for all people in the UK (for example)
• Strengths: provides some confidence the system is
capable of handling high capacity
• Weaknesses: simulated data can be unrepresentative; can
be difficult to create representative tests; can take a long
time to run

31
Stress Testing
• When: in systems that are intended to react in real-time e.g.
control systems, embedded systems, e.g. anti-lock brake
system
• What/How: usually we are interested in bursty traffic and
environmental extremes (e.g. lots of sensor messages
together with cold and wet conditions). Could build a test rig
to test the integration of the mechanical, electronics and
software. If changes are required in service then we can use
a real system.
• Why: these systems are most depended on in these kinds
of conditions - it is imperative that they function here.
• Strengths: this is an essential kind of testing for this class
of systems unavoidable.
• Weaknesses: test environment may be unrepresentative or
may omit a component (e.g. in a car the radio environment
is very noisy we may need to simulate this).
32
Usability Testing
• When: where the system has a significant user interface and it is
important to avoid user error
• What/How: we could construct a simulator in the case of
embedded systems or we could just have many users try the
system in a controlled environment.
• Why: there may be safety issues, we may want to produce
something more useable than competitors’ products...
• Strengths: in well-defined contexts this can provide very good
feedback – often underpinned by some theory e.g. estimates of
cognitive load.
• Weaknesses: some usability requirements are hard to express
and to test, it is possible to test extensively and then not know
what to do with the data.

33
Security Testing
• When: most systems that are open to the outside world and have
a function that should not be disrupted require some kind of
security test. Usually we are concerned to thwart malicious users.
• What/How: there are a range of approaches. One is to use
league tables of bugs/errors to check and review the code (e.g.
SANS top twenty-five security- related programming errors). We
might also form a team that attempts to break/break into the
system.
• Why: some systems are essential and need to keep running, e.g.
the telephone system, some systems need to be secure to
maintain reputation.
• Strengths: this is the best approach we have most of the effort
should go into design and the use of known secure components.
• Weaknesses: we only cover known ways in using checklists
and we do not take account of novelty using a team to try to break
does introduce this.

34
Performance Testing
• When: many systems are required to meet performance
targets laid down in a service level agreement (e.g. does
your ISP give you 2Mb/s download?).
• What/How: there are two approaches -
modelling/simulation, and direct test in a simulated
environment (or in the real environment).
• Why: often a company charges for a particular level of
service - this may be disputed if the company fails to
deliver. E.g. the VISA payments system guarantees 5s
authorisation time delivers faster and has low variance.
Customers would be unhappy with less.
• Strengths: can provide good evidence of the performance
of the system, modelling can identify bottlenecks and
problems.
• Weaknesses: issues with how representative tests are.
35
Reliability Testing
• When: we may want to guarantee some system will only fail
very infrequently (e.g. nuclear power control software we
might claim no more than one failure in 10,000 hours of
operation). This is particularly important in
telecommunications.
• What/How: we need to create a representative test set and
gather enough information to support a statistical claim
(system structured modelling supports demonstrating how
overall failure rate relates to component failure rate).
• Why: we often need to make guarantees about reliability in
order to satisfy a regulator or we might know that the market
leader has a certain reliability that the market expects.
• Strengths: if the test data is representative this can make
accurate predictions.
• Weaknesses: we need a lot of data for high-reliability
systems, it is easy to be optimistic. 36
Compliance Testing
• When: we are selling into a regulated market and to sell
we need to show compliance. E.g. if we have a C
compiler we should be able to show it correctly compiles
ANSI C.
• What/How: often there will be standardised test sets that
constitute good coverage of the behaviour of the system
(e.g. a set of C programs, and the results of running
them).
• Why: we can identify the problem areas and create tests
to check that set of conditions.
• Strengths: regulation shares the cost of tests across
many organisations so we can develop a very capable
test set.
• Weaknesses: there is a tendency for software producers
to orient towards the compliance test set and do much 37
worse on things outside the compliance test set.
Availabilty Testing
• When: we are interested in avoiding long down times we
are interested in how often failure occurs and how long it
takes to get going again. Usually this is in the context of a
service supplier and this is a Key Performance Indicator.
• What/How: similar to reliability testing – but here we
might seed errors or cause component failures and see
how long they take to fix or how soon the system can
return once a component is repaired.
• Why: in providing a critical service we may not want long
interruptions (e.g. 999 service).
• Strengths: similar to reliability.
• Weaknesses: similar to reliability – in the field it may be
much faster to fix common problems because of learning.

38
Documentation Testing
• When: most systems that have documentation should
have it tested and should be tested against the real
system. Some systems embed test cases in the
documentation and using the doc tests is an essential
part of a new release.
• What/How: test set is maintained that verifies the doc
set matches the system behaviour. Could also just get
someone to do the tutorial and point out the errors.
• Why: the user gets really confused if the system does
not conform to the documentation.
• Strengths: ensures consistency.
• Weaknesses: not particularly good on checking
consistency of narrative rather than examples.
39
Configuration Testing
• When: throughout the life of the system when the
software or hardware configuration is altered.
• What/How: usually we record a set of relationships or
constraints between different components e.g. libraries
and systems, hardware and drivers and if we change any
component we should check the configuration is
maintained.
• Why: configuration faults will often cause failures if they
are not corrected.
• Strengths: addresses an increasingly common source of
error. It is possible to automate this style of testing.
• Weaknesses: only as good as the record of
dependencies recorded in the system. There is no
universal approach but there are emerging standards to
record configurations. 40
Summaries for System testing
• There are a very wide range of potential tests
that should be applied to a system.
• Not all systems require all tests.
• Managing the test sets and when they should be
applied is a very complex task.
• The quality of test sets is critical to the quality of
a running implementation.

41
6.4. Acceptance Testing

42
Acceptance Testing
• Carried out by the customer on their premises.
• Testing is carried out independent of the
development organisation.
• The acceptance test marks the transition from
development before use to development in use
• Help the customer to decide whether or not to accept
the system

43
Kinds of Acceptance testing
• User acceptance testing
• Conducted by the customer to ensure that the
system satisfies the contractural acceptance
criteria
• Actual planing and execution of acceptance
tests do not to be undertaken directly by the
customer
• Consult a third-party to do this task
• Business acceptance testing
• Undertaken within the development
organisation
• The development organisation derives and
executes test cases from client’s contract

44
Acceptance Criteria
• Functional correctness and completeness
• All the features described in the requirement specification
must be present in the delivered system
• Accuracy
• Does the system provide correct results?
• Defined in terms of the magnitude of the error
• Data Integrity
• The preservation of data while it is trasmitted or stored such
that the value of data remains unchanged
• Data Conversion
• The conversion of one form of computer data to another
• Test programs or procedures that are used to convert data
from existing systems for use in replacement systems

45
Acceptance Criteria
• Usability
• How easy it is to use the system?
• How easy it is to learn?
• Performance
• The desired performance characteristics of the system must
be defined for the measured data to be useful
• Reliability and Availability
• What is the current failure rate of the software?
• What will be the failure rate if the customer continues
acceptance testing
• for a long time?
• How many defects are likely to be in the software?
• How much testing has to be performed to reach a particular
failure rate?

46
 Thank you
for your
attention!!!

47
1
 Software Quality Assurance
Đảm bảo chất lượng phần mềm
Lecture 7: Testing techniques and tools
Contents

• Regression Testing
• Exploratory Testing
• Agile testing
• Automation testing

3
7.1. Regression Testing

4
Regression Testing
• Regression testing is applied to code immediately after
changes are made
• Goal: to assure that the changes have not had
unintended consequences on the behaviour of the test
object
• Apply regression testing during
• Development phase
• After system have been upgraded or maintainted
• Regression testing is very important to monitor the
effect of changes
Why use regression testing?
• Good reasons:
• Bug fixes often break other things the developer isn’t
concentrating on
• Somtimes bug fixes don’t fix the bug
• Checking software still runs after making a change
• Discovery faulty localization
• Errors in build process
• Conforming to standard or regulators
• Bad reasons:
• Arguments in terms of replicability of results
• Arguments in terms of quality in analogy with a production line

6
Risks of change
• Bug regression testing: checks that a bug fix has removed
the symptoms of the bug that have been identified
• Old fix regression: checks that a new fix has not broken an
old fix: refactoring should limit this as old fixes are
refactored into the code.
• Functional regression: new code or fix has not broken
previously working code.
• Incremental Regression testing: regression testing as we
develop.
• Localisation Testing: tests if a product has been correctly
localised for a particular market.
• Build Testing: has an error been introduced in the field that
means the system will not build correctly.

7
Motivation for reusing tests
• In development (e.g. XP) tests play the role of
specifications so we want to keep them fixed and
reduce the cost of regression.
• In an established product:
• Using the same tests may help us manage risk since we
can focus tests on mitigating a particular risk.
• Some tests are good at uncovering likely errors so we
want to reuse.
• There may be economic motivations:
• Automated retest (replay or oracle).
• Replay with human inspection may reduce the need for
specialist technical time (e.g. in GUI testing – this is a
particularly common approach). The aim is to routinise
repeat testing.

8
Key questions of reuse
• Which test should we reuse?
• What is the cost of maintaining tests?
• Complex tests may make extensive use of the environment
• Complex to maintain
• Developing test architecture to support tests, e.g., web
services
• What is the cost of applying tests?
• What is the cost of applying regression tests?

9
Fault region model
• Systems have fault regions where their behaviour is does not
conform to the requirements.
• Tests are point executions of the system.
• Test specifications may specify a region in the input space

10
Clearing mines

11
Totally repeatable tests won’t clear
mines

12
Variable tests are often more effective

13
Economic perspective
• What is the best way to improve product quality?
• Maintain a regression test set
• Develop new tests
• It is possible to develop new tests for low value
events (e.g. patch bundles)
• What is the benefit of reusing tests?
• Tends to focus on core functionality of the system
• Perhaps takes a narrow view of the functionality
• Costs:
• How much does it cost to maintain tests?
• How much does it cost to create tests?

14
Support for refactoring
• Tests act as an executable specification.
• Tools like JUnit reduce the cost to the
developer.
• Tendency to focus on unit level behaviour.
• Tendency to focus on function over resource
use.
• Issues about how to integrate many unit level
test sets that have been created individually.

15
Risk Management
• Tests target critical behaviour the main hazards.
• For embedded systems we have good specifications
and it may be possible to infer more from a test
result.
• We can use combinations of old tests to exercise the
system more extensively on retest:
• More tests.
• More combinations of test.
• More variants.
• With a good specification we can see how the tests cover
the different behaviours of the system.
• We provide independent testers with a large armoury of
possible weapons to break the system.

16
7.2. Exploratory Testing

17
What is Exploratory Testing?
• A style of software testing that
• Emphasizes the personal freedom and
responsibility of the individual tester
• Continually optimize the value of her work
• Treat test-related learning, test design, test
execution, and test results interpretation as
mutually supportive activities that run in
parallel throughout the project
• Widely used in Agile models and is all
about discovery, investigation and
learning
18
Scripted vs Exploratory Testing

19
Scripted vs Exploratory Testing

20
Exploratory Testing
• Is not random testing but is ad-hoc testing with a
purpose of finding bugs
• Is structured and rigorous
• Is highly teachable and manageable
• Not a technique but an approach

21
How to do Exploratory Testing?
1. Create a bug taxonomy (classification)
• Categorize common types of faults found in the
past project
• Analyze the root cause analysis of the problems
or faults
• Find the risks and develop ideas to test the
application
2. Test Charter
• Suggest what to test
• How it can be tested
• What need to be looked
• Help determine how the end user could use the
system
22
How to do Exploratory Testing?
3. Time Box
• This method includes a pair of testers working
together not less than 90 minutes
• There should not be any interrupted time in those
90 minutes session
• Timebox can be extended or reduced by 45
minutes
• This session encourages testers to react on the
response from the system and prepare for the
correct outcome

23
How to do Exploratory Testing?
4. Review Results
• Evaluation of the defects
• Learning from the testing
• Analysis of coverage areas
5. Debriefing
• Compilation of the output results
• Compare the results with the charter
• Check whether any additional testing is needed

24
Variations of Exploratory Testing

25
7.3. Agile Testing

26
Software development
Agile software development (*)

READY DONE

(*) these slides are from: Dave Mattingly

Agile testing
• Imagine, Plan, Make, Test, Deliver
Agile Testing - TDD
• Test Driven Development
• Make it Fail
• Make it Work
• Make it Better
Agile Testing - TDD
Agile Testing - TDD
• Tools: csUnit, jUnit, nUnit, BusterJS
Agile testing - BDD
• Behavior Driven Development
• Given
• When
• Then
Agile testing - BDD
Agile testing - BDD
Agile testing - BDD
• Tools: Cucumber, RSpec, SpecFlow
Agile testing - ATDD
• Acceptance Test Driven Development
• Discuss

• Distill

• Develop

• Demonstrate
Agile testing - ATDD
• Discuss
• What is a valid password?
• What characters are mandatory?
• When should they change?
• Can changed passwords repeat?
• How will we know it works?
• What are some specific examples?
Agile testing - ATDD
• Distill
Agile testing - ATDD
• Develop
Agile testing - ATDD
• Demonstrate

• Tools: EasyB, FitNesse, JBehave, SpecTacular

Agile Testing - Auto
• Automated Regression Testing
• Simulates real-world experiences
• Eliminates repetitive tests
• Eases complex tests
Agile Testing - Auto
• Tools: Selenium, Silk, Concordion
Considerations
• TDD – implementation
• Is it working?
• BDD – system behavior
• Is it rights?
• ATDD – requirements
• Is it useful?
• Automated Regression – availability
• Is it reliable?
Considerations
• Adoption
• Promotion
• Bugs
• Documentation
• Versioning
• Notifications
Considerations
• Test everywhere

ATDD BDD TDD QA Auto

Considerations
• Applications
• Data
• Performance
• Availability
• Roles
• Accessibility
• Security
Considerations
7.4. Automation Testing

49
Testing tools by process

50
What to automate?
Test cases that are:
• Less volatile
• Repeatable
• High risk
• Easy to automate
• Manually difficult
• Boring and time consuming

51
Evolution of Test Automation

52
Automating different steps

53
Relationship of testing activities

54
Test automation promises
1.Efficient regression test
2.Run tests more often
3.Perform difficult tests (e.g. load, outcome
check)
4.Better use of resources
5.Consistency and repeatability
6.Reuse of tests
7.Earlier time to market
8.Increased confidence

55
Common problems
1.Unrealistic expectations
2.Poor testing practice
3.Expected effectiveness
4.False sense of security
5.Maintenance of automatic tests
6.Technical problems (e.g. Interoperability)
7.Organizational issues

56
Limits of automated testing
• Does not replace manual testing
• Manual tests find more defects than automated
tests
• Does not improve effectiveness
• Greater reliance on quality of tests
• Oracle problem
• Test automation may limit the software
development
• Costs of maintaining automated tests

57
New automation perspectives

58
 Thank you
for your
attention!!!

59
1
 Software Quality Assurance
Đảm bảo chất lượng phần mềm
Lecture 8: Software Reliability

2
Contents
• Definition of Reliability
• Reliability Criteria
• Operation Profile

3
8.1. Reliability

4
Definition of Reliability (1)
• The probability of failure-free operation of a software
system for a specified time in a specified environment
• The key elements:
• Probability of failure free operation
• Length of time of failure free operation
• A given execution environment

5
Definition of Reliability (2)
• Failure intensity is a measure of the reliability of a
software system operating in a given environment
• The lower the failure intensity of a software system, the
higher is its reliability

6
What is the difference between two Definitions?

7
Failures vs Faults
• Software Failure: an incorrect result with to the
specification or unexpected software behavior perceived by
users
• Software Fault: identified or hypothesized cause of the
software failure
• Failure – Effect vs Fault – Cause
• Defect: generic term referring to both failure and fault

8
 Time Intervals

• Execution time • The CPU time that is actually

spent by the computer in
executing the software
• The time people normally
• Calendar time experience in terms of years,
months, weeks, days…
• The elapsed time from start
• Clock time to end of computer execution
in running the software
• What is the most relevant ?

9
Questions to estimate the Software Reliability

• Questions about the Reliability in terms of time and failure:

• What is the time interval between two successive failures?
• How many failures have been observed within a certain time
interval, for example in the past one month or one year?
• What is the total number of failures observed so far?
• Answer these questions give an indication of the quality
level of a software

10
Time Interval between Failures
• What does the small time interval between successive
failures tell?
• System is failing frequently and the reliability level is too low
• Reliability metrics
• Mean Time to Failure (MTTF)
• Mean Time to Repair (MTTR)
• Mean Time between Failures (MTBF)
• MTBF = MTTF + MTTR

11
 15.1.4 Counting Failures in Periodic Intervals
Useful information can be gathered by monitoring the cumulative failure count
Relationship between MTTR, MTTF, MTBF
on a periodic basis. For example, we record the cumulative failure count every

Occurrences of failures Repairs performed

Start of
system operation

Time

MTTR

MTTF
MTBF

Figure 15.1 Relationship between MTTR, MTTF, and MTBF.

12
8.2. Mathematical
Model of Reliability

13
Mathematical Modelling of Software
Reliability
• Motivation
• Counting the number of failures observed so far and plotting the
data as a function of time to express the change in the reliability of
the system
• A rising graph of the cumulative number of failures shows that
there are more faults in the system
• The rate of rising of the graph is the rate at which failures are being
observed
• What is the meaning of small rate of rising of graph ?

14
Mathematical Functions of Failures
• Cumulative failures are counted in periodic intervals
• Count cumulative failures every month
• Time: CPU time (τ) or calendar time (t)
• Cumulative failure function 𝜇 (τ)
• The total number of failures observed until execution time τ from the
beginning of the system execution
• Failure intensity function 𝜆 (τ)
• The number of failures observed per unit time after τ time units of
executing the system from the beginning
!"($) $
•𝜆 𝜏 = 𝜇(𝜏) = ∫' 𝜆 𝑥 d𝑥
&$

15
8.3. Factors

16
Factors influencing software reliability
• The software reliability depends on two categories of
information:
• The way users operate the system – also known as operational
profile
• The number of faults present in the software system
• Size and Complexity of code
• Characteristics of Development Process
• Education, Experience, and Training of Personnel
• Operational environment

17
Applications of Software Reliability
• Comparison of Software Engineering Technologies
• What is the cost of adopting technology?
• How does the new technology affect the development schedule?
• What is the return from the new technology in terms of software
quality?
• Measuring the Progress of System Testing
• Percentage of test cases executed
• Percentage of successful execution of high-priority functional tests
• Controlling the System in Operation
• Better insight into Software Development Process

18
8.4. Operational Profile

19
Operational Profiles
• Operational profile or usage profile
• Describe how actual users operate a system
• Estimate the reliability of a system depends essentially on
how it will actually be used in the field
• Representation of Operational Profile
• Set of operations and its probability of occurrence
• Ex: 3 operations A, B, C with probability of occurrence: 50, 30 and 2%
• Operational profile: {(A, 0.5), (B, 0.3), (C, 0.02)}
• Two ways to represent:
• Tabular representation
• Graphical representation

20
Tabular representation of Operational
Profile
• Tabular form with 3 columns
• Ex: Library Information System
• 1st column: names of operations
• 2nd column: the frequency of using the operation
• 3rd column: the probability of using the operation

21
484
Example CHAPTER 15 SOFTWARE RELIABILITY

TABLE 15.1 Example of Operational Profile of Library Information System

Operation Operations per Hour Probability

Book checked out 450 0.45

Book returned in time 324 0.324
Book renewed 81 0.081
Book returned late 36 0.036
Book reported lost 9 0.009
.. .. ..
. . .
Total 1000 1.0

Example of Operational Profile of Library Information System

Account
management = 0.4

2
Administration = 0.1
Reporting = 0.6
22
1 Check out = 0.5
Graphical representation of Operational
Profile
• Tree structure consisting of nodes and branches
• Nodes represent attributes of operations
• Branches represent values of attributes with the associated
probability of occurrence
• Example: Library Information System

23
 Book returned late 36 0.036
Book reported lost 9 0.009
.. .. ..
. . .
Total 1000 1.0
Example
Account
management = 0.4

2
Administration = 0.1
Reporting = 0.6

1 Check out = 0.5

3 Renewal = 0.09 .....

User service = 0.9
Loss = 0.01

Return = 0.4
Delayed = 0.1
4

In time = 0.9

Figure 15.2 Graphical representation of operational profile of library information system.

Graphical Representation of Operational Profile of Library Information System
nodes, namely 1, 2, 3, and 4 are shown. Node 1 represents the scope attribute
of operations with two values, administration and user service. The scope of an
operation refers to whether an operation is for the administration of the infor-
mation system or for providing services to users. The probability of occurrence24
of the administration value of the scope attribute of an operation is 0.1, and the
probability of occurrence of the user service value is 0.9. Node 2 represents an
Which form to choose?

25
 Thank you
for your
attention!!!

26