Software Testing
Basic Concepts
Instructor: Dang Tu Linh
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
04e-BM/NS/HDCV/FSOFT v2/2
Agenda
Duration: 1.5 Hours
Purpose: Understand basic Test concepts
Audience: All Members
Content:
1. Introduction
2. Defect concepts
3. Test process
4. Test plan & test design
5. Testing strategy
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
04e-BM/NS/HDCV/FSOFT v2/2
Introduction
Objectives:
To arrive at a common understanding of the scope of testing
activity
To familiarize the audience with the terminology used in the
testing work
Understand the required test skills of developers
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
04e-BM/NS/HDCV/FSOFT v2/2
Typical Project Blues
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
04e-BM/NS/HDCV/FSOFT v2/2
Testing definition
Testing is the process of exercising or evaluating a system or
system component by manual or automated means to verify
that it satisfies specified requirements - (IEEE 83a)
Process:
Sequence of steps performed for a given purpose. (IEEE)
Software Process
A set of activities, methods, practices, and transformations
that people use to develop and maintain software and
associated products.(SEI-CMM)
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
04e-BM/NS/HDCV/FSOFT v2/2
Testing Objectives
Primary: Execute a program with the intent of finding
errors to
Determine whether system meets specifications
Determine whether system meets users needs
Secondary
Continuously improve the testing process
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
04e-BM/NS/HDCV/FSOFT v2/2
Misunderstandings
about testing
Testing is debugging
Testing is not the job of a programmer
If programmers were more careful testing would be
unnecessary
Testing never ends
Testing activities start only after the coding is complete
Testing is not a creative task
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
04e-BM/NS/HDCV/FSOFT v2/2
Theory & Reality
Theory
Need sufficient time
Baselined & freezed requirements
Automation testing
Have trained testers: test process, business flow,
Reality
Time Pressures to keep delivery date
Continuously updated requirements
Dont know when to stop: Update & update Requirement
continuously (CRs) Tester updates TC and re-tests
Manual testing
Lack of dedicated software test environment: share,
steal
Inexperienced testers without appropriate training
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
04e-BM/NS/HDCV/FSOFT v2/2
Defect concepts
Objectives:
Understand the definition of defect
Understand the cost of a bug to software
development
Understand the common types of defects and
their perceived severity
What is defect?
A defect is any error found by testing and
reviewing activities (All errors found by internal
reviewer, external reviewer and customer).
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
04e-BM/NS/HDCV/FSOFT v2/2
Where defects come from?
Products
Quality Control
Review, Test, Audit, Inspection
Processes
SRS, DD, Source code, Test cases, etc.
Requirements, Design, Coding, Test, etc.
Other sources:
Change requests, misunderstanding, carelessness,
planning, etc.
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
10
04e-BM/NS/HDCV/FSOFT v2/2
Cost of defects
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
11
04e-BM/NS/HDCV/FSOFT v2/2
FSoft Defect lifecycle
What: To log defect intoWhat:
DMS -To analyze What:
the causes
of the the
defect,
To correct
defect
with clear information: project,
find-out solution and
assign
to person
who isto
and
perform
unit testing
module, stage, QC activity,
responsible for correcting
it. the defect has been
make sure
severity, type, due date,- tester,
removed.
To accept the defect if its acceptable (by
created date ...
concession of authority
customer).
Who:orDeveloper
Who: Tester
Who: Project Leader
Defects status:
Defects status: ERROR
CORRECTED
Defects status: ASSIGNED
/ ACCEPTED
Step 1
Log defects
Step 2
Analyze defects
Step 3
Fix defects
What: To retest the function(s)
containing the defect and make
sure that the defect has been
corrected completely
Who: Tester
Defects status: CLOSED
Step 4
Re-test
Defect still exists in the product, change status to ERROR
Defect classification:
Open defects
Closed defects
Leakage
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
12
04e-BM/NS/HDCV/FSOFT v2/2
Fsoft Defect classification
Defect status:
Error
Assigned
Corrected
Cancelled
Accepted: by PM/PTL/Customer
Closed: by tester/PM/PTL
Leakage Defects:
Acceptance test
After Release test
After Release review
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
13
04e-BM/NS/HDCV/FSOFT v2/2
Fsoft Defect classification
Defect type
Functionality
o Requirement misunderstanding
o Feature missing
o Coding logic
o Business logic
User Interface
Performance
Design issue
Coding standard
Severity: How bad the problem is
(some time disagreement btw. Dev & Tester)
Fatal: the system is broken
Serious: can not work around
Medium: impact the function
Cosmetic: not impact the function
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
14
04e-BM/NS/HDCV/FSOFT v2/2
Fsoft Defect classification
Work product
Software module
Software Package
ADD
DDD
Test Cases
(UT/IT/ST)
Quality control
activity
Code review
Unit test
Integration test
System test
Document review
Inspection
Audit
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
15
Defect origin
Requirement
Design
Coding
Test
CM
Document control
Priority
Immediately
High priority
Normal priority
Low priority
04e-BM/NS/HDCV/FSOFT v2/2
Test Process
Objectives:
Understand the steps followed for a test
process
Understand test planning process
FSoft Test Process
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
16
04e-BM/NS/HDCV/FSOFT v2/2
Tester Competency
Soft skills (non-technical attributes)
Discipline and Perseverance
Testing is repetitive and requests a lot of manual effort
Have the ability to withstand pressures and workload
Say no to managers when quality is insufficient
Reading skills: study many docs of specs, design
Communication & Interpreter skills
Both Verbal and Written communication
Diplomatic skill
Be able to communicate with technical and nontechnical people, engineers, managers, customers
Negative thinking: foresee things that can go wrong,
evaluate risks
Attitude: test to break attitude
Time management and Effort prioritization
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
17
04e-BM/NS/HDCV/FSOFT v2/2
Tester Competency
Technical skills
Familiarity with software development
architectures, processes
Familiarity with testing methodologies
Very clearly articulate specs, designs, business
rules, inspection reports, configurations, code
changes, TP, TC, bug reports, user manuals,
Know how and where to look for bugs
Know how to report the defects effectively for
quickly reproduce
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
18
04e-BM/NS/HDCV/FSOFT v2/2
The Test Process
Project life
cycle
Initiation
Definition
Solution
Construction
Transition
Termination
Test process
Test
Planning
Test Preparation
Test Execution
Perform
testing
SRS study
Create
Test Plan
(by
Test leader)
Create
Test
Design
Review
&
Approva
l
Bug
Reportin
g
- Bug Fixing
- TC
Updating
- Develop
test cases,
test scripts
- Prepare
test data
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
Data Analysis
Test
Repor
t
- Analyze
root
Causes of
defects
- Identify
actions
Bug
Tracking
(Re-test)
19
04e-BM/NS/HDCV/FSOFT v2/2
Input/Output for testing
Input:
Customer requirements and Acceptance criteria
Change requests
Software Requirement Specification (SRS)
Design documents (ADD, DDD)
Programs (Modules)
Output:
Test documents: Test plan, Test cases and
procedures, Test script, Test data
Defect list
Test report (Pass/Fail of each TC, Summary report)
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
20
04e-BM/NS/HDCV/FSOFT v2/2
FSoft Test resources
Guideline: Test process & Test guideline
Templates for test documents
FSOFT tools
Test Plan
Test case specification
Test report
Defect analysis report
Defect tracking tool: DMS
Test Effort tracking tool: Timesheet
Test schedule: FSoft Insight
Test automation tools
Rational Robot (Functional & Performance test)
OpenSTA (Open source)
Witir (Open source)
FATA: Test Council
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
21
04e-BM/NS/HDCV/FSOFT v2/2
Test strategy
Objectives
Understand Unit Testing concept
Understand Integration Testing concept
Understand System Testing concept
Understand Acceptance Testing concept
Overview of Test types
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
22
04e-BM/NS/HDCV/FSOFT v2/2
V-Model
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
23
04e-BM/NS/HDCV/FSOFT v2/2
Testing stages
There are 4 stages of Testing:
Unit Test
Integration Test
System Test
Acceptance Test
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
24
04e-BM/NS/HDCV/FSOFT v2/2
Unit Test
Purpose: to verify that the
component/module functions properly
Check:
internal data structures
Logic
boundary conditions for input/output data
Normally: White box oriented
Doer: Development team
Metrics: Test coverage:
Statement
Branch
Path
Condition
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
25
04e-BM/NS/HDCV/FSOFT v2/2
Integration Test
Purpose:
To ensure that code is implemented and
designed properly
to take unit tested modules and build a
program structure that has been dictated by
design
IT only after UT
Combining the individual components to
uncover errors associated with interfacing
Normally: Black box oriented
Doer: independent Test team
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
26
04e-BM/NS/HDCV/FSOFT v2/2
System Test
Purpose: To ensure that the system does what
the requirement specifies
Types
Function Testing
Performance Testing
Installation Testing
Normally: Black box oriented
Doer: independent Test team
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
27
04e-BM/NS/HDCV/FSOFT v2/2
Acceptance Test
Customers way to verify that what was wanted
is what is built
Uncovers more than requirements discrepancies
Allows the customers to determine what they
really want, whether specified in the document
or not.
New problems may arise
Customers may decide that the problem as
changed and a different solution is needed
Normally: Black box oriented
Doer: Customer/independent Test team
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
28
04e-BM/NS/HDCV/FSOFT v2/2
Testing types
Functional test
User Interface: Layout
Performance test
Function testing
User Interface testing
Data and database integrity testing
Business cycle testing
Performance profiling
Load testing
Stress testing
Volume testing
Access control & Security test
Regression test
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
29
04e-BM/NS/HDCV/FSOFT v2/2
Test techniques/methods
White Box Testing
It guarantees that all independent paths within a
module have been exercised at least once
Exercise all logical decisions on their true an false sides
Execute all loops their boundaries and within their
operational bounds
Exercise internal data structures to assure their validity
Black Box Testing
Focuses on functional requirement of the software
Derive sets of input conditions that will fully exercise all
functional requirements for a program
Not an alternative to white box
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
30
04e-BM/NS/HDCV/FSOFT v2/2
Test Plan, Test Design
& Test Cases
Objectives
Understand the principles of test plan & test design
Understand the concepts for test case design
Understand the structure of test cases
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
31
04e-BM/NS/HDCV/FSOFT v2/2
Test Planning: Objectives
Test planning is to:
Define Test strategy:
What to test: list of requirements to test
How to test (Test techniques)
Test types
Test tool, .
Define the responsibilities
Decide test metrics & manage through metrics
When to Stop Testing?
Not good Unit Test
Low quality of code,
Meet requested Test coverage
All Test cases are tested and passed
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
32
04e-BM/NS/HDCV/FSOFT v2/2
Test Plan
What to do
Who does it: Test team, Development team
How to do it: Techniques
When to do it
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
33
04e-BM/NS/HDCV/FSOFT v2/2
Test Plan
Test Plan to define:
Scope of test: stages and types of test
Risks may affect testing: quantify the loss of the risk
Training needs for testers
Requirements to test: what will be tested
Testing strategy: how will the test be performed
Completion criteria (Pass/Fail)
Criteria to stop testing
Tools to be used for testing
Testing resource and environment (Responsibilities of
testers)
Test Milestone: schedule of testing activities
Test deliverables: outputs of testing
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
34
04e-BM/NS/HDCV/FSOFT v2/2
When to Stop Testing?
Not good UT: Defects of UT do not meet Defect plan
Low quality of code: too many defects (over the
number identified in planning phase/kick-off meeting)
Does not meet requested Test coverage (by
customer/PM).
It must be identified at planning phase.
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
35
04e-BM/NS/HDCV/FSOFT v2/2
Test design structure
Specifications
Basic flow
Alternative flow
Test Design
All test cases should be traceable to requirements
Design framework and rules for test cases (large/average items)
Plan number of test cases for normal, abnormal & boundary
data
Assign testers for module/function/large items
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
36
04e-BM/NS/HDCV/FSOFT v2/2
Test Design
Structure/Framework for Test cases (TC):
Modules or functions
Large items
GUI
Functions
Performance
Number of TC:
Total
For each module
For each type: GUI/Function
For normal/abnormal/boundary cases
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
37
04e-BM/NS/HDCV/FSOFT v2/2
Test Case
Test environment
Test condition
A test case includes:
Action (sample: click on Add button, on a link, input
birth day)
Input data (valid/invalid/boundary data)
Expected result (message, )
Good test case:
Structure of TC is clear and reasonable.
Follow requirement closely.
Cover all of cases that can occur.
Description and test conditions is brief and easy to
understand
Procedure and expected result is a step by step process
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
38
04e-BM/NS/HDCV/FSOFT v2/2
Testing Measurement
Objectives
Understand the need for metrics and measurement
in Testing
Understand the minimal metrics required for the
measurement of the testing activity
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
39
04e-BM/NS/HDCV/FSOFT v2/2
Examples of testing measurement?
Product: Weve tested 80% of the lines of code
Plan: Weve run 80% of the test cases
Results: Weve discovered 593 defects
Effort: Weve worked 80 hours a week on this for 4
months. Weve run 7,243 tests
Quality of Testing: Beta testers have found 30 defects
that we missed. Our regression tests seems ineffective
History across projects: At this milestone on previous
projects, we had fewer than 12% of the defects found
still open. We should be at that percentage on this
product too
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
40
04e-BM/NS/HDCV/FSOFT v2/2
How do we measure Testing in FSOFT?
Measurements:
Defects in DMS
Test effort in Timesheet
Defect rate in Fsoft Insight: Weighted defects/ project size (in
UCP)
Test coverage: number of executed test cases/ Total number
of test cases
Test successful coverage: number of test cases executed
successfully/total number of test cases
Defect removal efficiency
Metrics to assess tester performance:
Test effectiveness
Weighted defects/ Test execution effort
Leakage
Weighted defects found after release/ project size
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
41
04e-BM/NS/HDCV/FSOFT v2/2
Q&A
Copyright
2006 FPT
Software
FPT SOFTWARE
TRAINING
MATERIAL
Internal use
42
04e-BM/NS/HDCV/FSOFT v2/2
