Software Quality Assurance
Pavithra Subashini
Senior lecturer
Faculty of Computing
Psychology of testing
Why test?
• build confidence
• prove that the software is correct
• demonstrate conformance to requirements
• find faults
• reduce costs
• show system meets user needs
• assess the software quality
Confidence
Confidence
Fault
Faultsfound
found
Time
No faults found = confidence?
A traditional testing approach
Show that the system:
– does what it should
– doesn't do what it shouldn't
Goal: show working
Success: system works
Fastest achievement: easy test cases
Result: faults left in
A better testing approach
Show that the system:
– does what it shouldn't
– doesn't do what it should
Goal: find faults
Success: system fails
Fastest achievement: difficult test cases
Result: fewer faults left in
The testing paradox
Purpose of testing: to find faults
Finding faults destroys confidence
Purpose of testing: build confidence
The best way to build confidence
is to try to destroy it
Who wants to be a tester?
• A critical process
• Bring bad news (“your baby is ugly”)
• Under worst time pressure (at the end)
• Need to take a different view, a different
mindset (“What if it isn’t?”, “What could go
wrong?”)
• How should fault information be
communicated (to authors and managers?)
Tester’s have the right to:
– accurate information about progress and changes
– insight from developers about areas of the software
– delivered code tested to an agreed standard
– be regarded as a professional (no abuse!)
– find faults!
– challenge specifications and test plans
– have reported faults taken seriously (non-reproducible)
– make predictions about future fault levels
– improve your own testing process
Testers have responsibility to:
– follow the test plans, scripts etc. as documented
– report faults objectively and factually (no abuse!)
– check tests are correct before reporting s/w faults
– remember it is the software, not the programmer,
that you are testing
– assess risk objectively
– prioritise what you report
– communicate the truth
Independence
Test your own work?
– find 30% - 50% of your own faults
– same assumptions and thought processes
– see what you meant or want to see, not what is
there
– emotional attachment
• don’t want to find faults
• actively want NOT to find faults
Levels of independence
• None: tests designed by the person who wrote
the software
• Tests designed by a different person
• Tests designed by someone from a different
department or team (e.g. test team)
• Tests designed by someone from a different
organisation (e.g. agency)
• Tests generated by a tool (low quality tests?)
Fundamental test process
Test Planning - different levels
Test
Policy
Company level
Test
Strategy
High
HighLevel
Level Project level (IEEE 829)
Test Plan (one for each project)
Test Plan
Detailed Test stage level (IEEE 829)
Detailed
Detailed
Test Plan (one for each stage within a project,
Detailed
Test Plan
Test Plan e.g. Component, System, etc.)
Test Plan
The test process
Planning (detailed level)
check
specification execution recording
completion
Test planning
how the test strategy and project test plan
apply to the software under test
document any exceptions to the test strategy
– e.g. only one test case design technique needed
for this functional area because it is less critical
other software needed for the tests, such as
stubs and drivers, and environment details
set test completion criteria
Test specification
Planning (detailed level)
check
specification execution recording
completion
Identify conditions
Design test cases
Build tests
A good test case
• effective Finds faults
• exemplary
Represents others
• evolvable
Easy to maintain
• economic
Cheap to use
Test specification
test specification can be broken down into three
distinct tasks:
1. identify: determine ‘what’ is to be tested (identify
test conditions) and prioritise
2. design: determine ‘how’ the ‘what’ is to be
tested
(i.e. design test cases)
3. build: implement the tests (data, scripts, etc.)
Task 1: identify conditions
(determine ‘what’ is to be tested and prioritise)
list the conditions that we would like to test:
– use the test design techniques specified in the test plan
– there may be many conditions for each system function or
attribute
– e.g.
• “life assurance for a winter sportsman”
• “number items ordered > 99”
• “date = 29-Feb-2004”
prioritise the test conditions
– must ensure most important conditions are covered
Task 2: design test cases
(determine ‘how’ the ‘what’ is to be tested)
design test input and test data
– each test exercises one or more test conditions
determine expected results
– predict the outcome of each test case, what is
output, what is changed and what is not changed
design sets of tests
– different test sets for different objectives such as
regression, building confidence, and finding faults
Task 3: build test cases
(implement the test cases)
prepare test scripts
– less system knowledge tester has the more
detailed the scripts will have to be
– scripts for tools have to specify every detail
prepare test data
– data that must exist in files and databases at the
start of the tests
prepare expected results
– should be defined before the test is executed
Test execution
Planning (detailed level)
check
specification execution recording
completion
Execution
Execute prescribed test cases
– most important ones first
– would not execute all test cases if
• testing only fault fixes
• too many faults found by early test cases
• time pressure
– can be performed manually or automated
Test recording
Planning (detailed level)
check
specification execution recording
completion
Test recording 1
The test record contains:
– identities and versions (unambiguously) of
• software under test
• test specifications
Follow the plan
– mark off progress on test script
– document actual outcomes from the test
– capture any other ideas you have for new test cases
– note that these records are used to establish that all
test activities have been carried out as specified
Test recording 2
Compare actual outcome with expected outcome. Log
discrepancies accordingly:
– software fault
– test fault (e.g. expected results wrong)
– environment or version fault
– test run incorrectly
Log coverage levels achieved (for measures specified
as test completion criteria)
After the fault has been fixed, repeat the required
test activities (execute, design, plan)
Check test completion
Planning (detailed level)
check
specification execution recording
completion
Check test completion
• Test completion criteria were specified in the
test plan
• If not met, need to repeat test activities, e.g.
test specification to design more tests
Coverage too low
check
specification execution recording
completion
Coverage
OK
Test completion criteria
Completion or exit criteria apply to all levels of
testing - to determine when to stop
– coverage, using a measurement technique, e.g.
• branch coverage for unit testing
• user requirements
• most frequently used transactions
– faults found (e.g. versus expected)
– cost or time
Comparison of tasks Governs the
quality of tests
Planning Intellectual
one-off
Specification activity Good to
activity automate
Execute repeated
many times
Recording Clerical
