Testing Fundamentals
Testing Fundamentals
Contents
What is Quality ?
What Is Software Testing?
Why Testing?
Testing Lifecycle
Testing Principles
When to Start Testing
Quality Assurance Vs Testing
Complete Testing – Is it possible?
What is Quality ?
The totality of features and characteristics of a product or service that bear on its
ability to satisfy stated or implied needs - ISO 8402
Quality is Conformance to requirements - Crossby
Quality is Compliance to standard - W.E Perry
“The degree to which a system, component or process meets requirements”
or
“The degree to which a system, component or process meets customer or
user needs or expectations” - I.E.E.E.
Dimensions / Attributes of Software Quality
Portability
Efficiency Testability
Reliability Understandability
Usability Modifiability
(Boehm, 1978)
What and Why Testing?
What and Why Testing?
What Is Software Testing?
Testing is the process of executing a program with the intent of finding errors
Assume that the program contains errors and then test the program to find as many of
the error as possible
Testing is a destructive yet creative process which aims at establishing confidence
that a program does what it is supposed to do
Why Testing?
To Ensure Quality
Verifies that all requirements are implemented correctly (both for positive and
negative conditions)
Identifies defects before software deployment
Helps improve quality and reliability.
Makes software predictable in behavior.
Reduces incompatibility and interoperability issues.
Helps marketability and retention of customers.
Give programmers information they can use to prevent bugs and create useful
software
Are we playing with fire? Cost of Ineffective Testing
Time
Late Releases
Projects need to be reworked or abandoned
Money
Budget over-runs
Defects are 100 to 1000 times more costly to find and repair after
deployment
Quality
Developers unsure of product quality
Products released with undiscovered or unresolved defects
The Cost of Quality
Poor quality has costs. Many IT failures could be equated to
lost business. There is, however, a cost to ensuring quality
The chart below, displaying data from the Software Engineering Institute
at Carnegie Mellon University, underscores the importance of identifying
software errors prior to testing.
The graph above clearly indicates a quantum leap in software defect costs if those
defects are not identified until testing. If they are then not identified during testing,
the costs again increase substantially
The Cost of Quality ……..contd
Following are the two findings by Center for Empirically Based
Software Engineering Republished in January 2001, their analysis of
software defect reduction metrics includes these findings:
"Finding and fixing a software problem after delivery is often 100
times more expensive than finding and fixing it during the
requirements and design phase."
"Disciplined personal practices can reduce defect introduction rates
by up to 75 percent…When you couple this [Personal Software
Process] with the strongly compatible Team Software Process,
defect reduction rates can soar to factors of 10 or higher for an
organization that operates at a modest maturity level."
Testing Life Cycles
Testing Life Cycle
Testing Lifecycle
Requirements
Capture
Analysis The Scenarios Design and the Test Case
Development could normally start in parallel
with the Development Cycle.
Test planning & Test Execution Synchs up with the
Scenario Design Development Cycle during the functional
Testing Phases
New Version Test Case
Development
Defect Fixing
Cycle
Test Execution
Defects Test Cycle Closure
Test Result
Analysis
Development Process and Testing Process go on in parallel
Development Process
Development Testing
Business Detailed * High Level Detailed and Unit (Systems, SI &
Transition /
Analysis Requirements Design Design Testing UAT) Rollout
* Requirements could be defined along many dimensions e.g. : Functional Reqts, Usability Reqts, System Reqts, Performance Reqts, Quality Reqts, Technical Reqts, etc
Testing Process
Requirement Unit test Integration
capture planning testing
Testing IT results
Strategy
IT test planning Unit testing
review
Unit test results Functional Non-functional
Test analysis testing
System Test planning review testing
& design
Functional Non-functional
Test data Test bed setup
results review
results review
Defect tracking
The boxes in blue represent the tasks undertaken by Testing Team
Different stages of Testing Lifecycle
Test requirements identification
Validate for testability
Test Planning and Scenario design
Develop Test Objectives
Identify Test Items
Resources and Schedules
Test Case Development
Test Case Specification
Pre-requisites, post-requisites and Acceptance criteria
Different stages of Testing Lifecycle (Contd.)
Prepare Test Bed
Test Data Preparation
Test Environment Setup
Test Execution
Run Tests and Validate Results
Bug Reporting
Bug fixes and retesting
Test Result Analysis
Defect Analysis
Determining Test Set Coverage and Effectiveness
Testing principles
Testing principles
Testing Principles
Plan and test to find errors and NOT to prove that the program works fine.
Be creative and have an attitude to HACK / BREAK.
All tests should be traceable to the requirements
Exhaustive/complete testing is not feasible, so optimize tests based on priority and
criticality
Test Early and Iteratively as it evolves
“Divide and Conquer” – Begin “in the small” and progress towards testing “in the large”
Pareto principle holds true - The 80-20 rule
More the errors found already, it’s more likely to find more errors
Automate tests, wherever feasible
Create reusable test artifacts/scenarios
Fix only when a test cycle is completed
Testing is more efficient if done by an Unbiased third party
Testing Principles - continued
Testing must be planned – Testing Engineering
A good testing requires thinking about overall approach, designing test cases,
and establishing expected result for each case
The planning and care we expend on that case selection accounts for much
of the difference between good and poor testers
Examining a program to see if it does not do what it is supposed to do is only half of the
battle. The other half is seeing whether the program does what is not supposed to do
Test cases must be written not only for valid and expected, but also for invalid and
unexpected input conditions
Errors seem to come in clusters, and some sections seems to be more error-prone that
other sections.
E.g. a program with modules A and B. Five errors found in A and one in B. It is
likely that A still has more errors if A has not been purposely subjected to a more
rigorous test
Testing efforts may be focused against error-prone sections
Actors in Testing
Testers
• Role - Execute tests, record test scripts, maintain statistics and
metrics, check test data setup, test environment setup, execute re-tests
• Knowledge - Understanding the system being tested, awareness of
tools, how to progress against a plan
• Skills - Observation, accuracy, methodical, co-ordination, problem
solver, avoiding egos
• Experience - Following instructions, problem reporting and solving,
and relevant testing tools.
Developers
Users
Test Analyst
Test Manager
Test Artifacts
Test Plan : A formal test plan is a document that provides and records important information
about a test project, for example: Resources, Schedule & Timeline, Test Milestones, Use
cases and/or Test cases
Test Environment/Bed: An environment containing the hardware, instrumentation,
simulators, software tools, and other support elements needed to conduct a test
Test Case: A set of test inputs, execution conditions and expected results.
Test Data: The actual (set of) values used in the test or that are necessary to execute the
test.
Test Tools: Winrunner, Loadruner, Test Director, WebLoad etc. are widely used tools.
Test Scripts: is used to test a particular functionality (business rule). It may consist of 1 or
more Test Cases.
Test Log: A chronological record of all relevant details about the execution of a test
Bug Reports: Contains a summary of the bug, its priority and other details regarding the
bug.
When to Start Testing?
When to Start Testing?
When to Start Testing
Quality Assurance Vs Testing
Quality Assurance vs Testing
Quality Assurance - Preventing Bugs
The earlier in the process a bug is discovered and corrected, the
cheaper the correction.
Testing Isn’t Everything
Non-testing Activities
Prototyping – incomplete implementation that mimics the behavior we
think the users need.
Requirement analysis – checking the requirements for logical self-
consistency, for testability and for feasibility. Users can’t be expected
to provide valid requirements because they are not trained to do so.
Formal analysis, possibly mathematical
Design – ill-considered design may destroy the best requirements
Formal inspection – a process without formal inspection is seriously
flawed and depends too much on testing.
Complete Testing – Is it possible?
Impossibility of Complete Testing
If test completely, then at the end of testing, there cannot be any
undiscovered errors - impossible.
Main Reasons
The domain of possible inputs is too large to test
There are too many paths through the program to test
The user interface (and thus design) issues are too complex to completely
test
Difficult to find all design errors
From Kaner’s lecture notes on Black Box Software Testing
Too Many Inputs
Test all valid inputs
Test all invalid inputs – check everything you can enter at the keyboard
Test all edited inputs if the program lets you edit /change numbers- make
sure editing works
Test all variations on input timing
Try testing one event just before, just after, and in the middle of processing a second
event. Will they interfere with each other?
Don’t wait to enter numbers until the computer has printed a question mark and started
flashing its cursor at you
Enter numbers when it’s trying to display others, when it is adding them up, when it is
printing a message, whenever it is busy
As soon as you skip any input value, you have abandoned completing
testing
Too Many Combinations
Variables Interact
E.g. a program fails when the sum of a series of variables is too large.
Suppose the number of choices for the N variables are V1, V2,
through VN. The total number of possible combinations is V1 x V2 x . . .
x VN.
39,601 combinations of just two variables whose values could range only
between -99 and 99.
A case that isn’t so trivial - 318,979,564,000 possible combinations of the first
four moves in chess.
Too Many Paths
A program path can be traced through the code from the start of the
program to the program termination.
Two paths differ if the program executes different statements in each or executes the
same statements but in a different order
You have not completely tested the program unless you have exercised every path
Example 1: From The Art of Software Testing by Myers
Program starts at A.
From A it can go to B or C
From B it goes to X
From C it can go to D or E
From D it can go to F or G
From F or from G it goes to X
From E it can go to H or I
From H or from I it goes to X
From X the program can go to EXIT or back to A. It can go back to A no more than
19 times.
Too Many Paths - cont
B
A F
D
G
C X EXIT
H
E < 20 times
through the
I
loop
One path is ABX-Exit. There are 5 ways to get to X and then to the EXIT in one pass.
Another path is ABXACDFX-Exit. There are 5 ways to get to X the first time, 5 more to get
back to X the second time, so there are 5 x 5 = 25 cases like this.
There are 51 + 52 + ... + 519 + 520 = 1014 = 100 trillion paths through the program to test or
approximately one billion years to try every path (if one could write, execute and verify a
test case every five minutes ).
Difficult to Find Every Design Error
Specifications often contain errors
Accidents, e.g. 2+2=5
Deliberate – the designer thought he had a better idea, but didn’t
implement it
Many user interface failings are design errors – being in the specification
does not make them right
In summary – it is hard to find all errors
Q&A
Thank You
References
http://www.geocities.com/xtremetesting/
http://www.sqatester.com/
http://www.ontariofamilyhealthnetwork.gov.on.ca/
http://www.Kaner.com
http://www.testingFaqs.com
