Exploratory Testing
Mattias Göransson
Acknowledgements
James Bach, Satisfice, Inc.
Exploratory testing – statistics
There is no reliable study showing exactly
how efficient Exploratory Testing is…
Exploratory testing – facts of life
…so I will give you an example from a
large project where I acted as Test
Manager.
Exploratory testing – a real project (1/2)
Requirements in the form of Use Cases
All flows through them gave one positive + three
negative Test Cases.
Walkthroughs with the owners of these Use Cases.
For each Iteration the testers spent app. 90 days on
design, documentation and execution of these Test
Cases.
This resultet in app. 600 Defect reports
Exploratory testing – a real project (2/2)
After we had done all the work with the scripted
testing I was allowed to spend app. 5 days of each
testers time on Exploratory Testing.
This resulted in app. another 200 Defect reports.
Exploratory testing
Exploratory testing is simultaneous
learning, test design, and test execution.
Contrasting approaches
Scripted testing
• Tests are first designed and recorded.
• Then they may be executed at some later time or by a
different tester.
• Done well: 70% of the Testers time is spent on design &
documentation and 30% on Executing Tests
Exploratory testing
• Tests are designed and executed at the same time, and they
often are not recorded.
• Done well: 80-90 % of the Testers time is spent on Executing
Tests and 10-20% is spent on Documentation.
What is exploratory testing good for
Exploratory testing is useful when…
• it is not obvious what the next test should be
• we want to go beyond the obvious tests
• AND when we don’t know all every detail of every
requirement on the product under test.
When is Exploratory testing not to be used
Exploratory testing is not a method for
testing safety critical applications where
you up front know all the requirements.
Severity matrix – A mix is often best
2 4
Frequency
1 3
Impact
The core practice of a skilled ET Tester
Excellent exploratory testers…
• challenge constraints and negotiate their mission.
• alert their clients to project issues that prevent good testing.
• spontaneously coordinate and collaborate, as needed.
• train their minds to be cautious, curious, and critical.
• know how to design questions and experiments.
• know the difference between observation and inference.
• have developed resources and tools to enhance performance.
• adapt their test strategy to fit the situation.
• tolerate substantial ambiguity, indecision, and time pressure.
• take notes and report results in a useful and compelling way.
• have earned the trust placed in them.
Getting the most out of exploratory testing
(1/5)
Exploit inconsistency.
• Let yourself be distracted by anomalies and new ideas.
• Avoid repeating the exact same test twice.
Exploit the human factor.
• Encourage variability among testers.
• Exploit subject matter expertise.
• Use your confusion as a resource.
• Work in pairs or groups, whenever possible.
• Get to know your developers.
• Test in short bursts.
Getting the most out of exploratory testing
(2/5)
Learn the logic of testing
• correlation and causality
• design of experiments
• forward, backward, and lateral thinking
• biases, heuristics, and human error
• risk, benefit, and the meaning of “good enough”
Practice critical reading and interviewing
• Analyzing natural language specifications
• Analyzing and cross-examining a developer’s explanation.
Getting the most out of exploratory testing
(3/5)
Adopt a clear and consistent testing vocabulary
• bug, specification, risk, test strategy, test case, test plan
Learn to model a product rapidly
• flowcharting; data flows; state model
• matrices and outlines
• function/data square
• study the technology
Use a “grid search” strategy to control coverage
• Model the product in some way, then specify broad test areas
in terms of that model (not specific test cases). Keep track of
what areas you have and have not tested in.
Getting the most out of exploratory testing
(4/5)
Learn to take reviewable notes
• Take concise notes so that they don’t interrupt your work.
• Record at least what your strategy was, what you tested, what
problems you found, and what issues you have.
Practice responding to scrutiny
• Why did you test that?
• What was your strategy?
• How do you know your strategy was worthwhile?
Learn to spot obstacles to good testing
• Not enough information, yet.
• The product isn’t testable enough.
• Don’t have enough of the right test data.
Getting the most out of exploratory testing
(5/5)
Develop and use testing heuristics
• Try the heuristic test strategy model.
• Practice critiquing test techniques and rules of thumb.
• Notice the test ideas you use that lead to interesting results.
If you’re in a highly structured environment, consider
using session-based test management to measure
and control exploratory testing.
• Packages ET into chunks that can be tracked and measured.
• Protects the intuitive process; gives bosses what they want.
Exploratory testing
Exploratory testing is simultaneous
learning, test design, and test execution.
