Books
• Amazon returns 400+ hits for “software
CO531: Software Engineering Practice: testing” in the title
Testing • Recommended: How to Break Software by
James A. Whittaker, Addison Wesley, 2003.
or some of the things you should Price: £25.19 (from Amazon)
know about testing but never knew • Google returns about 175,000,000 hits to a
you had to ask! search on “software testing”
• Truth vs Anecdotes?
Testing Simple Example
• Why do we do it? • CO522 Example class
– Because we ain’t perfect • p(x) = 4096x6 – 36864x5 + 129280x4 –
• When do we do it? 222720x3 + 194224x2 – 78096x + 10395
– All the time
• Did anyone test their calculation method?
• How do we do it?
• How did/do you test this?
– Systematically and repeatedly
• When do we stop doing it? • Balance?
– When the software isn’t used any longer
1
Exhaustive testing Exhaustive testing ….
• Exhaustively test a method that takes two • … is generally not an option.
integer parameters. Assume
– 32-bit integers
– runs a test every 50 nanoseconds (10-9 secs)
• How long will this take?
General Introduction Dijkstra says…
• More next term • Program testing can be used to show the presence
of bugs, but never to show their absence!
• What is the point of testing? • You must not give the world what it asks for, but
– XP is very hot on testing what it needs.
• What does testing actually tell you? • We must not put mistakes into programs because
of sloppyness, we have to do it systematically and
• What can’t testing tell you? with care.
• Confidence building … but don’t forget • Object-oriented programming is an exceptionally
– that “Oh fcuk” moment bad idea which could only have originated in
California.
2
Errors --Nomenclature
• looking for bugs, errors, defects, faults,
failures, features, showstoppers …
• lost marks, wrong answers, dead people,
crashed aircraft, no passports, blue screens,
any of the above (Microsoft)
• Debugging – fixing it – safely!
• Removal cost increases with time
Why/How Do Errors/Bugs Occur Types of Testing
• Designed/written/tested/used by • Just talking about testing code
fallible/malicious people. • Unit testing vs System testing
• You made a silly error – a simple oversight • White/Glass box, grey box, black box
• You’re being pushed to complete a project • Iterative testing vs Big bang testing
• “Unforeseeable” circumstances occur
• Difficult/impossible to test some
circumstances for real
3
The Test First (TDD) Scheme
Story Why Test-First
Understand
Never write a line of • Feedback: instant on code changes
Add one failing test
functional code without • Task orientation: focuses on problem
a broken test. Kent Beck decomposition, maintains focus, provides
Add production
code for that test steady and measurable progress
• Quality: built-in and maintained via testing
Run ALL tests
• Low-level design: context for decisions
Story Next (classes/methods, names, interfaces, …)
Rework OK
Complete Story
Advocates and Sceptics What else is it?
• Advocates: increased productivity and • Test first coding is not a testing technique
better quality Ward Cunningham (being provocative!)
• Sceptics: Hard to learn, counter-productive • Analysis technique
• Quality: Interleaving tests with coding • Design technique
• Productivity: Not so obvious
• and, I believe, it’s also a testing technique!
• Studies: inconclusive
• Other factors: team dynamics, self-esteem,
courage
4
Unit Testing Unit Testing contd
• Usually tests just one particular method • Answers the question
– Is the code doing exactly what I think it should?
• Need confidence that low-level routines work • Note that this isn’t the question
before adding high level code – Does the code do what the requirements demand?
• Not concerned with formal validation of code • You need to ensure your code does what you want ALL
THE TIME
– Still need other forms of testing – Regression tests – all tests must pass NOT JUST THE NEW
• Wish to avoid the ripple effect ONES
• Unit testing helps with documentation
• Good unit testing – Tests are visible
– Better code design – Test are up-to-date and correct (because you keep running them!)
– Less time in the debugger – Code documentation (on the other hand) tends to drift
Unit Testing JUnit
• Testing isn’t free but ... • Nice to have help with unit tests
– Invest up-front to save time later
• BlueJ provides a neat interface to the JUnit
• Typical excuses testing framework.
– It takes too long to write tests
– Don’t have to write drivers
– It takes too long to run the tests
– It’s not my job to test my code – You have to know the answer to each test
– I don’t know what the code is supposed to do so I can’t – You can build the tests up
test it – You can run all the tests easily (and batches if
– I feel guilty putting testers and QA staff out of work you want to do a bit more work.
– I’m not allowed to run unit tests on a live system
5
Unit Tests vs Function Tests Generating Test Cases
• Unit tests are local • Problem dependent -- experience
– Include integration tests
• Some general rules
– Written by (and for) programmers
– equivalence classes – one example covers many
• Function testing = higher order testing
– Does code do what the user expects?
– edge cases – what happens at the limits (< | <=)
– XP has continual customer involvement – odd cases
– Iterative function testing = minimal surprises • Illegal cases
– Done by end user (as well as programmers) – comment (assume programmers read them)
– Communication
– trap (defensive programming)
• Others: stress, performance, security, installability
On Finding an Error Debugging
• Same for all reported errors • Methodical process of finding and reducing the number of
defects in a system.
– Communicate • Use tools – you’re not a sissy because you use a debugger
– Reproduce (the error not literally!) least – You are a plonker if you don’t
code/data = failing test • Debuggers are often quite sophisticated tools
– Debug (why, where) – Steep learning curve, but …
– … they do pay back time invested
– Repair (add new test, fix code, run regression) • Other tools might be available
– Learn (discover why and record) – Tools – Compiler flags to do additional checks
– Specific checks e.g., memory management
• Maintenance phase (more?)
6
Basic Guide to Debugging Basic Guide (contd.)
1. Recognise that there is a defect in your code 3. Why has the problem occurred?
• Check input data (program should already do this?) • Incorrect code may be a long way from the problem’s
appearance (code distance/time)
• Reproducibility
• E.g., incorrect value generated but problem not
• Identify the symptoms detected until it is used much later
2. Isolate the source of the problem • Are there similar sections of code elsewhere?
• What part of the system is causing the problem 4. Determine the fix
• Check inputs and outputs over smaller parts of the • Sometimes simple – one line
system • Sometimes catastrophic – major rewrite
• Experience counts here • Quick fix may be necessary
Basic Guide (contd.) Useful things to do
5. Fix, TEST, TEST and TEST 1. Have the correct mind-set
• Your code probably is broken
• Apply the fix 2. Add assert/debug statements into code so they can be
• Check it does correct the problem (TEST) activated when necessary
3. Use log files
• Check nothing else breaks (TEST)
4. Only ever make one substantial source change at a time
• Add new test (which one???) to regression • Helps prevent adding new bugs
(and unit ??) tests 5. Back out of changes that don’t fix the problem
• Problem not where you think it is
• Modified code never called
• Use a source code control system (Subversion/CVS)
7
Do I Only Add Failing Tests Testing GUIs
• Never write a line of functional code • Is a form of function testing
without a broken test. Kent Beck. • XP approach is not as successful (Kent
• Doesn’t say I can’t add tests that don’t fail! Beck says so – so it must be true ☺)
• Add test if • It’s hard to do
– they show a failure (need to write code) • It’s hard to automate
– there is something special about the test • Screen shots are worth a thousand words.
• Document all tests
Whittaker’s Book
• Read his book for a really useful list of
things to try when you perform both unit
and function testing.
• Try some of these attacks out on your
project code.
