Introduction to

Software Testing
 What is software Testing?
Define
i. software testing
ii. Computer bug
iii. Computer fault
 Bugs a.k.a. …
• Defect • Failure
• Fault • Inconsistency
• Problem • Product
• Error Anomaly
• Incident • Product
• Anomaly Incidence
• Variance • Feature :-)
 Defective Software
• We develop programs that contain
defects
– How many? What kind?
• Hard to predict the future, however…
it is highly likely, that the software we
(including you!) will develop in the
future will not be significantly better.
 Sources of Problems
• Requirements Definition: Erroneous, incomplete,
inconsistent requirements.
• Design: Fundamental design flaws in the software.
• Implementation: Mistakes in chip fabrication, wiring,
programming faults, malicious code.
• Support Systems: Poor programming languages,
faulty compilers and debuggers, misleading
development tools.
Sources of Problems (Cont’d)
• Inadequate Testing of Software:
Incomplete testing, poor verification,
mistakes in debugging.
• Evolution: Sloppy redevelopment or
maintenance, introduction of new flaws
in attempts to fix old flaws, incremental
escalation to inordinate complexity.
 Adverse Effects of
Faulty Software
• Communications: Loss or corruption of
communication media, non delivery of
data.
• Space Applications: Lost lives, launch
delays.
• Defense and Warfare: Misidentification
of friend or foe.
 Adverse Effects of Faulty
Software (Cont’d)
• Transportation: Deaths, delays,
sudden acceleration, inability to brake.
• Safety-critical Applications: Death,
injuries.
• Electric Power: Death, injuries, power
outages, long-term health hazards
(radiation).
 Adverse Effects of Faulty
Software (Cont’d)
• Money Management: Fraud, violation of
privacy, shutdown of stock exchanges and
banks, negative interest rates.
• Control of Elections: Wrong results
(intentional or non-intentional).
• Control of Jails: Technology-aided escape
attempts and successes, accidental release
of inmates, failures in software controlled
locks.
• Law Enforcement: False arrests and
imprisonments.
 Bug in Space Code
• Project Mercury’s FORTRAN code had the
following fault:
DO I=1.10 instead of ... DO I=1,10
• The fault was discovered in an analysis of
why the software did not seem to generate
results that were sufficiently accurate.
• The erroneous 1.10 would cause the loop to
be executed exactly once!
 Military Aviation Problems
• An F-18 crashed because of a missing
exception condition:
if ... then ... without the else clause
that was thought could not possibly
arise.
• In simulation, an F-16 program bug
caused the virtual plane to flip over
whenever it crossed the equator, as a
result of a missing minus sign to
indicate south latitude.
 Year Ambiguities
• In 1992, Mary Bandar received an
invitation to attend a kindergarten in
Winona, Minnesota, along with others
born in '88.
• Mary was 104 years old at the time.
 Year Ambiguities (Cont’d)
• Mr. Blodgett’s auto insurance rate
tripled when he turned 101.
• He was the computer program’s first
driver over 100, and his age was
interpreted as 1.
• This is a double blunder because the
program’s definition of a teenager is
someone under 20!
 Dates, Times, and Integers
• The number 32,768 = 215 has caused
all sorts of grief from the overflowing of
16-bit words.
• A Washington D.C. hospital computer
system collapsed on September 19,
15
1989, 2 days after January 1, 1900,
forcing a lengthy period of manual
operation.
 Shaky Math
• In the US, five nuclear power plants
were shut down in 1979 because of a
program fault in a simulation program
used to design nuclear reactor to
withstand earthquakes.
• This program fault was, unfortunately,
discovered after the power plants were
built!
 Bank Generosity
• A Norwegian bank ATM consistently
dispersed 10 times the amount
required.
• Many people joyously joined the queues
as the word spread.
 Bank Generosity (Cont’d)
• A software flaw caused a UK bank to
duplicate every transfer payment
request for half an hour. The bank lost
2 billion British pounds!
• The bank eventually recovered the
funds but lost half a million pounds in
potential interest.
 Discussion …
• Have you heard of other software bugs?
– In the media?
– From personal experience?
• Does this embarrass you as a future
software engineer?
 Specification
“if you can’t say it, you can’t do it”
• You have to know what your product is before
you can say if it has a bug.
• A specification defines the product being
created and includes:
– Functional requirements that describes the
features the product will support. E.g., on a word
processor
• Save, print, check spelling, change font, …
– Non-functional requirements are constraints on
the product. E.g,
• Security, reliability, user friendliness, platform, …
A software bug occurs when at
least one of these rules is true
• The software does not do something that the
specification says it should do.
• The software does something that the
specification says it should not do.
• The software does something that the
specification does not mention.
• The software does not do something that the
product specification does not mention but
should.
• The software is difficult to understand, hard to
use, slow …
Most bugs are not because of
mistakes in the code …
• Specification (~= 55%)
• Design (~= 25%)
• Code (~= 15%)
• Other (~= 5%)
 Relative cost of bugs
“bugs found later cost more to fix”
• Cost to fix a bug increases exponentially (10x)
– i.e., it increases tenfold as time increases
• E.g., a bug found during specification costs $1 to
fix.
• … if found in design cost is $10
• … if found in code cost is $100
• … if found in released software cost is $1000
 Bug Free Software
• Software is in the news for the wrong reason
– Security breach, Mars Lander lost, hackers getting
credit card information, etc.
• Why can’t software engineers develop
software that just works?
– As software gets more features and supports more
platforms it becomes increasingly difficult to make
it create bug-free.
 Discussion …
• Do you think bug free software is
unattainable?
– Are their technical barriers that make this
impossible?
– Is it just a question of time before we can
do this?
– Are we missing technology or processes?
 Goal of a software tester
• … to find bugs
• … as early in the software development
processes as possible
• … and make sure they get fixed.

• Advice: Be careful not to get get caught

in the dangerous spiral of unattainable
perfection.
 You now know …
• … what is a bug
• … the relationship between specification and
bugs
• … the cost of a bug relative to when it is
found
• … the unattainable goal of perfect software
• … the goal of the software tester
• … valuable attributes of a software tester
 The Software
Development Process
 Software is …
• requirements specification documents
• design documents
• source code
• test suites and test plans
• interfaces to hardware and software
operating environment
• internal and external documentation
• executable programs and their persistent
data
 Software effort
• Specification
• Product reviews
• Design
• Scheduling
• Feedback
• Competitive information acquisition
• Test planning
• Customer surveying
• Usability data gathering
• Look and feel specification
• Software architecture
• Programming
• …
 Discussion …
• What is software engineering?
• Where does testing occur in the
software development process?
 Customer requirements
• The software development team must
determine what the customer wants.
• How can you do this?
– Guess?
– Collect detailed information from surveys?
– Get feedback from a previous version of the
software?
– Read reviews in magazines?
– Get information about the competition?
– Other ways?
• The collected data is used to guide the
specification effort.
 Specification
“If you don't know where you're going any road will take you there”

• The specification takes the data from the

customer requirements and other sources
and defines:
– The features of the software (functional
requirements).
– The constraints on these features (non-functional
requirements).
• Specifications can be:
– formal (e.g., aerospace industry), rigid
– informal (e.g., a .com start up), on a cocktail
napkin or a whiteboard.
 Schedules
• The goals of scheduling are to know:
– What work needs to be completed?
– How much work is left to do?
– When will the work be finished?
– Who will finish each task?
– Other measurable queries.
• A Gantt chart is a popular type of bar chart
that illustrates a project schedule.
 Design
• Before coding begins on non-trivial software
projects, a set of design documents are
created to serve as blueprints.
– Software Architecture
– Data flow diagram
– State transition diagram
– Flowchart
– Commented code
 Source code … of course
• The ultimate specification of the
software!
• ‘Code is king’ philosophy is
still prevalent.
• Many programming
languages and tools to
choose from.
 Test documents
• Test plan
– Quality objectives, resource needs, schedules,
assignments, methods, etc.
• Test cases
– Inputs and expected outputs.
• Bug reports
– E.g., the Bugzilla web-based bug tracker.
• Test tools and automation
• Metrics, statistics, and summaries
– Number of unresolved bugs, mean time to repair a
bug, etc.
 Software Project Staff
• Project managers
– Write product specification, manage the schedule, critical
decision tradeoffs
• Software architects, system engineers
– Design the software, work closely with programmers
• Programmers, developers, coders
– Write code, fix bugs
• Testers, quality assurance staff
– Find bugs, document bugs, track progress of open bugs
• Technical writers
– Write manuals, on line documentation
• Configuration managers, builders
– Packaging and code, documents, and specifications
Software Development Lifecycles
• Waterfall
• Spiral
 You now know …
• … what is software
• … what is software engineering
• … what is the composition of a software
development organization
• … what are the major phases of a software
development project
• … how major phased are organized