Software Testing

Vũ Thị Hồng Nhạn

(vthnhan@vnu.edu.vn)

Dept. Software Engineering, FIT, UET

Vietnam National Univ., Hanoi

 Contents
 Development testing
 Test-driven development
 Release testing
 User testing

5/28/2025 Software testing Page 2

 Program testing
 Testing is intended to demonstrate that a program performs as expected and to
identify defects before the software is deployed

 When testing software, you execute the program using artificial (or simulated)
data

 You then examine the test results for errors, anomalies, or information related to
the program’s non-functional attributes

 Testing is part of a broader verification and validation process, which also includes
static validation technique

5/28/2025 Software testing Page 3

 Program testing goals
 To demonstrate that software meets its requirements

 For customer-specific software, this means there should be at least one test for
every requirement outlined in the requirement document

 For generic software products, this implies that tests should cover all system
features, as well as combination of these features, that will be included in the
product release

 To discover situations where the software behaves incorrectly, undesirably, or

does not conform to its specification

 Defect testing focuses o identifying undesirable system behaviors such as crashes,

unintended interactions with other systems, incorrect computations, and data
corruption
5/28/2025 Software testing Page 4
 Validation testing & defect testing
 The first goal is validation testing
 You expect the system to perform correctly using a given set of test cases that
reflects its expected use

 The second goal is defect testing

 in this case, the test cases are designed to expose defects.

 They can be deliberately obscure and don’t need to reflect how the system is
normally used

5/28/2025 Software testing Page 5

 An input-output model for program testing

5/28/2025 Software testing Page 6

 Verification vs. validation
 Verification: “are we building the product right?”
 The software should conform to its specification
 Validation: “are we building the right product?”
 The software should do what the user really needs

5/28/2025 Software testing Page 7

 Inspections and testing
 Software inspection involves analyzing the static representation of a system to

identify problems (static verification)

 It may be supplemented by tool-based analysis of documents and code

 Software testing, on the other hand, focuses on exercising the software and

observing its behavior during execution (dynamic verification)

 The system is run with test data, and its operational behavior is observed

5/28/2025 Software testing Page 8

 Inspections & testing...

5/28/2025 Software testing Page 9

 Software inspections
 These involves people examining the source representation with the aim of

discovering anomalies and defects

 Inspections don’t require the execution of the system and can therefore be

performed before implementation

 They may be applied to any representation of the system, such as requirements,

design, configuration data, test data, etc.

 Inspections have been shown to be an effective technique for discovering program

errors

5/28/2025 Software testing Page 10

 Advantages of inspections
 During testing, some errors may mask others, since inspection is a static process,
you don’t need to worry about interactions between errors

 Incomplete versions of a system can be inspected without incurring additional

costs
 However, if a program is incomplete you need to develop specialized test harness to
test the available components

 In addition to identifying program defects, inspections can also evaluate broader

quality attributes such as compliance with standards, portability, and
maintainability

5/28/2025 Software testing Page 11

 Inspections & testing...
 Are complementary, not opposing

 Both should be used throughout Verification and Validation (V&V) process

 Inspections can verify conformance to specifications but not necessarily to the

customer’s actual requirements

 Additional, inspections cannot assess non-functional characteristics such as

performance, usability, and so on

5/28/2025 Software testing Page 12

 A model of the software process

5/28/2025 Software testing Page 13

 Stages of testing
 Development testing – the system is tested during development to discover bugs

and defects

 Release testing - a separate testing team tests a complete version of the system

before it is released to users

 User testing – users or potential users test the system in their own environment

5/28/2025 Software testing Page 14

Development testing
 Development testing
 Development testing include all testing activities that are carried out by the team
developing the system. It typically involves the following levels:
 Unit testing
 this involves testing individual program units or object classes
 Unit testing should focus on verifying the functionality of individual objects and
methods
 Component testing
 Multiple individual units are integrated to form composite components
 Component testing should focus on validating the interfaces and interactions between
the integrated units within a component
 System testing
 This level involves integrating some or all components in the system and testing the
system as a whole
 System testing should focus on verifying the interactions between components and
ensuring the entire system behaves as expected

5/28/2025 Software testing Page 16

 Unit testing

 Is the process of testing individual components in isolation

 Units may include

 Individual methods within an object

 Object classes with several attributes and methods

 Composite components with defined interfaces to access their functionality

5/28/2025 Software testing Page 17

 Object class testing
 Complete test coverage of a class involves
 Testing all operations associated with the object

 Setting and querying all object attributes

 Exercising the object in all possible states

 However, inheritance makes it more difficult to design object class tests,

as the information to be tested is not localized

5/28/2025 Software testing Page 18

 The weather station object interface

5/28/2025 Software testing Page 19

 Weather station testing
 It’s necessary to define test cases for reportWeather, calibrateTest, and
shutdown

 Using a state model, identify sequences of state transitions to be tested,

along with the event sequences that trigger these transitions

 For example
 Shutdown  Running  Shutdown

 Configuring  Running  Testing  Transmitting  Running

 Running  Collecting > Running  Summarizing  Transmitting 

Running

5/28/2025 Software testing Page 20

 Automated testing
 Whenever possible, unit testing should be automated so that tests can be

executed and verified without manual intervention

 In automated unit testing, a test automation framework (e.g. Junit) is used to

write and run tests for your program

 Unit testing frameworks provide generic test classes that you can executed to

create specific test cases

 These frameworks can then run all the implemented tests and generate reports,

often through a graphical user interface (GUI)

5/28/2025 Software testing Page 21

 Automated test components
 A setup part, where you initialize the system with the test case – namely, the

inputs and expected outputs

 A call part, where you invoke the object or method to be tested

 An assertion part, where you compare the result of the call with the expected

result

 If the assertion evaluates to true, the test is considered successful; otherwise it has

failed

5/28/2025 Software testing Page 22

 Choosing unit test cases
 The test cases should demonstrate that, when used as expected, the component
being tested performs its intended functions correctly

 If there are any defects in the component, these should be revealed through the
test cases

 This leads to 2 types of unit test cases

 The first type reflects the normal operation of the program and should confirm that the
component behaves as expected

 The second type is based on testing experience and focuses on areas where common
problems tend to occur

 These test cases use abnormal or unexpected inputs to verify that the component
handles them appropriately and does not crash

5/28/2025 Software testing Page 23

 Testing strategies
 Partition testing involves identifying groups of inputs that share common
characteristics and should be processed in the same way
 You should select test cases from within each of these groups

 Guideline-based testing involves using established testing guidelines to

choose test cases
 These guidelines are based on previous experience and reflect common types
of errors that programmers often make when developing components

5/28/2025 Software testing Page 24

 Partition testing
 Input data and output results often fall into different classes, where all members
of a class are related

 Each of these classes is called an equivalence partition or domain, where the

program behaves in a similar way for each member

 Test cases should be selected from each partition

5/28/2025 Software testing Page 25

 Equivalence partitioning

5/28/2025 Software testing Page 26

 Equivalence partitioning

5/28/2025 Software testing Page 27

 Testing guidelines (sequences)
 Test software with sequences that contain only a single value

 Use sequences of different sizes in different tests

 Design tests so that the first, middle, and last elements of the sequence are

accessed

 Test with sequences of zero length

5/28/2025 Software testing Page 28

 General testing guidelines
 Choose inputs that force the system to generate all error messages

 Design inputs that cause input buffers to overflow

 Repeat the same input or series of input multiple times

 Force the system to generate invalid outputs

 Cause computation results to be excessively large or extremely small

5/28/2025 Software testing Page 29

 Component testing
 Software components are often composite components made up of several
interacting objects
 E.g., in the weather station, the reconfiguration component includes objects that handle
various aspects of the reconfiguration

 You access the functionality of these objects through a defined component

interface

 Therefore, testing composite components should focus on verifying that

component interface behaves according to its specification

 It can be assumed that unit tests on the individual objects within the component
have already been completed

5/28/2025 Software testing Page 30

 Interface testing
 objectives: to detect faults or invalid assumptions about interface

 Types of interfaces

 Parameter interfaces: data is passed from one method to another

 Shared memory interfaces: a black of memory is shared between procedures

 Procedural interfaces: a subsystem encapsulates a set of procedures that can be called

by other subsystems

 Message-passing interfaces: subsystems request services from other subsystems

through message exchanges

5/28/2025 Software testing Page 31

 Interface testing

5/28/2025 Software testing Page 32

 Interface errors
 interface misuse
 A calling component invokes another component but makes an error in using its
interface – for example, by passing parameters in the wrong order

 Interface misunderstanding
 A calling component makes incorrect assumptions about the behavior of the component
it is calling

 Timing errors
 The called and calling components operate at different speeds, leading to the use of
outdated or incorrected information

5/28/2025 Software testing Page 33

 Interface testing guidelines
 Design tests where parameters passed to a called procedure are at the extreme
ends of their ranges

 Always test pointer parameters with null pointers

 Design tests that intentionally cause the component to fail

 In shared memory system, vary the order in which components are activated

5/28/2025 Software testing Page 34

 System testing
 System testing during development involves integrating components to create a
version of the system and then testing the integrated system

 The focus of system testing is to test the interactions between components

 System testing ensures that components are compatible, interact correctly, and

transfer the right data at the right time across their interfaces

5/28/2025 Software testing Page 35

 Use-case testing
 The use cases developed to identify system interactions can be used as a basis for
system testing

 Each use case usually involves several system components, so testing the use case
forces these interactions to occur

 The sequence diagrams associated with the use case document the components
and interactions being tested

5/28/2025 Software testing Page 36

 Collect weather data sequence chart

5/28/2025 Software testing Page 37

 Test cases derived from sequence diagram

 A requests for a report should have an associated acknowledgement

 A report should ultimately be returned in response to the request

 You should create summarized data that can be used to verify that the report is
correctly organized

 A request for a report to the WeatherStation should result in a summarized report

being generated

 This can be tested by creating raw data that corresponds to the prepared
summary used for testing the summary

 The same raw data is also used to test the WeatherData objects

5/28/2025 Software testing Page 38

Test-driven development
 Test-driven development
 Test-driven development (TDD) is an approach to program development in which
testing and code development are interleaved

 In TDD, tests are written before the code

 You develop code incrementally, along with a test for each increment

 You don’t move on to the next increment until the code you have developed
passes its test

 TDD was introduced as part of agile methods, such as extreme programming

 However, it can also be applied with a plan-driven development process

5/28/2025 Software testing Page 40

 Test-driven development

5/28/2025 Software testing Page 41

 TDD process activities
 Start by identifying the required increments of functionality

 This increment should typically be small and implementable in just a few lines of
code

 Write a test for this functionality and implement it as an automated test

 Run the test, along with any other previously implemented tests

 Implement the functionality, then re-run the test

 Once all tests pass successfully, you can move on to implementing the next
increment of functionality

5/28/2025 Software testing Page 42

 Benefits of test-driven development
 Code coverage
 every code segment you write has at least one associated test, ensuring that all written
code is covered by at least one test
 Regression testing
 a regression test suites is developed incrementally as the program evolves, allowing
earlier functionality to be continuously verified
 Simplified debugging
 When a test fails, it should be clear where the problem lies – in the newly written code,
which can then be checked and modified accordingly
 System documentation
 The tests themselves act as a form of documentation, describing the expected behavior
of the code

5/28/2025 Software testing Page 43

 Regression testing
 is the process of testing a system to ensure that the recent changes have not
broken previously working code

 In a manual testing process, regression testing can be expensive ad time

consuming, but with automated testing, it becomes simple and straightforward

 All test are re-run every time a change is made to the program

 All tests must pass successfully before the change is committed

5/28/2025 Software testing Page 44

Release testing
 Release testing
 Is the process of testing a particular release of a system that is intended for use
outside the development team

 The primary goal of release testing is to convince the system’s supplier that it’s
good enough for deployment and use

 Therefore, release testing must demonstrate that the system delivers its specified
functionality, performance, and dependability, and that it doesn’t fail during
normal operation

 Release testing is typically a back-box testing process, in which tests are derived
solely from the system specification

5/28/2025 Software testing Page 46

 Release testing & system testing
 release testing is a form of system testing, but there are some important
differences

 A separate team, which has not been involved in the system’s development,
should be responsible for release testing

 System testing performed by the development team typically focuses on

discovering bugs in the system (i.e., defect testing)

 In contrast, the objective of release testing is to verify that the system meets its
requirements and is suitable for external use – this is known as validation testing

5/28/2025 Software testing Page 47

 Requirements based testing
 involves examining each requirement and developing one or more tests for it

 Examples – Mentcare system requirements

 If a patient is known to be allergic to a particular medication, then prescribing that
medication shall result in a warning message being issued to the system user

 If a prescriber choose to ignore an allergy warning, they shall be required to provide a

reason for doing so

5/28/2025 Software testing Page 48

 Requirements tests
 Set up a patient record with no known allergies
 Prescribe a medication that typically triggers allergic reaction
 Verify that no warning message is issued by the system
 Set up a patient record with a known allergy
 Prescribe the medication to which the patient is allergic, and verify that a warning is
issued by the system
 Set up a patient record with allergies to two or more drugs
 Prescribe each of these drugs separately and verify that the correct warning is issued for
each drug
 Prescribe two drugs that the patient is allergic to. Verify that two warnings are correctly
issued
 Prescribe a drug that trigger a warning and overrule that warning. Verify that the system
requires user to provide an explanation for overriding the warning

5/28/2025

Software testing Page 49

A usage scenario for the Mentcare system
 George is a nurse who specializes in mental healthcare. One of his responsibilities is to visit
patients at home to check that their treatment is effective and that they are not suffering from
medication side effects
 On a day for home visits, George logs into the Mentcare system and uses it to print his schedule of
home visits for that day, along with summary information about the patient to be visited. He
requests that the records for these patients be downloaded to his laptop. He is prompted for his
key phrase to encrypt the records on the laptop
 One of the patients that he visits is Jim, who is being treated with medication for depression. Jim
feels that the medication is helping him but believes that it has the side effect of keeping him
awake at night. George looks up Jim’s record and is prompted for his key phrase to decrypt the
record. He checks the drug prescribed and queries its side effects. Sleeplessness is a known side
effect so he notes the problem in Jim’s record and suggests that he visits the clinic to have his
medication changed. Jim agrees so George enters a prompt to call him when he gets back to the
clinic to make an appointment with a physician. George ends the consultation and the system re-
encrypts Jim’s record
 After finishing his consultations, George returns to the clinic and uploads the records of patients
visited to the database. The system generates a call list for George of those patients who he has to
contact for follow-up information and make clinic appointments

5/28/2025 Software testing Page 50

E.g….

Features tested by scenario

 Authentication by logging on to the system
 Downloading specified patient records to a laptop
 Home visit scheduling
 Encryption and decryption of patient records on a mobile device
 Record retrieval and modification
 Links with the drugs database that maintains side-effect information
 The system for call prompting

5/28/2025 Software testing Page 51

 Performance testing
 Part of release testing may involve evaluating the emergent properties of a

system, such as performance and reliability

 Performance testing typically involves planning a series of tests in which the load

is gradually increased until the system’s performance becomes unacceptable

 Stress testing is a specific type of performance testing where the system is

deliberately overload to observe its failure behavior

5/28/2025 Software testing Page 52

User testing
 User testing
 user or customer testing is a stage in the testing process where users or
customers provide input and offer feedback on system testing

 User testing is essential, even when comprehensive system and release testing
have already been performed
 This is because factors from the user’s working environment can be a significant impact
on the reliability performance, usability and robustness of the system

5/28/2025 Software testing Page 54

 Types of user testing
 Alpha testing
 Users of the software work with the development team to test the software at the
developer’s site

 Beta testing
 a release of the software is made available to users, allowing them to experiment with
it and report any issues they discovery to the system developers

 Acceptance testing
 Customers test the system to determine whether it is ready to be accepted and
deployed in the customer’s environment

5/28/2025 Software testing Page 55

 The acceptance testing process

5/28/2025 Software testing Page 56

 Stages in the acceptance testing process

 Define acceptance criteria

 Plan acceptance testing
 Derive acceptance tests
 Run acceptance tests
 Negotiate test results
 Reject/accept system

5/28/2025 Software testing Page 57

 Agile methods & acceptance testing

 In agile methods, the user or customer is considered part of the development

team and is responsible for making decisions about the acceptability of the system

 Tests are defined by the users and are integrated with other tests so that they are
run automatically whenever changes are made

5/28/2025 Software testing Page 58

 Key points
 Testing can only show the presence of errors in a program. It cannot
demonstrate that there’re no remaining faults
 Development testing is the responsibility of the software development
team.
 A separate team should be responsible for testing a system before it is released
to customers
 Development testing includes unit testing, in which you test individual
objects and methods component testing in which you test related groups of
objects and system testing, in which you test partial or complete systems

5/28/2025 Software testing Page 59

Key points
 When testing software, you should try to ‘break’ the software by using experience
and guidelines to choose types of test case that have been effective in
discovering defects in other systems
 Whenever possible, you should write automated tests. The tests are embedded
in a program that can be run every time a change is made to a system
 Test-first development is an approach to development where tests are written
before the code to be tested
 Scenario testing involves inventing a typical usage scenario and using this
to derive test cases
 Acceptance testing is a user testing process where the aim is to decide if the
software is good enough to be deployed and used in its operational
environment

5/28/2025 Software testing Page 60