Testing, Part 2

Introduction to Computer Programming

Lecture W9C

Prof. Rutwa Engineer,

Prof. Dan Zingaro,
Prof. Peter Dixon,
Prof. Randy Hickey,
Prof. Romina Piunno

Mathematical and Computational Sciences

University of Toronto Mississauga
1 / 19
Agenda

1. Motivation – limitations of Doctests

2. unittesting

3. Other levels of Testing

4. Tasks

2 / 19
Motivation: Limits of Doctests

Attempt to create doctests for Task 3 in W07C (pasted below).

1 def update_cell(row, col, v, filename) -> None:

2 '''Update the value <v> at <row>, <col> in the
3 CSV file, creating empty entries as needed.'''

What are some barriers that prevent you from creating doctests?

3 / 19
Motivation: Limits of Doctests

Doctests only compare printed output. A non-exhaustive list of

limitations of doctests:

1. Where functions that return NoneType

2. Where order/format of output is not deterministic

3. Situations where code set-up/tear-down is needed

4. When many test cases in docstring becomes impractical.

4 / 19
unittest

We will explore the use of the unittest module to implement

tests.

Some unit tests can be written with a doctest. For these, you
don’t have to use the unittest module to write a unit test.

But unittest provides additional functionality to set up tests that

make it easier to test things that do not have screen outputs (like
files).

5 / 19
Unit Testing

▶ Unit Testing is intended to isolate components of a program

and demonstrate that they are correct.

▶ Its purpose is to validate each unit of the software

separately, ensuring it performs as designed.

▶ Con: It may not catch issues when different pieces of code

interact (e.g. external modules). Hence, integration and
system testing ...

6 / 19
Code Framework

1 import unittest
2 class Test<function's name>(unittest.TestCase):
3 def test_<descriptive test case name>(self):
4 any number of lines of
5 code you need for set up/tear down
6 self.assert<option>(a, b, message)
7

8 As many tests as you want

.
.
9 .
10 unittest.main()

7 / 19
 Example 1

1 import unittest
2 MSG = 'abs(), input: {}, expected: {}, received: {}'
3 class TestAbs(unittest.TestCase):
4 def test_positive(self):
5 input_val = 8
6 expected = 8
7 result = abs(input_val)
8 self.assertEqual(expected, result,
9 MSG.format(input_val, expected, result))
10 unittest.main()

8 / 19
assert Methods
In the above, we saw a unittest that used assertEqual. Other
tests are available.

Method Checks that Negative Case

assertEqual(a, b) a == b assertNotEqual(a, b)

assertTrue(a) bool(a) is True assertFalse(a)

assertIs(a, b) a is b assertIsNot(a, b)

assertNone(a) a is None assertNotNone(a)

assertIn(a, b) a in b assertNotIn(a, b)

assertIsInstance(a, b) isinstance(a, b) assertNotIsInstance(a, b

9 / 19
Metric: Code Coverage
The goal is to test every piece of the code’s functionality, but it’s
hard to measure “functionality.”
One metric for test quality is code coverage.

A program with higher code coverage has more of its source code
executed by a particular test suite. There are python packages
that can measure this!

Knowing how much of your code is covered under testing can

help. But this heuristic doesn’t account for corner cases based on
values, so keep an eye out for those!
10 / 19
Open-Box Approach
Unit testing can be performed using a closed-box (last time) or
an open-box approach.

Open-box testing involves testing a system with full knowledge

of its implementation. Given some input, you are exercising a
specific path through the code to achieve some output.

More details: https://www.indeed.com/career-advice/career-development/what-is-open-box-testing

11 / 19
Testing Tips
▶ Each test case should be short and focused. This means
having more small tests are better than one long test.

▶ Write down test ideas as you’re coding! Often this is when

the best edge cases and testing scenarios are thought up.

▶ Helper functions/methods can improve testing

infrastructure.

▶ Test for robustness! A system change should only ever break

tests related to the change itself. Modularity of components
and reduction of dependencies are important!
12 / 19
Doctests as Unittests

Task 1
Take the function abs from last class, and construct unit tests
for it.

Task 2
Take the function indices from last class, and construct unit
tests for it.

13 / 19
No Equivalent Docttest

Task 3
Recall Task 2 from the lecture on CSVs (W07C):

def list_to_csv(xs, ys, zs, file_name) -> None

Create a unittest suite with a few test cases.

14 / 19
Recall: Why do we write tests?

1. To find bugs in the early stages of code development.

2. To ensure the code meets the design requirements for the

user’s needs.

3. Because bugs found earlier in the development cycle cost

less to fix than those found later (e.g. by an End User in
production).

4. Quality assurance of an application. Further, reputation and

integrity of that product.

15 / 19
Testing Overview
There are many types of testing ...

Sanity testing
Usability testing
Security testing
Performance testing
Smoke testing
Happy path testing
Gorilla testing
... who comes up with these names, anyway?

More descriptions: https://ca.indeed.com/career-advice/career-development/types-of-software-testing

16 / 19
Levels of Testing
Each of those types of testing takes a different approach or has a
different goal. When testing functionality, there are four levels:
1. Unit Testing: Does a module work? Often performed by
developers, followed by testers.

2. Integration Testing: Do sets of components work

together? Often performed by a testing team.

3. System Testing: Does the system, as a whole, work?

Often performed by a testing team.

4. Acceptance Testing: Does the product meet the business

criteria? Often performed by testers and end users.
17 / 19
Levels of Testing
1. Unit Testing: Does a module work? Often performed by
developers, then testers.

2. Integration Testing: Do sets of components work

together? Often performed by a testing team.

3. System Testing: Does the system, as a whole, work?

Often performed by a testing team.

4. Acceptance Testing: Does the product meet the business

criteria? Often performed by testers and end users.

We cover Unit Testing in this course. You’ll learn others in later

CS courses. 18 / 19
Next Time

1. Regular Expressions (REGEX)

19 / 19