Writing unit tests

Overview

Teaching: 30 min
Exercises: 10 min

Questions

What is a unit test?

How do I write and run unit tests?

How can I avoid test duplication and ensure isolation?

How can I run tests automatically and measure their coverage?

Objectives

Implement effective unit tests using pytest

Execute tests in Visual Studio Code

Explain the issues relating to non-deterministic code

Implement fixtures and test parametrisation using pytest decorators

Configure git hooks and pytest-coverage

Apply best-practices when setting up a new Python project

Recognise analogous tools for other programming languages

Apply testing to Jupyter notebooks

Introduction

Unit testing validates, in isolation, functionally independent components of a program.

In this lesson we’ll demonstrate how to write and execute unit tests for a simple scientific code.

This involves making some technical decisions…

Test frameworks

We’ll use pytest as our test framework. It’s powerful but also user friendly.

For comparison: testing using assert statements:

from temperature import to_fahrenheit

assert to_fahrenheit(30) == 86

Testing using the built-in unittest library:

from temperature import to_fahrenheit
import unittest

class TestTemperature(unittest.TestCase):
    def test_to_farenheit(self):
        self.assertEqual(to_fahrenheit(30), 86)

Testing using pytest:

from temperature import to_fahrenheit

def test_answer():
    assert to_fahrenheit(30) == 86

Why use a test framework?

Avoid reinventing the wheel - frameworks such as pytest provide lots of convenient features (some of which we’ll see shortly)
Standardisation leads to better tooling, easier onboarding etc

Projects that use pytest:

Learning by example

Reading the test suites of mature projects is a good way to learn about testing methodologies and frameworks

Code editors

We’ve chosen Visual Studio Code as our editor. It’s free, open source, cross-platform and has excellent Python (and pytest) support. It also has built-in Git integration, can be used to edit files on remote systems (e.g. HPC), and handles Jupyter notebooks (plus many more formats).

Demonstration of pytest + VS Code + coverage

Test discovery, status indicators and ability to run tests inline

Code navigation (“Go to Definition”)

The Test perspective and Test Output

Maximising coverage (assert recursive_fibonacci(7) == 13)

Test-driven development: adding and fixing a new test (test_negative_number)

A tour of pytest

Checking for exceptions

If a function invocation is expected to throw an exception it can be wrapped with a pytest raises block:

def test_non_numeric_input():
    with pytest.raises(TypeError):
        recursive_fibonacci("foo")

Parametrisation

Similar test invocations can be grouped together to avoid repetition. Note how the parameters are named, and “injected” by pytest into the test function at runtime:

@pytest.mark.parametrize("number,expected", [(0, 0), (1, 1), (2, 1), (3, 2)])
def test_initial_numbers(number, expected):
    assert recursive_fibonacci(number) == expected

This corresponds to running the same test with different parameters, and is our first example of a pytest decorator (@pytest). Decorators are a special syntax used in Python to modify the behaviour of the function, without modifying the code of the function itself.

Skipping tests and ignoring failures

Sometimes it is useful to skip tests (conditionally or otherwise), or ignore failures (for example if you’re in the middle of refactoring some code).

This can be achieved using other @pytest.mark annotations e.g.

@pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
def test_linux_only_features():
    ...

@pytest.mark.xfail
def test_unimplemented_code():
    ...

Refactoring

Code refactoring is “the process of restructuring existing computer code without changing its external behavior” and is often required to make code more amenable to testing.

Fixtures

If multiple tests require access to the same data, or a resource that is expensive to initialise, then it can be provided via a fixture. These can be cached by modifying the scope of the fixture. See this example from Devito:

@pytest.fixture
def grid(shape=(11, 11)):
    return Grid(shape=shape)

def test_forward(grid):
    grid.data[0, :] = 1.
    ...

def test_backward(grid):
    grid.data[-1, :] = 7.
    ...

This corresponds to providing the same parameters to different tests.

Tolerances

It’s common for scientific codes to perform estimation by simulation or other means. pytest can check for approximate equality:

def test_approximate_pi():
    assert 22/7 == pytest.approx(math.pi, abs=1e-2)

Random numbers

If your simulation or approximation technique depends on random numbers then consistently seeding your generator can help with testing. See random.seed() for an example or the pytest-randomly plugin for a more comprehensive solution.

doctest

pytest has automatic integration with the Python’s standard doctest module when invoked with the --doctest-modules argument. This is a nice way to provide examples of how to use a library, via interactive examples in docstrings:

def recursive_fibonacci(n):
    """Return the nth number of the fibonacci sequence

    >>> recursive_fibonacci(7)
    13
    """
    return n if n <=1 else recursive_fibonacci(n - 1) + recursive_fibonacci(n - 2)

Hands-on unit testing (10 min)

Getting started

Setting up our editor
If you haven’t already, see the setup guide for instructions on how to install Visual Studio Code and conda.

Download and extract this zip file. If using an ICT managed PC please be sure to do this in your user area on the C: drive i.e. C:\Users\your_username

Note that files placed here are not persistent so you must remember to take a copy before logging out

In Visual Studio Code go to File > Open Folder… and find the files you just extracted.

If you see an alert “This workspace has extension recommendations.” click Install All and then switch back to the Explorer perspective by clicking the top icon on the left-hand toolbar
Open Anaconda Prompt (Windows), or a terminal (Mac or Linux) and run:
conda env create --file "path_to_environment.yml"
The path_to_environment.yml can be obtained by right-clicking the file name in the left pane of Visual Studio Code and choosing “Copy Path”. Be sure to include the quotation marks. Right click on the command line interface to paste.
Important: After the environment has been created go to View > Command Palette in VS Code, start typing “Python: Select interpreter” and hit enter. From the list select your newly created environment “diffusion”
Running the tests

Open test_diffusion.py

You should now be able to click on Run Test above the test_heat() function and see a warning symbol appear, indicating that the test is currently failing. You may have to wait a moment for Run Test to appear.

Switch to the Test perspective by clicking on the flask icon on the left-hand toolbar. From here you can Run All Tests, and Show Test Output to view the coverage report (see Lesson 1 for more on coverage)

Important: If you aren’t able to run the test then please ask a demonstrator for help. It is essential for the next exercise.

Advanced topics

More pytest plugins

pytest-benchmark provides a fixture that can transparently measure and track performance while running your tests:

def test_fibonacci(benchmark):
    result = benchmark(fibonacci, 7)
    assert result == 13

pytest-benchmark example

Demonstration of performance regression via recursive and formulaic approaches to Fibonacci calculation (output)

pytest-notebook can check for regressions in your Jupyter notebooks (see also Jupyter CI)

Hypothesis provides property-based testing, which is useful for verifying edge cases):

from fibonacci import recursive_fibonacci
from hypothesis import given, strategies

@given(strategies.integers())
def test_recursive_fibonacci(n):
    phi = (5 ** 0.5 + 1) / 2
    assert recursive_fibonacci(n) == int((phi ** n - -phi ** -n) / 5 ** 0.5)

Taking testing further

Testing in other languages

We’ve discussed tools and approaches for Python but analogues exist for other languages
We (the RCS) have had success with:
- C++: Catch2
- Fortran: pFUnit
- R: testthat
See the Software Sustainability Institute’s Build and Test Examples for many more

Further resources

Key Points

Testing is not only standard practice in mainstream software engineering, it also provides distinct benefits for any non-trivial research software

pytest is a powerful testing framework, with more functionality than Python’s built-in features while still catering for simple use cases

Testing with VS Code is straightforward and encourages good habits (writing tests as you code, and simplifying test-driven development)

It is important to have a set-up you can use for every project - so that it becomes as routine in your workflow as version control itself

pytest has a myriad of extensions that are worthy of mention such as Jupyter, benchmark, Hypothesis etc

previous episode

Essential Software Engineering for Researchers

next episode

Writing unit tests

Overview

Introduction

Test frameworks

Learning by example

Code editors

Demonstration of pytest + VS Code + coverage

A tour of pytest

Checking for exceptions

Parametrisation

Skipping tests and ignoring failures

Refactoring

Fixtures

Tolerances

Random numbers

doctest

Hands-on unit testing (10 min)

Getting started

Setting up our editor

Running the tests

Advanced topics

More pytest plugins

pytest-benchmark example

Taking testing further

Testing in other languages

Further resources

Key Points

previous episode

next episode