# Introduction to Software Testing¶

Before we get to the central parts of the book, let us introduce essential concepts of software testing. Why is it necessary to test software at all? How does one test software? How can one tell whether a test has been successful? How does one know if one has tested enough? In this chapter, let us recall the most important concepts, and at the same time get acquainted with Python and interactive notebooks.

from bookutils import YouTubeVideo


This chapter (and this book) is not set to replace a textbook on testing; see the Background at the end for recommended reads.

## Simple Testing¶

Let us start with a simple example. Your co-worker has been asked to implement a square root function $\sqrt{x}$. (Let's assume for a moment that the environment does not already have one.) After studying the Newton–Raphson method, she comes up with the following Python code, claiming that, in fact, this my_sqrt() function computes square roots.

def my_sqrt(x):
"""Computes the square root of x, using the Newton-Raphson method"""
approx = None
guess = x / 2
while approx != guess:
approx = guess
guess = (approx + x / approx) / 2
return approx


Your job is now to find out whether this function actually does what it claims to do.

### Understanding Python Programs¶

If you're new to Python, you might first have to understand what the above code does. We very much recommend the Python tutorial to get an idea on how Python works. The most important things for you to understand the above code are these three:

1. Python structures programs through indentation, so the function and while bodies are defined by being indented;
2. Python is dynamically typed, meaning that the type of variables like x, approx, or guess is determined at run-time.
3. Most of Python's syntactic features are inspired by other common languages, such as control structures (while, if), assignments (=), or comparisons (==, !=, <).

With that, you can already understand what the above code does: Starting with a guess of x / 2, it computes better and better approximations in approx until the value of approx no longer changes. This is the value that finally is returned.

### Running a Function¶

To find out whether my_sqrt() works correctly, we can test it with a few values. For x = 4, for instance, it produces the correct value:

my_sqrt(4)

2.0


The upper part above my_sqrt(4) (a so-called cell) is an input to the Python interpreter, which by default evaluates it. The lower part (2.0) is its output. We can see that my_sqrt(4) produces the correct value.

The same holds for x = 2.0, apparently, too:

my_sqrt(2)

1.414213562373095


### Interacting with Notebooks¶

If you are reading this in the interactive notebook, you can try out my_sqrt() with other values as well. Click on one of the above cells with invocations of my_sqrt() and change the value – say, to my_sqrt(1). Press Shift+Enter (or click on the play symbol) to execute it and see the result. If you get an error message, go to the above cell with the definition of my_sqrt() and execute this first. You can also run all cells at once; see the Notebook menu for details. (You can actually also change the text by clicking on it, and corect mistaks such as in this sentence.)

from bookutils import quiz


### Quiz

What does my_sqrt(16) produce?

Try it out for yourself by uncommenting and executing the following line:

# my_sqrt(16)


Executing a single cell does not execute other cells, so if your cell builds on a definition in another cell that you have not executed yet, you will get an error. You can select Run all cells above from the menu to ensure all definitions are set.

Also keep in mind that, unless overwritten, all definitions are kept across executions. Occasionally, it thus helps to restart the kernel (i.e. start the Python interpreter from scratch) to get rid of older, superfluous definitions.

### Debugging a Function¶

To see how my_sqrt() operates, a simple strategy is to insert print() statements in critical places. You can, for instance, log the value of approx, to see how each loop iteration gets closer to the actual value:

def my_sqrt_with_log(x):
"""Computes the square root of x, using the Newton–Raphson method"""
approx = None
guess = x / 2
while approx != guess:
print("approx =", approx)  # <-- New
approx = guess
guess = (approx + x / approx) / 2
return approx

my_sqrt_with_log(9)

approx = None
approx = 4.5
approx = 3.25
approx = 3.0096153846153846
approx = 3.000015360039322
approx = 3.0000000000393214

3.0


Interactive notebooks also allow launching an interactive debugger – insert a "magic line" %%debug at the top of a cell and see what happens. Unfortunately, interactive debuggers interfere with our dynamic analysis techniques, so we mostly use logging and assertions for debugging.

### Checking a Function¶

Let's get back to testing. We can read and run the code, but are the above values of my_sqrt(2) actually correct? We can easily verify by exploiting that $\sqrt{x}$ squared again has to be $x$, or in other words $\sqrt{x} \times \sqrt{x} = x$. Let's take a look:

my_sqrt(2) * my_sqrt(2)

1.9999999999999996


Okay, we do have some rounding error, but otherwise, this seems just fine.

What we have done now is that we have tested the above program: We have executed it on a given input and checked its result whether it is correct or not. Such a test is the bare minimum of quality assurance before a program goes into production.

## Automating Test Execution¶

So far, we have tested the above program manually, that is, running it by hand and checking its results by hand. This is a very flexible way of testing, but in the long run, it is rather inefficient:

1. Manually, you can only check a very limited number of executions and their results
2. After any change to the program, you have to repeat the testing process

This is why it is very useful to automate tests. One simple way of doing so is to let the computer first do the computation, and then have it check the results.

For instance, this piece of code automatically tests whether $\sqrt{4} = 2$ holds:

result = my_sqrt(4)
expected_result = 2.0
if result == expected_result:
print("Test passed")
else:
print("Test failed")

Test passed


The nice thing about this test is that we can run it again and again, thus ensuring that at least the square root of 4 is computed correctly. But there are still a number of issues, though:

1. We need five lines of code for a single test
2. We do not care for rounding errors
3. We only check a single input (and a single result)

Let us address these issues one by one. First, let's make the test a bit more compact. Almost all programming languages do have a means to automatically check whether a condition holds, and stop execution if it does not. This is called an assertion, and it is immensely useful for testing.

In Python, the assert statement takes a condition, and if the condition is true, nothing happens. (If everything works as it should, you should not be bothered.) If the condition evaluates to false, though, assert raises an exception, indicating that a test just failed.

In our example, we can use assert to easily check whether my_sqrt() yields the expected result as above:

assert my_sqrt(4) == 2


As you execute this line of code, nothing happens: We just have shown (or asserted) that our implementation indeed produces $\sqrt{4} = 2$.

Remember, though, that floating-point computations may induce rounding errors. So we cannot simply compare two floating-point values with equality; rather, we would ensure that the absolute difference between them stays below a certain threshold value, typically denoted as $\epsilon$ or epsilon. This is how we can do it:

EPSILON = 1e-8

assert abs(my_sqrt(4) - 2) < EPSILON


We can also introduce a special function for this purpose, and now do more tests for concrete values:

def assertEquals(x, y, epsilon=1e-8):
assert abs(x - y) < epsilon

assertEquals(my_sqrt(4), 2)
assertEquals(my_sqrt(9), 3)
assertEquals(my_sqrt(100), 10)


Seems to work, right? If we know the expected results of a computation, we can use such assertions again and again to ensure our program works correctly.

(Hint: a true Python programmer would use the function math.isclose() instead.)

## Generating Tests¶

Remember that the property $\sqrt{x} \times \sqrt{x} = x$ universally holds? We can also explicitly test this with a few values:

assertEquals(my_sqrt(2) * my_sqrt(2), 2)
assertEquals(my_sqrt(3) * my_sqrt(3), 3)
assertEquals(my_sqrt(42.11) * my_sqrt(42.11), 42.11)


Still seems to work, right? Most importantly, though, $\sqrt{x} \times \sqrt{x} = x$ is something we can very easily test for thousands of values:

for n in range(1, 1000):
assertEquals(my_sqrt(n) * my_sqrt(n), n)


How much time does it take to test my_sqrt() with 100 values? Let's see.

We use our own Timer module to measure elapsed time. To be able to use Timer, we first import our own utility module, which allows us to import other notebooks.

import bookutils.setup

from Timer import Timer

with Timer() as t:
for n in range(1, 10000):
assertEquals(my_sqrt(n) * my_sqrt(n), n)
print(t.elapsed_time())

0.01648758299415931


10,000 values take about a hundredth of a second, so a single execution of my_sqrt() takes 1/1000000 second, or about 1 microseconds.

Let's repeat this with 10,000 values picked at random. The Python random.random() function returns a random value between 0.0 and 1.0:

import random

with Timer() as t:
for i in range(10000):
x = 1 + random.random() * 1000000
assertEquals(my_sqrt(x) * my_sqrt(x), x)
print(t.elapsed_time())

0.01900470902910456


Within a second, we have now tested 10,000 random values, and each time, the square root was actually computed correctly. We can repeat this test with every single change to my_sqrt(), each time reinforcing our confidence that my_sqrt() works as it should. Note, though, that while a random function is unbiased in producing random values, it is unlikely to generate special values that drastically alter program behavior. We will discuss this later below.

## Run-Time Verification¶

Instead of writing and running tests for my_sqrt(), we can also go and integrate the check right into the implementation. This way, each and every invocation of my_sqrt() will be automatically checked.

Such an automatic run-time check is very easy to implement:

def my_sqrt_checked(x):
root = my_sqrt(x)
assertEquals(root * root, x)
return root


Now, whenever we compute a root with my_sqrt_checked()$\dots$

my_sqrt_checked(2.0)

1.414213562373095


... we already know that the result is correct, and will so for every new successful computation.

Automatic run-time checks, as above, assume two things, though:

• One has to be able to formulate such run-time checks. Having concrete values to check against should always be possible, but formulating desired properties in an abstract fashion can be very complex. In practice, you need to decide which properties are most crucial, and design appropriate checks for them. Plus, run-time checks may depend not only on local properties, but on several properties of the program state, which all have to be identified.

• One has to be able to afford such run-time checks. In the case of my_sqrt(), the check is not very expensive; but if we have to check, say, a large data structure even after a simple operation, the cost of the check may soon be prohibitive. In practice, run-time checks will typically be disabled during production, trading reliability for efficiency. On the other hand, a comprehensive suite of run-time checks is a great way to find errors and quickly debug them; you need to decide how many such capabilities you would still want during production.

### Quiz

Does run-time checking give a guarantee that there will always be a correct result?

An important limitation of run-time checks is that they ensure correctness only if there is a result to be checked - that is, they do not guarantee that there always will be one. This is an important limitation compared to symbolic verification techniques and program proofs, which can also guarantee that there is a result – at a much higher (often manual) effort, though.

## System Input vs Function Input¶

At this point, we may make my_sqrt() available to other programmers, who may then embed it in their code. At some point, it will have to process input that comes from third parties, i.e. is not under control by the programmer.

Let us simulate this system input by assuming a program sqrt_program() whose input is a string under third-party control:

def sqrt_program(arg: str) -> None:
x = int(arg)
print('The root of', x, 'is', my_sqrt(x))


We assume that sqrt_program is a program which accepts system input from the command line, as in