Cheap fuzz testing

Testing is all about return on investment: how do we maintain an acceptable level of trust in the code with the least amount of effort? Fuzz testing verifies that the code does something sensible with a large number of randomly generated inputs. It is usually only used on safety- or security-critical code like cryptography libraries, because it can be a lot of work to both implement and run with sufficient amounts of input. But a light-weight variant can be used to improve the payoff of regular tests. As a bonus, it makes tests easier to read and write.

Prerequisites:

We already have a test suite, or we’re about to create one. In this article we’ll be adding fuzzing to existing tests to get more out of them, not writing “pure” fuzz tests.
We know how to write reliable tests. This technique will not be useful if we’re regularly committing tests which fail even though nothing was wrong with the code under test or succeed when the code under test is wrong. This is a normal part of learning to write tests, but introducing fuzzing can make an unreliable test suite even harder to maintain.

Implementation¹

Let’s say we have an existing test where we only care about some of the inputs:

def test_should_raise_exception_when_overbooking_leave() -> None:
    with raises(LeaveDeficitError):
        Employee(
            name="Jane Doe",
            office="Oslo",
            floor=1,
            leave_balance=2,
            salary_band=3,
        ).reserve_leave(3)

We can guess that we only care about the leave_balance argument here², but we still need the rest of the arguments if we’re going to test the code at this level. And in many situations the connection between the inputs and the check are much less obvious. For example, employees in some offices could be allowed go into leave deficit, because of local laws, or because their contract has a specific clause about this. So let’s correct for this, and introduce fuzzing at the same time. Enter generators:

import string
from random import choice, randrange


def random_string(length: int) -> str:
   """
   Includes ASCII printable characters and the first printable character from several Unicode
   blocks <https://en.wikipedia.org/wiki/List_of_Unicode_characters>.
   """
   characters = f"{string.printable}¡ĀƀḂəʰͰἀЀ–⁰₠℀⅐←∀⌀①─▀■☀🬀✁ㄅﬀ"
   return "".join(choice(characters) for _ in range(length))


def any_name() -> str:
    return random_string(20)


def any_office() -> str:
    return random_string(20)


def any_floor() -> int:
    return choice([-1, 1]) * randrange(20)


def any_salary_band() -> int:
    return randrange(10)


def test_should_raise_exception_when_overbooking_leave() -> None:
   with raises(LeaveDeficitError):
      Employee(
          name=any_name(),
          office=any_office(),
          floor=any_floor(),
          leave_balance=2,
          salary_band=any_salary_band(),
      ).reserve_leave(3)

This has achieved several things:

We now have a good chance of getting a test failure in the near future if someone maps a parameter to the wrong field, someone makes an error refactoring the parameters, we misunderstood which values are actually valid, or we have a bug in the production code related to any of these fields.
The test is easier to understand: 2 and 3 are the only literal numbers, so those are the only values relevant for the test.
We can easily change the any methods if for example salary_band becomes a string field.
We don’t need to come up with a bunch of irrelevant but valid values during testing. This is tedious and error-prone work, and we’re better off without it.

Some things to keep in mind:

Some of the generators have the same code, but don’t fall for the temptation to merge them. When the fields’ definition inevitably diverges, you’ll have a much easier time detecting and correcting test errors if you keep them separate.
Make sure each generator actually generates valid data, all the time. For example, randomising the string length is an obvious improvement in many cases, but should the underlying code actually accept empty strings?
The ideal generator should be able to return a large subset of the valid values, to maximise the code confidence.
The ideal generator should also generate values with a high chance of triggering a bug, such as the minimum/maximum allowed integers, zero, shortest/longest strings, and the like. This is often in conflict with the previous point - you can’t have a maximum spread and a high percentage of special values. One way around this is for the generator to randomly choose one of a handful of hardcoded special values in some percentage of the cases, and use the entire range in other cases, for example:
```
if random() >= 0.5:
    return choice(special_values)
else:
    return randrange(first, last)
```
One problem with the new approach as written is that it’s not really possible to reproduce errors - we end up relying on the error message to tell us which inputs cause which errors. We’ll want to use something like pytest-randomly with a seed taken from the environment (such as a job ID in a CI pipeline). That way, we can easily reproduce a run by reusing the seed printed by pytest at runtime.

I’ve used Python for the example code, but it should be easy to apply to any popular language. ↩
If the constructor doesn’t force keyword arguments by adding * as the first argument, this could be even harder to understand: Employee("Jane Doe", "Oslo", 1, 2, 3).reserve_leave(3). Working out which arguments are relevant now at minimum involves reading the constructor signature. ↩

Prerequisites:

Implementation1

Implementation¹