Fuzzy Assertions

Ashley FriezeNovember 24th, 2019Last Updated: November 22nd, 2019

0 71 2 minutes read

As discussed in other Test Smells, like Trail by Competitive Calculation, and It Passed Yesterday, it’s very easy to create obscure assertions that are either unreliable or don’t prove a huge amount.

While there are often good “bad reasons” for why we cannot write straightforward assertions, recognising when you are not writing one, and doing the best job possible to move back towards the norm is a very good idea.

Let’s look at a test in a fictional language:

// Note - this is is no specific language
test('adding user to database results in new user', {
 
    user = createUser('John', 'Smith')
 
    id = system.addUser(user)
 
    readUser = system.retrieveUser(id)
 
    assert(user.firstName).is('John')
    assert(user.lastName).is('Smith')
});

The above has many properties of a straightforward test:

The test data we’re using ‘John’ and ‘Smith’ is simple and right there in front of us
The system under test as an API which lends itself to testing
We’re asserting with precise discrete values that we were able to predict before the test
Any auto-generated things like id and maybe a userCreationDate (not shown) are not affecting our test

But What About…?

In the above example, there is the hint that maybe users are provided with an id and also a creation timestamp.

There might have been a temptation to try to write an assertion like this:

readUser = system.retrieveUser(id)
 
    assert(user).is(expectedUser)

Where expectedUser has somehow been set up to contain an exact match of the user that the system would generate.

General tip – where there’s a constructed object in a test called expected, there’s a high risk of trial by competitive calculation.

To achieve the ability to predict system generated things, we end up having to plug in our own key generator and mock clock etc into a system. That might be valuable, or it might be a load of test garbage.

Let’s Fuzzy Match

An alternative for the whole object match is to create something like this:

// still a fictional language
    readUser = system.retrieveUser(id)
 
    assert(user).matches([
        { obj.firstName == 'John' },
        { obj.lastName == 'Smith' },
        { obj.id instanceof UUID },
        { obj.creationDate - now() < inSeconds(5) }
    ])

In this made up test, there are some concrete assertions and then some more fuzzy assertions.

Fuzzy Matching is Cumbersome

The above solution shows how you can make relatively meaningful assertions on object type, approximate object value, maybe you could even do regex matches on the contents of fields… it allows you to assert for a value you can’t predict…

But…

The reason the above assertion got big is that we were doing a whole object match on something that can’t be fully matched.

This may be the best option available.

Alternatives

Do fuzzy matches in separate tests, one field at a time – avoid the whole object fuzzy match
Filter out fields that cannot be matched from comparing data – either by blanking, by copying from actual to expected, or by exclusion rules in the comparison
Rig the generators to produce predictable values
Write lower-level tests that can predict the values so you don’t have to rely on fuzzy matching at a higher level