I want to thank Steven Colbert for coining a word I can use in my title. Without him, all this would still be possible, had I not given up looking for a better word after a few minutes.
Tests are about trust. We expect them to be reliable. Reliable tests tell us everything is ok when they pass, and that something is wrong when they fail.
The problem is that life is not black and white, and tests are not just green and red. Tests can give false positive (fail when they shouldn’t) or false negative (pass when they shouldn’t) results. We’ve encountered the false positive ones before – these are the fragile, dependent tests.
The ones that pass, instead of failing, are the problematic ones. They hide the real picture from us, and erode our trust, not just in those tests, but also in others. After all, when we find out a problematic tests, who can say the others we wrote are not problematic as well?
Truthiness (how much we feel the tests are reliable) comes into play.
Dependency Injection Example
Or rather, injecting an example of a dependency.
Let’s say we have a service (or 3rd party library) our tested code uses. It’s slow and communication is unreliable. All the things that give services a bad name. Our natural tendency is to mock the service in the test. By mocking the service, we can test our code in isolation.
So, in our case, our tested Hotel class uses a Service:
public class Hotel
{
public string GetServiceName(Service service)
{
var result = service.GetName();
return "Name: " + result;
}
}To know if the method works correctly, we’ll write this test:
[TestMethod]public void GetServiceName_RoomService_NameIsRoom()
{
var fakeService = A.Fake<Service>();
A.CallTo(() => fakeService.GetName()).Returns("Room");
var hotel = new Hotel();
Assert.AreEqual("Name: Room", hotel.GetServiceName(fakeService));
}And everything is groovy.
Until, in production, the service gets disconnected and throws an exception. And our test says “B-b-b-but, I’m still passing!”.
The Truthiness Is Out There
Mocking is an example of obstructing the real behavior by prescriptive tests, but it’s just an example. It can happen when we test a few cases, but don’t cover others.
Here’s one of my favorite examples. What’s the hidden test case here?
public int Increment()
{
return counter++;
}Tests are code examples. They work to the extent of our imagination of “what can go wrong?” Like overflow, in the last case.
Much like differentiation, truthiness can not be examined by itself. The example works, but it hides a case we need another test for. We need to look at the collection of test cases, and see if we covered everything.
The solution doesn’t have to be a test of the same type. We can have a unit test for the service happy path, and an end-to-end test to cover the disconnection case. Of course, if you can think of other cases in the first place, why not unit test them?
So to level up your truthiness:
- Ideate. Before writing the tests, and if you’re doing TDD – the code, write a list of test cases. On a notebook, a whiteboard, or my favorite: empty tests.
- Reflect. Often, when we write a few test, new test cases come to mind. Having a visual image of the code can help think of other cases.
- Beware the mock. We use mocks to prescribe dependency behavior in specific cases. Every mock you make can be a potential failure point, so think about other cases to mock.
- Review. Do it in pairs. Four eyes are better than two.
Aim for higher truthiness. Higher trust in your tests will help you sleep better.
| Reference: | Test Attribute #8 – Truthiness from our JCG partner Gil Zilberfeld at the Geek Out of Water blog. |
