Writing Clean Tests – Trouble in Paradise

Petri KainulainenMarch 31st, 2015Last Updated: March 30th, 2015

2 70 5 minutes read

If our code has obvious faults, we are very motivated to improve it. However, at some point we decide that our code is “good enough” and move on.

Typically this happens when we think that the benefits of improving our existing code are smaller than the required work. Of course, if we underestimate our return of investment, we can make the wrong call, and it can hurt us.

This is what happened to me, and I decided to write about it so that you can avoid making the same mistake.

Writing “Good” Unit Tests

If we want to write “good” unit tests, we have to write unit tests that:

Test only one thing. A good unit test can fail for only one reason and can assert only one thing.
Are named properly. The name of the test method must reveal what went wrong if the test fails.
Mock external dependencies (and state). If a unit test fails, we know exactly where the problem is.

Additional Reading:
A Unit Test Should Test Only One Thing
Writing Clean Tests: Naming Matters
Writing Clean Tests: Divide and Conquer
Writing Clean Tests: To Verify Or Not To Verify

If we write unit tests that fulfil these conditions, we will write good unit tests. Right?

I used to think so. Now I doubt it.

The Road to Hell Is Paved with Good Intentions

I have never met a software developer who decided to write crappy unit tests. If a developer is writing unit tests, it is a lot more likely that he/she wants to write good unit tests. However, this doesn’t mean that the unit tests written by that developer are good.

I wanted to write unit tests that are both easy to read and maintain. I have even written a tutorial that describes how we can write clean tests. The problem is that the advice given in this tutorial is not good enough (yet). It helps us to get started, but it doesn’t show us how deep the rabbit hole really is.

The approach that is described in my tutorial has two major problems:

Naming Standards FTW?

If we use the “naming standard” that was introduced by Roy Osherove, we notice that it is surprisingly hard to describe the state under test and the expected behavior.

This naming standard works very well when we are writing tests for simple scenarios. The problem is that real software is not simple. Typically we end up naming our test methods by using one of these two options:

First, if we try to be as specific as possible, the method names of our test methods become way too looooooooong. In the end, we have to admit that we cannot be as specific as we would want to because the method names would take too much space.

Second, if we try to keep the method names as short as possible, the method names won’t really describe the tested state and the expected behavior.

It doesn’t really matter which option we choose because we will run into the following problem anyway:

If a test fails, the method name won’t necessarily describe want went wrong. We can solve this problem by using custom assertions, but they aren’t free.
It is hard to get a brief overview of the scenarios that are covered by our tests.

Here are the names of the test methods that we have written during the Writing Clean Tests tutorial:

registerNewUserAccount_SocialSignInAndDuplicateEmail_ShouldThrowException()
registerNewUserAccount_SocialSignInAndDuplicateEmail_ShouldNotSaveNewUserAccount()
registerNewUserAccount_SocialSignInAndUniqueEmail_ShouldSaveNewUserAccountAndSetSignInProvider()
registerNewUserAccount_SocialSignInAndUniqueEmail_ShouldReturnCreatedUserAccount()
registerNewUserAccount_SocialSignInAnUniqueEmail_ShouldNotCreateEncodedPasswordForUser()

These method names aren’t very long, but we must remember that these unit tests are written to test a simple registration method. When I have used this naming convention for writing automated tests for a real life software project, the longest method names have been twice as long as our longest example.

That is not very clean or readable. We can do a lot better.

There Is No Common Config

We have made our unit tests a lot better during this tutorial. Nevertheless, they still suffer from the fact that there is no “natural” way to share configuration between different unit tests.

This means that our unit tests contain a lot of duplicate code which configures our mock objects and creates other objects that are used in our unit tests.

Also, since there is no “natural” way to indicate that some constants are relevant only for specific test methods, we have to add all constants to the beginning of the test class.

The source code of our test class looks as follows (the problematic code is highlighted):

import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.ArgumentCaptor;
import org.mockito.Mock;
import org.mockito.invocation.InvocationOnMock;
import org.mockito.runners.MockitoJUnitRunner;
import org.mockito.stubbing.Answer;
import org.springframework.security.crypto.password.PasswordEncoder;
 
import static com.googlecode.catchexception.CatchException.catchException;
import static com.googlecode.catchexception.CatchException.caughtException;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Matchers.isA;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyZeroInteractions;
import static org.mockito.Mockito.when;
 
@RunWith(MockitoJUnitRunner.class)
public class RepositoryUserServiceTest {
 
    private static final String REGISTRATION_EMAIL_ADDRESS = "john.smith@gmail.com";
    private static final String REGISTRATION_FIRST_NAME = "John";
    private static final String REGISTRATION_LAST_NAME = "Smith";
    private static final SocialMediaService SOCIAL_SIGN_IN_PROVIDER = SocialMediaService.TWITTER;
 
    private RepositoryUserService registrationService;
 
    @Mock
    private PasswordEncoder passwordEncoder;
 
    @Mock
    private UserRepository repository;
 
    @Before
    public void setUp() {
        registrationService = new RepositoryUserService(passwordEncoder, repository);
    }
 
    @Test
    public void registerNewUserAccount_SocialSignInAndDuplicateEmail_ShouldThrowException() throws DuplicateEmailException {
        RegistrationForm registration = new RegistrationFormBuilder()
                .email(REGISTRATION_EMAIL_ADDRESS)
                .firstName(REGISTRATION_FIRST_NAME)
                .lastName(REGISTRATION_LAST_NAME)
                .isSocialSignInViaSignInProvider(SOCIAL_SIGN_IN_PROVIDER)
                .build();
 
        when(repository.findByEmail(REGISTRATION_EMAIL_ADDRESS)).thenReturn(new User());
 
        catchException(registrationService).registerNewUserAccount(registration);
 
        assertThat(caughtException()).isExactlyInstanceOf(DuplicateEmailException.class);
    }
 
    @Test
    public void registerNewUserAccount_SocialSignInAndDuplicateEmail_ShouldNotSaveNewUserAccount() throws DuplicateEmailException {
        RegistrationForm registration = new RegistrationFormBuilder()
                .email(REGISTRATION_EMAIL_ADDRESS)
                .firstName(REGISTRATION_FIRST_NAME)
                .lastName(REGISTRATION_LAST_NAME)
                .isSocialSignInViaSignInProvider(SOCIAL_SIGN_IN_PROVIDER)
                .build();
 
        when(repository.findByEmail(REGISTRATION_EMAIL_ADDRESS)).thenReturn(new User());
 
        catchException(registrationService).registerNewUserAccount(registration);
 
        verify(repository, never()).save(isA(User.class));
    }
 
    @Test
    public void registerNewUserAccount_SocialSignInAndUniqueEmail_
ShouldSaveNewUserAccountAndSetSignInProvider() throws DuplicateEmailException {
        RegistrationForm registration = new RegistrationFormBuilder()
                .email(REGISTRATION_EMAIL_ADDRESS)
                .firstName(REGISTRATION_FIRST_NAME)
                .lastName(REGISTRATION_LAST_NAME)
                .isSocialSignInViaSignInProvider(SOCIAL_SIGN_IN_PROVIDER)
                .build();
 
        when(repository.findByEmail(REGISTRATION_EMAIL_ADDRESS)).thenReturn(null);
 
        registrationService.registerNewUserAccount(registration);
 
        ArgumentCaptor<User> userAccountArgument = ArgumentCaptor.forClass(User.class);
        verify(repository, times(1)).save(userAccountArgument.capture());
 
        User createdUserAccount = userAccountArgument.getValue();
 
        assertThatUser(createdUserAccount)
                .hasEmail(REGISTRATION_EMAIL_ADDRESS)
                .hasFirstName(REGISTRATION_FIRST_NAME)
                .hasLastName(REGISTRATION_LAST_NAME)
                .isRegisteredUser()
                .isRegisteredByUsingSignInProvider(SOCIAL_SIGN_IN_PROVIDER);
    }
 
 
    @Test
    public void registerNewUserAccount_SocialSignInAndUniqueEmail_ShouldReturnCreatedUserAccount() throws DuplicateEmailException {
        RegistrationForm registration = new RegistrationFormBuilder()
                .email(REGISTRATION_EMAIL_ADDRESS)
                .firstName(REGISTRATION_FIRST_NAME)
                .lastName(REGISTRATION_LAST_NAME)
                .isSocialSignInViaSignInProvider(SOCIAL_SIGN_IN_PROVIDER)
                .build();
 
        when(repository.findByEmail(REGISTRATION_EMAIL_ADDRESS)).thenReturn(null);
 
        when(repository.save(isA(User.class))).thenAnswer(new Answer<User>() {
            @Override
            public User answer(InvocationOnMock invocation) throws Throwable {
                Object[] arguments = invocation.getArguments();
                return (User) arguments[0];
            }
        });
 
        User createdUserAccount = registrationService.registerNewUserAccount(registration);
 
        assertThatUser(createdUserAccount)
                .hasEmail(REGISTRATION_EMAIL_ADDRESS)
                .hasFirstName(REGISTRATION_FIRST_NAME)
                .hasLastName(REGISTRATION_LAST_NAME)
                .isRegisteredUser()
                .isRegisteredByUsingSignInProvider(SOCIAL_SIGN_IN_PROVIDER);
    }
 
    @Test
    public void registerNewUserAccount_SocialSignInAnUniqueEmail_ShouldNotCreateEncodedPasswordForUser() throws DuplicateEmailException {
        RegistrationForm registration = new RegistrationFormBuilder()
                .email(REGISTRATION_EMAIL_ADDRESS)
                .firstName(REGISTRATION_FIRST_NAME)
                .lastName(REGISTRATION_LAST_NAME)
                .isSocialSignInViaSignInProvider(SOCIAL_SIGN_IN_PROVIDER)
                .build();
 
        when(repository.findByEmail(REGISTRATION_EMAIL_ADDRESS)).thenReturn(null);
 
        registrationService.registerNewUserAccount(registration);
 
        verifyZeroInteractions(passwordEncoder);
    }
}

Some developers would claim that unit tests that look like the above example are clean enough. I understand this sentiment because I used to be one of them. However, these unit tests have three problems:

The essence of the case is not as clear as it could be. Because each test method configures itself before it invokes the tested method and verifies the expected outcome, our test methods become longer than necessary. This means that we cannot just take a quick peek at a random test method and figure out what it tests.
Writing new unit tests is slow. Because every unit test has to configure itself, adding new unit tests to our test suite is a lot slower than it could be. Another “unexpected” downside is that this kind of unit tests encourage people to practice copy-and-paste programming.
Maintaining these unit tests is a pain in the ass. We have to make changes to every unit test if we add a new mandatory field to the registration form or the change the implementation of the registerNewUserAccount() method. These unit tests are way too brittle.

In other words, these unit tests are hard to read, hard to write, and hard to maintain. We must do a better job.

Summary

This blog post has taught us four things:

Even though we think that we are writing good unit tests, that is not necessarily true.
If changing existing features is slow because we have to change many unit tests, we are not writing good unit tests.
If adding new features is slow because we have to add so much duplicate code to our unit tests, we are not writing good unit tests.
If we cannot see what situations are covered by our unit tests, we are not writing good unit tests.

The next part of this tutorial answers to this very relevant question: