JUnit in a Nutshell: Unit Test Assertion

Frank AppelSeptember 24th, 2014Last Updated: September 23rd, 2014

0 80 10 minutes read

This chapter of JUnit in a Nutshell covers various unit test assertion techniques. It elaborates on the pros and cons of the built-in mechanism, Hamcrest matchers and AssertJ assertions. The ongoing example enlarges upon the subject and shows how to create and use custom matchers/assertions.

Unit Test Assertion

Trust, but verify
Ronald Reagan

The post Test Structure explained why unit tests are usually arranged in phases. It clarified that the real testing aka the outcome verification takes place in the third phase. But so far we have only seen some simple examples for this, using mostly the built-in mechanism of JUnit.

As shown in Hello World, verification is based on the error type AssertionError. This is the basis for writing so called self-checking tests. A unit test assertion evaluates predicates to true or false. In case of false an AssertionError is thrown. The JUnit runtime captures this error and reports the test as failed.

The following sections will introduce three of the more popular unit test assertion variants.

Assert

The built-in assertion mechanism of JUnit is provided by the class org.junit.Assert. It offers a couple of static methods to ease test verification. The following snippet outlines the usage of the available method patterns:

fail();
fail( "Houston, We've Got a Problem." );

assertNull( actual );
assertNull( "Identifier must not be null.",
            actual );

assertTrue( counter.hasNext() );
assertTrue( "Counter should have a successor.",
            counter.hasNext() );

assertEquals( LOWER_BOUND, actual );
assertEquals( "Number should be lower bound value.", 
              LOWER_BOUND,
              actual );

Assert#fail() throws an assertion error unconditionally. This can be helpful to mark an incomplete test or to ensure that an expected exception has been thrown (see also the Expected Exceptions section in Test Structure).
Assert#assertXXX(Object) is used to verify the initialization state of a variable. For this purpose there exists two methods called assertNull(Object) and assertNotNull(Object).
Assert#assertXXX(boolean) methods test expected conditions passed by the boolean parameter. Invocation of assertTrue(boolean) expects the condition to be true, whereas assertFalse(boolean) expects the opposite.
Assert#assertXXX(Object,Object) and Assert#assertXXX(value,value) methods are used for comparison verifications of values, objects and arrays. Although it makes no difference in result, it is common practice to pass the expected value as first parameter and the actual as second.

All these types of methods provide an overloaded version, that takes a String parameter. In case of a failure this argument gets incorporated in the assertion error message. Many people consider this helpful to specify the failure reason more clearly. Others perceive such messages as clutter, making tests harder to read.

This kind of unit test assertion seems to be intuitive upon first sight. Which is why I used it in the previous chapters for getting started. Besides it is still quite popular and tools support failure reporting well. However it is also somewhat limited with respect to the expressiveness of assertions that require more complex predicates.

Hamcrest

A library that aims to provide an API for creating flexible expressions of intent is Hamcrest. The utility offers nestable predicates called Matchers. These allow to write complex verification conditions in a way, many developers consider easier to read than boolean operator expressions.

Unit test assertion is supported by the class MatcherAssert. To do so it offers the static assertThat(T, Matcher) method. The first argument passed is the value or object to verify. The second is the predicate used to evaluate the first one.

assertThat( actual, equalTo( IN_RANGE_NUMBER ) );

As you can see, the matcher approach mimics the flow of a natural language to improve readability. The intention is even made more clear by the following snippet. This uses the is(Matcher) method to decorate the actual expression.

assertThat( actual, is( equalTo( IN_RANGE_NUMBER ) ) );

MatcherAssert.assertThat(...) exists with two more signatures. First, there is a variant that takes a boolean parameter instead of the the Matcher argument. Its behavior correlates to Assert.assertTrue(boolean).

The second variant passes an additional String to the method. This can be used to improve the expressiveness of failure messages:

assertThat( "Actual number must not be equals to lower bound value.", 
             actual, 
             is( not( equalTo( LOWER_BOUND ) ) ) );

In a case of failure the error message for the given verification would look somewhat like this:

Hamcrest comes with a set of useful matchers. The most important ones are listed in the tour of common matchers section of the library’s online documentation. But for domain specific problems readability of a unit test assertion could often be improved, if an appropriate matcher was available.

For that reason the library allows to write custom matchers.

Let us return to the tutorial‘s example for a discussion of this topic. First we adjust the scenario to be more reasonable for this chapter. Assume that NumberRangeCounter.next() returns the type RangeNumber instead of a simple int value:

public class RangeNumber {
  
  private final String rangeIdentifier;
  private final int value;

  RangeNumber( String rangeIdentifier, int value  ) {
    this.rangeIdentifier = rangeIdentifier;
    this.value = value;
  }
  
  public String getRangeIdentifier() {
    return rangeIdentifier;
  }
  
  public int getValue() {
    return value;
  }
}

We could use a custom matcher to check, that the return value of NumberRangeCounter#next() is within the counter’s defined number range:

RangeNumber actual = counter.next();

assertThat( actual, is( inRangeOf( LOWER_BOUND, RANGE ) ) );

An appropriate custom matcher could extend the abstract class TypeSafeMatcher<T>. This base class handles null checks and type safety. A possible implementation is shown below. Note how it adds the factory method inRangeOf(int,int) for convenient usage:

public class InRangeMatcher extends TypeSafeMatcher<RangeNumber> {

  private final int lowerBound;
  private final int upperBound;

  InRangeMatcher( int lowerBound, int range ) {
    this.lowerBound = lowerBound;
    this.upperBound = lowerBound + range;
  }
  
  @Override
  public void describeTo( Description description ) {
    String text = format( "between <%s> and <%s>.", lowerBound, upperBound );
    description.appendText( text );
  }
  
  @Override
  protected void describeMismatchSafely(
    RangeNumber item, Description description )
  {
    description.appendText( "was " ).appendValue( item.getValue() );
  }


  @Override
  protected boolean matchesSafely( RangeNumber toMatch ) {
    return    lowerBound <= toMatch.getValue() 
           && upperBound > toMatch.getValue();
  }
  
  public static Matcher<RangeNumber> inRangeOf( int lowerBound, int range ) {
    return new InRangeMatcher( lowerBound, range );
  }
}

The effort may be a bit exaggerated for the given example. But it shows how the custom matcher can be used to eliminate the somewhat magical IN_RANGE_NUMBER constant of the previous posts. Besides the new type enforces compile time type-safety of the assertion statement. This means e.g. a String parameter would not be accepted for verification.

The following picture shows how a failing test result would look like with our custom matcher:

It is is easy to see in which way the implementation of describeTo and describeMismatchSafely influences the failure message. It expresses that the expected value should have been between the specified lower bound and the (calculated) upper bound¹ and is followed by the actual value.

It is a little unfortunate, that JUnit expands the API of its Assert class to provide a set of assertThat(…) methods. These methods actually duplicate API provided by MatcherAssert. In fact the implementation of those methods delegate to the according methods of this type.

Although this might look as a minor issue, I think it is worth to mention. Due to this approach JUnit is firmly tied to the Hamcrest library. This dependency leads now and then to problems. In particular when used with other libraries, that do even worse by incorporating a copy of their own hamcrest version…

Unit test assertion à la Hamcrest is not without competition. While the discussion about one-assert-per-test vs. single-concept-per-test [MAR] is out of scope for this post, supporters of the latter opinion might perceive the library’s verification statements as too noisy. Especially when a concept needs more than one assertion.

Which is why I have to add another section to this chapter!

AssertJ

In the post Test Runners one of the example snippets uses two assertXXX statements. These verify, that an expected exception is an instance of IllegalArgumentException and provides a certain error message. The passage looks similar like this:

Throwable actual = ...

assertTrue( actual instanceof IllegalArgumentException );
assertEquals( EXPECTED_ERROR_MESSAGE, actual.getMessage() );

The previous section taught us how to improve the code using Hamcrest. But if you happen to be new to the library you may wonder, which expression to use. Or typing may feel a bit uncomfortable. At any rate the multiple assertThat statements would add up to the clutter.

The library AssertJ strives to improve this by providing fluent assertions for java. The intention of the fluent interface API is to provide an easy to read, expressive programming style, that reduces glue code and simplifies typing.

So how can this approach be used to refactor the code above?

import static org.assertj.core.api.Assertions.assertThat;

Similar to the other approaches AssertJ provides a utility class, that offers a set of static assertThat methods. But those methods return a particular assertion implementation for the given parameter type. This is the starting point for the so called statement chaining.

Throwable actual = ...

assertThat( actual )
  .isInstanceOf( IllegalArgumentException.class )
  .hasMessage( EXPECTED_ERROR_MESSAGE );

While readability is to some extend in the eye of the beholder, at any rate assertions can be written in a more compact style. See how the various verification aspects relevant for the specific concept under test are added fluently. This programming method supports efficient typing, as the IDE’s content assist can provide a list of the available predicates for a given value type.

So you want to provide an expressive failure messages to the after-world? One possibility is to use describedAs as first link in the chain to comment the whole block:

Throwable actual = ...

assertThat( actual )
  .describedAs( "Expected exception does not match specification." )
  .hasMessage( EXPECTED_ERROR_MESSAGE )
  .isInstanceOf( NullPointerException.class );

The snippet expects a NPE, but assume that an IAE is thrown at runtime. Then the failing test run would provide a message like this:

Maybe you want your message to be more nuanced according to a given failure reason. In this case you may add a describedAs statement before each verification specification:

Throwable actual = ...

assertThat( actual )
  .describedAs( "Message does not match specification." )
  .hasMessage( EXPECTED_ERROR_MESSAGE )
  .describedAs( "Exception type does not match specification." )
  .isInstanceOf( NullPointerException.class );

There are much more AssertJ capabilities to explore. But to keep this post in scope, please refer to the utility’s online documentation for more information. However before coming to the end let us have a look at the in-range verification example again. This is how it can be solved with a custom assertion:

public class RangeCounterAssertion
  extends AbstractAssert<RangeCounterAssertion, RangeCounter>
{

  private static final String ERR_IN_RANGE_OF 
    = "Expected value to be between <%s> and <%s>, but was <%s>";
  private static final String ERR_RANGE_ID 
    = "Expected range identifier to be <%s>, but was <%s>";
  
  public static RangeCounterAssertion assertThat( RangeCounter actual ) {
    return new RangeCounterAssertion( actual );
  }
  
  public InRangeAssertion hasRangeIdentifier( String expected ) {
    isNotNull();
    if( !actual.getRangeIdentifier().equals( expected ) ) {
      failWithMessage( ERR_RANGE_ID, expected, actual.getRangeIdentifier()  );
    }
    return this;
  }
  
  public RangeCounterAssertion isInRangeOf( int lowerBound, int range ) {
    isNotNull();
    int upperBound = lowerBound + range;
    if( !isInInterval( lowerBound, upperBound ) ) {
      int actualValue = actual.getValue();
      failWithMessage( ERR_IN_RANGE_OF, lowerBound, upperBound, actualValue );
    }
    return this;
  }

  private boolean isInInterval( int lowerBound, int upperBound ) {
    return actual.getValue() >= lowerBound 
        && actual.getValue() < upperBound;
  }

  private RangeCounterAssertion( Integer actual ) {
    super( actual, RangeCounterAssertion.class );
  }
}

It is common practice for custom assertions to extend AbstractAssert. The first generic parameter is the assertion’s type itself. It is needed for the fluent chaining style. The second is the type on which the assertion operates.

The implementation provides two additional verification methods, that can be chained as in the example below. Because of this the methods return the assertion instance itself. Note how the call of isNotNull() ensures that the actual RangeNumber we want to make assertions on is not null.

The custom assertion is incorporated by its factory method assertThat(RangeNumber). Since it inherits the available base checks, the assertion can verify quite complex specifications out of the box.

RangeNumber first = ...
RangeNumber second = ...

assertThat( first )
  .isInRangeOf( LOWER_BOUND, RANGE )
  .hasRangeIdentifier( EXPECTED_RANGE_ID )
  .isNotSameAs( second );

For completeness here is how the RangNumberAssertion looks in action:

Unfortunately it is not possible to use two different assertion types with static imports within the same test case. Assumed of course, that those types follow the assertThat(...) naming convention. To circumvent this the documentation recommends to extend the utility class Assertions.

Such an extension can be used to provide static assertThat methods as entry point to all of a project’s custom assertions. By using this custom utility class throughout the project no import conflicts can occur. A detailled description can be found in the section Providing a single entry point for all assertions : yours + AssertJ ones of the online documentation about custom assertions.

Another problem with the fluent API is that single-line chained statements may be more difficult to debug. That is because debuggers may not be able to set breakpoints within the chain. Furthermore it may not be clear which of the method calls may have caused an exception.

But as stated by Wikipedia on fluent interfaces, these issues can be overcome by breaking statements into multiple lines as show in the examples above. This way the user can set breakpoints within the chain and easily step through the code line by line.

Conclusion

This chapter of JUnit in a Nutshell introduced different unit test assertion approaches like the tool’s built-in mechanism, Hamcrest matchers and AssertJ assertions. It outlined some pros and cons and enlarged upon the subject by means of the tutorial’s ongoing example. Additionally it was shown how to create and use custom matchers and assertions.

While the Assert based mechanism surely is somewhat dated and less object-oriented, it still has it advocates. Hamcrest matchers provide a clean separation of assertion and predicate definition, whereas AssertJ assertions score with a compact and easy to use programming style. So now you are spoilt for choice…

Please regard that this will be the last chapter of my tutorial about JUnit testing essentials. Which does not mean that there is nothing more to say. Quite the contrary! But this would go beyond the scope this mini-series is tailored to. And you know what they are saying: always leave them wanting more…