JUnit in a Nutshell: Test Runners

Frank AppelSeptember 5th, 2014Last Updated: September 4th, 2014

0 41 7 minutes read

The fourth chapter of my multi-part tutorial about JUnit testing essentials explains the purpose of the tool’s exchangable test runners architecture and introduces some of the available implementations. The ongoing example enlarges upon the subject by going through the different possibilities of writting parameterized tests.

Since I have already published an introduction to JUnit Rules, I decided to skip the announced sections on that topic. Instead I spend the latter a minor update.

Test Runners Architecture

Don’t be afraid to give up the good to go for the great.
John D. Rockefeller

In the previous posts we have learned to use some of the xUnit testing patterns [MES] with JUnit. Those concepts are well supported by the default behavior of the tool’s runtime. But sometimes there is a need to vary or supplement the latter for particular test types or objectives.

Consider for example integration tests, that often need to be run in specific environments. Or imagine a set of test cases comprising the specification of a subsystem, which should be composed for common test execution.

JUnit supports the usage of various types of test processors for this purpose. Thus it delegates at runtime test class instantiation, test execution and result reporting to such processors, which have to be sub types of org.junit.Runner.

A test case can specify its expected runner type with the @RunWith annotation. If no type is specified the runtime chooses BlockJUnit4ClassRunner as default. Which is responsible that each test runs with a fresh test instance and invokes lifecycle methods like implicit setup or teardown handlers (see also the chapter about Test Structure).

@RunWith( FooRunner.class )
public class BarTest {

The code snippet shows how the imaginary FooRunner is specified as test processor for the also imaginary BarTest.

Usually there is no need to write custom test runners. But in case you have to, Michael Scharhag has written a good explanation of the JUnit’s runner architecture recently.

It seems that usage of special test runners is straight forward, so let us have a look at a few:

Suite and Categories

Probably one of the best known processors is the Suite. It allows to run collections of tests and/or other suites in a hierarchically or thematically structured way. Note that the specifying class itself has usually no body implementation . It is annotated with a list of test classes, that get executed by running the suite:

@RunWith(Suite.class)
@SuiteClasses( { 
  NumberRangeCounterTest.class,
  // list of test cases and other suites
} )
public class AllUnitTests {}

However the structuring capabilities of suites are somewhat limited. Because of this JUnit 4.8 introduced the lesser known Categories concept. This makes it possible to define custom category types like unit-, integration- and acceptance tests for example. To assign a test case or a method to one of those categories the Category annotation is provided:

// definition of the available categories
public interface Unit {}
public interface Integration {}
public interface Acceptance {}

// category assignment of a test case
@Category(Unit.class)
public class NumberRangeCounterTest {
  [...]
}

// suite definition that runs tests
// of the category 'Unit' only
@RunWith(Categories.class)
@IncludeCategory(Unit.class)
@SuiteClasses( { 
  NumberRangeCounterTest.class,
  // list of test cases and other suites
} )
public class AllUnitTests {}

With Categories annotated classes define suites that run only those tests of the class list, that match the specified categories. Specification is done via include and/or exclude annotations. Note that categories can be used in Maven or Gradle builds without defining particular suite classes (see the Categories section of the JUnit documentation).

For more information on categories: John Ferguson Smart’s has written a detailled explanation about Grouping tests using JUnit categories.

Since maintenance of the suite class list and category annotations is often considered somewhat tedious, you might prefer categorising via test postfix names à la FooUnitTest instead of FooTest. This allows to filter categories on type-scope at runtime.

But this filtering is not supported by JUnit itself, why you may need a special runner that collects the available matching tests dynamically. A library that provides an appropriate implementation is Johannes Link‘s ClasspathSuite. If you happen to work with integration tests in OSGi environment Rüdiger‘s BundleTestSuite does something similar for bundles.

After this first impressions of how test runners can be used for test bundling let us continue the tutorial’s example with something more exciting.

Parameterized Tests

The example used throughout this tutorial is about writing a simple number range counter, which delivers a certain amount of consecutive integers, starting from a given value. Additionally a counter depends on a storage type for preserving its current state. For more information please refer to the previous chapters.

Now assume that our NumberRangeCounter, which is initialized by constructor parameters, should be provided as API. So we may consider it reasonable, that instance creation checks the validity of the given parameters.

We could specify the appropriate corner cases, which should be acknowledged with IllegalArgumentExceptions, by a single test each. Using the Clean JUnit Throwable-Tests with Java 8 Lambdas approach, such a test verifying that the storage parameter must not be null might look like this:

@Test
  public void testConstructorWithNullAsStorage() {
    Throwable actual = thrown( () -> new NumberRangeCounter( null, 0, 0 ) );
    
    assertTrue( actual instanceof IllegalArgumentException );
    assertEquals( NumberRangeCounter.ERR_PARAM_STORAGE_MISSING,
                  actual.getMessage() );
  }

Note that I stick with the JUnit build-in functionality for verification. I will cover the pro and cons of particular matcher libraries (Hamcrest, AssertJ) in a separate post.

To keep the post in scope I also skip the discussion, whether a NPE would be better than the IAE.

In case we have to cover a lot of corner cases of that kind, the approach above might lead to a lot of very similar tests. JUnit offers the Parameterized runner implementation to reduce such redundancy. The idea is to provide various data records for the common test structure.

To do so a public static method annotated with @Parameters is used to create the data records as a collection of object arrays. Furthermore the test case needs a public constructor with arguments, that match the data types provided by the records.

The parameterized processor runs a given test for each record supplied by the parameters method. This means for each combination of test and record a new instance of the test class is created. The constructor parameters get stored as fields and can be accessed by the tests for setup, exercise and verification:

@RunWith( Parameterized.class )
public class NumberRangeCounterTest {
  
  private final String message;
  private final CounterStorage storage;
  private final int lowerBound;
  private final int range;
  
  @Parameters
  public static Collection<Object[]> data() {
    CounterStorage dummy = mock( CounterStorage.class );
    return Arrays.asList( new Object[][] { 
      { NumberRangeCounter.ERR_PARAM_STORAGE_MISSING, null, 0, 0 }, 
      { NumberRangeCounter.ERR_LOWER_BOUND_NEGATIVE, dummy, -1, 0 },
       [...] // further data goes here... 
    } );
  }
  
  public NumberRangeCounterTest(
    String message, CounterStorage storage, int lowerBound, int range )
  {
    this.message = message;
    this.storage = storage;
    this.lowerBound = lowerBound;
    this.range = range;
  }
  
  @Test
  public void testConstructorParamValidation() {
    Throwable actual = thrown( () -> 
      new NumberRangeCounter( storage, lowerBound, range ) );
    
    assertTrue( actual instanceof IllegalArgumentException );
    assertEquals( message, actual.getMessage() );
  }

  [...]
}

While the example surely reduces test redundancy it is at least debatable with respect to readability. In the end this often depends on the amount of tests and the structure of the particular test data. But it is definitively unfortunate, that tests, which do not use any record values, will be executed multiple times, too.

Because of this parameterized tests are often kept in separate test cases, which usually feels more like a workaround than a proper solution. Hence a wise guy came up with the idea to provide a test processor that circumvents the described problems.

JUnitParams

The library JUnitParams provides the types JUnitParamsRunner and @Parameter. The param annotation specifies the data records for a given test. Note the difference to the JUnit annotation with the same simple name. The latter marks a method that provides the data records!

The test scenario above could be rewritten with JUnitParams as shown in the following snippet:

@RunWith( JUnitParamsRunner.class )
public class NumberRangeCounterTest {
  
  public static Object data() {
    CounterStorage dummy = mock( CounterStorage.class );
    return $( $( ERR_PARAM_STORAGE_MISSING, null, 0, 0 ),
              $( ERR_LOWER_BOUND_NEGATIVE, dummy, -1, 0 ) );  
  }
  
  @Test
  @Parameters( method = "data" )
  public void testConstructorParamValidation(
    String message, CounterStorage storage, int lowerBound, int range ) 
  {
    Throwable actual = thrown( () -> 
      new NumberRangeCounter( storage, lowerBound, range ) );
    
    assertTrue( actual instanceof IllegalArgumentException );
    assertEquals( message, actual.getMessage() );
  }
  
  [...]
}

While this is certainly more compact and looks cleaner on first glance, a few constructs need further explanation. The $(...) method is defined in JUnitParamsRunner (static import) and is a shortcut for creating arrays of objects. Once accustomed to it, data definition gets more readable.

The $ shortcut is used in the method data to create a nested array of objects as return value. Although the runner expects a nested data array at runtime, it is able to handle a simple object type as return value.

The test itself has an additional @Parameters annotation. The annotation’s method declaration refers to the data provider used to supply the test with the declared parameters. The method name is resolved at runtime via reflection. This is the down-side of the solution, as it is not compile-time safe.

But there are other use case scenarios where you can specify data provider classes or implicit values, which therefore do not suffer from that trade-off. For more information please have a look at the library’s quick start guide for example.

Another huge advantage is, that now only those tests run against data records that use the @Parameters annotation. Standard tests are executed only once. This in turn means that the parameterized tests can be kept in the unit’s default test case.

Wrap Up

The sections above outlined the sense and purpose of JUnit’s exchangable test runners architecture. It introduced suite and categories to show the basic usage and carried on with an example of how test runners can ease the task of writing data record related tests.

For a list of additional test runners the pages Test runners and Custom Runners at junit.org might be a good starting point. And if you wonder what the Theories runner of the title picture is all about, you might have a look at Florian Waibels post JUnit – the Difference between Practice and @Theory.

Next time on JUnit in a Nutshell I finally will cover the various types of assertions available to verify test results.