We recently learned about JUnit’s new extension model and how it allows us to inject customized behavior into the test engine. I left you with the promise to look at conditions. Let’s do that now!
Conditions allow us to define flexible criteria when tests should or shouldn’t be executed. Their official name is Conditional Test Execution.
Overview
Other posts in this series about JUnit 5:
Most of what you will read here and more can be found in the emerging JUnit 5 user guide. Note that it is based on an alpha version and hence subject to change.
Indeed, we are encouraged to open issues or pull requests so that JUnit 5 can improve further. Please make use of this opportunity! It is our chance to help JUnit help us, so if something you see here could be improved, make sure to take it upstream.
This post will get updated when it becomes necessary. The code samples I show here can be found on GitHub.
Extension Points For Conditions
Remember what we said about extension points? No? In short: There’s a bunch of them and each relates to a specific interface. Implementations of these interfaces can be handed to JUnit (with the @ExtendWith annotation) and it will call them at the appropriate time.
For conditions, two extension points are of interest: ContainerExecutionCondition and TestExecutionCondition.
public interface ContainerExecutionCondition extends Extension {
/**
* Evaluate this condition for the supplied ContainerExtensionContext.
*
* An enabled result indicates that the container should be executed;
* whereas, a disabled result indicates that the container should not
* be executed.
*
* @param context the current ContainerExtensionContext
*/
ConditionEvaluationResult evaluate(ContainerExtensionContext context);
}
public interface TestExecutionCondition extends Extension {
/**
* Evaluate this condition for the supplied TestExtensionContext.
*
* An enabled result indicates that the test should be executed;
* whereas, a disabled result indicates that the test should not
* be executed.
*
* @param context the current TestExtensionContext
*/
ConditionEvaluationResult evaluate(TestExtensionContext context);
}A ContainerExecutionCondition determines whether the tests in a container are executed or not. In the usual scenario with annotated test methods, the test class would be the container. In the same scenario, individual test method execution is determined by TestExecutionConditions.
(I’m saying “in the usual scenario” because different test engines might have very different interpretations of containers and tests. Classes and methods are just the most common ones.)
And that’s already pretty much it. Any condition should implement one or both of these interfaces and do the required checks in its evaluate implementation(s).
@Disabled
The easiest condition is one that is not even evaluated: We simply always disable the test if our hand-crafted annotation is present.
So let’s create @Disabled:
@Target({ ElementType.TYPE, ElementType.METHOD })
@Retention(RetentionPolicy.RUNTIME)
@ExtendWith(@DisabledCondition.class)
public @interface Disabled { }And the matching extension:
public class DisabledCondition
implements ContainerExecutionCondition, TestExecutionCondition {
private static final ConditionEvaluationResult ENABLED =
ConditionEvaluationResult.enabled("@Disabled is not present");
@Override
public ConditionEvaluationResult evaluate(
ContainerExtensionContext context) {
return evaluateIfAnnotated(context.getElement());
}
@Override
public ConditionEvaluationResult evaluate(
TestExtensionContext context) {
return evaluateIfAnnotated(context.getElement());
}
private ConditionEvaluationResult evaluateIfAnnotated(
AnnotatedElement element) {
Optional<Disabled> disabled = AnnotationUtils
.findAnnotation(element, Disabled.class);
if (disabled.isPresent())
return ConditionEvaluationResult
.disabled(element + " is @Disabled");
return ENABLED;
}
}Easy as pie, right? And correct, too, because it is almost the same as the real @Disabled implementation. There are only two small differences:
- The official annotation does not need to carry its own extension with it because it is registered by default.
- It can be given a reason, which is logged when the disabled test is skipped.
Small caveat (of course there’s one, what did you think?): AnnotationUtils is internal API but it is likely that its functionality will be officially available soon.
Now let’s try something less trivial.
@DisabledOnOs
Maybe we only want to run some tests if we are on the right operating system.
Simple Solution
Again, we start with the annotation:
@Target({ ElementType.TYPE, ElementType.METHOD })
@Retention(RetentionPolicy.RUNTIME)
@ExtendWith(OsCondition.class)
public @interface DisabledOnOs {
OS[] value() default {};
}This time it takes a value, or rather a bunch if values, namely the operating systems on which the test should not run. OS is just an enum with a value for each operating system. And it has a handy static OS determine() method, which, you guessed it, determines the operating system the code is running on.
With that, let’s turn to OsCondition. It has to check whether the annotation is present but also whether the current OS is one of those given to the annotation.
public class OsCondition
implements ContainerExecutionCondition, TestExecutionCondition {
// both `evaluate` methods forward to `evaluateIfAnnotated` as above
private ConditionEvaluationResult evaluateIfAnnotated(
AnnotatedElement element) {
Optional<DisabledOnOs> disabled = AnnotationUtils
.findAnnotation(element, DisabledOnOs.class);
if (disabled.isPresent())
return disabledIfOn(disabled.get().value());
return ENABLED;
}
private ConditionEvaluationResult disabledIfOn(OS[] disabledOnOs) {
OS os = OS.determine();
if (Arrays.asList(disabledOnOs).contains(os))
return ConditionEvaluationResult
.disabled("Test is disabled on " + os + ".");
else
return ConditionEvaluationResult
.enabled("Test is not disabled on " + os + ".");
}
}We can use it as follows:
@Test
@DisabledOnOs(OS.WINDOWS)
void doesNotRunOnWindows() {
assertTrue(false);
}Nice.
Less Ceremony
But we can do even better! Thanks to JUnit’s customizable annotations we can make this condition even smoother:
@TestExceptOnOs(OS.WINDOWS)
void doesNotRunOnWindowsEither() {
assertTrue(false);
}To implement @TestExceptOnOs, it would be great to just do this:
@Retention(RetentionPolicy.RUNTIME)
@Test
@DisabledOnOs(/* somehow get the `value` below */)
public @interface TestExceptOnOs {
OS[] value() default {};
}When executing a test and scanning for @DisabledOnOs in OsCondition::evaluateIfAnnotated, we would find it meta-annotated on @TestExceptOnOs and our logic would Just Work™. But I couldn’t find a way to make the OS values given to @TestExceptOnOs accessible to @DisabledOnOs. :( (Can you?)
The next best option is to simply use the same extension for the new annotation:
@Retention(RetentionPolicy.RUNTIME)
@ExtendWith(OsCondition.class)
@Test
public @interface TestExceptOnOs {
OS[] value() default {};
}Then we pimp OsCondition::evaluateIfAnnotated to include the new case…
private ConditionEvaluationResult evaluateIfAnnotated(
AnnotatedElement element) {
Optional<DisabledOnOs> disabled = AnnotationUtils
.findAnnotation(element, DisabledOnOs.class);
if (disabled.isPresent())
return disabledIfOn(disabled.get().value());
Optional<TestExceptOnOs> testExcept = AnnotationUtils
.findAnnotation(element, TestExceptOnOs.class);
if (testExcept.isPresent())
return disabledIfOn(testExcept.get().value());
return ConditionEvaluationResult.enabled("");
}… and we’re done. Now we can indeed use it as we hoped we could.
Polishing
Creating the inverted annotations (disabling if not on one of the specified operating systems) is just more of the same but with them, improved names, and static imports we could end up here:
@TestOn(WINDOWS)
void doesNotRunOnWindowsEither() {
assertTrue(false);
}Not bad, eh?
@DisabledIfTestFails
Let’s try one more thing – and this time we’ll make it really interesting! Assume there are a bunch of (integration?) tests and if one of them fails with a specific exception, other tests are bound to fail as well. So to save time, we’d like to disable them.
So what do we need here? Right off the bat it’s clear that we have to somehow collect the exceptions thrown during test execution. This has to be bound to the lifetime of the test class so we don’t disable tests because some exception flew in a totally different test class. And then we need a condition implementation that checks whether a specific exception was thrown and disables the test if so.
Collect Exceptions
Looking over the list of extension points we find “Exception Handling”. The corresponding interface looks promising:
/**
* ExceptionHandlerExtensionPoint defines the API for Extension Extensions
* that wish to react to thrown exceptions in tests.
*
* [...]
*/
public interface ExceptionHandlerExtensionPoint extends ExtensionPoint {
/**
* React to a throwable which has been thrown by a test method.
*
* Implementors have to decide if they
*
* - Rethrow the incoming throwable
* - Throw a newly constructed Exception or Throwable
* - Swallow the incoming throwable
*
* [...]
*/
void handleException(TestExtensionContext context, Throwable throwable)
throws Throwable;
}So we’ll implement handleException to store and then rethrow the exception.
You may remember what I wrote about extensions and state:
The engine makes no guarantees when it instantiates extension and how long it keeps instances around, so they have to be stateless. Any state they need to maintain has to be written to and loaded from a store that is made available by JUnit.
Ok, so we use the store; effectively a keyed collection of things we want to remember. We can access it via the extension context that is handed to most extensions’ methods. A little tinkering revealed that each context has its own store so we have to decide which one to access.
There is one context per test method ( TestExtensionContext) and for the whole test class ( ContainerExtensionContext). Remember that we want to store all exceptions thrown during the execution of all tests in a class but not more, i.e. not the ones thrown by other test classes. Turns out that the ContainerExtensionContext and its store are exactly what we need.
So here we go getting the container context and using it to store a set of thrown exceptions:
private static final Namespace NAMESPACE = Namespace
.of("org", "codefx", "CollectExceptions");
private static final String THROWN_EXCEPTIONS_KEY = "THROWN_EXCEPTIONS_KEY";
@SuppressWarnings("unchecked")
private static Set<Exception> getThrown(ExtensionContext context) {
ExtensionContext containerContext = getAncestorContainerContext(context)
.orElseThrow(IllegalStateException::new);
return (Set<Exception>) containerContext
.getStore(NAMESPACE)
.getOrComputeIfAbsent(
THROWN_EXCEPTIONS_KEY,
ignoredKey -> new HashSet<>());
}
private static Optional<ExtensionContext> getAncestorContainerContext(
ExtensionContext context) {
Optional<ExtensionContext> containerContext = Optional.of(context);
while (containerContext.isPresent()
&& !(containerContext.get() instanceof ContainerExtensionContext))
containerContext = containerContext.get().getParent();
return containerContext;
}Now adding an exception is simple:
@Override
public void handleException(TestExtensionContext context, Throwable throwable)
throws Throwable {
if (throwable instanceof Exception)
getThrown(context).add((Exception) throwable);
throw throwable;
}This is actually an interesting extension of its own. Maybe it could be used for analytics as well. Anyways, we will want to have a look the thrown exceptions so we need a public method for that:
public static Stream<Exception> getThrownExceptions(
ExtensionContext context) {
return getThrown(context).stream();
}With this any other extension can check which exceptions have been thrown so far.
Disable
The rest is much like before so let’s be quick about it:
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@ExtendWith(DisabledIfTestFailedCondition.class)
public @interface DisabledIfTestFailedWith {
Class<? extends Exception>[] value() default {};
}Note that we only allow this annotation on methods. Using it on test classes could make sense but let’s keep it simple for now. Accordingly we only implement TestExecutionCondition. After checking whether our annotation is present we call disableIfExceptionWasThrown with the user provided exception classes:
private ConditionEvaluationResult disableIfExceptionWasThrown(
TestExtensionContext context,
Class<? extends Exception>[] exceptions) {
return Arrays.stream(exceptions)
.filter(ex -> wasThrown(context, ex))
.findAny()
.map(thrown -> ConditionEvaluationResult.disabled(
thrown.getSimpleName() + " was thrown."))
.orElseGet(() -> ConditionEvaluationResult.enabled(""));
}
private static boolean wasThrown(
TestExtensionContext context, Class<? extends Exception> exception) {
return CollectExceptionExtension.getThrownExceptions(context)
.map(Object::getClass)
.anyMatch(exception::isAssignableFrom);
}Putting It Together
And this is how we use those annotations to disable tests if an exception of a specific type was thrown before:
@CollectExceptions
class DisabledIfFailsTest {
private static boolean failedFirst = false;
@Test
void throwException() {
System.out.println("I failed!");
failedFirst = true;
throw new RuntimeException();
}
@Test
@DisabledIfTestFailedWith(RuntimeException.class)
void disableIfOtherFailedFirst() {
System.out.println("Nobody failed yet! (Right?)");
assertFalse(failedFirst);
}
}Summary
Wow, that was a lot of code! But by now we really know how to implement conditions in JUnit 5:
- create the desired annotation and @ExtendWith your condition implementation
- implement ContainerExecutionCondition, TestExecutionCondition, or both
- check whether the new annotation is even present
- perform the actual checks and return the result
We have also seen that this can be combined with other extension points, how the store can be used to persist information, and that custom annotations can make using an extension much more elegant.
For more fun with flags extension points, check the next post in this series when we’ll be discussing parameter injection.
| Reference: | JUnit 5 – Conditions from our JCG partner Nicolai Parlog at the CodeFx blog. |
