Test Attribute #10 – Isolation

Gil ZilberfeldAugust 24th, 2014Last Updated: August 22nd, 2014

2 30 3 minutes read

This is last, final, and 10th entry in the ten commandments of test attributes that started here. And you should read all of them.

We usually talk about isolation in terms of mocking. Meaning, when we want to test our code, and the code has dependencies, we use mocking to fake those dependencies, and allow us to test the code in isolation.

That’s code isolation. But test isolation is different.

An isolated test can run alone, in a suite, in any order, independent from the other tests and give consistent results. We’ve already identified in footprint the different environment dependencies that can affect the result, and of course, the tested code has something to do with it.

Other tests can also create dependency, directly or not. In fact, sometimes we may be relying on the order of tests.

To give an example, I summon the witness for the prosecution: The Singleton.

Here’s some basic code using a singleton:

public class Counter
{
    private static Counter instance;
    private int count = 0;
    public static void Init()
    {
        instance = new Counter();
    }

    public static Counter GetInstance()
    {
        return instance;
    }

    public int GetValue()
    {
        return count++;
    }
}

Pretty simple: The static instance is initialized in a call to Init. We can write these tests:

[TestMethod]public void CounterInitialized_WorksInIsolation()
{
    Counter.Init();
    var result = Counter.GetInstance().GetValue();
    Assert.AreEqual(0, result);
}

[TestMethod]public void CounterNotInitialized_ThrowsInIsolation()
{
    var result = Counter.GetInstance().GetValue();
    Assert.AreEqual(1, result);
}

Note that the second passes when running after the first. But if you run it alone it crashes, because the instance is not initialized. Of course, that’s the kind of thing that gives singletons a bad name. And now you need to jump through hoops in order to check the second case.

By the way, we’re not just relying on the order of the tests – we’re relying on the way the test runner runs them. It could be in the order we’ve written them, but not necessarily.

While singletons mostly appear in the tested code, test dependency can occur because of the tests themselves. As long as you keep state in the test class, including mocking operations, there’s a chance that you’re depending on the order of the run.

Do you know this trick?

public class MyTests: BaseTest {
    ///...

Why not put all common code in a base class, then derive the test class from it?

Well, apart of making readabilty suffer, and debugging excruciating, we now have all kinds of test setup and behavior that are located in another shared place. It may be that the test itself does not suffer interference from other tests, but we’re introducing this risk by putting shared code in the base class. Plus, you’ll need to no more about initialization order. And what if the base class is using a singleton? Antics ensue.

Test isolation issues show themselves very easily, because once they are out of order (ha-ha), you’ll get the red light. The problem is identifying the problem, because it may seem like an “irreproducible problem”.

In order to avoid isolation problems:

Check the code. If you can identify patterns of usage like singelton, be aware of that and put it to use: either initialize the singleton before the whole run, or restart it before every test.
Rearrange. If there are additional dependencies (like our counter increase), start thinking about rearranging the tests. Because the way the code is written, you’re starting to test more than just small operations.
Don’t inherit. Test base classes create interdependence and hurt isolation.
Mocking. Use mocking to control any shared dependency.
Clean up. Make sure that tests clean up after themselves. Or, instead before every run.

Isolation issues in tests are very annoying, because especially in unit tests, they can be easily avoided. Know the code, understand the dependencies, and never rely on another test to set up the state needed for the current one.

Reference:

Test Attribute #10 – Isolation from our JCG partner Gil Zilberfeld at the Geek Out of Water blog.