Why Most Unit Testing is Waste???

A rebuttal of points raised in this article by James O Coplien.

It’s worth noting that James O Coplien is a well respected father of modern software engineering, so it’s odd to be writing such a piece.

Write the wrong sort of code, and the wrong sorts of tests will annoy you.

The myth about good code structure

The article starts with an incorrect statement about how code used to be well structured, and that you’d trace the lower functions from the business case.

I’d contest that building implementation directly from business requirements does not naturally lead to good code structures. A solution to a problem is often based around abstractions and simplifications, or powerful patterns that can be applied to a business problem.

Similarly, when we apply approaches like DRYing out our software, or reducing cyclomatic complexity and, mainly, function size, then our lower level software becomes abstract building blocks. Blocks of single responsibility are provably very good.

These structures need only come into existence when the business problem we’re solving requires them. Critically, there should be automated tests that reflect the business cases. This is where BDD helps.

Object Oriented Programming Loses the Context

The idea that the various object oriented techniques lose us the ability to statically analyse the software is both true and false.

It’s true that a series of modules may need to be executed to understand their dynamic behaviour. It’s probably untrue to believe that any non-trivial alternative approach is any easier to follow. I’m presently working with some monolithic ASP pages. It’s all there in the page, with limited modularisation. It’s harder than well formed, well named components brought together.

The case against polymorphism, implied by the article, is terrifyingly backwards.

However, perhaps the point being made here is that unit testing becomes necessary to assemble loosely coupled modules together in a way that’s less relevant with older procedural programming. Who knows what we’d do with request and global state in the latter at test time, though.

Unit Tests are Unlikely to Test More Than One Trillionth of the Functionality of any Given Method

This is nonsense.

It’s true that there are some methods you can write that are so innately complex that you have no chance of trying out every permutation of pathways through them.

It’s also true that proving that all pathways are followed does not entirely equate to proof that the method works bug free.

Test driven development takes the approach that we write a test case first, and this creates the need for well formed software to achieve that outcome. We do it in small increments, refactoring as we go, and we prefer small single responsibility, single level of abstraction methods. When you add all these up, the unit tests we write for the lowest level functions enable us to exhaustively test them, especially because the exact boundaries we need are the thing we think of first with TDD.

Adding test automation AFTER the fact, to code structures which are naturally less manageable, will behave as predicted, and waste time.

So don’t.

Smaller Functions Don’t Encapsulate Algorithms

The argument that you shouldn’t break a large function down into smaller ones doesn’t end well.

If you fracture code randomly while refactoring down from larger things to smaller things, then it ends badly. I’m yet to find monolithic code which looks easier to reason about that its equivalent reasonably refactored alternative.

I’ve seen code over-refactored, and I’ve seen some patterns that increase the number of awkward boundaries possible in an algorithm.

But the argument that you can’t manage a broken down algorithm, and that you’re just gaming the tests is backwards.

What is Good Coverage?

It’s a myth to assume that we’re aiming for 100% coverage. That said, I tend to achieve high coverage.

High coverage is a relatively weak metric of code quality.

However, low coverage – i.e. less than 80% – is a very strong metric. It implies a few things:

• The developers don’t care about testing
• We’re probably not doing TDD
• We have higher cyclomatic complexity
• We may be writing code that’s redundant
• We’re using coding patterns that come with extraneous edge cases
• We have boilerplate bloat without needing it

The waste that the article speaks of refers to what happens when developers cargo cult the process of software testing. The purpose of TDD is to drive features, quality and design into the software. The idea that someone has to use this function makes us see it differently and often produce it better. We have to suffer the indignity of using our own software, do we produce something better rounded.

Of course high coverage requirements can lead to some seemingly wasteful practices. To make sonar happy, we occasionally add something that doesn’t seem to add much value…

But every test we write is a stake in the ground. It pins some functionality or behaviour down and gives us early warning if our assumptions stop being met in the future.

Cut the Unit Tests and Go For More Integration Testing

The well known test pyramid begs to differ here.

Coplien argues that too much worry around unit testing may come from a lack of integration, and that you can measure the ratio of unit test code to actual code to determine the fear factor. He argues you should cut the unit testing and integrate more.

Integration tests require a complex universe to set up to start with, and then a complex analysis of outcomes to complete them. Usually they require more steps, and they’re more brittle as systems change.

Why is this better?

The idea that there might be too many lines of unit test code is an interesting one, though. On the one hand, this is part of the challenge of writing good unit tests. See the Test Smells list for a lot more on this. There’s a dilemma. You do invest time and lines of code into the construction of test automation – you’re doing that to document and nail down the behaviour of the system… but then it sits there.

But running tests should be quick and cheap and should help debugging. It also removes the need to wait for the application to start to do 90% of the things we need to do to believe the software is likely to work.

If your tests have no relevance to the likelihood of the system working, then you’re doing something wrong.

Throw Away Tests That Never Fail

This is almost a good idea. If a test doesn’t fail, then perhaps it’s testing nothing important. It might be testing some boilerplate, or an area of the code that’s seldom visited…

But the cost of running a test is essentially 0. Test suites are slowed down more by writing the wrong sorts of tests than writing lots of small simple ones.

We should throw away tests that:

• Test implementation rather than behaviour
• Duplicate other tests
• Are impossible to understand – we should replace them with clearer tests as we refactor

Keeping Tests Up To Date Reduces Velocity

In TDD this makes no sense. We always write tests first, and this implies keeping them up to date.

But a feature we add may contradict an existing test. That’s good. We can then update that test – perhaps not add a new one, or at least not start afresh with a new one.

If all our functionality is in a mass of glue in some hotspot, then any small change there will imply a lot of damaged test blast radius.

If software change has a huge blast radius, then your software design is poor.

The same is also true for test structures. Testing the implementation, or very implementation-dependent tests, often found in UI testing, also turns out to be a case of the code structure not being optimal for velocity.

The open closed principle probably helps us here.

As does abstraction.

Tracing Tests Back to Business Requirements

If this unit test fails, what business requirement can’t we meet?

Great point. It sounds to me like the unit test is a likely harbinger of an integration test that might also fail… or maybe we’re in the category of boundary conditions that are hard to contrive in any form of testing, but easy to manage in a unit test.

There’s huge value in this. We cannot achieve the permutational complexity to manage everything from the system-sized black box testing, but we can easily do it down at the unit test.

That we can’t relate each unit test failure to a business requirement is not necessarily an issue. The components of the system should be meaningful in their own right. If not, then we have badly designed code, not a testing problem.

Unit Tests are Assertions in Disguise

It used to be the case that you’d assert the production code as you went along. If an assertion failed, the production code would crash and you could get a bug report from the logs.

Some companies insist on assertion like things at runtime. For example:

void myFunction(Input one, Input two) {
   Objects.requireNonNull(one);
   Objects.requireNonNull(two);

   // now proceed knowing there'll be no unexplained 
   // null pointer exceptions
}

I didn’t really enjoy using the above structures, because it felt like code bloat, but it does something useful. It brings assumptions/errors about runtime screw ups to the very front, so they fail in a useful way before doing damage.

I resolved my discomfort with the above by knowing that I could ensure these failures were more likely to happen at unit test time, so I could fix them, rather than at actual runtime.

There’s limited value in waiting for a bug report, when we can meaningfully explore our code as we write it, to most drive bugs out before we even start the application.

Create System Tests with Feature Not Code Coverage

I agree with this. This is where ATDD or BDD can help. Feature coverage, impossible to measure, is the real metric of test coverage.

Debugging is Not Testing

Agreed… almost.

A consultant I met a while back boasted of never using the debugger, because they just wrote tests and the tests found the issue.

I always feel bad running a debugger these days, as it’s an admission that I can’t think of a unit test that would drive a stake into the issue that’s going wrong.

I feel worse when I’m debugging the app as it runs. Most of my time running a debugger is spend debugging a test – a smaller quicker thing to run – and the outcome is usually to write a test I’d missed from a particular edge case, and then make the necessary change to get all tests green again.

We Are Trying to Get The Computer to Think

The author is suggesting people adopt a test approach of:

• Believe your tests are right because they’re more thorough
• Then just hack code until it goes green, failing fast and often, and experimenting until the computer tells you you’re right

In some cases, trivial ones, I’d argue that this is exactly the method. I don’t really want to waste too much time agonising over operator precedence or where something is a plus or a minus in a calculation. I’d rather get this stuff tuned empirically against meaningful tests that make me think about my goals, not any particular implementation.

However, the best advice for any developer is the same as Merlin gives to King Arthur in the musical Camelot. “Arthur: don’t forget to think!”.

Conclusion

If you write code and tests badly. If you set up a war between a tester and a developer, fought over red/green test automation. If you fail to maintain tests, or write the wrong sort of test, then your tests will feel like a waste, because they’re not, themselves, functioning as either test or user feature.

If you do things as well as you can, driving good design and features into a system, and keeping them there with well thought out tests, over well thought out implementation, then it’ll speed you up overall.

There are some minor test rituals that we may do to appease coverage gods, but they pay off in other ways.

There’s also a cost to maintaining tests, especially flickering ones.

However testing allows you to manage complexity by driving good practices into your software.

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design

and many more ....

I agree to the Terms and Privacy Policy

Sign up

Thank you!

We will contact you soon.