Remember the days when bugs were just part of life? I mean a big part.
If you recall those dark ages, we’ve spent most of our time fixing bugs and introducing others. In our spare time, we got to work on new stuff.
Today is different, there are a lot less bugs. So we get to work more on the new stuff, right?
Or have we found a new replacement for them?
Test Maintenance Is the New Bug
You wake up one morning, and you discover that you spend 25% on maintaining tests. True story.
This doesn’t happen in one day, you feel more work on tests creeping up, and you feel it’s just natural. Part of the work.
Then one day you actually look at the numbers. And someone needs to pick you off the floor.
Nobody Writes Bad Tests Intentionally
So, where do these tests come from?
These tests are the creaky windows in the attic. One gust of wind, and you hear a loud bang. You need to go up there and fix them. Again.
Much like those windows, they don’t give you much protection. Since they break once in a while, you get accustomed to them creaking and banging, you don’t notice them anymore. Which is bad, since sometimes they let some real menaces into the house. Like real bugs. But you won’t know because you’re sure it’s the ol’ window moving in its place.
So until you face reality, it’s not clear how much they cost us. It could amount to a lot.
Back to Test World
Let’s call these bad tests the “wasteful” tests.
Which begs the question: If those cause waste, when are tests valuable? What exactly is this value?
This is hard to quantify. A test is valuable if it catches a bug. It gets points for that. So, ridiculous as it may sound, the time spent on fixing bugs the tests found is an indicator to the value of the test. We can call those tests the “valuable” tests.
Now, here’s the Test Waste ratio:
The denominator is the total time spent due to tests breaking, the valuable and wasteful. The numerator is the wasteful work.
If your wasted time is large, the ratio gets closer to 1. If it’s small, it gets closer to zero.
You want to get a lower ratio as possible. It’s not really up to you, but it’s an indicator. And those numbers can actually be measured.
So Much For Theory
What do you think? Does it make sense? Is it helpful in any way? Do I stand a chance for a Nobel prize?
Let me know.
PS. Next week, I’m going to be at Belgium Testing Days. Apart from the Spaceship I’m smuggling through customs, I’m going to be part of an interactive session, talking about the ROI of test automation.
I would argue that tests can be useful in other, more subtle, ways then catching bugs. * Documentation of behavior, which should save time for the next developer to understand the system * A good test coverage gives the Courage to refactor code to make it Clean. Clean code should be easier to understand and improve further. * Tests should drive the design to a more testable state. A testable system is easier to run in isolation. It should lead to a system that is easier to investigate. This is all secondary effects to having the system under test, but… Read more »