Overreaching Unit Testing
Steven Sanderson has written “the benefit of unit testing is correlated with the non-obviousness of the code under test.” I largely agree with this sentiment as a general guideline. I see little value in unit testing trivial “get” and “set” methods. Some methods are more readily evaluated via code review than via unit test.
The concept of code coverage can be a useful one as long as it’s not taken too far. Code coverage appears to provide high return for the effort for a while, but there comes a point of diminishing returns when gaining additional code coverage comes at much greater cost and may not be worth that cost. It’s also important to recognize that even the often highly expensive 100% code coverage typically means only all lines of code were executed and does not check all possible paths through the code.
All Code’s Unit Testability is Not Equal
The post Selective Unit Testing – Costs and Benefits clearly articulates well the differences in difficulty (cost) and advantages (benefits) of unit testing of different types of code. In cases where the advantages/benefits of a unit test are high and the cost/effort is low, the value of unit testing is obvious. On the opposite extreme, there are types of code that receive little benefit from unit testing.
There Are Other Effective Types of Testing
I have seen multiple disparate groups of developers build and unit test their respective code bases flawlessly and then suffer through the pain of trying to integrate their thoroughly unit tested code with the other groups’ thoroughly unit tested code. This happens when the involved groups have written comprehensive unit tests based on their own assumptions and without regard for functional and integration tests. A certain minimum number of unit tests are always necessary, but there may be cases where less important unit tests should not displace better or more comprehensive integration tests. When looking at code coverage, I prefer to look at code coverage from all types of tests rather than simply code coverage provided via unit tests.
In his insightful column Breaking Away From The Unit Test Group Think, Cedric Beust (created TestNG and wrote Next Generation Java Testing) wrote:
I also question all the attention that unit tests are receiving at the expense of the other kinds of tests (functional, integration, etc.). The truth is that functional tests serve your users, while unit tests serve you — the developer. A unit test is just a convenience that allows you to track down bugs faster. At the end of the day, the reason you write tests is to make sure that your users will be able to be productive with your application, not to make sure that you can debug faster.
I also like how Igal Tabachnik puts it in the post Where unit testing fails: “The road to successful unit testing begins with understanding what unit testing is, but most importantly – what it isn’t.”
Unit Testing is Not the Sole Means to the End
Not only are there other valuable types of testing, there are also other valuable software development tactics and methodologies for producing high-quality software the meets or exceeds customer expectations. Unit testing is a valuable part of this overall approach, but should not diminish or overshadow other approaches such as design and code inspections, appropriate collaboration, code analyzers, and so forth.
Sacrificing Other -ilities for Testability
As valuable as I believe unit testing is, the thing I probably find most frustrating about unit testing in Java is having to compromise a desired language design feature in production code for the sake of unit testing. To be sure, there are many cases where unit testing encourages better practices in the production code. For example, unit testing encourages smaller methods, a feature that is generally considered a strength in code maintainability and readability. However, there are times when I want to make my class final or a method final or private and this can be orthogonal to unit testability.
It galls me to have to give up long-term design considerations and benefits to make testing possible. I want the software I deliver to be of high quality and enjoy readability and maintainability. It is difficult to have to give some of this up in some cases in the name of testability. Fortunately, unit testing often forces better design. But in the cases when I must choose between elegant design and production code with hacked test code to test that elegant design or hacked design and production code to accommodate elegant test code, remembering that unit testing is not the end itself provides the appropriate frame of reference for making that call.
The good news is that modern unit testing frameworks are steadily reducing the types of desirable traits in production code that must be compromised to support unit testing. Unit test frameworks such as PowerMock use bytecode manipulation to overcome most of these issues.
Group Think
Whether we call it the Lemming Effect or peer pressure or group think, there’s no question that we in the software development community tend to chase fads and shiny things. Unit testing has been part of software development for many years, but the increasing emphasis on it over the last decade has benefited software development. Unit testing has a long history in software development and has long since proven itself to not be a fad. However, I sometimes feel that some developers (particularly those who don’t realize how long unit testing has been around) think it’s something new that the rest of us don’t understand. In their zeal to promote it as a “new thing,” they actually make the act of unit testing more important than the goal of unit testing.
It is often the case in software development that evangelists and defenders of a particular product, language, framework, or technology cannot allow for any hint of a weakness or disadvantage to be discussed in relation to their favorite item. Any suggestion that their preferred product might not be the best thing since sliced bread is met with derision and scoffing. This ridiculous and unrealistic stance reduces constructive discourse about the advantages, disadvantages, opportunities, risks, and costs associated with a given approach. In the case of unit testing, this might be compounded by the fact that this practice was met with some skepticism by a large number of developers for numerous years and strong evangelism to change that has been largely successful but doesn’t know now how to contain itself.
Conclusion
This post has assumed constrained resources and schedules. For those who have the luxury of not facing constrained resources and short timetables, there may be sufficient time to write and maintain all of the tests for all types of code no matter how trivial and still deliver a final product in time. However, many of us are in circumstances where costs and benefits of different aspects of software development must be evaluated. In such cases, it is important to keep focus on the true long-term goal and use effective unit testing to achieve this goal while not allowing unit testing itself to become the end goal. As with most things in software development, the level and type of unit testing should based on experienced judgment of the value gained versus the cost expended. Unit testing is a means to an end; it is not the end itself.
Referenced and Other Related Resources
- Unit Testing is a Means to an End from our JCG partner Dustin Marxat the Inspired by Actual Events blog .
- Breaking Away From The Unit Test Group Think (30 August 2011)
- Selective Unit Testing – Costs and Benefits (4 November 2009)
- Is Unit Testing worth the effort? (January 2009)
- From Podcast 38 (31 January 2009)
- You Might Be Looking At Unit Testing All Wrong (11 February 2009)
- How to Quantify the Value of Unit Testing (7 September 2011)
- Unit Testing Myths and Practices (5 January 2012)
- The evil unit test. (27 January 2012)
- Where unit testing fails (24 June 2011)
- Unit tests lie: that’s why I love them (21 October 2011)
- Your Unit Tests Lie to You (17 February 2009)
- Ten Reasons to Write Unit Tests (31 May 2010)
