JUnit vs TestNG: The testing frameworks showdown
Testing is an inseparable part of the software release cycle in well balanced developer teams. And it wasn’t always like that. Unit tests, integration test, system tests and others weren’t always around. Today, we’re lucky to be in a place in time where testing matters and the value is known to most of the stakeholders.
In this post, we put testing at the center and explore Java testing through the eyes of JUnit and TestNG, while comparing their main features and use cases.
Huge thanks goes out to Sasson Shmuel, OverOps’s test automation engineer, for his help in writing the post.
So… Given a clean slate, what would you choose?
1. Meet the frameworks
JUnit and TestNG are without a doubt the most popular testing frameworks out there. In fact, when we looked into the libraries GitHub’s top Java projects use, both of them made the top 20.
JUnit ranked first with a presence in 62% of the projects, and TestNG was in at #20 with 6%. The Maven repository shows similar results with JUnit rocking the popularity charts at #1 with 42,484 uses, and TestNG at #15 with 3,873 uses. This is also because JUnit is added by default in many Maven archetypes.
Versions and some recent history
The latest JUnit version is 4.12, released on December 2014, with JUnit 5 expected to reach GA status in late 2016. That’s real soon and we also expect it to become a hot topic in conferences.
JUnit has been one of the drivers for the adoption of Test Driven Development, and originally developed by Kent Beck and Erich Gamma. In 2013, the torch was passed on to the JUnit team:
i just transfered the junit repo on github to junit-team. *sniff sniff* my baby's all grown up.
— Kent Beck 🌻 (@KentBeck) February 6, 2013
An early JUnit 5 version is already available for you to play around with and you can also contribute to its development on GitHub. A fun fact is that the new version development was made possible through a crowdfunding campaign.
As to TestNG, which was created by Cedric Beust, the latest version at the time of writing is 6.9.10 from December 2015. It first showed up around the time JUnit 3 was still around, and offered new features that didn’t exist at the time with a design that’s focused on wider testing use cases. Some of these features, like the use of annotations, were added in JUnit 4, and this is the version that we’re focusing on here, with an outlook towards the upcoming JUnit 5 release.
2. Writing tests
Both TestNG and JUnit are based on a behavior similar to Java Assertions, which were added back in Java 4. And we can just use assertions directly, so… why bother with a testing framework?
It’s easy to get tempted starting off with simple Java assertions, but as the project grows, testing quickly becomes non-trivial and it makes a lot of sense to use a framework to manage it. Moreover, the learning curve is pretty quick and the main concepts are simple yet powerful. JUnits code base is so lightweight, that Martin Fowler was famously quoted:
“Never in the field of software development have so many owed so much to so few lines of code”
Both JUnit and TestNG follow the xUnit conventions, yet have quite a few distinctions that we’d like to highlight here. Groups, parallelism, parameterized test, and dependencies:
Groups
TestNG offers additional annotations to those available in JUnit. Probably the most notable is the ability to run code before / after groups of test cases. Also, single tests can belong to multiple groups and then run in different contexts (like slow or fast tests). It’s practically another layer in between a test case and a test suite:
@Test(groups = { "sanity", "insanity" })A similar feature exists in JUnit Categories but lacks the @BeforeGroups / @AfterGroups TestNG annotations that allow initializing the test / tearing it down. btw, looks like JUnit 5 is going to deprecate Categories and introduce a new concept that will be called a Tag:
@Tag("sanity")
@Tag("insanity")
void testSomething() {
...
}But as it looks right now, no @BeforeTag / @AfterTag annotations in sight.
Parallelism
If you’d like to run the same test in parallel on multiple threads, TestNG has you covered with a simple to use annotation while JUnit doesn’t offer a simple way to do so out of the box. The TestNG implementation would look like:
@Test(threadPoolSize = 3, invocationCount = 9)
public void testSomething() {
...
}Meaning, 3 threads and 9 invocations of the same test. You can also run whole suites in parallel if you specify it in TestNG’s XML run configurations. While with JUnit you’d have to write a custom runner method, and feed the same testing parameters multiple times. Which brings us to the next bullet point.
Parameterized / Data driven testing
This is the problem of feeding different test inputs to the same test case, which both TestNG and JUnit solve, but use different approaches. The basic idea is the same, creating a 2D array, Object[][] that includes the parameters.
However, in addition to supplying the parameters through the code, the TestNG @DataProvider can also support XML for feeding in data, CSVs, or even plain text files.
A feature that exists in JUnit and misses on TestNG is the ability to use different combinations between several arguments. This provides a shortcut to long parameter list and explained in JUnit Theories.
Dependencies between groups / methods
Since JUnit was built for unit tests, and TestNG had a wider array of tests in mind, they also differ in their approach to dependencies between tests.
TestNG allows you to declare dependencies between tests, and skip them if the dependency test didn’t pass:
@Test(dependsOnMethods = { "dependOnSomething" })
public void testSomething() {
...
}This functionality doesn’t exist in JUnit, BUT can be emulated using assumptions. A failed assumption, results in an ignored test which is skipped.
Bottom line: Different developers would have different expectations from their framework of choice. TestNG seems to provide greater flexibility out of the box compared to JUnit.
3. Running tests
When writing tests, we don’t need to include a main method since the frameworks run these tests for us with their own main method that manages the execution of the individual tests.
In case you do need a custom runner, JUnit provides the @RunWith annotation that lets you use your own runner. Bypassing the default runner is also possible with TestNG but not as straightforward as with JUnit. Yet it’s worth noting that TestNG support XML run configurations that prove to be useful in many use cases.
As to actually running tests, both frameworks have CLI support, running through ANT, and plug-ins available to your IDE of choice with pretty similar functionality. Although JUnit comes out of the box with JDT (Eclipse Java Development Tools) so there’s that.
Bottom line: JUnit is slightly more flexible with its @RunWith annotation.
4. Reporting on the results
Test results interest a lot of people. It’s not just the developer who runs them. This is when reports come into play and both frameworks have an answer for this issue.
TestNG reports are generated by default to a test-output folder that includes html reports with all of the test data, passed/failed/skipped, how long did they run, which input was used and the complete test logs. In addition, it also exports everything to an XML file which can be used to construct your own report template.
On the JUnit front, all of this data is also available via XML but there’s no out of the box report and you need to rely on plugins.
Bottom line: TestNG provides out of the box reports, JUnit only exports it to XML
5. Automating test runs
Both frameworks can be used to create automated test runs with tools like Jenkins, Travis CI and Teamcity. Plugins are also available for creating reports out of the test data and sending it to whoever might be interested by your channel of choice. For example, using TestNG with Jenkins, and integrating it with email and Slack.
We’ve also previously written about some of the most useful Slack integrations for developers, check it out.
Bottom line: Automation for the win. Both JUnit and TestNG probably have the integrations you’re looking for.
6. Killing off the failed test
Whoops, we have a failed test. What happens next? If it’s granular enough it should be relatively easy to find the cause for the failure, but reality often has different plans for us.
Although it’s mainly built for production environments, a good part of our users also have OverOps set up to monitor their testing environment. Whenever a test fails, they’re able to view the variables that caused the failure, across all the methods that led there.
Bottom line: For taking your environment to its next level, check out OverOps and see how it can help you troubleshoot errors.
7. Mocking, matching and other frameworks
A blog post about Java testing, JUnit and TestNG cannot be complete without mentioning some of the complementary testing libraries out there.
In this section we’re doing a quick overview of these categories with the most popular libraries according to GitHub’s top Java projects.
Mocking
A good practice is to run unit tests in isolation, but what if the component that you’re testing depends on other complex objects? This is where mocking comes in and let’s you create mock objects that emulate the behavior you need from other parts of the system:
Matching
The assertions provided with JUnit and TestNG are pretty basic, using a matcher contributes to their readability and adds more capabilities for you to choose from:
- Hamcrest (#34, 4.12% of projects)
- AssertJ (#55 ,2.72% of projects )
- Another library worth checking out is Google’s Truth, on which we wrote as part of a post about interesting Java projects from Google (beyond Guava).
And… Spock
Spock is an up and coming Groovy (not THAT kind of groovy) testing framework that’s based on JUnit. The idea behind it is to consolidate this ecosystem of test libraries to a single framework with a simple Groovy DSL. We’ve covered some of its functionality in a post about Groovy and its uses for Java. For a deeper comparison, check out these slides from Kostis Kapelonis.
Bottom line: Testing usually doesn’t stop with a single framework, get to know the other libraries in the game.
8. Community
JUnit has more history to it and a considerably larger user base, it basically defined the standard for Java unit testing. As such, it has the largest user base and it’s fairly easy to find an answer or someone to ask about any issue that you might have.
TestNG might have fewer users, but its community doesn’t fall short (although its website could use a facelift). And… both could benefit from some logo guidelines. Any designers in the crowd?
Here are some more useful links for JUnit:
- JUnit 4 site and GitHub repo
- JUnit 5 site and GitHub repo
And TestNG:
- TestNG site and GitHub repo
- TestNG discussion group
Bottom line: You won’t get lost with either of them.
Final Thoughts
Java testing has come a long way and we’re happy to be in a time where testing matters. The main takeaway here is creating meaningful test.
With that said…
Which framework are you using and why? What would you choose given a clean slate? Let us know in the comments section below.
| Reference: | JUnit vs TestNG: Which Testing Framework Should You Choose? from our JCG partner Alex Zhitnitsky at the OverOps blog. |
I am big fan of TestNG so I am trying to push it on every project I stumble across. To be honest I don’t understand why is JUnit so popular in Java community. I haven’t check JUnit 5 with it’s new features but having DataProvider, easy and non-static @BeforeClass method annotation (instead of Junit static one) always beats JUnit bigger community support and usage.