Unit Testing – Cost vs. Benefit

Kapil Viren AhujaAugust 26th, 2014Last Updated: August 25th, 2014

6 112 9 minutes read

I have been a big fan of unit testing for a very long time; my blog is ridden with posts about it. If I look at how automated unit testing has progressed in my projects over the last decade I am noticing that the inclination to do unit testing is just not there. Recently, I came across this Podcast series between Martin Fowler, Kent Beck and DHH which was in response to DHH’s post TDD is dead, long live Testing. The Podcast did’t reveal something new to me but it helped enlighten me. I spent the last evening introspecting and thinking how I have progressed TDD/Unit Testing in general over my projects and the trend I see is that the ones where I have been more involved as a developer or a senior developer or even an architect the effort to make unit testing has been there, BUT unit testing (not TDD) didnt work very well in last big project. And since I have taken up the responsibilities of a Solution Architect and as with my responsibilities I have moved away from Code a bit, I see the next layer not doing unit testing very much.

As I spent the time thinking what could be the causes, this is where the Podcast series helped a lot. The reasons that I can attribute to Unit Testing (not TDD) just not happening on projects recently

Writing Automated Unit Tests is just not as fun as Coding is
Writing Automated Unit Tests is just not as fun as Coding is

You get the point! I compare the cases when I first did unit testing back in 2004 (my first project with an IT Consultancy Organization) and then the last one in 2012 and I see why is that was the case. The environment in which I was doing unit testing back in 2004 and then in 2012 (8 years apart) were very different. Back in the days, I was an Enterprise/Core Java/.Net developer, and my life was to deal with things like making Database calls, writing services – business logic, validations, yada yada yada. The part which involved writing automated test cases for Web layer was minimal because the code that went into Web Controller (MVC frameworks we used varied) was only view. And we used Functional/System testing approach to test the UI. The responsibility of creating those selinium test cases were not with the QA Function but still with me as the developer. So the whole testing was “what makes sense” – we did TDD on services code and then functional testing (post coding) on Front-end.
But the most important part was that as a developer I was supposed to make sure that nothing breaks when I committed the code. The CI tools were not that advanced back in the days, but they were there. We used Ant for build and there was a task in there which was responsible for running test cases else failed the build (I even coded the ant build script). We were not able to integrate Selenium test runner ant script so that was a manual step, but we had to run a whole suite to ensure everything works. The phenomena “Self Testing Code” existed and we followed it to the core. The were build break fines place – 50 INR as fine of we failed a break; 3 consecutive breaks and it went upto 500 INR; 20 INR for any regression defects introduced (QA had their own suits to keep us in control). When I coded and when I committed the code, I knew that at night no one is going to call me to tell me that other members of my team is stuck. I was working in India and I had a team working from Boston and i knew that when I left and when they came in they wont be gated because i fucked up a compilation or broke something – they can just go about doing their work and be productive. Everyone did so and we all were happy bunch of developers/team. We did not have Continuous Delivery, but once story was done, a couple of rounds of QA testing and it was demo able to the clients. Out test coverage was 80%+ and we had a solid test suite. The suite however took 2 hours to run (I will talk about is very soon as that was the part which make Unit Testing a pleasure).

That was the last time, I saw the workflow work so very well. Surprisingly, that was also the first time I was doing it.

Fast forward 2007-08; I was the architect on a 40 people team (Developers, QA, Site Developers) and we were working on a web development – some heavy web development. It was using Backbase, JSF on JBoss, talking to MySQL via hibernate, everything wrapped around Spring. We started with the project and like before I wanted to do full blown TDD (not just Unit Testing). We started, but very soon (Sprint 1) there was feedback from developers that unit testing was too tough, it takes more time than it takes time to code. I was not the architect and I had produced a Golden Copy as a reference point which the team was using; so the feedback team was giving me made no sense. I let the team struggle on it for another sprint thinking they just need time to come around it, but when it started to hit our velocity points, I was forced to see what was going on in that Unit Testing Departments and what I saw shocked the soul out of me. The automated tests were there for a formality – the tests did test something, they provided some coverage but they were monstrous. They were ridden with mocks (almost) everywhere. No wonder it took so much time for the team to write test cases. For everything that needed to test they had to create a mock. A bit of it was lack of experience as well where team did not create helpers classes to provide for those common mocks, but trying to mock things like HTTPSession, HTTPRequest, Data Adapters for MySQL logic – aka every collaborator was in there. Well they did the right thing (per Unit Testing definition) – test the unit itself, so they mocked even the classes making calls to databases. It make so much harder to test a functionality. We quickly turned it around and set some ground rules

We will not use mocks
If we find a scenario where mocks could not be avoided we will not write automated unit tests for it

What this meant was a) this was not Unit Testing in it’s purest form (I don’t even know what Unit Testing is in it’s purest form, but what definitions and my seniors told me) and b) we had a bit of less coverage. But, I was okay with those consequences. The outcome was that team was not needed to write stuff that was used for just testing; it made our whole test suite a bit slow because it now made calls to the DB, but DB was local so it was not so slow – the team started to spend less time on unit testing, we wrote lesser tests and achieved better coverage to code. This wasn’t unit testing, but we did achieve Self Testing Code. Our CI build was now once again telling us if a commit broke something else. The fun was coming back into the game and we started to make progress. Then in Jan 2008, we decided to switch out Backbase and move over to Flex and this needed a redesign not just in our front-end but also in our back-end services. When we put together a team to redesign and reimplement the services they were given a solid test suite of 300 odd tests. The implementation was thrown away a few interfaces were changed. We fixed the test suite only (We made a decision to let our QA team fix the test suite – which need a QA person to code/script in Java which caused a lot of pushback, but I won that battle) the interface part and then started writing those services again and all we had to do was to make those 300 test cases pass. It was a breeze; we didn’t even need a QA team to test our services. Our test suite was good enough to do it.

We never achieved perfection, but then I was able to meet the goals.

Fast Forward 2012-2013; I was the architect on Adobe CQ 5.5 project and the frameworks had changed. There were no services we were writing. The traditional MVC framework was all not there at the developers disposal. I saw the code and there was monstrosity of JSP having business logic in scriplets, single tiered Java classes which did everything from getting data, manipulating them, validating inputs, sending outputs in very fluid formats like Maps (ValueMaps to be specific to Sling implementation). Amongst all this we made some hard design decisions like no scriptlets or business logic in JSPs. Hence there was now going to be Java Code, but there were numerous sling and JCR dependencies that we had to deal with. I once again wanted to do what I had done before – No MOCKS/STUBS. But, the frameworks to be able to do that were’t at my disposal back then. They existed but the community around then was thin, support was less and whatever existed was not going to work. And I ended up deciding to do Mocks. 3 months and it was exactly what was in 2008 – it wasn’t working.

The mocks were so bad that we could change the logic and tests would still pass.

What had happened that developers just mangle everything and tests were not even testing state or behavior. They just went for coverage. The tools were better this time and we had sonar. The idea was to write a test case and make it pass and show an increased covered in sonar. We had used mockit and it seemed like a good idea at the time, but just didn’t work. I understand mocks but i just don’t get them. We setup some state and we check for that state – all makes sense. But when everything is coming from Collaborators and the object you are testing is just aggregating and assembling data into a ValueObject (Map in our case), testing that assembly makes no sense. It’s like saying a developer can code the following wrong.

outputData.put("athletes", athleteList);

Well, if a developer can’t get a bunch of basic java statements right, well they have no business coding; let’s send them into a training and save ourselves from a world of pain.

Fast Forward 2014, I am not a solution architect and not focusing a whole lot of coding per say. I love to code, but it’s not my day job (i would like coding to be my day job some day). But, now I see teams struggle a lot to get unit testing even off of the ground. The environment is similar to what I had back in 2012, but now I do have a solution and I think there is a way of doing it without Mocks. Yet, the struggle is there, developers dont see this as fun any more – it looks like a burden.

The Cost vs. Benefit argument is a valid one in every scenario, and in case of our software development lifecycle where getting any sort of confidence is just too tough, as a developer the need to have something to fall back on and knowing that we are not breaking a system with my next commit is huge. In my development lifecycle, we can probably still live with some regression, but as we reach application maturity, as we start to reach stabilization not having a Self Testing Code makes us like Mammoths (not elephants) who move ever so slowly. In the current world of marketing, where we have clients who want to run campaigns in next 2 weeks, we can’t be slow in how soon we release code. You can only be relevant in the industry if you can move quickly, achieve the Continuous Delivery or at least reach a point when you can get releases out in production with a reasonable speed. Those days where releases used to happen once every 6 months are gone; or at least gone in the environment/market I am operating.

We have to have Self Testing Code, We have to Unit Test, We have to have feedback loops and We have to have feedback loop as soon as possible. There is no avoidance; let’s understand and embrace Or be extinct in a few years.

Reference:

Unit Testing – Cost vs. Benefit from our JCG partner Kapil Viren Ahuja at the Scratch Pad blog.