In this short series about acceptance testing I previously wrote about collaboration between testers and developers and how acceptance tests help to define clear requirements of a system.
Where the previous article were on the theoretical side, let’s now take a look at using acceptance testing as a practice in our development process. I prefer using Acceptance Test Driven Development, or ATDD.
In a Nutshell
With ATDD you use acceptance tests to guide the implementation path of a feature. You write a failing test scenario and the required glue code. When the scenario fails with the correct message (i.e. the test tool can run the acceptance test and the message correctly describes the missing feature) you start implementing the feature.
Like in TDD use the simplest way possible to make the scenario pass, but on a feature level. In TDD you would pass a test in the simplest possible way and then improve from there. In ATDD you just implement the feature in the simplest way possible to pass the scenario. Simply stated, if you’ve got the right set of scenarios the feature is implemented when the last one passes.
[Note that I deliberately chose not to use the word done, as after implementing the feature often we need to fiddle with the UI a bit more.]
Example Problem
For an example we’re going to build a shopping cart for Harry Potter book sale kata using ATDD. Customers get a discount based on the number of books in the series they buy. This exercise is taken from a list of kata from the Coding Dojo website. For a full description you can read: http://codingdojo.org/cgi-bin/wiki.pl?KataPotter.
The first 5 books from the Harry Potter series are on sale. One copy of any of the books costs $8. If you buy two different books from the series you get a 5% discount, if you buy three different books you get a 10% discount, if you buy 4 different books your discount is 20%, and if you buy all 5 you get a 25% discount.
Let’s pair up and build this feature together.
Discuss and Distill
The cycle for building a new feature goes through a few phases. Before we think about writing production code we have to get ourselves some scenarios. Remember how we talked about the team meeting with a business expert in the previous article? These activities form the first phases in the development cycle: discuss and distill.
Our team arranges a meeting with a business expert to discuss the shopping cart. We have prepared some questions, so this meeting can be very effective. Our team knows better than to discuss technical details in this meeting. We do not want to waste any of the business expert’s valuable time. At the end of the meeting we have taken note of examples describing the behavior of the shopping cart. For clarity we decide to list the books using roman numerals.
| I | II | III | IV | V | Formula | $ |
|---|---|---|---|---|---|---|
| 1 | 1 * 8 | 8.00 | ||||
| 1 | 1 | 2 * 8 * 0.95 | 15.20 | |||
| 1 | 1 | 1 | 3 * 8 * 0.9 | 21.60 | ||
| 1 | 1 | 1 | 1 | 4 * 8 * 0.8 | 25.60 | |
| 1 | 1 | 1 | 1 | 1 | 5 * 8 * 0.75 | 30.00 |
| 2 | 2 * 8 | 16.00 | ||||
| 2 | 1 | 2 * 8 * 0.95 + 1 * 8 | 23.20 | |||
| 2 | 2 | 2 | 1 | 1 | 4 * 8 * 0.8 + 4 * 8 * 0.8 | 51.20 |
Most of these lines are pretty much self-explanatory. The last line shows that there are 2 sets of 4 books. This might be surprising because you’d expect a set of 5 books plus a set of 3 books. The reason for this is that the discount must be optimized for the customer. The best possible discount in this scenario is to purchase 2 sets of 4 books as that is $0.40 cheaper than a set of 5 books and a set of 3 books.
After the meeting we distill acceptance tests from these examples and save them as Cucumber scenarios. The scenario for the first example reads:
Scenario: buy single book without discount When I buy 1 copy of "Harry Potter I" Then I must pay $8
The scenario for the second example turns out to be much alike the first one:
Scenario: buy two distinct books When I buy 1 copy of "Harry Potter II" Then I must pay $15.20
Obviously the remaining scenarios can be added in the same way. But a pattern is emerging. It would be pretty tedious and repetitive to have to copy and paste the exact same scenarios only to use different values. Besides, it would be nice if the example table we took from meeting with the business expert could be used in the acceptance test as well. That would make our scenarios more useful as documentation.
Cucumber provides a Scenario Outline feature to help out. Scenario Outlines more concisely express these examples through the use of a template with placeholders and an example table. We try a scenario outline for the former two scenarios and then decide to add the rest of the scenarios using the outline.
Scenario Outline: buy discounted Harry Potter books When I buy <I> copies of "Harry Potter I" And I buy <II> copies of "Harry Potter II" And I buy <III> copies of "Harry Potter III" And I buy <IV> copies of "Harry Potter IV" And I buy <V> copies of "Harry Potter V" Then I must pay $<Total> Examples: | I | II | III | IV | V | Total | | 1 | 0 | 0 | 0 | 0 | 8.00 | | 1 | 1 | 0 | 0 | 0 | 15.20 | | 1 | 1 | 1 | 0 | 0 | 21.60 | | 1 | 1 | 1 | 1 | 0 | 25.60 | | 1 | 1 | 1 | 1 | 1 | 30.00 | | 2 | 0 | 0 | 0 | 0 | 16.00 | | 2 | 1 | 0 | 0 | 0 | 23.20 | | 2 | 2 | 2 | 1 | 1 | 51.20 |
That’s more like it. Using the outline our examples read just like the table we draw earlier.
Obviously we’re not done at this stage, although we have successfully discussed the feature and distilled acceptance scenarios that discussion. We now have a complete set of acceptance scenarios. Our scenarios are a complete representation of the problem according to the business expert. The notation format of the scenarios is simple enough for the business expert to review them.
Now our team is ready for the implementation phase.
Implement
First thing we need is the glue code so that examples can be run and fail with a reasonable error message. Luckily Cucumber helps us creating the necessary glue functions by generating stubs. After a first run without glue code we can copy and paste to get to pending steps.
An example of such a generated function for Java is listed below. This is the glue code for a single step in a Cucumber scenario, also known as a step definition.
When("^I buy (\\d+) copies of \"([^\"]*)\"$")
public void I_buy_copies_of(int arg1, String arg2) throws Throwable {
// Express the Regexp above with the code you wish you had
throw new PendingException();
}Now that all steps are pending we start implementing the steps to make the first scenario fail with the appropriate messages. To get these steps to compile we need to add some skeleton production code, which we’ll later enhance with the implementation. This is interesting because at this time the API we wish we’d have flows rather natural from the acceptance tests.
When the first scenario fails with a message that describes the missing feature, we implement the feature. Then we move on to the next, and so on, until all features pass. We keep our code clean by using TDD.
The actual implementation of the step definitions and the code isn’t that interesting for the sake of this article. As I don’t wish to withold these details as well I did the exercise and put the code on Github.
Review
The last phase of building a new feature is to review it. I’m not talking about code reviews here, these take place in the implementation phase.
To wrap up our new feature we do some exploratory testing. Exploratory testing is finding out how the system actually works by using it freestyle. I.e. there are not test scripts to follow. Instead test cases are invented on the fly and immediately executed against the system. If you wish to learn more about exploratory testing I can recommend you to read Elisabeth Hendrickson her fantastic book Explore It!
In our review phase we write down any inconsistencies we find through exploratory testing.
To finish our feature we demo it to the business experts and collect their feedback. Combined with any results from exploratory testing, this is important input for business experts. Only they can decide how the system should behave.
More Examples
Personally I find ATDD a very pleasant way for the entire team to work. While having the benefits of clear requirements you enjoy the fact that you grow an automated regression suite that you can (and should) run every build. This regression suite gains the team a lot of confidence not to be afraid to make changes anywhere. As soon as something breaks, a test will fail, and the team knows what to fix.
If you like to see more examples of ATDD you should read ATDD by Example, in which Markus Gärtner pairs up with you to practice ATDD while working example problems.
