Is Behaviour Driven Development (BDD) right for API testing?

The primary goal of BDD can be summarized as follows: to reduce misunderstandings between those who define software requirements and those who implement the software.

The best source for the history and origins of BDD can be found on the Cucumber web site . Cucumber is just one tool that implements an approach to BDD, but it is not the only one. But many teams assume that because they are using Cucumber as a tool – it means that they are successfully “doing BDD”. As the creators of Cucumber themselves have lamented about for years, this is a huge mistake.

Cargo Cult

To be more specific, a very common misconception is that if you use the keywords “Given”, “When” and “Then” in an automated test – it means that the team will magically enjoy the supposed benefits of BDD. This may sound far-fetched, but if you are leading a team that claims to be doing BDD, I recommend that you walk the floor a bit, and ask the lesserexperienced engineers what their understanding is. You may get some interesting insights into how the team is thinking about BDD.

I remember a point very early in my career where I switched to a shiny new unit-testing framework that had things like “given()” and “when()” and “then()” in the syntax. I remember that virtuous feeling of satisfaction. Having just read about this great approach called BDD in some article or blog, I was now part of that exclusive club! I proudly declared to some colleagues that I was doing BDD. It was probably a few years later when I realized how mis-informed I was.

The feeling that badly done BDD evokes in me is the term “Cargo Cult Programming”. If you haven’t heard of the term, this Wikipedia entry explains the rather hilarious origin. “Cargo cult” is a great way to refer to a phenomenon which is all too prevalent in our field. Which is when teams hear about some “best practice”, adopt a bunch of ceremonies without truly understanding the fundamentals, and then stand back expecting to promptly enjoy the rewards and good things that should ensue.

Yeah, that never ends well.

BDD is not for testing code that already exists

Did you know that in its true form, BDD means that you should write your scenarios before a single line of code is written?

You should pause reading for a moment and think deeply about the implications of the above statement.

BDD is mostly about having conversations with your domain experts, business users or product owners on what the software requirements are. BDD also encourages that that discussion should result in “examples” which help in fleshing out the business-rules involved.

Instead of just descriptive text, examples with real data map a lot better to how programmers think about writing code and tease out the edge-cases

In other words, if your software delivery team is trying to automate tests for an existing system, BDD is not what you should be doing! It doesn’t matter if the system is partially done or not, BDD is just going to slow you down.

There is an argument to be made that BDD results in “executable specifications” and much more readable tests. This is why some teams choose BDD even though they are writing BDD scenarios “after the fact”.

But as we will see below, there are elegant ways to achieve “readable” test reports that serve as documentation of what the system does. You don’t need to formally adopt BDD, and your test-scripts and test-reports will still be readable, even by non-programmers.

BDD for API testing

The end-user or consumer of an API is typically another programmer. Another way to look at this is that APIs are how computers to talk to each other, and expressing what APIs should do - is best done using concepts that are closer to code than natural language.

I have a strong point of view that BDD has very little (and perhaps negative) value for API tests. The big difference between a UI test (human facing) vs an API test (machine facing) is that an API test has a clear "contract" that you are coding to. This contract is best expressed in technical terms (JSON / schema) instead of the deliberate abstraction needed when you do BDD the right way.

For more insights on how API testing is simpler than UI testing, read our free e-book:

Navigating the Brave New World of API Testing.

But I want my tests and reports to be readable!

If you are not using BDD, how can you ensure that your API tests are readable? Ideally, your test-reports should:

• serve as documentation of how your APIs work,
• and include examples of how to call them.

Here is where a mature automation tool that has built-in HTML reporting can add value. Shown below is part of a Karate test that exercises the “Restful-Booker” API playground. This is a realistic simulation of an API that allows the consumer to book a hotel reservation.

The test script is on the left and the test-report on the right. Since there are comments added before each business-operation, the test and test-report provide the best of both worlds: you not only get full-control over the API calls, payload-data and JSON assertions – but you also get a very readable (and hence maintainable) test.

Comments appear clearly in the report, in-line with the API calls that were made. The tests and test-reports can be easily read top-to bottom and give you a good sense on what business functionality is invoked.

Observe how there is a good simulation of an end-user workflow, and you can see response data (e.g. the “bookingid”) “chained” into the next request multiple times.

The test-data (or in BDD terminology, the “example”) for the business scenario can be clearly viewed at the start of the test. JSON happens to be an elegant, human-readable way of concisely expressing scenario data, and Karate takes advantage of this throughout the framework.

For those who are familiar with Cucumber’s “Scenario Outline”, note that Karate also offers you the same human-friendly way of defining Examples in a tabular format. Some teams really like this way of doing data-driven test automation aligned with the BDD concept of examples. All of this can be done without the need to worry about whether a test step has to be prefixed with “Given”, “When” or “Then”.

Look Ma, No Step Definitions!

For teams that have experience with BDD tools, what surprises them the most is that Karate does not require any step-definitions to be implemented behind the scenes. Step-definition “glue code” is known to be one of the “hidden costs” of BDD tools, and Karate eliminates this layer completely. What you see in the test is all that you need to write (or read). The built-in keywords for API testing and JSON assertions take care of most of your API testing needs.

Parting Thoughts

While we focused on whether BDD is appropriate for API testing in this article, it may also help you evaluate if BDD initiatives in your organization are structured correctly, whether the right people in charge, and whether they are delivering the value that you expect.