The Test-Automation Capability Map

Test Automation is serious business. With teams under pressure to ship more often for competitive advantage, ensuring the best-quality end-user experience is key.

The choice of a test-automation tool can make the difference between a team that has high velocity - or one that lacks confidence to release.

We have seen many teams that have made the mistake of writing an in-house test-automation framework. I have personally experienced the pain of developers who have inherited tests in such a framework multiple times.

Unfortunately, teams continue to write their own framework for many reasons. One is that development skills are valued more than domain-knowledge or knowledge about the end-user. Creating a test-framework “from scratch” is over-hyped, fuelled by a multitude of influencers and bloggers.

A reason that few like to talk about is that it is easier to write some code instead of thinking through what it takes to actually test a given piece of software. You would have seen cases where a team spends months building a test-framework, only for a new team to come along and promptly decide that a re-write is required.

To help teams make the right choice, we have created this graphic to emphasize a few points. It can be used as a handy reference to compare test-automation tools and discuss the relative importance of the capabilities for your specific team.

Please pass this on to those who are responsible for driving standards and evaluating tools in your enterprise.

To summarize:

• Test Automation requires a large set of fundamental capabilities. Are you sure you can implement all these in-house, or would you be better-off using an industry standard solution?
• Does the tool pay attention to Developer Experience? Keep in mind that tests must be maintained for as long as your systems are in production.
• Does the tool bring developers and QA closer together or force them into different user-interfaces and workflows?
• Can your tests be version-controlled just like your code? To be specific, can you see the history and the “diffs” between previous versions?
• Is your test data within your firewall or going into “somebody else’s cloud”?
• Is it easy to simulate end-user-flows and the core capabilities (your reason to exist) of your business?
• Is there a mature community of users or someone that you can reach out to for support?
• How useful is the documentation? Does it go into detail? Are there enough examples?
• How easy is it for someone new to the tool to start using it productively?

The list of capabilities in the graphic is a result of many iterations and discussions. Do let us know if any test-aspect is missing, or if you have any other feedback!

And by the way, the boxes in blue are things that Karate does today.

Happy Testing!