Your coverage is green. Would your tests notice if the code broke?

The suite that ran everything and checked nothing

Here is the failure mode we keep seeing, and it is getting worse, not better. A team points an AI assistant at their API and asks for tests. The assistant is good at its job: within an hour, every endpoint has a test, and coverage is high. The suite goes green. Everyone moves on.

Look inside those tests and a pattern shows up. A status code assertion here. A field-exists check there. And, most dangerous of all, assertions that compare the response to itself: read the value from the server, then assert the server returns that value. That test kills every bug on paper and none in practice, because it passes no matter what the server says.

Nothing in your tooling flags this. Your tests grade your API; nothing grades your tests. A suite that would miss a real bug looks exactly like one that would catch it, right up until the day it matters.

Break things on purpose, then count

The honest way to grade a test suite is old, and it has a name: mutation testing. Deliberately introduce a defect, run the suite, and see if anything fails. A caught defect is a kill. A missed one is a survivor. The survivors are the interesting part: each one is a bug that could ship today without a single test going red.

The catch is that the entire mutation-testing ecosystem, PIT, Stryker, mutmut and friends, works at the source level. It mutates your code and grades your unit tests, and it needs your source and your build to do it. If what you have is an API suite testing a service over HTTP, that machinery does not apply. And the other tools that sound adjacent do a different job: fuzzers mutate the request to find bugs in your API, not weaknesses in your tests, and fault injectors like WireMock or Toxiproxy break responses on purpose but leave "did your test notice?" entirely to you. No score, no verdict.

Mutation testing at the API boundary

Karate now does this at the API boundary, black-box, without your source. The engine perturbs your provider's behavior two ways: it seeds a realistic defect into a behavioral model of the provider (a threshold moves, a branch inverts, a rounding rule changes), or it tampers the live response in flight (a field nulled, a value shifted, an item dropped from a list). Then it replays your suite against each perturbation and scores what your assertions caught.

What comes back is not just a number. Survivors arrive as a ranked worklist in the language of your domain:

And the score is built so it cannot be gamed. The headline number is an independent score: it counts only assertions with no detected dependence on what the server answered. A suite of copied-from-the-response echoes scores zero, which is what it deserves. The gap between the raw score and the independent score is itself a finding: it measures how much of your suite is checking the server against the server.

What the score means, stated plainly

Two honesty rails, because a number invites over-reading. First, the score grades assertion strength: would your tests notice a change in behavior. It never grades whether your expected values are correct; a test that asserts a bug will kill the mutant of that bug. Second, the claim about the category is scoped: as far as we can find, no shipped tool grades an API suite's assertions by perturbing provider behavior without the system's source. The near-misses are real tools doing different jobs, and academic work exists with different methods. We state it that way because that is what the survey found.

The third axis

We think of this as the third adequacy axis for a test suite, next to two we already measure:

• Covered: you exercised it. The endpoint was called, the requirement has a test against its name.
• Exercised: honestly, not faked. The evidence is real execution, so green that was never actually earned gets surfaced.
• Hardened: you would notice if it broke. The axis neither coverage nor provenance can see.

A suite can max out the first two and still be soft on the third. That is precisely the shape of an AI-generated suite, and it is why we built this now.

Start with what you already run

The minimum is the suite you already have and a base URL: the response-tamper lane needs nothing else to earn a grade. An API spec adds schema-aware mutants. A rulebook adds business-logic mutants, the ones shaped like real defects in your domain. Requirements let each survivor name the acceptance criterion it leaves undefended. Anything missing narrows the run and the report says so; nothing is refused.

The full story, including the fault feed that turns the same machinery around to grade your own service's resilience against a misbehaving dependency, is on the API mutation testing page.