Karate seeds a defect into a model of your API's behavior, or tampers the live response in flight, then replays your suite and scores which defects your assertions caught. The ones they missed come back as a ranked worklist.
To be clear about a crowded word: fuzzers mutate the request to find bugs in your API. PIT and Stryker mutate source code to grade unit tests. Karate perturbs the provider's answer to grade your tests. Black-box, no source code needed.
# 24 defects seeded into provider behavior,
# your suite replayed against each one
caught ............... 17
survived ............... 7
survivor "discount cap raised 500 -> 5000"
no test would notice
survivor "renewal date off by one day"
no test would notice
survivor "empty claims list on GET /claims"
no test would notice
# independent score excludes assertions
# that echo the server's own answerThe Problem
Green means your tests ran. Coverage means they touched the endpoint. Neither answers the question that actually bites: would your assertions notice if the behavior broke?
An AI-generated suite reaches high coverage fast. That is what it is optimized for: exercise the endpoint, get a green, move on.
The observed failure pattern in the field: the suite hits every endpoint and asserts almost nothing. A status code here, a field-exists there.
Your tests grade your API. Nothing grades your tests. So a suite that would miss a real bug looks exactly like one that would catch it.
The only honest way to grade a test suite is to break things on purpose and count what it caught.
How It Works
Your suite's expectations are frozen before anything changes, so a test cannot quietly recompute its expected value to match the tampered answer. Then the provider's behavior is perturbed, two ways.
If your provider has a rulebook, an executable model of its decision logic, Karate seeds a realistic defect into it: a threshold moves, a branch inverts, a rounding rule changes. The mutated model answers your suite. Every seeded defect is a bug that could ship.
No rulebook? The response from your real provider or mock is tampered on the way back: a field nulled, a value shifted, an item dropped from a list. If your suite still passes, that tampering is a bug it would have missed.
The minimum is a suite and a base URL: the response-tamper lane needs nothing else to earn a grade. Each artifact you add widens the run. An API spec enables schema-aware mutants. A rulebook enables business-logic mutants, the ones shaped like real defects in your domain. Requirements let each survivor name the acceptance criterion it leaves undefended.
An absent artifact is a disclosed narrowing, never a refusal. The report states what the run could and could not perturb.
The Score
The obvious way to game a mutation score is to assert whatever the server said. Karate detects that, and the headline number excludes it.
The independent score
A test that reads the response and asserts the value it just read will pass no matter what the server returns. It kills every mutant on paper and none in practice. The independent score counts only assertions whose expected values do not derive from the response being checked, so a suite of copied-from-the-response echoes scores zero, which is what it deserves.
Both numbers are reported, raw and independent, side by side. The gap between them is itself a finding: it measures how much of your suite is checking the server against the server.
The mutation score grades assertion strength: would your tests notice a change in behavior. It never grades whether your expected values are correct. A test that asserts a bug kills the mutant of that bug. That is the known limit of mutation testing everywhere it is practiced, and this page will not claim otherwise.
The Output
A score tells you there is a problem. A survivor tells you which one, in the language of your domain.
Survivor
business-logic
No test would notice the discount cap dropping
The cap moved from 500 to 5000 in the mutated model. Every scenario that touches discounts still passed. The pricing behavior your business depends on is undefended.
Survivor
response-tamper
No test would notice the claims list coming back empty
The tampered response dropped every item from the array. The suite checks the status code and the shape, so it stayed green.
Survivors are ranked, so the first thing on the list is the defect your suite is most exposed to. With requirements wired in, each survivor also names the acceptance criterion it leaves undefended, which turns a test-quality report into a risk report.
The Fault Feed
Everything above grades your tests. Point the perturbation at a dependency instead, and it grades your service.
The Karate mock standing in for a dependency your service consumes starts feeding it faults: a 500 where a 200 belongs, a connection that dies mid-response, a field that arrives corrupted.
A fault your service tolerates is a pass. A fault it mishandles, a crash, a silent corruption, a hung request, is a finding that no recorded contract could express, because contracts describe agreement and this is about disagreement.
One artifact, two jobs. The paired run proves your mock faithful when it behaves. The fault feed grades what happens when it does not.
See how Karate mocks workWhere It Fits
Coverage, provenance, and hardening answer three different questions about the same suite. Karate measures all three.
You exercised it. The endpoint was called, the requirement has a test against its name.
API coverage →Honestly, not faked. The evidence is real execution, not a claim, so green that was never actually earned gets surfaced.
Trust & governance →You would notice if it broke. The axis this page is about, and the one neither coverage nor provenance can see.
The Landscape
The scoped claim, from a two-sweep primary-source survey in August 2026: no shipped tool grades an API suite's assertions by perturbing provider behavior without the system's source. The near-misses are worth naming, because each one is a good tool doing a different job.
| WireMock, Toxiproxy, Mountebank | Inject faults into responses, but emit no score. Whether your test noticed is left entirely to you. |
| Pact, Microcks | Score the provider's conformance to a contract, not the strength of your suite's assertions. Different measured subject. |
| PIT, Stryker, mutmut, Diffblue | Real mutation testing, of unit tests, requiring your source and build. They cannot grade a black-box API suite. |
| API fuzzers | Mutate the request to find bugs in your API. The word overlaps; the job does not. |
Academic work on grading generated API tests exists and is different in method. The claim here is scoped to shipped tools, and we will keep it scoped.
The paired run measures divergence: does your mock still behave like the real provider? Mutation measures adequacy: would your suite notice if either one broke? A team serious about its API estate wants both numbers, and they come from the same suite.
See contract testingFAQ
Grading a test suite by deliberately changing the provider's behavior and counting what the suite caught. A seeded defect your assertions catch is a kill; one they miss is a survivor. It answers the question coverage cannot: would your tests notice if the behavior broke?
Fuzzing mutates the request to find bugs in your API. This perturbs the response to find weaknesses in your tests. A fuzzer grades your service; mutation testing grades your assertions.
No. Everything happens at the API boundary. The minimum is your existing suite and a base URL. A spec, a rulebook, and requirements each widen the run, and every absence is disclosed rather than refused.
No. It grades assertion strength, whether your tests would notice a change, never whether your expected values are right. A test that asserts a bug kills that bug's mutant. That limit is inherent to mutation testing, and we state it.
Yes. The fault feed points the perturbation at the dependency your service consumes: your mock of it misbehaves on purpose, and your service's resilience gets graded. A tolerated fault is a pass, a mishandled one is a finding.
This says they would notice if the code broke. Graded at the API boundary, without your source, from the suite you already run.