Point Karate at the suite you already have, name your mock and your real provider, and it runs both legs in one session, in one pinned order, and returns the measured set of differences.
No pact file. No broker. No consumer sign-off. No provider-state fixtures.
# Your existing functional suite.
# The target is a variable. That is it.
Background:
* url baseUrl
Scenario: create and fetch a policy
Given path 'policies'
And request { holder: 'ACME', premium: 2400 }
When method post
Then status 201
And match response.premium == 2400
# One session runs it twice:
# leg A -> your mock
# leg B -> the real provider
# and classifies every scenario.The Problem
Your suite is green against the mock by construction. That is what a mock is for. It says nothing about whether the teams building against it are building against the truth.
Nobody ships a breaking change on purpose. It accumulates: a default changes, a field goes nullable, an error path starts returning 422 instead of 400.
The provider moves. The mock does not, because nothing is watching it. Every team building against that mock is now building against last quarter's behaviour.
The document describes operations nobody calls and omits behaviour everybody depends on. It stays green because a document cannot fail a test.
API drift is what happens between releases. This is how you measure it.
Where It Belongs
Teams reaching for a contract-testing tool before the rungs below it hold are buying the top of a pyramid with nothing underneath.
The provider is verified, and the double is verified against the provider.
Load, auth, and abuse cases are tested rather than assumed.
Bad input, missing fields, and error paths are covered, not just the happy case.
You assert the values that come back, not merely that a 200 came back.
The endpoints are exercised at all, and something checks the result.
The advantage of doing this in Karate is that every rung uses the same syntax. You are not adopting a new tool at level five, you are pointing tests you already have at a second target.
Shapes vs Behaviour
One of them checks that the response has the right fields. The other checks that the answer is right. They are not the same job, and only one of them tells you whether a stand-in can replace a provider.
Does the response have the right fields, the right types, the right status code? This catches a rename. It misses almost everything else.
The same suite, with the same assertions about behaviour, run against your mock and against the real provider. Nothing is downgraded to a schema check to make contract testing convenient.
A tool that lets you skip real functional testing during contract testing is taking a shortcut. Peter Thomas, who created Karate, has a published word for that shortcut: a cop-out. Low-fidelity contract tests check shapes. They do not check behaviour, and the question you actually need answered is behavioural.
Can a consuming team use the mock instead of the provider, or not?
The Paired Run
Both legs run in the same session, in the same pinned scenario order, so the comparison is like for like. Nothing in your suite is contract-testing-shaped except a tag and a variable.
The stateful test double your consuming teams develop against every day.
The service itself, in whichever environment you can reach it.
Every scenario classified, with both compared values kept on each difference.
The OpenAPI document is the shared artifact, not a consumer's recorded expectations. That is the difference between this and a Pact-style workflow, and it is why there is no pact file to publish, no broker to operate, and no consumer sign-off gate. Pact is an honest answer to a different question. If your teams want each consumer to declare what it depends on, that is what consumer-driven contract testing is for. If you want to know whether your stand-in behaves like the real thing, this is the shorter path.
The Outcomes
Three of them are findings. One of those three is a finding that no schema-based or example-based approach can produce, even in principle.
The finding your assertions cannot see
Both runs go green. Every schema check passes on both sides. And the two services still returned different answers, because your assertions were not tight enough to look. This is a finding about your test suite, not about either service, and it is the one that quietly invalidates a contract-testing programme built on shape checks.
A schema-based contract test cannot report this. Not because of an implementation gap, but because both responses satisfy the schema by definition.
Finding
staleMock
Your mock is behind. Teams developing against it are building on behaviour the real service no longer has.
Finding
mockBug
Your double cannot simulate something the real service does. Any test that relies on it is testing a narrower world than production.
agreed |
Both legs behaved the same way |
staleMock |
The mock is behind the provider |
mockBug |
The mock cannot do what the provider does |
unassertedDivergence |
Both passed, and the responses still differ |
agreedFailure |
Both legs failed the same way |
notMeasured |
Nothing was compared here, which is not the same as agreement |
A class with nothing in it still reports its zero. An empty divergence list means nothing was compared, not that the two targets agreed, and the report says so in those words.
See It For Yourself
No licence, no install, no sign-up. This is the reference kit's own report, regenerated by CI.
The mock and the provider gave the same answers in 12 compared scenarios. This run compared 5 of 7 operations. It did not test 2.
12
scenarios matched
2
operations not tested
54
expected differences, waived
Note what a good result still tells you. It reached rung 3, verified against provider, and it is capped there because 2 of the 7 declared operations were never exercised. The report names them, cancelPolicy and getClaim, and hands you the work: add a test for each. The 54 waived differences come from 3 ignore rules covering minted ids and quote references, each carrying a reason and an owner, none of them stale.
The report does not just grade you. It hands you the work, by name.
Add a test for openapi:cancelPolicy
This run did not exercise it, so nothing here reports whether the mock matches the provider for it.
Add a test for openapi:getClaim
Same again. An operation nothing compared is reported, never assumed to agree.
Review 3 ignore rules
They waived 54 differences. Each waiver carries a reason and an owner.
Record which deployment you tested
The wire shows only that the responder was not one of our mocks. Provider identity is not independently verified, and the report says so rather than implying otherwise.
Freshness is computed when you read the file, not when it was written. If the run ages past its window or a binding stops matching, the claim is withdrawn rather than quietly carried forward, so the report you are looking at may say so.
The Precondition
A test double is only useful if it can hold state. Without it you can never just use the mock instead of the provider, which is the entire reason to have one. A stub that answers with the same canned payload no matter what you send it cannot stand in for a service. It can only stand in for a screenshot of one.
Contract testing only means something if your mock is stateful. A shape-matcher has nothing to prove and nothing to disprove.
Karate mocks are stateful by default: the Background block runs once at startup, so variables defined there survive for the life of the server. A working CRUD double is a dozen lines. The paired run is then what proves that double can stand in.
The Two Numbers
How many of the compared operations showed no difference, and how much of the declared API the run actually touched. One without the other is a lie by omission.
Of the operations that were compared, how many showed no difference between the two legs.
How much of the declared API the run actually reached. The report names the operations it never tested, individually, so the gap is a work item rather than a footnote.
A pass or fail delta is a lower bound on behavioural divergence, never a measurement of it. Any single percentage claiming to say how faithful a mock is has quietly discarded that distinction. So the output here is a measured divergence set, not a number.
For the same reason, an agreement proportion never appears without its denominator. Reporting no differences across five of forty operations as a clean bill of health is exactly the misreading this design exists to prevent.
What The Evidence Entitles You To Claim
The whole-API rung is the floor of the per-operation rungs, and it stays capped while any declared operation is untested.
The double was compared against the real provider across the declared surface, and the differences are accounted for.
The suite ran against the real service, so something other than your own mock has answered.
The suite ran and passed, but only a stand-in ever answered it.
The responses match the document. Nothing has been established about behaviour.
Zero of zero reads not checked, never passed. Every rung carries the counts it rests on.
Waivers & Freshness
Generated ids and timestamps will always differ. That is not drift, and it should not drown the findings that are.
Getting Started
You can start without one. A team with a Karate mock and a Karate suite is already doing contract testing: write the stateful mock, write the functional tests, point the same tests at the mock and at the real service, and you learn whether the double can stand in. No document required.
What a spec adds is the automated, measured version. The document fixes operation identity and gives the scope number its denominator. Start today with the mock and the tests you already have, and add a spec when you want the measurement. Spec-less paired runs are on the way, where the proportion and the rung read not measured rather than being guessed.
That is the best position to be in, because you reuse them. There is no pact file, no second artifact to keep in sync, and no broker to run. The target becomes a variable and the suite carries a tag.
The whole surface is agent-callable over MCP. An LLM can write the suite, mint the paired run, read the divergence set, and propose the fix, without leaving its own chat.
Most tools tell you the API matches its description. Karate tells you whether your stand-in can actually replace it, and names what it did not check.