Your tests are green. Two of your seven operations were never compared.

Every test report is built to answer one question: did anything fail? It is the wrong question to stop at, because a report that only knows how to say pass or fail has no vocabulary for the third answer, which is nobody looked.

That third answer is where the risk lives. Below is a real run, published on every commit, with no login and no sales call in front of it.

The result

Read the result word again. Not passed. Matched, where tested. The qualifier is doing real work, and most reports would have dropped it.

What a normal report would have said

Twelve scenarios ran. Twelve passed. Rendered the usual way, that is a green tick and a 100% badge, and you would move on.

But 100% of what? The suite exercised 5 of the 7 operations the OpenAPI document declares. The other two were never called, so nothing in that run knows anything about them. They were not fine. They were not broken. They were unexamined, which is a different state, and the honest report keeps it separate.

An agreement figure without its denominator is the most common way a test report misleads a room. "100% divergence-free" over five of forty operations reads identically to genuine coverage on a slide.

It names the gap

The part worth stealing, whatever tool you use, is that the report does not stop at flagging the gap. It hands you the work by name:

Two named operations is a ticket someone can pick up this afternoon. "Coverage is at 71%" is not.

The 54 differences that are not failures

Fifty-four responses differed between the mock and the provider, and none of them are a problem. They are minted identifiers: a surrogate primary key, a quote reference, a claim's policy reference. Each target generates its own, so they will never match, and pretending otherwise would drown every real finding in noise.

Three ignore rules cover them, and each carries a reason and an owner. Crucially the rule re-classifies the difference rather than hiding it: both values are still recorded, and a rule that stops matching anything is flagged stale so the list cannot quietly rot. In this run, none are stale.

The distinction that matters: a waived difference is one you have accounted for. A suppressed difference is one you have stopped seeing.

The claim it will not make

The run reached rung 3 of 4, verified-against-provider. Not rung 4. It is capped there, and the report says exactly why: the pair did not exercise 2 of the 7 declared operations, so the artifact cannot claim the top rung. That limit holds even though every operation it did compare was divergence-free.

A tool willing to cap its own best-case output is telling you something about how the rest of its numbers were arrived at.

What we would want you to take from this

• Report the denominator, always. An agreement proportion without the scope it was measured over is a lie by omission, however accurate the numerator.
• Not measured is not zero, and it is not pass. It needs its own state, or it silently becomes one of the other two.
• Name the gap, do not score it. cancelPolicy and getClaim are actionable. A percentage is not.
• Expected differences get a reason and an owner. Otherwise the ignore list becomes the place findings go to die.

None of this requires our tooling. It requires deciding that a test report exists to tell you what you actually know, which is a narrower and more useful thing than what passed.

How the paired run works, the six outcome classes, and the rung ladder are covered on contract testing. The same refusal to score what was not measured, applied to a specification rather than a run, is API governance.