Spec drift is a hard error, and that is the point

The pilot goes fine for three weeks. Then a flow that has never been touched starts producing a screen with an empty section in it, and the investigation ends somewhere unhelpful: a field that the spec says is required has been optional in production since a refactor last spring. Nobody broke anything. The document and the API simply stopped agreeing, some time ago, and nothing until now had a reason to notice.

That is OpenAPI spec drift, and if you are putting an assistant in front of customers it stops being a documentation-hygiene issue and becomes the thing your feature is built on. Your spec is the integration — which is excellent when it is true, and is the whole failure when it is not.

The good news is that this is the most checkable prerequisite in the entire project. You can find out where you stand in an afternoon, before committing to anything.

The short version

1. Drift is any disagreement between your published spec and what the live API does.
2. It is common. An industry study of 650 million calls found roughly three quarters of production APIs varied from their published specs.
3. A chatbot hides drift, because it narrates whatever came back. A layer that renders structure cannot, because it planned the structure in advance.
4. Design for the loud failure: a missing operation should stop startup, a bad argument should be a structured rejection, an unexpected response should be a rejected record rather than a half-drawn screen.
5. If nobody validates your spec against live responses today, that is the work to do first — and it is a week, not a quarter.

What OpenAPI spec drift actually is

Drift is the gap that opens between a description and an implementation when only one of them is under test. An endpoint gains a field, a required parameter quietly becomes optional, an enum grows a value, a 200 starts returning a different envelope on Tuesdays. Each change is small, none of them break a client that was written by a human who reads release notes, and together they make the document a work of historical fiction.

The scale is worth knowing before you assume your own API is fine. APIContext's study OpenAPI Specifications in the Real World analysed 650 million API calls across more than 10,000 endpoints. It found that 75% of production APIs tested had variances to their published OpenAPI specifications, with over half of those specs not updated in the previous six months. Reported by Nordic APIs, checked 21 August 2026.

Three quarters is not a rounding error, and it is not a comment on anybody's engineering discipline. It is what happens when a document is generated once, published, and then serves an audience that mostly does not complain.

Why a chatbot hides drift and a screen exposes it

This is the part missing from everything else written on the subject, and it decides how worried you should be.

A text assistant is tolerant of drift by construction. It calls an operation, gets JSON back, and writes a sentence about it. If the response has an extra field, it ignores it. If a field is missing, it writes a slightly vaguer sentence. If the whole envelope changed shape, it will usually still find something to say. Nothing errors. The output looks fine and is quietly less correct than it was last month, which is the failure mode nobody catches because there is nothing to catch.

An assistant that answers with a screen rather than a paragraph cannot do that. It decided, before the call returned, that this answer is a comparison of three items with four rows — because the response schema said the data would support one. If the data does not, there is no graceful version. There is a comparison with a missing row, or there is a rejection.

Tolerance is not a feature here. An interface that degrades quietly under drift is an interface that tells you nothing is wrong for six months.

So the design question is not "how do we survive drift" but "how do we make drift impossible to ignore." The answer is to fail hard, early, and with a message that names what disagreed.

The three places drift shows up

The operation is gone, or renamed. Somebody deprecated getInvoices in favour of getBillingInvoices and updated the spec. Your allowlist still names the old id. Nothing about this is subtle, and it is the easiest of the three to catch — provided something is looking.

The arguments changed. A parameter became required, or its type narrowed from string to enum. The model builds arguments from the spec's schema, so it will construct a request that the spec permits and the API rejects. This is the one that produces the most confusing debugging session, because the model looks like it is at fault and it is doing exactly what it was told.

The response changed. The most consequential and the most invisible. Fields added are harmless; fields removed, made optional, or retyped are not, because they are what the answer's structure was planned against. An array that used to hold objects with a total now holds objects with amount, and every screen that was going to be a metric row becomes an empty one.

Notice that the first two are about the request half of the spec, which is the half everyone already tests. The third is about the response half, which is the half most contract-testing setups touch least and which matters most once the output is structured.

Failing loudly, on purpose

Here is what that looks like implemented, at three points in a turn.

At startup, a missing operation stops everything. The allowlist is a list of operationId strings. When the spec is parsed, any id on that list with no matching operation is an error, not a shorter tool list:

const missing = options.include.filter(
  (operationId) => !tools.some((tool) => tool.name === operationId),
);
if (missing.length) {
  throw new Error(`OpenAPI operations not found: ${missing.join(", ")}`);
}

A silently shorter tool list is the worst available outcome: the assistant keeps working, gets worse at one category of question, and no alarm goes anywhere. Refusing to boot is rude and correct.

At call time, bad arguments are a structured rejection. The tool's inputSchema is built from the spec's parameters, and the model's arguments are validated against it before any HTTP request happens. A mismatch returns field-level detail rather than a stack trace:

{
  "ok": false,
  "error": "invalid_arguments",
  "fields": ["/accountId is required", "/period must be equal to one of the allowed values"]
}

Those messages are the drift report. If they cluster on one operation, the spec and the API disagree about that operation's inputs, and you now know which.

At render time, an unexpected shape is rejected rather than patched. Every model-emitted record is validated against the wire contract before it reaches the device, and an invalid record is dropped rather than repaired — the argument for rejecting instead of repairing applies to drift for the same reason it applies to model error. A record built from data that did not arrive fails its own required-fields check, and a screen with a missing section is visible in a way that a subtly wrong sentence is not.

None of these three is clever. Together they convert a class of silent wrongness into a class of loud, attributable failure, which is the only trade worth making here.

What this still does not protect you from

Semantic drift. The shape is identical and the meaning changed. status used to mean the payment's status and now means the invoice's. Every schema check passes; every answer is wrong. No amount of validation catches this, and it is the strongest argument for keeping a human in the allowlist review.

Fields that are optional in the spec and always present in staging. Your test data is the happy path. A field that is technically optional will be there in every environment you look at and absent for the one customer whose account was migrated in 2019.

Enums that grow. A new status value is a compatible change to the API team and an unhandled case downstream. Specs are usually updated for this; the update usually lands after the deploy.

A spec that was never true. Drift implies the document was accurate once. Some specs are generated from route decorators and have never described a response body at all. That is not drift, it is absence, and it needs writing rather than reconciling.

How to find out in an afternoon

1. Pick your five most-contacted-about operations. Not the interesting ones — the ones behind the questions your support queue actually receives.
2. Call each one against production with a real account, and save the response.
3. Validate each response against the documented schema. Any JSON Schema validator will do; the point is the diff, not the tool.
4. Score what you find. Extra fields are fine. Missing required fields, retyped fields and changed envelopes are the ones that matter, and they should be zero.
5. If it is not zero, put the check in CI before you build anything else. Validating live responses against the spec on every deploy is a day of work and it is the entire prerequisite.

If four of five pass, you are in good shape and the exceptions are a ticket. If two of five pass, the assistant project is not the next thing you should do — and finding that out this week rather than in week nine of a pilot is worth more than any feature.

Frequently asked questions

What is OpenAPI specification drift?

Drift is any divergence between a published API description and the live implementation it describes — a renamed operation, a parameter that changed requirement, a response field that was removed or retyped. It accumulates because specs are usually generated or written once and then only updated when someone notices, which is rarely.

How do I check my spec matches the live API?

Send real requests to the live API and validate the responses against the documented schemas, rather than against mocks. Mock-based contract tests confirm your test doubles match the spec and say nothing about production. Start with the handful of operations that matter most, then run the check on every deploy.

Do I need contract testing before adding an AI assistant?

If the assistant reads your spec at runtime, yes, and the reason is that the spec stops being documentation and becomes an input. Anything planning behaviour from an inaccurate description will behave inaccurately. A basic live-response validation in CI covers most of the risk and is much smaller than a full contract-testing programme.

Is a generated spec good enough?

For request parameters, usually. For responses, often not — generators frequently emit a bare object type with no properties when the code has no response annotations. That is enough to call an endpoint and not enough to plan an answer from it, so check what your generator produced rather than assuming it described anything.

Where to start

Run the five-operation check this week, before any vendor conversation, including this one. It costs an afternoon and it answers the only question that gates everything downstream.

If the result is clean, the rest is straightforward and we would rather show you the contract and the integration format than describe them. If the result is not clean, fix that first — we will say the same thing on a call, and it is better to hear it now. Either way, hello@uzori.ai will read your spec against a live response and tell you honestly which one you are.